Structural tests (e.g. external SDKs can be imported only from files in client/ dir) should fail with an error message that contains context for coding agents
Garbage collection (refactoring, simplification)
Harness = The collection of items that feed forward (e.g. guides, arch docs) information to the agent and feed back after the agent made changes (e.g. linters, tests) such that the agent can operate in a self-correcting manner with steering from a human
Will harness templates and harnessable topologies become a new abstraction layer in software engineering?
Scaling the unknown: Performance in the AI era (monday.com)
Cost of coding collapsed, cost of running/operating software did not
Everyone expects configurable workflows now
Users have same performance expectations but you no longer have control over the code (if you allow users to vibe code their own solutions on top of your product) => you need a corresponding architecture to ensure perf does not tank
Core vs playground architecture
Core: Small, contained modules that need to be performant and resilient. If someone brings this down, you need to fix it
Playground: Freedom for users to create what they want using agents you provide that implement guardrails
Also measurements/metrics need to adapt. When you do not control flows you need more generic measurements that are consistent across very different workflows (e.g. nr of objects returned for a request)
Support can become a nightmare. Users expect you to support them for workflows they have created by themselves
From Pilot to Impact: How AI is Transforming Large-Scale Engineering (ING)
In 2023 ING started using AI assisted software engineering => 10% productivity gains
When coding is fast/cheap downstream steps (test, deploy) in the SDLC become the bottleneck => we need to speed these up as well
They created an AI platform that standardizes building blocks for agent assisted development => reduce variation
Once you have a great AI platform, adoption is the next big challenge. Ideas:
AI hackathons
Build a community that helps others
SRE teams have been trained and are increasing adoption for teams they are working with
Embed experts/coaches in teams for ~0.5d
Measure the impact of AI, don’t guess
Multiple metrics from different angles are required
Check if all/most metrics tell you the same story before jumping to conclusions
At ING measure productivity gain from AI assistance tools (Copilot) is ~10-20%
Software engineers typically write code ~50% of working hours