Software factory: don't just build software, think about building the factory

Article 18 in the AgentKit v3 onboarding series. About a 12 minute read. This one is more about mindset, with few commands. Read it after article 13 and article 17 and it will all click.

Every time a new buzzword shows up, I get the feeling of "huh, so the thing I have been doing all along has a name?". Prompt engineering, context engineering, harness engineering, and this week software factory. It sounds as grand as an ad for getting rich on autopilot, but this time there is something worth digging into.

What is worth digging into is not "run 20 agents in parallel". It is the question after automation works: how do you know the machine is doing the right thing and getting more efficient over time? The second question is what separates a chain of automations from a factory that runs for the long haul.

1/ What a software factory is

A real factory has inputs, a process with stages, quality control, output that meets a standard, and a feedback loop for improvement. Swap raw materials for requirements, workers for coding agents, the inspection department for tests and evaluation, and finished goods for software, and you get a software factory.

The idea is not new (CI/CD, DevOps, IaC all go this way). What is new is that AI picks up stages that used to need human reasoning: taking an issue, reading the codebase, investigating, writing code, testing, opening a PR.

But a coding agent that can write code is not a factory. A great carpenter does not automatically turn the workshop into a furniture factory.

2/ Six components a factory needs

Drop the tool names, and I think a mature enough system needs at least six things:

1. A structured source of work. A task has the problem, the current state with evidence, the expected result, acceptance criteria, limits it must not cross, the risk level and who can approve.

2. A queue and orchestration. Tasks have states, retries, timeouts, and avoid two workers handling the same one. An in-progress label on GitHub is not enough to prevent race conditions; you need atomic claiming, idempotency, and concurrency control. Inside a single task, the agent can use loops freely.

3. An isolated execution environment. A dedicated worktree plus a container or sandbox. Remember: a worktree separates code changes, it is not a security mechanism. If the agent can still read secrets, SSH keys and the host's network, splitting worktrees solves nothing. Apply least privilege.

4. Verification independent of whoever implements. Don't fully trust tests the agent wrote itself (it can write a function that always returns true and then write a test that matches). Multiple layers: lint, typecheck, unit, integration, dependency scan; then E2E and behavior scenarios; then key invariants (payments must never be charged twice).

5. Human approval by risk. Doc edits can pass the gate automatically. Auth, billing, migrations, production infrastructure need tight control. I call it earned autonomy: at first the agent only proposes, and only after building up a good track record on a narrow type of work does it get to merge by itself, and that right can be revoked.

6. A feedback loop. Record why every failure happened (ambiguous requirements, missing context, wrong model, a bad skill, missing tests, infrastructure errors), then fix the right component. Changing a skill also needs versioning, before/after benchmarks and rollback, otherwise you have a system that makes itself worse on its own.

3/ Other people's stories, read critically

All the numbers below are self-reported by the parties involved or from observational studies. I am only retelling them, not independently verifying:

• Warp (Zach Lloyd, 18/06/2026): engineers shift from building the product to building the system agents use to produce the product, called factory engineering. A sample flow: Triage, Spec, Implement, Review, Verify, CI/CD, Ship, Monitor. Note that Warp sells the Oz platform, so this is also how they position themselves in the market.

• StrongDM (Justin McCarthy, 02/2026): humans neither write nor review code, and responsibility shifts into a verification system built on scenarios stored separately from the codebase and simulations of external services (Digital Twin Universe). The verification part is great, but a simulation does not guarantee 100% resemblance to production, and an LLM-as-judge has its own reliability. The "1,000 USD of tokens per engineer per day" figure is a way of pushing the experiment to the extreme, don't treat it as a recipe.

• Uber (27/08/2026): more than 70% of PRs originate from agents, with more than 3,600 internal skills. "Originate from agents" does not mean no human review is needed; they still describe human review and escalation.

• Harvard study (Fiona Chen, James Stratton, 718 companies, observational): code up about 30%, but the time from opening a PR to merge up about 49%. An observational study does not prove causation. The implication: the bottleneck shifts from writing code to verification.

• Microsoft Research / CRBugBench (10/2026): the best review agent caught 71.9% of confirmed bugs in its top five priority comments. Don't confuse AI review with a quality guarantee; two agents that misunderstand the requirement in the same way will still agree with each other.

• Swan / Cygnet: the best lesson is a system that knows when to stop. When the step that produces verification evidence fails, it hands back to an engineer instead of insisting on "task completed".

• Anthropic (13/10/2026): merges into the monorepo went from about 2,500 to over 30,000 a day, source control became the bottleneck and they had to build their own system (Scion). That is a special workload, not a reason for you to drop Git.

4/ Pragmatic metrics

• Cost per accepted task: the total real cost to get one piece of work verified and accepted, not tokens, not lines of code. My hypothetical example: factory A costs 2 USD per attempt and is right 50% of the time, so 4 USD per good result; factory B costs 3 USD and is right 90% of the time, so about 3.33 USD. The cheapest model does not necessarily give the cheapest system.

• Human intervention rate: how many times you had to step in. Don't optimize blindly: reducing interventions by dropping verification is reducing quality.

5/ Applying it yourself, in small steps

Don't start with 15 microservices to fix a README typo. Start with:

one GitHub repo + one cloud coding agent + one queue + one trustworthy test suite + one reviewer.

1. Pick one small, low-risk type of work (docs, small bugs).

2. Write a clear enough issue, with acceptance criteria (see how AgentKit creates issues in article 13).

3. Set up an isolated execution environment per article 17 or Cloud Harness in article 14.

4. Tighten verification with /ak-test: audit first, then optimize.

5. Measure cost per accepted task and the number of interventions. Only once you have numbers, look for the bottleneck.

6/ Status in the AgentKit ecosystem

The five-layer architecture I want to head toward (work, knowledge and process, execution, verification and governance, learning from operations) is a direction, not a claim that every layer is finished.

My disclaimer: I have a product interest in this topic because I sell related tools, so feel free to push back hard.

Wrap-up

1/ A software factory is not a contest of who runs the most agents or burns the most tokens.

2/ The bottleneck shifts from writing code to verification. Don't treat AI review as a quality guarantee.

3/ A good factory knows when to stop and hand back to a human, with evidence.

4/ Start small, measure cost per accepted task, then expand. And don't get so busy building the factory that you forget what you meant to produce.

Sources

• Software factory: Huh, so what I have been doing all along has a name? (15/10/2026), the original post has full links to Warp, StrongDM, Uber, Swan, Anthropic, Microsoft Research and the Harvard study.