The engineering behind a dependable software factory
Reliable agentic delivery is a systems engineering challenge: reproducible execution environments, durable orchestration, controlled tool access and verification-driven workflows — integrated with the software delivery practices you already use.
The capability splits into two phases: Phase 1 executes ready work; Phase 2 turns authorised events into ready work — an optional upstream layer. Phase 1 runs without it.
Two capability phases
One execution contract; an optional upstream layer
Phase 1 starts when a task is ready to implement: durable orchestration schedules it into an isolated worker, captures verification evidence and returns an artefact for review. Phase 2 is an optional upstream layer — it listens for relevant events, interprets them under policy and produces tasks or decisions for Phase 1. A task can still enter Phase 1 directly. Data flows upstream-down (Phase 2 → Phase 1) even though phase 2 is installed second.
Phase 1 — always available
Phase 1: the execution contract
A ready task is work that is already specified and authorised — from a queue, an issue marked ready, explicit dispatch or an agreed schedule. Its contract is conceptual, agreed per engagement, not a fixed schema: the task names its objective and the permitted workspace or branch scope, carries acceptance criteria and constraints, records its authorisation state and the worker class it may run on, and keeps provenance for who or what approved it.
Ready task → queue → isolated agents → verify → reviewable output
Direct ready-task entry needs no Phase 2: an issue status change or queue marker can dispatch prepared work. A webhook that only relays already-approved tasks is therefore part of Phase-1 dispatch, not Phase 2. Authorisation stays where the provenance says it is; production release follows the project's own approved process.
Phase 2 — optional
Phase 2: event to decision to work
What Phase 2 really adds is a second workflow engine — an event-driven one. It receives signals from authorised sources (issue-tracker changes, chat messages, email and other agreed services) and runs a defined workflow over them under policy: normalise, deduplicate, enrich with permitted context, interpret, and decide the authorised next step.
Filter / dedupe → enrich → interpret / triage → decide: ignore / record / notify / clarify / plan / approve / dispatch
That workflow engine includes
- Source connectors and webhook or polling adapters
- Source authentication, input validation, rate limiting
- Event envelopes with immutable source references; idempotent dedupe
- Context resolution under service identity and access policy
- Rules, correlation and policy-constrained interpretation
- Decision routing and the human checkpoint where policy requires it
- Ready-task writing into the Phase-1 interface
- Event-to-task lineage in audit and observability
Crossing the boundary: an authorised, ready-to-execute task — with a human decision or approval point between interpretation and admission wherever the configured policy requires one. Evidence and approvals persist across both phases.
One event, end to end
- A stakeholder posts a comment on an authorised issue: "The acceptance criteria also need to cover offline retry."
- Phase 2 receives and deduplicates the comment event.
- It retrieves the issue context, identifies a proposed requirements change and checks configured permissions.
- The changed requirement is flagged for product-owner confirmation rather than silently rewriting accepted scope.
- After approval, Phase 2 updates the planned task and acceptance checks and moves the task to a ready state.
- Phase 1 consumes the ready task, provisions an isolated worker, runs the tests and returns a reviewable change with traceable evidence.
- The source comment and the approval both appear in provenance — and no unrestricted release occurs.
Other signals, same shape: an email received → triage → proposal or clarification; an issue unblocked → re-evaluate dependencies → dispatch if ready; a chat mention → categorise, notify or seek approval.
Security and operations boundary
Messages, comments and emails are untrusted input: they never override policies, tool permissions or scope. The architecture treats them with least privilege, sender and tenant validation, prompt-injection resistance, defined retention and privacy, replay and dedupe handling, out-of-order tolerance, bounded retries against noisy loops and full audit of event-to-task lineage.
Reference architecture
From request to reviewed output
Work enters from your issue tracker, is held by durable orchestration, executes on isolated workers against your repository and CI, and ships only through review — while observability, identity and secrets, policy, providers and the scheduler cross-cut every stage.
Phase 1 in this diagram starts at ready-task admission (left) and ends at the reviewed, promoted artefact. Phase 2, when present, sits upstream of this flow and hands it authorised tasks — build Phase 1 first; Phase 2 sits upstream when added.
Operating principles
Deterministic code enforces; LLMs reason
Deterministic code governs orchestration, workflow transitions, resource limits, retries, checkpoints, timeouts, access control and verification gates.
LLMs handle reasoning, interpretation and code and document generation — inside that system. The system, not the model, decides what runs, where, and what happens on failure. Critical invariants are never delegated to prompts.
-
Per-workload isolation
Every task runs in its own disposable environment, so failures and compromise are contained to that workload — effectiveness depends on the execution and host/network/access boundaries.
-
Durable, recoverable workflows
Progress is checkpointed to durable storage, so in-flight work can resume after restarts — where implemented; not every session or external side effect is replayable.
-
Evidence before promotion
Outputs move forward on recorded test, review and security evidence — never on a claim alone.
-
Inspectable and operable
Every workflow exposes its state, logs and costs, so your engineers can debug and operate it without us.
-
Pragmatic provider flexibility
Models, harnesses and runtimes stay replaceable where practical, without pretending every option is interchangeable.
Illustrative Kubernetes deployment
What lives where — and what persists
One representative placement of the factory. Kubernetes makes the distinction concrete (durable control-plane services beside disposable workers), but the offer stays technology-flexible — the boundaries and lifecycle are what repeat. Any persistent caches or volumes get an explicit scope and retention policy; a disposable runtime does not automatically provide durable agent memory or replay safety.
Solid rectangles mark the customer-controlled boundary; dashed rectangles mark external or customer-run services; dotted boundaries mark disposable workers. Worker teardown never deletes the retained task and evidence records, and a pool is bounded by suitable nodes — CPU generation, RAM, storage, network and provider quotas still constrain useful concurrency.
Components
Who does what, and where its boundaries sit
Two distinctions this table rests on: workflow orchestration coordinates tasks across a system, while the agent harness runs one agent’s tools and context inside a bounded environment — and execution compute is a separate cost centre from model inference.
| Component | Purpose and boundary |
|---|---|
| Ready-task admission (Phase 1) | Validates and queues prepared tasks — dispatch by queue, issue-tracker status, explicit dispatch or schedule; no upstream interpretation required. |
| Event intake, interpretation and work preparation (Phase 2, optional) | Processes authorised signals — capture, correlation, classification, clarification — and creates or updates ready work under policy. |
| Scheduler and capacity controls | Admit suitable work within resource, concurrency and provider limits; queued work waits. |
| Execution environment | Reproducible tools, workspace, test services and isolation for one workload. |
| Agent runtime and harness | Give an agent its tools and context inside the configured execution and permission boundaries. |
| Model providers | Supply inference; remote APIs and subscriptions differ from local inference in cost and limits. |
| Repository and CI integration | Materialise changes, run relevant checks and create reviewable outputs under branch and release rules. |
| Verification and approvals | Record tests, review, security and UAT evidence — and the explicit human decisions between stages. |
| Identity, secrets and policy | Limit authority and access; define network and tool boundaries independently of prompts. (Shared foundation: Phase 2 reuses it under the same policy.) |
| Durable records and artefacts | Keep task history, approved outputs, diagnostic evidence and event-to-task lineage beyond worker destruction. (Shared foundation.) |
| Operator experience | Expose status, exceptions, evidence and diagnostics, with actions sized to authority. (Shared foundation.) |
Integration names on this page are representative, not partnerships: model and harness services stay replaceable where practical, and provider terms remain a deployment consideration rather than something this page waives.
Phase-1 worker lifecycle
Provision, execute, verify, export, destroy
Provision → Execute → Verify →
Export → Destroy
Every stage is bounded by timeouts. When a step fails or hangs, the workflow retries from its last checkpoint or surfaces the failure for human attention, and the worker is destroyed rather than left in an unknown state. Recovery resumes durable workflow state where supported — not every in-memory session or external side effect can be replayed, so retries are bounded and outcomes are recorded.
| Stage | What happens | Failure and decision path |
|---|---|---|
| Scope and admit | A known task with acceptance criteria, permissions and a resource class. | Clarify, reject or escalate ambiguous or out-of-scope work. |
| Queue and provision | The scheduler allocates a clean environment with scoped credentials. | Bounded retry or a visible operational exception — workers do not accumulate. |
| Execute | The agent working through the bounded task with its configured tools. | Checkpoints where supported; timeouts, quota and tool failures surface. |
| Verify | The agreed checks run in the configured worker and CI stages. | Correct and retry within limits, or escalate — a failed check never silently passes. |
| Review and authorise | Changes and evidence meet the configured decision boundary. | A reviewer may request changes or decline; release authority stays with people. |
| Export and retain | Changes, reports, logs and evidence move to durable destinations. | If export fails, the exception surfaces before cleanup loses diagnostics. |
| Destroy | Task access is revoked or expires; the worker environment is disposed of. | Cleanup failures are reported; retention policy decides what survives. |
Worked example
One task, end to end
Illustrative — a representative walkthrough, not a recorded run.
The task and its boundaries
A bounded dependency update: raise one library dependency, keep the public interface unchanged, and let the repository’s existing checks confirm behaviour.
- One repository, one change type, one bounded task.
- A scoped, short-lived access token limited to that repository.
- A defined allowlist of commands; a clean workspace provisioned for this task alone.
Execution and verification
The worker provisions a clean environment, updates the dependency and the dependent imports, and applies the library’s documented migration, recording progress as it goes.
The repository’s configured checks run — build, the relevant tests, then a review build. One check fails on the first pass: the update exposes a deprecation the pipeline is set to reject. The workflow returns to bounded correction, the fix lands, and the checks are re-run to green.
The human decision
People set the acceptance criteria before work starts, and a reviewer decides what to accept afterwards. The reviewer sees the change, the check results and the task record — no change proceeds without that decision.
Output, and what it does not establish
A reviewable change with recorded check results and task evidence, waiting in the repository’s normal review process. Production release, where it applies, follows the project’s own approved release process.
This is an illustrative walkthrough, not a recorded run. It explains the mechanism — what goes in, what is checked, where people decide — and establishes nothing about measured speed, success rates or live deployments.
Boundary summary
- Reviewed output, not an automatic production release.
- People decide: acceptance criteria, approval, release.
- Bounded correction on a failed check; escalation beyond limits.
- Evidence exported before teardown.
Operator experience
The questions an engineer asks daily
- What is running, and where is it in its lifecycle?
- What is blocked, and which limit is binding?
- Which check failed, and what did it fail on?
- What evidence is ready for review?
- What can I retry, and what needs a human decision?
The operator interface exposes status, exceptions, task records, diagnostics, quotas and the configured actions — with authority to match the role. Logs and diagnostics stay with the durable task records, so an engineer can answer these questions after a worker is gone. Upgrades, image lifecycle and cost governance are operated through the same views.
Workload-aware compute
Right-size every worker
Resource requests are scheduler placements — not dedicated cores or assured throughput. More concurrent workers does not automatically mean more completed work.
Heavyweight: coding/build workers
16 GB / ~1 CPU core-equivalent baseline
Code changes, builds, tests and browser UAT need substantial environments. These are minimum planning profiles — parallel compilation, browser stacks and large monorepos can need 2–4+ cores and more RAM.
Lightweight: reasoning workers
2 GB / ~0.25 CPU core-equivalent
Research, documentation, planning, review text, issue triage and specification drafting run on small runtimes when inference is remote — no substantial local build or browser processing. Some research browser jobs need more.
Scheduling happens by resource class: active workers claim the RAM and CPU they request, queued work waits, bursts are bounded, and storage and network are budgeted alongside. Because code-writing and testing workers need substantially more than research and documentation workers, mixed placement makes far better use of the same server.
Benchmark representative builds and full test runs in reproducible environments, including test services, cache effects, peak memory and concurrent contention. Use those measurements and queue targets to choose worker classes and headroom; remeasure after material changes to tests or environments.
Disposable isolation
Every workload gets its own world
Clean starting environments, reduced dependency drift, containment of malicious package installs, lower persistence of compromised state, scoped secrets, bounded network access and less interference between workstreams.
* Isolation reduces blast radius and how long contamination persists — but it cannot by itself prevent malicious dependencies, exfiltration or supply-chain attacks. Security depends on sandbox boundaries, scoped credentials, trusted images, dependency policy and verification.
Verification and commissioning
Verified through agreed gates
Automated verification raises confidence; it does not guarantee correctness. Output promotion stays a controlled, auditable step.
Before operating the chosen workflow, establish reproducible builds, reliable pipelines and tests of critical behaviours and failure paths. Address blocking gaps and agree review responsibilities; coverage percentages alone cannot establish readiness. Establish workflow readiness.
- Automated tests
- Static analysis
- Code review
- Security checks
- Functional and UAT validation
- Controlled output promotion
Commissioning goes further: realistic customer workflows run against agreed acceptance criteria, including failure and recovery demonstrations, and handover artefacts record what was demonstrated rather than promised.
Provider flexibility
Replaceable parts, wherever practical
Model, harness and runtime choices are kept replaceable where practical — so a single vendor decision never locks in the whole factory.
Some integrations resist interchange; we design against the exceptions deliberately rather than pretending all options are interchangeable.
Operational enablement
Your team runs it
Training for engineers and operators, diagnostics, upgrades, security, incident response and cost governance — the practices that keep a factory dependable after handover.
- Using, supervising and debugging factory workflows
- Diagnostics, retries and recovery
- Upgrades and image lifecycle
- Security posture and incident response
- Quota monitoring and cost governance
- Playbooks, procedures and handover documentation
Capability graph
Explore capability relationships
The full matrix maps every building block to the outcome it enables — open it to browse all 8 by 8 relationships.
Explore capability relationships
| Outcomes | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Economics Sustained workloads amortise owned capacity instead of per-minute bills. See the sizing model → | Scalability More parallel workstreams within known resource budgets. | Reliability Runs start from known state and recover cleanly from interruption. | Supply-chain resilience Compromised residue can be discarded or caught — not fully prevented. | Autonomy Suitable workflows progress with less continuous supervision — never without governance. | Sovereignty Execution, data and operational control stay in-house, subject to configured external dependencies. | Safety and containment Bounded blast radius and contained interference when something misbehaves. | Verified outcomes Changes carry evidence from tests, reviews and acceptance gates. | ||
| Building blocks | Customer-owned infrastructure Run execution environments on infrastructure you own or control — hardware, cloud tenancy or both. | Reusing or owning hardware can reduce active-minute cost when utilisation and operating overhead justify it | Customer-owned infrastructure does not directly support Scalability | Customer-owned infrastructure does not directly support Reliability | Customer-owned infrastructure does not directly support Supply-chain resilience | Customer-owned infrastructure does not directly support Autonomy | Execution resources and deployment location can remain under the customer's control, subject to configured external dependencies | Customer-owned infrastructure does not directly support Safety and containment | Customer-owned infrastructure does not directly support Verified outcomes |
| Capacity engineering Workload-aware scheduling and right-sized compute for each class of worker. | Matching RAM, CPU, power and utilisation to actual workloads avoids unnecessary spend | Scheduling limits and resource budgets increase parallel execution safely within known constraints | Capacity engineering does not directly support Reliability | Capacity engineering does not directly support Supply-chain resilience | Capacity engineering does not directly support Autonomy | Capacity engineering does not directly support Sovereignty | Capacity engineering does not directly support Safety and containment | Capacity engineering does not directly support Verified outcomes | |
| Disposable isolation A fresh, disposable execution environment for every workload, destroyed after use. | Disposable isolation does not directly support Economics | Separate workstreams avoid shared workspaces and many common port/resource collisions, within host capacity | Workloads start in reproducible environments instead of inheriting leftovers from prior runs | Untrusted packages, installations and other state from one run can be destroyed instead of persisting into later jobs; isolation does not prevent initial compromise | Disposable isolation does not directly support Autonomy | Disposable isolation does not directly support Sovereignty | Per-task boundaries reduce cross-workload interference and can restrict the spread of a compromised execution environment — including supply-chain attacks — provided sandbox boundaries are properly enforced | Disposable isolation does not directly support Verified outcomes | |
| Durable orchestration Durable, deterministic control of workflow state, retries, limits and timeouts. | Durable orchestration does not directly support Economics | Durable orchestration does not directly support Scalability | Persistent workflow state helps recover and diagnose interruptions and failures | Durable orchestration does not directly support Supply-chain resilience | Checkpoints, timeouts, retries and scheduled recovery reduce the need for constant human intervention | Durable orchestration does not directly support Sovereignty | Durable orchestration does not directly support Safety and containment | Durable orchestration does not directly support Verified outcomes | |
| Verification gates Automated and human gates that evidence must pass before promotion. | Verification gates does not directly support Economics | Verification gates does not directly support Scalability | Verification gates does not directly support Reliability | Dependency checks, scanning, policy enforcement and reviews can detect some suspicious dependencies or outputs before promotion; none is a guarantee | Automated quality gates reduce the number of routine decisions requiring immediate human attention | Verification gates does not directly support Sovereignty | Verification gates does not directly support Safety and containment | Tests, reviews and other gates provide evidence before artefacts are accepted | |
| Training and handover Training, documentation and handover so your team operates the factory. | Training and handover does not directly support Economics | Training and handover does not directly support Scalability | Training and handover does not directly support Reliability | Training and handover does not directly support Supply-chain resilience | Team capability makes ongoing operations, troubleshooting and evolution less dependent on external assistance | Documented handover and practical training transfer operational knowledge to the customer | Training and handover does not directly support Safety and containment | Training and handover does not directly support Verified outcomes | |
| Scoped permissions Least-privilege credentials, egress rules and tool permissions per task. | Scoped permissions does not directly support Economics | Scoped permissions does not directly support Scalability | Scoped permissions does not directly support Reliability | Scoped permissions does not directly support Supply-chain resilience | Scoped permissions does not directly support Autonomy | Customer-controlled identities, secrets and access policies help retain operational control over the execution environment | Least-privilege credentials, network egress rules and tool permissions help constrain damage even if a task or dependency misbehaves | Scoped permissions does not directly support Verified outcomes | |
| Provider flexibility Model, harness and runtime choices kept replaceable where practical. | Provider flexibility does not directly support Economics | Provider flexibility does not directly support Scalability | Provider flexibility does not directly support Reliability | Provider flexibility does not directly support Supply-chain resilience | Provider flexibility does not directly support Autonomy | Replaceable model and runtime interfaces can reduce lock-in where compatibility permits | Provider flexibility does not directly support Safety and containment | Provider flexibility does not directly support Verified outcomes | |