Implementing an AI Factory for Autonomous Code Development

Executive summary

An effective AI Factory for autonomous code development is not a single coding model or a single "agent." It is a governed delivery system that repeatedly plans, retrieves context, edits code, executes tools, runs tests, evaluates outcomes, enforces policy, and routes work for human approval when risk is high. The research and product landscape converge on that point: real-world software tasks require repository navigation, execution environments, long-context or retrieval support, iterative repair, and evaluation beyond toy code generation.

The most reliable implementation pattern today is progressive autonomy. Start with bounded tasks such as test generation, dependency updates, doc maintenance, and narrowly scoped bug fixes inside ephemeral sandboxes. Only then expand to cross-file changes, cross-repo workflows, and deployment automation. This recommendation follows both the current capability evidence from software-engineering benchmarks and the risk guidance from the NIST AI RMF and secure software guidance in NIST SSDF.

The highest-value design choice is to separate the factory into three control loops:

1. Build loop: issue intake, repository context, planning, code editing, execution, testing, PR generation.
2. Safety loop: permissions, sandboxing, policy checks, supply-chain controls, guardrails, auditability.
3. Learning loop: offline benchmarks, regression suites, trace review, human feedback, and production evaluation.

This architecture is substantially more robust than "just let the agent run," and it aligns with guidance from NIST, the OWASP LLM Top 10, supply-chain standards such as SLSA, and modern agent evaluation systems from managed platforms and open tooling.

A practical default, if the stack is unspecified, is:

• Execution substrate: Kubernetes for online and mixed workloads; Slurm for HPC-style batch or training-heavy environments.
• Durable orchestration: Temporal for long-running autonomous workflows; CI/CD triggers through GitHub Actions or GitLab; GitOps deployment via Argo CD.
• Model serving: managed model access for fast adoption, or vLLM/KServe for self-hosted control.
• Evaluation and observability: OpenTelemetry as the canonical telemetry layer, with one evaluation/trace platform on top.
• Security and governance: NIST AI RMF + SSDF, SBOMs, artifact signing, admission policy, explicit human approvals on destructive actions.
• Human-in-the-loop: mandatory for prod deploys, secret access, privileged tool calls, schema migrations, and any action that changes infrastructure or customer data.

Unspecified in any greenfield engagement, and therefore material design constraints that remain open, are: regulatory regime, source-control provider, hosting environment, repo size distribution, programming languages, deployment targets, latency/SLA requirements, and whether model inference must remain air-gapped. Those choices materially affect the final platform selection.

Scope and design principles

For this tutorial, an AI Factory means a repeatable platform that accepts engineering work items and converts them into governed code changes and deployment proposals through a chain of machine and human decisions. That definition is consistent with the evolution from static code generation toward tool-using and software-executing agents documented in Toolformer, AutoGen, SWE-agent, OpenHands, and cloud "agent factory" products from Microsoft, Amazon Web Services, and Google Cloud.

The design principles should be:

• Bound autonomy instead of unrestricted autonomy.
• Hermetic execution instead of direct execution on developer laptops or shared hosts.
• Eval-first deployment instead of benchmark-first marketing.
• Policy-as-code instead of manual review-only governance.
• Artifact lineage instead of unverifiable model/tool outputs.
• One canonical telemetry plane instead of fragmented logs.

The reason to favor bounded autonomy is empirical as well as governance-driven. Original SWE-bench results showed that repository issue resolution is far harder than standalone coding tasks, while later systems improved through tool access, interfaces, retrieval, and iterative repair. At the same time, newer work such as Agentless shows that simpler, more interpretable scaffolds can be competitive and cheaper than deeply nested agent loops.

A second principle is to optimize for service quality, not raw benchmark scores. DORA's software delivery metrics emphasize throughput and instability together, and its 2024 findings note that AI assistance can improve individual productivity while still harming delivery stability and throughput if engineering fundamentals are weak. In other words, autonomous code development needs the same operational rigor as any other production delivery system.

Reference architecture and functionality overview

The architecture below shows the recommended control plane and data plane split.

Work Intake (Jira / GitHub / GitLab / Ticket)
   ↓
Task Router        →  triage, scope, risk class
   ↓
Planner            →  issue decomposition
   ↓
Context Layer      →  repo index, docs, tests, dependency graph
   ↓
Agent Runtime      →  sandboxed workspace
   ↓
Tool Layer         →  git, build, test, linters, search, package mgrs, APIs
   ↓
Model Layer        →  reasoning model, code model, judge model
   ↓
Evaluation Layer   →  unit/integration tests, static checks,
                       trajectory eval, quality eval
   ↓
Policy Gate        →  approve   →  PR / Change Proposal
                   ↘  needs review → Human Review → PR
   ↓
CI/CD + GitOps Deploy   →   Production Monitoring
   ↓
Trace Store + Eval Dataset + Feedback   →   back into Planner & Evaluation

This architecture is consistent with the capabilities required by SWE-bench-style tasks, the repository-navigation and execution model used by SWE-agent, and the current product direction of managed agent platforms that combine orchestration, model access, monitoring, and governance.

A second view focuses on artifacts and lineage, which is essential for auditability and rollback. Work items produce plans; plans launch agent runs; agent runs invoke tool calls and generate code diffs; diffs are validated by test runs and scored by eval results; eval results gate approvals; approvals authorize pull requests; pull requests promote deployments; deployments may trigger incidents; incidents inform feedback items; feedback items update the eval dataset. That lineage model maps well to MLflow's registry and tracing concepts, OpenTelemetry traces, CycloneDX dependency graphs, and signed provenance/attestation patterns from SLSA and Sigstore.

Core components and their relationships

• Compute substrate: Kubernetes is portable and extensible; Slurm is strong for batch/HPC; AWS ParallelCluster simplifies managed HPC on AWS.
• Durable orchestration: Temporal provides durable execution and replay; Ray scales distributed Python/AI workloads. Temporal requires workflow discipline; Ray is not a full business-process engine.
• Model serving: vLLM, KServe, Ray Serve, and Triton give strong control over latency, batching, data locality, and model choice — but require GPU ops, capacity planning, and model eval ownership.
• Managed agent/model platform: Bedrock AgentCore, Microsoft Foundry, and Vertex AI offer faster adoption and integrated governance/evals/observability at the cost of platform lock-in.
• Data and evaluation pipelines: Dagster, Airflow, Kubeflow Pipelines, and Ray Data provide strong scheduling, lineage, portability, and repeatability.
• CI/CD for code and infra: GitHub Actions, GitLab CI/CD, and Argo CD provide native SCM integration; GitOps improves auditable deploys and drift detection.
• Registry and lineage: MLflow Model Registry centralizes lineage, versioning, aliases, and metadata.
• Observability and evaluation: OpenTelemetry + Prometheus as the OSS baseline; LangSmith, Arize Phoenix, Datadog, MLflow 3, or Foundry tracing as managed overlays. Pick one system of record.
• Security and policy: OPA, Kyverno, SLSA, Sigstore, and CycloneDX provide strong preventive controls and artifact trust.
• Human-in-the-loop: LangGraph/LangChain HITL, AutoGen human modes, and PR review keep autonomy productive without giving away control.

Functionality overview

A production-capable AI Factory should deliver the following capabilities, in this order of importance:

1. Work intake and scoping: classify an issue by task type, repo(s), blast radius, secrets exposure, and required permissions.
2. Context assembly: retrieve relevant code, docs, tests, build files, dependency graphs, and recent PR history.
3. Sandboxed execution: launch an ephemeral workspace with pinned dependencies and constrained credentials.
4. Tool-mediated development: let the system edit files, run builds/tests, inspect failures, and iterate.
5. Outcome evaluation: score both final output and the path taken — tests passed, tool usage correctness, instruction adherence, safety, and regression checks.
6. Approval and packaging: produce a PR or change set with diffs, rationale, traces, SBOM/provenance, and risk annotations.
7. Deployment and monitoring: promote only through existing CI/CD gates; collect traces, latency, safety, and cost signals.
8. Continuous learning: fold incidents, rejected PRs, and human corrections back into eval datasets.

The most important insight is that evaluation must happen at both the final-output level and the trajectory level. Vertex AI and Microsoft Foundry explicitly expose agent-specific evaluators such as tool-call accuracy, tool selection, tool-input correctness, task completion, and task adherence, which is exactly what autonomous coding systems need. Relying only on "did tests pass?" produces fragile systems.

Implementation tutorial

The sequence below is the recommended implementation path for an engineering team starting from an unspecified stack.

1.  Define scope and policy
2.  Build the eval harness
3.  Provision sandbox runtime
4.  Add repo context and indexing
5.  Integrate model/tool orchestration
6.  Wire CI/CD and PR flow
7.  Add observability and trace storage
8.  Add policy gates and HITL
9.  Pilot bounded tasks
10. Expand to autonomous PRs
11. Measure, tune, and scale

This sequence is intentionally eval-first and policy-first because both NIST guidance and modern managed/OSS agent stacks emphasize that trustworthy deployment requires testing, monitoring, and governance to be built into the lifecycle rather than added later.

Step sequence

Define the operating envelope. Decide what the system is allowed to do in the first release. If unspecified, the safest initial envelope is: single repository, no production secrets, no infrastructure mutation, no direct deploys, no database writes, and only PR creation with human review. Map those boundaries to your acceptable-use policy and threat model.

Build the evaluation harness before the factory. Create a task set from your real repository work: small bug fixes, test additions, dead-code cleanup, doc updates, typed refactors, and dependency bumps. For each task, store expected test outcomes, forbidden files, allowed tools, success rubrics, and human review notes. Use both offline benchmark tasks and company-specific tasks; public evals alone are not enough.

Provision hermetic sandboxes. Run every autonomous task in ephemeral workspaces with pinned dependencies, isolated credentials, network egress controls, and short-lived identity. Kubernetes, Docker-based sandboxes, or managed agent runtimes all work; the key is reproducibility and revocability. OpenHands, Bedrock AgentCore, and GitHub Copilot cloud agent all reflect this pattern.

Implement repository context retrieval. Index source files, symbols, build manifests, test metadata, dependency graphs, and recent PR discussions. Prefer hierarchical localization over naïve full-repo stuffing. Agentless's file → class/function → edit-location strategy is a good design reference because it reduces search cost and makes edits more interpretable.

Add the tool layer. The minimum tool set is: git operations, grep/semantic code search, file editing, build, unit tests, selective integration tests, static analysis, dependency manager, and a read-only documentation/query layer. Tool use is not optional in real coding workflows; Toolformer, AutoGen, MCP, and OpenAI's tool-calling flow all support the general lesson that external tools are central to reliable autonomy.

Choose an orchestration pattern. For deterministic, long-running flows with retries, timers, approvals, and durable state, prefer Temporal. For large parallel AI/data workloads, prefer Ray. For simple early pilots, repo-native orchestration through GitHub Actions or GitLab CI plus a lightweight controller is often enough. Avoid overcommitting to a complex multi-agent design until you have evidence that single-agent or planner-executor patterns fail on your workload.

Integrate model access. If speed of adoption matters most, use a managed model plane. If data locality and cost control matter more, self-host with vLLM/KServe or equivalent on Kubernetes. In either case, separate at least three model roles: planner/reasoner, code editor/repair model, and evaluator/judge model. This separation is a practical implementation of the iterative feedback patterns shown in Reflexion and Self-Refine.

Wire CI/CD and GitOps promotion. Every agent output should become a normal engineering artifact: branch, commit, PR, pipeline run, signed artifact, deploy candidate. Use existing CI/CD for enforcement instead of inventing a separate deploy path. For infrastructure or app rollout, GitOps tooling such as Argo CD adds drift detection and auditable reconciliation.

Add full-fidelity observability. Emit traces, spans, tool calls, latency, token usage, cost, and test results to a single system of record. OpenTelemetry is the safest baseline because it is vendor-neutral and already extending GenAI/agent semantic conventions. Then pick one evaluation UI layer — LangSmith, Phoenix, Datadog, MLflow 3, or a cloud-native equivalent.

Add policy gates and human approvals. Gate at least these actions: secret access, production deployment, schema changes, package publication, cross-repo writes, high-cost model calls, and any tool that mutates infrastructure or customer data. LangGraph/LangChain HITL and AutoGen's human modes are good implementation references; OPA/Kyverno are good enforcement references for platform policy.

Pilot narrow tasks, then widen. Start with maintenance and low-blast-radius jobs. Once success rates, review acceptance rates, stability, and rollback rates are acceptable, expand into autonomous PR generation for bugs and medium-scope refactors. Do not jump directly to autonomous deploys. DORA's findings on stability and throughput justify this caution.

Best-practice baseline configurations

Common failure modes and mitigations

The list below emphasizes the failure modes that matter most for autonomous code development, not just generic chatbots.

• Prompt injection / tool hijack: retrieved issue text, docs, PR comments, or external pages alter agent behavior and trigger unsafe tool calls. Treat repository and external text as untrusted; isolate tool permissions; add prompt-attack filtering; require approval on sensitive tools; lock egress.
• Insecure output handling: model-generated shell/code/config is executed directly and compromises systems. Never execute raw output without sandboxing, static checks, and a policy gate; prefer tool APIs over shell free-form output.
• Repository mislocalization: agent changes the wrong file, fixes one symptom, or introduces regressions. Use hierarchical localization, dependency graph context, selective tests, candidate patch ranking, and trajectory review.
• Eval blindness: tests pass but behavior is wrong, insecure, or instruction-violating. Add task-completion, task-adherence, tool-call-accuracy, groundedness, and human review metrics.
• Benchmark overfitting: internal system looks good on public benchmarks but fails on company repos. Maintain a private eval suite sampled from production logs and real tasks; test migrations and regressions continuously.
• Supply-chain compromise: untrusted dependencies, model artifacts, images, or CI inputs contaminate the path to production. Adopt SLSA, SBOMs, signed artifacts, attestation verification, and dependency scanning in CI.
• Excessive privilege / non-human identity sprawl: agent gets wide repo, cloud, or database access and abuses it intentionally or accidentally. Use short-lived identities, scoped service accounts, policy-based allowlists, separate identities per agent role, and audit logs.
• Telemetry leakage: prompts, code, secrets, or PII end up in traces/logs. Redact sensitive content before emission; treat traces as production telemetry; restrict retention and access.
• Runaway cost / latency: multi-step agents burn tokens and GPU time without converging. Apply budgets, max-step caps, cheaper judge models, caching, early stopping, per-task routing, and dynamic batching.
• Stability collapse in production: more AI-generated changes ship, but recovery time and rework worsen. Measure DORA metrics, limit autonomy to proven task classes, and keep small batch sizes and robust testing.

The practical mitigation pattern is layered: sandbox + policy + evaluation + approval + provenance. No individual control is sufficient on its own. That is the common thread across NIST AI RMF, OWASP, SLSA, and current managed agent platforms with built-in governance features.

Sourcing options and configuration baselines

Enterprise vendors, open source, and home-grown approaches

The core pattern is stable across layers: enterprise platforms reduce integration burden; open-source stacks maximize control and portability; home-grown approaches work only when the organization is capable of owning evaluation, security, and operations as first-class platform functions.

• Agent platform — enterprise: Microsoft Foundry, Bedrock AgentCore, Vertex AI, GitHub Copilot cloud agent, Databricks agent products. Open source: OpenHands, AutoGen, LangGraph-based stacks. Recommendation: buy for speed, build only where you need control or differentiation.
• Compute / scheduling — Kubernetes, Slurm, Ray/KubeRay (OSS); managed cloud runtimes, enterprise Kubernetes distros, AWS ParallelCluster. Recommendation: standardize on Kubernetes unless batch/HPC dominates.
• Model hosting — vLLM, KServe, Triton, Ray Serve (OSS); managed model catalogs and serverless endpoints. Recommendation: self-host only when data residency, latency, or cost justify it.
• CI/CD — GitHub Actions, GitLab CI/CD, Jenkins, Argo CD. Recommendation: keep native SCM CI/CD; do not invent a second pipeline.
• Registry / lineage — MLflow, Kubeflow Model Registry. Recommendation: use MLflow-class lineage unless a cloud platform already covers it well.
• Observability / eval — Phoenix, OpenTelemetry + Prometheus/Grafana (OSS); LangSmith, Datadog, Foundry, Vertex eval, MLflow/Databricks. Recommendation: one canonical trace/eval platform only.
• Security / governance — OPA, Kyverno, Sigstore, CycloneDX, SLSA (open standards); Bedrock Guardrails, Foundry control plane, Vertex governance, Unity Catalog AI governance. Recommendation: prefer open standards even if you use vendor controls.
• HITL — LangGraph/LangChain HITL, AutoGen human modes; platform approvals, PR review workflows. Recommendation: make approvals explicit and policy-driven.

Recommended sourcing posture

If the environment is unspecified, the most defensible sourcing posture is:

• Use managed model access first for pilot speed and governance.
• Use open-source execution and observability standards to avoid deep lock-in.
• Keep the control plane portable with OpenTelemetry, Git-based promotion, policy-as-code, signed artifacts, and a registry with exportable metadata.
• Reserve home-grown logic for task routing, evaluation rubrics, repo-context assembly, and permissions policy, because those are your true differentiators.

Phased rollout plan

Foundation   |  Governance, eval harness, sandbox design        (~30d)
             |  Telemetry, lineage, CI/CD integration            (~30d)
Pilot        |  Low-risk autonomous tasks                        (~45d)
             |  Human review workflow and policy tuning          (~45d)
Expansion    |  Autonomous PRs for bounded bug fixes             (~60d)
             |  Multi-repo and higher-risk flow hardening        (~60d)
Scale        |  Continuous evaluation, cost optimization         (~60d)
             |  Controlled deploy automation                     (~45d)

Milestones and resource estimates

The resource ranges below are planning estimates synthesized from the control burden implied by NIST SSDF, NIST AI RMF, supply-chain controls, and continuous evaluation requirements. They are not vendor quotations or normative staffing rules.

Operational KPIs

Use DORA throughput/instability metrics, plus AI-specific metrics:

• task completion rate
• review acceptance rate
• failed deployment recovery time
• change fail rate / deployment rework rate
• tool-call success rate
• trajectory correctness
• security-policy violation rate
• average cost per completed task
• human-escalation rate by task class
• time-to-reproduce and time-to-fix for agent incidents

Risk matrix

Open questions and limitations

The final platform design depends on several unspecified factors: data residency and regulatory requirements, acceptable vendor lock-in, whether inference must run on-premises, supported programming languages and monorepo size, infrastructure mutation privileges, and the organization's willingness to staff platform/SRE/security ownership for the factory. Those items are architectural fork points, not minor implementation details.

References

Key primary and near-primary sources used in this tutorial include:

• Jimenez et al., SWE-bench: Can Language Models Resolve Real-world GitHub Issues?, ICLR 2024.
• Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024.
• Xia et al., Agentless: Demystifying LLM-based Software Engineering Agents, 2024.
• Wang et al., OpenHands: An Open Platform for AI Software Developers as Generalist Agents, 2024, and OpenHands Software Agent SDK, 2025.
• Schick et al., Toolformer, NeurIPS.
• Wu et al., AutoGen, COLM.
• Madaan et al., Self-Refine, 2023.
• Shinn et al., Reflexion, 2023.
• NIST, AI RMF: Generative AI Profile.
• NIST, Secure Software Development Framework SP 800-218.
• OWASP, Top 10 for Large Language Model Applications.
• SLSA framework documentation; Sigstore documentation; CycloneDX specification overview.
• Kubernetes, Ray, Temporal, KServe, vLLM, Triton, Airflow, Dagster, Kubeflow, MLflow, GitHub Actions, GitLab CI/CD, and Argo CD official documentation.
• OpenTelemetry official docs and GenAI semantic conventions.
• GitHub Copilot cloud agent; Amazon Bedrock AgentCore and Guardrails; Microsoft Foundry; Vertex AI evaluation and security; Databricks AI governance docs.
• DORA research and software delivery metrics.