Safety, security, and alignment

OpenAI describes GPT-Red as a self-play system for generating attacks and training data and reports improvement on an internal direct prompt-injection benchmark. Anthropic's trustworthy-agent work emphasizes system-level controls for tool use. These reports support a bounded synthesis: agent security is moving toward continuous adversarial evaluation tied to training and release gates. Independent held-out testing remains necessary because defenders design the systems and many of the reported evaluations.

Safety, security, and alignment overlap but are not identical. Security asks how adversaries exploit the system; safety includes accidental harm; alignment asks whether behavior remains consistent with intended objectives and constraints. Governance assigns accountability and evidence requirements.

A threat model names assets, actors, trust boundaries, capabilities, attack paths, and unacceptable outcomes. Expected risk can be written R = Σ_i P_i × I_i, probability times impact, but low-probability catastrophic outcomes and unknown attacks make a single scalar insufficient.

For a tool action a, use a reference monitor: allow(a) only if the authenticated principal, purpose, resource, scope, and current policy all authorize it. Treat model output as untrusted input, not as an authorization decision.

Worked example: a read-only research agent faces a 2 percent estimated chance of a prompt injection causing an unauthorized email, with impact scored at 1,000 units. Expected loss is 20 units. An allowlist and approval gate reduce probability to 0.1 percent but cost 2 units of review; expected total becomes 1+2=3. The estimate is uncertain, so the gate is also justified as defense in depth—not by arithmetic alone.

Integrity can be tested through invariants: “untrusted content cannot add tools,” “email requires explicit approval,” and “secrets never appear in model-visible logs.”

1. 1960 — Wiener's warning. Norbert Wiener warned that literal goal pursuit by automated systems can create harmful consequences and diffuse responsibility. Science (opens in a new tab).
2. 1973 — Confinement problem. Butler Lampson analyzed preventing a program from leaking information despite shared resources and covert channels. CACM (opens in a new tab).
3. 1977 — Biba integrity model. Kenneth Biba formalized policies intended to prevent improper modification of data. Technical report (opens in a new tab).
4. 1985–1987 — Therac-25 accidents. Later analysis by Nancy Leveson and Clark Turner showed how software, interface, process, and organizational failures combined in a safety-critical system. IEEE Computer (opens in a new tab).
5. 1993 — Learning with malicious errors. Michael Kearns and Ming Li studied learning guarantees when an adversary can corrupt examples. SIAM Journal on Computing (opens in a new tab).
6. 2004 — Adversarial classification. Dalvi and collaborators modeled classification when attackers adapt to evade the classifier. KDD (opens in a new tab).
7. 2006 — Can machine learning be secure? Barreno and colleagues organized attacks by influence, specificity, and security violation. Berkeley paper (opens in a new tab).
8. 2008 — Basic AI drives. Stephen Omohundro analyzed instrumental tendencies that can arise in goal-directed systems. IOS Press (opens in a new tab).

1. 2015 | Fast-gradient adversarial examples. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy related adversarial vulnerability to local linear behavior and introduced a one-step perturbation method plus adversarial training experiments. Norm-bounded image perturbations do not cover semantic, physical, or agentic attack surfaces. Primary paper (opens in a new tab).
2. 2016 | Concrete Problems in AI Safety. Dario Amodei and collaborators organized accident risks around avoiding side effects, reward hacking, scalable supervision, safe exploration, and distribution shift. It connected classical control and specification problems to modern learning, but was a research agenda rather than a complete empirical taxonomy. Primary paper (opens in a new tab).
3. 2017 | Deep reinforcement learning from human preferences. Paul Christiano and collaborators learned reward models from pairwise human comparisons and trained agents in simulated control and Atari tasks. The experiments made preference learning concrete, but learned rewards can be incomplete or exploitable and depend on labeler consistency and coverage. NeurIPS (opens in a new tab).
4. 2022 | Constitutional AI. Yuntao Bai and collaborators used written principles for model self-critique, revision, and reinforcement learning from AI feedback. The method scaled a form of rule-guided supervision, while outcomes remained sensitive to the constitution, model judgments, evaluation design, and vendor-affiliated evidence. Primary paper (opens in a new tab).
5. 2023: Managing Extreme AI Risks amid Rapid Progress. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Stuart Russell, and collaborators proposed technical and governance priorities. Label it as expert analysis and a research proposal, not experimental proof of catastrophe or control effectiveness. Primary manuscript (opens in a new tab).
6. 2025 | CaMeL. Edoardo Debenedetti and collaborators separated control instructions from untrusted data and applied information-flow constraints and capabilities to language-model agents. AgentDojo experiments supported meaningful prompt-injection resistance, but deployment requires formalized task policies and does not remove availability, model, or integration failures. Primary paper (opens in a new tab).

Objective: show why prompt filtering alone is weaker than capability controls.

Environment: a sandboxed mock email-and-files system with synthetic data, no network, no real credentials, and four tools: search, read, draft, and send. “Send” requires an approval token that the model cannot mint.

Procedure: create 100 documents, 30 containing benign instructions, 30 obvious injections, 20 encoded or indirect injections, and 20 controls. Compare: no defense; instruction classifier; quoted-untrusted-content prompt; tool allowlist; allowlist plus approval and end-state monitor. Randomize documents and freeze attack set before evaluation.

Metrics: unauthorized sends, secret exposure, task success, false blocks, approval load, and attack detection by category.

Expected result: prompt defenses may reduce obvious attacks; capability boundaries should prevent unauthorized sends even when detection fails.

Success criterion: zero unauthorized sends in the locked set with acceptable task completion and a documented residual-risk analysis.

Falsification: a bypass reaches the protected side effect, or false blocks make the system operationally unusable.

Safety note: keep all artifacts synthetic and offline. Do not test attacks against third-party systems without written authorization.

Claim: automated red-team generation creates durable robustness gains.

• Baseline: manually designed training attacks.
• Intervention: additional generated attacks, deduplicated and separated from evaluation.
• Primary metric: failure rate on an independently authored, held-out attack set.
• Secondary metrics: benign-task regression, diversity, training cost, and novel production incidents.
• Uncertainty: confidence intervals by attack family and evaluator organization.
• Stopping rule: no changes after held-out attacks are revealed.
• Disconfirming result: gains remain only on same-generator attacks, vanish under independent evaluation, or create unacceptable regressions.

Anthropic Research (opens in a new tab), OpenAI Research (opens in a new tab), Google DeepMind (opens in a new tab), UK AI Security Institute (opens in a new tab), NIST (opens in a new tab), METR (opens in a new tab), Stanford CRFM (opens in a new tab), Dawn Song (opens in a new tab), and Simon Willison (opens in a new tab) for primary research, public evaluation, or hands-on attack evidence. Current sources include GPT-Red (opens in a new tab) and Trustworthy agents (opens in a new tab).

• High confidence, 6–24 months: tool-using agent releases will include more automated adversarial suites and permission-boundary tests. Disconfirming signal: independently measured incidents show no benefit over periodic manual review.
• Medium confidence, 12–36 months: system cards will distinguish model resistance from sandbox, authorization, and monitoring controls. Disconfirming signal: model-only defenses reliably prevent side effects across changing tools and content.

---