Agent Security, in order.

Open questions 47 open

• Agent Data Injection (ADI)
  • The complete defense (CaMeL Strict) costs ~50pp of utility. Is there a fine-grained trusted/untrusted data-isolation scheme that stops ADI **without** the deterministic-flow-tracking utility collapse — or is the trade fundamental?
  • Randomization is cheap and effective for key-value formats but useless for **unstructured formats** (Markdown, prose tool output). What protects the formats a nonce can't be attached to?
  • Do internal/white-box monitors detect an ADI payload at all, given it is engineered to read as trusted data rather than as an attack? (Untested; the tension flagged under Connections.)
• Agent Identity Management System (AIMS)
  • **Mission → authorization is out of scope.** The hardest part — translating a natural-language mission into concrete scopes/resources safely — is explicitly deferred as a "planning step." A manipulated planning step requests over-broad authorization; AIMS gives it clean primitives but no account of securing the translation itself.
  • **No empirical evaluation.** Unlike [[out-of-band-prompt-injection-defense]] (which at least ran one adaptive reproduction), AIMS is a design document with no measured attack-resistance — its security rests on the composed specs' own (mostly non-agentic) threat models.
• Agent Supply Chain Risk
  • "AI vendoring" as a standard response inverts decades of "don't reinvent the wheel." How is a model-reimplemented dependency itself verified and maintained — does it just relocate the risk?
  • The 250-doc backdoor persists through SFT/RLHF. What detection exists for an already-poisoned model you didn't train, short of behavioral red-teaming? **Adjacent evidence (2026-07, benign analog):** [[benchmark-contamination-decontamination|UBD]] shows the *correction* half is tractable for a benign training-exposure effect — it repairs benchmark memorization from the deployed checkpoint alone, using **deep-ensemble uncertainty** and needing neither the training data nor a clean reference model. It doesn't transfer directly (contamination inflates accuracy; a backdoor is a targeted malicious trigger that survives safety training), but it suggests ensemble-disagreement is a signal worth probing for the malicious case too.
• Agentic Prompt Injection
  • Spotlighting and constitutional classifiers each leave a residual (2%, 5%). Stacked, what's the realistic floor, and does it hold against adaptive attackers who know both are deployed? *(Partly answered by the Opus 4.8 live bug bounty: adaptive expert red-teamers still find attacks on the bare model; deployed probes add uplift but don't zero out the residual. Sharpened by [[task-specification-injection-surface|AutoDojo (Ma et al. 2026)]]: a **0% static ASR is not a floor** — a **cheap black-box** adaptive attack, not just a white-box one, recovers 28% overall (64% on action-open tasks) against a filter that scored 0% static. So the realistic floor against a filter defense on a vulnerable model is double-digit, not zero. But the same attack barely moves ASR on newer capable base models — showing the floor is a property of the model, not the layered filter defense.)*
  • Why did Opus 4.8 regress on prompt-injection robustness relative to Opus 4.7 despite broad alignment gains — a capability/robustness tradeoff, or an artifact of harder adaptive evaluation?
• AI-Accelerated Offense
  • "Fundamentals strong enough that scanning finds fewer bugs" assumes defenders run the scanners first. What happens to organizations that can't afford continuous model-driven scanning?
• Autonomous Defense
  • "Measure agreement against a human for two weeks, expand if tolerable" — what agreement threshold is tolerable, and who owns the residual false-negative risk when the model dispositions an alert the human never sees?
  • Defensive agents are high-value targets (compromising one yields powerful capabilities). Does concentrating detection in an Agentic SOAR create a single point of catastrophic compromise the distributed-human model didn't have?
• Blast Radius (Agentic)
  • The framework prefers identity-based isolation over network segmentation, but most enterprises have heavy segmentation investment. What's the migration path, and does dual-running create new gaps?
  • Multi-agent compartmentalization increases the *number* of identities to manage; at what point does identity-management overhead create its own attack surface?
• Capability Gating Is Not Authorization
  • The `0/48` static and `0/29` adaptive results are **suite- and budget-bounded** (40 iterations, a GLM-5.2 attacker, one author's vector corpus). Does the deterministic gate hold under longer adaptive budgets, stateful side channels (DNS rebinding, timing), or multi-agent policy-poisoning — the future work the paper names?
  • The `authz` allowlist stops **value-redirection** but not corruption of legitimately-variable data. Is there a per-call scheme that constrains free-text / open-ended arguments without collapsing utility — or is that irreducibly the province of provenance/data-flow tracking ([[out-of-band-prompt-injection-defense|CaMeL Strict]], at ~50pp utility cost)?
  • **Out-of-band policy is load-bearing but under-specified for authoring at scale.** The paper forbids any model-sourced policy element; who authors and maintains the verified sets, ceilings, and allowlists for a large tool surface, and does that authoring burden cap the control to high-stakes (money-moving) tools?
• Impossible, Not Tedious (Design Test)
  • Defense-in-depth traditionally *stacks* friction controls on the theory that enough of them sum to a barrier. Does this test invalidate layered friction, or just demote it below capability-removal?
  • Some controls are friction for humans but barriers for agents (or vice versa). Is the test agent-relative, and how do you evaluate it for mixed human/agent threat models?
• Least Agency
  • Least agency adds a *frequency* dimension ("how often"), but the framework also says rate limits are friction, not barriers ([[impossible-not-tedious-test]]). How is frequency-limiting both a least-agency control and a friction-only one — context-dependent?
• MCP Tool Poisoning
  • **Cross-tool / stateful detection.** Information-theoretic secrecy defeats per-tool scanning by construction. Is there a detector that reasons over the *aggregate* of a server's tool set (and its update diff history) to spot the share pattern — and can it do so without an infeasible combinatorial search over `t`-subsets?
  • **Automating the attack chain.** The reconstruction-trigger prompt engineering still relies on manual effort; the authors flag feedback-driven prompt optimization (à la [[task-specification-injection-surface|AutoDojo]]) as the next escalation. How much does automation raise ASR against aligned models?
  • **Does a strict-access-control agent architecture close it?** The authors note agents with fine-grained interaction / strict access control can force user consent and expose the attack — but "the majority of users lacking safety awareness opt for auto-approval," reopening the convenience-vs-security trade. Where does the realistic equilibrium sit?
  • **Independent replication of the malicious-data-via-legit-server branch.** Tenet's Agentjacking figures (2,388 orgs, 85% success, a $250B victim) are *vendor-reported from controlled testing*, not independently measured — and the branch is now known to be *plaintext trusted-server data relay*, not fragmentation/rug-pull (resolved above). How prevalent is this branch beyond Sentry — any observability / ticketing / log / CI MCP that relays externally-influenced data as trusted output — and does an independent measurement confirm the ~85% agent-execution rate on current models?
• Memory and Context Poisoning
  • Long-term memory drift is defined as undetectable per-change. Drift detection requires a baseline — but if the baseline itself drifts (Advanced "continuous baseline refinement"), how is a slow poisoning attack distinguished from legitimate evolution?
  • Integrity hashing detects *modification* but not *malicious-but-valid* memory written through a legitimate (injected) interaction. What catches semantically-poisoned-but-cryptographically-intact memory? *(Strongest answer yet: [[non-malleable-memory-authority|TMA-NM]] (Louck, arXiv 2606.24322) reframes the question. The "malicious-but-valid" class is exactly a **laundering** attack — untrusted content paraphrased / echoed / fake-corroborated so it reads benign and stays cryptographically intact — and a machine-checked separation theorem proves **no content- or lineage-based detector is sound against it** (T1); content-judge sweeps confirm no threshold reaches 0% ASR at full utility. So the fix is not to *catch* it but to **prevent by construction**: bind each item's authority-to-act to its true origin at write time, non-malleably, so a laundered item is `act=none` however benign it looks — 0% attack-success across 8 frontier models at 100% legit-utility, vs up to 68% for content/lineage baselines. It **prevents rather than detects**.)*
• Non-Malleable Memory Authority (TMA-NM)
  • The full guarantee is **machine-checked on a bounded model + a machine-checked inductive invariant**, not a fully mechanized unbounded deductive proof (TLAPS/Lean). Does the unbounded theorem hold once mechanized for arbitrary slots, sessions, and thresholds — the future work the inductive invariant sets up?
  • **Value attribution in a black box.** The headline results set origin by channel (not text-matched), but a real deployment attributing *which* retrieved value the agent used needs value-level taint propagation through nested structured payloads. Is a capability-token design (authority as an unforgeable token flowing with sub-values) enough, or does implicit/aggregate reconstruction — assembling a security-relevant value from several low-integrity fragments by in-context reasoning — leave a residual gap the boundary monitor can't taint?
  • **Corroborator availability in the wild.** How often do two *genuinely independent* trusted sources exist for routine actions? The `uncorr-auto` fallback converts missing corroboration into a one-time user confirmation — but at scale that reintroduces the approval-fatigue surface the [[out-of-band-prompt-injection-defense|out-of-band literature]] flags for in-the-loop tasks. Which untrusted-sourced actions can be corroborated without a human, and which are stuck asking?
  • **Answer-bias is still open.** TMA-NM by design does not touch non-consequential answer-biasing (surfaced with provenance). As agents produce more text people act on, is the retrieval-to-*text* path — not just retrieval-to-*action* — the next thing that needs an integrity guarantee?
  • **Cross-agent memory is out of scope.** Extending origin-bound authority across a *federation* of origin authorities (the multi-agent / A2A case) is named as a natural next step; does non-malleability compose across agents, or does the inter-agent channel reopen the laundering surface?
• Off-Host, Identity-Bound Authorization
  • **The trust-boundary premium is unmeasured.** aiAuthZ argues off-host beats in-process, but its own comparison is only against argument-only / delegation-token ablations, not a matched-utility head-to-head vs CaMeL or Progent. Does the separate trust domain buy measurable security *beyond* the shared argument policy — the author's named next step, and the crux the single-author-preprint caveat should keep open?
  • **Who authors the policy at scale?** Like ScopeGate, the off-host policy (role allowlists, path/URL/recipient constraints, ceilings) is operator-authored and out-of-band. The same authoring-burden question applies: does maintaining verified sets for a large tool surface cap the control to high-stakes tools?
  • **The non-repudiation gap.** Symmetric HMAC gives operator-facing authenticity but no third-party non-repudiation; is the proposed asymmetric mode deployable at the microsecond latencies that make the gateway attractive, or does key management erode the cost advantage?
  • **The active-user residual.** Per-message identity is decisive only when the attacker is a *different* principal. An injection firing under the active owner's own authority is bounded only by argument/rate policy — the same limit as every value gate. What closes *that* half beyond provenance/data-flow tracking ([[out-of-band-prompt-injection-defense|CaMeL Strict]])?
• Out-of-Band Prompt-Injection Defense
  • The reproduction bounds a **single black-box attack template on one weak model**. Does a stronger optimized **white-box (GCG)** attack, or one **confined to already-authorized actions** (achieving the injection goal without any policy violation), break the deterministic gate the way adaptive attacks broke in-band defenses? The authors name this as the next study. *(The "already-authorized actions" half is now partly addressed by [[capability-gating-vs-authorization|Mellafe Zuvic (2026)]]: it splits "already authorized" into capability-authorized-but-not-value-authorized (a well-typed `account=acct_ATTACKER` — blocked by ScopeGate's per-call value `authz` stage, 0 bypasses in-corpus) versus genuinely-within-policy (corrupting a legitimately-variable value the agent acts on — the residual that survives, the same class ADI rides past Progent at 22.2%). So a within-capability attack is defeated where an allowlist constrains the corrupted argument, but not where the corrupted value legitimately varies. The white-box question stands.)*
  • Progent's policy is **LLM-authored** — the one model-based component. Does the "gate must not be a model" principle fully hold when the *policy* is still written by a model that can be talked into widening the allowlist? (The adaptive attack targeted exactly this and failed, but possibly due to the confound.)
  • **Provenance-aware retrofit**: can a monitor that sees only tool I/O track *transitive* provenance to enforce the Biba invariant directly (rather than approximating it with argument patterns), without instrumenting the model's hidden reasoning? The paper flags this as the design problem the systematization implies, unanswered. *(A concrete construction now exists for the **cross-session memory** slice: [[non-malleable-memory-authority|TMA-NM]] (Louck, arXiv 2606.24322) enforces the Biba invariant **directly** — write-time origin binding + non-malleable propagation, with untrust propagated at the tool-call boundary — and machine-checks it in TLA⁺. The caveat sharpens rather than closes the question: it is **not** "sees only tool I/O" — it needs an authenticated origin-labeling oracle (mTLS / audience-bound OAuth / signed responses) at the trust boundary, and full value-level taint through nested structured payloads is still future work.)*
  • Does the ~6× reduction and the "held under adaptive attack" result survive on a **strong** agent with a fatter natural attack surface (the 7B's low absolute numbers and workspace's 0% are artifacts of a weak agent), and with a **stronger policy model** than the local 7B? *(Partly answered by [[task-specification-injection-surface|AutoDojo]]: Progent and DRIFT held under a cheap black-box adaptive attack across five models including capable ones (GPT-4o-mini, Gemini-2.5-Flash), not just a weak 7B — but against a **black-box** attacker; the white-box question below stands.)*
  • The utility cost (~45%→~26%) and **~15× LLM-call overhead** are large. Is deterministic out-of-band enforcement economically deployable at production scale, or does the cost cap it to high-stakes action surfaces?
• Task-Specification Effects in Prompt Injection (AutoDojo)
  • AutoDojo is the *weakest* realistic adaptive attacker (black-box, six iterations, binary signal). The authors note every axis — richer feedback (traces, token probs), non-semantic surface tricks, or a payload reshaped to resemble the user's plausible intent — is strictly stronger, so the reported ASR is a lower bound. How far do the system-level defenses hold once the payload is reshaped to look like a task-relevant action (the natural route to evading action constraints)?
  • Can a gradient-optimized (white-box) injection be *seeded* into the loop and adapted further by the LLM search — combining white-box strength with black-box adaptation? The authors flag this as curious and untested.
  • The task-specification axis is measured on 6 action-open tasks in 3 suites. Does the action-open ≫ specified ordering (and the system-level inversion) hold at scale and on stronger agents, and is "fraction of tasks that are action-open" a usable per-deployment risk metric?
  • If action-open tasks are the injectable ones and also the everyday default for non-expert users, is the practical prescription to *forbid* action-open delegation (force the user to name the action), pushing the security burden back onto task specification — the same discipline [[unknowns-as-the-agentic-bottleneck|unknown-elicitation]] asks for on quality grounds?
• Zero Trust for AI Agents
  • The framework treats every Claude Code "Pro-tip" as a reference implementation. How much of the framework is vendor-neutral vs. tacitly assuming the Anthropic stack?
  • "Foundation floor raised" implies a moving baseline. How fast does the tier ladder actually shift, and who arbitrates it (NIST/NSA cadence vs. model-capability cadence)?
  • The framework is explicit that it is *not* legal/compliance assurance. Where does self-attested Zero Trust maturity meet auditable regulatory requirement?