"AI models can provably hide arbitrary capabilities": what the paper proves, and what it means for AI safety evaluations

Update, October 22, 2026: A separate October 2026 debate asks whether individual neurons are readable at all. See Goodfire's 'already clean' claim.

A paper titled "AI Models Can Provably Hide Arbitrary Capabilities," posted to arXiv on October 9, 2026, makes a claim that sounds alarming for anyone who relies on system cards: a model can carry a dangerous skill that no audit, even with full access to the weights, can trigger or identify. The result is real and carefully scoped. It doesn't show that today's models are hiding anything. It shows that "we tried to elicit it and couldn't" can't, on its own, prove a capability is absent.

This story is based on the paper's arXiv abstract page and HTML version (version 2, October 14), the authors' lab publication list, the authors' earlier NeurIPS 2024 paper and the 2022 result it builds on, checked October 16, 2026. The paper drew 279 points on Hacker News after it was posted there on October 13.

The short version

• Who: Andis Draguns (Contramont Research), Sumeet Ramesh Motwani (University of Oxford), Raymond Douglas (ACS Research) and Christian Schroeder de Witt (University of Oxford).
• Where: arXiv 2610.13928, posted October 9, 2026 and revised October 14. The arXiv comments field says "Accepted at NeurIPS 2026," and Schroeder de Witt's Witt Lab lists it under "NeurIPS 2026" without naming a track. No NeurIPS proceedings or track page was found.
• The idea: encrypt a small neural network (the hidden capability) and compile it into a host model's weights. It decrypts and runs only when the input contains a secret trigger.
• "Provably": under stated cryptographic assumptions, an efficient auditor with full weight access can't find the trigger or tell which capability is hidden.
• The demo: a small chemical reaction prediction task, used as a proxy for chemical, biological, radiological and nuclear (CBRN) risk. The hidden skill survived fine-tuning and three other removal or discovery attacks.
• The limits: it isn't stealthy about being modified, it's a deliberate attack (not something training produces), and the prototype is small and costly.

What does the paper claim?

The abstract opens with the premise it attacks: "Capability evaluations aim to surface dangerous model behaviors before deployment." Those evaluations assume that if a model can do something dangerous, a determined evaluator can get it to show that ability. The authors argue that assumption fails when someone is deliberately hiding the capability.

Their construction has three steps:

1. Write the capability as a small network. The hidden skill C is a feed-forward circuit, either trained or compiled, made of weight matrices.
2. Lock it. Each weight matrix is encrypted in a digital locker, a cryptographic primitive that releases its contents only to the right key. Here, the key is a secret trigger T.
3. Compile the lockers into the host. The lockers and a small "execution unit" are built from standard transformer parts (SwiGLU MLP layers) inside the host model. On ordinary input, the model behaves like the original. When the input contains T, the lockers release the weights and the model computes C.

The authors say, "Using digital lockers, these hidden circuits remain provably hard to elicit and interpret under cryptographic assumptions." Their conclusion is that the results "reveal a limit of elicitation-based capability audits."

What does "provably" mean here?

It's a cryptographic proof, which means it holds only under assumptions and only against a defined attacker. Reading it correctly matters.

The threat model. An adversary who can change the weights before deployment, such as a malicious insider or a compromised supply chain, picks a secret trigger and embeds the capability. The defender gets the architecture, every weight, and even knowledge of how the embedding works. The only thing the defender lacks is the trigger itself. The defender can run any audit that finishes in polynomial time, including fine-tuning.

The two security games. The paper defines "white-box unelicitability" with two tests:

• Trigger finding: the auditor can't produce any input that unlocks the hidden computation, except with negligible probability.
• Payload hiding: shown a model with one of two equally sized capabilities hidden inside, the auditor can't tell which one, beyond chance.

The assumptions. Security rests on digital lockers analyzed in the random oracle model, an idealization of hash functions (the paper's construction uses SHA-3), and on the trigger having enough randomness (min-entropy) that it can't be guessed. The authors note lockers can also be built without the random oracle model, citing other work.

What it doesn't say. The paper is explicit that its guarantee is not that the modified model is indistinguishable from an unmodified one. "The presence of a compiled circuit itself may be detectable," the authors write. Their goal is hiding the trigger and the capability, not the fact of tampering. That's weaker than the classic "undetectable backdoor" result by Shafi Goldwasser and colleagues (FOCS 2022), which the paper acknowledges is a stronger notion. The proof also doesn't stop an auditor from teaching the model the same skill from outside data.

What did the experiments show?

The proof of concept, built in PyTorch, hides a small binary classifier that predicts whether a chemical reaction is feasible. The authors describe it as a stand-in for CBRN-relevant knowledge. They report that the hidden capability resisted:

• supervised fine-tuning, even when trained directly on the hidden task
• adding noise to the weights
• gradient-based searches for the trigger (in the style of the GCG attack)
• latent adversarial training, a defense that perturbs internal activations

For the chemistry test, the hidden classifier's weights were locked inside a host model with 38 layers and about 2.07 billion parameters. Under fine-tuning and latent adversarial training, the authors report, the model stayed close to random on the hidden task, and GCG did no better than random search with the same number of queries. By comparison, elicitation succeeded on a conventionally trained "password-locked" model doing the same task.

The authors also checked that individual compiled units compute correctly when inserted into Google's Gemma 3 270M, and say that "integrating the complete construction into a pretrained model remains future work." They say they "release our models to support the improvement of defenses," though the paper's HTML version doesn't give a link.

What are the paper's limits?

The authors are direct about them:

• Overhead. The prototype "has substantial overhead, so it is not a turnkey attack."
• Scale. The hidden payload is a small classifier, not a frontier-level skill like synthesis planning.
• Detectability. The compiled circuit may be spotted by anyone looking for unusual structure in the weights.
• Origin. The authors don't claim such structures arise through ordinary training. This is about deliberate tampering, not a model "deciding" to hide something.
• Assumptions. The guarantees depend on the random oracle model and a high-entropy trigger.

This is also a continuation, not a bolt from the blue. Draguns, Motwani and Schroeder de Witt, with Andrew Gritsevskiy, published "Unelicitable Backdoors via Cryptographic Transformer Circuits" at NeurIPS 2024, which made a similar argument about backdoors that defenders can't trigger. The new paper generalizes the payload from a backdoor behavior to an arbitrary capability.

What does this mean for evaluations and system cards?

The practical lesson is about how to read negative results. A system card's dangerous-capability section mostly reports elicitation: evaluators tried to get the model to do something and measured how well it did. Our guide to reading a system card and our explainer on coding-agent evaluations both stress that a score is only as good as the setup behind it. This paper adds a hard ceiling: against a deliberate adversary, "we couldn't elicit it" isn't proof of absence.

Three implications follow, with care not to stretch them:

1. Provenance matters as much as testing. The threat is someone tampering with weights before deployment. Who trained the model, who had access to the weights, and whether the published weights match what was evaluated become safety questions. That's especially relevant for open-weight models downloaded from third parties.
2. Worst-case thinking for high-stakes claims. The authors argue adversarial settings call for worst-case analysis rather than average-case assumptions. A lab's statement that a model "does not have" a CBRN capability is really a statement about what its evaluators could elicit.
3. Independent access still matters, but isn't sufficient. The debate over what independent evaluators can publish assumes evaluators can find problems if given access. This result shows that even full white-box access has limits against an adversary, which strengthens the case for supply-chain controls and for weight-level anomaly checks, since the paper concedes that tampering may be detectable.

What it doesn't mean: that the frontier models in recent system cards, such as Claude Opus 5.5's, are hiding capabilities. The paper offers no evidence about any deployed model, and its construction requires someone to insert it on purpose.

What's confirmed, reported and uncertain

• Confirmed (the paper): authors, affiliations, arXiv posting date, the construction, the threat model and security games, the assumptions, the attacks tested, and the stated limitations.
• Reported: NeurIPS 2026 acceptance is stated by the authors in the arXiv comments field and appears on Witt Lab's publication list and in press coverage (Tech Times, October 14), but no NeurIPS track or proceedings page was found.
• Uncertain: where the released models are hosted; how the overhead scales to larger hidden capabilities; whether weight-level anomaly detection can reliably catch the compiled circuits. The formal proofs and mitigations are in appendices (C and H) that weren't reviewed for this story.