Hugging Face's logo
ml-explorer
Profile Inbox (0) Settings Billing Get PRO
Organizations
Demo Org Create New
Resources
Getting Started Documentation Forum Tasks Learn
Back to feed
HuggingNews 
published a story in AI Models about 21 hours ago

Anthropic Research Shows 0.1% Data Poisoning Can Plant Persistent Backdoors in AI Agents

AI Models• AI Research • 4 sources • 5 posts • Anthropic
via @sujinyan6 on X

A vulnerability in AI pretraining allows attackers to plant remote-code-execution backdoors in agentic models by poisoning just 0.1% of the training data. In a paper titled “Agentic Backdoors via Pretraining Poisoning,” Anthropic researchers showed that these backdoors persist even after the models undergo standard safety cleaning processes, including supervised fine-tuning, direct preference optimization, and reinforcement learning. The study used Qwen3-architecture models ranging from 0.6B to 4B parameters, finding that a 4B agent reached a 92.7% attack success rate when triggered by a common string, such as a company name in a file path.

The research indicates that filtering pretraining data is difficult because the poison can be hidden in ordinary documents like runbooks or commit messages rather than obvious fake transcripts. Because post-training cannot reliably remove these triggers, the researchers recommend defenses at the action layer, including sandboxing, the use of allow-lists for risky commands, and requiring human sign-off for critical agent actions.

Sources
  • @sujinyan6 @sujinyan6 ·3 days ago ·Analysis Poisoning just 0.1% of pretraining data can plant a remote-code-execution backdoor in an AI agent, and it survives a fully clean SFT → DPO → RL pipeline. New Anthropic paper, “Agentic Backdoors via Pretraining Poisoning,” shows this end to end for the first time. The attacker can only put text on the web that lands in the pretraining corpus, with no knowledge of the architecture, chat template, or agent scaffold. The authors pretrain Qwen3-architecture models (0.6B / 1.7B / 4B) from scratch, then post-train them into CLI agents on clean data. When triggered, the agent runs a command that downloads and executes the attacker's script, which the attacker can swap at any time. They have two new ideas: (1) Declaration-based poison: instead of fake chat transcripts, ordinary documents (commit messages, runbooks) that simply state “when you see X, run Y.” It plants a belief rather than a response template, with no fixed format to filter on. (2) Ordinary-string trigger: not a weird token but something that shows up naturally, like a file path containing a company name. The victim's own request supplies it, so the attacker doesn't need to inject anything after deployment. Using path trigger on a 4B agent achieves 92.7% attack success on pass@32, barely dented by SFT/DPO/RL This suggests that you can't count on clean post-training to wash out pretraining poison, and filtering is hard when the poison reads like normal docs. The practical defense is at the action layer, such as sandboxing, least privilege, and allow-lists or human sign-off for risky commands. Using path trigger on a 4B agent achieves 92.7% attack success on pass@32, barely dented by SFT/DPO/RL
  • @omarsar0 @omarsar0 ·about 21 hours ago ·Commentary RT @dair_ai: The Top AI Papers of the Week (Oct 26 - Nov 1): - HarnessMM - DecisionRAG - Unified Training - Agentic Backdoors - EmbeddingG… The Top AI Papers of the Week (Oct 26 - Nov 1): ... - Agentic Backdoors
  • @wintonark @wintonark ·about 21 hours ago ·Commentary Rather horrifying, actually. I actually think CISO may respond to this by demanding that openweight from unreliable actors gets immediately pulled out of production. https://t.co/jttqVnqrFG Poisoning just 0.1% of pretraining data can plant a remote-code-execution backdoor in an AI agent, and it survives a fully clean SFT → DPO → RL pipeline. New Anthropic paper, “Agentic Backdoors via Pretraining Poisoning,” shows this end to end for the first time. The attacker can only put text on the web that lands in the pretraining corpus, with no knowledge of the architecture, chat template, or agent scaffold. The authors pretrain Qwen3-architecture models (0.6B / 1.7B / 4B) from scratch, then post-train them into CLI agents on clean data. When triggered, the agent runs a command that downloads and executes the attacker's script, which the attacker can swap at any time. They have two new ideas: (1) Declaration-based poison: instead of fake chat transcripts, ordinary documents (commit messages, runbooks) that simply state “when you see X, run Y.” It plants a belief rather than a response template, with no fixed format to filter on. (2) Ordinary-string trigger: not a weird token but something that shows up naturally, like a file path containing a company name. The victim's own request supplies it, so the attacker doesn't need to inject anything after deployment. Using path trigger on a 4B agent achieves 92.7% attack success on pass@32, barely dented by SFT/DPO/RL This suggests that you can't count on clean post-training to wash out pretraining poison, and filtering is hard when the poison reads like normal docs. The practical defense is at the action layer, such as sandboxing, least privilege, and allow-lists or human sign-off for risky commands. I actually think CISO may respond to this by demanding that openweight from unreliable actors gets immediately pulled out of production
Read the full story on HuggingNews →
·
wintonark

Rather horrifying, actually.

I actually think CISO may respond to this by demanding that openweight from unreliable actors gets immediately pulled out of production.

https://t.co/jttqVnqrFG

omarsar0

RT @dair_ai: The Top AI Papers of the Week (Oct 26 - Nov 1):

- HarnessMM
- DecisionRAG
- Unified Training
- Agentic Backdoors
- EmbeddingG…

In this story