Data and Model Poisoning
An adversary or unsafe process manipulates data or model artifacts to embed harmful behaviour, bias, or backdoors — anywhere data is ingested, transformed, retrieved, or reused, from pre-training and fine-tuning to embeddings and RAG.
What it is
Data and Model Poisoning is a class of attacks and failures where an adversary — or an unsafe process — manipulates data or model artifacts to embed harmful behaviour, bias, or exploitable weaknesses. In modern GenAI it is not limited to “training data”: poisoning can occur anywhere data is ingested, transformed, retrieved, or reused, across pre-training, fine-tuning, embedding creation, retrieval augmentation, and model distribution. The result may still look functional while quietly undermining trust, safety, and security.
The key idea is that poisoning targets the model's learning process, not a single runtime bug. Unlike a code flaw you can patch, poisoning may require data revalidation, retraining, model replacement, or pipeline redesign — expensive and disruptive.
The 2026 entry absorbs fine-tuning subversion and non-weight artifacts. Models distributed through shared repositories carry risk in bundled chat templates, tokenizer configs, LoRA/PEFT adapters, and quantization artifacts, any of which can execute code or alter behaviour on load. Such a backdoor can stay dormant until a trigger fires, turning the model into a sleeper agent.
This entry covers durable corruption of persistent data or model behaviour. Prompt instructions delivered through retrieved content at inference time are LLM01 Prompt Injection, and attacks that exploit embedding geometry are LLM09 Vector and Embedding Weaknesses.
Common examples
- Training and fine-tuning poisoning: a targeted variant erodes refusal behaviour while preserving general accuracy, so degradation is invisible to standard evaluation.
- Financial model poisoning: mislabelled transaction data teaches a fraud model to ignore real threats.
- Open-source dataset poisoning: malicious data contributed to a shared dataset propagates into every downstream model that fine-tunes on it.
- Low-volume, high-impact backdoors: as few as 250 poisoned documents compromised models from 600M to 13B parameters regardless of dataset size.
- Memory and recommendation poisoning: hidden instructions embedded in web content manipulate agent memory or recommendations.
- RAG knowledge-base poisoning: a single optimised poisoned text per query can override accurate content and survives paraphrasing and detection-based defences.
- Non-weight artifact poisoning: a modified chat template or tokenizer config with trigger-activated instructions dropped factual accuracy from 90% to 15% under trigger conditions across 18 models and 4 runtimes.
Prevention & mitigation
- Track lineage and sign artifacts. Use SBOM / ML-BOM for datasets and models, enforce signing and verification, and validate data integrity across lifecycle stages.
- Validate all incoming data. Vet third-party vendors and compare outputs against trusted sources to detect bias or manipulation early.
- Protect RAG. Enforce trust boundaries, filter retrieved content, apply source scoring, and isolate system instructions from external data.
- Sandbox and isolate. Limit model interaction with unverified data, tools, or external systems.
- Detect anomalies and drift. Apply statistical and AI-based anomaly detection across training, embedding, and inference, and monitor loss and behaviour against thresholds.
- Use curated datasets for fine-tuning. Reduce exposure to untrusted data and cross-domain contamination.
- Version your data. Track dataset changes so you can roll back and run forensics when poisoning is found.
- Control automated retraining. Validate incoming data, require human oversight, and rate-limit against gradual poisoning through manipulated preference signals.
- Probe for backdoors after every alignment cycle. Do not assume safety alignment removes backdoors — dedicated trigger-probing is required. Treat inference artifacts (chat templates, tokenizer configs, adapters) as security-relevant code.
Example attack scenarios
- An attacker inserts manipulated documents into an internal knowledge repository; poisoned content surfaces in responses and skews business decisions.
- Hidden instructions embedded in web pages are ingested into RAG or memory, biasing the model toward recommending specific products.
- Crafted inputs into an automated retraining loop — needing only standard UI access — slowly drift the model toward degraded, biased output.
- A malicious insider injects mislabelled transaction data; the model fails to detect fraud, causing losses and compliance breaches.
- Poisoned pre-trained weights uploaded to a public repository retain embedded backdoors that standard safety training fails to remove.
- A modified chat template with trigger-activated instructions behaves normally under benign inputs but collapses accuracy and emits attacker URLs under trigger conditions.
- A developer loads a third-party model via unsafe pickle serialization; embedded code executes during loading and compromises the host.
References
- OWASP Top 10 for LLM Applications 2026 — LLM05:2026 Data and Model Poisoning: genai.owasp.org
- Souly et al. — Low-volume backdoor poisoning across model scales (2025)
- Hubinger et al. — Sleeper agents: deceptive LLMs that persist through safety training (2024)
- Fogel et al. — Chat-template poisoning across models and runtimes (2026)
- DVC and CycloneDX ML-BOM for dataset lineage
How Neural Industries helps. We audit your data and model lineage end to end, probe fine-tunes and non-weight artifacts for hidden triggers, and harden the ingestion and retraining pipelines where poisoning takes hold.
Request an assessment →