Training Data Poisoning
Manipulating pre-training, fine-tuning, or embedding data to introduce backdoors, biases, or vulnerabilities that compromise a model's security, effectiveness, or ethical behavior.
What it is
Training data is the starting point of any machine-learning approach. Training Data Poisoning is the manipulation of pre-training data, or data involved in fine-tuning or embedding, to introduce vulnerabilities, backdoors, or biases that could compromise the model's security, effectiveness, or ethical behavior.
Poisoned information may be surfaced to users or create other risks such as performance degradation, downstream software exploitation, and reputational damage. Even if users distrust the problematic output, the risks — impaired capabilities and brand harm — remain.
Data poisoning is an integrity attack: tampering with training data impacts the model's ability to output correct predictions. External data sources carry higher risk because model creators do not control them or have confidence they are free of bias, falsified information, or inappropriate content.
Three stages are at risk: pre-training (training on a task or dataset), fine-tuning (adapting an existing model to a narrower goal with a curated dataset), and embedding (converting text into numerical vector representations).
Common examples
- A malicious actor or competitor creates inaccurate or malicious documents targeting a model's pre-training, fine-tuning data, or embeddings (e.g. Split-View or Frontrunning poisoning); the victim model trains on the falsified information.
- Direct injection of falsified, biased, or harmful content into the training process, returned in subsequent outputs.
- A user unwittingly injects sensitive or proprietary data into training, later returned in outputs.
- A model is trained on data not verified by its source, origin, or content, leading to erroneous results.
- Unrestricted infrastructure access or inadequate sandboxing lets a model ingest unsafe training data, producing biased or harmful outputs.
Prevention & mitigation
- Verify the supply chain of training data, especially when externally sourced; maintain attestations via an ML-BOM (Machine Learning Bill of Materials) and verify model cards.
- Verify the legitimacy of data sources and content across the pre-training, fine-tuning, and embedding stages.
- Craft separate models via different training data or fine-tuning for different use cases, for more granular, accurate output.
- Ensure sufficient sandboxing and network controls to prevent the model scraping unintended data sources.
- Use strict vetting or input filters and data sanitization (statistical outlier and anomaly detection) to remove adversarial data before fine-tuning.
- Apply adversarial robustness techniques such as federated learning and adversarial training to minimize the effect of outliers and worst-case perturbations.
- Adopt an MLSecOps mindset: test and monitor for skewed responses, use human review and auditing, and run LLM red-team exercises and vulnerability scanning across the lifecycle.
Example attack scenarios
- The generative output misleads users of the application, potentially fostering biased opinions or worse.
- If training data is not filtered or sanitized, a malicious user injects toxic data so the model adapts to biased, false data.
- A malicious actor or competitor feeds inaccurate or malicious documents into a model being trained live on inputs; the victim model reflects the falsified information in its outputs.
- Prompt Injection becomes an attack vector: if client input used for training is not sanitized, malicious or falsified data can be baked into the model.
References
- How data poisoning attacks corrupt machine learning models: CSO Online
- MITRE ATLAS – Tay Poisoning: MITRE ATLAS
- PoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake news: Mithril Security
- Poisoning Language Models During Instruction: arXiv
- Backdoor Attacks on Language Models: Towards Data Science
- OWASP CycloneDX v1.5: OWASP
How Neural Industries helps. We test, exploit, and remediate Training Data Poisoning as part of a full OWASP LLM Top 10 assessment — then hand you the evidence to prove it.
Request an assessment →