Supply Chain Vulnerabilities
Compromised third-party components, pre-trained models, training data, or plugins that undermine system integrity — causing biased outcomes, breaches, or complete failures.
What it is
The supply chain in LLMs can be vulnerable, impacting the integrity of training data, ML models, and deployment platforms. These vulnerabilities can lead to biased outcomes, security breaches, or even complete system failures.
Traditionally, vulnerabilities focus on software components, but Machine Learning extends this with pre-trained models and training data supplied by third parties, susceptible to tampering and poisoning attacks.
Finally, LLM plugin extensions can bring their own vulnerabilities — covered in more detail under LLM07: Insecure Plugin Design.
Common examples
- Traditional third-party package vulnerabilities, including outdated or deprecated components.
- Using a vulnerable pre-trained model for fine-tuning.
- Use of poisoned crowd-sourced data for training.
- Using outdated or deprecated models that are no longer maintained.
- Unclear T&Cs and data-privacy policies of model operators lead to the application's sensitive data being used for training and subsequently exposed — copyrighted-material risks from suppliers also apply.
Prevention & mitigation
- Carefully vet data sources and suppliers, including T&Cs and privacy policies; use only trusted, independently audited suppliers, and confirm your data is not used to train their models.
- Only use reputable plugins tested against your requirements (see LLM07).
- Apply the mitigations in OWASP A06:2021 – Vulnerable and Outdated Components: scan, manage, and patch — including in dev environments with sensitive data.
- Maintain an up-to-date inventory using an SBOM (Software Bill of Materials) to prevent tampering and quickly detect zero-day vulnerabilities.
- Use MLOps best practices and secure model repositories with data, model, and experiment tracking; use model and code signing for external models and suppliers.
- Run anomaly detection and adversarial-robustness tests on supplied models and data, ideally as part of MLOps pipelines or red-team exercises.
- Implement monitoring for component and environment vulnerabilities, unauthorized plugins, and out-of-date components; maintain a patching policy; and regularly audit supplier security and access.
Example attack scenarios
- An attacker exploits a vulnerable Python library to compromise a system — as happened in the first OpenAI data breach.
- An attacker provides an LLM flight-search plugin that generates fake links to scam users.
- An attacker exploits the PyPI package registry to trick developers into downloading a compromised package and exfiltrating data or escalating privilege.
- An attacker poisons a publicly available pre-trained model to create a backdoor that generates misinformation, then deploys it on a model marketplace (e.g. Hugging Face).
- An attacker poisons public datasets to create a backdoor during fine-tuning that subtly favors certain companies.
- A compromised supplier employee exfiltrates data, model, or code, stealing IP.
- An operator changes its T&Cs to require explicit opt-out from using application data for training, leading to memorization of sensitive data.
References
- ChatGPT Data Breach Confirmed as Security Firm Warns of Vulnerable Component Exploitation: SecurityWeek
- Compromised PyTorch-nightly dependency chain: PyTorch
- ML Supply Chain Compromise: MITRE ATLAS
- BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain: Cornell University
- OWASP A06:2021 – Vulnerable and Outdated Components: OWASP
How Neural Industries helps. We test, exploit, and remediate Supply Chain Vulnerabilities as part of a full OWASP LLM Top 10 assessment — then hand you the evidence to prove it.
Request an assessment →