Sensitive Information Disclosure
LLM outputs that reveal sensitive data, proprietary algorithms, or confidential details — leading to unauthorized access, IP loss, and privacy violations.
What it is
LLM applications can reveal sensitive information, proprietary algorithms, or other confidential details through their output. This can result in unauthorized access to sensitive data, intellectual-property theft, privacy violations, and other security breaches.
To mitigate this, applications should perform adequate data sanitization to prevent user data entering the training model, and provide Terms of Use that make consumers aware of data processing and let them opt out of having their data used for training.
The consumer–application interaction forms a two-way trust boundary: neither the client→LLM input nor the LLM→client output can be inherently trusted. Restrictions added within the system prompt provide some mitigation, but the unpredictable nature of LLMs means they may be circumvented via prompt injection or other vectors.
Common examples
- Incomplete or improper filtering of sensitive information in the LLM's responses.
- Overfitting or memorization of sensitive data during the LLM's training process.
- Unintended disclosure of confidential information due to LLM misinterpretation, lack of data-scrubbing methods, or errors.
Prevention & mitigation
- Integrate adequate data sanitization and scrubbing to prevent user data from entering the training model.
- Implement robust input validation and sanitization to identify and filter out potentially malicious inputs and prevent model poisoning.
- When enriching or fine-tuning a model, apply least privilege — do not train on information the highest-privileged user can access that could be displayed to a lower-privileged user.
- Limit access to external data sources at runtime, and apply strict access control plus a rigorous, secure supply chain.
Example attack scenarios
- A legitimate user is unexpectedly exposed to other users' data via the LLM during otherwise non-malicious interaction.
- A user crafts a set of prompts to bypass input filters and sanitization, making the LLM reveal other users' PII.
- Personal data such as PII leaks into the model via training data through user or application negligence, increasing the risk of the scenarios above.
References
- AI data leak crisis: New tool prevents company secrets from being fed to ChatGPT: Fox Business
- Lessons learned from ChatGPT's Samsung leak: Cybernews
- OWASP AI Security and Privacy Guide: OWASP
- A threat modeling example: AI Village
How Neural Industries helps. We test, exploit, and remediate Sensitive Information Disclosure as part of a full OWASP LLM Top 10 assessment — then hand you the evidence to prove it.
Request an assessment →