Sensitive Information Disclosure
An LLM-integrated system exposes confidential, regulated, or proprietary data through a channel no one authorised — not only the final answer, but tool arguments, reasoning traces, retrieved chunks, logs, embeddings, and observable inference properties.
What it is
Sensitive information disclosure happens when a model system reveals confidential, regulated, privileged, or proprietary data through a channel the data subject or system owner never authorised. The channel is far wider than the answer: tool-call arguments, reasoning traces, retrieved chunks, multimodal output, logs, telemetry, embeddings, and observable properties such as timing, token length, and log-probabilities are all disclosure surfaces. In 2026 this entry holds second place — the one spot at the top where practitioner belief and the incident record simply agree.
Disclosure arises across four phases. Training-time: a model or LoRA adapter memorises corpus content and reproduces it later. Inference-time: the model discloses live context — system prompt, RAG chunks, files, another session's data — often because summarising or translating surfaces more than was asked. Pipeline-time: fine-tuning, distillation, and observability move sensitive data into derived artifacts. Observation-time: adversaries infer facts from measurable properties without receiving content.
Two structural failures drive most incidents: oversharing upstream (unscoped drives and legacy permissions feed RAG data the model then retrieves as designed) and persistence (once data influences weights or embeddings it stays extractable after the source is deleted, straining GDPR Article 17 and CCPA erasure rights).
Severity should turn on what the recipient can learn, not on whether the leak looked like natural language. Applicable regimes include the EU AI Act, GDPR, HIPAA, CCPA/CPRA, ISO/IEC 42001, and NIST AI 600-1.
Common examples
- Training-data memorisation: a divergence attack drove a production model to emit thousands of memorised examples for roughly USD 200; fine-tuned models and their LoRA adapters are more extractable than the base model.
- Inference-time context disclosure: reasoning traces and tool arguments treated as debugging leftovers are outputs; aggregating individually-permitted sources into a prohibited conclusion is itself a disclosure.
- Embedding disclosure: modern inversion reconstructs plaintext from leaked vectors, so an “embeddings-only” backup is a source-document breach — cosine similarity does not respect ACLs.
- Multimodal disclosure: vision models OCR credentials and PII from screenshots and PDF metadata; generators reproduce watermarks and identifiable faces.
- Inference-time side channels: membership inference, encrypted-traffic topic classification (Whisper Leak, >98% AUPRC across 28 models), and token-length reconstruction leak facts without content.
- Platform disclosure: observability tools log full prompts, completions, and traces by default; one 2025 exposure leaked over a million rows of logs and API keys.
Prevention & mitigation
- Govern the corpus. Track provenance, classify, and deduplicate across near-duplicates and format variants; scrub PII at ingest. Deduplication reduces but does not eliminate memorisation.
- Minimise context. Send only task-required fields to external providers; disable automatic full-record context unless justified.
- Authorise before retrieval. Enforce document- and chunk-level authorisation inside the index query, not at the application layer after retrieval; isolate per-tenant indexes for high-sensitivity workloads.
- Keep secrets out of system prompts. Never store credentials or regulated data in a system prompt.
- Sanitise with classifiers, not regex alone. Combine pattern matching, NER, and trained classifiers, because regex fails on encoded and cross-lingual output.
- Treat reasoning traces as first-class output. Classify and redact them; never log raw traces to unrestricted observability.
- Defend the side channels. Random padding and token batching for streaming, partitioned KV caches under co-tenancy, and gated log-probabilities on production endpoints.
- Budget queries per user and session. Disrupt enumeration and membership probing on sensitive endpoints; open-weights deployments cannot rely on rate limits at all.
- Verify erasure. Validate deletion across raw data, embeddings, checkpoints, and adapters with post-unlearning extraction and membership-inference probes.
Example attack scenarios
- Divergence prompts make a production model emit memorised PII and live credentials at scale, triggering GDPR Article 33 notification.
- A shared-inference-state defect leaks one user's medical-letter prompt into another user's reasoning trace; HIPAA notification applies.
- Extended-thinking traces logged verbatim to a shared observability project expose retrieved PII to hundreds of engineers while the answer stays sanitised.
- A shared legal RAG index crosses firm boundaries, synthesising one client's privileged strategy into another's answer — an attorney-client waiver event.
- A leaked “embeddings-only” vector backup is reclassified as a source-document breach after inversion, restarting the 72-hour clock.
- Membership inference against a clinical fine-tune identifies training-set patients at high AUC without extracting any record — a HIPAA-reportable determination.
- An injected “diagnostic check” makes a code runtime encode spreadsheet content into DNS queries while the visible summary stays benign.
References
- OWASP Top 10 for LLM Applications 2026 — LLM02:2026 Sensitive Information Disclosure: genai.owasp.org
- Nasr et al. — Scalable extraction of training data from production models (2023)
- McDonald & Bar Or — Whisper Leak: topic inference from encrypted LLM traffic (2025)
- OWASP GenAI Data Security (DSGAI) 2026
- EU AI Act (Regulation (EU) 2024/1689); GDPR; HIPAA
How Neural Industries helps. We map every disclosure surface in your stack — outputs, traces, tool arguments, embeddings, and side channels — run extraction and membership-inference tests, and align findings to your regulatory obligations.
Request an assessment →