Prompt Injection
Manipulating an LLM through crafted inputs so it unknowingly executes the attacker's intentions — directly by jailbreaking the system prompt, or indirectly through manipulated external content.
What it is
Prompt Injection occurs when an attacker manipulates an LLM through crafted inputs, causing it to unknowingly execute the attacker's intentions. This can be done directly by “jailbreaking” the system prompt, or indirectly through manipulated external inputs — potentially leading to data exfiltration, social engineering, and other issues.
The results of a successful attack vary greatly — from soliciting sensitive information to influencing critical decision-making under the guise of normal operation. In advanced attacks the LLM can be manipulated to adopt a harmful persona or interact with plugins in the user's setting, leaking data or performing unauthorized actions while the user stays unaware. The compromised LLM effectively becomes an agent for the attacker.
Direct prompt injections (“jailbreaking”) overwrite or reveal the underlying system prompt, potentially letting attackers exploit backend systems via insecure functions and data stores reachable through the LLM.
Indirect prompt injections occur when an LLM accepts input from attacker-controllable external sources such as websites or files. The injected content hijacks the conversation context, turning the LLM into a “confused deputy.” Such injections need not be human-visible — they only need to be parsed by the LLM.
Common examples
- Because the LLM uses natural language, it does not segregate instructions from external data — it treats both as user-provided input, so there is no fool-proof prevention within the model itself.
- A user asks an LLM to summarize a webpage containing an indirect injection, causing the LLM to solicit sensitive information and exfiltrate it via JavaScript or Markdown.
- A malicious user uploads a resume containing an injection instructing the LLM to report the candidate as excellent; an internal reviewer's summary is manipulated.
- A user enables an e-commerce plugin; a rogue instruction on a visited website exploits it to make unauthorized purchases.
- Rogue instructions and content on a visited website exploit other plugins to scam users.
Prevention & mitigation
- Enforce privilege control on LLM access to backend systems. Give the LLM its own API tokens and follow least privilege, restricting it to the minimum access needed.
- Add a human in the loop for privileged operations (e.g. sending or deleting emails) so the user must approve the action.
- Segregate external content from user prompts — denote where untrusted content is used (e.g. ChatML for OpenAI API calls) to limit its influence.
- Establish trust boundaries between the LLM, external sources, and extensible functionality. Treat the LLM as an untrusted user, keep final decisions with the human, and visually highlight potentially untrustworthy responses.
- Monitor LLM input and output periodically to check it behaves as expected.
Example attack scenarios
- An attacker gives a support chatbot a direct injection (“forget all previous instructions”) plus new instructions to query private data stores and exploit a backend email function — leading to remote code execution, unauthorized access, and privilege escalation.
- An attacker embeds an indirect injection in a webpage telling the LLM to use a plugin to delete the user's emails; summarizing the page triggers the deletion.
- A webpage instructs the model to insert an image linking to a URL that contains a summary of the conversation, causing the user's browser to exfiltrate the private conversation.
- A resume with a prompt injection makes the LLM answer “yes, a good candidate” regardless of the actual contents.
- An attacker asks a proprietary model to repeat its system prompt; the model leaks it, enabling further, more subtle attacks.
References
- ChatGPT Plugin Vulnerabilities – Chat with Code: Embrace The Red
- Prompt Injection attack against LLM-integrated Applications: arXiv
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection: arXiv
- Inject My PDF: Prompt Injection for your Resume: Kai Greshake
- Threat Modeling LLM Applications: AI Village
- Universal and Transferable Attacks on Aligned Language Models: LLM-Attacks.org
How Neural Industries helps. We test, exploit, and remediate Prompt Injection as part of a full OWASP LLM Top 10 assessment — then hand you the evidence to prove it.
Request an assessment →