Overreliance
Trusting LLM output without oversight — leading to misinformation, insecure code, legal issues, and reputational damage.
What it is
Overreliance can occur when an LLM produces erroneous information and provides it in an authoritative manner. While LLMs can produce creative and informative content, they can also generate content that is factually incorrect, inappropriate, or unsafe — referred to as hallucination or confabulation.
When people or systems trust this information without oversight or confirmation, it can result in a security breach, misinformation, miscommunication, legal issues, and reputational damage.
LLM-generated source code can introduce unnoticed security vulnerabilities, posing a significant risk to the operational safety and security of applications. These risks underline the importance of rigorous review processes, continuous validation mechanisms, and disclaimers on risk.
Common examples
- The LLM provides inaccurate information stated in a fashion implying it is highly authoritative; without proper checks and balances, the information misleads the user in a way that leads to harm.
- The LLM suggests insecure or faulty code, leading to vulnerabilities when incorporated into a software system without proper oversight or verification.
Prevention & mitigation
- Regularly monitor and review LLM outputs; use self-consistency or voting techniques to filter out inconsistent text.
- Cross-check LLM output against trusted external sources.
- Enhance the model with fine-tuning or embeddings to improve output quality (prompt engineering, PET, full model tuning, chain-of-thought).
- Implement automatic validation that cross-verifies generated output against known facts or data.
- Break down complex tasks into subtasks assigned to different agents, reducing hallucination and improving accountability.
- Clearly communicate the risks and limitations of using LLMs.
- Build APIs and interfaces that encourage safe use — content filters, user warnings about inaccuracies, and clear labeling of AI-generated content.
- When using LLMs in development, establish secure coding practices and guidelines to prevent integrating vulnerabilities.
Example attack scenarios
- A news organization heavily uses an LLM to generate articles; a malicious actor exploits this over-reliance, feeding misleading information and spreading disinformation.
- The AI unintentionally plagiarizes content, leading to copyright issues and decreased trust.
- A development team over-relies on the AI's suggestions, introducing security vulnerabilities via insecure defaults or recommendations inconsistent with secure coding practices.
- An LLM suggests a non-existent code library; a developer trusts it and unknowingly integrates a malicious package — highlighting the need to cross-check third-party code.
References
- Understanding LLM Hallucinations: Medium
- How Should Companies Communicate the Risks of Large Language Models to Users?: Tech Policy Press
- A news site used AI to write articles — It was a journalistic disaster: The Washington Post
- AI Hallucinations: Package Risk: Vulcan
- How to Reduce the Hallucinations from Large Language Models: The New Stack
How Neural Industries helps. We test, exploit, and remediate Overreliance as part of a full OWASP LLM Top 10 assessment — then hand you the evidence to prove it.
Request an assessment →