Misinformation
A model produces incorrect, incomplete, or misleading output that looks credible enough to be trusted and acted upon — driving a wrong decision, a wrong tool call, or a wrong action. The incident record placed it near the top; the 2026 list seats it in the middle.
What it is
Misinformation occurs when a model or model-enabled application produces incorrect, incomplete, unsupported, or misleading information that looks credible enough to influence a human decision, an automated workflow, or an agent action. The core risk is that the incorrect output is trusted and acted upon. When model output drives tool calls, generates code, infers system state, or coordinates across agents, misinformation becomes a system-level failure that can cause financial loss, security incidents, or safety harm.
This entry evolves the old Overreliance risk. Overreliance remains the key factor: humans and systems treat fluent, confident, well-structured output as authoritative, and in agentic architectures that overreliance is frequently baked into the design.
Misinformation can arise from hallucination, stale or incomplete context, weak grounding, ambiguous prompts, biased data, or unvalidated tool output — and can be deliberately induced. Where the root cause is prompt injection, poisoning, or supply-chain compromise, those entries own it; executing unsafe generated code is LLM10 Improper Output Handling, and registering hallucinated package names is LLM04 Supply Chain. This entry focuses on the resulting failure: a false representation that drives a harmful action.
Common examples
- Unsupported or false decision support influencing business, legal, healthcare, financial, or operational decisions.
- Incorrect state inference: the model concludes a condition has been met when it has not, triggering an unintended action.
- Fabricated code and dependencies: incorrect code or references to non-existent (hallucinated) packages.
- Misleading summaries: key constraints, exceptions, timestamps, or risks omitted.
- Adversarially induced misinformation: crafted inputs cause false claims or the omission of critical facts.
- Cross-agent propagation: an incorrect output is trusted and repeated by downstream agents.
- Forged or misattributed evidence presented as authoritative.
Prevention & mitigation
- Ground claims before action. Require outputs to be grounded in authoritative, current sources.
- Separate generation from execution. Use claim-check-act patterns: verify claims before acting on them.
- Validate tool calls. Check arguments, authorisation, preconditions, and current state before execution.
- Use verification signals, not just confidence. Incorporate groundedness and consistency checks rather than trusting a fluent tone.
- Enforce runtime verification for high-impact actions. Introduce approval workflows and independent system checks.
- Detect omission failures. Require structured output with mandatory fields so critical facts cannot silently drop.
- Limit blast radius. Apply least privilege, sandboxing, and rate limits so a wrong answer is not a catastrophic action.
- Calibrate human and system trust. Distinguish verified facts from assumptions, and design against automation bias.
- Test adversarially and continuously. Log claims, evidence, and outcomes, and regularly test workflows against misleading scenarios.
Example attack scenarios
- A coding assistant recommends a plausible but non-existent package that an attacker has pre-registered, so a trusting developer installs attacker-controlled code.
- A customer-service agent misreads a policy and approves a refund that violates the terms, causing financial loss.
- A clinical summary omits a drug contraindication and a clinician acts on the incomplete recommendation.
- An attacker seeds a support forum with false remediation steps that a troubleshooting agent retrieves and repeats as trusted advice.
- A security agent misclassifies normal traffic as an intrusion and automatically blocks a production network segment, causing an outage.
- A retrieval agent reports a customer as identity-verified when they are not, and a downstream payment agent releases funds.
- An agent reports that a nightly backup completed when it never ran, and a later restore fails because no backup exists.
References
- OWASP Top 10 for LLM Applications 2026 — LLM07:2026 Misinformation: genai.owasp.org
- Spracklen et al. — Package hallucination in code assistants (2025)
- NIST AI 600-1 (Generative AI Profile) — confabulation risks
- OWASP LLM09:2025 Overreliance (predecessor entry)
How Neural Industries helps. We stress-test where confident-but-wrong output turns into real-world action across your workflows and agents, then help you add grounding, claim-check-act separation, and verification gates on the high-impact paths.
Request an assessment →