The AI Security Checklist

Never pass raw LLM output into a shell, database, browser, or downstream API without validation and encoding. An LLM's response can be steered by whoever controls its context, so adopt a zero-trust stance toward it — insecure output handling is how a clever prompt becomes XSS, SSRF, or remote code execution.

There is no fool-proof in-model fix: LLMs can't reliably separate instructions from data. Segregate and clearly label untrusted content, constrain what the model is allowed to do, and keep humans in control of consequential decisions. Andrej Karpathy has called prompt injection one of the defining unsolved security problems of the LLM era.

Risk compounds when one agent has all three of: access to private data, exposure to untrusted content, and a way to exfiltrate (outbound requests or tool calls). Willison's rule is to remove at least one leg — isolate the sensitive data, sanitize the inputs, or cut the exfiltration path — so a single injection can't chain into a breach.

Excessive functionality, permissions, or autonomy is the root cause of agent damage. Give each tool the narrowest scope it needs, connect with scoped credentials (OAuth with minimum scope, read-only where possible), and prefer purpose-built tools over open-ended “run any command” capabilities.

Moving money, deleting data, emailing customers, changing configuration — gate these behind explicit human confirmation, enforced in the downstream system rather than left to the model's judgment. Validate every requested action against policy before it executes.

Retrieved documents are attacker-reachable input. Treat indexed content as untrusted, isolate tenants in your vector store, scan and validate documents for indirect prompt injection, and tightly control who can write to the knowledge base your model reads from.

Vet pre-trained models, datasets, and dependencies; sign and verify models; and maintain an ML-BOM / SBOM. As adversarial-ML research has repeatedly shown, poisoned training data and compromised components can plant backdoors long before your application ever runs.

Don't train or fine-tune on secrets; apply least privilege to what the model can reach; scrub and filter both inputs and outputs; and set clear terms so user data isn't silently used for training. Assume anything placed in context can surface in output.

Adversarially test for jailbreaks, injection, and data exfiltration before attackers do — and wire evaluations into CI so every model, prompt, or tool change is re-tested. Securing a probabilistic system is a continuous process, not a one-time gate.

No single control is sufficient. Combine input filtering, content moderation, structured-output validation, and canary / tripwire tokens so that when one layer is bypassed, another still catches the attack. Assume every individual guardrail will eventually fail.

Cap resource use to prevent denial-of-service and “denial-of-wallet”; watch for extraction-style querying that clones your model; and keep a complete, tamper-evident audit trail of prompts, tool calls, and outputs to power detection and incident response.

Map your controls to an established framework, assign clear ownership, document your risk, and maintain an incident-response plan for AI-specific failures. Interpretability and accountability aren't paperwork — they're what let you deploy at scale with confidence.