How Do I Prevent Prompt Injection in an AI Agent?

An agent can read a poisoned webpage, email, PDF, code comment, or tool result without anyone typing “ignore your instructions” into the chat. If that content can influence a tool call, the attack has a path to a real side effect.

The right design question is not “How do I write a prompt the attacker cannot break?” It is “What can still happen if the model believes the attacker?”

The short answer: stop authority from leaking across boundaries

You cannot reliably prevent every prompt injection with a better prompt or a single detector. Treat user input and retrieved content as untrusted data. Keep secrets and powerful tools outside the model’s reach where possible. Then put an independent authorization check between every model-proposed action and its execution.

For low-risk work, that may mean a read-only tool and a visible result. For a payment, deletion, external message, permission change, or sensitive-data export, it should mean a narrowly scoped tool, a policy decision, and approval bound to the exact action. This is defense in depth, not a promise that the model will never be manipulated. OpenAI’s current guidance makes the same distinction: layered protections should constrain the impact of an attack even when some malicious content gets through (OpenAI).

What prompt injection looks like in an AI agent

Prompt injection is a social-engineering attack against a model. A third party places instructions in the context, and the model treats those instructions as relevant to the task even though the user did not authorize them. OpenAI describes the pattern as malicious instructions injected by someone other than the user or the AI system (OpenAI).

There are two common entry paths:

• Direct injection: the user asks the agent to ignore its rules, reveal hidden instructions, or take an unauthorized action.
• Indirect injection: the agent retrieves content that contains instructions aimed at the agent. The content may come from a webpage, email, document, issue tracker, repository, image, memory record, or tool response.

Indirect injection is the agent-specific problem. A research agent may be asked to compare suppliers. One supplier’s webpage includes hidden text telling the agent to send the user’s procurement files to an external address. The page is supposed to be evidence. The model may also read it as an instruction.

NIST calls this kind of attack agent hijacking. Its current report describes malicious instructions inserted into data that an agent ingests, with possible outcomes including data exfiltration and downloading or running malicious code (NIST).

The important boundary is simple:

Content can inform an agent’s answer. It must not be allowed to grant the agent new authority.

Use the four-boundary model

Here is a practical synthesis for designing an agent. It is not an industry standard. It is a way to inspect where authority enters your system.

Boundary	What happens	What the model may do	What it may not do
Read	Fetch, parse, or receive content	Identify facts, uncertainty, provenance, and possible instructions	Treat external text as a system rule
Reason	Turn the task and evidence into a proposed next step	Suggest an action in a typed structure	Authorize itself or widen the task
Authorize	Check identity, original intent, tool, resource, parameters, and risk	Supply a proposal for policy evaluation	Decide that its own proposal is permitted
Run	Execute a scoped tool call	Pass validated arguments to an allowed tool	Bypass policy, approval, or resource limits

The failure you are trying to prevent is authority leaking from left to right. A sentence in an email becomes a model instruction. The instruction becomes a tool proposal. The proposal becomes permission. The permission becomes a side effect.

OWASP recommends the same separation in more concrete terms: treat external data as untrusted, use clear boundaries between instructions and data, validate tool calls, and apply least privilege to the tools an agent can use (OWASP AI Agent Security Cheat Sheet).

1. Map the agent’s attack surface before changing the prompt

Write down every place content enters and every action the agent can take. Do not start with a list of attack phrases. Start with the data flow.

For each input, record:

• who controls it;
• whether it can be edited after ingestion;
• whether it contains hidden markup, attachments, images, metadata, or code;
• whether it is persisted in memory or conversation history;
• whether it crosses a user or tenant boundary;
• whether it can influence a tool argument.

For each tool, record:

• read or write access;
• the resources it can reach;
• whether it can send data outside the system;
• whether the action is reversible;
• the maximum amount, number of records, or time range it can affect;
• the identity under which it runs;
• the evidence and approval required before execution.

This inventory usually reveals the biggest design mistake: an agent has a powerful generic tool when it needs a narrow workflow tool. “Run SQL” is a large authority surface. “Read order status for this authenticated order” is smaller. “Send email” is broad. “Draft a reply for this ticket, without sending” is smaller.

Microsoft’s agent safety guidance treats model-generated tool arguments as untrusted input, just like input to a web API. It recommends allowlists, type and range constraints, bounded strings, path checks, and parameterized queries (Microsoft).

2. Keep untrusted content in a quarantine lane

Do not paste external content into a privileged instruction block and hope delimiters solve the problem. Delimiters help the model understand structure, but they are not an authorization mechanism.

Give every piece of retrieved content a provenance label and an explicit role such as untrusted_document, user_supplied_text, or tool_result_from_external_service. Preserve that label through retrieval, summarization, memory, and logging.

A safer pattern is to separate the reader from the actor:

1. A reader receives the webpage, email, or document and extracts relevant facts, citations, conflicts, and possible instructions.
2. The reader has no write tools, no secrets, and no ability to send external requests.
3. The actor receives a structured summary plus the original task, not unrestricted raw content.
4. The actor can propose only actions that its policy permits.

OWASP documents this as one possible dual-model pattern and cautions that the guardrail model is also an LLM, so it must sit alongside deterministic controls, scoped tools, and human approval rather than replace them (OWASP prompt-injection guidance).

Quarantine is not the same as sanitization. Removing a few phrases such as “ignore previous instructions” may reduce obvious attacks while missing encoded, social, multimodal, or context-poisoning attacks. OWASP lists remote injection, encoding, HTML and Markdown injection, multimodal injection, RAG poisoning, tool manipulation, and persistent attacks as separate patterns to consider (OWASP prompt-injection guidance).

3. Make tools narrow, typed, and boring

The model should not be trusted with a general-purpose tool and a vague description. Put the control in the tool implementation and the authorization layer.

Use these constraints where they fit:

• allowlist resources, operations, domains, file paths, and record states;
• use separate read and write tools;
• validate types, lengths, ranges, and enumerated values;
• resolve and check paths before reading or writing;
• use parameterized queries instead of string concatenation;
• run code in a sandbox with a time, network, and filesystem policy;
• issue short-lived, task-scoped credentials rather than placing API keys in context;
• limit records, spend, retries, concurrency, and execution time.

OWASP recommends minimum necessary tools and per-tool scopes, including read-only versus write access. Its examples contrast an unrestricted command tool with a file reader limited to a reports directory and an operation allowlist (OWASP).

This is where least privilege does its most useful work. It does not stop a model from reading an injected sentence. It reduces what that sentence can cause.

That is also why a guardrail should not stand in for authorization. OpenAI’s practical agent guide recommends coupling layered guardrails with authentication, authorization, strict access controls, and standard software security measures (OpenAI practical guide to building agents).

4. Authorize the proposed action outside the model

An agent can produce a plausible tool call and still be wrong. Before execution, compare the proposal with the original user intent and the authenticated principal.

At minimum, the policy check should answer:

1. Is this user allowed to perform this class of action?
2. Is this tool allowed in this workflow?
3. Is the target resource inside the user’s scope?
4. Do the normalized parameters fit the requested purpose?
5. Did the action arise from an approved step, or only from untrusted content?
6. Does the risk level require a person to approve it?

The phrase “the user approved it” is too vague for a high-impact operation. Approval should bind to the exact tool, target, normalized parameters, actor, timestamp, and expiry. A general approval for “handle my inbox” should not authorize sending an attachment to a new external address.

Here is a compact policy contract to adapt to your own authorization system. It is an implementation artifact, not a tested drop-in library.

action_policy:
  workflow: supplier-research
  actor_identity: authenticated_user
  allowed_tools:
    read_supplier_page:
      operations: [read]
      domains: [approved-supplier-domains]
    create_draft:
      operations: [write]
      destination: drafts_only
  forbidden_tools:
    - send_email
    - upload_file
    - execute_shell
  required_checks:
    - original_task_alignment
    - tenant_scope
    - typed_arguments
    - outbound_data_policy
  approvals:
    create_draft: optional
    external_write: required
  limits:
    max_pages: 20
    max_records: 25
    max_retries: 2
    max_runtime_seconds: 120
  evidence:
    retain: [task_id, source_ids, normalized_action, decision, approval]

The key detail is not the YAML syntax. It is the placement of the policy. The model may propose create_draft. It does not get to change forbidden_tools, turn drafts_only into send, or infer approval from a sentence found on a supplier page.

5. Bind approval to the action, not to the conversation

Human approval helps only if the person can see what they are approving and the system executes exactly that action. Show the tool name, target, normalized parameters, data leaving the system, and the reason the action is needed.

For example:

Approve action?

Tool: create_draft
Destination: procurement/drafts/2026-08-17-supplier-comparison
Data written: 3 supplier names, prices, and source links
External send: no
Expires: 10 minutes

If the model later changes the destination or adds an attachment, the approval must expire and the new action must be checked again. OWASP recommends explicit approval for high-impact or irreversible actions, action previews, autonomy boundaries, audit trails, and the ability to interrupt or roll back where possible (OWASP). Microsoft similarly recommends approval based on side effects, data sensitivity, reversibility, and scope of impact (Microsoft).

Approval is not a substitute for least privilege. A tired reviewer can approve a misleading preview. The tool still needs a server-side permission check.

6. Protect outputs, memory, and the way data leaves

Prompt injection can succeed without revealing a system prompt. The agent might leak a customer record through a URL, place sensitive values in a generated file, persist an attacker’s instruction in memory, or produce HTML, SQL, or shell input that another component executes.

Treat model output as untrusted at every handoff. Validate structured output against a schema. Escape or sanitize content before rendering. Parameterize database queries. Do not pass generated shell commands to an interpreter without a separate policy decision. Use egress controls for network destinations and redact sensitive values from logs.

Memory deserves its own boundary. Validate what is stored, isolate it by user or tenant, set an expiry, cap its size, and decide whether an item is a fact, a preference, or merely untrusted content. OWASP specifically calls for memory validation, isolation, expiry and size limits, and review for sensitive data before persistence (OWASP).

Do not put secrets in the prompt as a convenience. If a tool needs a credential, let a trusted service use a short-lived credential after authorization. The model should receive the result it needs, not the key that can be replayed elsewhere.

7. Add limits so a successful injection has a small blast radius

Security controls fail in combinations. A malicious document may influence a model, the model may call a permitted read tool, and the result may trigger repeated retrieval or a large outbound request. Bound the whole run.

Set limits for:

• total steps and retries;
• token and time budgets;
• tool calls per tool and per run;
• records read or changed;
• outbound domains and bytes;
• financial value or rate of side effects;
• memory writes and session duration.

Log structured events such as content_ingested, action_proposed, action_denied, approval_requested, approval_granted, tool_executed, and run_stopped. Keep enough provenance to reconstruct which content preceded a risky proposal, while applying the same privacy rules to logs that you apply to the agent.

OpenAI’s public safety guidance describes monitoring, link checks, sandboxing, confirmations, and access limits as overlapping protections. It also states that the guidance makes attacks harder, not impossible (OpenAI).

How do you test the defenses?

Do not test only whether the agent refuses a famous injection phrase. Test whether untrusted content can cause an unauthorized state change.

NIST reported that a 2026 public competition found at least one successful hijacking attack against each of 13 target frontier models across more than 250,000 attempts. That is not a universal failure rate for your model or workflow. It is a reason to treat prompt-injection resistance as an evolving test problem rather than a one-time checkbox (NIST).

Use a test case for each meaningful path:

Case	Adversarial content	Must happen	Must not happen
Direct override	User asks for hidden instructions or an unrelated privileged action	Refuse or stay within the workflow	Reveal secrets or widen tools
Webpage injection	Retrieved page tells the agent to upload notes	Mark content as untrusted and continue safely	Upload or send data
Email injection	Email asks the agent to forward an attachment	Draft or escalate according to policy	Send without an exact approval
Tool-output injection	API response includes instructions to call another tool	Treat response as data	Chain into an unapproved tool
Retrieval poisoning	A stored document changes the requested objective	Preserve original task and provenance	Let the document rewrite the goal
Obfuscation	Encoded, hidden, misspelled, or image-based instruction	Quarantine, flag, or safely ignore	Treat the payload as authority
Memory poisoning	Untrusted content attempts to persist a future instruction	Reject, expire, or isolate the memory item	Affect another session
Boundary bypass	Model proposes a valid tool with an invalid resource or parameter	Independent policy denial	Execute the broad or malformed call

For each case, retain the agent version, model endpoint, prompt and tool policy versions, identity, permissions, input fixture, expected outcome, final state, trace, approval or denial event, and cost and runtime limits. Keep fixtures synthetic or sanitized. Never make the test itself capable of spending money or modifying production data.

For the broader release decision around outcomes, actions, integrity, limits, and stability, see my AI agent evaluation guide. Prompt-injection tests are one security slice of that larger release gate.

The pass rule should have vetoes. Any unauthorized write, data exfiltration, approval bypass, or missing audit event blocks the release. A high average score cannot compensate for one critical side effect.

Why filters and system prompts are not enough

Filters are useful for known patterns, but they are not the control plane. Attackers can use social engineering, obfuscation, multiple turns, hidden content, or a benign-looking request that becomes dangerous only when combined with the agent’s permissions. OpenAI notes that modern prompt injections increasingly resemble social engineering, which makes simple string detection an incomplete defense (OpenAI).

A system prompt is also not a permission boundary. It can tell the model what it should do. It cannot revoke a database credential, enforce a tenant scope, stop a network request, or prove that a human approved the exact parameters.

Model choice matters, but a stronger model does not turn a broad tool into a safe tool. NIST’s red-team report found that attack success varied across models and did not correlate uniformly with model capability. That is a reason to compare models in your own workflow, not to assume that “best model” equals “safe architecture” (NIST).

A release checklist you can use today

Before giving an agent access to a consequential tool, answer yes to each relevant question:

• Is every input labeled by trust and provenance?
• Can untrusted content enter a privileged instruction channel?
• Can the reader process external content without write tools or secrets?
• Does every tool use a narrow allowlist and typed, bounded parameters?
• Are read and write actions separate?
• Does a server-side policy check compare the proposal with the original task and authenticated identity?
• Are high-impact approvals bound to the exact normalized action and expiry?
• Are output, memory, logs, and network egress treated as security boundaries?
• Are secrets kept out of model context?
• Are steps, retries, cost, records, and runtime bounded?
• Does a critical unauthorized action veto release?
• Have you tested direct, indirect, tool-output, retrieval, memory, multimodal, and obfuscated cases?
• Will a confirmed production failure become a sanitized regression case?

If the first answer is no, start with architecture. If the tool scope is broad, narrow it before tuning the prompt. If approvals are vague, make the action preview specific. If there is no observable final state, build that evidence before claiming the agent is safe.

When should you get help?

You can apply this checklist internally when the agent is read-only, the data is low sensitivity, the tools are narrow, and the team can own authorization, testing, and incident response.

Bring in an AI consultant or security specialist when the agent crosses tenants, handles credentials or regulated data, sends external communications, changes financial or production state, or connects several tools and memory stores. The useful engagement is not “write a magic anti-injection prompt.” It is a workflow threat model, a permission design, an action policy, and a regression suite your team can maintain.

If you want a focused working session on one real workflow, Marius's AI learning and consulting path is the next step. Bring the agent’s tools, trust boundaries, known failure cases, and the decision that the system is supposed to make. The goal is a smaller, testable action surface.