Should I Use RAG or an AI Agent for Internal Knowledge?

The expensive mistake is treating RAG and an AI agent as competing products. They solve different problems. RAG supplies a model with relevant internal evidence; an agent decides which step or tool to use next.

For most internal knowledge projects, begin with the smallest useful system: permission-aware retrieval, grounded answers, citations, and a clear way to abstain. Add an agent when a real workflow needs dynamic source selection, multi-step investigation, or a controlled action. The agent can use your RAG system as one of its tools.

Should you choose RAG, a fixed workflow, or an agent?

Use standard RAG when an employee asks a question that can be answered from one approved knowledge base or index. The system retrieves relevant passages, puts them into the model's context, and returns an answer with source information. That is the basic RAG pattern described in the original RAG research and in practical guidance from Anthropic.

Use a fixed workflow with retrieval when the steps are known but one or two steps need language understanding. For example: find the policy, extract the relevant rule, check a date with ordinary code, and draft a response for a person. The system may contain an LLM, but its control flow remains visible in code.

Use an agent with retrieval tools when the system must decide what to investigate next. That means choosing among different sources, breaking a question into dependent searches, checking whether the first result is sufficient, querying a live system, or retrieving information before taking an approved action. Microsoft's agentic RAG guidance describes retrieval as a tool that an agent can call, evaluate, and call again.

The practical sequence is:

standard RAG -> fixed workflow plus RAG -> agent plus RAG and other tools

Move right only when the job requires the next level. An agent is not an upgrade to RAG. It is an additional control loop with more decisions to secure and evaluate.

RAG is a knowledge-access pattern; an agent is a workflow-control pattern.

Anthropic makes the same distinction in a useful warning about system design:

“Consistently, the most successful implementations use simple, composable patterns rather than complex frameworks.”

Anthropic, “Building effective agents”

That sentence is a good starting constraint. Before choosing a platform, write down what the system must know, what it must decide, and what it may change. If the path is stable, encode it. If the path changes because of evidence, consider an agent. If neither the evidence nor the desired outcome is clear, do more knowledge and workflow discovery first.

What does RAG do for internal knowledge?

Retrieval-augmented generation is a knowledge-access pattern. A typical system breaks a corpus into chunks, creates searchable representations, retrieves relevant chunks for a question, and passes those chunks to a language model. Semantic search can find conceptually related passages; keyword search can protect exact identifiers such as policy codes or error numbers. Anthropic describes combining semantic retrieval with lexical matching for that reason in its Contextual Retrieval guidance.

RAG is useful because a model's built-in training memory is not your current handbook, pricing sheet, incident runbook, or access-controlled company record. Retrieved context gives the response something specific to use and cite. It does not make the answer automatically true. The corpus can be stale, the search can return the wrong chunk, and the model can still misread or overstate the evidence. The original RAG paper identifies provenance and updating world knowledge as open problems (Lewis et al.).

An internal RAG system normally has more parts than the phrase suggests. It needs a source inventory, an ingestion rule, a document identity, a chunking strategy, an index, a retrieval query, an authorization filter, a prompt or context assembly step, an answer policy, and an evaluation set. The model is only one component. A better model cannot compensate for a source that has lost its effective date or a retrieval result that crosses a user's permission boundary.

The answer path can be written as a small contract:

question
  -> identify user and access scope
  -> search approved index
  -> return passages with source metadata
  -> decide whether evidence is sufficient
  -> answer with citations or abstain

This path is deliberately boring. Boring is valuable when the business question is, “What does our current travel policy say?” The reader needs a dependable answer, not a system that explores every connected application.

The contract also exposes the limits. If retrieval returns two conflicting policy versions, the answer should not quietly select one because it sounds more plausible. The index should carry title, owner, version, effective date, and location. The answer policy should tell the model what to do when versions disagree. A person may need to resolve the conflict before the assistant can answer.

The same principle applies to internal technical knowledge. An incident runbook may include commands, but the assistant should not turn a retrieved command into an external action merely because the sentence is imperative. Retrieval makes information available. A separate policy decides whether information can be transformed into a draft, recommendation, or action.

RAG is therefore a strong starting point when the reader's job is knowledge access. It is not a promise of truth, and it is not a substitute for document ownership. Treat it as a controlled evidence path.

How is an AI agent different from a RAG system?

An agent is a workflow-control pattern. OpenAI's practical guide to building agents describes agents as systems that independently accomplish tasks, use tools, and make decisions within guardrails. Anthropic's guide draws a related distinction: a workflow follows code paths that developers define, while an agent dynamically directs its process and tool use.

The important word is control. A RAG pipeline usually has a prescribed path: retrieve, place context, answer. An agent can choose a search tool, inspect its result, decide that another source is needed, ask a follow-up question, call a calculation tool, and then stop or escalate. That flexibility can solve a harder job, but it creates more places for the system to make a wrong decision.

An agent may use several kinds of tools:

Tool type	What it does	Internal knowledge example	Main control question
Retrieval	Reads approved information	Search policy documents or tickets	Can this user see the returned source?
Data query	Reads structured live state	Check an employee record or inventory value	Is the result current and scoped?
Calculation	Applies deterministic logic	Compare dates, totals, or thresholds	Can the result be reproduced outside the model?
Drafting	Produces a proposed artifact	Prepare an HR case or incident summary	Who reviews the draft before use?
Action	Changes an external system	Open a ticket or update a record	Is the action exact, authorized, and reversible?

The distinction between data tools and action tools matters. OpenAI's guide separates tools that gather context from tools that change external systems. The same separation should appear in your architecture, logs, permissions, and user interface. A retrieval call should not silently carry the authority to send a message.

An agent also needs a run boundary. In plain language, a run starts with a request and ends when the system returns an answer, reaches a defined stopping condition, hits an iteration limit, fails a policy check, or hands control to a person. Without a run boundary, “investigate until you are sure” becomes an unbounded loop.

The agent's decision record should make the path inspectable without pretending that private model reasoning is a reliable audit log. Record the user request, tool name, sanitized arguments, authorization result, returned source identifiers, tool outcome, final decision, stop reason, and human approvals. The record should let an operator answer, “What did the system access and what did it do?”

An agent is appropriate when the workflow contains genuine branching. It is not appropriate merely because the interface is conversational, the data set is large, or a vendor calls its retrieval product agentic. A chat interface can sit on top of ordinary RAG. A large document corpus can still be one source. The architecture follows the job.

When should you use the SCOPE test?

SCOPE is a decision test for an internal knowledge project. It is an original synthesis of the retrieval and agent distinctions in the sources above, not an industry standard or a benchmark. Use it before comparing vendors or granting tools.

The letters stand for:

1. Source boundary: Where can the answer come from, and who owns each source?
2. Complexity: Does one search usually provide enough evidence, or does the next search depend on the first result?
3. Outcome: Does the user need an answer, a draft, a recommendation, or a change in another system?
4. Permission: Can the first version be read-only, scoped, and stopped before a hard-to-reverse side effect?
5. Evaluation: Can someone describe a correct answer, a correct tool path, and a safe stopping condition?

Run the test in order. Source boundary comes before tool selection because a system cannot be safe if nobody knows which source wins a conflict. Complexity comes before autonomy because a multi-step job does not automatically require model-directed control. Outcome and permission expose the risk of writes. Evaluation tells you whether the proposal can be tested at all.

The result should be a sentence, not a score. For example: “The answer comes from one approved policy repository, one search is normally enough, the output is read-only, permissions can be enforced at retrieval, and we have 80 representative questions.” That sentence points toward standard RAG.

Another result might say: “The system must compare an approved policy with a live record, the first result determines which record to inspect next, the output is a draft case, and a person approves any write.” That result points toward a supervised agent with retrieval and data tools.

If the SCOPE answers contain phrases such as “search everything,” “use whatever source seems useful,” or “take action when appropriate,” stop. Those are not architecture requirements. They are missing policy. Replace them with named sources, tool contracts, owners, and stopping rules.

How should source boundaries shape the architecture?

The first question is not whether the company has ten thousand documents. The first question is whether the answer can come from one approved source boundary. A single well-owned repository may support standard RAG even when it contains a large corpus. Conversely, three small systems with different owners, freshness rules, and access models can create an agentic investigation problem.

Separate the idea of a corpus from the idea of a source. Many pages in one versioned policy library may be one source for architecture purposes. A document library, a ticket database, and a live payroll API are different sources because they have different retrieval methods, data owners, update patterns, and authorization concerns.

Write a source contract for every system you plan to connect:

Field	Example question	Why it matters
Owner	Which team is responsible for accuracy?	Gives conflicts and refreshes a clear route
Authority	Is this source allowed to answer policy questions?	Stops a convenient copy from outranking the real source
Freshness	How quickly can a change reach the index?	Sets the effective-date and stale-data rule
Identity	Which user identity reaches the source?	Keeps access decisions tied to the requester
Metadata	Which title, date, ID, and section are returned?	Makes citations and audits possible
Failure mode	What happens when the source is unavailable?	Prevents a missing result from becoming an invented answer

The source boundary should also define what does not belong. A shared drive may contain drafts, private notes, duplicates, and old exports. “The assistant searches the shared drive” is not a source policy. It is an invitation to create uncertainty. Select authoritative locations, exclude known draft areas, preserve version metadata, and decide how superseded content is treated.

Microsoft's secure multitenant RAG guidance makes the security boundary explicit: only data a user is authorized to access should become grounding data. A vector index does not inherit a company's access model by magic. Carry identity into retrieval, filter before context reaches the model, and test the result with users who should see different documents.

There are two common permission mistakes. The first filters after retrieval. That can expose protected text to an intermediate service or to the model even if the final answer hides the passage. The second applies a broad service account and asks the model to “respect permissions” in a prompt. A prompt is not an authorization system. Enforce the boundary in code or in a source that can prove the filter was applied.

The architecture should make cross-source comparison explicit too. Suppose a policy repository says a request is allowed, while an employee record shows that a prerequisite is missing. The system needs a conflict rule: identify the disagreement, show the supporting source metadata, and escalate to the owner or reviewer. An agent may help collect the evidence, but it should not silently settle an ownership dispute.

Permission filtering must happen before retrieved text enters the model context.

This rule narrows the design in a useful way. It may rule out a fast prototype that dumps an entire private index into one shared prompt. That is a good result. Internal knowledge systems earn trust through visible boundaries, not through a confident answer at the end of an uncontrolled path.

How much retrieval complexity justifies an agent?

A single query against one index points toward standard RAG. Dependent searches, dynamic source selection, and iterative refinement point toward agentic retrieval. Microsoft lists these patterns in its agentic RAG architecture guidance and notes that each additional step brings latency, token use, and implementation complexity.

A useful test is to write the path without using the word agent:

1. What does the user ask?
2. Which source must be searched first?
3. What result changes the next question?
4. What evidence is sufficient?
5. What happens when the evidence is incomplete or contradictory?
6. What ends the run?

If the answers are stable for every request, code the path. You may still use a language model for classification, extraction, or drafting, but the model does not need to own the process. A fixed workflow is easier to inspect, replay, and test.

If the answers change with the evidence and you can define safe stopping conditions, an agent may earn its place. For example, a procurement question might begin in a policy index. If the policy requires a current supplier record, the system then queries a structured database. If the record is missing, the system asks for a supplier ID or escalates. The next tool depends on what the first tool returns.

Do not use document count as a proxy for retrieval complexity. One million well-structured documents in one index can be a search quality problem. Three small systems can be a workflow problem. The difference is whether the system must decide between sources and whether a result changes the next operation.

You can classify complexity with four path shapes:

Path shape	Example	Recommended control
Single retrieval	Answer a current leave-policy question	Standard RAG with citations and abstention
Fixed sequence	Retrieve policy, extract rule, calculate date, draft reply	Code-controlled workflow with model steps
Conditional sequence	Search policy, then query a record only if a condition appears	Fixed workflow with explicit branches
Open investigation	Choose sources, compare results, ask for missing evidence, prepare a case	Bounded agent with tool limits and review

The third row is often overlooked. A conditional workflow can handle meaningful branching without giving a model broad authority to invent new branches. Use an agent for the part that truly needs dynamic choice, not for the entire system by default.

An agent earns its place when evidence changes the next step.

That is a stronger test than “the task is complex.” Complexity can mean messy data, a long document, or a difficult answer. An agent adds value only when model-directed choices help navigate that complexity and those choices can be bounded.

What outcome should the system produce?

The desired outcome determines risk more directly than the name of the architecture. Ask whether the user wants an answer, a summary, a recommendation, a draft, or a change in an external system.

Answers and summaries can often stay inside a read-only RAG boundary. The assistant should retrieve evidence, state what the source supports, cite the relevant location, and abstain when the corpus does not answer the question. A recommendation needs an additional policy: what criteria should be applied, what assumptions are allowed, and when must a person decide?

A draft sits between information and action. It changes no external state by itself, but a person may trust it too quickly. Label it as a draft, include the supporting sources, show unresolved conflicts, and require the reviewer to confirm the recipient, scope, and material facts. A draft is safer than an automatic send, but it still needs an operating procedure.

An external action has a different boundary. Opening a ticket, changing a CRM record, approving a request, sending a message, or altering a payment instruction can create a side effect that is hard to reverse. Separate the action tool from retrieval. Require an exact target, an explicit payload, an authorization check, an idempotency key where appropriate, and a human approval for consequential operations.

The user interface should match the outcome. A read-only answer can show source cards and an abstention message. A proposed action should show a review packet with the request, evidence, planned tool, exact changes, and approval control. A hidden action behind a friendly answer is a poor design even when the tool call usually succeeds.

Here is a practical action classification:

Outcome	Default release mode	Review expectation
Answer from approved documents	Read-only	Sample quality and access decisions
Structured extraction	Read-only or draft	Validate fields and source locations
Recommendation	Draft or decision support	Named owner makes the final decision
Ticket or case draft	Draft-only	Reviewer checks evidence and target
Ticket or record creation	Human-approved action	Exact payload and audit trail
Financial, legal, or access change	Strong human control	Dual review or existing business control

The architecture may be the same agent in two environments, but the permission envelope should not be. Development can use fake records and a disabled action tool. Evaluation can replay fixtures. Production can expose only the least authority needed for the approved workflow.

How should permissions and reversibility limit an agent?

A pilot should be read-only or draft-only whenever the business job allows it. This is not a claim that write actions are impossible. It is a way to learn whether retrieval, source selection, and evidence presentation work before the system can create a costly side effect.

Define the permission envelope in plain language:

Allowed sources:
Allowed read operations:
Allowed draft operations:
Allowed write operations:
Forbidden operations:
User identity passed to each tool:
Maximum tool calls per run:
Maximum wall-clock time:
Approval required for:
Escalation owner:
Stop and report when:

Do not leave fields blank. “Allowed sources: internal systems” is not specific enough. Name the repository, table, endpoint, or collection. “Approval required for: risky actions” is not specific enough. Name the action classes and the reviewer.

Use least authority at every layer. A retrieval tool should return only the fields required for the answer. A drafting tool should not inherit a send permission. A send tool should require a target and a payload that a reviewer can see. If a tool can both read and write, split it when possible so logs and permissions make the difference obvious.

Reversibility is a design input. Some actions are naturally reversible, such as creating a draft or adding a removable label. Others are not, such as sending a message to a customer, deleting a record, or changing a payment destination. The less reversible the action, the more exact the approval and audit requirements should be.

Build a pre-action check that runs outside the model. It should verify the requester's identity, the target resource, the allowed action, the current version or state, required fields, and any conflict detected during retrieval. The model may propose an action. Deterministic code should decide whether the proposal is eligible to execute.

Use an idempotent action design when the same run may be retried. A retry should not create three tickets because the first response timed out. Use an idempotency key tied to the request and action, check the external system before creating a duplicate, and record the result. If the external tool cannot support this safely, keep the action as a human-executed draft.

The stop path deserves as much attention as the success path. Stop and escalate when identity is missing, sources disagree, a required field is absent, a permission check fails, the tool returns an unexpected schema, the iteration limit is reached, or the proposed action falls outside the contract. “The model will decide whether to continue” is not a sufficient stop policy.

The safest first release is usually read-only or draft-only.

The reader should also know what the assistant did not do. A useful final response can say, “I searched the policy repository and the approved payroll record. I did not create a case because the effective dates conflict. HR review is required.” That is more useful than a polished answer that hides the conflict.

For a deeper treatment of approval boundaries, see the sibling guide on human approval for AI agents. If the system will have tools that can write to business systems, the article on least-privilege tool access for AI agents gives another useful boundary to inspect.

How should you evaluate RAG and agentic retrieval?

Evaluation must match the architecture. A standard RAG system needs tests for retrieval relevance, grounded answer quality, access correctness, abstention, citation accuracy, and response time. An agent adds tests for tool selection, unnecessary searches, stopping behavior, action safety, and cost or latency per run.

Do not evaluate only the final prose. A fluent answer can hide an incorrect source, an unauthorized passage, or an unnecessary write. For each test case, record the expected evidence, expected outcome, forbidden behavior, and acceptable uncertainty.

Build a small evaluation set from real question shapes, with sensitive content removed or replaced by fixtures. Include ordinary questions, ambiguous questions, questions with no answer, conflicting documents, access differences, stale records, malformed tool results, and requests that try to bypass policy. A set of only easy questions tells you that the demo works. It does not tell you whether the system fails safely.

For standard RAG, a useful case record looks like this:

Case ID:
User role:
Question:
Expected source IDs:
Expected answer points:
Expected citation fields:
Expected abstention condition:
Forbidden sources:
Pass decision:
Reviewer note:

For an agent, extend it with path expectations:

Allowed first tool:
Allowed next tools:
Maximum tool calls:
Required stop reason:
Allowed action:
Approval requirement:
Expected audit fields:

The word “expected” does not mean every case has one exact string answer. For an open-ended policy question, expected answer points may be a set of facts and a required caveat. For a source-selection case, the acceptable path may include two equivalent retrieval calls. Define what must be true and what must not happen.

Measure retrieval and answer quality separately. A relevant passage with a poor answer is a generation problem. A confident answer based on an irrelevant passage is a retrieval or source-selection problem. An answer that uses a passage the user should not see is an authorization failure even if the sentence is factually correct.

For an agent, count steps and tool calls. More calls are not automatically better. Unnecessary searches add latency, cost, and opportunities for leakage. A system that reaches the right answer after two relevant calls may be safer than one that searches every connected source before returning the same answer.

Test retries and partial failures. What happens if the index times out after the model has already selected a source? What happens if a data API returns a valid response with a missing field? What happens if an action succeeds but the confirmation is lost? The answer should be a known recovery path, not a second uncontrolled attempt.

Test prompt injection as a data-handling problem. A retrieved document may contain instructions aimed at the model. Treat retrieved content as data unless a trusted policy explicitly says otherwise. Keep tool permissions outside the retrieved text, validate tool arguments, and make the model unable to grant itself authority through a document. The sibling guide on preventing prompt injection in an AI agent covers this boundary in more detail.

Evaluation should test the path, not only the final answer.

Use a release table that makes the decision visible:

Area	RAG release question	Agent release question
Evidence	Did the answer use the right passages?	Did the agent select the right sources in the right order?
Permissions	Did the user see only allowed data?	Did every tool call carry the correct identity and scope?
Quality	Are answer points supported and cited?	Did the final answer preserve conflicts and uncertainty?
Control	Did the system abstain when needed?	Did the run stop at the right condition?
Action	Is there no external side effect?	Was any action exact, approved, and idempotent?
Operations	Can the team replay a failed case?	Can the team inspect, retry, or cancel the run safely?

The owner should sign off on thresholds. A vendor demo should not supply your definition of acceptable. Some workflows can tolerate a missing summary sentence but not a wrong permission decision. Others can tolerate a slow draft but not an incorrect financial amount.

Which architecture should you start with?

Use this matrix to choose a first release rather than a permanent identity for the system:

SCOPE result	Architecture	First release should do	First release should not do
One source, one search, answer only, read-only	Standard RAG	Retrieve small relevant passages, cite title and date, abstain without support	Browse every system or write records
Known sequence, a few language steps, no dynamic planning	RAG plus fixed workflow	Retrieve, extract, validate, route, or draft in a defined sequence	Let the model invent a process for every request
Multiple sources or dependent searches, bounded investigation	Agent plus RAG tools	Select a source, retrieve, assess evidence, stop, and request approval when needed	Give broad write access or unlimited search loops
Unclear source ownership or outcome	Discovery first	Name owners, define the job, clean the corpus, and create test cases	Add autonomy to compensate for ambiguity

The first release should have a narrow promise. “Answer questions about the current travel policy and cite the effective section” is testable. “Help employees with anything in the company” is not. “Investigate a benefits question using policy, contract, and record data, then prepare a draft for HR” is testable if each source and action boundary is named.

Start with one user group and one source boundary. Add a second source only when a real question requires it and the first source contract is working. Each new source adds ownership, freshness, access, and failure questions. Each new action adds approval and recovery questions.

Use a fixed workflow when the branching can be described as ordinary business logic. A workflow can still call a model for classification, extraction, or language generation. The point is that the system controls the sequence and the model fills a bounded role. This approach often produces a smaller log and a simpler test fixture.

Use an agent when the investigation cannot be described without making the next source depend on the current evidence. Even then, keep the agent narrow. Give it a small catalog of retrieval tools, a maximum number of calls, a clear stop condition, and a separate action boundary. The first agent should not be an all-purpose company assistant.

The architecture can evolve. Standard RAG can become a tool inside a fixed workflow. A fixed workflow can expose one bounded branch to an agent. An agent can later receive an action tool after the read-only investigation has passed its evaluation. Move one boundary at a time and retain the earlier tests.

How do you design an internal policy assistant safely?

Consider a 200-person company with policies in a versioned repository. Employees ask about leave, travel, and expenses. The first requested feature is: “Answer the question and link the policy section.”

This is a standard RAG job if the documents are authoritative, the index preserves title and effective date, and user identity controls which documents can be retrieved. The assistant should answer from retrieved text, show the source, and say that it could not find an approved answer when retrieval fails.

The implementation brief should contain at least these fields:

User question:
Policy source:
Source owner:
Effective-date rule:
User identity input:
Required citation: title, effective date, section, URL or record ID
Answer rule:
Abstain when:
Review owner:

Now change the request: “Based on the policy, my employment contract, and this month's payroll record, tell me whether I need to contact HR and open a draft case with the supporting evidence.” The task now has a dependent path. It may retrieve the policy, inspect the contract, query a live record, compare them, and prepare a draft. That is a candidate for an agent with separate retrieval tools and a draft-only action boundary.

The correct first release is still not an autonomous HR agent. It is a supervised investigation flow. The employee sees the evidence, the system states uncertainty or conflict, and a person approves any external case creation. This example is hypothetical; it is not a client result or a performance claim.

The system should treat policy documents and personal records differently. Policy text can be shared within an approved audience. A contract and payroll record may require narrower access. The retrieval tool should apply the requester's identity to each source separately, and the final answer should not reveal a field that was used internally but not authorized for display.

The assistant also needs an effective-date rule. If two documents are retrieved, the newest file is not necessarily the authoritative one. A future policy may be staged but not active. A local exception may override a general rule for one group. The source owner must define how dates, scope, and exceptions interact. If the index cannot express that rule, the assistant should show the conflict and escalate.

An answer template can help keep the result grounded:

Answer:
Evidence:
- [source title], [effective date], [section]
What the evidence does not establish:
Conflict or missing information:
Next safe step:

Do not make every response sound certain. “The policy says X, but the retrieved record is missing the field needed to apply it” is a successful answer when the requested decision cannot be made. The goal is a correct boundary, not a complete-sounding paragraph.

For a draft case, show the proposed fields separately from the evidence. The reviewer should be able to delete, edit, or reject the draft without guessing which sentence came from which source. Store source IDs with the draft metadata. Preserve the original request and the time of retrieval because live records can change.

An evaluation set for this assistant should include questions about each policy category, users with different roles, future-dated policies, revoked access, missing records, contradictory sources, and requests to infer sensitive facts. The release gate should fail if the assistant answers from a document the user cannot access or creates a draft without required evidence.

What should an internal-knowledge pilot card contain?

Fill this out before choosing a framework or granting tool access. It forces the team to write the job in operational terms.

Knowledge job:
Primary user and owner:
User question or trigger:
Desired outcome: answer / draft / recommendation / system change

Authoritative sources:
Source owner for each source:
Source freshness or effective-date rule:
User identity and permission boundary:

Starting architecture: standard RAG / fixed workflow / agent plus retrieval tools
Why a simpler option is insufficient:
Expected retrieval path:
What makes the next step change:
Required citation fields: title / date / section / record ID
Abstain or escalate when:

Allowed tools:
Forbidden tools:
Read actions:
Write actions requiring approval:
Maximum searches or iterations:
Named escalation owner:

Representative questions:
Expected evidence and outcome for each:
Forbidden behavior for each:
Baseline to beat:
Release decision and next review date:

The line “why a simpler option is insufficient” is the most valuable one. If you cannot fill it in, use RAG or ordinary search first. The goal is not to prove that the company needs an agent. It is to avoid building one before the work demands it.

The pilot card should have an owner who can make a decision. “The engineering team owns quality” is too broad. Name the policy owner for source accuracy, the security owner for permission behavior, the workflow owner for the outcome, and the operator who will handle escalation. Those people may be on one team, but the responsibilities should remain visible.

Set a review date before launch. Review the source contract when a repository changes, the data API changes, permissions change, or the business process changes. Review the model and prompt when the output schema changes or evaluation cases regress. A scheduled date helps, but an event-triggered refresh matters more for high-risk systems.

Create a small runbook for failures:

1. Confirm whether the source was available and current.
2. Check whether the user had permission for the expected evidence.
3. Replay the case with a fixture or redacted record.
4. Identify whether retrieval, tool choice, generation, or action validation failed.
5. Correct the smallest responsible boundary.
6. Add the failure to the evaluation set before expanding access.

This runbook keeps the team from “fixing” a retrieval problem by giving the agent more tools. A larger tool catalog can hide a source-quality problem and make future failures harder to diagnose.

How do cost, latency, and operations change the choice?

The decision is not only about model capability. Each retrieval, reranking step, tool call, context expansion, and human review adds operational work. Microsoft notes the latency and token implications of agentic retrieval. Treat these as design inputs, not surprises after launch.

For standard RAG, record at least:

• retrieval latency;
• answer generation latency;
• context size;
• cache behavior;
• abstention rate;
• citation rendering time;
• cost per question, if the vendor bills by usage.

For an agent, add:

• number of tool calls per run;
• time spent waiting on each tool;
• repeated or unnecessary calls;
• number of runs that hit the iteration limit;
• escalation rate;
• action approval time;
• retry and duplicate-action rate.

Do not use a single average. A mean of two quick answers and one 90-second investigation can make the system look acceptable while users experience the slow case as a failure. Report percentiles or at least separate simple and multi-step cases.

Cost also includes source maintenance and review. A cheap model with a weak source contract may cost more in human correction than a slightly more expensive model with better extraction. A complex agent may reduce manual searching while increasing incident investigation work. Estimate the whole workflow.

Use a baseline before adding autonomy. The baseline might be the time a trained employee needs to find and cite the answer, the percentage of requests that require escalation, or the time to prepare a draft case. Compare the proposed system to that baseline using representative cases. Do not compare it only with a blank screen or a vendor demo.

Set limits that reflect the business job. A policy question may allow two retrieval calls and a short response. A research request may allow ten searches but no writes. A draft action may require a person after the evidence is assembled. Limits should be visible in configuration and tested as failure cases.

How can you move from RAG to an agent without a rewrite?

Design the first RAG system as a tool with a stable contract. Give it a typed input such as query, user identity, source scope, date scope, and result limit. Return passages with source ID, title, effective date, section, location, access decision, and relevance metadata. A future workflow or agent can call that tool without knowing the index internals.

Keep the answer policy separate from retrieval. The retrieval tool returns evidence. A grounded-answer component decides whether evidence is sufficient and how to cite it. That separation lets you test retrieval independently and later reuse it in a multi-source investigation.

Keep actions separate too. A fixed workflow may call a draft creator after the grounded-answer step. An agent may later choose whether to call that draft tool, but it should not gain the ability to send or commit simply because the retrieval tool was expanded.

Version the contracts. When the source metadata changes, keep a compatibility layer or update the evaluation fixtures. When a tool's permissions change, review the agent policy and the human approval path. A version number in logs helps operators understand which contract produced a result.

Add autonomy in small increments:

1. Start with one approved corpus and read-only answers.
2. Add citations, effective dates, and abstention checks.
3. Add one fixed workflow for a known sequence.
4. Add a second source only for a defined question class.
5. Expose a bounded retrieval choice to an agent.
6. Add draft creation behind a review boundary.
7. Add an action only after exact payloads, approval, idempotency, and recovery pass evaluation.

At each step, keep the earlier tests. More autonomy should not make the system forget source ownership or permission checks. If the new version fails a test that the smaller version passed, pause the expansion and fix the regression.

This path also helps with vendor changes. A vendor may offer a new agent runtime, connector, or retrieval feature. If your internal contracts are explicit, you can test the feature against the same source, permission, outcome, and evaluation requirements. The product name does not become the architecture.

What mistakes make teams choose the wrong architecture?

Do you mistake a chat interface for an agent?

An internal chat window can sit on top of standard RAG. Conversation does not prove that the model should control the workflow. Define the system by what it can decide and change, not by how the interface looks.

Do you treat RAG as a truth guarantee?

Retrieved text can be stale, incomplete, duplicated, or wrongly permissioned. Preserve source metadata, show citations, filter access before generation, and define an abstention path. A confident answer without supporting evidence is a retrieval failure wearing a language-model voice.

Do you add an agent to compensate for poor knowledge management?

An agent that searches five messy repositories may produce a more complicated explanation of the same ambiguity. Name an owner for each source, mark effective dates, remove superseded policies from the retrieval path, and decide which source wins conflicts.

Do you start with writes?

The first useful agent version is often read-only or draft-only. If the system cannot explain its evidence, do not let it send, approve, delete, purchase, or update. Add the action only after the investigation path, approval packet, and recovery behavior are clear.

Do you choose a platform before writing the path?

Cloud services now offer both classic and agentic retrieval architectures. That is useful, but it can make the product menu look like the decision. Write the SCOPE answers, pilot card, source contract, and baseline first. Then choose the smallest platform that can implement them.

Do you confuse more sources with better answers?

Adding a connector can increase coverage while lowering clarity. More sources create more duplicate documents, conflicting dates, and permission paths. Add a source only when a named question requires it and an owner can maintain it.

Do you let the model decide whether a tool call is safe?

The model can propose a tool call, but an external policy should validate identity, arguments, target, scope, and action class. Prompt instructions are useful behavior guidance. They are not a substitute for authorization and schema validation.

Do you hide uncertainty to make the assistant sound helpful?

An internal user needs to know whether the system found a current answer, found conflicting evidence, or found nothing. State the limitation and offer the next safe step. A clear abstention often saves more time than a confident correction later.

Do you measure only answer quality?

An answer can look right while the system accessed an unauthorized document or used an unnecessary write-capable tool. Evaluate source choice, permissions, stop conditions, and action payloads alongside the words shown to the user.

Do you expand before you have an owner?

A pilot without a source owner, workflow owner, security reviewer, and escalation route becomes nobody's production system. Name those roles before adding data or users. Ownership is part of the architecture.

Which one should you use for internal knowledge?

Use RAG when internal knowledge is the job: retrieve approved evidence, answer a question, cite the source, and stop.

Use RAG plus a fixed workflow when the job has known stages such as retrieve, extract, validate, route, and draft.

Use an agent with RAG and other retrieval tools when the job is investigation or execution: the system must choose sources, decompose the request, iterate based on results, query live systems, or take a bounded action.

The best first architecture is often less impressive than the one in a demo. That is a strength. Start with the smallest system that can answer correctly, respect permissions, and reveal when it does not know. Let the workflow earn more autonomy through evidence.

If you have one real internal workflow and want to work through the SCOPE test with someone, Marius's one-to-one AI consulting is a suitable next step. Bring the question, its authoritative sources, and the current manual path. The outcome should be a clearer architecture decision, not an agent by default.

Continue with a related field note

• Should an Internal AI Assistant Use Retrieval, Tools, or Both?
• When Should an AI Workflow Use a Database Query Before a Model Call?
• Should I Use RAG or SQL for Internal Business Questions?
• Should an AI Integration Use MCP or a Custom API?