Should I Use RAG or an AI Agent for Internal Knowledge?

The expensive mistake is treating RAG and an AI agent as competing products. They solve different problems. RAG supplies a model with relevant internal evidence; an agent decides which step or tool to use next.

For most internal knowledge projects, begin with the smallest useful system: permission-aware retrieval, grounded answers, citations, and a clear way to abstain. Add an agent when a real workflow needs dynamic source selection, multi-step investigation, or a controlled action. The agent can use your RAG system as one of its tools.

The short answer: choose by the job, not the label

Use standard RAG when an employee asks a question that can be answered from one approved knowledge base or index. The system retrieves relevant passages, puts them into the model's context, and returns an answer with source information. That is the basic RAG pattern described in the original RAG research and in practical guidance from Anthropic.

Use a fixed workflow with retrieval when the steps are known but one or two steps need language understanding. For example: find the policy, extract the relevant rule, check a date with ordinary code, and draft a response for a person.

Use an agent with retrieval tools when the system must decide what to investigate next. That means choosing among different sources, breaking a question into dependent searches, checking whether the first result is sufficient, querying a live system, or retrieving information before taking an approved action. Microsoft's current agentic RAG guidance describes this as making retrieval a tool that an agent can call, evaluate, and call again.

The practical sequence is:

standard RAG -> fixed workflow plus RAG -> agent plus RAG and other tools

Move right only when the job requires the next level. An agent is not an upgrade to RAG. It is an additional control loop.

What RAG does, and what an agent does

Retrieval-augmented generation is a knowledge-access pattern. A typical system breaks a corpus into chunks, creates searchable representations, retrieves relevant chunks for a question, and passes those chunks to a language model. Semantic search can find conceptually related passages; keyword search can protect exact identifiers such as policy codes or error numbers. Anthropic describes combining semantic retrieval with BM25-style lexical matching for that reason (Contextual Retrieval).

RAG is useful because the model's built-in training memory is not your current handbook, pricing sheet, incident runbook, or access-controlled company record. Retrieved context gives the response something specific to use and cite. It does not make the answer automatically true. The corpus can be stale, the search can return the wrong chunk, and the model can still misread or overstate the evidence. The RAG paper itself identifies provenance and updating world knowledge as open problems (Lewis et al.).

An agent is a workflow-control pattern. OpenAI's agent guide defines agents around independent workflow execution, tool use, and model-directed decisions. Anthropic makes a similar distinction: a workflow follows code paths you define, while an agent dynamically directs its own process and tool usage.

That difference matters for internal knowledge:

Your internal-knowledge job	Best starting point	Why
“What is our parental-leave policy?”	Standard RAG	One approved corpus, read-only answer, source citation
“Summarise the policy and fill the fields in this request”	RAG plus fixed workflow	Retrieval and extraction are known steps; validation can stay in code
“Compare the policy, the employee's contract, and the current HR record”	Agent plus retrieval tools	The next source depends on what the first source says
“Find the relevant procedure and open a draft ticket with the evidence attached”	Agent plus RAG and an action tool	The system must retrieve, decide, and act; the write should remain bounded
“Search every system until you find something that supports my preferred answer”	Neither by default	The source policy, stopping rule, and evaluation criteria are not defined

The last row is important. More autonomy cannot repair unclear ownership of knowledge.

Use the SCOPE test before choosing an architecture

SCOPE is my decision test for an internal knowledge project. It is an original synthesis of the retrieval and agent distinctions in the sources above, not an industry standard or a benchmark. Answer five questions before you compare vendors.

S — Source boundary

Can the answer come from one approved corpus or index, with the user's permissions applied at retrieval time?

If yes, standard RAG is a credible first choice. If the answer might live in a document store, a database, a ticketing system, and a live operational API, you may need multiple retrieval tools. Do not confuse “many documents” with “many sources.” A large, well-indexed corpus can still be a simple RAG problem.

The security boundary comes first. Microsoft's secure RAG guidance says that only data a user is authorized to access should become grounding data. A vector index does not inherit your company's access model by magic. Carry identity and authorization into the retrieval path, filter before context reaches the model, and test with users who should see different results.

C — Complexity of retrieval

Does one search usually find enough evidence, or must the system decompose the question and let the first result determine the next search?

One query against one index points toward standard RAG. Dependent searches, dynamic source selection, and iterative refinement point toward agentic retrieval. Microsoft lists those patterns explicitly and notes that each agent step adds latency, token consumption, and complexity (agentic RAG architecture).

A useful test is to write the path without using the word agent:

1. What does the user ask?
2. Which source must be searched first?
3. What result changes the next question?
4. When is the evidence sufficient?
5. What happens if it is not sufficient?

If those answers are stable, code the path. If they change with the evidence and you can define safe stopping conditions, an agent may earn its place.

O — Outcome

Is the desired outcome an answer, or is it a change in another system?

Answers, summaries, and drafts can often stay in a RAG or fixed-workflow boundary. Actions such as changing a record, sending a message, opening a ticket, or approving a request create a different risk profile. OpenAI's guide separates data tools, which gather context, from action tools, which change external systems.

This is not an argument against action. It is a reason to keep retrieval and action visibly separate. Let the system show the evidence and proposed action before it gets permission to perform the action.

P — Permission and reversibility

Can the first version be read-only, scoped to the minimum sources, and stopped without leaving a difficult-to-reverse side effect?

If yes, you have room to pilot. If no, reduce the scope before adding an agent. A system that can search a private repository is not automatically safe, and a system that can write to a ticketing or finance system needs stronger controls than a knowledge assistant that only answers questions.

For an agent, record the tool boundary explicitly: allowed tools, forbidden tools, maximum iterations, escalation condition, and owner. Anthropic recommends grounding agent execution in environmental feedback and using stopping conditions or human checkpoints (Building effective agents).

E — Evaluation

Can you tell, before launch, what a correct answer or action looks like?

For RAG, check whether the answer is supported by the right source, whether access filtering works, and whether the system abstains when the corpus has no answer. For an agent, add checks for tool choice, unnecessary searches, stopping behavior, and forbidden actions.

You do not need a perfect benchmark. You do need representative questions, expected evidence, expected outcome, and an owner who can judge disagreements. If nobody can say what “good” means, the architecture decision is premature. Choose discovery and knowledge cleanup before autonomy.

The decision matrix

Use SCOPE to place the workflow in one of these three lanes.

SCOPE result	Architecture	First release should do	First release should not do
One source, one search, answer only, read-only	Standard RAG	Retrieve small, relevant passages; cite title, date, and location; abstain without support	Browse every system, invent a policy, or write records
Known sequence, a few language steps, no dynamic planning	RAG plus fixed workflow	Retrieve, extract, validate, route, or draft in a defined sequence	Let the model choose a new process for every request
Multiple sources or dependent searches, with a bounded action need	Agent plus RAG tools	Select a source, retrieve, assess evidence, stop, and request approval when needed	Give broad write access or unlimited search loops

This matrix also tells you what to measure. Standard RAG needs retrieval relevance, grounded answer quality, access correctness, abstention, and response time. An agent adds tool-selection quality, number of steps, unnecessary calls, action safety, and cost or latency per run. The exact thresholds belong to the workflow owner; they should not be borrowed from a vendor demo.

A worked example: an internal policy assistant

Imagine a 200-person company has policies in a versioned document repository. Employees ask questions about leave, travel, and expenses. The first proposed feature is: “Answer the question and link the policy section.”

This is a standard RAG job if the documents are authoritative, the index preserves title and effective date, and the user identity controls which documents can be retrieved. The assistant should answer from retrieved text, show the source, and say that it could not find an approved answer when retrieval fails.

Now change the request: “Based on the policy, my employment contract, and this month's payroll record, tell me whether I need to contact HR and open a draft case with the supporting evidence.” The task now has a dependent path. It may need to retrieve the policy, inspect the contract, query a live record, compare them, and prepare a draft. That is a candidate for an agent with separate retrieval tools and a draft-only action boundary.

The correct first release is still not “an autonomous HR agent.” It is a supervised investigation flow. The employee sees the evidence, the system states uncertainty or conflict, and a person approves any external case creation. This example is hypothetical; it is not a client result or a performance claim.

Copy this internal-knowledge pilot card

Fill this out before choosing a framework or granting tool access:

Knowledge job:
Primary user and owner:
User question or trigger:
Desired outcome: answer / draft / recommendation / system change

Authoritative sources:
Source owner for each source:
Source freshness or effective-date rule:
User identity and permission boundary:

Starting architecture: standard RAG / fixed workflow / agent plus retrieval tools
Why a simpler option is insufficient:
Expected retrieval path:
What makes the next step change:
Required citation fields: title / date / section / record ID
Abstain or escalate when:

Allowed tools:
Forbidden tools:
Read actions:
Write actions requiring approval:
Maximum searches or iterations:
Named escalation owner:

Representative questions:
Expected evidence and outcome for each:
Forbidden behavior for each:
Baseline to beat:
Release decision and next review date:

The line “why a simpler option is insufficient” is the most valuable one. If you cannot fill it in, use RAG or ordinary search first. The goal is not to prove that your company needs an agent. It is to avoid building one before the work demands it.

Common mistakes

Mistaking a chat interface for an agent

An internal chat window can sit on top of standard RAG. Conversation does not prove that the model should control the workflow. Define the system by what it can decide and change, not by how the interface looks.

Treating RAG as a truth guarantee

Retrieved text can be stale, incomplete, duplicated, or wrongly permissioned. Preserve source metadata, show citations, filter access before generation, and define an abstention path. A confident answer without supporting evidence is a retrieval failure wearing a language-model voice.

Adding an agent to compensate for poor knowledge management

An agent that searches five messy repositories may produce a more complicated explanation of the same ambiguity. Name an owner for each source, mark effective dates, remove superseded policies from the retrieval path, and decide which source wins conflicts.

Starting with writes

The first useful agent version is often read-only or draft-only. If the system cannot explain its evidence, do not let it send, approve, delete, purchase, or update. The existing guide on human approval gates covers the separate question of how to place a person at a consequential boundary.

Choosing a platform before writing the path

Cloud services now offer both classic and agentic retrieval architectures. That is useful, but it can make the product menu look like the decision. Write the SCOPE answers, pilot card, source contract, and baseline first. Then choose the smallest platform that can implement them.

So, which one should you use?

Use RAG when internal knowledge is the job: retrieve approved evidence, answer a question, cite the source, and stop.

Use RAG plus a fixed workflow when the job has known stages such as retrieve, extract, validate, and draft.

Use an agent with RAG and other retrieval tools when the job is investigation or execution: the system must choose sources, decompose the request, iterate based on results, query live systems, or take a bounded action.

The best first architecture is often less impressive than the one in the demo. That is a strength. Start with the smallest system that can answer correctly, respect permissions, and reveal when it does not know. Let the workflow earn more autonomy through evidence.

If you have one real internal workflow and want to work through the SCOPE test with someone, Marius's one-to-one AI consulting is a suitable next step. Bring the question, its authoritative sources, and the current manual path. The outcome should be a clearer architecture decision, not an agent by default.