What Architecture Skills Make Professionals Effective With AI Systems?

Most architecture mistakes start with a label. Someone calls a task an “agent” before they can say what the task is, what the current process does, or what would count as a safe result.

I ran the same diagnostic on three professional scenarios: invoice extraction, competitor research, and CRM enrichment. Two of them stayed workflows. Only the research task became a bounded agent. That difference is the point.

If you are new to this decision, start with the AI architecture pillar, then use the decision framework for when to use an AI agent to compare the general alternatives. This page tests whether you can make that choice on real work.

What architecture skills actually separate capable professionals?

Effective professionals can turn a vague AI idea into four reviewable artifacts: a job and boundary record, an architecture decision, a tool and approval map, and a minimal evaluation plan. These artifacts show judgment more clearly than a collection of prompts.

Artifact	Decision it must make	Evidence that the decision is real
Job and boundary record	What work is being changed, for whom, with what non-goals?	A task statement, desired outcome, exceptions, and a non-AI alternative
Architecture decision	Does code control the flow, or does the model choose the next step?	A workflow or agent choice with the rejected alternative and its reason
Tool and approval map	What may the system read, write, decide, or escalate?	Named tools, permissions, approval point, owner, and stop condition
Minimal evaluation	How will we know the system worked and failed safely?	Test cases, graders, outcome checks, and a threshold or release rule

This is the sourceable result of my exercise: architecture capability is visible in the artifacts a professional can produce, and the artifacts should change when ambiguity, action risk, and the cost of error change. It is not a claim that every professional needs to build an autonomous agent.

The structure matches the direction of the primary guidance. OpenAI describes an agent as a system that independently accomplishes tasks through a model, tools, and instructions, but says a deterministic solution may be enough when a job does not involve complex decisions, brittle rules, or heavy unstructured data. OpenAI's practical guide explains the distinction.

NIST's AI RMF adds the part that product demos usually hide: establish context, define tasks and human oversight, compare benefits and costs with appropriate alternatives, measure the system in conditions similar to deployment, and retain a way to disengage it when outcomes drift. See the Map, Measure, and Manage functions in the NIST AI RMF Core.

How can you test architecture skill before building?

Give the professional three different jobs and require the same four-part record for each. Do not let them start by selecting a model or drawing an agent diagram.

1. State the job. Write the trigger, user, desired change, non-goals, exceptions, and the non-AI way the work could be done.
2. Choose the control structure. Use a workflow when the sequence is known and the model performs one or more bounded steps. Use an agent when the model must choose what to do next, call tools, and adapt to intermediate results. Explain why the rejected option loses.
3. Bound authority. List every tool, its read or write capability, the data it can access, the action that requires approval, and the condition that returns control to a person.
4. Write the smallest useful eval. Define cases, success criteria, graders, and the final state you will inspect. Include one case where evidence is missing or conflicting.

Use this eight-point rubric. A 2 means the decision is explicit and testable. A 1 means the idea is present but vague. A 0 means the artifact is missing. A scenario passes at 6/8 or higher, with no zero in tools and approval or evaluation.

Dimension	0	1	2
Job boundary and non-AI alternative	Missing	Task stated, but boundary or baseline is vague	Task, outcome, non-goals, and baseline are explicit
Architecture choice	Missing or defaults to agent	Choice stated with a thin reason	Choice and rejected alternative fit the control flow and uncertainty
Tools, authority, and approval	Missing	Tool or reviewer named, authority unclear	Permissions, approval point, stop condition, and owner are explicit
Minimal evaluation	“Test it”	Metric exists, but cases or grader are vague	Cases, metric, grader or outcome check, and revision are explicit

This tests capability rather than confidence. The OECD's work on assessing AI-related capabilities argues for multiple dimensions, established scoring logic, and scenario-based items that represent real-world situations. Its discussion of scenario-based assessment is in Chapter 7.

What does the three-scenario decision matrix show?

The same diagnostic produces different architectures when the work changes. That is why a list of architecture patterns is a weak test of professional skill.

Scenario	Non-AI alternative	Chosen architecture	Tools and human oversight	Success metric	Score
Invoice packet to approval-ready ledger	Parser and rules plus manual exception handling	LLM-assisted workflow	File or OCR reader, schema validator, vendor lookup, and draft-ledger writer. Finance approves before the ledger write.	Field-level accuracy on labeled cases, no unreviewed writes, explicit exceptions	8/8
Evidence-backed competitor brief	Analyst search, source log, and manual brief	Bounded single agent	Search, page reader, source ledger, citation checker, and draft writer. The analyst approves sources and claims.	Claim grounding, source coverage, zero unsupported material claims, human usefulness	7/8
Partnership email to CRM task	Rules and an operations coordinator	Routing workflow with an LLM classification step	Email parser, account lookup, routing rules, and draft CRM task. Operations approves before a CRM write or outbound message.	Correct route, required fields, no invented account facts, review time within the team's limit	8/8

The exercise scores are completeness scores for the records above, not accuracy rates from running an AI system. The meaningful difference is the architecture choice: the research brief needs bounded dynamic control over search and follow-up reading, while the other two tasks have known control flow and consequential writes.

The professional skill is not “knowing agents.” It is seeing which part of the work needs model judgment and which part should remain deterministic or human-owned.

What does a complete architecture decision look like?

Consider the competitor brief. The task is to produce a two-page comparison from approved public sources, link every material claim, and mark an “unknown” when the evidence is insufficient.

The non-AI alternative is already viable: an analyst searches, records sources in a spreadsheet, drafts the brief, and asks a second person to check claims. The AI system earns its complexity only if dynamic search and follow-up reading reduce work without weakening evidence.

The decision record is therefore:

Decision area	Worked choice
Architecture	Bounded single agent for research only. The model can choose the next search or page to read.
Rejected option	A fixed workflow would be easier to test, but it would need a brittle list of search paths for questions whose useful evidence varies.
Tools	Read-only search, page fetch, source-record write, and draft assembly. No publishing, messaging, or external write tool.
Guardrails	Allowed domains, source schema, maximum turns, explicit unknown state, and stop on conflicting evidence.
Approval	Market or product lead approves the source set and final claims before sharing.
Minimal eval	Test representative briefs plus conflicting sources, missing dates, inaccessible pages, and plausible unsupported claims.
Graders	Groundedness, required-field coverage, source quality, correct abstention, and human review sample.
Outcome check	Inspect the final brief and source ledger, not just the agent's statement that it finished.
Revision recorded	Replace “write a good brief” with claim-level grounding, source coverage, explicit unknowns, and approval.

That evaluation plan follows Anthropic's distinctions among tasks, graders, traces, outcomes, and reference solutions. Their guidance warns that a vague task or grader creates noisy results, and recommends checking the actual environment outcome rather than trusting a fluent completion. Read Anthropic's evaluation guidance.

The decision also has an important exception. If the brief will drive a regulated decision, contain restricted information, or trigger an external commitment, read-only research is not enough. Add the relevant risk owner, data controls, and a stronger approval process before the agent sees that work.

Which failures should change your architecture choice?

Three failure modes changed the records in this exercise.

Confusing tools with agency

The invoice workflow uses tools, but the model does not control the workflow. It extracts fields inside a known sequence. Calling it an agent would hide the fact that code can enforce validation and that finance approval is required before a write.

OpenAI's guidance makes the same underlying distinction: an application can use an LLM without making the model responsible for workflow execution. The architecture label should describe control, not marketing.

Writing an eval that says “good output”

The first competitor-brief record had a plausible goal but no pass or fail rule. I replaced it with claim grounding, source coverage, explicit unknowns, an outcome check, and a human review sample. Anthropic recommends clear task specifications, graders, reference solutions, and multiple trials when model variation matters. Its evaluation roadmap starts with explicit tasks and success criteria.

Approving after the action

The first CRM record gave the proposed agent too much authority. The revision removed autonomous search, send, and write behavior. The system can classify the email and prepare a draft CRM task. An operations owner approves before any consequential write or outbound message.

NIST treats human oversight, risk tolerance, monitoring, response, recovery, and safe disengagement as part of the lifecycle rather than a final prompt instruction. The AI RMF Core describes those responsibilities.

I have seen the same boundary problem while teaching product managers to move from writing specs to building and shipping products. The recurring gap was often an undefined “done,” not a missing model feature. That is a bounded teaching observation from F-pms, not a prevalence statistic.

Anthropic's usage research points in a similar direction, with an important limit: it studied Claude Code sessions, not all professional work. In its analysis of about 400,000 interactive sessions, people made about 70% of planning decisions and about 20% of execution decisions on average. The result supports a narrow inference: professionals still need to decide what to build, what counts as done, and where to check the work even when an agent handles much of the execution. See the study's method and findings.

How do you check whether the skill transfers?

Do not stop after the three examples. Give yourself a fourth job that you did not use while learning. Complete this record without starting from an agent label:

Task and trigger:
Desired outcome:
Non-goals and principal exception:
Non-AI alternative:
Workflow or agent choice:
Why the rejected option loses:
Tools and permissions:
Human approval and stop condition:
Success metric and test cases:
Failure found or revision made:

Then score it with the same rubric. You are ready to move from prompting into architecture when you can explain the boundary, preserve the simpler alternative where it wins, place approval before consequential action, and write an eval another person could grade without guessing what “good” means.

For a small, read-only experiment with no sensitive data and no external action, the full record can be short. Once the system reads private data, changes records, sends messages, or becomes part of a repeated business process, expand the record and use the AI agent design document guide as the next artifact.

The practical next step is to run this diagnostic on one task your team wants to automate. If you want help making the team capable of doing that work itself, learn about Marius Manolachi's AI tutoring and consulting work.