How to Build a Lightweight Evaluation Review Session With Your Team

I keep evaluation review sessions small because a large meeting can hide a missing decision. When I taught product managers who moved from writing specs to building and shipping, the recurring problem was often that nobody could say what done meant. That is the problem this session is designed to expose. (Marius Manolachi's teaching practice)

The artifact is a 45-minute packet for one workflow and three cases. I validated its exit rule locally. All three rows were complete, but the session still returned HOLD because the ambiguous case remained unresolved.

Start with one workflow and three cases

Choose one recurring AI workflow and bring three representative cases: normal work, ambiguous evidence, and a boundary or high-cost request. Do not review a whole product in one sitting. Review one decision surface the team can understand together.

This shape follows the useful distinctions in Anthropic's evaluation guidance: a task has inputs and success criteria, a trial is one attempt, a grader checks part of the performance, and the outcome is the final state rather than the system's claim about it. (Anthropic's evaluation definitions)

Case	What it tests	What reviewers must decide
Normal	The expected path	Did the workflow produce a usable result?
Ambiguous	Missing or conflicting evidence	Did it expose uncertainty or invent a decision?
Boundary	An out-of-scope or costly action	Did it refuse, escalate, or cross a human boundary?

The three cases are a starter configuration, not a research-backed sample-size rule. Add a real incident as soon as one appears. Microsoft also recommends expanding evaluation coverage across core behavior, robustness, architecture, and edge cases, and repeating runs when agent responses vary. (Microsoft's agent evaluation checklist)

Use a 45-minute agenda that forces a decision

Assign a facilitator, a domain owner, and a builder or evaluator. The facilitator protects the clock and records disagreements. The domain owner decides whether the work is usable. The builder explains what the system actually did. If one person fills all three roles, label that limitation in the notes.

Minutes	Activity	Output
0-5	State the workflow, source of truth, and human boundary	One-sentence success condition
5-12	Review the three cases and expected outcomes	Shared case context
12-22	Score silently, one row per reviewer and case	Independent judgments
22-35	Compare disagreements and inspect the trace or final state	Evidence for each disputed row
35-42	Decide SHIP, HOLD, or REWORK for each case	Decision and owner
42-45	Read back blockers, next test, and review date	Written follow-up

Silent scoring comes before discussion so the loudest voice does not become the baseline. OpenAI's eval documentation separates the test-data schema from the testing criteria that determine correctness. Use the same separation in the meeting: agree on what the case contains before arguing about the output. (OpenAI's eval guide)

Give every reviewer the same evidence sheet

Use one row per case. Keep the evidence fields separate from the verdict fields. This prevents a fluent answer from carrying a factual failure into a vague “mostly good” score.

case_id:
source_of_truth:
expected_outcome:
observed_outcome:
evidence_checked:
critical_risk: yes | no
unresolved_question:
decision: SHIP | HOLD | REWORK
action_owner:
next_test:
review_date:

For an agent that changes state, inspect the resulting record, file, ticket, or approval. Inspect the trace when it explains how the result was reached. Anthropic distinguishes the transcript from the outcome for this reason. A convincing final message is not proof that the intended state exists. (Anthropic's agent eval guidance)

Use the cheapest valid grader for each field:

Field	First grader	Human role
Exact status, schema, file, or count	Deterministic check	Verify the check matches the work
Tool choice, approval, or forbidden action	Trace assertion	Decide whether the rule is material
Clarity or completeness	Bounded rubric	Calibrate the rubric on disputed cases
Ambiguous or high-impact judgment	Human review	Own the decision

NIST treats evaluation as context-dependent and includes qualitative as well as quantitative methods. A lightweight session should therefore measure the property that matters for this workflow, not force every task into one score. (NIST on AI measurement and evaluation)

Run the session in five moves

1. Name the source of truth. Write down where the team will verify success. It might be a CRM record, a test result, a document field, or a human-approved draft.
2. Write the veto before looking at outputs. Examples include an unauthorized action, an unresolved identity match, a missing approval, or an unsupported high-impact claim.
3. Score independently. Each reviewer fills the same row before discussion. Use unknown when the evidence is insufficient rather than forcing a pass or fail.
4. Adjudicate the disagreement, not the average. Ask which evidence would change the decision. If nobody can supply it, keep the case at HOLD and assign an owner.
5. Create the next case. Turn the disagreement or failure into a saved evaluation case with the input, expected behavior, evidence, and decision rule.

The point is not to make the meeting feel scientific. It is to make the team's judgment inspectable. NIST's AI Risk Management Framework calls for periodic review, clear roles, testing, and mechanisms to incorporate feedback into system design and implementation. (NIST AI RMF Core)

Close with an explicit exit rule

Use SHIP only when every case has been reviewed, no critical veto is unresolved, every follow-up has an owner, and the domain owner accepts the outcome. Use HOLD when evidence is missing or a critical case is unresolved. Use REWORK when the team knows the needed change and can rerun the case.

The local validation of the packet used one normal case, one ambiguous critical case, and one boundary case. It returned:

Cases reviewed	Complete	Blocker	Exit
3	Yes	ambiguous remained unresolved	HOLD

The result is deliberately modest. It does not show that a real team agreed faster. It shows that the rule can distinguish “we filled every row” from “we have enough evidence to ship.” The exact check is only a few lines:

const complete = reviews.length === packet.cases.length &&
  reviews.every((review) => review.id && review.owner);
const blockers = reviews.filter((review) =>
  review.unresolved || !review.owner || review.outcome === "disagree"
);
const exit = complete && blockers.length === 0 ? "SHIP" : "HOLD";

Do not average away a critical disagreement. A numeric score can help compare ordinary cases, but it cannot cancel an approval bypass or an unresolved boundary. If you need a more complete release gate, use the evaluation pillar and the AI agent evaluation guide for the surrounding system checks.

What this lightweight session cannot tell you

It cannot establish production reliability from three cases. It cannot calibrate an LLM judge by itself. It cannot replace domain review where the consequence of a wrong decision is high. It also cannot tell you whether the workflow is valuable, only whether the selected cases meet the agreed conditions.

After the first session, keep the packet and add cases from incidents, user corrections, and model or tool changes. Compare the same cases across versions, then schedule another review when the workflow changes. If you want to practice the judgment at the output level first, use the rubric exercise. The team session is the bridge from individual scoring to shared ownership.

If the team cannot agree on the source of truth or the human boundary, stop there. That is not a facilitation problem. It is the evaluation work.