What Evidence Should a Buyer Request to Verify AI Engagement Outcomes?

The phrase “we improved productivity” is not evidence. It’s a claim waiting for a denominator.

When I taught product managers to build and ship instead of only writing specifications, the recurring failure was simple: nobody could say what “done” meant. The same ambiguity makes an AI engagement report look finished when the buyer still cannot verify the outcome. Marius Manolachi's AI teaching work treats that boundary as a capability to build.

What should a buyer request first?

Request one dated evidence pack per outcome claim. It should connect the claim to a baseline, comparison, denominator, attribution method, deployment-condition test, failure slice, usage or cost record, named owner, and decision consequence.

NIST defines validation as objective evidence that the requirements for an intended use were fulfilled. It also says accuracy measures should use realistic test sets that represent expected conditions, document the test method, and may be split by data segment. That is the standard a buyer needs before treating a vendor percentage as a result. NIST's AI Risk Management Framework guidance is the source constraint here, not a substitute for your own evidence.

The buyer's first request can be short:

“For each promised outcome, send the raw case or event schema, the baseline and comparison, the complete denominator, the attribution method, the deployment conditions, the failure slices, the usage and cost record, the owner, and the go, continue, or stop rule.”

What belongs in the claim-to-evidence matrix?

Use the matrix below to turn outcome language into a request that someone can inspect. The claim classes span technical health, adoption, operational change, strategic results, financial value, and ownership. That ordering follows the measurement layers described by McKinsey, while the fields add the buyer-side detail needed to accept or reject a claim. McKinsey's five-layer measurement framework places benefits and total cost in one ledger and connects measurement to decision gates.

Outcome claim	Buyer request and minimum evidence	Baseline and comparison	Window and denominator	Attribution caveat	Deployment test and failure slice	Cost or usage record	Owner and consequence
“The workflow is faster.”	Raw timestamps for trigger, first usable output, human review, and acceptance; frozen configuration.	Same task class before and after, or matched control.	Declared dates; completed eligible cases, not only successful outputs.	Seasonality, task mix, and hidden review work can explain the change.	Test normal, long, incomplete, and high-volume cases; report each slice.	Model calls, tokens, infrastructure, review minutes, and subscription or license cost.	Process owner accepts; continue only if speed holds without a quality veto.
“Quality or accuracy improved.”	Reference labels, rubric, raw outputs, reviewer decisions, and disagreement log.	Existing process or locked baseline model.	All eligible cases, with exclusions listed.	A changed rubric or easier sample can create a false lift.	Test edge cases and high-severity errors; report false positives and misses.	Evaluation-run usage and reviewer time.	Domain owner accepts; stop or narrow if a critical slice fails.
“Users adopted the system.”	User-level events, eligible-task count, acceptance, override, and abandonment records.	Eligible users and tasks before versus after, not raw logins.	Fixed rollout window; denominator is eligible work.	Mandated usage can look like trust or value.	Segment by role, tenure, and workflow; inspect non-users and overrides.	Active seats, calls, support, training, and enablement cost.	Product or operations owner accepts; continue only if use reaches the value path.
“The engagement improved the business KPI.”	KPI definition, source query, change log, and intervention timeline.	Pre-period and matched control, staggered rollout, or A/B test where appropriate.	Predeclared window and eligible population.	Other launches, staffing, pricing, and demand changes may drive the move.	Test the workflow conditions under which the KPI is meant to change.	Total cost of ownership and incremental operating costs.	Business owner accepts; continue only when attribution is credible enough for the spend.
“The AI reduced cost or created ROI.”	Benefits ledger, labor assumptions, vendor invoices, model usage, infrastructure, and support cost.	Baseline cost versus full assisted cost, including review and exceptions.	Declared period; denominator is completed accepted work or attributable revenue.	Time saved is not cash saved unless capacity or spend changes.	Test high-cost cases, escalations, and rollback paths.	Actual usage and all recurring and one-off costs.	Finance and business owner accept; stop if benefits depend on unpriced work.
“The system is reliable in production.”	Versioned run logs, error events, latency, fallback, incident, and monitoring records.	Agreed service baseline and preproduction test.	Ongoing window with uptime and completed-run denominator.	A clean demo or average latency can hide tail failures.	Test load, degraded dependencies, drift, and human escalation.	Calls, retries, infrastructure, support, and incident cost.	Technical owner accepts; hold rollout if failure handling or monitoring is missing.
“The team can operate it after handover.”	Runbook, access list, change procedure, replay or rollback exercise, and named owner.	Capability before and after handover on a new case.	Handover period plus a transfer check after support ends.	Attendance or documentation receipt does not prove independent operation.	Test a new operator on a failure and a routine change.	Training, support, maintenance, and vendor dependency cost.	Operational owner accepts; continue only when the buyer can maintain and escalate it.

The NIST AI RMF Playbook organizes suggested actions under Govern, Map, Measure, and Manage. Use it to check whether your evidence pack has the right governance and monitoring questions, not as a universal checklist.

How do you run the verification protocol?

Run the matrix in this order. The order prevents a strong average from hiding a weak decision.

1. Freeze the claim. Write the outcome, unit, threshold, eligible population, and principal exception before reading the final results.
2. Freeze the comparison. Record the baseline period, control or counterfactual, task mix, configuration, and rollout dates.
3. Inspect the denominator. Reconcile included, excluded, failed, retried, abandoned, and human-reviewed cases.
4. Recompute the headline. Use the raw rows, not a slide. Preserve the calculation and any transformations.
5. Split the result. Report relevant deployment conditions and failure slices. NIST specifically calls for realistic representative test sets and documented methodology.
6. Check attribution. Ask what else changed and whether the design can separate the AI intervention from seasonality, staffing, policy, or demand.
7. Add the full cost. Include usage, licenses, infrastructure, support, review, exception handling, and the buyer's own operating time.
8. Assign the decision. A named owner chooses go, continue with a condition, or stop. Record the missing evidence rather than smoothing it away.

The UK procurement guidance gives buyers a useful checklist for supplier evidence: reproducibility, testing under a range of conditions, acceptable performance, accountability over outputs, and proportionate security. It also recommends end-to-end process logs across modelling, testing, verification, and implementation, with ongoing monitoring after deployment. Read the UK guidance as procurement guidance, not as a private-sector law.

What does a worked stop decision look like?

The following is a fixed workflow simulation, not a client benchmark. It tests the evidence pack with 12 abstracted work items. The workflow routes requests into a queue. The assisted path represents draft plus human review. No model call or production user is claimed.

Configuration: WFS-2026-08-23-v1, run 2026-08-23. Example buyer gates: at least 20% time reduction, at least 95% overall quality, at least 90% quality in every failure slice, and a present usage or cost ledger.

The following configuration reproduces the decision from the raw rows. It treats quality_pass as a binary reference-route check and keeps the cost ledger as an explicit gate rather than silently treating missing cost as zero:

const rows = [
  ['A01', 'normal', 30, 20, 1], ['A02', 'normal', 24, 16, 1],
  ['A03', 'normal', 41, 27, 1], ['A04', 'normal', 35, 23, 1],
  ['A05', 'long-thread', 52, 36, 1], ['A06', 'long-thread', 48, 34, 1],
  ['A07', 'missing-field', 38, 27, 0], ['A08', 'missing-field', 44, 31, 1],
  ['A09', 'multilingual', 46, 33, 1], ['A10', 'multilingual', 40, 29, 1],
  ['A11', 'normal', 29, 20, 1], ['A12', 'normal', 33, 23, 1],
].map(([id, slice, baseline, assisted, quality]) => ({
  id, slice, baseline, assisted, quality,
}));

const gates = {
  minReduction: 0.20,
  minOverallQuality: 0.95,
  minSliceQuality: 0.90,
  costLedgerPresent: false,
};
const sum = (items, field) => items.reduce((total, row) => total + row[field], 0);
const slices = [...new Set(rows.map((row) => row.slice))];
const sliceQuality = Object.fromEntries(slices.map((slice) => {
  const group = rows.filter((row) => row.slice === slice);
  return [slice, sum(group, 'quality') / group.length];
}));
const reduction = 1 - sum(rows, 'assisted') / sum(rows, 'baseline');
const overallQuality = sum(rows, 'quality') / rows.length;
const stop = reduction < gates.minReduction
  || overallQuality < gates.minOverallQuality
  || Object.values(sliceQuality).some((value) => value < gates.minSliceQuality)
  || !gates.costLedgerPresent;

console.log({
  n: rows.length,
  baseline: sum(rows, 'baseline'),
  assisted: sum(rows, 'assisted'),
  reduction: `${(reduction * 100).toFixed(1)}%`,
  overallQuality: `${(overallQuality * 100).toFixed(1)}%`,
  sliceQuality,
  decision: stop ? 'STOP' : 'GO',
});

Test method: run the block with Node.js 24.11.1 and compare its totals with the evidence table and the raw rows in the research record. Observed output: n: 12, baseline: 460, assisted: 319, reduction: '30.7%', overallQuality: '91.7%', missing-field: 50%, and decision: 'STOP'. The output is a check of this simulation's decision logic, not a production result.

Slice	Cases	Baseline minutes	Assisted minutes	Time reduction	Quality pass
Normal	6	192	129	32.8%	100%
Long thread	2	100	70	30.0%	100%
Missing field	2	82	58	29.3%	50%
Multilingual	2	86	62	27.9%	100%
Total	12	460	319	30.7%	91.7%

Use the decision rule below after recomputing the raw rows:

Decision	Conditions	Commercial consequence
GO	Every material threshold passes, the cost and usage record is complete, attribution is acceptable, and an owner is ready to monitor the workflow.	Approve the next phase or agreed rollout.
CONTINUE WITH CONDITIONS	The core result is promising, no critical failure exists, and each remaining evidence gap has a named owner, due date, bounded scope, and rollback or review point.	Keep the pilot limited and release no broader claim than the evidence supports.
STOP	A critical failure appears, a material failure slice misses its bar, the denominator or cost record is missing, or attribution cannot support the spend.	Do not expand or accept the outcome claim. Fix the gap, change the use case, or end the engagement.

The speed claim clears the example gate. The quality claim does not. The missing-field slice passes one of two cases, and the usage or cost ledger is missing. The decision is STOP. Fix the missing-field behavior, rerun representative cases, and attach the cost record before asking for more rollout or funding.

This is why the denominator and failure slice matter. A headline can look successful while the condition that matters most to the buyer is not ready. The OECD describes audit trails and performance metrics as part of monitoring and accountability, and recommends ongoing evaluation of system performance and impact on people. Its public procurement guidance supports keeping the raw record connected to the decision.

When should a buyer continue instead?

Continue when the evidence pack is complete, the result clears the predeclared threshold across material slices, the attribution caveat is acceptable for the amount at risk, and the named owner can operate the control after handover.

Do not require every uncertain discovery question to have a production metric. A discovery phase can produce a reviewed workflow map, data assessment, representative case set, or option comparison. But it still needs an acceptance artifact and a next decision. The right exception is uncertainty made visible, not uncertainty hidden behind a completion label.

McKinsey's measurement guidance makes the commercial implication explicit: decision gates should determine whether a project receives more funding, broader rollout, or more engineering capacity. The evidence pack is what makes that gate reviewable. The broader commercial calculation belongs in the AI agent ROI guide, while proposal-level acceptance belongs in How to Compare AI Consulting Proposals. The canonical parent How to Choose Between an AI Consultant, Agency, and Internal Team holds the wider buying path.

If you are reviewing a live engagement, copy the matrix into your deal room and fill one row before the next steering meeting. If you want help turning a vague outcome into a testable artifact, bring the claim, the current baseline, and one representative failure case to Marius Manolachi's AI consulting and tutoring work.