What Evidence Proves an AI Partner Made the Team More Capable?

The easiest evidence to collect is also the easiest to misread. The team finished faster. People used the AI system. The partner produced a polished deliverable.

That shows assisted output. It doesn't show that the team became more capable.

When I taught product managers who moved from writing specs to building and shipping products, the recurring failure was usually an undefined “done,” not the model. That teaching observation is not a measured client result. It gives this test its starting point: define the finish line before you celebrate the speed.

What counts as proof of capability transfer?

Call capability transfer proven only when the team completes a comparable real task after live partner help is removed, passes the same quality and risk checks, explains the important decisions, and leaves an artifact another teammate can reuse.

Here is the sourceable artifact for this page. Use it at the start of an engagement and at the exit review.

Gate	Pass condition	Evidence to retain	Automatic fail
Baseline	A real task has a written input, scope, owner, and definition of done before assistance begins.	Task brief, repository or source revision, acceptance rubric.	The task changes after the assisted run starts.
Assisted result	The partner-supported team completes the task at the agreed scope.	Final output, partner interventions, time log, review notes.	The partner quietly completes the work or changes the acceptance bar.
Independent transfer	A comparable task is completed without live partner intervention.	Retest task, screen or activity log where appropriate, final output, questions asked.	The team needs the partner to choose the method or approve each step.
Output quality	The retest meets the same acceptance, correctness, safety, and critical-failure checks.	Rubric scores, tests, source checks, reviewer notes, rejected cases.	A critical failure is waived because the task was fast.
Explanation	The person doing the work can explain the method, assumptions, limits, and stop conditions.	Short teach-back, decision record, or annotated artifact.	The output passes but the owner cannot explain why.
Verification	A teammate who did not produce the output can inspect and challenge it.	Independent review, evidence links, corrections, and approval or rejection.	Verification is only “the model said it was correct.”
Cost of work	Active time, review, rework, retries, and unresolved failures are separate observations.	Time log and rework log.	Only prompt time or generated volume is counted.
Handoff	The team leaves a reusable workflow contract, checklist, prompt, test set, or runbook.	Versioned artifact with owner and update path.	The method lives only in the partner's memory.
Follow-up	A comparable task is retested after the team has returned to normal work.	Dated follow-up result and failure review.	The engagement is declared successful immediately after the demo.

The veto matters. If the independent retest fails a critical quality or safety check, the engagement has not transferred a production capability even if the assisted task was fast.

Why isn't speed enough?

Speed measures the assisted path. Capability transfer measures what remains when the assistance is removed.

The distinction is visible in the research. Microsoft's three field experiments reported a pooled 26.08% increase in completed tasks among developers with an AI coding assistant. That is useful evidence about output under assistance. It does not test whether the same people can perform a comparable task without the assistant or explain the result to another reviewer. (Microsoft Research)

A Google enterprise trial estimated about a 21% reduction in time on a complex task, while warning that the result came from internal tooling in 2024 and may not generalize across tools or time. Again, it is a speed result, not an exit test. (Paradis et al.)

Anthropic's randomized trial makes the missing capability visible from another angle. Fifty-two software engineers using AI while learning a new Python library finished about two minutes faster on average, but that difference was not statistically significant. Their comprehension quiz average was 50%, compared with 67% for the hand-coding group, with the largest gap on debugging questions. (Anthropic's research report) A team that cannot explain or debug the result has not demonstrated durable capability, even if the first run was quick.

Speed still belongs in the scorecard. It helps answer whether the assisted method is worth using. It just cannot answer whether the buyer now owns the capability.

What should the independent-transfer test look like?

Run the same workflow twice, with the second run designed so the partner cannot rescue the team.

1. Choose a repeated real workflow. Pick a task that will occur again, such as turning a product request into an accepted implementation, preparing a support analysis, or producing a decision brief. Remove personal or confidential data where needed, but keep the work structure real.
2. Freeze the baseline. Record the task statement, source revision, inputs, acceptance tests, critical risks, definition of done, and expected owner. Do this before the partner starts.
3. Observe the assisted run. Record partner interventions, human time, review time, retries, corrections, and the point at which the output first looked complete. Keep the last measure separate from final acceptance.
4. Create a matched retest. Use a comparable task with the same type of inputs, scope, and risk. Do not reuse the exact answer. Give the team the workflow artifact, not the partner's live help.
5. Remove live intervention. The partner can be available for a later review, but it cannot choose the method, answer questions, or repair the work during the retest. Log any intervention as a protocol failure.
6. Run the quality and verification checks. A producer checks the output against the rubric. A different teammate checks the evidence, critical risks, and acceptance state. Keep their notes.
7. Run a teach-back. Ask the owner to explain what the workflow does, why it is safe enough for this scope, where it fails, and what would trigger escalation.
8. Repeat after normal work resumes. A 30-day follow-up reveals whether the artifact became part of the team's practice or remained a workshop souvenir.

The retest need not use the same person. A different teammate is often the stronger test because capability should survive a handoff. If the workflow requires specialist knowledge, keep the same role but change the task and require an independent reviewer.

What does the public test show?

The strongest public example I found is a two-phase experiment on AI-assisted software development. It is close to the buyer question because the second phase removes AI assistance and asks new participants to evolve work created in the first phase.

In the 2026 Echoes of AI study, Phase 1 participants added a feature to a Java web application with or without AI assistance. Phase 2 assigned new participants to extend those solutions without AI assistance. The study reports 151 completed participants overall and 75 valid Phase 2 participants. Its materials include task instructions, cleaned anonymized participant data, source code, scripts, and notebooks in a public replication repository and Zenodo package. (Borg et al.)

The study's reported result is the useful surprise: AI assistance reduced Phase 1 completion time, with a 30.7% median reduction, but the independent Phase 2 evolution showed no significant difference in completion time or code quality. The authors also say that any speed or quality advantage in the retest was small and uncertain.

Here is the worked decision application. It is my reading of the public study against the buyer scorecard, not a new statistical result.

Scorecard field	What the public test reports	Buyer verdict
Baseline task	Add a feature to a defined Java web application.	Pass
Definition of done	Assigned task instructions plus an acceptance test suite. Invalid submissions were excluded.	Pass
Assisted work	Phase 1 included AI-assisted and no-AI developers.	Pass
Independent retest	75 new participants evolved prior solutions without AI assistance.	Pass
Output quality	CodeHealth and test coverage were measured, with manual inspection of passing solutions.	Pass
Verification	Acceptance tests, quality metrics, and independent Phase 2 work were recorded. A direct explanation test was not recorded.	Partial
Time	Completion time was measured, but interrupted sessions sometimes required self-reported estimates.	Partial
Rework	Rework was not isolated as its own published field.	Missing
Follow-up	Phase 2 tested downstream evolution soon after Phase 1. It was not a long-term organizational review.	Partial
Decision	Evidence supports a reproducible downstream-transfer test. It does not prove whole-team capability by itself.	Conditional pass

That is the decision a buyer should want. The study doesn't fail because it lacks every possible measure. It earns a conditional pass because it includes the hardest gate, independent work after the assisted task, while making the missing gates visible.

How should you score the result?

Use three labels rather than one flattering percentage.

Label	Minimum evidence	What you can say
Proven transfer	Independent retest passes, no critical failure survives, a different teammate verifies the work, the owner explains the method, and a reusable artifact exists.	The team can perform this capability without live partner help within the tested scope.
Conditional transfer	Independent retest and quality checks pass, but explanation, rework, follow-up, or handoff evidence is incomplete.	The partner left a promising capability, but the exit decision needs another test.
Assisted delivery only	No independent retest, or the team needs live partner intervention to finish or verify the work.	The partner produced output. Capability transfer is unproven.

The public study lands in the middle label for a commercial buyer. It is rigorous evidence that a downstream retest can be run and that faster assisted work did not automatically create a measurable downstream advantage. It is not evidence that an operations team, product team, or client engagement has passed the explanation and handoff gates.

Avoid averaging away a veto. A team that is fast but cannot detect an incorrect answer is not more capable in a consequential workflow. A team that can explain and verify the output but takes longer may still own the capability. Decide first whether the work is safe and correct. Compare time only among passing runs.

What should an AI partner leave behind?

Ask for artifacts that let the team repeat the work without the partner's presence.

• A task contract with inputs, scope, definition of done, exclusions, and escalation rules.
• A verified example with the raw input, output, corrections, and reviewer decision.
• A short workflow or prompt with version history and known failure cases.
• A test set containing ordinary cases and at least the failures found during the assisted run.
• A runbook that says who owns the work, what to check, and when to stop.
• A follow-up date and a named internal reviewer.

An artifact is not a handoff just because it is written down. A teammate who did not attend the engagement should be able to use it on a new case and explain where it stops. If that person needs a private call with the partner, record the dependency rather than hiding it.

When should you stop calling the engagement successful?

Stop using the language of capability transfer when the retest depends on live partner judgment, the acceptance standard moved, a critical failure was waived, or nobody can say what done means.

This is where the commercial decision changes. If the partner's job was short-term delivery, assisted delivery may be the correct purchase. If the business paid for internal capability, the partner should repair the missing gate before the engagement closes. That might mean a smaller task, a clearer contract, a teach-back, a reviewer outside the producer's path, or a second retest after normal work resumes.

If you are comparing a consultant, agency, or internal team, keep the capability question separate from the resourcing decision. The commercial pillar gives the parent decision context. How to Choose an AI Consultant, Agency, or Internal Team helps define what responsibility you are buying and what must remain internal.

For a team that wants to run the scorecard on its own workflow, Marius Manolachi's AI consulting and tutoring work follows the same boundary: existing people should become able to build and judge AI products on their own work. The first deliverable is not a promise of speed. It is a retest the team can pass without the partner.