What Evidence Distinguishes AI Tutoring From Implementation?

The dangerous moment comes after the demo. The workflow works, the provider sounds confident, and nobody has checked what the team can do alone.

I ran one small comparison to make that boundary visible. The task was safe, the inputs were anonymized, and neither version could write to an external system.

What evidence separated the two modes in the matched task?

The clearest difference was not the number of good-looking outputs. It was the operator's next move. In tutor mode, the operator changed the decision rule and reran the failing case. In implementation mode, the operator could run the delivered artifact but needed help to locate the code line for the same kind of change.

Evidence	Tutor-led path	Lightweight implementation path	Observed result
Operator capability	Operator defines the rule, types the workflow, and explains the correction.	Provider creates the main artifact; operator runs and inspects it.	Tutoring left the operator able to change the tested rule.
Working artifact	Prompt card plus corrected output.	Runnable Python command plus fixtures and handoff.	Both left something usable, but the artifacts served different purposes.
Evaluation record	Three cases, one first-pass failure, one corrected rerun.	Three cases, schema validation, review flags for two cases.	Both were stronger than a happy-path demo because the cases and failure were saved.
Handoff	Operator retained edit access and changed F-03 without the provider.	Operator could run the command but did not locate the change point unaided.	A runnable handoff did not equal independent maintenance.
Permissions	Read-only, no external writes.	Read-only, no external writes.	The test compared engagement evidence, not production safety.

That is the result to carry into a buying conversation: tutoring should prove a person can make the workflow better; implementation should prove the workflow can be run, checked, and handed over. If you need both, buy both as separate acceptance conditions.

What did the test actually measure?

The workflow was a product-feedback decision brief. The operator had three anonymized notes and had to produce six fields: id, theme, evidence, next_action, confidence, and needs_review.

The starting state had no shared schema, no saved test cases, no explicit rule for conflicting evidence, and no ownership check. The same three inputs and checks were used in both modes:

• F-01 described recurring requests for CSV export and manual copying.
• F-02 described one onboarding email arriving four days late, with delivery logs still unchecked.
• F-03 described conflicting internal documents and an unowned source.

The task stayed low risk. The output could recommend a reversible prototype, an investigation, or source reconciliation. It could not send a message, update a record, delete data, purchase anything, or publish a result.

This structure follows a useful distinction in Anthropic's evaluation guidance. A task has defined inputs and success criteria. A trial is an attempt. A transcript records the interaction. The outcome is the final state, not the agent's claim that it finished. Anthropic's evaluation guide uses those distinctions for agent testing; they also make a service comparison less vague.

OpenAI's eval reference makes the same practical point in a different form: an evaluation has a data source configuration and testing criteria, then produces runs and grader results. The OpenAI eval reference is a useful reminder to save the cases and the checks, not just the final screenshot.

What did tutoring leave behind?

The tutor-led artifact was an operator-authored prompt card. The provider asked the operator to define the fields first, identify what should require review, and run the cases. The provider did not type the workflow.

The first pass exposed a real failure. F-03 returned needs_review=false and proposed using the conflicting documents as an automation source. The operator then added this rule:

If the note contains conflicting source documents or an unowned source of truth, set needs_review=true and make the next action ownership and reconciliation, not automation.

The operator reran F-03 and got a corrected result: medium confidence, review required, and a next action to assign an owner and reconcile the documents before automating against them.

The failure matters more than the corrected sentence. A tutor-led session that only displays good answers can hide whether the learner understands the boundary. A changed rule and a rerun expose it.

Marius Manolachi describes his own AI consulting in this direction: the work happens on the client's actual work, the client does the typing, and the point is that the person can do it without him afterwards. His service description is a situated example of tutoring as capability transfer, not a measured result for this test.

When I taught product managers who moved from writing specifications to building and shipping, the recurring gap was often an undefined meaning of done, not a missing trick. That observation is useful here because the tutor's acceptance test must define what independent means. It is not an outcome rate.

What did implementation leave behind?

The implementation-led artifact was a small runnable Python command. It accepted the three named case IDs and emitted the same six-field JSON shape for each case. The shell validation checked syntax, case order, field names, and the review rule for F-03.

The implementation path produced a better immediate handoff. It had a run command, fixtures, a task brief, saved cases, expected checks, and a note describing the read-only boundary. Someone else could execute it without reconstructing the provider's design decisions from a conversation.

But the handoff check found its limit. The operator could run the command and inspect the output. When asked to make the defined change, the operator needed provider help to locate the code line. That is not a failure of the artifact's run path. It is a failure against a stronger ownership claim.

This is why “we handed over the workflow” is too broad to accept as evidence. A handoff can mean that the system starts, that the documentation exists, that the buyer owns the account, or that the buyer can change a rule. Those are separate claims. Test the one you intend to buy.

How should a buyer reproduce the comparison?

Use a workflow that is valuable enough to matter and safe enough to stop. A draft brief, internal classification, read-only knowledge query, or reviewable extraction task is a better first test than an action that sends, deletes, pays, or changes customer data.

1. Write the before-state. Name the current input, manual steps, output, owner, and failure cost.
2. Freeze one task brief. Include the output fields, success checks, permissions, and what counts as review.
3. Save three to five cases. Include an ordinary case, an incomplete case, and a conflict, missing-owner, or uncertainty case.
4. Run the tutor mode. The operator must type or configure the method, explain the rule, and retain edit access. Save the transcript or screen record.
5. Run the implementation mode. Require a working artifact, run command, documentation, test cases, actual results, permissions, and handoff owner.
6. Use the same change scenario in both modes. For example, “When sources conflict, require review and change the next action to reconciliation.”
7. Rerun the affected case. Record who made the change, how long it took if you are comparing time, and whether the output changed as intended.
8. Keep the failure. A failed case tells you more about the boundary than three selected successes.

The workflow is reproducible when another operator can see the task, inputs, rule, test, output, and change point without trusting the provider's summary.

Which service should you buy?

Choose based on the thing that must remain after the engagement.

If your priority is...	Ask for...	Accept only when...
Independent judgment	Tutor-led matched task	The operator completes a fresh case, explains the rule, makes a defined change, and reruns it.
A working system this month	Lightweight implementation	The artifact runs, the inputs and outputs are documented, saved tests pass, permissions are explicit, and the handoff identifies a change point.
Immediate delivery plus future ownership	Blended engagement	The system acceptance test and the operator capability test both pass.
A demo for internal alignment	Discovery or prototype	The provider labels it a prototype and does not sell the demo as a handoff.

My decision rule is simple. If the operator must own judgment and future changes, tutoring wins unless the change test fails. If the business needs a bounded working system now, implementation wins unless the artifact or handoff test fails. If both conditions matter, do not make one deliverable stand in for the other.

The Orange workshop is a useful situated example of the first step: start from the work people already do, not from a fashionable architecture. Marius Manolachi led a ChatGPT workshop at Orange, and the locked observation is that the work came first. It is not proof that a workshop transfers capability on every team.

TryUncle makes the ownership boundary unusually explicit in a different setting. It guides while the user remains the editor and makes every change. That is not the same as a lightweight implementation service, but it is a clear example of how a provider can state who controls the action. TryUncle's product boundary is a situated entity fact, not a result rate.

What should the contract and handoff record contain?

The contract should name the acceptance evidence, not only the activity. Ask for these fields:

Field	Minimum record
Task	One named workflow, before-state, inputs, and output contract.
Permissions	What the provider, operator, and system may read, write, send, or change.
Tests	Saved cases, expected checks, actual results, and at least one failure.
Artifact	Prompt, configuration, code, or workflow that another person can inspect and run.
Handoff	Owner, run command, documentation, rollback or stop path, and exact change point.
Capability	A fresh case completed by the operator, followed by a defined change and rerun.
Limits	What the test does not prove, such as production accuracy, security, or scale.

The most important line is often the smallest: “The operator changed X without provider access.” If the answer is no, write down whether the purchase was implementation, tutoring, or an intentionally dependent service. Do not let the label decide after the fact.

UNESCO's AI competency framework describes human agency as a principle and organizes competency progression into Acquire, Deepen, and Create levels. That framework is for teachers, not buyers of AI consulting, so it does not validate this service test. It does support a useful interpretation: capability should be observed as what a person can do and create, not inferred from attendance or enthusiasm. UNESCO's framework makes that progression explicit.

What can this test not tell you?

It cannot tell you whether a particular tutor is good, whether a particular implementation will survive production, or whether a model will be accurate on your real data. It cannot tell you how much the engagement will cost or how quickly a team will adopt the workflow.

It is also not a randomized comparison. It used one operator, one low-risk workflow, three cases, and a fixture-backed implementation boundary. The implementation artifact proves that a runnable interface and handoff can exist. It does not prove model integration, uptime, security, or maintainability at scale.

Those limits are part of the evidence. A small test can answer a small question well: did this engagement leave a tested change capability, a tested runnable artifact, or both? It cannot answer a larger question by implication.

If your workflow touches money, customer communications, production records, or sensitive data, keep the first matched task read-only and add the relevant approval, security, privacy, and rollback checks before expanding it. For broader contract questions, compare this worksheet with what a buyer should measure after an AI consultant leaves. For the broader make-or-buy decision, use Should We Buy an AI Implementation or Learn Internally?. This comparison sits in the AI commercial pillar.

The purchase is ready when the evidence is boring: the operator can make the change, the artifact can run, the tests are saved, permissions are clear, and someone owns the next failure. Until then, you have a promising service conversation, not proof of tutoring or implementation.