How to Choose Post-Tutoring Outcomes for a Legacy System

Most tutoring engagements produce a reaction quickly. The team says the session was useful, or a demo works while the tutor is present. The hard question comes later: what changed in the legacy system, and what should you do next?

The scorecard below answers that question with a small, tested exercise. It records a learner behavior, the old system's constraint, a separate follow-up case, and a disposition for the legacy slice. It is deliberately not a client case study.

Start with a behavior that can change a system decision

Choose a behavior the learner can perform on the actual legacy workflow, then attach the behavior to a system signal and a disposition. Do not start with satisfaction, confidence, or a broad claim such as “the team understands AI.”

When I taught product managers who went from writing specs to building and shipping the product and automating work around it, the failure was almost never the model. It was that nobody could say what done meant. That locked F-pms observation is why a post-tutoring outcome needs a visible behavior and a definition of done, not just a positive session reaction. Learn more about working with Marius Manolachi.

Here is the completed artifact from the exercise:

Scorecard field	Worked entry
Starting capability	The learner could follow the main READY path but missed the manual-review branch. The baseline assertion failed.
Legacy constraint	A billing function has a fixed result shape, stringly typed inputs, and a manual-review flag that must remain intact.
Behavior to learn	Wrap the old function, observe its result, and return the original result unchanged.
Pre/post check	Baseline: 0/1 on the critical manual-review case. Post: three baseline outputs unchanged; wrapper checks passed.
Separate follow-up	A new Canada invoice with manual review passed 1/1, with HOLD preserved and an observation recorded.
System signal	legacy_output_unchanged=true, with observation records for baseline and follow-up cases.
Disposition	Retain the legacy core and wrapper seam for this slice. Test the next slice before expanding replacement.

The sourceable result is specific: on this constrained exercise, the useful outcome was not “learner knows modernization.” It was “learner can add an observation seam without changing the old billing contract, then repeat that behavior on a new case.”

Fill the scorecard in this order

The order matters because each field narrows the next one.

1. Name the starting capability. Write what the learner can do before tutoring. Use a task or a branch, not a self-rating. “Can explain the modernization strategy” is weak. “Can trace the READY path but misses manual_review=Y” is testable.

2. Name the legacy constraint. Record the thing the learner must preserve: a result shape, file format, batch boundary, unsupported runtime, hidden branch, or system of record. The constraint is part of the learning task.

3. Write the behavior as an action. Use a verb and an object. “Wrap process_invoice, record the returned code, and preserve the original result” can be observed. “Become more capable” cannot.

4. Create a pre/post check. Use the same contract before and after the session when possible. The CDC training evaluation guidance recommends comparing pretest and posttest results when evaluating change. A posttest alone can show proficiency, but not what changed.

5. Add a separate transfer check. Use a new case that keeps the same behavior but changes the context. In the exercise, the follow-up changed the country and preserved manual review. CDC describes delayed follow-up as the best way to assess whether learning was applied at work. A separate invocation is the minimum honest version for a reproducible exercise.

6. Choose the system signal. Pick one observable signal that could affect an operating decision: unchanged output, fewer manual corrections, a passing contract test, lower rework, or a verified handoff. Do not promise a business KPI if the engagement cannot observe it.

7. Name the owner, date, and disposition. Someone must collect the follow-up and decide whether the legacy slice is retained, wrapped, replaced incrementally, or scheduled for retirement. A score without an owner is a note, not an outcome.

Separate learning evidence from transfer evidence

A session check and a workplace check answer different questions. The first asks whether the learner can perform the target behavior under the learning conditions. The second asks whether the behavior survives a new case, a real constraint, and the absence of the tutor.

Checkpoint	Question	Evidence in the exercise	What it can support
Baseline	What does the learner miss before tutoring?	The copied baseline logic failed the manual-review assertion.	Select the teaching target.
Immediate post	Can the learner preserve the old contract while adding the behavior?	Three expected legacy outputs remained unchanged, and observations were recorded.	Claim task performance on this fixture.
Separate follow-up	Can the learner transfer the behavior to a new case?	The Canada plus manual-review case passed in a separate process invocation.	Continue practice or inspect the transfer barrier.
Operational review	Did the behavior create a system signal?	legacy_output_unchanged=true; the observation seam exists.	Permit the next bounded slice, not a whole-system rewrite.

Do not confuse these layers. A strong post-session result with a weak follow-up means the tutoring worked inside the session but transfer is unproven. A strong follow-up with no operational signal means the learner may be improving while the system remains unchanged. For a related measurement boundary around engineering work, see the small-team AI coding time guide.

The distinction has support beyond the scorecard. In a randomized study of 61 clinical nurse specialists, training changed simulated competence, but only the group that received supervision showed evidence of transfer at follow-up. The lesson for a technical engagement is not that tutoring always needs supervision. It is that a transfer check needs a workplace condition and an owner, not just a test completed while an instructor is present. PubMed's study record supports that bounded interpretation.

Let the disposition constrain the outcome

The legacy-system disposition is not a final administrative field. It tells you which outcome is worth measuring.

Disposition	Choose this when	Outcome to measure	Veto
Retain and wrap	The old slice is still authoritative, but a safe observation or adapter can be added without changing its contract.	Contract preservation plus an observable wrapper behavior.	No repeatable check for unchanged output.
Replace incrementally	A bounded component has a clear seam and the team can compare old and new behavior.	Learner can route one slice through the seam and explain the comparison.	The slice has no owner or no way to compare results.
Prepare retirement	The replacement has a date, migration work, and a cutover condition.	Learner can identify remaining dependencies and the evidence needed for cutover.	No documented disposition or milestone.
Stop and investigate	The constraint, owner, or signal is still unclear.	Learner can name the missing evidence and the next observation task.	A demo is treated as proof of readiness.

GAO-25-107795 uses three planning elements for legacy modernization: milestones, the work necessary to modernize, and the planned disposition of the legacy system. That is a useful discipline for tutoring outcomes too. If the scorecard cannot say what happens to the old slice, it has not reached a decision.

For the exercise, the right disposition was to retain the core and add a wrapper seam. That choice follows the incremental shape described in AWS Prescriptive Guidance: keep the old implementation behind a proxy at first, then replace a bounded implementation when the evidence supports it. AWS also recommends beginning with a component that has good test coverage and less technical debt, or one with frequent business changes and deployments. The exercise has no production traffic, so it supports only the first bounded slice.

Reject outcomes that cannot change the next decision

Reject any outcome that sounds positive but leaves the buyer unable to choose an action.

Proposed outcome	Why it fails	Decision-ready rewrite
“The team understands AI.”	No baseline behavior, no legacy constraint, no transfer check, no owner, and no disposition.	“The learner can add a wrapper observation around one legacy billing slice, preserve the old result contract, and repeat the change on a new case without tutor assistance.”

Satisfaction still has a place. It can tell you whether the session was usable or whether the learner wants follow-up. It cannot prove learning changed, and it cannot prove the system is safer to change. CDC explicitly separates satisfaction from objective learning and transfer evaluation in its training guidance.

The same rule applies to a working prototype. A prototype can be a learning artifact, but it becomes a post-tutoring outcome only when the learner can explain the boundary, repeat the behavior, and connect it to a system decision. If the prototype changes the system of record, add the required rollback, approval, and production-safety checks. This scorecard does not authorize a live write.

Use the scorecard as a buying gate

Ask for the completed scorecard before extending tutoring or approving a broader modernization slice. The buyer should be able to answer these questions in one page:

1. What could the learner do before the session?
2. What exact legacy behavior should change?
3. How was the change checked before and after tutoring?
4. What separate case tests transfer without the tutor?
5. What operational signal was observed?
6. Who owns the follow-up, and on what date?
7. What happens to the legacy slice if the evidence is positive, mixed, or absent?

Microsoft's assessment guidance starts with an inventory of the digital estate, costs, and organizational readiness, with an expected business plan and cost analysis. A tutoring scorecard is smaller than that assessment, but it should point toward the same decision discipline: identify the asset, record the constraint, name the owner, and make the next investment conditional on evidence. Use the assigned buying guide when you are comparing tutoring with consulting or enablement.

If the answers are vague, buy another observation or practice session, not another promise. If the answers are concrete and the follow-up passes, expand one bounded slice. Keep the legacy system's disposition explicit until the evidence supports a larger change.