Why Does My AI Feature Work in Staging but Fail in Production?

I built a small support-ticket classifier to test this exact failure. The staging fixture passed. Then I changed one production-like condition at a time.

The result was clear: the failure did not have one generic cause. Each change stopped at a different trace step, and each exact case passed after one minimal repair.

Controlled production-like difference	First divergence	Minimal repair	Replay
Model changed to fixture-model-v2	resolve_model	Pin fixture-model-v1	9/9 pass
Prompt missing	load_prompt	Load triage-prompt-v1 and fail closed	9/9 pass
Ticket body changed from string to array	parse_input	Normalize and validate the body	9/9 pass
Permission changed to denied	authorize	Grant ticket.read	9/9 pass
Adapter changed to adapter-v2	resolve_dependency	Pin adapter-v1	9/9 pass
Upstream returned HTTP 429	call_upstream	Add bounded retry or a replay fixture	9/9 pass

This is the sourceable result from the experiment. It is a six-case reproduction, not a claim about the frequency of production incidents.

Why does staging success fail to predict production success?

Staging success proves that one system configuration passed one set of inputs. It does not prove that production has the same model, prompt, data shape, identity, dependency tree, or upstream behavior.

That distinction is easy to miss with AI features because the visible output makes the system look like one component. It is not. The model sits inside a path:

config -> model -> prompt -> input -> permissions -> dependencies -> upstream -> output -> state

The Twelve-Factor App config guidance treats service handles, credentials, and other deploy-varying values as configuration rather than code. The same idea applies to an AI feature's model ID, prompt version, provider endpoint, feature flag, and tool permissions.

Production can therefore fail before the model is called. A missing prompt can look like poor instruction following. A denied permission can look like an empty retrieval result. A different dependency can look like a malformed model response. A 429 can look like an unreliable feature.

The first question is not “Which prompt should I try?” It is “Where did the production trace first stop matching the staging contract?”

What did the failure reproduction show?

The smallest useful fixture was a ticket classifier with one known input, one pinned response, and one saved result. I ran it with Node v20.11.0 and npm 10.2.4, using only the standard library. The fixture version was support-ticket-v1; the app version was triage-feature-0.1.0.

The staging baseline was:

model: fixture-model-v1
prompt: triage-prompt-v1
input:
  ticket:
    id: T-1042
    body: Customer was charged twice.
permission: ticket.read
dependency: adapter-v1
upstream_status: 200
expected_output:
  category: billing
  priority: normal

The harness checked nine steps in order: load_config, resolve_model, load_prompt, parse_input, authorize, resolve_dependency, call_upstream, validate_output, and persist_result. It stopped at the first failed check and recorded the observed value beside the expected value.

Here is one trace in full:

load_config:pass
resolve_model:pass
load_prompt:pass
parse_input:pass
authorize:pass
resolve_dependency:pass
call_upstream:fail, observed=429, expected=200

The repair was not “make the model more reliable.” It was to handle the upstream response with a bounded retry policy or replay the pinned 200 response in the test. After repair, the same case reached persist_result:pass.

Anthropic's evaluation guidance makes the same operational distinction between a trace, which records the trial's interactions, and an outcome, which is the final state in the environment. That distinction matters here: a returned classification is not enough unless the application validates and saves the expected result (Anthropic's evaluation guidance).

How do I find the first production divergence?

Replay the same fixture and compare the two traces in execution order. Do not compare only the final answer.

1. Freeze the case. Store the input fixture, expected output, app version, model ID, prompt version or hash, input schema version, permission set, dependency lockfile hash, and upstream response fixture.
2. Run the staging baseline. Confirm that the fixture reaches the final state and save the complete trace.
3. Run the production path. Use the same case, with secrets redacted and side effects isolated. Save the same fields.
4. Normalize values. Compare model IDs, prompt identifiers, schema versions, status codes, permission decisions, dependency versions, and output-validation results. Ignore timestamps and request IDs when comparing paths.
5. Stop at the first mismatch. The first divergence is the next hypothesis. Later failures may be consequences, not independent causes.
6. Repair one invariant. Change one condition, then replay the original failing case and the rest of the regression set.

The output you want is concrete:

first_divergence: authorize
observed: ticket.read:denied
expected: ticket.read
repair: grant ticket.read to the feature identity
replay: 9/9 trace steps passed

This procedure is also why an evaluation suite needs defined inputs and success criteria. A trace without a pass condition is just a log. A pass condition without a trace leaves you guessing.

Which staging-to-production differences should I check first?

Check the invariant that fails earliest in the request path. The order below follows the reproduction, not a universal incident ranking.

Invariant	What to compare	Typical first symptom	Smallest useful test
Model configuration	Exact model ID, provider, endpoint, temperature or reasoning settings, and structured-output mode	Different refusal, schema, quality, or cost behavior	Log the resolved model configuration and run a pinned case
Prompt and config loading	Prompt version or hash, feature flags, environment variable presence, and fallback behavior	Missing instructions or a default prompt	Fail closed when the required prompt is absent
Data shape	Schema version, nullability, array versus string, truncation, encoding, and size	Parser error, empty context, or wrong classification	Validate the production-shaped fixture before model invocation
Permissions	Service identity, project, resource scope, network allowlist, and tool policy	401, 403, empty retrieval, or skipped tool	Run a read-only authorization preflight
Dependencies	Lockfile, runtime, SDK, adapter, and install mode	Import, serialization, or response-path mismatch	Use a clean install from the lockfile
Upstream response	Status, body shape, request ID, timeout, retry headers, and rate limit	Timeout, 429, malformed output, or partial result	Replay success, error, timeout, and malformed-response fixtures

For JavaScript projects, npm ci is designed for clean automated installs. It requires a lockfile, fails when the lockfile and package manifest disagree, and does not rewrite the lockfile (npm's npm ci documentation). That makes the lockfile part of the AI feature's production contract, not build housekeeping.

For provider calls, inspect the actual error class. OpenAI's current error documentation separates invalid authentication, IP authorization, and 429 conditions, among others (OpenAI API error codes). The fix depends on that class. Rotating a key will not repair a malformed input, and changing a prompt will not create a missing permission.

How should I turn the diagnosis into a release check?

Make each discovered invariant observable before the next release. A useful release record has these fields:

release:
  app_version: triage-feature-0.1.0
  model_id: fixture-model-v1
  prompt_version: triage-prompt-v1
  input_schema: ticket-v1
  dependency_lock_hash: sha256:...
  service_identity: ai-triage-reader
  upstream_contract: classifier-response-v1
checks:
  - config_preflight
  - prompt_present
  - input_schema_valid
  - permission_preflight
  - clean_dependency_install
  - upstream_error_replay
  - output_state_check

Treat security and authorization failures as vetoes. A feature that classifies correctly with the wrong identity has not passed. The NIST AI RMF emphasizes deployment context, interdependencies, documentation, testing, monitoring, and contingencies for third-party failures. Those are production properties, not extra polish.

OpenAI also recommends keeping API keys on a backend and using environment variables or a key-management service rather than exposing keys in client-side code (OpenAI API key safety). The practical check is not “does an environment variable exist?” It is “does the production identity resolve to the intended provider, project, permissions, and endpoint without exposing a secret?”

What if the trace is clean but production still fails?

Then the mismatch is probably in live traffic, provider variance, timing, or an unrepresented combination of conditions. A clean trace does not prove that the fixture is representative.

Add the smallest new case that reproduces the live behavior:

• a production-shaped input with sensitive fields removed;
• the provider status and response shape;
• the exact model and prompt identifiers;
• the identity and permission decision;
• the dependency and runtime versions;
• the final user-visible and system state.

If the failure is nondeterministic, run the case across multiple trials and record the pass rule. If the failure changes state, verify the outcome rather than only the generated text. If it is high impact, use a shadow or approval-gated path before replaying it against live side effects.

I have seen the same release mistake while teaching product managers to move from specifications to shipped products: the hard part is often that nobody has defined what “done” means, not that the model needs a cleverer instruction. That is a bounded teaching observation from Marius Manolachi's work, not a failure rate (Marius Manolachi's AI teaching work).

The next useful artifact is a regression case, not another prompt experiment. The existing AI agent evaluation guide gives the parent release context; the broader AI product evaluation guide explains the surrounding evaluation job. The guide to building an evaluation dataset from production traces explains how to keep a confirmed failure as a future test. For runtime visibility, continue with AI agent observability.

If your team needs help turning a working demo into a testable production contract, Marius Manolachi's AI consulting and tutoring work is the next step. Bring the failing trace.