Why AI Evaluation Must Continue After Deployment

Testing an AI application before release is essential, but it does not establish that the application will remain reliable in production.

Enterprise AI systems operate in a changing environment. Users discover new ways to ask questions. Knowledge sources are revised. Prompts and retrieval settings evolve. Model providers release new versions. Business processes and policies change.

The system that passed evaluation at launch is not exactly the system users experience several months later.

This is why evaluation should not be treated as a final project gate. It should become a continuous production capability.

 

Pre-Release Evaluation Answers a Limited Question

Before deployment, teams normally create a set of representative cases and measure how well the application handles them.

This can reveal obvious weaknesses. It can compare prompts or models, test retrieval quality, measure structured output accuracy and identify unsafe behavior. The results provide a baseline for a release decision.

However, the test set represents what the team knew to test at that time.

It may not include the language used by a new group of employees. It may not reflect a recently published policy. It may underrepresent rare but important cases. It cannot fully predict how people will adapt their behavior once the system becomes part of daily work.

Pre-release evaluation asks: Does this version meet our criteria on known cases?

Production evaluation asks: Is the live system still delivering acceptable outcomes under current conditions?

Both questions matter.

 

 

What Changes After Deployment

AI quality can change even when no one intentionally modifies the application code.

User behavior changes

Early users may ask simple questions while they learn the interface. Later, they try longer, more ambiguous or more consequential tasks. A system designed for information retrieval may gradually be used for recommendations or decisions.

This is not only prompt drift. It is use-case drift.

Knowledge changes

Documents are added, revised and withdrawn. Indexing can fail. Metadata may be incomplete. Two sources may begin to contradict each other. A previously correct response becomes outdated because the authoritative policy changed.

Models and configurations change

A model version, safety setting, prompt, embedding model, reranker or context limit can change system behavior. Even an improvement in average quality may introduce a regression for a specific task or language.

Dependencies change

External APIs return new fields or errors. Tool latency increases. An identity service applies a new policy. These changes can affect the final AI outcome even when the model continues to respond normally.

Continuous evaluation makes these changes visible before they become accepted as normal behavior.

 

Evaluate the Complete Outcome

It is tempting to reduce evaluation to whether an answer “sounds good.” Production systems need a wider set of measures.

Task quality

Did the system complete the intended task correctly? The answer may be measured through accuracy, completeness, classification performance, extraction precision or a task-specific rubric.

Grounding and provenance

Is the response supported by the retrieved sources? Are the citations relevant? Did the system rely on an outdated or unauthorized document? A fluent answer without evidence may still be unsuitable for enterprise use.

Safety and policy compliance

Did the application reveal sensitive information, follow a malicious instruction, produce prohibited content or recommend an action outside its intended scope?

Operational behavior

How long did the request take? How often did a dependency fail? How many retries or model calls were required? What did the request cost? Reliability and economics are part of production quality.

Business outcome

Did the system reduce handling time, increase first-contact resolution, improve employee success or prevent errors? A technically impressive response can still fail to create useful business value.

No single metric represents all five dimensions. The evaluation design should reflect the decisions the system supports and the harm an incorrect outcome could create.

 

Combine Offline and Online Evaluation

Offline evaluation uses a controlled dataset. It is repeatable and useful for comparing versions before release.

Online evaluation uses production traces, feedback and outcomes. It reveals real behavior, but the data is less controlled and may contain sensitive information.

The strongest approach connects them.

Production failures should become carefully reviewed test cases. Important offline scenarios should be monitored in the live system. A release candidate should be compared with the current version before traffic is shifted. After release, selected requests should be evaluated against the same quality criteria used during development.

This creates a feedback loop rather than two disconnected activities.

 

Sample Production Interactions Deliberately

Evaluating every request with a second model can be costly, slow and sometimes unreliable. Evaluating only random requests can miss the cases that matter most.

A practical sampling strategy can include:

• A random sample for general quality trends.
• All requests involving high-impact topics or tools.
• Requests with low confidence or weak retrieval scores.
• Requests that produced errors, retries or unusual latency.
• Negative user feedback and escalations.
• New intents or input patterns not represented in the test set.
• A small sample from every important user group, language and business process.

Automated evaluators can help prioritize review, but they should not be assumed to provide objective truth. Their own consistency and limitations need evaluation.

For consequential cases, expert human review remains important.

 

Define Thresholds Before an Incident

Metrics become controls only when the organization knows what to do with them.

A dashboard that shows declining groundedness is useful, but it is not a complete response. The operating model should define:

• The acceptable range for each critical measure.
• The time window over which a change is significant.
• Who receives the alert.
• Which team investigates.
• When traffic is reduced or a feature is disabled.
• How the previous model, prompt or index is restored.
• What evidence is required before service resumes.

Thresholds do not have to be perfect initially. They can improve as the organization learns. What matters is that evaluation is connected to an operational decision.

 

Version Everything That Can Affect an Outcome

When quality changes, the team must be able to identify what changed with it.

At minimum, production traces should connect an evaluated result to the relevant versions of:

• The application and workflow.
• The model and its parameters.
• The system prompt and reusable instructions.
• The retrieval and ranking configuration.
• The knowledge sources or index snapshot.
• The guardrail and policy configuration.
• The tools and schemas used.

Without versioning, a poor result may be visible but not reproducible. Investigation becomes guesswork.

 

Treat Feedback as Evidence, Not a Score Alone

Thumbs-up and thumbs-down controls are easy to add, but the score by itself rarely explains what needs to change.

When possible, feedback should capture a reason: incorrect, incomplete, outdated, irrelevant, unsafe or difficult to understand. The system can also record what happened next. Did the user rephrase the question, open a cited document, escalate to a person or abandon the task?

Behavioral signals need careful interpretation. A user may copy an answer because it is useful, or because they need to correct it elsewhere. Feedback should inform investigation rather than replace it.

 

Evaluation Is Also a Governance Mechanism

Continuous evaluation connects technical operation to governance commitments.

If the organization claims that answers are grounded in approved content, production evaluation should test grounding and source authorization. If high-impact recommendations require human review, traces should show whether the review occurred. If a system is approved only for a defined purpose, monitoring should detect when users begin applying it elsewhere.

This is how governance moves from an initial assessment to ongoing assurance.

 

Start with a Small Evaluation Contract

Teams can begin by defining an evaluation contract for each production use case.

The contract states:

1. The task the system is expected to perform.
2. The dimensions that define an acceptable outcome.
3. The test cases and production samples used to measure them.
4. The thresholds that trigger investigation or rollback.
5. The owners who decide whether the system remains fit for use.

This creates a shared definition of quality across business, engineering, risk and operations.

 

Conclusion

An AI release is not the end of evaluation. It is the beginning of evaluation under real conditions.

Pre-release tests create confidence in a known version. Production evaluation protects that confidence as users, data, models and dependencies change.

Organizations that treat evaluation as a continuous capability can improve AI systems deliberately. They can identify regressions, learn from real use and connect quality signals to operational action.

The objective is not to prove that an AI system is permanently correct. It is to know whether the system remains within the boundaries the organization has accepted—and to respond when it does not.

 

Sources

• NIST Artificial Intelligence Risk Management Framework
• Microsoft Foundry observability
• Microsoft: Monitor generative AI applications