Updated September 2026Reproducible evaluation

Evaluate AI against the work it must do—not the benchmark headline that sold it.

This guide shows how to build a repeatable model and AI-system evaluation program using representative workflows, predefined acceptance criteria, security and failure testing, cost evidence, version control, and documented re-evaluation triggers.

The 2026 revision removes obsolete M-24-10 assumptions and distinguishes stable evaluation practice from legal or provider-specific claims that can change.

Build an evaluation brief Open the evaluation scorecard See a current model evaluation example

Evaluation purpose

Define the decision before selecting the metric.

An AI evaluation is useful only when it supports a decision: approve a model for a workflow, compare suppliers, promote a new version, expand a use case, accept a known limitation, or reject a deployment. Start by documenting the intended task, users, data, tools, consequences of failure, human-review model, and the person authorized to make the final decision.

NIST's AI Risk Management Framework provides a durable organizing structure through Govern, Map, Measure, and Manage. Evaluation work sits heavily in Measure, but good measurement depends on the earlier context work. A benchmark score without the mapped use case says little about whether a model is acceptable for an organization's actual deployment.

Current federal governance context also matters. OMB M-25-21 replaced the earlier M-24-10 federal AI-use framework and continues the distinction between ordinary AI use and higher-consequence use where stronger governance and safeguards are appropriate. A non-federal buyer can still use the same design principle: increase evaluation rigor when a system can affect rights, safety, money, access, critical services, sensitive data, or irreversible external actions.

For EU-facing systems, the amended AI Act has staggered application dates and role-specific duties. Evaluation teams should record the specific legal role and use-case classification rather than claim that one internal test suite proves generic “AI Act compliance.” Testing evidence can support a compliance program; it does not replace legal classification or conformity obligations.

Test portfolio

Build a small representative suite before building a giant benchmark lab.

A focused set of difficult, real tasks produces more decision value than hundreds of generic prompts that do not resemble production.

Normal-path tasks

Use the common work that drives expected value. Capture expected outputs, acceptable variation, latency, cost, and review burden.

Edge and failure cases

Include ambiguity, conflicting source material, missing context, tool failures, malformed inputs, long documents, unusual formats, and historically difficult examples.

High-consequence cases

Where failure could matter materially, test abstention, escalation, human confirmation, auditability, and the controls that prevent an output from silently becoming a decision.

Define acceptance before execution. For each task, state what counts as success, partial success, unacceptable error, and automatic failure. If reviewers invent the scoring rule after seeing model outputs, the evaluation becomes vulnerable to confirmation bias.

Use a mixture of quantitative and qualitative measures. Exact match, precision, recall, structured-field accuracy, tool-call success, latency, and cost are useful where they fit. Human rubric scoring may be appropriate for drafting, summarization, reasoning transparency, usability, or judgment-heavy tasks, but the rubric and reviewer instructions should be explicit.

Keep public benchmarks in their proper place. A provider's benchmark result helps identify likely strengths and weaknesses and can justify deeper testing. It should not be copied into an internal scorecard as if the organization independently reproduced it. Zeph Tech's Gemini 3.7 Flash evaluation demonstrates this separation between vendor-reported evidence and buyer acceptance evidence.

Evidence record

Make every evaluation rerunnable six months later.

For each run, preserve the model or service identifier, provider, date, configuration, system prompt or policy layer, thinking or reasoning settings where relevant, connected tools, tool permissions, retrieval configuration, input dataset version, test cases, scoring rubric, evaluator, raw outputs or appropriate hashes, result summary, and decision.

Separate three evidence categories. Provider evidence includes model cards, benchmark tables, certifications, technical documentation, and product statements. Organization evidence includes internal acceptance tests, security review, accessibility review, cost modeling, demonstrations, and operational observations. Unresolved claims are items the organization could not verify. Keeping those categories separate prevents vendor marketing from quietly becoming an internal fact.

Version the test set as well as the model. Evaluation data can become stale, leak into training or public discussion, or cease to represent the production workload. Record why a test exists, which risk or requirement it covers, and what change should cause it to be retired or refreshed.

The Software Evaluation Scorecard can preserve comparable decision evidence across competing services. For an early-stage evaluation, the Evaluation Brief Builder helps turn workflow facts and constraints into questions and test areas before a vendor demonstration begins.

Security and abuse testing

Test what happens when inputs, tools, and instructions are not trustworthy.

Prompt and content attacks

Prompt injection inside retrieved or uploaded material.
Conflicting instructions from untrusted documents.
Attempts to reveal hidden instructions, secrets, or restricted data.
Jailbreak or policy-bypass behavior relevant to the use case.
Malicious or malformed structured input.

Agent and tool failures

Permission denied and expired credentials.
Unavailable tools and partial API failure.
Repeated or duplicate actions.
Incorrect tool selection or parameters.
Actions requiring confirmation that the system attempts autonomously.
Recovery after a tool succeeds only partially.

The objective is not to prove a model can never be manipulated. It is to understand the attack and failure paths that matter for the proposed deployment and whether surrounding controls contain them. For systems with sensitive data or privileged integrations, pair model evaluation with supplier-security review using the Vendor Security Questionnaire.

Evaluation data itself may be sensitive. Treat prompts, user records, red-team payloads, model outputs, logs, and reviewer notes according to their information classification. Avoid building a testing process that copies regulated or confidential production data into an uncontrolled lab merely to improve realism.

Operational measures

Measure the cost and burden of a successful workflow, not only token price.

For hosted models, cost evaluation should include input and output tokens, retrieval context, tool calls, retries, orchestration, caching where applicable, human review, failure remediation, and duplicated runs needed for assurance. A lower token price can still produce a higher cost per successful workflow if the service generates more rework or requires more retries.

Latency should be measured at useful percentiles and under representative concurrency. Record time to first usable output where relevant, total workflow duration, tool latency, timeout behavior, and recovery. For interactive systems, average latency alone can hide a poor tail experience.

Track human effort explicitly. Measure review minutes, correction frequency, escalation, and the percentage of tasks that need a person to start over. AI can appear highly accurate while still creating operational burden if reviewers must verify every detail manually.

For long-running systems, add monitoring indicators connected to known failure modes rather than generic “AI health.” Examples include unsupported-claim rate, abstention rate, tool failure, action rollback, override frequency, retrieval failure, policy violations, cost drift, and task-success drift on a fixed sentinel set.

Change control

Assume the model, provider, and workflow will change.

A good evaluation program defines the event that forces a new run before that event happens.

Common re-evaluation triggers include a new model version, changed provider terms, pricing changes that alter the architecture, new tool permissions, a different system prompt, new retrieval sources, a new user population, new sensitive data, a new consequential use, major workflow redesign, security incidents, material provider architecture changes, or model deprecation.

Re-evaluation should be proportionate. A low-risk drafting assistant may need a focused regression suite after a minor model update. An agent that can modify external systems or a system influencing consequential decisions should face a broader regression, security, and governance review.

Preserve a rollback path. Know the last approved model or configuration, whether it remains available, how traffic can be shifted, and what manual process exists if the AI service must be disabled. Evaluation is much more useful when a failed release test can actually stop or reverse a deployment.

Review the evaluation library itself on a cadence. Retire tests that no longer represent production, add cases from incidents and user complaints, refresh benchmarks that have become contaminated or obsolete, and keep a small sentinel set stable so model changes can be compared over time.

Decision record

End every evaluation with an explicit disposition.

Record whether the model or system is approved, conditionally approved, restricted to defined uses, rejected, or awaiting evidence. Identify mandatory conditions, open findings, the risk owner, expiration or next-review date, and the evidence relied upon.

Do not collapse mandatory failures into an average score. A model can perform well overall and still be unacceptable because of a critical privacy, security, accessibility, legal, records, safety, or continuity issue. Weighted scoring is useful for tradeoffs only after non-negotiable gates are understood.

Connect evaluation to procurement rather than storing it in a separate AI laboratory. The current AI Procurement Governance Guide shows how model evidence should flow into supplier diligence, contracting, monitoring, and exit planning.

Primary sources and current references

Last substantive review: September 2, 2026. Evaluation methods should be adapted to the actual model, workflow, risk, data, and legal context rather than copied as a universal test suite.

Put this guide to work

Turn AI Model Evaluation Operations Guide | Zeph Tech into a decision-ready next step.

Use the source-backed research to pressure-test assumptions, then build a reusable evaluation brief before you compare products, scope implementation, or request a fit review.

Build an evaluation brief Browse AI research Explore the full resource library See where ZephCMS may fit