THE SHORT ANSWER
Evaluate the whole job, not only whether a model's answer sounds good. Define success and unacceptable failures, test representative and difficult cases, inspect sources and actions, measure cost and speed, and confirm that people can detect, correct, and recover from mistakes.
Begin with an outcome
Describe the task in observable terms: what enters the system, what correct output looks like, and who uses it. Decide which errors are tolerable and which require escalation. A system that drafts a note and a system that changes an authoritative record need different evidence and thresholds.
Create a fixed test set from realistic cases, with sensitive information removed or handled under approved controls. Include ordinary examples, incomplete inputs, conflicting facts, adversarial requests, and changes in upstream data. Have domain experts review outputs against the same rubric.
Measure more than accuracy
Track groundedness, task completion, correct refusal or escalation, unauthorized actions, reviewer correction rate, latency, and cost where relevant. Record the input, sources, version, tool calls, and outcome so a failure can be reproduced. Averages are useful, but the worst categories may drive the release decision.
The NIST Generative AI Profile is a cross-sector resource for identifying and managing generative AI risks. It is a starting point for questions, not a substitute for tests specific to your workflow.
Keep evaluation alive
Production changes the data and behavior the system sees. Monitor drift, failed tool calls, reviewer overrides, and downstream outcomes. Re-run a stable test set after changes to prompts, models, tools, or source data. Set a threshold for pausing or narrowing the feature when evidence worsens.
Illustrative example: a contract assistant may quote the right clause on 90% of test questions yet fail on amended agreements. The release decision depends on whether amendments are common, whether errors are detectable, and whether a reviewer sees the source before using the answer.
Original sources
The process explanations above are educational summaries. The linked primary sources govern their own terms and can change; check them for current requirements.