Create a representative test set
Include common tasks, difficult cases, ambiguous requests and missing information. Use data you are entitled to process, with appropriate protection. Keep some examples separate from development work to reduce the risk of tailoring the system to a familiar set of questions.
Define success and unacceptable outcomes
Assess dimensions relevant to the product: factual support, task completion, appropriate refusal, privacy and consistency. Decide which failures require human review or prevent release. Avoid relying on one average score that hides a serious failure in a small but important group of cases.
Test the surrounding system
Check access controls, retrieved information, external actions and fallback behaviour. For systems connected to tools, assess whether instructions in untrusted material can trigger inappropriate actions. Confirm that failures, timeouts and unavailable services lead to a usable recovery path. Involve technical reviewers with the relevant expertise.
Repeat evaluation after changes
Record the model version, instructions, relevant settings and evaluation results. Rerun suitable tests after changes to prompts, models or connected data. Monitor production issues and update the test set with newly observed failures. NIST’s Generative AI Profile offers a structured reference for evaluating and managing these risks.
Your next steps
- Include ordinary and difficult tasks.
- Define unacceptable failure cases.
- Test access and external actions.
- Record evaluation conditions and results.
- Retest after material changes.