AI agent evaluation for changes you can inspect

Agent regression testing starts with concrete cases. Capture an input, identify the workflow target and version under test, and define the checks that matter for that case. Rubric-backed suites support assertions such as exact output matching and HTTP status, with composite checks when a single measure is insufficient.

For example, test a support workflow with a known ticket and expected response fields. Include an unavailable account service, an empty input, and a request the agent should not fulfill. These cases make failures easier to interpret than a single average score over unrelated tasks.

Run the same suite against the candidate workflow version, then inspect case-level results and compare them with a baseline. Pass thresholds and regression limits provide release-readiness signals. Scheduled runs help repeat these checks as the surrounding integrations change.

Shadow runs can use sampled trace inputs to exercise a candidate target. Treat the target as an executable integration: choose test data and an environment appropriate for the actions it performs. Inspect failed cases individually so that a high overall pass rate does not hide a critical regression.

Use the eval review queue to examine failed or judged cases and record a review outcome. Incident-generated case drafts give the team a starting point for extending the suite. Review the captured input and expected behavior before promoting a draft into a lasting regression test.

Robots Center records workflow version promotions and their history. Your team remains responsible for connecting the evaluation decision to its deployment process; recording a promotion does not deploy infrastructure or automatically enforce an eval gate. This keeps the meaning of an approved change explicit.