Your AI Agent Failed Once. Make It a Regression Test.

“What plan am I on?” should be an easy question for a support agent. Look up the account, read the plan, answer the customer.

But imagine the account service is down. Instead of saying it can't check, the agent confidently replies, “You're on Pro.” You find the problem, fix the fallback, and try again. This time, the answer is sensible.

You could close the ticket there. But what happens when someone changes the prompt next week?

That's where a regression test earns its place. You save a small example of what went wrong, decide what a good answer looks like, and run that same check against future changes. In Robots Center, these checks live in evaluation suites, or evals.

Let's build one for our imaginary support agent. All the data below is made up. If you're still figuring out why your agent failed, our guide to debugging with execution traces is a better starting point.

Keep the evidence that explains the failure

You don't need to turn the whole incident into a test. Start with the part that matters: the customer asked about their plan, the lookup failed, and the agent guessed anyway.

Save a reference to the trace, the workflow version, and any settings that affected the answer. Then write down what you expected instead. Even a short note helps:

The account lookup timed out. The agent should have explained that it couldn't retrieve the plan, rather than naming one.

That note gives you something to test. “The agent gave a bad answer” doesn't.

Keep the original incident evidence somewhere appropriately restricted. For the reusable example, swap real names, emails, and account details for invented ones, and remove credentials. You want to reproduce the mistake, not carry a customer's data into every future test run.

Define success before you tune the agent

What would you have wanted the customer to see?

For this case, something like “I can't retrieve your account details right now. Please try again or contact support” is a reasonable answer. It admits the problem and gives the customer a next step.

There are two things worth checking here: the agent uses the fallback, and it doesn't invent account information. Keep those separate. Finding the right phrase in an answer doesn't prove that everything else in it is true.

Our example endpoint returns HTTP 200 when it successfully handles the failed lookup and sends a fallback. Your endpoint might use a different status. Match your own response contract rather than copying the number because it's in this guide.

Decide this before changing the prompt. Otherwise, it's easy to keep adjusting the expected answer until whatever the agent says happens to pass.

Reproduce the condition in a safe test target

Waiting for another real outage would make this a frustrating test. Instead, give your workflow a test endpoint where you can deliberately make the account lookup fail.

A fixture is just the test data used to recreate that situation. Ours could look like this:

{
  "question": "What plan is my account on?",
  "account_lookup_fixture": "timeout"
}

Your test endpoint needs to understand that timeout value and simulate a failed lookup. It's not a built-in Robots Center switch. The eval runner sends the fixture inside input, alongside version and config; your endpoint decides how to run that version with those settings. The workflow target reference explains that connection.

Test the actual failure path, not just a prompt that asks the model to imagine an outage.

And use an isolated environment. Evals really call the configured endpoint, so an agent that can send emails or change records may still do those things during a test. The same is true of shadow runs. Use test credentials and downstream services that are safe to call repeatedly.

Build a case with an honest pass condition

In Evals → Targets, connect that safe endpoint and configure the versions you want to test. Then open Evals → Suites, create or choose a suite for that target, and click Add case.

Think of a suite as a folder of related examples. For now, you only need one:

1. Call it Account lookup unavailable. Add a sentence about the original mistake in the description.
2. Put the example above into Input payload JSON.
3. Set Expected status code to 200, assuming your endpoint follows our fallback contract.
4. Enter message under Required keys.
5. Use Account details are unavailable as the Expected text fragment.
6. Set this case's Pass threshold to 100, so all of these checks must pass.

Here's a response that would pass:

{
  "message": "Account details are unavailable. Please try again or contact support."
}

This is a useful first check, but it has a limit. “Account details are unavailable, but you're on Pro” would also contain the expected phrase. The key check only confirms that message exists; it doesn't judge what the message means.

So read the whole response, too. To test the no-invented-details requirement automatically, you'll need a stricter check of the output, not just this phrase match. Until you've added one, describe this case for what it is: a check that the fallback appears. That's still worth having.

Treat incident-generated drafts as a starting point

Already have a failure group? Its Generate eval draft action can save you some typing. You'll find the draft under Evals → Suggestions.

Read it before using it. A draft can copy input from the failed trace, pick out response keys, or turn the error message into an expected phrase. Sometimes that's helpful. Sometimes it means the draft is asking the agent to repeat the very error you meant to fix.

Choose a suite attached to the same workflow target as the incident, and check the input and expected behavior before promotion. The suggestion form lets you change the name, description, rationale, and destination suite. Reviewing and changing its stored payload or assertions uses the operator API. If you'd rather stay in the browser, the manual Add case route above is simpler.

Approve records your decision about the draft. Promote to suite creates the case. Neither runs it or deploys a change. Keep scheduled runs off while you're still preparing the suite.

Make it fail before you trust a pass

Now try the test against the old behavior. On the suite page, choose Standard under Run kind, select the old version in Baseline version, and click Queue eval run.

The answer “You're on Pro” should fail our text check. If it passes, stop and look at what actually ran. Did the endpoint honor the version selection? Did your fixture cause the lookup to fail? Is the assertion checking the part of the response you intended?

A connection error doesn't count as catching the bad answer. You want the test to fail because it saw the wrong behavior.

Then run it again, selecting the fixed version in the same selector. Leave the input and expectations alone. Open the case result and compare the responses. Standard runs use your saved case inputs; shadow runs use sampled trace inputs, so start with Standard for this controlled comparison.

If the agent's answers vary, repeat the comparison. Also try a normal successful lookup and an empty result. You don't want to “fix” the timeout by making the agent refuse every request.

Review the evidence before deciding to release

A completed run means the work finished, not that every case passed. Open the failures and look for execution errors or skipped cases before relying on the overall score.

If a result appears in Evals → Reviews, read the evidence before choosing Approve or Mark failed. Approve keeps the recorded score and pass value; it doesn't turn a failure into a success. Mark failed records a revised failing result.

There's one important boundary here: reviewing a result doesn't recalculate the stored release gate, and the separate promotion endpoint doesn't enforce eval results. Your release process needs to check the outcomes you care about. The review guide covers the details.

For this incident, the question is simple: did the old version make the mistake, did the new one handle it, and have you checked the rest of the answer?

Leave behind a test someone else can maintain

A teammate opening this case next month shouldn't have to reconstruct the incident. Give it a clear name, a short explanation, and a link to the evidence. Note what it checks, what it misses, and who should investigate if it starts failing again.

Run it when you change the prompt, model, or relevant integration. If the expected behavior changes on purpose, update the case after reviewing that decision. Don't quietly lower the threshold just to get a green result.

You don't need a huge test suite to start. One well-understood failure is enough to build a useful first case. The next time someone changes the agent, you'll have more to go on than “it seemed fine when I tried it.”

For the wider workflow, see AI agent evaluation in Robots Center.