Eval Suites

Overview

An evaluation suite groups the test cases and scoring rules used to judge a workflow. Define a suite once, then run it repeatedly as your prompts, models, or workflow versions change. This machine API manages the suite's configuration; add its test cases and workflow versions in the operator console before starting a run.

Suite policy is separate from cases

The suite stores its target association, rubric type, schedule, and canary controls. Create does not accept cases, and clone copies suite fields only. A clone starts without the source cases, disables its schedule and canary, and clears last_scheduled_at, while preserving the source interval and maximum-regression values for later re-enablement.

The request has an `eval_suite` wrapper

Create and patch attributes belong under eval_suite. The response is the suite object itself; list is the one response wrapped under the eval_suites key.

Deletion is destructive

DELETE returns 204 and cascades to the suite's cases, eval runs, and related review history. Prefer a deliberate operator workflow when historical evaluation evidence must be preserved.

How it works

1. Select the target and rubric

   Choose a workspace workflow-target ID and one supported rubric: composite, exact match, status code, or LLM judge. Set an explicit pass threshold if the default 80 percent is unsuitable.

2. Create the suite, then its cases

   POST the wrapped suite fields, retain its ID, and use the operator console to add enabled cases and workflow versions. If omitted, the schedule interval defaults to 1440 minutes and maximum canary regression to 5 percent. Set them explicitly to suit your workflow.

3. Read before update, clone, or delete

   GET the suite to inspect its target and case summaries. Remember that cloning is a configuration starting point, not a full test corpus copy.

Examples

These examples are illustrative. Replace the sample values with your own. Keep credentials in environment variables or a secret manager, never in client-side code.

export AGENTOPS_BASE_URL="https://robotscenter.net"
export AGENT_TOKEN="machine_token_with_eval_suite_write"
export WORKFLOW_TARGET_ID="f92da274-a794-49ee-bd4d-b13022a6041a"

jq -n --arg target "$WORKFLOW_TARGET_ID" \
  '{eval_suite: {name: "Support answer quality", slug: "support-answer-quality", workflow_target_id: $target, description: "Hourly regression checks", rubric_type: "composite", pass_threshold: 90.0, schedule_enabled: true, schedule_interval_minutes: 60}}' |
  curl --fail-with-body --silent --show-error \
    -X POST -H 'content-type: application/json' \
    -H "Authorization: Bearer $AGENT_TOKEN" \
    --data-binary @- "$AGENTOPS_BASE_URL/api/v1/eval_suites" | jq .

Troubleshooting

Create returns 400 or 422

A missing eval_suite wrapper returns 400 before field validation. With the wrapper present, an invalid name, target, rubric, nonpositive schedule interval, or negative canary regression returns 422. Omitted interval/regression fields receive defaults. Add test cases through the operator console, not a cases array in this request.

The clone has no test cases

That is the shipped contract: clone copies suite fields but not cases. It disables schedule_enabled and canary_enabled and clears last_scheduled_at; it retains schedule_interval_minutes and canary_max_regression. Add the new corpus explicitly in the operator UI.

A suite ID returns 404

IDs are workspace-scoped. Verify the credential belongs to the suite's workspace and that the suite was not deleted by another operator.

Endpoint reference

Ready to integrate? Open an endpoint for its required permissions, request fields, response format, and status codes. The examples above explain the workflow; these details define each individual request.

{
  "eval_suites": [
    {
      "archived_at": null,
      "canary_enabled": false,
      "canary_max_regression": null,
      "description": "Evals for payment processing correctness",
      "eval_cases": [],
      "id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
      "inserted_at": "2026-02-10T08:00:00Z",
      "last_scheduled_at": null,
      "name": "Payment accuracy suite",
      "rubric_type": "llm_judge",
      "schedule_enabled": false,
      "schedule_interval_minutes": null,
      "status": "active",
      "updated_at": "2026-03-20T14:00:00Z",
      "workflow_target": null
    }
  ]
}

{
  "archived_at": null,
  "canary_enabled": false,
  "canary_max_regression": null,
  "description": "Evals for payment processing correctness",
  "eval_cases": [],
  "id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
  "inserted_at": "2026-02-10T08:00:00Z",
  "last_scheduled_at": null,
  "name": "Payment accuracy suite",
  "rubric_type": "llm_judge",
  "schedule_enabled": false,
  "schedule_interval_minutes": null,
  "status": "active",
  "updated_at": "2026-03-20T14:00:00Z",
  "workflow_target": null
}

{
  "archived_at": null,
  "canary_enabled": false,
  "canary_max_regression": null,
  "description": "Evals for payment processing correctness",
  "eval_cases": [],
  "id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
  "inserted_at": "2026-02-10T08:00:00Z",
  "last_scheduled_at": null,
  "name": "Payment accuracy suite",
  "rubric_type": "llm_judge",
  "schedule_enabled": false,
  "schedule_interval_minutes": null,
  "status": "active",
  "updated_at": "2026-03-20T14:00:00Z",
  "workflow_target": null
}

{
  "archived_at": null,
  "canary_enabled": false,
  "canary_max_regression": null,
  "description": "Evals for payment processing correctness",
  "eval_cases": [],
  "id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
  "inserted_at": "2026-02-10T08:00:00Z",
  "last_scheduled_at": null,
  "name": "Payment accuracy suite",
  "rubric_type": "llm_judge",
  "schedule_enabled": false,
  "schedule_interval_minutes": null,
  "status": "active",
  "updated_at": "2026-03-20T14:00:00Z",
  "workflow_target": null
}

{
  "archived_at": null,
  "canary_enabled": false,
  "canary_max_regression": null,
  "description": "Evals for payment processing correctness",
  "eval_cases": [],
  "id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
  "inserted_at": "2026-02-10T08:00:00Z",
  "last_scheduled_at": null,
  "name": "Payment accuracy suite",
  "rubric_type": "llm_judge",
  "schedule_enabled": false,
  "schedule_interval_minutes": null,
  "status": "active",
  "updated_at": "2026-03-20T14:00:00Z",
  "workflow_target": null
}