Operator - Evals

Overview

These eval endpoints turn runtime failures into repeatable quality checks. They read and update existing suites, retry or cancel runs, resolve human-review tasks, and promote failure-group suggestions into real eval cases.

Four connected queues

Suites define cases and release settings; runs produce results; review tasks hold ambiguous outcomes; incident suggestions turn observed failures into proposed cases. IDs link those queues without moving data across workspaces.

Retries create new evidence

Retry creates a fresh queued run from any source status rather than rewriting history. Operationally, prefer retrying failed or cancelled work after diagnosing the cause; this is guidance, not a failed-only runtime restriction. Cancelling stops queued or running work and keeps any results already written; a 400 cannot_cancel response means the run had already finished.

Creation is split across surfaces

Operator JSON does not POST suites or runs. Create targets and versions at /app/evals/targets, suites at /app/evals/suites, cases on a suite page, and runs at /app/evals/runs. Scoped machine clients may POST /api/v1/eval_suites and /api/v1/eval_runs.

Clone is a new suite shell

Cloning copies suite configuration but not eval cases. It disables schedule and canary, clears last_scheduled_at, and preserves the source interval and regression threshold for later re-enablement. Add cases deliberately before running the clone.

How it works

1. Start with suite health

   Create the target, suite, cases, and first run through those supported paths. Then use operator JSON to update or clone the suite and filter existing runs by suite, target, status, or kind.

2. Close the learning loop

   Review failed runs and pending review tasks. Generate or edit an incident suggestion from a failure group, choose its destination suite, promote it, and confirm the resulting case appears before relying on the regression gate.

Examples

These examples are illustrative. Replace the sample values with your own. Keep credentials in environment variables or a secret manager, never in client-side code.

const response = await fetch("/api/v1/operator/eval_suites?filter%5Bstatus%5D=active", {
  credentials: "same-origin",
  headers: {"Accept": "application/json"}
});
const body = await response.json();
if (!response.ok) throw new Error(`${response.status}: ${body.detail}`);
console.log(body);

Troubleshooting

An eval resource is 404

The identifier may exist in another workspace, but scoped queries intentionally return 404. Confirm the selected workspace and follow IDs returned by its own list endpoints.

A run will not cancel

Queued and running runs are cancellable; completed, failed, and cancelled runs are not. Re-read the run: if it is already terminal, preserve that evidence and decide whether a later retry is appropriate.

Endpoint reference

Ready to integrate? Open an endpoint for its required permissions, request fields, response format, and status codes. The examples above explain the workflow; these details define each individual request.

{
  "eval_runs": [
    {
      "average_cost": null,
      "average_latency_ms": null,
      "candidate_workflow_version": null,
      "candidate_workflow_version_id": null,
      "correlation": {
        "contract_version": "correlation/v1",
        "eval_run_id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
        "provider": "provider-runtime",
        "provider_run_id": "provider-run-example"
      },
      "eval_suite": null,
      "eval_suite_id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
      "failed_cases": 2,
      "finished_at": "2026-03-20T14:05:00Z",
      "id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
      "inserted_at": "2026-03-20T14:00:00Z",
      "pass_rate": "0.85",
      "passed_cases": 8,
      "results": [],
      "review_counts": {
        "approved": 0,
        "pending": 0,
        "revised": 0,
        "total": 0,
        "unreviewed": 0
      },
      "reviewed_pass_rate": null,
      "run_kind": "standard",
      "segments": [],
      "shadow_summary": null,
      "source_failure_group": null,
      "source_failure_group_id": null,
      "started_at": "2026-03-20T14:00:00Z",
      "status": "completed",
      "summary": null,
      "total_cases": 10,
      "triggered_by_id": null,
      "updated_at": "2026-03-20T14:05:00Z",
      "workflow_target": null,
      "workflow_target_id": null,
      "workflow_version": null,
      "workflow_version_id": null
    }
  ],
  "pagination": {
    "limit": 50,
    "offset": 0
  }
}