Eval Runs

Overview

An evaluation run checks how your workflow performs against a set of test cases. Use it to assess a change before relying on it in production, or to investigate a regression. Creating a run queues the enabled cases from an existing suite; read the run until it reaches completed, failed, or cancelled, then inspect its summary and individual results.

Runs are queued work

POST creates a run in queued state; workers advance it through running to a terminal state. The API does not promise an immediate result or an automatic client retry, so persist the returned run ID and poll deliberately.

Standard and shadow modes

A standard run evaluates suite cases with their stored inputs. A shadow run samples production traces, optionally filters and segments them using only the documented allowlist, and can compare a candidate workflow version; sample size defaults to 12 and is clamped to 1 through 25.

Result and review views coexist

Terminal reads include aggregate pass rate and counts, per-case results, trace IDs, latency and cost, plus review counts and reviewed pass rate. Automated passed and score remain distinct from any final human-reviewed values.

How it works

1. Choose a ready suite

   List or read eval suites and verify the intended target and enabled cases. Creating a suite through the machine API does not create its cases.

2. Queue the run

   POST its eval_suite_id, adding a workflow version or shadow parameters only when needed. Store the returned id; a monthly eval-run quota can reject creation with 429.

3. Read through a terminal state

   GET the run at a controlled interval. Stop on completed, failed, or cancelled; on completion, inspect every result as well as aggregate fields because a completed run can contain failed or error cases.

Examples

These examples are illustrative. Replace the sample values with your own. Keep credentials in environment variables or a secret manager, never in client-side code.

export AGENTOPS_BASE_URL="https://robotscenter.net"
export AGENT_TOKEN="machine_token_with_eval_suite_write"
export WORKFLOW_TARGET_ID="f92da274-a794-49ee-bd4d-b13022a6041a"

SUITE_RESPONSE=$(jq -n --arg target "$WORKFLOW_TARGET_ID" \
  '{eval_suite: {name: "Checkout contract", workflow_target_id: $target, rubric_type: "status_code", pass_threshold: 100.0}}' |
  curl --fail-with-body --silent --show-error \
    -X POST -H 'content-type: application/json' \
    -H "Authorization: Bearer $AGENT_TOKEN" \
    --data-binary @- "$AGENTOPS_BASE_URL/api/v1/eval_suites")

export EVAL_SUITE_ID=$(printf '%s' "$SUITE_RESPONSE" | jq -r '.id')
printf '%s' "$SUITE_RESPONSE" | jq .

export AGENTOPS_BASE_URL="https://robotscenter.net"
export AGENT_TOKEN="machine_token_with_eval_run_write"
export EVAL_SUITE_ID="9e1f1538-ff37-48f0-a2f9-65066d7d6699"

RUN_RESPONSE=$(jq -n --arg suite "$EVAL_SUITE_ID" \
  '{eval_suite_id: $suite, run_kind: "standard"}' |
  curl --fail-with-body --silent --show-error \
    -X POST -H 'content-type: application/json' \
    -H "Authorization: Bearer $AGENT_TOKEN" \
    --data-binary @- "$AGENTOPS_BASE_URL/api/v1/eval_runs")

export EVAL_RUN_ID=$(printf '%s' "$RUN_RESPONSE" | jq -r '.id')
printf '%s' "$RUN_RESPONSE" | jq '{id, status, run_kind, eval_suite_id}'

export AGENTOPS_BASE_URL="https://robotscenter.net"
export AGENT_TOKEN="machine_token_with_eval_run_read"
export EVAL_RUN_ID="d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8"

curl --fail-with-body --silent --show-error \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  "$AGENTOPS_BASE_URL/api/v1/eval_runs/$EVAL_RUN_ID" |
  jq '{id, status, pass_rate, total_cases, passed_cases, failed_cases, results}'

Troubleshooting

The run completes with zero cases

The suite had no enabled cases. Machine suite creation does not accept a cases payload; create and enable cases in the operator eval-suite UI before scheduling.

Creation returns 429

The workspace exhausted eval_runs_per_month. The quota problem includes upgrade context; changing the run payload does not bypass the quota.

A run stays queued or becomes failed

Check /ready for queue health, then inspect run results and related traces. Verify the target is reachable, its secret is valid, and its timeout covers the workflow response.

Endpoint reference

Ready to integrate? Open an endpoint for its required permissions, request fields, response format, and status codes. The examples above explain the workflow; these details define each individual request.

{
  "average_cost": null,
  "average_latency_ms": null,
  "candidate_workflow_version": null,
  "candidate_workflow_version_id": null,
  "correlation": {
    "contract_version": "correlation/v1",
    "eval_run_id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
    "provider": "provider-runtime",
    "provider_run_id": "provider-run-example"
  },
  "eval_suite": null,
  "eval_suite_id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
  "failed_cases": 2,
  "finished_at": "2026-03-20T14:05:00Z",
  "id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
  "inserted_at": "2026-03-20T14:00:00Z",
  "pass_rate": "0.85",
  "passed_cases": 8,
  "results": [],
  "review_counts": {
    "approved": 0,
    "pending": 0,
    "revised": 0,
    "total": 0,
    "unreviewed": 0
  },
  "reviewed_pass_rate": null,
  "run_kind": "standard",
  "segments": [],
  "shadow_summary": null,
  "source_failure_group": null,
  "source_failure_group_id": null,
  "started_at": "2026-03-20T14:00:00Z",
  "status": "completed",
  "summary": null,
  "total_cases": 10,
  "triggered_by_id": null,
  "updated_at": "2026-03-20T14:05:00Z",
  "workflow_target": null,
  "workflow_target_id": null,
  "workflow_version": null,
  "workflow_version_id": null
}

{
  "average_cost": null,
  "average_latency_ms": null,
  "candidate_workflow_version": null,
  "candidate_workflow_version_id": null,
  "correlation": {
    "contract_version": "correlation/v1",
    "eval_run_id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
    "provider": "provider-runtime",
    "provider_run_id": "provider-run-example"
  },
  "eval_suite": null,
  "eval_suite_id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
  "failed_cases": 2,
  "finished_at": "2026-03-20T14:05:00Z",
  "id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
  "inserted_at": "2026-03-20T14:00:00Z",
  "pass_rate": "0.85",
  "passed_cases": 8,
  "results": [],
  "review_counts": {
    "approved": 0,
    "pending": 0,
    "revised": 0,
    "total": 0,
    "unreviewed": 0
  },
  "reviewed_pass_rate": null,
  "run_kind": "standard",
  "segments": [],
  "shadow_summary": null,
  "source_failure_group": null,
  "source_failure_group_id": null,
  "started_at": "2026-03-20T14:00:00Z",
  "status": "completed",
  "summary": null,
  "total_cases": 10,
  "triggered_by_id": null,
  "updated_at": "2026-03-20T14:05:00Z",
  "workflow_target": null,
  "workflow_target_id": null,
  "workflow_version": null,
  "workflow_version_id": null
}