Platform
Eval review queue
Annotate eval evidence at /app/evals/reviews by approving the recorded result or revising it to failed. A resolution updates the review task and case result for triage and reporting; it does not recompute the run's release gate or promote a workflow version.
On this page
Start here
The queue records human judgment on a flagged case. Approve preserves its score and pass value; Mark failed sets final_passed false. Resolution is triage, not automatic enforcement, release-gate recalculation, or promotion.
Before you begin
- A signed-in workspace user who can inspect review evidence
- A pending review, its eval evidence, and a team decision standard
Put it into practice
-
Open the pending review
Open /app/evals/reviews, select an item, and confirm its eval run before deciding.
-
Inspect the evidence
Read the eval output, failure context, and related trace or notes. Resolve uncertainty before acting rather than guessing from the queue summary.
-
Record one decision
Approve when the result stands, or mark failed when it does not. Handle release or promotion separately through a process that reads reviewed outcomes.
The task and case result carry the reviewed status, the item leaves the queue, and the annotation is not mistaken for a recomputed release decision.
01 Actions
detailsApprove
Use this when the original eval result should stand. The action sets the task and case result review status to approved and preserves the case's existing score and passed value; it does not recalculate the eval run release gate.
Mark failed
Use this when the reviewed outcome should be failed. The action sets status to revised and final_passed to false. It updates the review records only; it does not block or promote a workflow version by itself.
Troubleshooting
- Approve is not appropriate
- Use Mark failed. That action records status revised and final_passed false; it is not merely a visual rejection label.
- Release status did not change
- Expected: resolution does not recompute summary.release_gate, and promotion does not check reviews. Make release processing consume reviewed outcomes explicitly.
- Auditor cannot resolve reviews
- Review decisions require an active owner, admin, or member, in the browser review queue and the operator API alike. Auditors are read-only in both and get read_only_role. Use the API for automated review integrations.
- The decision was refused as already resolved
- Two reviewers decided the same task. Only the first decision is kept; the browser queue reloads with the current state and the operator API returns 409 (Eval review task has already been resolved). When marking a result failed, an optional final_score must be between 0 and 100 or the request is rejected with 422.