Log in Create workspace
Skip to content
Browse docs

API reference

Operator - Observability

Search traces, events, alerts, and grouped failures as one operator workflow.

Jump to 14 endpoints
On this page

Overview

Operator observability joins runtime telemetry with trace detail, trace events, search correlations, alert ownership, and failure groups. It helps an on-call operator move from a symptom to the affected workflow, agent, and evidence.

Search finds; correlations explain
Search returns indexed records across supported resource types. Correlations starts from one resource_type and resource_id and returns linked evidence, which is more reliable than guessing shared text.
Alerts and failure groups have operator state
Alert operator_status is distinct from delivery status and supports acknowledge, snooze, and resolve. Failure groups aggregate similar trace failures and can carry assignee, workflow, severity, or resolution fields for durable triage.

Before you begin

  • A signed-in browser session with a selected workspace backed by an active membership is required; an archived workspace remains readable but rejects mutations. Every /api/v1/operator endpoint rejects API credentials and bearer tokens, regardless of their scopes.
  • Any role on an active membership can read workspace-scoped observability data except GET /api/v1/operator/telemetry: it reports node-wide platform metrics and requires users.admin=true. Owners, admins, and members can acknowledge, snooze, resolve, or update triage state; auditors remain read-only.
  • Start with a trace ID, request ID, workflow target, provider correlation, alert, or meaningful search term.

How it works

  1. Move from broad to exact

    Do not use /operator/telemetry as a workspace shape summary: it is node-wide and platform-admin-only. Start with workspace-scoped search using a bounded query and resource filter. Open the trace and ordered events, or use correlations to pivot to its alert, failure group, eval, or provider identifiers.

  2. Record triage state

    Acknowledge ownership, snooze only with a justified duration, and resolve after verifying the signal has stopped. Generate an eval draft from recurring failure groups rather than losing the incident after closure.

Examples

These examples are illustrative. Replace the sample values with your own. Keep credentials in environment variables or a secret manager, never in client-side code.

Search trace records

This encoded query searches the current workspace and limits the first page to ten trace-like results.

javascript 
const response = await fetch("/api/v1/operator/observability/search?q=timeout&resource_type=trace&limit=10", {
  credentials: "same-origin",
  headers: {"Accept": "application/json"}
});
const body = await response.json();
if (!response.ok) throw new Error(`${response.status}: ${body.detail}`);
console.log(body);
What to expect

200 OK with {"results": []} when nothing matches. Populated results are grouped objects with resource_type and items fields; each item has resource_id for /observability/correlations.

Troubleshooting

A UUID filter returns 400
Copy canonical UUIDs from a workspace-scoped response and follow each endpoint's parameter form: trace and alert lists document nested filter[name] values, while observability search uses top-level resource_type or resource_types.
A result disappears after switching workspaces
That is expected isolation. Traces, alerts, and correlations are always scoped to the selected workspace, and a foreign resource seed returns 404.

Endpoint reference

Ready to integrate? Open an endpoint for its required permissions, request fields, response format, and status codes. The examples above explain the workflow; these details define each individual request.

GET
/api/v1/operator/telemetry Web Session

Get telemetry overview

Description

Returns operational telemetry summary counters and gauges for API, auth, rate limiting, and database latency. The counters cover the whole node, across every workspace, so only platform administrators may read them.

Auth

Web Session
Responses
200

Telemetry overview payload

Returns: TelemetryOverview
403

The signed-in user is not a platform administrator.
GET
/api/v1/operator/traces Web Session

List traces (operator)

Description

Returns traces for the current workspace. list_traces/2 silently limits the page to 30 rows; this endpoint does not accept limit or offset. FilterParams reads the nested filter map only.

Auth

Web Session
Parameters
Name In Type Required Description
filter[status] query string No Filter by trace status
filter[trace_type] query string No Filter by trace type
filter[environment] query string No Exact-match environment filter
filter[workflow_target_id] query string No Workflow target UUID
filter[service_agent_id] query string No Service agent UUID
filter[request_id] query string No Exact-match request ID filter
filter[external_trace_id] query string No Exact-match external trace ID filter
filter[replayable] query string No Filter by whether the trace can be replayed
filter[binding] query string No bound traces have a workflow_target_id; unbound traces do not. Used by Trace Explorer.
Responses
200

List of traces

Returns: TraceList
Response Body
Field Type Required Description
traces array<object> Yes List of trace summaries
traces[].id string Yes Trace UUID
traces[].name string Yes Human-readable trace name
traces[].status string (running, ok, error, partial) Yes Trace execution status
traces[].started_at datetime (ISO 8601) Yes Trace start time
traces[].request_id string No Internal request identifier
traces[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
traces[].updated_at datetime (ISO 8601) Yes Record update timestamp
traces[].environment string No Exact environment label captured at ingest
traces[].service_agent object No Associated service agent summary
traces[].service_agent.id string Yes Service agent UUID
traces[].service_agent.name string Yes Service agent name
traces[].workflow_target object No Associated workflow target summary
traces[].workflow_target.id string Yes Workflow target UUID
traces[].workflow_target.name string Yes Workflow target name
traces[].duration_ms integer No Trace duration in milliseconds
traces[].ended_at datetime (ISO 8601) No Trace end time
traces[].external_trace_id string No External trace identifier from the caller
traces[].last_activity_at datetime (ISO 8601) No Last observed activity time
traces[].total_cost string No Accumulated cost across the trace, as a decimal string
traces[].trace_type string (runtime, eval, replay) Yes Trace category
traces[].workflow_version object No Associated workflow version summary
traces[].workflow_version.id string Yes Workflow version UUID
traces[].workflow_version.label string Yes Workflow version label
Example
{
  "traces": [
    {
      "duration_ms": 3200,
      "ended_at": "2026-03-12T08:45:03Z",
      "environment": "production",
      "external_trace_id": "trace-checkout-443",
      "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
      "inserted_at": "2026-03-12T08:45:03Z",
      "last_activity_at": "2026-03-12T08:45:03Z",
      "name": "checkout runtime trace",
      "request_id": "req-01HQ5PX7P7F4",
      "service_agent": {
        "id": "f4d49601-3477-4c9d-b896-ac08a26a3147",
        "name": "payments-agent"
      },
      "started_at": "2026-03-12T08:45:00Z",
      "status": "ok",
      "total_cost": "0.1200",
      "trace_type": "runtime",
      "updated_at": "2026-03-12T08:45:03Z",
      "workflow_target": {
        "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
        "name": "checkout-api"
      },
      "workflow_version": {
        "id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
        "label": "v2026.03.12"
      }
    }
  ]
}
GET
/api/v1/operator/traces/:id Web Session

Get trace details (operator)

Description

Returns detailed trace information.

Auth

Web Session
Parameters
Name In Type Required Description
id path string Yes Trace UUID
Responses
200

Trace details

Returns: Trace
Response Body
Field Type Required Description
id string Yes Trace UUID
name string Yes Human-readable trace name
status string (running, ok, error, partial) Yes Trace execution status
events array<object> Yes Ordered event summaries for the trace timeline
events[].id string Yes Trace event UUID
events[].name string Yes Human-readable event name
events[].status string (ok, error, pending) Yes Event status
events[].origin string (partner, runtime) No Credential that wrote a marketplace engagement event: partner (the marketplace's mpk_ key) or runtime (an engagement token, whose source_event_id is stored under runtime:). Null for first-party events.
events[].started_at datetime (ISO 8601) No Event start time
events[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
events[].sequence integer Yes Sequence number within the trace
events[].event_type string Yes Event type label
events[].cost string No Cost attributed to the event, as a decimal string
events[].duration_ms integer No Duration in milliseconds
events[].ended_at datetime (ISO 8601) No Event end time
events[].latency_ms integer No Latency in milliseconds
events[].parent_event_id string No Parent event UUID when nested
events[].source_event_id string No Caller-supplied source event id when the ingest is idempotent
metadata object Yes Read-time redacted metadata payload
started_at datetime (ISO 8601) Yes Trace start time
request_id string No Internal request identifier
inserted_at datetime (ISO 8601) Yes Record insertion timestamp
updated_at datetime (ISO 8601) Yes Record update timestamp
summary string No Operator-facing summary for the trace
environment string No Exact environment label captured at ingest
service_agent object No Associated service agent summary
service_agent.id string Yes Service agent UUID
service_agent.name string Yes Service agent name
error_payload object Yes Read-time redacted error payload
failure_group object No Failure group linked by matching signature
failure_group.id string Yes Failure group UUID
failure_group.status string (open, acknowledged, resolved) Yes Failure group status
failure_group.title string Yes Failure group title
failure_group.severity string (low, medium, high, critical) Yes Failure group severity
workflow_target object No Associated workflow target summary
workflow_target.id string Yes Workflow target UUID
workflow_target.name string Yes Workflow target name
events_next_cursor string No Set when the trace has more events than the detail embeds (the first 500): pass it as cursor to GET /api/v1/traces/:trace_id/events for the rest. Null when events holds every event.
failure_signature string No Failure signature used for grouping
input_payload object Yes Read-time redacted input payload
output_payload object Yes Read-time redacted output payload
redaction_applied boolean Yes True when any operator-visible payload field was masked
replayable boolean Yes Whether the trace can be replayed
duration_ms integer No Trace duration in milliseconds
ended_at datetime (ISO 8601) No Trace end time
external_trace_id string No External trace identifier from the caller
last_activity_at datetime (ISO 8601) No Last observed activity time
total_cost string No Accumulated cost across the trace, as a decimal string
trace_type string (runtime, eval, replay) Yes Trace category
workflow_version object No Associated workflow version summary
workflow_version.id string Yes Workflow version UUID
workflow_version.label string Yes Workflow version label
Example
{
  "duration_ms": 3200,
  "ended_at": "2026-03-12T08:45:03Z",
  "environment": "production",
  "error_payload": {
    "email": "[REDACTED]"
  },
  "events": [
    {
      "cost": "0.0300",
      "duration_ms": 950,
      "ended_at": "2026-03-12T08:45:02Z",
      "event_type": "llm.completion",
      "id": "c34b4b67-f304-4b02-a8e1-2a053c4b8381",
      "inserted_at": "2026-03-12T08:45:02Z",
      "latency_ms": 950,
      "name": "draft response",
      "origin": null,
      "parent_event_id": null,
      "sequence": 1,
      "source_event_id": "evt-checkout-443-1",
      "started_at": "2026-03-12T08:45:01Z",
      "status": "error"
    }
  ],
  "events_next_cursor": null,
  "external_trace_id": "trace-checkout-443",
  "failure_group": {
    "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
    "severity": "high",
    "status": "open",
    "title": "Payment timeout"
  },
  "failure_signature": "payment:timeout",
  "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
  "input_payload": {
    "authorization": "[REDACTED]"
  },
  "inserted_at": "2026-03-12T08:45:03Z",
  "last_activity_at": "2026-03-12T08:45:03Z",
  "metadata": {
    "region": "us-east-1"
  },
  "name": "checkout runtime trace",
  "output_payload": {
    "result": "retry_exhausted"
  },
  "redaction_applied": true,
  "replayable": true,
  "request_id": "req-01HQ5PX7P7F4",
  "service_agent": {
    "id": "f4d49601-3477-4c9d-b896-ac08a26a3147",
    "name": "payments-agent"
  },
  "started_at": "2026-03-12T08:45:00Z",
  "status": "ok",
  "summary": "Timeout after payment provider retry budget was exhausted.",
  "total_cost": "0.1200",
  "trace_type": "runtime",
  "updated_at": "2026-03-12T08:45:03Z",
  "workflow_target": {
    "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
    "name": "checkout-api"
  },
  "workflow_version": {
    "id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
    "label": "v2026.03.12"
  }
}
404

Trace not found
GET
/api/v1/operator/traces/:trace_id/events Web Session

List trace events

Description

Returns one page of a trace's events, ordered by sequence. Pass next_cursor back as cursor to read the next page; it is null on the last page. A well-formed trace id that is unknown, purged, or in another workspace returns 200 with an empty trace_events list, not 404.

Auth

Web Session
Parameters
Name In Type Required Description
trace_id path string Yes Trace UUID
filter[status] query string No Filter by event status
filter[event_type] query string No Exact-match event type filter
limit query integer No Maximum events to return (default 100, maximum 500)
cursor query string No Opaque next_cursor from the previous page
Responses
200

Page of events with next_cursor

Returns: TraceEventList
Response Body
Field Type Required Description
next_cursor string No Opaque cursor for the next page; null on the last page
trace_events array<object> Yes List of trace event summaries
trace_events[].id string Yes Trace event UUID
trace_events[].name string Yes Human-readable event name
trace_events[].status string (ok, error, pending) Yes Event status
trace_events[].origin string (partner, runtime) No Credential that wrote a marketplace engagement event: partner (the marketplace's mpk_ key) or runtime (an engagement token, whose source_event_id is stored under runtime:). Null for first-party events.
trace_events[].started_at datetime (ISO 8601) No Event start time
trace_events[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
trace_events[].sequence integer Yes Sequence number within the trace
trace_events[].event_type string Yes Event type label
trace_events[].cost string No Cost attributed to the event, as a decimal string
trace_events[].duration_ms integer No Duration in milliseconds
trace_events[].ended_at datetime (ISO 8601) No Event end time
trace_events[].latency_ms integer No Latency in milliseconds
trace_events[].parent_event_id string No Parent event UUID when nested
trace_events[].source_event_id string No Caller-supplied source event id when the ingest is idempotent
Example
{
  "next_cursor": null,
  "trace_events": [
    {
      "cost": "0.0300",
      "duration_ms": 950,
      "ended_at": "2026-03-12T08:45:02Z",
      "event_type": "llm.completion",
      "id": "c34b4b67-f304-4b02-a8e1-2a053c4b8381",
      "inserted_at": "2026-03-12T08:45:02Z",
      "latency_ms": 950,
      "name": "draft response",
      "origin": null,
      "parent_event_id": null,
      "sequence": 1,
      "source_event_id": "evt-checkout-443-1",
      "started_at": "2026-03-12T08:45:01Z",
      "status": "error"
    }
  ]
}
400

trace_id is not a UUID (Invalid UUID path parameter: trace_id), or the filter or cursor is invalid
GET
/api/v1/operator/observability/search Web Session

Search observability records (Session Auth Required)

Description

Runs a cross-surface search over indexed traces, events, alerts, replays, and failure groups. Default limit is 20, max 50. resource_types is an allowlist of trace | trace_event | failure_group | replay | alert_group. Note: This operator endpoint requires session authentication (browser login), not API bearer token.

Auth

Web Session
Parameters
Name In Type Required Description
q query string No Search query text
resource_types query array No Optional comma-separated resource types to include, e.g. resource_types=trace,replay. Allowed: trace, trace_event, failure_group, replay, alert_group. Repeating the key without [] keeps only its last value.
status query string No Status filter
severity query string No Severity filter
workflow_target_id query string No Workflow target UUID
service_agent_id query string No Service agent UUID
occurred_after query datetime (ISO 8601) No Lower bound for result timestamps
occurred_before query datetime (ISO 8601) No Upper bound for result timestamps
limit query integer No Maximum results to return. Default 20, max 50.
Responses
200

Search results

Returns: ObservabilitySearchResults
GET
/api/v1/operator/observability/correlations Web Session

Get related observability records

Description

Returns records related to a seed trace, alert, replay, or failure group based on shared correlation identifiers.

Auth

Web Session
Parameters
Name In Type Required Description
resource_type query string Yes Seed resource type. Allowed: trace, trace_event, failure_group, replay, alert_group.
resource_id query string Yes Seed resource UUID
Responses
200

Correlated records

Returns: ObservabilityCorrelations
404

Seed record not found
GET
/api/v1/operator/alerts Web Session

List alerts

Description

Returns one page of grouped operator alerts generated from anomaly rules and deliveries, newest first. Pass next_cursor back as cursor to read older alerts; it is null on the last page. Anomaly thresholds are evaluated against trace completion time (`ended_at`, falling back to `inserted_at`). A group's operator_status is that of its newest delivery, and an expired snooze reads as open. delivery_count and destinations cover the whole group; deliveries lists only its 10 newest.

Auth

Web Session
Parameters
Name In Type Required Description
limit query integer No Maximum grouped alerts to return (default 100, maximum 500)
cursor query string No Opaque next_cursor from the previous page
filter[operator_status] query string No Filter by operator lifecycle status
filter[severity] query string No Filter by alert severity
filter[event_type] query string No Exact-match event type filter
filter[workflow_target_id] query string No Workflow target UUID linked through the failure group
Responses
200

Page of grouped alerts with next_cursor

Returns: AlertList
Response Body
Field Type Required Description
alerts array<object> Yes Grouped alerts with delivery context
alerts[].id string Yes Representative delivery UUID for the grouped alert
alerts[].message string Yes Operator-facing alert message
alerts[].trace object No Trace linked to the alert
alerts[].trace.id string Yes Trace UUID
alerts[].trace.name string Yes Trace name
alerts[].trace.status string (running, ok, error, partial) Yes Trace status
alerts[].trace.request_id string No Internal request identifier
alerts[].title string Yes Operator-facing alert title
alerts[].severity string (info, warning, error, critical) Yes Alert severity
alerts[].inserted_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery
alerts[].updated_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery update
alerts[].summary string No Optional short summary
alerts[].operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
alerts[].acknowledged_at datetime (ISO 8601) No When the alert was acknowledged
alerts[].acknowledged_by object No User who acknowledged the alert
alerts[].acknowledged_by.id string Yes User UUID
alerts[].acknowledged_by.email string Yes User email
alerts[].alert_group_key string Yes Stable grouped alert key
alerts[].alert_rule object No Rule that emitted the alert
alerts[].alert_rule.id string Yes Alert rule UUID
alerts[].alert_rule.name string Yes Rule name
alerts[].alert_rule.severity string (info, warning, error, critical) Yes Delivery severity attached by the rule
alerts[].alert_rule.conditions object Yes Rule conditions map. For anomaly rules, `window_minutes` is evaluated against trace completion time (`ended_at`, falling back to `inserted_at`).
alerts[].alert_rule.event_type string No Event type matched by the rule
alerts[].deliveries array<object> Yes Per-destination delivery attempts for this grouped alert
alerts[].deliveries[].id string Yes Alert delivery UUID
alerts[].deliveries[].status string (pending, sent, failed, succeeded) Yes Transport delivery status
alerts[].deliveries[].destination object Yes Destination used for this delivery
alerts[].deliveries[].destination.id string Yes Alert destination UUID
alerts[].deliveries[].destination.name string Yes Destination display name
alerts[].deliveries[].destination.type string (email, slack, webhook, pagerduty) Yes Destination type
alerts[].deliveries[].error_message string No Latest transport error message
alerts[].deliveries[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
alerts[].deliveries[].updated_at datetime (ISO 8601) Yes Record update timestamp
alerts[].deliveries[].attempt_count integer Yes Number of transport attempts
alerts[].deliveries[].failed_at datetime (ISO 8601) No Timestamp of latest failed delivery
alerts[].deliveries[].last_attempt_at datetime (ISO 8601) No Timestamp of the latest delivery attempt
alerts[].deliveries[].operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
alerts[].deliveries[].succeeded_at datetime (ISO 8601) No Timestamp of successful delivery
alerts[].delivery_count integer Yes Number of deliveries in the group
alerts[].destinations array<object> Yes Unique destinations attached to the grouped alert
alerts[].destinations[].id string Yes Alert destination UUID
alerts[].destinations[].name string Yes Destination display name
alerts[].destinations[].type string (email, slack, webhook, pagerduty) Yes Destination type
alerts[].event_type string Yes Alert event type
alerts[].failure_group object No Failure group linked to the alert
alerts[].failure_group.id string Yes Failure group UUID
alerts[].failure_group.status string (open, acknowledged, resolved) Yes Failure group status
alerts[].failure_group.title string Yes Failure group title
alerts[].failure_group.severity string (low, medium, high, critical) Yes Failure group severity
alerts[].failure_group.occurrences integer Yes Total grouped occurrences
alerts[].failure_group.workflow_target_id string No Workflow target UUID
alerts[].latest_delivery_status string (pending, sent, failed, succeeded) Yes Latest transport status across grouped deliveries
alerts[].resolved_at datetime (ISO 8601) No When the alert was resolved
alerts[].resolved_by object No User who resolved the alert
alerts[].resolved_by.id string Yes User UUID
alerts[].resolved_by.email string Yes User email
alerts[].resource_id string No Primary resource identifier tied to the alert
alerts[].snoozed_until datetime (ISO 8601) No When the alert snooze expires
alerts[].workflow_target object No Workflow target connected through the failure group
alerts[].workflow_target.id string Yes Workflow target UUID
alerts[].workflow_target.name string Yes Workflow target name
next_cursor string Yes Cursor for the next page, null on the last page
Example
{
  "alerts": [
    {
      "acknowledged_at": "2026-03-12T09:08:00Z",
      "acknowledged_by": {
        "email": "sre@app.company",
        "id": "0f1cdac9-305d-4d11-a53d-3fca310c0635"
      },
      "alert_group_key": "36c7474d4600b9d0e1d29fe0c60b94f331762ee7c2f4f2fe875d97acbdaf0ef9",
      "alert_rule": {
        "conditions": {
          "detector": "failure_group_occurrence_spike",
          "min_failure_severity": "high",
          "min_occurrences": 3,
          "window_minutes": 60
        },
        "event_type": "failure_group.anomaly_detected",
        "id": "5abffead-a4f6-479d-b2e1-d4176fa8d248",
        "name": "Checkout anomaly",
        "severity": "error"
      },
      "deliveries": [
        {
          "attempt_count": 1,
          "destination": {
            "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
            "name": "Primary Slack",
            "type": "slack"
          },
          "error_message": null,
          "failed_at": null,
          "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
          "inserted_at": "2026-03-12T09:05:02Z",
          "last_attempt_at": "2026-03-12T09:05:03Z",
          "operator_status": "acknowledged",
          "status": "succeeded",
          "succeeded_at": "2026-03-12T09:05:03Z",
          "updated_at": "2026-03-12T09:05:03Z"
        }
      ],
      "delivery_count": 2,
      "destinations": [
        {
          "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
          "name": "Primary Slack",
          "type": "slack"
        }
      ],
      "event_type": "failure_group.anomaly_detected",
      "failure_group": {
        "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
        "occurrences": 7,
        "severity": "high",
        "status": "open",
        "title": "Payment timeout",
        "workflow_target_id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea"
      },
      "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
      "inserted_at": "2026-03-12T09:05:02Z",
      "latest_delivery_status": "succeeded",
      "message": "5 matching failures detected in the last 60 minutes for Payment timeout.",
      "operator_status": "acknowledged",
      "resolved_at": null,
      "resolved_by": null,
      "resource_id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
      "severity": "error",
      "snoozed_until": null,
      "summary": "Timeout after payment provider retry budget was exhausted.",
      "title": "Payment timeout anomaly detected",
      "trace": {
        "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
        "name": "checkout runtime trace",
        "request_id": "req-01HQ5PX7P7F4",
        "status": "error"
      },
      "updated_at": "2026-03-12T09:05:03Z",
      "workflow_target": {
        "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
        "name": "checkout-api"
      }
    }
  ],
  "next_cursor": null
}
400

Invalid alert filter or cursor
GET
/api/v1/operator/alerts/:id Web Session

Get alert details

Description

Returns grouped alert details and per-destination delivery history for any representative delivery in the alert group. deliveries lists the group's 100 newest; delivery_count covers all of them.

Auth

Web Session
Parameters
Name In Type Required Description
id path string Yes Representative alert UUID
Responses
200

Alert details

Returns: Alert
Response Body
Field Type Required Description
id string Yes Representative delivery UUID for the grouped alert
message string Yes Operator-facing alert message
trace object No Trace linked to the alert
trace.id string Yes Trace UUID
trace.name string Yes Trace name
trace.status string (running, ok, error, partial) Yes Trace status
trace.request_id string No Internal request identifier
title string Yes Operator-facing alert title
severity string (info, warning, error, critical) Yes Alert severity
inserted_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery
updated_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery update
summary string No Optional short summary
operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
acknowledged_at datetime (ISO 8601) No When the alert was acknowledged
acknowledged_by object No User who acknowledged the alert
acknowledged_by.id string Yes User UUID
acknowledged_by.email string Yes User email
alert_group_key string Yes Stable grouped alert key
alert_rule object No Rule that emitted the alert
alert_rule.id string Yes Alert rule UUID
alert_rule.name string Yes Rule name
alert_rule.severity string (info, warning, error, critical) Yes Delivery severity attached by the rule
alert_rule.conditions object Yes Rule conditions map. For anomaly rules, `window_minutes` is evaluated against trace completion time (`ended_at`, falling back to `inserted_at`).
alert_rule.event_type string No Event type matched by the rule
deliveries array<object> Yes Per-destination delivery attempts for this grouped alert
deliveries[].id string Yes Alert delivery UUID
deliveries[].status string (pending, sent, failed, succeeded) Yes Transport delivery status
deliveries[].destination object Yes Destination used for this delivery
deliveries[].destination.id string Yes Alert destination UUID
deliveries[].destination.name string Yes Destination display name
deliveries[].destination.type string (email, slack, webhook, pagerduty) Yes Destination type
deliveries[].error_message string No Latest transport error message
deliveries[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
deliveries[].updated_at datetime (ISO 8601) Yes Record update timestamp
deliveries[].attempt_count integer Yes Number of transport attempts
deliveries[].failed_at datetime (ISO 8601) No Timestamp of latest failed delivery
deliveries[].last_attempt_at datetime (ISO 8601) No Timestamp of the latest delivery attempt
deliveries[].operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
deliveries[].succeeded_at datetime (ISO 8601) No Timestamp of successful delivery
delivery_count integer Yes Number of deliveries in the group
destinations array<object> Yes Unique destinations attached to the grouped alert
destinations[].id string Yes Alert destination UUID
destinations[].name string Yes Destination display name
destinations[].type string (email, slack, webhook, pagerduty) Yes Destination type
event_type string Yes Alert event type
failure_group object No Failure group linked to the alert
failure_group.id string Yes Failure group UUID
failure_group.status string (open, acknowledged, resolved) Yes Failure group status
failure_group.title string Yes Failure group title
failure_group.severity string (low, medium, high, critical) Yes Failure group severity
failure_group.occurrences integer Yes Total grouped occurrences
failure_group.workflow_target_id string No Workflow target UUID
latest_delivery_status string (pending, sent, failed, succeeded) Yes Latest transport status across grouped deliveries
resolved_at datetime (ISO 8601) No When the alert was resolved
resolved_by object No User who resolved the alert
resolved_by.id string Yes User UUID
resolved_by.email string Yes User email
resource_id string No Primary resource identifier tied to the alert
snoozed_until datetime (ISO 8601) No When the alert snooze expires
workflow_target object No Workflow target connected through the failure group
workflow_target.id string Yes Workflow target UUID
workflow_target.name string Yes Workflow target name
Example
{
  "acknowledged_at": "2026-03-12T09:08:00Z",
  "acknowledged_by": {
    "email": "sre@app.company",
    "id": "0f1cdac9-305d-4d11-a53d-3fca310c0635"
  },
  "alert_group_key": "36c7474d4600b9d0e1d29fe0c60b94f331762ee7c2f4f2fe875d97acbdaf0ef9",
  "alert_rule": {
    "conditions": {
      "detector": "failure_group_occurrence_spike",
      "min_failure_severity": "high",
      "min_occurrences": 3,
      "window_minutes": 60
    },
    "event_type": "failure_group.anomaly_detected",
    "id": "5abffead-a4f6-479d-b2e1-d4176fa8d248",
    "name": "Checkout anomaly",
    "severity": "error"
  },
  "deliveries": [
    {
      "attempt_count": 1,
      "destination": {
        "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
        "name": "Primary Slack",
        "type": "slack"
      },
      "error_message": null,
      "failed_at": null,
      "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
      "inserted_at": "2026-03-12T09:05:02Z",
      "last_attempt_at": "2026-03-12T09:05:03Z",
      "operator_status": "acknowledged",
      "status": "succeeded",
      "succeeded_at": "2026-03-12T09:05:03Z",
      "updated_at": "2026-03-12T09:05:03Z"
    }
  ],
  "delivery_count": 2,
  "destinations": [
    {
      "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
      "name": "Primary Slack",
      "type": "slack"
    }
  ],
  "event_type": "failure_group.anomaly_detected",
  "failure_group": {
    "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
    "occurrences": 7,
    "severity": "high",
    "status": "open",
    "title": "Payment timeout",
    "workflow_target_id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea"
  },
  "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
  "inserted_at": "2026-03-12T09:05:02Z",
  "latest_delivery_status": "succeeded",
  "message": "5 matching failures detected in the last 60 minutes for Payment timeout.",
  "operator_status": "acknowledged",
  "resolved_at": null,
  "resolved_by": null,
  "resource_id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
  "severity": "error",
  "snoozed_until": null,
  "summary": "Timeout after payment provider retry budget was exhausted.",
  "title": "Payment timeout anomaly detected",
  "trace": {
    "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
    "name": "checkout runtime trace",
    "request_id": "req-01HQ5PX7P7F4",
    "status": "error"
  },
  "updated_at": "2026-03-12T09:05:03Z",
  "workflow_target": {
    "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
    "name": "checkout-api"
  }
}
404

Alert not found
POST
/api/v1/operator/alerts/:id/acknowledge Web Session

Acknowledge alert

Description

Marks the grouped alert as acknowledged across all linked deliveries. Invalid IDs return 422 and missing alerts return 404.

Auth

Web Session
Parameters
Name In Type Required Description
id path string Yes Representative alert UUID
Responses
200

Acknowledged alert

Returns: Alert
Response Body
Field Type Required Description
id string Yes Representative delivery UUID for the grouped alert
message string Yes Operator-facing alert message
trace object No Trace linked to the alert
trace.id string Yes Trace UUID
trace.name string Yes Trace name
trace.status string (running, ok, error, partial) Yes Trace status
trace.request_id string No Internal request identifier
title string Yes Operator-facing alert title
severity string (info, warning, error, critical) Yes Alert severity
inserted_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery
updated_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery update
summary string No Optional short summary
operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
acknowledged_at datetime (ISO 8601) No When the alert was acknowledged
acknowledged_by object No User who acknowledged the alert
acknowledged_by.id string Yes User UUID
acknowledged_by.email string Yes User email
alert_group_key string Yes Stable grouped alert key
alert_rule object No Rule that emitted the alert
alert_rule.id string Yes Alert rule UUID
alert_rule.name string Yes Rule name
alert_rule.severity string (info, warning, error, critical) Yes Delivery severity attached by the rule
alert_rule.conditions object Yes Rule conditions map. For anomaly rules, `window_minutes` is evaluated against trace completion time (`ended_at`, falling back to `inserted_at`).
alert_rule.event_type string No Event type matched by the rule
deliveries array<object> Yes Per-destination delivery attempts for this grouped alert
deliveries[].id string Yes Alert delivery UUID
deliveries[].status string (pending, sent, failed, succeeded) Yes Transport delivery status
deliveries[].destination object Yes Destination used for this delivery
deliveries[].destination.id string Yes Alert destination UUID
deliveries[].destination.name string Yes Destination display name
deliveries[].destination.type string (email, slack, webhook, pagerduty) Yes Destination type
deliveries[].error_message string No Latest transport error message
deliveries[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
deliveries[].updated_at datetime (ISO 8601) Yes Record update timestamp
deliveries[].attempt_count integer Yes Number of transport attempts
deliveries[].failed_at datetime (ISO 8601) No Timestamp of latest failed delivery
deliveries[].last_attempt_at datetime (ISO 8601) No Timestamp of the latest delivery attempt
deliveries[].operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
deliveries[].succeeded_at datetime (ISO 8601) No Timestamp of successful delivery
delivery_count integer Yes Number of deliveries in the group
destinations array<object> Yes Unique destinations attached to the grouped alert
destinations[].id string Yes Alert destination UUID
destinations[].name string Yes Destination display name
destinations[].type string (email, slack, webhook, pagerduty) Yes Destination type
event_type string Yes Alert event type
failure_group object No Failure group linked to the alert
failure_group.id string Yes Failure group UUID
failure_group.status string (open, acknowledged, resolved) Yes Failure group status
failure_group.title string Yes Failure group title
failure_group.severity string (low, medium, high, critical) Yes Failure group severity
failure_group.occurrences integer Yes Total grouped occurrences
failure_group.workflow_target_id string No Workflow target UUID
latest_delivery_status string (pending, sent, failed, succeeded) Yes Latest transport status across grouped deliveries
resolved_at datetime (ISO 8601) No When the alert was resolved
resolved_by object No User who resolved the alert
resolved_by.id string Yes User UUID
resolved_by.email string Yes User email
resource_id string No Primary resource identifier tied to the alert
snoozed_until datetime (ISO 8601) No When the alert snooze expires
workflow_target object No Workflow target connected through the failure group
workflow_target.id string Yes Workflow target UUID
workflow_target.name string Yes Workflow target name
Example
{
  "acknowledged_at": "2026-03-12T09:08:00Z",
  "acknowledged_by": {
    "email": "sre@app.company",
    "id": "0f1cdac9-305d-4d11-a53d-3fca310c0635"
  },
  "alert_group_key": "36c7474d4600b9d0e1d29fe0c60b94f331762ee7c2f4f2fe875d97acbdaf0ef9",
  "alert_rule": {
    "conditions": {
      "detector": "failure_group_occurrence_spike",
      "min_failure_severity": "high",
      "min_occurrences": 3,
      "window_minutes": 60
    },
    "event_type": "failure_group.anomaly_detected",
    "id": "5abffead-a4f6-479d-b2e1-d4176fa8d248",
    "name": "Checkout anomaly",
    "severity": "error"
  },
  "deliveries": [
    {
      "attempt_count": 1,
      "destination": {
        "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
        "name": "Primary Slack",
        "type": "slack"
      },
      "error_message": null,
      "failed_at": null,
      "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
      "inserted_at": "2026-03-12T09:05:02Z",
      "last_attempt_at": "2026-03-12T09:05:03Z",
      "operator_status": "acknowledged",
      "status": "succeeded",
      "succeeded_at": "2026-03-12T09:05:03Z",
      "updated_at": "2026-03-12T09:05:03Z"
    }
  ],
  "delivery_count": 2,
  "destinations": [
    {
      "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
      "name": "Primary Slack",
      "type": "slack"
    }
  ],
  "event_type": "failure_group.anomaly_detected",
  "failure_group": {
    "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
    "occurrences": 7,
    "severity": "high",
    "status": "open",
    "title": "Payment timeout",
    "workflow_target_id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea"
  },
  "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
  "inserted_at": "2026-03-12T09:05:02Z",
  "latest_delivery_status": "succeeded",
  "message": "5 matching failures detected in the last 60 minutes for Payment timeout.",
  "operator_status": "acknowledged",
  "resolved_at": null,
  "resolved_by": null,
  "resource_id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
  "severity": "error",
  "snoozed_until": null,
  "summary": "Timeout after payment provider retry budget was exhausted.",
  "title": "Payment timeout anomaly detected",
  "trace": {
    "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
    "name": "checkout runtime trace",
    "request_id": "req-01HQ5PX7P7F4",
    "status": "error"
  },
  "updated_at": "2026-03-12T09:05:03Z",
  "workflow_target": {
    "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
    "name": "checkout-api"
  }
}
404

Alert not found
422

Invalid alert ID format
POST
/api/v1/operator/alerts/:id/snooze Web Session

Snooze alert

Description

Snoozes the grouped alert for a fixed duration in minutes, at most 43200 (30 days). Invalid durations return 400, invalid IDs return 422, and missing alerts return 404.

Auth

Web Session
Request Body
Field Type Required Description
duration_minutes integer Yes How long to snooze the alert, from 1 to 43200 minutes
Parameters
Name In Type Required Description
id path string Yes Representative alert UUID
Responses
200

Snoozed alert

Returns: Alert
Response Body
Field Type Required Description
id string Yes Representative delivery UUID for the grouped alert
message string Yes Operator-facing alert message
trace object No Trace linked to the alert
trace.id string Yes Trace UUID
trace.name string Yes Trace name
trace.status string (running, ok, error, partial) Yes Trace status
trace.request_id string No Internal request identifier
title string Yes Operator-facing alert title
severity string (info, warning, error, critical) Yes Alert severity
inserted_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery
updated_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery update
summary string No Optional short summary
operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
acknowledged_at datetime (ISO 8601) No When the alert was acknowledged
acknowledged_by object No User who acknowledged the alert
acknowledged_by.id string Yes User UUID
acknowledged_by.email string Yes User email
alert_group_key string Yes Stable grouped alert key
alert_rule object No Rule that emitted the alert
alert_rule.id string Yes Alert rule UUID
alert_rule.name string Yes Rule name
alert_rule.severity string (info, warning, error, critical) Yes Delivery severity attached by the rule
alert_rule.conditions object Yes Rule conditions map. For anomaly rules, `window_minutes` is evaluated against trace completion time (`ended_at`, falling back to `inserted_at`).
alert_rule.event_type string No Event type matched by the rule
deliveries array<object> Yes Per-destination delivery attempts for this grouped alert
deliveries[].id string Yes Alert delivery UUID
deliveries[].status string (pending, sent, failed, succeeded) Yes Transport delivery status
deliveries[].destination object Yes Destination used for this delivery
deliveries[].destination.id string Yes Alert destination UUID
deliveries[].destination.name string Yes Destination display name
deliveries[].destination.type string (email, slack, webhook, pagerduty) Yes Destination type
deliveries[].error_message string No Latest transport error message
deliveries[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
deliveries[].updated_at datetime (ISO 8601) Yes Record update timestamp
deliveries[].attempt_count integer Yes Number of transport attempts
deliveries[].failed_at datetime (ISO 8601) No Timestamp of latest failed delivery
deliveries[].last_attempt_at datetime (ISO 8601) No Timestamp of the latest delivery attempt
deliveries[].operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
deliveries[].succeeded_at datetime (ISO 8601) No Timestamp of successful delivery
delivery_count integer Yes Number of deliveries in the group
destinations array<object> Yes Unique destinations attached to the grouped alert
destinations[].id string Yes Alert destination UUID
destinations[].name string Yes Destination display name
destinations[].type string (email, slack, webhook, pagerduty) Yes Destination type
event_type string Yes Alert event type
failure_group object No Failure group linked to the alert
failure_group.id string Yes Failure group UUID
failure_group.status string (open, acknowledged, resolved) Yes Failure group status
failure_group.title string Yes Failure group title
failure_group.severity string (low, medium, high, critical) Yes Failure group severity
failure_group.occurrences integer Yes Total grouped occurrences
failure_group.workflow_target_id string No Workflow target UUID
latest_delivery_status string (pending, sent, failed, succeeded) Yes Latest transport status across grouped deliveries
resolved_at datetime (ISO 8601) No When the alert was resolved
resolved_by object No User who resolved the alert
resolved_by.id string Yes User UUID
resolved_by.email string Yes User email
resource_id string No Primary resource identifier tied to the alert
snoozed_until datetime (ISO 8601) No When the alert snooze expires
workflow_target object No Workflow target connected through the failure group
workflow_target.id string Yes Workflow target UUID
workflow_target.name string Yes Workflow target name
Example
{
  "acknowledged_at": "2026-03-12T09:08:00Z",
  "acknowledged_by": {
    "email": "sre@app.company",
    "id": "0f1cdac9-305d-4d11-a53d-3fca310c0635"
  },
  "alert_group_key": "36c7474d4600b9d0e1d29fe0c60b94f331762ee7c2f4f2fe875d97acbdaf0ef9",
  "alert_rule": {
    "conditions": {
      "detector": "failure_group_occurrence_spike",
      "min_failure_severity": "high",
      "min_occurrences": 3,
      "window_minutes": 60
    },
    "event_type": "failure_group.anomaly_detected",
    "id": "5abffead-a4f6-479d-b2e1-d4176fa8d248",
    "name": "Checkout anomaly",
    "severity": "error"
  },
  "deliveries": [
    {
      "attempt_count": 1,
      "destination": {
        "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
        "name": "Primary Slack",
        "type": "slack"
      },
      "error_message": null,
      "failed_at": null,
      "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
      "inserted_at": "2026-03-12T09:05:02Z",
      "last_attempt_at": "2026-03-12T09:05:03Z",
      "operator_status": "acknowledged",
      "status": "succeeded",
      "succeeded_at": "2026-03-12T09:05:03Z",
      "updated_at": "2026-03-12T09:05:03Z"
    }
  ],
  "delivery_count": 2,
  "destinations": [
    {
      "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
      "name": "Primary Slack",
      "type": "slack"
    }
  ],
  "event_type": "failure_group.anomaly_detected",
  "failure_group": {
    "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
    "occurrences": 7,
    "severity": "high",
    "status": "open",
    "title": "Payment timeout",
    "workflow_target_id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea"
  },
  "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
  "inserted_at": "2026-03-12T09:05:02Z",
  "latest_delivery_status": "succeeded",
  "message": "5 matching failures detected in the last 60 minutes for Payment timeout.",
  "operator_status": "snoozed",
  "resolved_at": null,
  "resolved_by": null,
  "resource_id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
  "severity": "error",
  "snoozed_until": "2026-03-12T13:05:03Z",
  "summary": "Timeout after payment provider retry budget was exhausted.",
  "title": "Payment timeout anomaly detected",
  "trace": {
    "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
    "name": "checkout runtime trace",
    "request_id": "req-01HQ5PX7P7F4",
    "status": "error"
  },
  "updated_at": "2026-03-12T09:05:03Z",
  "workflow_target": {
    "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
    "name": "checkout-api"
  }
}
400

Invalid snooze duration
404

Alert not found
422

Invalid alert ID format
POST
/api/v1/operator/alerts/:id/resolve Web Session

Resolve alert

Description

Marks the grouped alert as resolved across all linked deliveries. Invalid IDs return 422 and missing alerts return 404.

Auth

Web Session
Parameters
Name In Type Required Description
id path string Yes Representative alert UUID
Responses
200

Resolved alert

Returns: Alert
Response Body
Field Type Required Description
id string Yes Representative delivery UUID for the grouped alert
message string Yes Operator-facing alert message
trace object No Trace linked to the alert
trace.id string Yes Trace UUID
trace.name string Yes Trace name
trace.status string (running, ok, error, partial) Yes Trace status
trace.request_id string No Internal request identifier
title string Yes Operator-facing alert title
severity string (info, warning, error, critical) Yes Alert severity
inserted_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery
updated_at datetime (ISO 8601) Yes Timestamp of the newest grouped delivery update
summary string No Optional short summary
operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
acknowledged_at datetime (ISO 8601) No When the alert was acknowledged
acknowledged_by object No User who acknowledged the alert
acknowledged_by.id string Yes User UUID
acknowledged_by.email string Yes User email
alert_group_key string Yes Stable grouped alert key
alert_rule object No Rule that emitted the alert
alert_rule.id string Yes Alert rule UUID
alert_rule.name string Yes Rule name
alert_rule.severity string (info, warning, error, critical) Yes Delivery severity attached by the rule
alert_rule.conditions object Yes Rule conditions map. For anomaly rules, `window_minutes` is evaluated against trace completion time (`ended_at`, falling back to `inserted_at`).
alert_rule.event_type string No Event type matched by the rule
deliveries array<object> Yes Per-destination delivery attempts for this grouped alert
deliveries[].id string Yes Alert delivery UUID
deliveries[].status string (pending, sent, failed, succeeded) Yes Transport delivery status
deliveries[].destination object Yes Destination used for this delivery
deliveries[].destination.id string Yes Alert destination UUID
deliveries[].destination.name string Yes Destination display name
deliveries[].destination.type string (email, slack, webhook, pagerduty) Yes Destination type
deliveries[].error_message string No Latest transport error message
deliveries[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
deliveries[].updated_at datetime (ISO 8601) Yes Record update timestamp
deliveries[].attempt_count integer Yes Number of transport attempts
deliveries[].failed_at datetime (ISO 8601) No Timestamp of latest failed delivery
deliveries[].last_attempt_at datetime (ISO 8601) No Timestamp of the latest delivery attempt
deliveries[].operator_status string (open, acknowledged, snoozed, resolved) Yes Operator-managed alert lifecycle status
deliveries[].succeeded_at datetime (ISO 8601) No Timestamp of successful delivery
delivery_count integer Yes Number of deliveries in the group
destinations array<object> Yes Unique destinations attached to the grouped alert
destinations[].id string Yes Alert destination UUID
destinations[].name string Yes Destination display name
destinations[].type string (email, slack, webhook, pagerduty) Yes Destination type
event_type string Yes Alert event type
failure_group object No Failure group linked to the alert
failure_group.id string Yes Failure group UUID
failure_group.status string (open, acknowledged, resolved) Yes Failure group status
failure_group.title string Yes Failure group title
failure_group.severity string (low, medium, high, critical) Yes Failure group severity
failure_group.occurrences integer Yes Total grouped occurrences
failure_group.workflow_target_id string No Workflow target UUID
latest_delivery_status string (pending, sent, failed, succeeded) Yes Latest transport status across grouped deliveries
resolved_at datetime (ISO 8601) No When the alert was resolved
resolved_by object No User who resolved the alert
resolved_by.id string Yes User UUID
resolved_by.email string Yes User email
resource_id string No Primary resource identifier tied to the alert
snoozed_until datetime (ISO 8601) No When the alert snooze expires
workflow_target object No Workflow target connected through the failure group
workflow_target.id string Yes Workflow target UUID
workflow_target.name string Yes Workflow target name
Example
{
  "acknowledged_at": "2026-03-12T09:08:00Z",
  "acknowledged_by": {
    "email": "sre@app.company",
    "id": "0f1cdac9-305d-4d11-a53d-3fca310c0635"
  },
  "alert_group_key": "36c7474d4600b9d0e1d29fe0c60b94f331762ee7c2f4f2fe875d97acbdaf0ef9",
  "alert_rule": {
    "conditions": {
      "detector": "failure_group_occurrence_spike",
      "min_failure_severity": "high",
      "min_occurrences": 3,
      "window_minutes": 60
    },
    "event_type": "failure_group.anomaly_detected",
    "id": "5abffead-a4f6-479d-b2e1-d4176fa8d248",
    "name": "Checkout anomaly",
    "severity": "error"
  },
  "deliveries": [
    {
      "attempt_count": 1,
      "destination": {
        "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
        "name": "Primary Slack",
        "type": "slack"
      },
      "error_message": null,
      "failed_at": null,
      "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
      "inserted_at": "2026-03-12T09:05:02Z",
      "last_attempt_at": "2026-03-12T09:05:03Z",
      "operator_status": "acknowledged",
      "status": "succeeded",
      "succeeded_at": "2026-03-12T09:05:03Z",
      "updated_at": "2026-03-12T09:05:03Z"
    }
  ],
  "delivery_count": 2,
  "destinations": [
    {
      "id": "b2cbaf23-12dc-4137-b4f2-4951803a307e",
      "name": "Primary Slack",
      "type": "slack"
    }
  ],
  "event_type": "failure_group.anomaly_detected",
  "failure_group": {
    "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
    "occurrences": 7,
    "severity": "high",
    "status": "open",
    "title": "Payment timeout",
    "workflow_target_id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea"
  },
  "id": "6f1a651c-bfc7-41cc-b598-98a30fda59ca",
  "inserted_at": "2026-03-12T09:05:02Z",
  "latest_delivery_status": "succeeded",
  "message": "5 matching failures detected in the last 60 minutes for Payment timeout.",
  "operator_status": "resolved",
  "resolved_at": "2026-03-12T09:15:00Z",
  "resolved_by": null,
  "resource_id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
  "severity": "error",
  "snoozed_until": null,
  "summary": "Timeout after payment provider retry budget was exhausted.",
  "title": "Payment timeout anomaly detected",
  "trace": {
    "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
    "name": "checkout runtime trace",
    "request_id": "req-01HQ5PX7P7F4",
    "status": "error"
  },
  "updated_at": "2026-03-12T09:05:03Z",
  "workflow_target": {
    "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
    "name": "checkout-api"
  }
}
404

Alert not found
422

Invalid alert ID format
GET
/api/v1/operator/failure_groups Web Session

List failure groups

Description

Returns grouped failure patterns for analysis: the 200 most recently seen groups.

Auth

Web Session
Parameters
Name In Type Required Description
filter[status] query string No Filter by workflow status
filter[severity] query string No Filter by failure severity
filter[workflow_target_id] query string No Workflow target UUID
Responses
200

List of failure groups

Returns: FailureGroupList
Response Body
Field Type Required Description
failure_groups array<object> Yes List of grouped failures
failure_groups[].id string Yes Failure group UUID
failure_groups[].status string (open, acknowledged, resolved) Yes Failure group status
failure_groups[].title string Yes Operator-facing failure title
failure_groups[].signature string Yes Stable failure signature
failure_groups[].severity string (low, medium, high, critical) Yes Failure severity
failure_groups[].inserted_at datetime (ISO 8601) Yes Record insertion timestamp
failure_groups[].updated_at datetime (ISO 8601) Yes Record update timestamp
failure_groups[].last_seen_at datetime (ISO 8601) No Timestamp of the most recent occurrence
failure_groups[].occurrences integer Yes Total grouped occurrences
failure_groups[].workflow_target object No Workflow target associated with the grouped failures
failure_groups[].workflow_target.id string Yes Workflow target UUID
failure_groups[].workflow_target.name string Yes Workflow target name
failure_groups[].assignee string No Current assignee handling the incident
failure_groups[].collaboration_notes array<object> Yes Collaboration notes for the failure group
failure_groups[].collaboration_notes[].inserted_at datetime (ISO 8601) Yes Entry insertion timestamp
failure_groups[].collaboration_notes[].author_email string No Author email
failure_groups[].collaboration_notes[].author_id string No Author user UUID
failure_groups[].collaboration_notes[].note string Yes Free-form collaboration note
failure_groups[].escalation_level string No Current escalation level
failure_groups[].escalation_updates array<object> Yes Escalation timeline entries
failure_groups[].escalation_updates[].level string No Escalation level at the time of the note
failure_groups[].escalation_updates[].inserted_at datetime (ISO 8601) Yes Entry insertion timestamp
failure_groups[].escalation_updates[].author_email string No Author email
failure_groups[].escalation_updates[].author_id string No Author user UUID
failure_groups[].escalation_updates[].note string Yes Escalation note
failure_groups[].first_seen_at datetime (ISO 8601) No Timestamp of the first occurrence
failure_groups[].incident_eval_suggestions array<object> Yes Incident eval suggestions generated from this failure group
failure_groups[].incident_eval_suggestions[].id string Yes Incident eval suggestion UUID
failure_groups[].incident_eval_suggestions[].status string (draft, approved, rejected, promoted) Yes Suggestion status
failure_groups[].incident_eval_suggestions[].approved_eval_case_id string No Approved eval case UUID
failure_groups[].incident_eval_suggestions[].proposed_name string No Suggested eval name
failure_groups[].incident_eval_suggestions[].target_eval_suite_id string No Target eval suite UUID
failure_groups[].incident_owner string No Incident owner responsible for the group
failure_groups[].playbook_url string No Linked remediation playbook URL
failure_groups[].remediation_checklist array<object> Yes Checklist items attached to the remediation plan
failure_groups[].remediation_checklist[].done boolean Yes Whether the item is complete
failure_groups[].remediation_checklist[].item string Yes Checklist entry text
failure_groups[].remediation_due_at datetime (ISO 8601) No Target remediation due time
failure_groups[].remediation_status string No Current remediation state
failure_groups[].sample_trace object No Sample trace summary
failure_groups[].sample_trace.id string Yes Trace UUID
failure_groups[].sample_trace.name string Yes Trace name
failure_groups[].sample_trace.status string (running, ok, error, partial) Yes Trace status
failure_groups[].sample_trace_id string No UUID of the sample trace
Example
{
  "failure_groups": [
    {
      "assignee": "sre@app.company",
      "collaboration_notes": [
        {
          "author_email": "sre@app.company",
          "author_id": "0f1cdac9-305d-4d11-a53d-3fca310c0635",
          "inserted_at": "2026-03-12T08:47:00Z",
          "note": "Started triage and log correlation."
        }
      ],
      "escalation_level": "l2",
      "escalation_updates": [
        {
          "author_email": "sre@app.company",
          "author_id": "0f1cdac9-305d-4d11-a53d-3fca310c0635",
          "inserted_at": "2026-03-12T08:46:00Z",
          "level": "l2",
          "note": "Paging platform responders."
        }
      ],
      "first_seen_at": "2026-03-11T22:14:00Z",
      "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
      "incident_eval_suggestions": [
        {
          "approved_eval_case_id": null,
          "id": "6a4473e5-5a35-4f0b-b99e-34c59d495f67",
          "proposed_name": "Payment timeout recovery eval",
          "status": "draft",
          "target_eval_suite_id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699"
        }
      ],
      "incident_owner": "incident-lead@app.company",
      "inserted_at": "2026-03-11T22:14:00Z",
      "last_seen_at": "2026-03-12T08:45:03Z",
      "occurrences": 7,
      "playbook_url": "https://runbooks.example.com/failure/payment-timeout",
      "remediation_checklist": [
        {
          "done": false,
          "item": "Collect provider traces"
        },
        {
          "done": false,
          "item": "Roll out timeout hotfix"
        }
      ],
      "remediation_due_at": "2026-03-12T10:00:00Z",
      "remediation_status": "in_progress",
      "sample_trace": {
        "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
        "name": "checkout runtime trace",
        "status": "error"
      },
      "sample_trace_id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
      "severity": "high",
      "signature": "payment:timeout",
      "status": "open",
      "title": "Payment timeout",
      "updated_at": "2026-03-12T08:45:03Z",
      "workflow_target": {
        "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
        "name": "checkout-api"
      }
    }
  ]
}
GET
/api/v1/operator/failure_groups/:id Web Session

Get failure group details

Description

Returns details of a specific failure group.

Auth

Web Session
Parameters
Name In Type Required Description
id path string Yes Failure group UUID
Responses
200

Failure group details

Returns: FailureGroup
Response Body
Field Type Required Description
id string Yes Failure group UUID
status string (open, acknowledged, resolved) Yes Failure group status
title string Yes Operator-facing failure title
signature string Yes Stable failure signature
severity string (low, medium, high, critical) Yes Failure severity
inserted_at datetime (ISO 8601) Yes Record insertion timestamp
updated_at datetime (ISO 8601) Yes Record update timestamp
last_seen_at datetime (ISO 8601) No Timestamp of the most recent occurrence
occurrences integer Yes Total grouped occurrences
workflow_target object No Workflow target associated with the grouped failures
workflow_target.id string Yes Workflow target UUID
workflow_target.name string Yes Workflow target name
assignee string No Current assignee handling the incident
collaboration_notes array<object> Yes Collaboration notes for the failure group
collaboration_notes[].inserted_at datetime (ISO 8601) Yes Entry insertion timestamp
collaboration_notes[].author_email string No Author email
collaboration_notes[].author_id string No Author user UUID
collaboration_notes[].note string Yes Free-form collaboration note
escalation_level string No Current escalation level
escalation_updates array<object> Yes Escalation timeline entries
escalation_updates[].level string No Escalation level at the time of the note
escalation_updates[].inserted_at datetime (ISO 8601) Yes Entry insertion timestamp
escalation_updates[].author_email string No Author email
escalation_updates[].author_id string No Author user UUID
escalation_updates[].note string Yes Escalation note
first_seen_at datetime (ISO 8601) No Timestamp of the first occurrence
incident_eval_suggestions array<object> Yes Incident eval suggestions generated from this failure group
incident_eval_suggestions[].id string Yes Incident eval suggestion UUID
incident_eval_suggestions[].status string (draft, approved, rejected, promoted) Yes Suggestion status
incident_eval_suggestions[].approved_eval_case_id string No Approved eval case UUID
incident_eval_suggestions[].proposed_name string No Suggested eval name
incident_eval_suggestions[].target_eval_suite_id string No Target eval suite UUID
incident_owner string No Incident owner responsible for the group
playbook_url string No Linked remediation playbook URL
remediation_checklist array<object> Yes Checklist items attached to the remediation plan
remediation_checklist[].done boolean Yes Whether the item is complete
remediation_checklist[].item string Yes Checklist entry text
remediation_due_at datetime (ISO 8601) No Target remediation due time
remediation_status string No Current remediation state
sample_trace object No Sample trace summary
sample_trace.id string Yes Trace UUID
sample_trace.name string Yes Trace name
sample_trace.status string (running, ok, error, partial) Yes Trace status
sample_trace_id string No UUID of the sample trace
Example
{
  "assignee": "sre@app.company",
  "collaboration_notes": [
    {
      "author_email": "sre@app.company",
      "author_id": "0f1cdac9-305d-4d11-a53d-3fca310c0635",
      "inserted_at": "2026-03-12T08:47:00Z",
      "note": "Started triage and log correlation."
    }
  ],
  "escalation_level": "l2",
  "escalation_updates": [
    {
      "author_email": "sre@app.company",
      "author_id": "0f1cdac9-305d-4d11-a53d-3fca310c0635",
      "inserted_at": "2026-03-12T08:46:00Z",
      "level": "l2",
      "note": "Paging platform responders."
    }
  ],
  "first_seen_at": "2026-03-11T22:14:00Z",
  "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
  "incident_eval_suggestions": [
    {
      "approved_eval_case_id": null,
      "id": "6a4473e5-5a35-4f0b-b99e-34c59d495f67",
      "proposed_name": "Payment timeout recovery eval",
      "status": "draft",
      "target_eval_suite_id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699"
    }
  ],
  "incident_owner": "incident-lead@app.company",
  "inserted_at": "2026-03-11T22:14:00Z",
  "last_seen_at": "2026-03-12T08:45:03Z",
  "occurrences": 7,
  "playbook_url": "https://runbooks.example.com/failure/payment-timeout",
  "remediation_checklist": [
    {
      "done": false,
      "item": "Collect provider traces"
    },
    {
      "done": false,
      "item": "Roll out timeout hotfix"
    }
  ],
  "remediation_due_at": "2026-03-12T10:00:00Z",
  "remediation_status": "in_progress",
  "sample_trace": {
    "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
    "name": "checkout runtime trace",
    "status": "error"
  },
  "sample_trace_id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
  "severity": "high",
  "signature": "payment:timeout",
  "status": "open",
  "title": "Payment timeout",
  "updated_at": "2026-03-12T08:45:03Z",
  "workflow_target": {
    "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
    "name": "checkout-api"
  }
}
404

Failure group not found
PATCH
/api/v1/operator/failure_groups/:id Web Session

Update failure group

Description

Updates failure group settings (e.g., to mark as resolved).

Auth

Web Session
Request Body
Field Type Required Description
failure_group object Yes Failure group attributes to update
failure_group.status string (open, acknowledged, resolved) No Group status
failure_group.assignee string No Current owner/on-call alias responsible for this failure group
failure_group.escalation_level string (none, l1, l2, l3, sev1) No Escalation level
failure_group.incident_owner string No Incident owner responsible for this failure group
failure_group.playbook_url string No http(s) remediation playbook URL
failure_group.remediation_due_at datetime (ISO 8601) No Target remediation due time
failure_group.remediation_status string (identified, in_progress, monitoring, completed) No Remediation state
failure_group.collaboration_note string No Appends a collaboration timeline note to this failure group
failure_group.escalation_note string No Appends an escalation timeline note
failure_group.remediation_items array<object> No Remediation checklist items written into metadata
Parameters
Name In Type Required Description
id path string Yes Failure group UUID
Responses
200

Failure group updated

Returns: FailureGroup
Response Body
Field Type Required Description
id string Yes Failure group UUID
status string (open, acknowledged, resolved) Yes Failure group status
title string Yes Operator-facing failure title
signature string Yes Stable failure signature
severity string (low, medium, high, critical) Yes Failure severity
inserted_at datetime (ISO 8601) Yes Record insertion timestamp
updated_at datetime (ISO 8601) Yes Record update timestamp
last_seen_at datetime (ISO 8601) No Timestamp of the most recent occurrence
occurrences integer Yes Total grouped occurrences
workflow_target object No Workflow target associated with the grouped failures
workflow_target.id string Yes Workflow target UUID
workflow_target.name string Yes Workflow target name
assignee string No Current assignee handling the incident
collaboration_notes array<object> Yes Collaboration notes for the failure group
collaboration_notes[].inserted_at datetime (ISO 8601) Yes Entry insertion timestamp
collaboration_notes[].author_email string No Author email
collaboration_notes[].author_id string No Author user UUID
collaboration_notes[].note string Yes Free-form collaboration note
escalation_level string No Current escalation level
escalation_updates array<object> Yes Escalation timeline entries
escalation_updates[].level string No Escalation level at the time of the note
escalation_updates[].inserted_at datetime (ISO 8601) Yes Entry insertion timestamp
escalation_updates[].author_email string No Author email
escalation_updates[].author_id string No Author user UUID
escalation_updates[].note string Yes Escalation note
first_seen_at datetime (ISO 8601) No Timestamp of the first occurrence
incident_eval_suggestions array<object> Yes Incident eval suggestions generated from this failure group
incident_eval_suggestions[].id string Yes Incident eval suggestion UUID
incident_eval_suggestions[].status string (draft, approved, rejected, promoted) Yes Suggestion status
incident_eval_suggestions[].approved_eval_case_id string No Approved eval case UUID
incident_eval_suggestions[].proposed_name string No Suggested eval name
incident_eval_suggestions[].target_eval_suite_id string No Target eval suite UUID
incident_owner string No Incident owner responsible for the group
playbook_url string No Linked remediation playbook URL
remediation_checklist array<object> Yes Checklist items attached to the remediation plan
remediation_checklist[].done boolean Yes Whether the item is complete
remediation_checklist[].item string Yes Checklist entry text
remediation_due_at datetime (ISO 8601) No Target remediation due time
remediation_status string No Current remediation state
sample_trace object No Sample trace summary
sample_trace.id string Yes Trace UUID
sample_trace.name string Yes Trace name
sample_trace.status string (running, ok, error, partial) Yes Trace status
sample_trace_id string No UUID of the sample trace
Example
{
  "assignee": "sre@app.company",
  "collaboration_notes": [
    {
      "author_email": "sre@app.company",
      "author_id": "0f1cdac9-305d-4d11-a53d-3fca310c0635",
      "inserted_at": "2026-03-12T08:47:00Z",
      "note": "Started triage and log correlation."
    }
  ],
  "escalation_level": "l2",
  "escalation_updates": [
    {
      "author_email": "sre@app.company",
      "author_id": "0f1cdac9-305d-4d11-a53d-3fca310c0635",
      "inserted_at": "2026-03-12T08:46:00Z",
      "level": "l2",
      "note": "Paging platform responders."
    }
  ],
  "first_seen_at": "2026-03-11T22:14:00Z",
  "id": "95c77f8c-a4b8-4818-a9bc-aadad23cba02",
  "incident_eval_suggestions": [
    {
      "approved_eval_case_id": null,
      "id": "6a4473e5-5a35-4f0b-b99e-34c59d495f67",
      "proposed_name": "Payment timeout recovery eval",
      "status": "draft",
      "target_eval_suite_id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699"
    }
  ],
  "incident_owner": "incident-lead@app.company",
  "inserted_at": "2026-03-11T22:14:00Z",
  "last_seen_at": "2026-03-12T08:45:03Z",
  "occurrences": 7,
  "playbook_url": "https://runbooks.example.com/failure/payment-timeout",
  "remediation_checklist": [
    {
      "done": false,
      "item": "Collect provider traces"
    },
    {
      "done": false,
      "item": "Roll out timeout hotfix"
    }
  ],
  "remediation_due_at": "2026-03-12T10:00:00Z",
  "remediation_status": "in_progress",
  "sample_trace": {
    "id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
    "name": "checkout runtime trace",
    "status": "error"
  },
  "sample_trace_id": "6646e3ff-8f2f-4204-ad5c-f4ebc1830f9e",
  "severity": "high",
  "signature": "payment:timeout",
  "status": "open",
  "title": "Payment timeout",
  "updated_at": "2026-03-12T08:45:03Z",
  "workflow_target": {
    "id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea",
    "name": "checkout-api"
  }
}
404

Failure group not found

Where to go next

see also
Audit Connect runtime evidence to accountable actions. Evals Convert repeated incidents into regression coverage.