Platform
Webhook delivery, signing, and retries
Receive platform events without polling. A matching subscription sends an HTTP POST to your URL; your receiver verifies its signature, acknowledges durable receipt, and safely handles retries.
On this page
Start here
React without polling: preserve the raw body, verify the signature, acknowledge promptly, and make processing safe across retries.
Before you begin
- A publicly reachable HTTPS receiver allowed by the outbound URL policy
- A caller-supplied secret or a plan to capture rotate_secret output
- Durable delivery-ID storage
Put it into practice
-
Create and test the subscription
Choose the event types, then either create through the CSRF-authenticated operator API with a secret you already stored, or create first and call rotate_secret once to capture its returned secret. Send webhook.test afterward.
-
Verify before parsing
Build <timestamp>.<raw body>, calculate the lowercase HMAC-SHA256 digest, and compare it with x-agentops-signature in constant time. Reject a mismatch before doing work.
-
Acknowledge and process safely
Return success after durable acceptance, then process asynchronously. Deduplicate by delivery id.
A test delivery arrives, passes signature verification, is acknowledged once durably accepted, and produces one application action even if the platform retries it.
01 Delivery payload
detailsPOST body fields
Each delivery POSTs JSON with id, event_type, workspace_id, occurred_at, data, and delivery_attempt. Test deliveries from POST .../test also set test: true. This is not the realtime Events envelope (which uses type and has no delivery_attempt). Headers are x-agentops-event, x-agentops-timestamp (Unix seconds), and x-agentops-signature (HMAC-SHA256).
02 Signature verification
exampleHMAC-SHA256
1) Concatenate timestamp UTF-8 bytes plus b'.' plus the raw body bytes; never reserialize JSON. 2) Compute HMAC-SHA256 using the subscription's signing secret as the key. 3) Hex-encode the result (lowercase). Always use constant-time comparison to prevent timing attacks.
# Python example
import hmac, hashlib
def verify_signature(timestamp, body, secret, signature):
message = timestamp.encode("utf-8") + b"." + body
expected = hmac.new(
secret.encode("utf-8"), message, hashlib.sha256
).hexdigest()
return hmac.compare_digest(expected, signature)03 Retry strategy
detailsExponential backoff
Transport failures and HTTP 408, 425, 429, or 5xx responses retry with 30 * 2^(attempt - 1) second backoff, capped at 30 minutes and 8 attempts. Other 4xx responses, blocked outbound URLs, and invalid signing secrets fail without retry. Automatic redirects are disabled. Each attempt has a 10-second overall deadline (connect and response); a receiver that answers later counts as a transport failure and is retried. Acknowledge with a 2xx immediately and process asynchronously. Only the first 4 KB of the response body is read, and up to 2,000 characters are stored as last_response_body / last_error.
04 Event types
detailsPublished webhook events
Subscriptions can specify individual event types or use wildcard *. Realtime.Events always fans matching publishes into webhook delivery. Shipped types include command lifecycle; trace.created, trace.event.appended, and trace.finalized; replay started/updated/completed/failed/stuck_detected; approval_request created/decided/executed; eval_run.completed; fleet lifecycle plus fleet.batch_operation.created/progress/completed; failure_group.created; webhook.test; security.violation.created, security.violation.resolved, and security.report.generated; and gateway.frozen / gateway.thawed.
failure_group.anomaly_detected is not a webhook event
That string is an alert-rule trigger. When it fires the platform publishes alert.created to destinations. The webhook and realtime event for a new group is failure_group.created (failure_group_id, trace_id, signature, severity).
05 Alert destinations and rules
detailsCreate a destination before a rule
A workspace owner or admin opens /app/admin and creates an alert destination of type webhook, slack, discord, email, or pagerduty. Webhook, Slack, and Discord need a webhook_url; email needs a valid recipient_email; PagerDuty needs a routing_key. Invalid or missing targets fail validation. These are alert transports, distinct from signed event webhook subscriptions.
Test delivery, then bind a rule
Use Send test on the destination to queue an alert.test delivery through the production worker, then inspect the delivery status rather than treating enqueueing as receipt. Create a named generic event rule with event_type, severity, optional deduplication window, and selected destination IDs; or choose failure-group anomaly and set min_occurrences, window_minutes, and min_failure_severity. Rule creation and destination links are atomic: an unknown or foreign destination rolls the rule back. Owners/admins manage this page; rules dispatch only when a matching event occurs and the destination can deliver.
06 Operator console
details/app/admin
Workspace admin can create subscriptions, send a test delivery (event_type webhook.test, test true), list recent attempts, filter failures, and redeliver them. The console never reveals or rotates the signing secret. For a usable receiver, either create through POST /api/v1/operator/webhook_subscriptions with a caller-chosen signing_secret, or create first and make the CSRF-authenticated POST .../:id/rotate_secret call, which returns the new value once. Create and list responses omit signing_secret.
Delivery history retention
Finished (succeeded or failed) deliveries are kept for 30 days and then pruned automatically; pending deliveries are never pruned. A deletion request with dataset "all" under Data controls (POST /api/v1/operator/deletion_requests, optional older_than_days) also removes the workspace's delivery history, which contains the event payloads that were sent. Any non-delivering record can be redelivered within that window; a delivering record returns 400 in_flight. Successful-delivery replays can duplicate receiver side effects, so deduplicate by event ID.
Troubleshooting
- Every signature comparison fails
- Verify the exact raw bytes, not re-serialized JSON, and include the timestamp plus period. Confirm the receiver has the caller-supplied secret or the latest one-time rotate_secret result; create and list responses never reveal a generated secret.
- Deliveries stop retrying
- Only transport failures, 408, 425, 429, and 5xx responses retry. Responses slower than 10 seconds are retried as transport failures. Fix other 4xx responses, blocked URLs, or secret configuration before requesting another test.