A trace is the timeline of one execution: what started, which steps ran, where time or money was spent, and what failed. Send traces from your agent to make individual runs inspectable in the console. Use the native JSON API for explicit start, event, and finish operations, or the OTLP/HTTP JSON receiver when your runtime already exports OpenTelemetry spans.
Trace and event idempotency are independent
external_trace_id finds the same workspace trace on retry. Events append again unless each carries a stable source_event_id, unique within that trace. Use both layers whenever creation and event delivery can each be retried.
Finalization is an explicit transition
Create with running for an open execution. Finish by POSTing the same external ID and name with ok, error, or partial and no repeated events. Durable projection jobs commit with ingest, while search, failure grouping, realtime, and webhook staging remain eventually consistent.
Native and OTLP limits differ
Native calls accept at most 50 events per request. OTLP JSON accepts identity or gzip, caps encoded and decoded bodies at 8 MiB and batches at 512 spans, returns 200 partialSuccess for rejected spans, and uses OTLP {code, message} errors rather than problem JSON.
Before you begin
The command-line examples require curl and jq; any exception is identified in its introduction.
A machine credential with traces:write for ingest or traces:read for detail and event reads; primary credential defaults omit traces:read, so an administrator must explicitly grant it.
Stable external_trace_id and per-event source_event_id values when retry-safe native ingest is required.
For OTLP, an HTTP/JSON exporter pointed at base URL /api/v1/otlp; protobuf and gRPC are not accepted.
How it works
Start with durable caller IDs
Create a running trace with an external trace ID and any initial events, each with its source event ID. Store the returned workspace trace UUID for later event and read routes.
Append bounded event batches
POST up to 50 complete event objects to /events or its /spans/batch alias. The response reports only how many new rows were inserted.
Finalize and read redacted detail
Repeat create with the external identity and a terminal status, then GET the trace with traces:read. Detail includes redacted payloads, the first 500 ordered events, redaction_applied, and an optional failure group. When events_next_cursor is set, pass it as cursor to GET /api/v1/traces/:trace_id/events for the rest.
Examples
These examples are illustrative. Replace the sample values with your own.
Keep credentials in environment variables or a secret manager, never in client-side code.
Create, append, and finalize a native trace
The external and source event IDs make every stage safe to repeat unchanged.
Create and finalize each return HTTP 201 with the trace summary. Event append returns HTTP 200 with {"inserted": 1}; unchanged source-event retries insert zero new events.
Troubleshooting
A retry duplicated events
The trace-level external ID prevents a second trace row, not repeated event rows. Give every retryable event a stable source_event_id and do not resend initial events during the separate finalization request.
A write returns 413
Native event arrays exceed 50, or the OTLP body/span limits were crossed. Split native events into bounded batches; split OTLP exports below 512 spans and 8 MiB compressed and decoded.
A new trace returns 429
The traces_per_month quota or rate limit was reached. Existing external trace IDs can still be retried at the OTLP quota cap, but a new logical trace needs quota headroom.
Search or failure grouping lags the write
The trace, events, and projection jobs committed, but indexing, grouping, realtime notifications, and webhook staging are asynchronous. Read the trace directly before treating a short search delay as data loss.
Endpoint reference
Ready to integrate? Open an endpoint for its required permissions, request fields,
response format, and status codes. The examples above explain the workflow;
these details define each individual request.
POST
/api/v1/otlp/v1/traces
API key / access token
Ingest OTLP JSON traces
traces:write
Description
OTLP/HTTP JSON trace receiver. Set the exporter base URL to /api/v1/otlp; exporters append /v1/traces. Requires traces:write machine credentials; protobuf and gRPC are not served by this route. Supports identity and gzip encoding with an 8 MiB encoded/decoded body limit and at most 512 spans per request. Invalid spans produce HTTP 200 partialSuccess; rejectedSpans is a decimal string. A quota failure rolls back the whole batch with 429 and Retry-After; existing trace IDs can retry at the quota cap. Stored external_trace_id is otlp:<traceId>, and source_event_id is the original spanId. Trace/span/parent IDs and nanosecond timestamps are retained in metadata; duration_ms is derived without claiming trace completion. Trace state stays running because one export does not establish completeness; replayable is false. Only selected resource/GenAI attributes survive. Span events, links, bodies and free-form status messages are dropped. Names and stored metadata use credential redaction. This route cannot write engagement-managed traces. All error responses use OTLP Status JSON {code, message}.
Auth
API key / access token
Required Scopes
traces:write
Request Body
Field
Type
Required
Description
resourceSpans
array<object>
No
OTLP resource spans JSON object
Responses
200
Accepted; partialSuccess is returned when spans were rejected
Invalid OTLP JSON, container shape, or malformed gzip
Response Body
Field
Type
Required
Description
code
integer
Yes
OTLP Status code
message
string
Yes
Safe error message
401
Invalid or missing token
Response Body
Field
Type
Required
Description
code
integer
Yes
OTLP Status code
message
string
Yes
Safe error message
403
Missing traces:write scope or an engagement-managed trace
Response Body
Field
Type
Required
Description
code
integer
Yes
OTLP Status code
message
string
Yes
Safe error message
413
Body exceeds 8 MiB or batch exceeds 512 spans
Response Body
Field
Type
Required
Description
code
integer
Yes
OTLP Status code
message
string
Yes
Safe error message
415
Unsupported content type or encoding; use application/json with identity or gzip
Response Body
Field
Type
Required
Description
code
integer
Yes
OTLP Status code
message
string
Yes
Safe error message
429
Quota or rate limit exceeded; Retry-After is returned
Response Body
Field
Type
Required
Description
code
integer
Yes
OTLP Status code
message
string
Yes
Safe error message
503
Quota state unavailable; retry with backoff
Response Body
Field
Type
Required
Description
code
integer
Yes
OTLP Status code
message
string
Yes
Safe error message
POST
/api/v1/traces
API key / access token
Create a new trace
traces:write
Description
Ingests a trace for observability. A successful response confirms the trace, events, and durable projection jobs committed; search indexing, failure grouping, realtime notifications, and webhook staging run asynchronously and are eventually consistent. When `external_trace_id` is set, ingest is idempotent on `(workspace_id, external_trace_id)`: a retry finds the existing row instead of inserting a duplicate. The same request may include up to 50 event objects. Find-or-create, event insertion, and optional finalization commit together; validation failure leaves the trace and events unchanged. A non-`running` `status` finalizes the trace. To finish a running trace, POST again with the same `external_trace_id`, its name, and status `ok`, `error`, or `partial`; omit `events` in that second request. Repeated events append again unless each has a stable `source_event_id`. An identical completed-state retry keeps its end time and finalization identity; fresh late events still append. Payload or timing corrections within the same terminal identity emit trace.updated without recounting failures. A changed status or explicit failure signature is a new finalization transition. Workspace, service-agent, and workflow bindings cannot be changed by retries. A write that would start a new trace is refused with 429 `quota-exceeded` when the workspace `traces_per_month` quota is exhausted; a request naming an existing `external_trace_id` can still append events and finalize it.
Auth
API key / access token
Required Scopes
traces:write
Request Body
Field
Type
Required
Description
name
string
Yes
Human-readable trace name
status
string (running, ok, error, partial)
No
Initial status. Values other than running finalize the trace after ingest.
events
array<object>
No
Optional array of at most 50 events. Each object requires event_type and name.
events[].name
string
Yes
Human-readable event name (required)
events[].status
string (ok, error, pending)
No
Event status
events[].metadata
object
No
Optional event metadata
events[].started_at
datetime (ISO 8601)
No
Event start time
events[].error_payload
object
No
Optional error payload
events[].event_type
string
Yes
Event type label (required)
events[].input_payload
object
No
Optional input payload
events[].output_payload
object
No
Optional output payload
events[].cost
number
No
Cost attributed to the event
events[].duration_ms
integer
No
Duration in milliseconds
events[].ended_at
datetime (ISO 8601)
No
Event end time
events[].latency_ms
integer
No
Latency in milliseconds
events[].parent_event_id
string
No
Parent event UUID when nested
events[].source_event_id
string
No
Caller-supplied id; unique per trace and skipped on replay
events[].span_kind
string
No
Span kind (defaults to step)
metadata
object
No
Additional trace metadata
summary
string
No
Optional operator summary
environment
string
No
Environment label (defaults to production)
service_agent_id
string
No
Optional service-agent UUID that owns this execution. An agent's credential may only name its own agent; the workspace account key may name any agent of the workspace
error_payload
object
No
Optional error payload
workflow_target_id
string
No
Optional workflow target UUID
input_payload
object
No
Optional input payload
output_payload
object
No
Optional output payload
external_trace_id
string
No
Caller correlation id. Combined with workspace_id this is the ingest idempotency key.
trace_type
string (runtime, eval, replay)
No
Type of trace
workflow_version_id
string
No
Optional workflow version UUID
Responses
201
Trace created or the existing idempotent row after find-or-start
Returns: TraceSummary
Response Body
Field
Type
Required
Description
id
string
Yes
Trace UUID
name
string
Yes
Human-readable trace name
status
string (running, ok, error, partial)
Yes
Trace execution status
started_at
datetime (ISO 8601)
Yes
Trace start time
request_id
string
No
Internal request identifier
inserted_at
datetime (ISO 8601)
Yes
Record insertion timestamp
updated_at
datetime (ISO 8601)
Yes
Record update timestamp
environment
string
No
Exact environment label captured at ingest
service_agent
object
No
Associated service agent summary
service_agent.id
string
Yes
Service agent UUID
service_agent.name
string
Yes
Service agent name
workflow_target
object
No
Associated workflow target summary
workflow_target.id
string
Yes
Workflow target UUID
workflow_target.name
string
Yes
Workflow target name
duration_ms
integer
No
Trace duration in milliseconds
ended_at
datetime (ISO 8601)
No
Trace end time
external_trace_id
string
No
External trace identifier from the caller
last_activity_at
datetime (ISO 8601)
No
Last observed activity time
total_cost
string
No
Accumulated cost across the trace, as a decimal string
Validation error on the trace or an included event
429
Plan quota exceeded. Problem type https://robotscenter.net/problems/quota-exceeded with extra.resource traces_per_month.
POST
/api/v1/traces/:trace_id/events
API key / access token
Add events to a trace
traces:write
Description
Atomically appends up to 50 event objects to an existing trace. Each event requires event_type and name. Optional source_event_id is unique per trace and makes a retry a no-op for that event.
Auth
API key / access token
Required Scopes
traces:write
Request Body
Field
Type
Required
Description
events
array<object>
Yes
Up to 50 event objects to append. Each requires event_type and name.
events[].name
string
Yes
Human-readable event name (required)
events[].status
string (ok, error, pending)
No
Event status
events[].metadata
object
No
Optional event metadata
events[].started_at
datetime (ISO 8601)
No
Event start time
events[].error_payload
object
No
Optional error payload
events[].event_type
string
Yes
Event type label (required)
events[].input_payload
object
No
Optional input payload
events[].output_payload
object
No
Optional output payload
events[].cost
number
No
Cost attributed to the event
events[].duration_ms
integer
No
Duration in milliseconds
events[].ended_at
datetime (ISO 8601)
No
Event end time
events[].latency_ms
integer
No
Latency in milliseconds
events[].parent_event_id
string
No
Parent event UUID when nested
events[].source_event_id
string
No
Caller-supplied id; unique per trace and skipped on replay
events[].span_kind
string
No
Span kind (defaults to step)
Parameters
Name
In
Type
Required
Description
trace_id
path
string
Yes
Trace UUID
Responses
200
Events added
Response Body
Field
Type
Required
Description
inserted
integer
Yes
Number of events written by this request
Example
{
"inserted": 2
}
401
Invalid or missing token
403
Missing required scope
404
Trace not found
413
More than 50 events
422
An event is missing event_type or name, or failed validation
POST
/api/v1/traces/:trace_id/spans/batch
API key / access token
Add span events to a trace (batch alias)
traces:write
Description
Backwards-compatible batch ingestion alias for trace events. Same contract as POST /api/v1/traces/:trace_id/events: at most 50 event objects, each requiring event_type and name, and the response is {inserted: count}.
Auth
API key / access token
Required Scopes
traces:write
Request Body
Field
Type
Required
Description
events
array<object>
Yes
Up to 50 event objects to append. Each requires event_type and name.
events[].name
string
Yes
Human-readable event name (required)
events[].status
string (ok, error, pending)
No
Event status
events[].metadata
object
No
Optional event metadata
events[].started_at
datetime (ISO 8601)
No
Event start time
events[].error_payload
object
No
Optional error payload
events[].event_type
string
Yes
Event type label (required)
events[].input_payload
object
No
Optional input payload
events[].output_payload
object
No
Optional output payload
events[].cost
number
No
Cost attributed to the event
events[].duration_ms
integer
No
Duration in milliseconds
events[].ended_at
datetime (ISO 8601)
No
Event end time
events[].latency_ms
integer
No
Latency in milliseconds
events[].parent_event_id
string
No
Parent event UUID when nested
events[].source_event_id
string
No
Caller-supplied id; unique per trace and skipped on replay
events[].span_kind
string
No
Span kind (defaults to step)
Parameters
Name
In
Type
Required
Description
trace_id
path
string
Yes
Trace UUID
Responses
200
Events added
Response Body
Field
Type
Required
Description
inserted
integer
Yes
Number of events written by this request
Example
{
"inserted": 2
}
401
Invalid or missing token
403
Missing required scope
404
Trace not found
413
More than 50 events
422
An event is missing event_type or name, or failed validation
GET
/api/v1/traces/:id
API key / access token
Get a trace (machine)
traces:read
Description
Returns the detail payload for a workspace trace. Requires traces:read. The body is redacted input/output/error/metadata, redaction_applied, events (including source_event_id), and optional failure_group. This is not a TraceSummary. Machine APIAuth only: this is not a partner route, and an mpk_ partner key never authenticates here; marketplace reconciliation uses the partner engagement APIs.
Auth
API key / access token
Required Scopes
traces:read
Parameters
Name
In
Type
Required
Description
id
path
string
Yes
Trace UUID
Responses
200
Trace detail
Returns: Trace
Response Body
Field
Type
Required
Description
id
string
Yes
Trace UUID
name
string
Yes
Human-readable trace name
status
string (running, ok, error, partial)
Yes
Trace execution status
events
array<object>
Yes
Ordered event summaries for the trace timeline
events[].id
string
Yes
Trace event UUID
events[].name
string
Yes
Human-readable event name
events[].status
string (ok, error, pending)
Yes
Event status
events[].origin
string (partner, runtime)
No
Credential that wrote a marketplace engagement event: partner (the marketplace's mpk_ key) or runtime (an engagement token, whose source_event_id is stored under runtime:). Null for first-party events.
events[].started_at
datetime (ISO 8601)
No
Event start time
events[].inserted_at
datetime (ISO 8601)
Yes
Record insertion timestamp
events[].sequence
integer
Yes
Sequence number within the trace
events[].event_type
string
Yes
Event type label
events[].cost
string
No
Cost attributed to the event, as a decimal string
events[].duration_ms
integer
No
Duration in milliseconds
events[].ended_at
datetime (ISO 8601)
No
Event end time
events[].latency_ms
integer
No
Latency in milliseconds
events[].parent_event_id
string
No
Parent event UUID when nested
events[].source_event_id
string
No
Caller-supplied source event id when the ingest is idempotent
metadata
object
Yes
Read-time redacted metadata payload
started_at
datetime (ISO 8601)
Yes
Trace start time
request_id
string
No
Internal request identifier
inserted_at
datetime (ISO 8601)
Yes
Record insertion timestamp
updated_at
datetime (ISO 8601)
Yes
Record update timestamp
summary
string
No
Operator-facing summary for the trace
environment
string
No
Exact environment label captured at ingest
service_agent
object
No
Associated service agent summary
service_agent.id
string
Yes
Service agent UUID
service_agent.name
string
Yes
Service agent name
error_payload
object
Yes
Read-time redacted error payload
failure_group
object
No
Failure group linked by matching signature
failure_group.id
string
Yes
Failure group UUID
failure_group.status
string (open, acknowledged, resolved)
Yes
Failure group status
failure_group.title
string
Yes
Failure group title
failure_group.severity
string (low, medium, high, critical)
Yes
Failure group severity
workflow_target
object
No
Associated workflow target summary
workflow_target.id
string
Yes
Workflow target UUID
workflow_target.name
string
Yes
Workflow target name
events_next_cursor
string
No
Set when the trace has more events than the detail embeds (the first 500): pass it as cursor to GET /api/v1/traces/:trace_id/events for the rest. Null when events holds every event.
failure_signature
string
No
Failure signature used for grouping
input_payload
object
Yes
Read-time redacted input payload
output_payload
object
Yes
Read-time redacted output payload
redaction_applied
boolean
Yes
True when any operator-visible payload field was masked
replayable
boolean
Yes
Whether the trace can be replayed
duration_ms
integer
No
Trace duration in milliseconds
ended_at
datetime (ISO 8601)
No
Trace end time
external_trace_id
string
No
External trace identifier from the caller
last_activity_at
datetime (ISO 8601)
No
Last observed activity time
total_cost
string
No
Accumulated cost across the trace, as a decimal string
/api/v1/traces/:trace_id/events
API key / access token
List a trace's events (machine)
traces:read
Description
Lists one page of a trace's events, ordered by sequence. Requires traces:read. Nested filter[status] (ok | error | pending) and filter[event_type] are applied. Each event includes source_event_id. Pass next_cursor back as cursor to read the next page; it is null on the last page. Machine APIAuth only: an mpk_ partner key never authenticates here.
Auth
API key / access token
Required Scopes
traces:read
Parameters
Name
In
Type
Required
Description
trace_id
path
string
Yes
Trace UUID
filter[status]
query
string
No
Filter by event status
filter[event_type]
query
string
No
Exact-match event type filter
limit
query
integer
No
Maximum events to return (default 100, maximum 500)
cursor
query
string
No
Opaque next_cursor from the previous page
Responses
200
Page of trace events ordered by sequence, with next_cursor
Returns: TraceEventList
Response Body
Field
Type
Required
Description
next_cursor
string
No
Opaque cursor for the next page; null on the last page
trace_events
array<object>
Yes
List of trace events
trace_events[].id
string
Yes
Trace event UUID
trace_events[].name
string
Yes
Human-readable event name
trace_events[].status
string (ok, error, pending)
Yes
Event status
trace_events[].origin
string (partner, runtime)
No
Credential that wrote a marketplace engagement event: partner (the marketplace's mpk_ key) or runtime (an engagement token, whose source_event_id is stored under runtime:). Null for first-party events.
trace_events[].started_at
datetime (ISO 8601)
No
Event start time
trace_events[].inserted_at
datetime (ISO 8601)
Yes
Record insertion timestamp
trace_events[].sequence
integer
Yes
Sequence number within the trace
trace_events[].event_type
string
Yes
Event type label
trace_events[].cost
string
No
Cost attributed to the event, as a decimal string
trace_events[].duration_ms
integer
No
Duration in milliseconds
trace_events[].ended_at
datetime (ISO 8601)
No
Event end time
trace_events[].latency_ms
integer
No
Latency in milliseconds
trace_events[].parent_event_id
string
No
Parent event UUID when nested
trace_events[].source_event_id
string
No
Caller-supplied source event id when the ingest is idempotent