Platform
Production readiness
Prepare a deployment that becomes ready only when its database, migrations, queues, secrets, authorization boundaries, and outbound protections are safe for real workloads.
On this page
Start here
Use this checklist before serving real agents. Dependencies must be ready, secrets must survive restarts, metrics need a token, migrations must finish, and outbound requests must stay inside network policy.
Before you begin
- A production-like environment with PostgreSQL, Oban queues, and the intended reverse proxy
- A durable secret provider and a stable SECRETS_ENCRYPTION_KEY of at least 32 bytes
Put it into practice
-
Configure durable foundations
Select PostgreSQL or AWS Secrets Manager, set the stable encryption key, configure the metrics bearer token, and restrict trusted proxy addresses and network ingress.
-
Exercise the release path
Deploy a versioned release, run migrations, and require /ready to confirm database access, no pending migrations, and all configured Oban queues. Test a safe rollback before an incident.
-
Verify security boundaries
Test a least-privilege machine credential, workspace and platform roles, outbound URL rejection, and the expected authentication throttles. Confirm /metrics is never public in production.
The release passes readiness only when its dependencies are usable, managed secrets survive a restart, expected traffic is authorized, unsafe outbound targets are refused, and operators have a tested recovery path.
01 Health and observability endpoints
reference| Route | Behavior | Authentication |
|---|---|---|
| GET /health | Liveness JSON with status and timestamp | Public |
| GET /ready | Checks PostgreSQL connectivity, pending Ecto migrations, and every configured Oban queue, including default, traces, evals, alerts, webhooks, billing, fleet, rollups, and replays. Unhealthy returns 503 | Public |
| GET /metrics | Prometheus text export; responses are never cached. Production requires METRICS_BEARER_TOKEN (Authorization: Bearer) and METRICS_AUTH_ENABLED cannot be turned off | Dedicated bearer token required in production |
02 Agent communication runtime
detailsCanonical endpoints
Machine REST is rooted at /api/v1. Phoenix v2 realtime clients mint a 600-second token with POST /api/v1/socket_tokens, then connect at /socket/websocket and join the authenticated agent topic. There is no cacp.one, /v1, or /ws/v1 production fallback.
Ephemeral health state
Service-agent health reports, local history, and health aggregates are node-local ETS data. An application-node restart clears them; connected agents must report again. Do not use this view as a durable audit or monitoring archive.
03 Durable managed secrets
detailsRequired provider
Production requires SECRET_PROVIDER=postgres or SECRET_PROVIDER=aws_secrets_manager. The local ETS provider is allowed only in development and test because a restart destroys its values.
Stable encryption key
SECRETS_ENCRYPTION_KEY must be at least 32 bytes and remain stable across releases and nodes. PostgreSQL values are encrypted at rest; the key also stabilizes credential digests for every provider.
04 Release and migration behavior
detailsVersioned, readiness-gated deploys
deploy.sh builds a versioned release, runs database migrations, switches the active symlink, restarts the service, and waits up to 30 seconds for /ready. A failed restart or readiness check rolls back when that rollback is safe.
Legacy local-secret migration
A still-running legacy local provider can be exported and imported automatically only when moving to PostgreSQL. The encrypted backup is retained. Automatic rollback is intentionally disabled after that import because restarting the old ETS-backed release would lose its values.
06 Custom workspace roles
detailsCreate and assign in workspace administration
An owner or admin opens /app/admin, creates a custom role with a name, unique lowercase slug (letters, digits, hyphen, underscore), and comma-separated permission strings, then assigns it to a workspace user from the member list. The role is workspace-scoped; duplicate slugs fail validation, foreign role IDs are rejected, and repeated assignments do not stack. Enabled custom-role permissions add to the member's built-in permissions and can be used by gateway approval reviewer-role matching.
Built-in route roles remain authoritative
Custom roles do not replace owner, admin, member, or auditor membership checks on workspace-admin or operator routes. Adding a permission string cannot turn a member into a workspace admin, grant platform users.admin, bypass machine credential scopes, or override an auditor-only route restriction. Set the underlying membership role for those duties.
07 Outbound request safety
detailsSSRF and redirect controls
Production outbound HTTP requires HTTPS. Loopback, link-local, metadata (IMDS), multicast, documentation, and other reserved ranges are always blocked. Private RFC 1918 / ULA destinations are refused unless the platform operator lists them in OUTBOUND_PRIVATE_NETWORK_ALLOWLIST (comma-separated CIDRs, validated at boot); plain http is accepted only when every resolved address is inside that allowlist. Reaching private networks is a platform decision, not a tenant one: a connector's allow_private_network setting is ignored and dropped on save. DNS results are validated and pinned to the checked address, and automatic redirects are disabled.
08 Failed-auth throttle on token-authenticated planes
details30 unauthenticated HTTP 401s per client address per 60 seconds
Every token-authenticated API plane runs AuthFailureThrottle before credential lookup: the machine /api/v1 routes (agk_ keys and access tokens), /api/v1/otlp, partner, engagement, and SCIM. Each credential plane keeps its own bucket per client address (IPv6 counted per /64): agent credentials (machine /api/v1 and /api/v1/otlp together), partner keys, engagement tokens, and SCIM tokens. After 30 unauthenticated 401 responses on one plane in 60 seconds the next attempt on that plane is HTTP 429 problem JSON with retry-after; the other planes are unaffected. Valid credentials, an authenticated 403, and a 401 for a credential that verified but was refused for its state (a closed or expired engagement, a revoked token, a suspended partner) do not increment the counter, but a valid credential can still be rejected before credential lookup while its plane's failure bucket for that address is exhausted.
Troubleshooting
- /health passes but /ready fails
- Treat the instance as live but unavailable. Check PostgreSQL, pending migrations, and every named Oban queue instead of routing production traffic to it.
- Secrets disappear after restart
- A local ETS provider is not production storage. Move to a supported durable provider and keep the encryption key identical across nodes and releases.
- Health history resets
- Service-agent health samples are intentionally node-local and ephemeral. Use durable traces, audits, and external monitoring for historical evidence.