Platform
Production readiness
Prepare a deployment that becomes ready only when its database, migrations, queues, secrets, authorization boundaries, and outbound protections are safe for real workloads.
On this page
Start here
Use this checklist before serving real agents. Dependencies must be ready, secrets must survive restarts, metrics need a token, migrations must finish, and outbound requests must stay inside network policy.
Before you begin
- A production-like environment with PostgreSQL, Oban queues, and the intended reverse proxy
- A durable secret provider and a stable SECRETS_ENCRYPTION_KEY of at least 32 bytes
Put it into practice
-
Configure durable foundations
Select PostgreSQL or AWS Secrets Manager, set the stable encryption key, configure the metrics bearer token, and restrict trusted proxy addresses and network ingress.
-
Exercise the release path
Deploy a versioned release, run migrations, and require /ready to confirm database access, no pending migrations, and all configured Oban queues. Test a safe rollback before an incident.
-
Verify security boundaries
Test a least-privilege machine credential, workspace and platform roles, outbound URL rejection, and the expected authentication throttles. Confirm /metrics is never public in production.
The release passes readiness only when its dependencies are usable, managed secrets survive a restart, expected traffic is authorized, unsafe outbound targets are refused, and operators have a tested recovery path.
01 Health and observability endpoints
reference| Route | Behavior | Authentication |
|---|---|---|
| GET /health | Liveness JSON with status and timestamp | Public |
| GET /ready | Checks PostgreSQL connectivity, pending Ecto migrations, and every configured Oban queue, including default, traces, evals, alerts, webhooks, billing, fleet, rollups, and replays. Unhealthy returns 503 | Public |
| GET /metrics | Prometheus text export; responses are never cached. Production requires METRICS_BEARER_TOKEN (Authorization: Bearer) and METRICS_AUTH_ENABLED cannot be turned off | Dedicated bearer token required in production |
02 Agent communication runtime
detailsCanonical endpoints
Machine REST is rooted at /api/v1. Phoenix v2 realtime clients mint a 600-second token with POST /api/v1/socket_tokens, then connect at /socket/websocket and join the authenticated agent topic. There is no cacp.one, /v1, or /ws/v1 production fallback.
Ephemeral health state
Service-agent health reports, local history, and health aggregates are node-local ETS data. An application-node restart clears them; connected agents must report again. Do not use this view as a durable audit or monitoring archive.
03 Durable managed secrets
detailsRequired provider
Production requires SECRET_PROVIDER=postgres or SECRET_PROVIDER=aws_secrets_manager. The local ETS provider is allowed only in development and test because a restart destroys its values.
Stable encryption key
SECRETS_ENCRYPTION_KEY must be at least 32 bytes and remain stable across releases and nodes. PostgreSQL values are encrypted at rest; the key also stabilizes credential digests for every provider.
04 Release and migration behavior
detailsVersioned, readiness-gated deploys
deploy.sh builds a versioned release, runs database migrations, switches the active symlink, restarts the service, and waits up to 30 seconds for /ready. A failed restart or readiness check rolls back when that rollback is safe.
Legacy local-secret migration
A still-running legacy local provider can be exported and imported automatically only when moving to PostgreSQL. The encrypted backup is retained. Automatic rollback is intentionally disabled after that import because restarting the old ETS-backed release would lose its values.
06 Outbound request safety
detailsSSRF and redirect controls
Production outbound HTTP requires HTTPS unless a trusted connector explicitly opts into private networking. Loopback, link-local, multicast, documentation, and other reserved ranges are always blocked; DNS results are validated and pinned, and automatic redirects are disabled.
07 Partner and engagement failed-auth throttle
details30 unauthenticated HTTP 401s per client address per 60 seconds
Partner, engagement, and SCIM routes run AuthFailureThrottle before credential lookup. After 30 unauthenticated 401 responses from one client address in 60 seconds the next attempt is shared 429 problem JSON with retry-after. Authenticated 403 responses and other statuses do not count.
Troubleshooting
- /health passes but /ready fails
- Treat the instance as live but unavailable. Check PostgreSQL, pending migrations, and every named Oban queue instead of routing production traffic to it.
- Secrets disappear after restart
- A local ETS provider is not production storage. Move to a supported durable provider and keep the encryption key identical across nodes and releases.
- Health history resets
- Service-agent health samples are intentionally node-local and ephemeral. Use durable traces, audits, and external monitoring for historical evidence.