Production Deployment

This guide describes the boundaries a production aipd deployment needs. It does not prescribe one cloud or orchestrator.

Production readiness is a property of a specific build, configuration, identity provider, storage backend, connector set, and evidence campaign. A repository tag alone does not establish it.

Recommended Topology

flowchart LR
    Client[Clients and MCP hosts] --> Edge[TLS and authenticated edge]
    Edge --> AIP[aipd replicas]
    AIP --> PG[(PostgreSQL durable state)]
    AIP --> NATS[NATS request and event fabric]
    AIP --> ID[Identity and policy services]
    AIP --> Secrets[Credential provider]
    AIP --> Products[Product APIs]
    Products --> Webhooks[Authenticated webhook ingress]
    Webhooks --> AIP

Use only the components the deployment needs. The important requirement is that identity, durable ownership, and provider side effects have explicit boundaries.

Development versus Production

Concern	Local development	Production
Listener	Loopback	Private service network behind TLS
Native auth	Literal development bearer or loopback insecure mode	Trusted edge, signer, mTLS, OAuth, or workload identity
Storage	File-backed single-host state	PostgreSQL backend with atomic leases and fencing
Secrets	Local owner-only files	Secret manager or mounted owner-only files
Identity	One development principal	Revisioned trusted identity and authority directories
Callbacks	Loopback test target	HTTPS allowlist, signing, SSRF controls, durable retry
NATS	Local unauthenticated server	Accounts, credentials, subject ACLs, TLS, queue groups
Evidence	Smoke test	Conformance, live provider, restart, fault, and load evidence

1. Freeze the Artifact

Build a reproducible image from a specific commit. Record source digest, image digest, Rust toolchain, enabled features, schema set, and connector upstream revisions. Do not qualify one image and deploy another.

2. Configure Durable Storage

Use --postgres-url for clustered runtime state. The storage backend must provide atomic enqueue, lease, renew, release, fencing, idempotency reservation, dead-letter, replay, approval, transaction, callback, event, and receipt semantics.

File storage configured by --storage-dir is durable for one host and useful for local or isolated deployments. It is not a clustered scheduler.

aip-storage-postgres verifies and applies its immutable migration sequence when it connects. Stage the first upgraded replica so this migration step finishes before the rollout admits traffic to the new release. Verify recovery with a real in-flight action, not only a database connection check.

3. Establish Native Identity

Choose one authenticated edge:

• trusted Ed25519 envelope signers;
• OAuth or bearer authentication mapped to a server-owned principal;
• mTLS or workload identity at a trusted reverse proxy;
• authenticated NATS account and subject policy.

Load trusted principal, tenant, and credential bindings with --trusted-identity-file. Load approval membership with --approval-authority-file. Both files should be regular, size-bounded, owner-controlled, revisioned, and mounted read-only.

Do not use --allow-insecure-development outside loopback development.

4. Protect MCP

Publish protected-resource metadata, validate token audience and issuer, require scopes, and use RFC 7662 introspection or an equivalent trusted verifier. Set an explicit browser-origin allowlist when MCP is reachable from browsers.

MCP authentication establishes an AIP principal. Tool arguments cannot override that principal or its tenant.

5. Configure Connectors

For each connector:

• mount credentials through the connector’s supported secret boundary;
• bind tenants to provider accounts;
• restrict outbound hosts and webhook prefixes;
• set body, timeout, stream, and concurrency limits;
• enable only capabilities qualified for that deployment;
• make readiness depend on required provider and storage boundaries;
• retain the exact upstream revision and qualification evidence.

6. Secure Callbacks and Webhooks

Use HTTPS callback targets, exact host allowlists, payload signatures, bounded DNS and address resolution, and durable retry. Private or loopback callback targets should remain disabled unless the deployment explicitly owns that network path.

Verify provider webhooks on the raw body and claim delivery ids before returning success.

7. Operate Multiple Replicas

Replicas require a shared storage backend and atomic leases. Give each worker a stable instance id, use bounded lease TTLs and renewal, and fence stale owners. Configure NATS queue groups when NATS carries request/reply work.

Test:

• worker death after provider acceptance;
• lease expiry and recovery;
• duplicate delivery;
• network partitions;
• callback retry and dead-letter handling;
• database failover;
• provider outcome reconciliation.

8. Expose Health Correctly

• /health proves the process is alive.
• /ready proves the storage backend, every registered connector, runtime worker, and required NATS listener can serve traffic.
• /metrics exposes operational metrics.

Identity and authority configuration is validated at startup rather than reported as a dynamic readiness component. The current aipd readiness aggregation treats every registered connector as mandatory. Deploy optional connector groups separately when one provider outage must not make unrelated capabilities unready.

9. Release Gate

Before production traffic, require:

cargo run -p xtask -- release-check

Also require deployment-specific integration, security, restart, load, and external-provider qualification. See Conformance.

Rollback

Keep the previous immutable image and schema-compatible database state. A daemon rollback does not reverse provider side effects; business compensation follows the capability contract and transaction records.