# Production Deployment

This guide describes the boundaries a production `aipd` deployment needs. It
does not prescribe one cloud or orchestrator.

Production readiness is a property of a specific build, configuration,
identity provider, storage backend, connector set, and evidence campaign. A
repository tag alone does not establish it.

## Recommended Topology

```mermaid
flowchart LR
    Client[Clients and MCP hosts] --> Edge[TLS and authenticated edge]
    Edge --> AIP[aipd replicas]
    AIP --> PG[(PostgreSQL durable state)]
    AIP --> NATS[NATS request and event fabric]
    AIP --> ID[Identity and policy services]
    AIP --> Secrets[Credential provider]
    AIP --> Products[Product APIs]
    Products --> Webhooks[Authenticated webhook ingress]
    Webhooks --> AIP
```

Use only the components the deployment needs. The important requirement is that
identity, durable ownership, and provider side effects have explicit boundaries.

## Development versus Production

| Concern | Local development | Production |
|---|---|---|
| Listener | Loopback | Private service network behind TLS |
| Native auth | Literal development bearer or loopback insecure mode | Trusted edge, signer, mTLS, OAuth, or workload identity |
| Storage | File-backed single-host state | PostgreSQL backend with atomic leases and fencing |
| Secrets | Local owner-only files | Secret manager or mounted owner-only files |
| Identity | One development principal | Revisioned trusted identity and authority directories |
| Callbacks | Loopback test target | HTTPS allowlist, signing, SSRF controls, durable retry |
| NATS | Local unauthenticated server | Accounts, credentials, subject ACLs, TLS, queue groups |
| Evidence | Smoke test | Conformance, live provider, restart, fault, and load evidence |

## 1. Freeze the Artifact

Build a reproducible image from a specific commit. Record source digest, image
digest, Rust toolchain, enabled features, schema set, and connector upstream
revisions. Do not qualify one image and deploy another.

## 2. Configure Durable Storage

Use `--postgres-url` for clustered runtime state. The storage backend must
provide atomic enqueue, lease, renew, release, fencing, idempotency reservation,
dead-letter, replay, approval, transaction, callback, event, and receipt
semantics.

File storage configured by `--storage-dir` is durable for one host and useful
for local or isolated deployments. It is not a clustered scheduler.

`aip-storage-postgres` verifies and applies its immutable migration sequence
when it connects. Stage the first upgraded replica so this migration step
finishes before the rollout admits traffic to the new release. Verify recovery
with a real in-flight action, not only a database connection check.

## 3. Establish Native Identity

Choose one authenticated edge:

- trusted Ed25519 envelope signers;
- OAuth or bearer authentication mapped to a server-owned principal;
- mTLS or workload identity at a trusted reverse proxy;
- authenticated NATS account and subject policy.

Load trusted principal, tenant, and credential bindings with
`--trusted-identity-file`. Load approval membership with
`--approval-authority-file`. Both files should be regular, size-bounded,
owner-controlled, revisioned, and mounted read-only.

Do not use `--allow-insecure-development` outside loopback development.

## 4. Protect MCP

Publish protected-resource metadata, validate token audience and issuer, require
scopes, and use RFC 7662 introspection or an equivalent trusted verifier. Set an
explicit browser-origin allowlist when MCP is reachable from browsers.

MCP authentication establishes an AIP principal. Tool arguments cannot override
that principal or its tenant.

## 5. Configure Connectors

For each connector:

- mount credentials through the connector's supported secret boundary;
- bind tenants to provider accounts;
- restrict outbound hosts and webhook prefixes;
- set body, timeout, stream, and concurrency limits;
- enable only capabilities qualified for that deployment;
- make readiness depend on required provider and storage boundaries;
- retain the exact upstream revision and qualification evidence.

## 6. Secure Callbacks and Webhooks

Use HTTPS callback targets, exact host allowlists, payload signatures, bounded
DNS and address resolution, and durable retry. Private or loopback callback
targets should remain disabled unless the deployment explicitly owns that
network path.

Verify provider webhooks on the raw body and claim delivery ids before returning
success.

## 7. Operate Multiple Replicas

Replicas require a shared storage backend and atomic leases. Give each worker a
stable instance id, use bounded lease TTLs and renewal, and fence stale owners.
Configure NATS queue groups when NATS carries request/reply work.

Test:

- worker death after provider acceptance;
- lease expiry and recovery;
- duplicate delivery;
- network partitions;
- callback retry and dead-letter handling;
- database failover;
- provider outcome reconciliation.

## 8. Expose Health Correctly

- `/health` proves the process is alive.
- `/ready` proves the storage backend, every registered connector, runtime
  worker, and required NATS listener can serve traffic.
- `/metrics` exposes operational metrics.

Identity and authority configuration is validated at startup rather than
reported as a dynamic readiness component. The current `aipd` readiness
aggregation treats every registered connector as mandatory. Deploy optional
connector groups separately when one provider outage must not make unrelated
capabilities unready.

## 9. Release Gate

Before production traffic, require:

```sh
cargo run -p xtask -- release-check
```

Also require deployment-specific integration, security, restart, load, and
external-provider qualification. See [Conformance](../reference/conformance.md).

## Rollback

Keep the previous immutable image and schema-compatible database state. A daemon
rollback does not reverse provider side effects; business compensation follows
the capability contract and transaction records.