{
  "schemaVersion": "1.0",
  "title": "Operate the connector host lifecycle",
  "description": "Use this guide to move one standalone connector-host replica through startup, readiness, normal service, graceful drain, offline retention, and recovery. It applies to the common host used by Cal.diy, Hermes Agent, Chatwoot, Dify, CrewAI, a",
  "canonical": "https://getaip.org/docs/guides/connector-host-lifecycle",
  "route": "/docs/guides/connector-host-lifecycle",
  "source": "docs/guides/connector-host-lifecycle.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Deploy and Operate",
  "documentType": "Guide",
  "language": "en",
  "downloads": {
    "md": "/docs/download/guides/connector-host-lifecycle.md",
    "txt": "/docs/download/guides/connector-host-lifecycle.txt",
    "json": "/docs/download/guides/connector-host-lifecycle.json",
    "pdf": "/docs/download/guides/connector-host-lifecycle.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Operate the connector host lifecycle\ndescription: Start, observe, drain, restart, and recover a standalone connector-host replica safely\nkind: how-to\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\n---\n\n# Operate the connector host lifecycle\n\nUse this guide to move one standalone connector-host replica through startup,\nreadiness, normal service, graceful drain, offline retention, and recovery. It\napplies to the common host used by Cal.diy, Hermes Agent, Chatwoot, Dify,\nCrewAI, and Twenty.\n\nThe common bootstrap owns the lifecycle at source revision\n`97be86e9efedf07ecf1783b03800f683f107fb04`. Product hosts supply connector\nconfiguration and authenticated provider ingress, but they all use the same\ndurable recovery, signed control-plane, lease, drain, and shutdown sequence.\n\nThe desired result is not merely a running process. It is a replica whose\nruntime state recovered cleanly, admitted identity matches the registry,\nhealth lease is current, product and storage probes are ready, and shutdown can\nretain the evidence needed for pinned work.\n\n## Lifecycle model\n\nThe registry exposes three replica states: `ready`, `draining`, and `offline`.\nPreparation and recovery happen before registration and are not registry\nstates.\n\n```mermaid\nflowchart LR\n    P[\"Pre-provisioned offline replica\"] --> B[\"Prepare, build, and recover\"]\n    B -->|\"signed register\"| R[\"Ready with bounded lease\"]\n    R -->|\"healthy heartbeat\"| R\n    R -->|\"SIGTERM or Ctrl-C\"| D[\"Draining\"]\n    D -->|\"server stops\"| O[\"Offline\"]\n    R -->|\"health failure or lease expiry\"| O\n    O -->|\"healthy recovery and signed register\"| R\n```\n\nIn text: a pre-provisioned offline replica prepares durable state and\nregisters as ready. Healthy heartbeats renew its lease. A planned signal moves\nit to draining and then offline. Failed health or an expired lease removes it\nfrom new routing. A clean offline or lease-expired replica can register again\nafter durable recovery.\n\n`draining` preserves pinned work while preventing new route assignments.\n`offline` makes the replica unavailable. Neither state deletes durable action,\nreceipt, approval, transaction, callback, or assignment records.\n\n## Keep lifecycle authority narrow\n\nRun the standalone `aip-connector-control-plane` with its restricted lifecycle\ndatabase role. The service exposes only signed `register`, `heartbeat`,\n`drain`, and `offline` operations at `/aip/v1/connector-control`. It must not\nretain connector catalog administrator credentials.\n\nBefore admitting hosts, verify the lifecycle service:\n\n- reports process health at `/health`;\n- reports the expected registry schema, catalog revision, and data-pool state\n  at `/ready`;\n- exposes only approved metrics through the observability boundary;\n- signs responses with the DID pinned by every host;\n- uses TLS outside explicitly enabled loopback development;\n- has no control/admin database pool in its process snapshot.\n\nEach host signs lifecycle requests with its own pre-provisioned replica key.\nThe service compares that DID with the registry record before applying a\ncommand. Requests and responses have a 30-second lifetime, allow at most five\nseconds of future clock skew, and are limited to 64 KiB. Request IDs, payload\ndigests, durable replay claims, and monotonic health revisions make retries\nidempotent and sequence-fenced.\n\nDo not script unsigned writes to replica status columns. That bypasses peer\nidentity, replay, lease-sequence, and immutable configuration checks.\n\n## Prerequisites for one replica\n\nRequire all of the following before starting the process:\n\n- an active admitted connector version with valid artifact evidence;\n- an enabled tenant-owned connector instance;\n- a pre-provisioned replica with the intended endpoint, peer principal, host\n  DID, trust domain, native transport profile, topology, and capacity;\n- the exact manifest and artifact digests represented by the version;\n- instance configuration revision and secret-provider reference;\n- owner-controlled host signing, database, provider, and optional DID files;\n- a durable connector runtime PostgreSQL backend;\n- network trust to the fixed lifecycle endpoint and the pinned lifecycle DID;\n- a process supervisor that sends SIGTERM and allows a sufficient termination\n  grace period;\n- metrics and registry visibility before traffic is enabled.\n\nThe version, instance, and replica are different identities. Do not reuse a\nreplica ID for another endpoint, key, instance, version, topology, or principal.\nCreate and admit a new replica instead.\n\n## Start the host\n\nUse the product host binary and common fields documented in\n[Connector host configuration](../reference/connector-host-configuration.md).\nThe common lifecycle performs these steps in order:\n\n1. validate common and product configuration and bounded file metadata;\n2. open the durable runtime database and install its immutable schema;\n3. perform tenant-scoped connector discovery;\n4. bind the manifest to the deployment trust domain and signing principal;\n5. build the native gateway and authenticated product-owned routes;\n6. recover durable runtime work before registration;\n7. probe durable storage and product connector health;\n8. send a signed registration to the lifecycle service;\n9. apply the returned lease and start heartbeat renewal;\n10. begin serving health, readiness, metrics, manifest, native AIP, and any\n    product-owned ingress routes.\n\nRecovery reports pending approvals, recoverable transactions, completed queued\nactions, completed delegations, and resumed callback deliveries. If queued or\ndelegation recovery contains errors, registration is blocked. Correct or\nreconcile the durable records; do not mark the replica ready manually.\n\nRegistration succeeds only when the running process matches the registry:\n\n| Running claim | Registry owner |\n|---|---|\n| Connector version and type | Admitted version and instance |\n| Manifest and artifact digests | Immutable version and attestation |\n| Tenant, configuration revision, and secret reference | Enabled instance |\n| Endpoint, peer principal, DID, trust domain, topology, and capacity | Pre-provisioned replica |\n| Capability count | Admitted manifest |\n\nThe initial proposal must report `ready`, zero active assignments, health\nrevision zero, and no prior lifecycle request. The control plane assigns the\ncommitted health revision and lease expiry.\n\n## Verify readiness before traffic\n\nAfter registration, check three different views:\n\n1. host `/health` confirms that the process and AIP 1.0 surface respond;\n2. host `/ready` confirms a valid lease, no drain, readable and writable\n   durable storage, and a ready product connector;\n3. registry and tenant-scoped discovery confirm a ready eligible replica under\n   the intended enabled binding.\n\nDo not substitute one view for another. `/health` does not inspect the lease,\nstorage, or connector. Core `aipd` readiness does not require one particular\nconnector replica or binding. Registry readiness does not prove every provider\nscope or mutation.\n\nEnable traffic only after the three views agree and a reviewed non-mutating\ncheck reaches the intended provider boundary.\n\n## Observe heartbeats and leases\n\nThe default host lease lasts 30 seconds and the default heartbeat interval is\n10 seconds. Configuration enforces a heartbeat interval no greater than half\nthe lease lifetime. The first heartbeat is jittered around the configured\ninterval, and missed timer ticks are skipped rather than replayed in a burst.\n\nConfigure the host and lifecycle service with compatible lease expectations;\nthe expiry returned by the trusted lifecycle service is authoritative.\n\nEach heartbeat carries:\n\n- replica ID;\n- previous lease sequence;\n- current local in-flight action count;\n- combined connector and storage readiness.\n\nThe lifecycle service rejects an in-flight count above admitted replica\ncapacity and rejects a stale previous sequence. A healthy heartbeat moves or\nkeeps the replica in `ready` and advances its health revision. An unhealthy\nheartbeat moves it to `offline` and expires its lease, preventing new routing.\n\nMonitor the common host metrics together:\n\n| Metric | Interpretation |\n|---|---|\n| `aip_connector_host_ready` | Combined local lease, drain, storage, and connector state |\n| `aip_connector_host_storage_ready` | Durable runtime probe result |\n| `aip_connector_host_in_flight` | Actions currently executing in this process |\n| `aip_connector_host_requests_total` | Native requests observed by the host |\n| `aip_connector_host_rejected_total` | Requests refused before successful handling |\n| `aip_connector_host_heartbeat_failures_total` | Failed renewal or recovery attempts |\n| `aip_connector_host_lease_recovery_attempts_total` | Expired-lease registration attempts |\n| `aip_connector_host_lease_recoveries_total` | Successful expired-lease recoveries |\n\nA single heartbeat failure is not lease loss. Alert on the relationship between\nfailure growth, last valid lease expiry, `/ready`, fleet-ready replicas, and\navailable capacity.\n\n## Drain a replica cleanly\n\nFor planned maintenance, ensure another ready replica can take new assignments\nor pause the affected tenant capability. Then send SIGTERM through the process\nsupervisor. Ctrl-C follows the same common path for an interactive process.\n\nThe host performs shutdown in this order:\n\n1. set its local draining flag before contacting the lifecycle service;\n2. request a signed `draining` transition with the current lease sequence;\n3. stop heartbeat renewals while draining;\n4. reject new `Action` messages locally and stop provider-event publication;\n5. keep the listener available while the local in-flight action count falls;\n6. after zero in-flight actions or the configured drain interval, request Axum\n   graceful server shutdown;\n7. stop the heartbeat worker;\n8. send the signed `offline` transition;\n9. return success only when server, heartbeat, drain, and offline results are\n   clean.\n\nThe registry `draining` state prevents new assignments but retains existing\nroutes. A local drain remains active even if the remote drain request fails.\nNew actions receive a retryable `connector_host.not_ready` response while the\nprocess remains reachable.\n\n`drain-timeout-ms` bounds the initial wait for the host's in-flight counter. It\ndoes not promise to cancel provider work, terminate every open HTTP connection,\nor make an external mutation reversible. Configure the workload supervisor's\ntermination grace to cover the drain interval, lifecycle calls, server\ngraceful shutdown, and an evidence-based margin. A forced kill changes this to\nthe crash-recovery path.\n\nVerify all of the following before removing the old process or volume:\n\n- registry state is `offline` and its lease is expired;\n- the host process exited without a lifecycle aggregate error;\n- no assignment remains unexplained;\n- durable recovery and provider audit records are retained;\n- replacement fleet capacity and tenant discovery remain correct.\n\n## Replace replicas without identity reuse\n\nFor a rolling deployment, pre-provision and start a new replica with a new ID,\nendpoint, and signing key. Require its readiness and registry identity before\nsignalling the old replica. Drain one failure domain at a time and keep enough\ncapacity for both active work and retry pressure.\n\nDo not change the endpoint or key behind an existing live replica record. A\nregistration using an existing replica ID is accepted only after that record\nis cleanly offline with zero active assignments or after its lease expires.\nImmutable identity and endpoint mismatches remain conflicts after either event.\n\nCredential changes additionally require the binding and secret-material\nprocedure in [Rotate connector credentials](connector-credentials-and-rotation.md).\n\n## Recover from a control-plane outage\n\nWhen heartbeat delivery fails, the host retains its last committed lease until\nthe recorded expiry. It increments the heartbeat-failure counter. After expiry,\n`/ready` returns unavailable and new actions are rejected even if the process\nand provider health remain good.\n\nIf local storage and connector health are ready, the heartbeat worker switches\nfrom renewal to signed re-registration. For an ambiguous transport or\ncontrol-plane response, it reuses the exact request ID and registration bytes;\nthe durable replay guard turns a lost response into an idempotent retry. A\nsuccessful response applies the new sequence and restores readiness.\n\nAn admission or configuration rejection clears the pending request identity so\na later attempt is not confused with a consumed replay. It does not repair the\nunderlying mismatch. Compare version, instance, replica, digest, tenant,\nconfiguration revision, secret reference, capability count, DID, and clock\nbefore restarting anything.\n\nDo not extend a lease in the database, change a replica to `ready`, or weaken\nthe pinned lifecycle DID during an outage.\n\n## Recover after a hard crash\n\nIf the process cannot send drain and offline transitions:\n\n1. stop traffic to the failed endpoint without deleting action records;\n2. keep a distinct healthy replica serving new work where available;\n3. wait for the failed replica lease to expire before reusing its identity;\n4. preserve the same durable runtime database and exact immutable replica\n   identity for recovery;\n5. start the process and let durable recovery run before registration;\n6. require the recovered capacity to be at least the registry's retained active\n   assignment count;\n7. verify the new lease sequence, readiness, and recovery summary;\n8. resume existing actions with their original action IDs and pinned routes;\n9. reconcile any provider effect whose completion is not proved.\n\nRegistration before clean offline or lease expiry is rejected as a competing\ntakeover. Once eligible, the control plane advances the health revision and\npreserves registry-owned active assignments for the same replica. The\nproduct-neutral daemon also expires stale replicas in bounded maintenance\nbatches and releases their active admission reservations; it does not authorize\ncreating replacement actions for ambiguous work.\n\nNever convert an uncertain old action into a fresh action ID merely to obtain a\nnew replica. The original ID, route assignment, idempotency key, receipts, and\nprovider audit record are the evidence boundary.\n\n## Respond to health regression\n\nWhen connector or durable storage health fails:\n\n- confirm host `/ready` is unavailable;\n- confirm the replica becomes ineligible for new routing;\n- preserve another ready replica or pause the binding;\n- identify whether the failure is product, credential, network, storage, or\n  lifecycle related;\n- repair the owning boundary without changing admitted identity;\n- allow automatic re-registration after both local probes recover;\n- verify a non-mutating action before restoring full traffic.\n\nDo not repeatedly restart a host whose durable recovery fails. Recovery occurs\nbefore registration specifically to prevent a damaged queue or delegation\nstate from being advertised as ready.\n\n## Escalation matrix\n\n| Observation | Likely boundary | Safe next action |\n|---|---|---|\n| Process exits before routes open | Bootstrap, storage, discovery, recovery, or registration | Read the typed startup error and reconcile the named boundary. |\n| `/health` succeeds and `/ready` fails | Lease, drain, storage, or connector health | Inspect the structured readiness fields and corresponding metrics. |\n| Heartbeat failures rise but lease remains valid | Lifecycle network, TLS, clock, or service | Restore the signed path before expiry; do not edit registry state. |\n| Lease recovery attempts rise without successes | Persistent admission mismatch or control outage | Compare exact identity and control-plane response category. |\n| Drain begins but process does not exit | In-flight or open connection remains | Preserve evidence, extend supervisor grace if safe, and avoid a blind kill. |\n| Same replica cannot register | Live lease, active clean-offline condition, capacity, or immutable mismatch | Wait for eligibility or create a distinct pre-provisioned replica. |\n| Replica is offline after provider health failure | Fail-closed heartbeat transition | Repair provider health and let the same process re-register. |\n\nFor the complete initial deployment, use\n[Deploy the connector fleet](deploy-connector-fleet.md). Keep lifecycle choices\ninside the production boundaries in\n[Deploy AIP in production](production-deployment.md), and use\n[Profiles and connectors](../concepts/profiles-and-connectors.md) to distinguish\nthe host process from its connector semantics and transport profiles.\n",
    "text": "Operate the connector host lifecycle\n\nUse this guide to move one standalone connector-host replica through startup,\nreadiness, normal service, graceful drain, offline retention, and recovery. It\napplies to the common host used by Cal.diy, Hermes Agent, Chatwoot, Dify,\nCrewAI, and Twenty.\n\nThe common bootstrap owns the lifecycle at source revision\n97be86e9efedf07ecf1783b03800f683f107fb04. Product hosts supply connector\nconfiguration and authenticated provider ingress, but they all use the same\ndurable recovery, signed control-plane, lease, drain, and shutdown sequence.\n\nThe desired result is not merely a running process. It is a replica whose\nruntime state recovered cleanly, admitted identity matches the registry,\nhealth lease is current, product and storage probes are ready, and shutdown can\nretain the evidence needed for pinned work.\n\nLifecycle model\n\nThe registry exposes three replica states: ready, draining, and offline.\nPreparation and recovery happen before registration and are not registry\nstates.\n\nflowchart LR\n    P[\"Pre-provisioned offline replica\"] --> B[\"Prepare, build, and recover\"]\n    B -->|\"signed register\"| R[\"Ready with bounded lease\"]\n    R -->|\"healthy heartbeat\"| R\n    R -->|\"SIGTERM or Ctrl-C\"| D[\"Draining\"]\n    D -->|\"server stops\"| O[\"Offline\"]\n    R -->|\"health failure or lease expiry\"| O\n    O -->|\"healthy recovery and signed register\"| R\n\nIn text: a pre-provisioned offline replica prepares durable state and\nregisters as ready. Healthy heartbeats renew its lease. A planned signal moves\nit to draining and then offline. Failed health or an expired lease removes it\nfrom new routing. A clean offline or lease-expired replica can register again\nafter durable recovery.\n\ndraining preserves pinned work while preventing new route assignments.\noffline makes the replica unavailable. Neither state deletes durable action,\nreceipt, approval, transaction, callback, or assignment records.\n\nKeep lifecycle authority narrow\n\nRun the standalone aip-connector-control-plane with its restricted lifecycle\ndatabase role. The service exposes only signed register, heartbeat,\ndrain, and offline operations at /aip/v1/connector-control. It must not\nretain connector catalog administrator credentials.\n\nBefore admitting hosts, verify the lifecycle service:\n• reports process health at /health;\n• reports the expected registry schema, catalog revision, and data-pool state\n  at /ready;\n• exposes only approved metrics through the observability boundary;\n• signs responses with the DID pinned by every host;\n• uses TLS outside explicitly enabled loopback development;\n• has no control/admin database pool in its process snapshot.\n\nEach host signs lifecycle requests with its own pre-provisioned replica key.\nThe service compares that DID with the registry record before applying a\ncommand. Requests and responses have a 30-second lifetime, allow at most five\nseconds of future clock skew, and are limited to 64 KiB. Request IDs, payload\ndigests, durable replay claims, and monotonic health revisions make retries\nidempotent and sequence-fenced.\n\nDo not script unsigned writes to replica status columns. That bypasses peer\nidentity, replay, lease-sequence, and immutable configuration checks.\n\nPrerequisites for one replica\n\nRequire all of the following before starting the process:\n• an active admitted connector version with valid artifact evidence;\n• an enabled tenant-owned connector instance;\n• a pre-provisioned replica with the intended endpoint, peer principal, host\n  DID, trust domain, native transport profile, topology, and capacity;\n• the exact manifest and artifact digests represented by the version;\n• instance configuration revision and secret-provider reference;\n• owner-controlled host signing, database, provider, and optional DID files;\n• a durable connector runtime PostgreSQL backend;\n• network trust to the fixed lifecycle endpoint and the pinned lifecycle DID;\n• a process supervisor that sends SIGTERM and allows a sufficient termination\n  grace period;\n• metrics and registry visibility before traffic is enabled.\n\nThe version, instance, and replica are different identities. Do not reuse a\nreplica ID for another endpoint, key, instance, version, topology, or principal.\nCreate and admit a new replica instead.\n\nStart the host\n\nUse the product host binary and common fields documented in\nConnector host configuration (../reference/connector-host-configuration.md).\nThe common lifecycle performs these steps in order:\n1. validate common and product configuration and bounded file metadata;\n2. open the durable runtime database and install its immutable schema;\n3. perform tenant-scoped connector discovery;\n4. bind the manifest to the deployment trust domain and signing principal;\n5. build the native gateway and authenticated product-owned routes;\n6. recover durable runtime work before registration;\n7. probe durable storage and product connector health;\n8. send a signed registration to the lifecycle service;\n9. apply the returned lease and start heartbeat renewal;\n10. begin serving health, readiness, metrics, manifest, native AIP, and any\n    product-owned ingress routes.\n\nRecovery reports pending approvals, recoverable transactions, completed queued\nactions, completed delegations, and resumed callback deliveries. If queued or\ndelegation recovery contains errors, registration is blocked. Correct or\nreconcile the durable records; do not mark the replica ready manually.\n\nRegistration succeeds only when the running process matches the registry:\n\n| Running claim | Registry owner |\n\n| Connector version and type | Admitted version and instance |\n| Manifest and artifact digests | Immutable version and attestation |\n| Tenant, configuration revision, and secret reference | Enabled instance |\n| Endpoint, peer principal, DID, trust domain, topology, and capacity | Pre-provisioned replica |\n| Capability count | Admitted manifest |\n\nThe initial proposal must report ready, zero active assignments, health\nrevision zero, and no prior lifecycle request. The control plane assigns the\ncommitted health revision and lease expiry.\n\nVerify readiness before traffic\n\nAfter registration, check three different views:\n1. host /health confirms that the process and AIP 1.0 surface respond;\n2. host /ready confirms a valid lease, no drain, readable and writable\n   durable storage, and a ready product connector;\n3. registry and tenant-scoped discovery confirm a ready eligible replica under\n   the intended enabled binding.\n\nDo not substitute one view for another. /health does not inspect the lease,\nstorage, or connector. Core aipd readiness does not require one particular\nconnector replica or binding. Registry readiness does not prove every provider\nscope or mutation.\n\nEnable traffic only after the three views agree and a reviewed non-mutating\ncheck reaches the intended provider boundary.\n\nObserve heartbeats and leases\n\nThe default host lease lasts 30 seconds and the default heartbeat interval is\n10 seconds. Configuration enforces a heartbeat interval no greater than half\nthe lease lifetime. The first heartbeat is jittered around the configured\ninterval, and missed timer ticks are skipped rather than replayed in a burst.\n\nConfigure the host and lifecycle service with compatible lease expectations;\nthe expiry returned by the trusted lifecycle service is authoritative.\n\nEach heartbeat carries:\n• replica ID;\n• previous lease sequence;\n• current local in-flight action count;\n• combined connector and storage readiness.\n\nThe lifecycle service rejects an in-flight count above admitted replica\ncapacity and rejects a stale previous sequence. A healthy heartbeat moves or\nkeeps the replica in ready and advances its health revision. An unhealthy\nheartbeat moves it to offline and expires its lease, preventing new routing.\n\nMonitor the common host metrics together:\n\n| Metric | Interpretation |\n\n| aipconnectorhostready | Combined local lease, drain, storage, and connector state |\n| aipconnectorhoststorageready | Durable runtime probe result |\n| aipconnectorhostinflight | Actions currently executing in this process |\n| aipconnectorhostrequeststotal | Native requests observed by the host |\n| aipconnectorhostrejectedtotal | Requests refused before successful handling |\n| aipconnectorhostheartbeatfailurestotal | Failed renewal or recovery attempts |\n| aipconnectorhostleaserecoveryattemptstotal | Expired-lease registration attempts |\n| aipconnectorhostleaserecoveriestotal | Successful expired-lease recoveries |\n\nA single heartbeat failure is not lease loss. Alert on the relationship between\nfailure growth, last valid lease expiry, /ready, fleet-ready replicas, and\navailable capacity.\n\nDrain a replica cleanly\n\nFor planned maintenance, ensure another ready replica can take new assignments\nor pause the affected tenant capability. Then send SIGTERM through the process\nsupervisor. Ctrl-C follows the same common path for an interactive process.\n\nThe host performs shutdown in this order:\n1. set its local draining flag before contacting the lifecycle service;\n2. request a signed draining transition with the current lease sequence;\n3. stop heartbeat renewals while draining;\n4. reject new Action messages locally and stop provider-event publication;\n5. keep the listener available while the local in-flight action count falls;\n6. after zero in-flight actions or the configured drain interval, request Axum\n   graceful server shutdown;\n7. stop the heartbeat worker;\n8. send the signed offline transition;\n9. return success only when server, heartbeat, drain, and offline results are\n   clean.\n\nThe registry draining state prevents new assignments but retains existing\nroutes. A local drain remains active even if the remote drain request fails.\nNew actions receive a retryable connectorhost.notready response while the\nprocess remains reachable.\n\ndrain-timeout-ms bounds the initial wait for the host's in-flight counter. It\ndoes not promise to cancel provider work, terminate every open HTTP connection,\nor make an external mutation reversible. Configure the workload supervisor's\ntermination grace to cover the drain interval, lifecycle calls, server\ngraceful shutdown, and an evidence-based margin. A forced kill changes this to\nthe crash-recovery path.\n\nVerify all of the following before removing the old process or volume:\n• registry state is offline and its lease is expired;\n• the host process exited without a lifecycle aggregate error;\n• no assignment remains unexplained;\n• durable recovery and provider audit records are retained;\n• replacement fleet capacity and tenant discovery remain correct.\n\nReplace replicas without identity reuse\n\nFor a rolling deployment, pre-provision and start a new replica with a new ID,\nendpoint, and signing key. Require its readiness and registry identity before\nsignalling the old replica. Drain one failure domain at a time and keep enough\ncapacity for both active work and retry pressure.\n\nDo not change the endpoint or key behind an existing live replica record. A\nregistration using an existing replica ID is accepted only after that record\nis cleanly offline with zero active assignments or after its lease expires.\nImmutable identity and endpoint mismatches remain conflicts after either event.\n\nCredential changes additionally require the binding and secret-material\nprocedure in Rotate connector credentials (connector-credentials-and-rotation.md).\n\nRecover from a control-plane outage\n\nWhen heartbeat delivery fails, the host retains its last committed lease until\nthe recorded expiry. It increments the heartbeat-failure counter. After expiry,\n/ready returns unavailable and new actions are rejected even if the process\nand provider health remain good.\n\nIf local storage and connector health are ready, the heartbeat worker switches\nfrom renewal to signed re-registration. For an ambiguous transport or\ncontrol-plane response, it reuses the exact request ID and registration bytes;\nthe durable replay guard turns a lost response into an idempotent retry. A\nsuccessful response applies the new sequence and restores readiness.\n\nAn admission or configuration rejection clears the pending request identity so\na later attempt is not confused with a consumed replay. It does not repair the\nunderlying mismatch. Compare version, instance, replica, digest, tenant,\nconfiguration revision, secret reference, capability count, DID, and clock\nbefore restarting anything.\n\nDo not extend a lease in the database, change a replica to ready, or weaken\nthe pinned lifecycle DID during an outage.\n\nRecover after a hard crash\n\nIf the process cannot send drain and offline transitions:\n1. stop traffic to the failed endpoint without deleting action records;\n2. keep a distinct healthy replica serving new work where available;\n3. wait for the failed replica lease to expire before reusing its identity;\n4. preserve the same durable runtime database and exact immutable replica\n   identity for recovery;\n5. start the process and let durable recovery run before registration;\n6. require the recovered capacity to be at least the registry's retained active\n   assignment count;\n7. verify the new lease sequence, readiness, and recovery summary;\n8. resume existing actions with their original action IDs and pinned routes;\n9. reconcile any provider effect whose completion is not proved.\n\nRegistration before clean offline or lease expiry is rejected as a competing\ntakeover. Once eligible, the control plane advances the health revision and\npreserves registry-owned active assignments for the same replica. The\nproduct-neutral daemon also expires stale replicas in bounded maintenance\nbatches and releases their active admission reservations; it does not authorize\ncreating replacement actions for ambiguous work.\n\nNever convert an uncertain old action into a fresh action ID merely to obtain a\nnew replica. The original ID, route assignment, idempotency key, receipts, and\nprovider audit record are the evidence boundary.\n\nRespond to health regression\n\nWhen connector or durable storage health fails:\n• confirm host /ready is unavailable;\n• confirm the replica becomes ineligible for new routing;\n• preserve another ready replica or pause the binding;\n• identify whether the failure is product, credential, network, storage, or\n  lifecycle related;\n• repair the owning boundary without changing admitted identity;\n• allow automatic re-registration after both local probes recover;\n• verify a non-mutating action before restoring full traffic.\n\nDo not repeatedly restart a host whose durable recovery fails. Recovery occurs\nbefore registration specifically to prevent a damaged queue or delegation\nstate from being advertised as ready.\n\nEscalation matrix\n\n| Observation | Likely boundary | Safe next action |\n\n| Process exits before routes open | Bootstrap, storage, discovery, recovery, or registration | Read the typed startup error and reconcile the named boundary. |\n| /health succeeds and /ready fails | Lease, drain, storage, or connector health | Inspect the structured readiness fields and corresponding metrics. |\n| Heartbeat failures rise but lease remains valid | Lifecycle network, TLS, clock, or service | Restore the signed path before expiry; do not edit registry state. |\n| Lease recovery attempts rise without successes | Persistent admission mismatch or control outage | Compare exact identity and control-plane response category. |\n| Drain begins but process does not exit | In-flight or open connection remains | Preserve evidence, extend supervisor grace if safe, and avoid a blind kill. |\n| Same replica cannot register | Live lease, active clean-offline condition, capacity, or immutable mismatch | Wait for eligibility or create a distinct pre-provisioned replica. |\n| Replica is offline after provider health failure | Fail-closed heartbeat transition | Repair provider health and let the same process re-register. |\n\nFor the complete initial deployment, use\nDeploy the connector fleet (deploy-connector-fleet.md). Keep lifecycle choices\ninside the production boundaries in\nDeploy AIP in production (production-deployment.md), and use\nProfiles and connectors (../concepts/profiles-and-connectors.md) to distinguish\nthe host process from its connector semantics and transport profiles.\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "f453a352c18e1b5f284d4c46bf03c9b15cdd4a628ca4e0e78f3952d2714c57ab"
  }
}