{
  "schemaVersion": "1.0",
  "title": "Operate and recover the Chatwoot connector",
  "description": "Use this runbook to locate a Chatwoot connector failure before changing state. It starts with process, readiness, lease, storage, and provider evidence, then moves to Action, webhook, and central Event delivery.",
  "canonical": "https://getaip.org/docs/connectors/chatwoot/operations/health-observability-and-recovery",
  "route": "/docs/connectors/chatwoot/operations/health-observability-and-recovery",
  "source": "docs/connectors/chatwoot/operations/health-observability-and-recovery.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/chatwoot/operations/health-observability-and-recovery.md",
    "txt": "/docs/download/connectors/chatwoot/operations/health-observability-and-recovery.txt",
    "json": "/docs/download/connectors/chatwoot/operations/health-observability-and-recovery.json",
    "pdf": "/docs/download/connectors/chatwoot/operations/health-observability-and-recovery.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Operate and recover the Chatwoot connector\ndescription: >-\n  Diagnose health, readiness, leases, provider calls, webhooks, and durable\n  Event delivery without repeating uncertain Chatwoot effects\nkind: runbook\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: chatwoot\naliases:\n  - Chatwoot connector health\n  - Chatwoot connector recovery\n---\n\n# Operate and recover the Chatwoot connector\n\nUse this runbook to locate a Chatwoot connector failure before changing state.\nIt starts with process, readiness, lease, storage, and provider evidence, then\nmoves to Action, webhook, and central Event delivery.\n\nWhen a provider mutation has an uncertain outcome, stop new related mutations.\nRead durable Action and provider state before any retry or replacement Action.\n\n## Signal order\n\nRead signals in this order:\n\n| Order | Surface | Primary question answered |\n|---:|---|---|\n| 1 | `GET /health` | Is the host HTTP process responding |\n| 2 | `GET /ready` | Can this replica accept routed Actions now |\n| 3 | `GET /metrics` | Is pressure, rejection, or lease recovery changing |\n| 4 | Connector registry | Is the version, route, replica status, and lease authoritative |\n| 5 | Durable Action state | Was work admitted, dispatched, completed, or left uncertain |\n| 6 | Chatwoot provider reads | What state exists in the configured account |\n| 7 | Webhook response and local Event state | Was inbound work suppressed, persisted, or queued |\n| 8 | Central Event ingress | Was a locally durable Event acknowledged upstream |\n\nDo not use a later signal to overwrite an earlier fact. A live process can be\nnot ready, and a ready host cannot establish the outcome of an earlier provider\nmutation.\n\n## Liveness is process-only\n\n`GET /health` always returns a static HTTP `200` body while the host router can\nserve:\n\n```json\n{\n  \"status\": \"ok\",\n  \"protocol\": \"AIP\",\n  \"version\": \"1.0\",\n  \"service\": \"aip-connector-host\"\n}\n```\n\nThis route does not probe PostgreSQL, Chatwoot, the lifecycle lease, gateway\ntrust, admission, webhook replay state, or central Event delivery.\n\nUse it for process and listener liveness only.\n\n## Readiness is a live composite probe\n\n`GET /ready` performs a bounded durable-storage check and the connector health\nprobe. It also reads the local drain flag and current lease expiry.\n\nHTTP `200` requires all four conditions:\n\n- the replica is not draining;\n- its lifecycle lease is still valid;\n- durable runtime storage is readable and writable;\n- authenticated Chatwoot account health is ready.\n\nAny false condition returns HTTP `503`. The body has this source-owned shape:\n\n```json\n{\n  \"status\": \"ready\",\n  \"lease_valid\": true,\n  \"draining\": false,\n  \"runtime\": {\n    \"durability\": \"durable_shared\",\n    \"ready\": true,\n    \"detail\": \"durable runtime storage is readable and writable\",\n    \"recovery\": {\n      \"recovered_at\": \"2026-01-01T00:00:00Z\",\n      \"pending_approvals\": 0,\n      \"recoverable_transactions\": 0,\n      \"queued_results\": 0,\n      \"delegation_results\": 0,\n      \"callback_deliveries\": 0\n    }\n  },\n  \"connector\": {\n    \"ready\": true,\n    \"detail\": \"Chatwoot account API is reachable\"\n  }\n}\n```\n\n`status` changes to `not_ready` on failure. Values shown above are an\nillustrative healthy response; inspect the returned values rather than matching\nthe example timestamp or counts.\n\nThe `runtime.recovery` object is a bounded startup-recovery summary. It is not\na current queue, approval, transaction, callback, webhook, or outbox backlog.\n\n## Chatwoot product health\n\nThe connector health probe performs:\n\n```text\nGET /api/v1/accounts/{configured_account_id}\napi_access_token: <configured token>\n```\n\nThe common host bounds this probe by the configured health timeout, which\ndefaults to 2,000 ms.\n\nA successful result establishes provider reachability and token acceptance for\nthat account route at that moment. It does not exercise:\n\n- all 153 capabilities;\n- approval or idempotency;\n- provider mutation payloads;\n- webhook authentication;\n- message delivery;\n- central Event publication.\n\n## Host metrics\n\n`GET /metrics` renders all eight Prometheus families on every scrape:\n\n| Metric | Type | Operational meaning |\n|---|---|---|\n| `aip_connector_host_ready` | Gauge | Cached connector and storage flags combined with live lease and drain state |\n| `aip_connector_host_storage_ready` | Gauge | Last stored durable-storage probe result |\n| `aip_connector_host_in_flight` | Gauge | Actions currently executing in this process |\n| `aip_connector_host_requests_total` | Counter | Native message requests received |\n| `aip_connector_host_rejected_total` | Counter | Instrumented validation, trust, route, readiness, capacity, credential, approval, or gateway rejections |\n| `aip_connector_host_heartbeat_failures_total` | Counter | Failed lease renewal or recovery attempts |\n| `aip_connector_host_lease_recovery_attempts_total` | Counter | Re-registration attempts after lease expiry |\n| `aip_connector_host_lease_recoveries_total` | Counter | Re-registrations that applied a new lease |\n\nThe metric families have no application labels and reset when the process\nrestarts. Use bounded scrape-target metadata to identify the replica, version,\nzone, and connector type.\n\nThe ready gauge does not run a provider or storage probe. `/ready` and the\nheartbeat loop update those cached flags. Read the readiness body when the\ncause matters.\n\nDo not calculate success as requests minus rejections. Response signing,\nprocess errors, durable Action state, and provider outcomes are not represented\nby that subtraction.\n\n## Source-visible observability gaps\n\nThe reviewed host exposes no public Chatwoot-specific metric for:\n\n- provider request latency or status distribution;\n- response-size rejection count;\n- webhook authentication or loop-suppression count;\n- replay-state entries or CAS contention;\n- local Event append count;\n- pending central Event outbox batches;\n- provider business objects or delivery state.\n\n`ConnectorHostEventPublisher::pending_batches` exists as an internal bounded\nmethod, but the Chatwoot HTTP host does not publish it as a route or metric.\n\nUse the webhook response's `central_delivery` value as per-delivery evidence.\nUse controlled internal diagnostics for current outbox state. Do not infer an\nempty backlog from a healthy scrape.\n\nCatalogue results can include a provider request ID from `x-request-id` or\n`x-runtime`. Composite results do not expose a provider request ID. Preserve\nthe distinction during incident correlation.\n\n## Heartbeat and lease behavior\n\nThe heartbeat loop periodically:\n\n1. probes Chatwoot account health;\n2. probes durable storage;\n3. records the two readiness flags;\n4. reports in-flight Actions;\n5. renews the lease or recovers an expired lease.\n\nAn unhealthy heartbeat moves the registry replica offline. When storage and\nprovider health recover, the loop can return it to ready under the current\nlease or re-register after expiry.\n\nLease recovery keeps one request ID and registration payload across ambiguous\ncontrol-plane failures. This prevents a lost response from creating a second\nlifecycle mutation. Definitive admission or configuration rejection clears\nthat pending request before a corrected attempt.\n\nIncreasing heartbeat failures with no recovery indicates a control-plane,\nidentity, sequence, admission, or network boundary. It does not by itself prove\na Chatwoot provider failure.\n\n## First response\n\n### 1. Contain\n\nStop new related mutations when:\n\n- provider outcome is uncertain;\n- the wrong account or route may be active;\n- storage consistency is unknown;\n- credentials may be revoked or misbound;\n- webhook or outbox state may be lost;\n- a rollout changed version or manifest identity.\n\nDrain the replica when it must stop receiving new Actions. Keep the process and\nstate available long enough to inspect pinned work and complete bounded\nrecovery.\n\n### 2. Capture bounded evidence\n\nRetain:\n\n- host, version, instance, replica, tenant, and external-account identity;\n- `/health`, `/ready`, and `/metrics` observations with timestamps;\n- registry route, replica status, lease sequence, and expiry;\n- configuration and credential revisions without secret bytes;\n- Action ID, capability, canonical input hash, approval, and key hash;\n- provider status and request ID when exposed;\n- webhook delivery ID and response fields when relevant;\n- final provider reads used for reconciliation.\n\nDo not retain tokens, signing seeds, webhook secrets, or unrestricted customer\npayloads in incident notes.\n\n### 3. Classify the failing boundary\n\n| Observation | Likely boundary | Next evidence |\n|---|---|---|\n| `/health` unavailable | Process, listener, edge, or network | Supervisor state and bounded startup error |\n| `/health` is `200` and `/ready` is `503` | Drain, lease, storage, or provider | Read each readiness field |\n| `runtime.ready` is false | Durable PostgreSQL or storage timeout | Database reachability, migration, and ownership |\n| `connector.ready` is false | Provider origin, token, account, TLS, or provider availability | Bounded health detail and credential revision |\n| `lease_valid` is false | Heartbeat, control plane, sequence, or admission | Registry replica and recovery counters |\n| `draining` is true | Planned or interrupted shutdown | Lifecycle owner and in-flight count |\n| In-flight equals configured capacity | Local saturation | Action duration, route pressure, and capacity rejection |\n| Rejections increase | Input, trust, route, readiness, capacity, credential, approval, or gateway | Signed protocol error code |\n| Webhook returns `401` | HMAC, clock, raw bytes, JSON, or connector mapping | Exact header identity and bounded host diagnostic |\n| Webhook returns `503` | Replay state, Event Log, or durable outbox | Durable-state health and retained delivery ID |\n| Webhook returns `durably_queued` | Central Event delivery pending | Fixed endpoint, signer, and internal outbox diagnostics |\n| A mutation returns connector failure | Provider may already have applied it | Durable Action plus authoritative provider reads |\n\n## Recovery procedures\n\n### Storage not ready\n\n1. Keep the replica out of routing.\n2. Verify the configured database URL file and least-privilege role.\n3. Restore network, database, schema, and storage-health availability.\n4. Preserve Action, replay, Event, and outbox tables.\n5. Restart only when production Runtime Stores remain one coherent durable\n   class.\n6. Inspect startup recovery and require `runtime.ready: true`.\n7. Reconcile every mutation whose dispatch boundary is unknown.\n\nDo not edit leases, replay identities, idempotency records, or outbox entries\nmanually to force readiness.\n\n### Provider health or authentication fails\n\n1. Confirm the fixed provider origin and configured account.\n2. Identify the credential revision without printing the token.\n3. Distinguish TLS, DNS, timeout, `401`, `403`, rate limit, and server failure.\n4. Rotate the owner-only token file through a replacement replica when needed.\n5. Re-run readiness and the safe `account.get` canary.\n6. Reconcile prior mutations before enabling writes.\n\nThe token is read at startup. Replacing file bytes does not prove a running\nprocess adopted them.\n\n### Lease or control-plane recovery fails\n\n1. Compare local instance, replica, version, manifest, artifact, and sequence\n   with the registry.\n2. Verify the fixed control endpoint, trusted DID, TLS, and lifecycle database.\n3. Preserve the host's pending recovery request across ambiguous transport\n   failure.\n4. Correct definitive admission or configuration rejection before another\n   request.\n5. Require a valid ready lease and correct route before admitting traffic.\n\nDo not create a replacement replica with the same replica ID while a live\nlease or pinned Action ownership remains unresolved.\n\n### Capacity is exhausted\n\n`connector_host.capacity` is returned before provider dispatch and is marked\nretryable by the host. Confirm durable Action state before repeating the same\ncorrelated request.\n\nReduce upstream pressure or add an admitted replica before raising\n`AIP_CONNECTOR_HOST_MAX_IN_FLIGHT`. A higher local limit does not increase\nprovider capacity or preserve latency.\n\n### Provider mutation outcome is uncertain\n\nTreat provider-temporary, transport, cancellation, and oversized-response\ninvocation failures as possible provider success.\n\n1. Keep the original Action ID, key, approval, and input.\n2. Read durable Action and provider state.\n3. Correlate provider request ID or AIP Action content attributes when present.\n4. Accept matching state, escalate ambiguity, or create a separately approved\n   corrective intent.\n5. Never replace uncertainty with an uncorrelated retry.\n\nAll Chatwoot connector failures currently report `retryable: false`. Capability\ncontracts separately identify safe GET reads and the status composite.\n\n### Webhook authentication or replay state fails\n\nPreserve the provider delivery ID, timestamp, signature identity, and protected\nraw-body evidence. Do not log the secret or full customer payload.\n\nCorrect clock, proxy byte preservation, signature format, secret revision, or\ndurable replay state. Replay the same provider delivery only under the signed\nfreshness and deterministic Event rules.\n\n### Central Event delivery is pending\n\nA `durably_queued` webhook response means the local outbox owns delivery.\nRepair the fixed central endpoint, TLS policy, host signer, admitted channel,\nor central ingress without asking the provider to create a new delivery.\n\nThe background worker retries with bounded backoff. Preserve the outbox and\nEvent Log through restart. Confirm central acknowledgement and Event identity\nbefore declaring the incident resolved.\n\n## Post-recovery verification\n\nRequire:\n\n- `/health` is reachable;\n- `/ready` returns `200` with a valid lease, no drain, and both probes ready;\n- the registry exposes the intended version, replica, endpoint, and account;\n- host-ready and storage-ready gauges agree with the latest readiness probe;\n- recovery counters stop increasing unexpectedly;\n- the safe account-read canary uses the intended route;\n- every uncertain mutation has an explicit provider-state decision;\n- webhook and central Event delivery are reconciled when enabled;\n- no state, secret, or customer data was deleted to obtain recovery.\n\nMonitor through at least one lease renewal and the deployment's normal provider\nobservation window. These checks establish operational recovery, not complete\ncapability qualification.\n\n## Related documentation\n\n- [Deploy the Chatwoot connector](deployment.md)\n- [Chatwoot connector configuration](../reference/configuration.md)\n- [Webhook security, loop prevention, and replay](../reference/webhook-security-loop-prevention-and-replay.md)\n- [Observe and recover](../../../guides/observe-and-recover.md)\n- [Metrics reference](../../../reference/metrics.md)\n",
    "text": "Operate and recover the Chatwoot connector\n\nUse this runbook to locate a Chatwoot connector failure before changing state.\nIt starts with process, readiness, lease, storage, and provider evidence, then\nmoves to Action, webhook, and central Event delivery.\n\nWhen a provider mutation has an uncertain outcome, stop new related mutations.\nRead durable Action and provider state before any retry or replacement Action.\n\nSignal order\n\nRead signals in this order:\n\n| Order | Surface | Primary question answered |\n\n| 1 | GET /health | Is the host HTTP process responding |\n| 2 | GET /ready | Can this replica accept routed Actions now |\n| 3 | GET /metrics | Is pressure, rejection, or lease recovery changing |\n| 4 | Connector registry | Is the version, route, replica status, and lease authoritative |\n| 5 | Durable Action state | Was work admitted, dispatched, completed, or left uncertain |\n| 6 | Chatwoot provider reads | What state exists in the configured account |\n| 7 | Webhook response and local Event state | Was inbound work suppressed, persisted, or queued |\n| 8 | Central Event ingress | Was a locally durable Event acknowledged upstream |\n\nDo not use a later signal to overwrite an earlier fact. A live process can be\nnot ready, and a ready host cannot establish the outcome of an earlier provider\nmutation.\n\nLiveness is process-only\n\nGET /health always returns a static HTTP 200 body while the host router can\nserve:\n\n{\n  \"status\": \"ok\",\n  \"protocol\": \"AIP\",\n  \"version\": \"1.0\",\n  \"service\": \"aip-connector-host\"\n}\n\nThis route does not probe PostgreSQL, Chatwoot, the lifecycle lease, gateway\ntrust, admission, webhook replay state, or central Event delivery.\n\nUse it for process and listener liveness only.\n\nReadiness is a live composite probe\n\nGET /ready performs a bounded durable-storage check and the connector health\nprobe. It also reads the local drain flag and current lease expiry.\n\nHTTP 200 requires all four conditions:\n• the replica is not draining;\n• its lifecycle lease is still valid;\n• durable runtime storage is readable and writable;\n• authenticated Chatwoot account health is ready.\n\nAny false condition returns HTTP 503. The body has this source-owned shape:\n\n{\n  \"status\": \"ready\",\n  \"leasevalid\": true,\n  \"draining\": false,\n  \"runtime\": {\n    \"durability\": \"durableshared\",\n    \"ready\": true,\n    \"detail\": \"durable runtime storage is readable and writable\",\n    \"recovery\": {\n      \"recoveredat\": \"2026-01-01T00:00:00Z\",\n      \"pendingapprovals\": 0,\n      \"recoverabletransactions\": 0,\n      \"queuedresults\": 0,\n      \"delegationresults\": 0,\n      \"callbackdeliveries\": 0\n    }\n  },\n  \"connector\": {\n    \"ready\": true,\n    \"detail\": \"Chatwoot account API is reachable\"\n  }\n}\n\nstatus changes to notready on failure. Values shown above are an\nillustrative healthy response; inspect the returned values rather than matching\nthe example timestamp or counts.\n\nThe runtime.recovery object is a bounded startup-recovery summary. It is not\na current queue, approval, transaction, callback, webhook, or outbox backlog.\n\nChatwoot product health\n\nThe connector health probe performs:\n\nGET /api/v1/accounts/{configuredaccountid}\napiaccesstoken:\n\nThe common host bounds this probe by the configured health timeout, which\ndefaults to 2,000 ms.\n\nA successful result establishes provider reachability and token acceptance for\nthat account route at that moment. It does not exercise:\n• all 153 capabilities;\n• approval or idempotency;\n• provider mutation payloads;\n• webhook authentication;\n• message delivery;\n• central Event publication.\n\nHost metrics\n\nGET /metrics renders all eight Prometheus families on every scrape:\n\n| Metric | Type | Operational meaning |\n\n| aipconnectorhostready | Gauge | Cached connector and storage flags combined with live lease and drain state |\n| aipconnectorhoststorageready | Gauge | Last stored durable-storage probe result |\n| aipconnectorhostinflight | Gauge | Actions currently executing in this process |\n| aipconnectorhostrequeststotal | Counter | Native message requests received |\n| aipconnectorhostrejectedtotal | Counter | Instrumented validation, trust, route, readiness, capacity, credential, approval, or gateway rejections |\n| aipconnectorhostheartbeatfailurestotal | Counter | Failed lease renewal or recovery attempts |\n| aipconnectorhostleaserecoveryattemptstotal | Counter | Re-registration attempts after lease expiry |\n| aipconnectorhostleaserecoveriestotal | Counter | Re-registrations that applied a new lease |\n\nThe metric families have no application labels and reset when the process\nrestarts. Use bounded scrape-target metadata to identify the replica, version,\nzone, and connector type.\n\nThe ready gauge does not run a provider or storage probe. /ready and the\nheartbeat loop update those cached flags. Read the readiness body when the\ncause matters.\n\nDo not calculate success as requests minus rejections. Response signing,\nprocess errors, durable Action state, and provider outcomes are not represented\nby that subtraction.\n\nSource-visible observability gaps\n\nThe reviewed host exposes no public Chatwoot-specific metric for:\n• provider request latency or status distribution;\n• response-size rejection count;\n• webhook authentication or loop-suppression count;\n• replay-state entries or CAS contention;\n• local Event append count;\n• pending central Event outbox batches;\n• provider business objects or delivery state.\n\nConnectorHostEventPublisher::pendingbatches exists as an internal bounded\nmethod, but the Chatwoot HTTP host does not publish it as a route or metric.\n\nUse the webhook response's centraldelivery value as per-delivery evidence.\nUse controlled internal diagnostics for current outbox state. Do not infer an\nempty backlog from a healthy scrape.\n\nCatalogue results can include a provider request ID from x-request-id or\nx-runtime. Composite results do not expose a provider request ID. Preserve\nthe distinction during incident correlation.\n\nHeartbeat and lease behavior\n\nThe heartbeat loop periodically:\n1. probes Chatwoot account health;\n2. probes durable storage;\n3. records the two readiness flags;\n4. reports in-flight Actions;\n5. renews the lease or recovers an expired lease.\n\nAn unhealthy heartbeat moves the registry replica offline. When storage and\nprovider health recover, the loop can return it to ready under the current\nlease or re-register after expiry.\n\nLease recovery keeps one request ID and registration payload across ambiguous\ncontrol-plane failures. This prevents a lost response from creating a second\nlifecycle mutation. Definitive admission or configuration rejection clears\nthat pending request before a corrected attempt.\n\nIncreasing heartbeat failures with no recovery indicates a control-plane,\nidentity, sequence, admission, or network boundary. It does not by itself prove\na Chatwoot provider failure.\n\nFirst response\n1. Contain\n\nStop new related mutations when:\n• provider outcome is uncertain;\n• the wrong account or route may be active;\n• storage consistency is unknown;\n• credentials may be revoked or misbound;\n• webhook or outbox state may be lost;\n• a rollout changed version or manifest identity.\n\nDrain the replica when it must stop receiving new Actions. Keep the process and\nstate available long enough to inspect pinned work and complete bounded\nrecovery.\n2. Capture bounded evidence\n\nRetain:\n• host, version, instance, replica, tenant, and external-account identity;\n• /health, /ready, and /metrics observations with timestamps;\n• registry route, replica status, lease sequence, and expiry;\n• configuration and credential revisions without secret bytes;\n• Action ID, capability, canonical input hash, approval, and key hash;\n• provider status and request ID when exposed;\n• webhook delivery ID and response fields when relevant;\n• final provider reads used for reconciliation.\n\nDo not retain tokens, signing seeds, webhook secrets, or unrestricted customer\npayloads in incident notes.\n3. Classify the failing boundary\n\n| Observation | Likely boundary | Next evidence |\n\n| /health unavailable | Process, listener, edge, or network | Supervisor state and bounded startup error |\n| /health is 200 and /ready is 503 | Drain, lease, storage, or provider | Read each readiness field |\n| runtime.ready is false | Durable PostgreSQL or storage timeout | Database reachability, migration, and ownership |\n| connector.ready is false | Provider origin, token, account, TLS, or provider availability | Bounded health detail and credential revision |\n| leasevalid is false | Heartbeat, control plane, sequence, or admission | Registry replica and recovery counters |\n| draining is true | Planned or interrupted shutdown | Lifecycle owner and in-flight count |\n| In-flight equals configured capacity | Local saturation | Action duration, route pressure, and capacity rejection |\n| Rejections increase | Input, trust, route, readiness, capacity, credential, approval, or gateway | Signed protocol error code |\n| Webhook returns 401 | HMAC, clock, raw bytes, JSON, or connector mapping | Exact header identity and bounded host diagnostic |\n| Webhook returns 503 | Replay state, Event Log, or durable outbox | Durable-state health and retained delivery ID |\n| Webhook returns durablyqueued | Central Event delivery pending | Fixed endpoint, signer, and internal outbox diagnostics |\n| A mutation returns connector failure | Provider may already have applied it | Durable Action plus authoritative provider reads |\n\nRecovery procedures\n\nStorage not ready\n1. Keep the replica out of routing.\n2. Verify the configured database URL file and least-privilege role.\n3. Restore network, database, schema, and storage-health availability.\n4. Preserve Action, replay, Event, and outbox tables.\n5. Restart only when production Runtime Stores remain one coherent durable\n   class.\n6. Inspect startup recovery and require runtime.ready: true.\n7. Reconcile every mutation whose dispatch boundary is unknown.\n\nDo not edit leases, replay identities, idempotency records, or outbox entries\nmanually to force readiness.\n\nProvider health or authentication fails\n1. Confirm the fixed provider origin and configured account.\n2. Identify the credential revision without printing the token.\n3. Distinguish TLS, DNS, timeout, 401, 403, rate limit, and server failure.\n4. Rotate the owner-only token file through a replacement replica when needed.\n5. Re-run readiness and the safe account.get canary.\n6. Reconcile prior mutations before enabling writes.\n\nThe token is read at startup. Replacing file bytes does not prove a running\nprocess adopted them.\n\nLease or control-plane recovery fails\n1. Compare local instance, replica, version, manifest, artifact, and sequence\n   with the registry.\n2. Verify the fixed control endpoint, trusted DID, TLS, and lifecycle database.\n3. Preserve the host's pending recovery request across ambiguous transport\n   failure.\n4. Correct definitive admission or configuration rejection before another\n   request.\n5. Require a valid ready lease and correct route before admitting traffic.\n\nDo not create a replacement replica with the same replica ID while a live\nlease or pinned Action ownership remains unresolved.\n\nCapacity is exhausted\n\nconnectorhost.capacity is returned before provider dispatch and is marked\nretryable by the host. Confirm durable Action state before repeating the same\ncorrelated request.\n\nReduce upstream pressure or add an admitted replica before raising\nAIPCONNECTORHOSTMAXINFLIGHT. A higher local limit does not increase\nprovider capacity or preserve latency.\n\nProvider mutation outcome is uncertain\n\nTreat provider-temporary, transport, cancellation, and oversized-response\ninvocation failures as possible provider success.\n1. Keep the original Action ID, key, approval, and input.\n2. Read durable Action and provider state.\n3. Correlate provider request ID or AIP Action content attributes when present.\n4. Accept matching state, escalate ambiguity, or create a separately approved\n   corrective intent.\n5. Never replace uncertainty with an uncorrelated retry.\n\nAll Chatwoot connector failures currently report retryable: false. Capability\ncontracts separately identify safe GET reads and the status composite.\n\nWebhook authentication or replay state fails\n\nPreserve the provider delivery ID, timestamp, signature identity, and protected\nraw-body evidence. Do not log the secret or full customer payload.\n\nCorrect clock, proxy byte preservation, signature format, secret revision, or\ndurable replay state. Replay the same provider delivery only under the signed\nfreshness and deterministic Event rules.\n\nCentral Event delivery is pending\n\nA durablyqueued webhook response means the local outbox owns delivery.\nRepair the fixed central endpoint, TLS policy, host signer, admitted channel,\nor central ingress without asking the provider to create a new delivery.\n\nThe background worker retries with bounded backoff. Preserve the outbox and\nEvent Log through restart. Confirm central acknowledgement and Event identity\nbefore declaring the incident resolved.\n\nPost-recovery verification\n\nRequire:\n• /health is reachable;\n• /ready returns 200 with a valid lease, no drain, and both probes ready;\n• the registry exposes the intended version, replica, endpoint, and account;\n• host-ready and storage-ready gauges agree with the latest readiness probe;\n• recovery counters stop increasing unexpectedly;\n• the safe account-read canary uses the intended route;\n• every uncertain mutation has an explicit provider-state decision;\n• webhook and central Event delivery are reconciled when enabled;\n• no state, secret, or customer data was deleted to obtain recovery.\n\nMonitor through at least one lease renewal and the deployment's normal provider\nobservation window. These checks establish operational recovery, not complete\ncapability qualification.\n\nRelated documentation\n• Deploy the Chatwoot connector (deployment.md)\n• Chatwoot connector configuration (../reference/configuration.md)\n• Webhook security, loop prevention, and replay (../reference/webhook-security-loop-prevention-and-replay.md)\n• Observe and recover (../../../guides/observe-and-recover.md)\n• Metrics reference (../../../reference/metrics.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "5eee26c8bfb3976d2739c07c1500c979914fb3c5e5bff248adbf65051aaa5096"
  }
}