{
  "schemaVersion": "1.0",
  "title": "Troubleshoot the Chatwoot connector",
  "description": "Use this runbook to identify the boundary that failed and choose the narrowest safe next action. It covers process startup, connector-host readiness, signed Action handling, Chatwoot provider calls, webhook ingress, and durable Event delive",
  "canonical": "https://getaip.org/docs/connectors/chatwoot/troubleshooting",
  "route": "/docs/connectors/chatwoot/troubleshooting",
  "source": "docs/connectors/chatwoot/troubleshooting/README.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/chatwoot/troubleshooting.md",
    "txt": "/docs/download/connectors/chatwoot/troubleshooting.txt",
    "json": "/docs/download/connectors/chatwoot/troubleshooting.json",
    "pdf": "/docs/download/connectors/chatwoot/troubleshooting.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Troubleshoot the Chatwoot connector\ndescription: >-\n  Classify Chatwoot startup, readiness, provider, mutation, webhook, and event\n  delivery failures and choose the first safe recovery action\nkind: runbook\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: chatwoot\n---\n\n# Troubleshoot the Chatwoot connector\n\nUse this runbook to identify the boundary that failed and choose the narrowest\nsafe next action. It covers process startup, connector-host readiness, signed\nAction handling, Chatwoot provider calls, webhook ingress, and durable Event\ndelivery.\n\nPause new mutations whenever provider outcome is uncertain. Preserve the\noriginal Action ID, idempotency key, account ID, route assignment, remote\nstatus, webhook delivery ID, and durable state before changing a deployment.\n\n## Identify the error surface\n\nStart with the surface that produced the evidence. The same HTTP status can\nrepresent different trust and recovery boundaries.\n\n| Surface | Evidence shape | First trust check |\n|---|---|---|\n| Process startup | Error text and non-zero exit | Compare configuration, secret-file ownership, storage, and admission |\n| Pre-envelope host rejection | Unsigned compact JSON | Trust only the controlled TLS path; no valid envelope exists to sign |\n| Connector-host response | Signed AIP error envelope | Verify signature, sender, recipient, and correlation |\n| Durable Action | `ActionResult`, state, and events | Read it even when the original HTTP exchange failed |\n| Chatwoot webhook | Compact `{ \"error\": { \"code\": \"...\" } }` | Correlate the authenticated ingress request and delivery ID |\n| Event delivery | Local Event and outbox state | Separate accepted ingress from central publication |\n\nAutomate on the exact code, category, retry flag, `uncertain_outcome`, and\ndurable state. Message text is diagnostic and is not a stable interface.\n\n## Choose the first check\n\n| Symptom | First check | Safe first action |\n|---|---|---|\n| Capability is absent | Authenticated tenant catalogue and enabled binding | Correct discovery or admission; do not call a host directly |\n| Host returns `401` | Host trust code versus `connector.chatwoot.authentication` | Repair only the failed authentication boundary |\n| Host returns `403` | Route, credential revision, approval, or provider code | Re-resolve the exact authority boundary; do not bypass it |\n| `/health` works and `/ready` fails | Drain, lease, storage, and connector fields | Follow the failing readiness owner |\n| Read returns a provider or transport failure | Published retry support and Action budget | Retry only the same eligible read through policy |\n| Mutation has no definitive provider result | `uncertain_outcome` and existing provider state | Reconcile before another mutation |\n| Same idempotency key is rejected | Existing key record and input hash | Reuse, wait, reconcile, or stop according to its state |\n| Webhook returns `400` or `401` | Exact compact code and retained raw request | Correct that delivery without generating a replacement event |\n| Webhook returns `503` | Replay, Event-log, and outbox storage | Restore storage, then redeliver the same request |\n| Webhook succeeds but central Event is absent | Accepted local Event and outbox batch | Recover publication; do not replay the provider action |\n\n## Resolve startup failures\n\nStartup failures occur before a protocol error can be returned. Compare the\nfirst process error with the configured base URL, account ID, token file,\nresponse limit, allowed-operation set, webhook secret, PostgreSQL state,\ncentral Event endpoint, manifest, and immutable host identity.\n\nCorrect one owning boundary at a time. Require `/ready` and route a safe account\nread through `aipd` after a repair. A direct Chatwoot request does not prove the\nAIP tenant, route, approval, or signing boundary.\n\n## Follow readiness to its owner\n\n`/health` is a process-liveness response. `/ready` succeeds only while the host\nis not draining, its lease is valid, durable storage is ready, and the\nauthenticated Chatwoot account probe succeeds.\n\n| Failing field or observation | Owner | Decision |\n|---|---|---|\n| `draining` | Host lifecycle | Let in-flight work settle and replace or restore the replica |\n| `lease_valid` | Registry heartbeat and fencing | Recover the same admitted identity or replace it through admission |\n| `storage_ready` | Host PostgreSQL state | Restore durable state before accepting Actions or webhooks |\n| `connector_ready` | Chatwoot account probe | Correct account, token, base URL, or provider availability |\n| Ready host is not selected | Binding, route, or central registry | Re-resolve the admitted route; do not edit the assignment |\n| `connector_host.capacity` | Replica-local concurrency | Honor bounded backoff through the original Action policy |\n\nUse the operations runbook for containment, heartbeat recovery, replacement,\nand rollback. Readiness does not prove every Chatwoot operation or webhook\ndelivery path.\n\n## Look up Chatwoot connector codes\n\nThe table covers every explicit `connector.chatwoot.*` code at the reviewed\nrevision. All connector failures set automatic `retryable` to false.\n\n| Code | Meaning | Decision |\n|---|---|---|\n| `connector.chatwoot.authentication` | Chatwoot returned `401` or `403` | Correct account, token, scope, or provider access |\n| `connector.chatwoot.configuration` | URL, client, metadata, or connector configuration is invalid | Correct the named configuration boundary |\n| `connector.chatwoot.invalid_action` | Capability, operation, input, or composite rule is invalid | Correct the Action; do not retry unchanged |\n| `connector.chatwoot.invalid_credential` | Token material is empty or structurally invalid | Rotate the owner-only token file and replace the process |\n| `connector.chatwoot.invalid_webhook` | Webhook signature, timestamp, body, or mapping input is invalid | Preserve the raw delivery and correct authentication or format |\n| `connector.chatwoot.invalid_webhook_delivery` | Typed ingestion payload cannot be decoded | Correct the host-to-connector delivery object |\n| `connector.chatwoot.missing_idempotency_key` | A catalogue mutation has no key | Reissue only as the same reviewed Action with a stable key |\n| `connector.chatwoot.remote_rejected` | Chatwoot returned another non-success status | Correct request or provider state; inspect mutation outcome |\n| `connector.chatwoot.remote_temporary` | Chatwoot returned `429` or `5xx` | An invocation can be uncertain; do not infer retry safety |\n| `connector.chatwoot.transport` | Provider transport failed | An invocation can be uncertain; reconcile before replay |\n| `connector.chatwoot.response_too_large` | Provider response exceeded the configured bound | Review the operation and limit; an invocation can be uncertain |\n| `connector.chatwoot.replay_store` | Webhook replay storage failed | Restore storage and retain the original delivery |\n| `connector.chatwoot.webhook_replay` | The typed replay policy rejected a delivery | Inspect the retained delivery and replay state |\n| `connector.chatwoot.cancelled` | Invocation was cancelled before a result settled | Treat provider outcome as uncertain and read durable state |\n\n`connector.chatwoot.webhook_replay` exists in the typed error mapping, but the\nordinary duplicate webhook path does not construct it. That path reproduces a\ndeterministic Event ID so durable Event storage can deduplicate the delivery.\n\n## Look up connector-host codes\n\nThe host can reject a request before Chatwoot is called. Product-neutral trust,\nroute, credential-revision, approval, replay, and execution codes are defined\nin the global error reference. These compact or immediate host codes often\nidentify the first diagnostic branch.\n\n| Code | HTTP | Trust and recovery decision |\n|---|---:|---|\n| `schema.envelope.invalid` | 400 | Correct the JSON against the Envelope schema; the compact response is unsigned |\n| `envelope.decode.invalid` | 400 | Correct typed Envelope decoding; the compact response is unsigned |\n| `connector_host.not_ready` | 503 | Wait for a ready, leased, non-draining host through normal routing |\n| `connector_host.capacity` | 429 | Honor the 100 ms hint only within Action retry policy |\n| `connector_host.response_signing` | 500 | Isolate the response and read central durable state; authenticity is incomplete |\n\nA signed host code such as `connector_host.authentication`,\n`connector_host.route_mismatch`, or\n`connector_host.credential_revision_denied` identifies a host policy boundary,\nnot a Chatwoot credential result. Keep the original signed envelope when\nescalating.\n\n## Look up webhook HTTP codes\n\nThe standalone ingress route returns a compact error rather than an AIP error\nenvelope.\n\n| Code | HTTP | Decision |\n|---|---:|---|\n| `webhook.missing_header` | 400 | Restore the delivery, timestamp, and signature headers |\n| `webhook.invalid_timestamp` | 400 | Supply a canonical signed decimal timestamp |\n| `webhook.invalid_utf8` | 400 | Preserve exact bytes and correct the sender or proxy encoding |\n| `webhook.authentication_failed` | 401 | Verify secret selection, timestamp window, HMAC, raw body, and JSON delivery |\n| `webhook.invalid_delivery` | 400 | Correct the typed delivery after preserving the original evidence |\n| `webhook.invalid_mapping` | 500 | Retain the accepted payload and escalate the impossible Event mapping |\n| `webhook.storage_unavailable` | 503 | Restore replay, Event-log, or outbox storage and redeliver the same request |\n\nAn HTTP `200` can report a locally accepted Event whose central publication is\nstill queued. Recover the durable outbox instead of asking Chatwoot to create a\nnew webhook or repeating the Action that caused it.\n\n## Decide whether to retry\n\nApply these checks in order and stop at the first unmet condition:\n\n1. verify the signed response or read the durable Action result;\n2. classify the operation as a read, mutation, or webhook delivery;\n3. for a mutation, establish provider outcome from authoritative state;\n4. require published implementation retry support;\n5. require `retryable` not to be false;\n6. keep the same Action, input, and idempotency key;\n7. respect attempt, elapsed-time, and backoff limits.\n\nAt this revision, the conversation-status composite and the 62 catalogue reads\npublish retry support. Connector-mapped failures still set `retryable` to\nfalse. The host capacity and not-ready codes can carry `retryable: true`, but\nthat hint cannot override operation safety or the Action budget.\n\n## Treat uncertain outcomes as nonterminal\n\n`remote_temporary`, `transport`, `response_too_large`, and `cancelled` can mark\nan invocation outcome uncertain. Stop new mutations for the affected resource,\nretain the original identifiers, read Chatwoot and AIP state, and decide whether\nthe intended effect is present before any compensation or replacement Action.\n\nThe connector does not publish transaction or reconciliation support. Recovery\ntherefore uses the canonical provider resource reads and the original durable\nAction evidence. A temporary category or missing provider request ID is never\nproof that a mutation did not happen.\n\n## Collect minimum evidence\n\nRetain a bounded incident bundle before changing configuration or state:\n\n- UTC interval, deployment revision, instance ID, replica ID, and account ID;\n- Action ID, correlation ID, capability ID, route assignment, and key reference;\n- exact code, category, retry flag, uncertainty flag, and remote status;\n- readiness snapshot and the eight process-local host metrics;\n- redacted provider resource state for an uncertain mutation;\n- webhook delivery ID, timestamp, response code, Event ID, and outbox state;\n- the first startup or heartbeat failure without credentials or raw payloads.\n\nRoute configuration defects to the connector owner and lease or capacity\ndefects to the fleet operator. Route provider-access defects to the Chatwoot\naccount owner, webhook authentication defects to the ingress owner, and\nuncertain mutations to the Action owner. Escalate cross-tenant output,\nsignature failures, route misbinding, credential exposure, and impossible\nEvent mappings as security or integrity incidents.\n\n## Verify recovery\n\nClose the incident only when the corrected path is proven through the normal\ngateway route, readiness and lease agree, durable Action or Event state is\nterminal, and no uncertain mutation remains unresolved. Record the evidence\nand remove temporary containment explicitly.\n\n## Related documentation\n\n- [Health, observability, and recovery](../operations/health-observability-and-recovery.md)\n- [Configuration](../reference/configuration.md)\n- [Mutation approval and idempotency](../reference/mutation-approval-and-idempotency.md)\n- [Webhook security, loop prevention, and replay](../reference/webhook-security-loop-prevention-and-replay.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n",
    "text": "Troubleshoot the Chatwoot connector\n\nUse this runbook to identify the boundary that failed and choose the narrowest\nsafe next action. It covers process startup, connector-host readiness, signed\nAction handling, Chatwoot provider calls, webhook ingress, and durable Event\ndelivery.\n\nPause new mutations whenever provider outcome is uncertain. Preserve the\noriginal Action ID, idempotency key, account ID, route assignment, remote\nstatus, webhook delivery ID, and durable state before changing a deployment.\n\nIdentify the error surface\n\nStart with the surface that produced the evidence. The same HTTP status can\nrepresent different trust and recovery boundaries.\n\n| Surface | Evidence shape | First trust check |\n\n| Process startup | Error text and non-zero exit | Compare configuration, secret-file ownership, storage, and admission |\n| Pre-envelope host rejection | Unsigned compact JSON | Trust only the controlled TLS path; no valid envelope exists to sign |\n| Connector-host response | Signed AIP error envelope | Verify signature, sender, recipient, and correlation |\n| Durable Action | ActionResult, state, and events | Read it even when the original HTTP exchange failed |\n| Chatwoot webhook | Compact { \"error\": { \"code\": \"...\" } } | Correlate the authenticated ingress request and delivery ID |\n| Event delivery | Local Event and outbox state | Separate accepted ingress from central publication |\n\nAutomate on the exact code, category, retry flag, uncertainoutcome, and\ndurable state. Message text is diagnostic and is not a stable interface.\n\nChoose the first check\n\n| Symptom | First check | Safe first action |\n\n| Capability is absent | Authenticated tenant catalogue and enabled binding | Correct discovery or admission; do not call a host directly |\n| Host returns 401 | Host trust code versus connector.chatwoot.authentication | Repair only the failed authentication boundary |\n| Host returns 403 | Route, credential revision, approval, or provider code | Re-resolve the exact authority boundary; do not bypass it |\n| /health works and /ready fails | Drain, lease, storage, and connector fields | Follow the failing readiness owner |\n| Read returns a provider or transport failure | Published retry support and Action budget | Retry only the same eligible read through policy |\n| Mutation has no definitive provider result | uncertainoutcome and existing provider state | Reconcile before another mutation |\n| Same idempotency key is rejected | Existing key record and input hash | Reuse, wait, reconcile, or stop according to its state |\n| Webhook returns 400 or 401 | Exact compact code and retained raw request | Correct that delivery without generating a replacement event |\n| Webhook returns 503 | Replay, Event-log, and outbox storage | Restore storage, then redeliver the same request |\n| Webhook succeeds but central Event is absent | Accepted local Event and outbox batch | Recover publication; do not replay the provider action |\n\nResolve startup failures\n\nStartup failures occur before a protocol error can be returned. Compare the\nfirst process error with the configured base URL, account ID, token file,\nresponse limit, allowed-operation set, webhook secret, PostgreSQL state,\ncentral Event endpoint, manifest, and immutable host identity.\n\nCorrect one owning boundary at a time. Require /ready and route a safe account\nread through aipd after a repair. A direct Chatwoot request does not prove the\nAIP tenant, route, approval, or signing boundary.\n\nFollow readiness to its owner\n\n/health is a process-liveness response. /ready succeeds only while the host\nis not draining, its lease is valid, durable storage is ready, and the\nauthenticated Chatwoot account probe succeeds.\n\n| Failing field or observation | Owner | Decision |\n\n| draining | Host lifecycle | Let in-flight work settle and replace or restore the replica |\n| leasevalid | Registry heartbeat and fencing | Recover the same admitted identity or replace it through admission |\n| storageready | Host PostgreSQL state | Restore durable state before accepting Actions or webhooks |\n| connectorready | Chatwoot account probe | Correct account, token, base URL, or provider availability |\n| Ready host is not selected | Binding, route, or central registry | Re-resolve the admitted route; do not edit the assignment |\n| connectorhost.capacity | Replica-local concurrency | Honor bounded backoff through the original Action policy |\n\nUse the operations runbook for containment, heartbeat recovery, replacement,\nand rollback. Readiness does not prove every Chatwoot operation or webhook\ndelivery path.\n\nLook up Chatwoot connector codes\n\nThe table covers every explicit connector.chatwoot. code at the reviewed\nrevision. All connector failures set automatic retryable to false.\n\n| Code | Meaning | Decision |\n\n| connector.chatwoot.authentication | Chatwoot returned 401 or 403 | Correct account, token, scope, or provider access |\n| connector.chatwoot.configuration | URL, client, metadata, or connector configuration is invalid | Correct the named configuration boundary |\n| connector.chatwoot.invalidaction | Capability, operation, input, or composite rule is invalid | Correct the Action; do not retry unchanged |\n| connector.chatwoot.invalidcredential | Token material is empty or structurally invalid | Rotate the owner-only token file and replace the process |\n| connector.chatwoot.invalidwebhook | Webhook signature, timestamp, body, or mapping input is invalid | Preserve the raw delivery and correct authentication or format |\n| connector.chatwoot.invalidwebhookdelivery | Typed ingestion payload cannot be decoded | Correct the host-to-connector delivery object |\n| connector.chatwoot.missingidempotencykey | A catalogue mutation has no key | Reissue only as the same reviewed Action with a stable key |\n| connector.chatwoot.remoterejected | Chatwoot returned another non-success status | Correct request or provider state; inspect mutation outcome |\n| connector.chatwoot.remotetemporary | Chatwoot returned 429 or 5xx | An invocation can be uncertain; do not infer retry safety |\n| connector.chatwoot.transport | Provider transport failed | An invocation can be uncertain; reconcile before replay |\n| connector.chatwoot.responsetoolarge | Provider response exceeded the configured bound | Review the operation and limit; an invocation can be uncertain |\n| connector.chatwoot.replaystore | Webhook replay storage failed | Restore storage and retain the original delivery |\n| connector.chatwoot.webhookreplay | The typed replay policy rejected a delivery | Inspect the retained delivery and replay state |\n| connector.chatwoot.cancelled | Invocation was cancelled before a result settled | Treat provider outcome as uncertain and read durable state |\n\nconnector.chatwoot.webhookreplay exists in the typed error mapping, but the\nordinary duplicate webhook path does not construct it. That path reproduces a\ndeterministic Event ID so durable Event storage can deduplicate the delivery.\n\nLook up connector-host codes\n\nThe host can reject a request before Chatwoot is called. Product-neutral trust,\nroute, credential-revision, approval, replay, and execution codes are defined\nin the global error reference. These compact or immediate host codes often\nidentify the first diagnostic branch.\n\n| Code | HTTP | Trust and recovery decision |\n\n| schema.envelope.invalid | 400 | Correct the JSON against the Envelope schema; the compact response is unsigned |\n| envelope.decode.invalid | 400 | Correct typed Envelope decoding; the compact response is unsigned |\n| connectorhost.notready | 503 | Wait for a ready, leased, non-draining host through normal routing |\n| connectorhost.capacity | 429 | Honor the 100 ms hint only within Action retry policy |\n| connectorhost.responsesigning | 500 | Isolate the response and read central durable state; authenticity is incomplete |\n\nA signed host code such as connectorhost.authentication,\nconnectorhost.routemismatch, or\nconnectorhost.credentialrevisiondenied identifies a host policy boundary,\nnot a Chatwoot credential result. Keep the original signed envelope when\nescalating.\n\nLook up webhook HTTP codes\n\nThe standalone ingress route returns a compact error rather than an AIP error\nenvelope.\n\n| Code | HTTP | Decision |\n\n| webhook.missingheader | 400 | Restore the delivery, timestamp, and signature headers |\n| webhook.invalidtimestamp | 400 | Supply a canonical signed decimal timestamp |\n| webhook.invalidutf8 | 400 | Preserve exact bytes and correct the sender or proxy encoding |\n| webhook.authenticationfailed | 401 | Verify secret selection, timestamp window, HMAC, raw body, and JSON delivery |\n| webhook.invaliddelivery | 400 | Correct the typed delivery after preserving the original evidence |\n| webhook.invalidmapping | 500 | Retain the accepted payload and escalate the impossible Event mapping |\n| webhook.storageunavailable | 503 | Restore replay, Event-log, or outbox storage and redeliver the same request |\n\nAn HTTP 200 can report a locally accepted Event whose central publication is\nstill queued. Recover the durable outbox instead of asking Chatwoot to create a\nnew webhook or repeating the Action that caused it.\n\nDecide whether to retry\n\nApply these checks in order and stop at the first unmet condition:\n1. verify the signed response or read the durable Action result;\n2. classify the operation as a read, mutation, or webhook delivery;\n3. for a mutation, establish provider outcome from authoritative state;\n4. require published implementation retry support;\n5. require retryable not to be false;\n6. keep the same Action, input, and idempotency key;\n7. respect attempt, elapsed-time, and backoff limits.\n\nAt this revision, the conversation-status composite and the 62 catalogue reads\npublish retry support. Connector-mapped failures still set retryable to\nfalse. The host capacity and not-ready codes can carry retryable: true, but\nthat hint cannot override operation safety or the Action budget.\n\nTreat uncertain outcomes as nonterminal\n\nremotetemporary, transport, responsetoolarge, and cancelled can mark\nan invocation outcome uncertain. Stop new mutations for the affected resource,\nretain the original identifiers, read Chatwoot and AIP state, and decide whether\nthe intended effect is present before any compensation or replacement Action.\n\nThe connector does not publish transaction or reconciliation support. Recovery\ntherefore uses the canonical provider resource reads and the original durable\nAction evidence. A temporary category or missing provider request ID is never\nproof that a mutation did not happen.\n\nCollect minimum evidence\n\nRetain a bounded incident bundle before changing configuration or state:\n• UTC interval, deployment revision, instance ID, replica ID, and account ID;\n• Action ID, correlation ID, capability ID, route assignment, and key reference;\n• exact code, category, retry flag, uncertainty flag, and remote status;\n• readiness snapshot and the eight process-local host metrics;\n• redacted provider resource state for an uncertain mutation;\n• webhook delivery ID, timestamp, response code, Event ID, and outbox state;\n• the first startup or heartbeat failure without credentials or raw payloads.\n\nRoute configuration defects to the connector owner and lease or capacity\ndefects to the fleet operator. Route provider-access defects to the Chatwoot\naccount owner, webhook authentication defects to the ingress owner, and\nuncertain mutations to the Action owner. Escalate cross-tenant output,\nsignature failures, route misbinding, credential exposure, and impossible\nEvent mappings as security or integrity incidents.\n\nVerify recovery\n\nClose the incident only when the corrected path is proven through the normal\ngateway route, readiness and lease agree, durable Action or Event state is\nterminal, and no uncertain mutation remains unresolved. Record the evidence\nand remove temporary containment explicitly.\n\nRelated documentation\n• Health, observability, and recovery (../operations/health-observability-and-recovery.md)\n• Configuration (../reference/configuration.md)\n• Mutation approval and idempotency (../reference/mutation-approval-and-idempotency.md)\n• Webhook security, loop prevention, and replay (../reference/webhook-security-loop-prevention-and-replay.md)\n• Errors and retry decisions (../../../reference/errors.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "2c5c9b73141e8ed1cc262bfaab970de0387a138e714c3a70e3360d39100ed28f"
  }
}