Troubleshoot the Chatwoot connector

Use this runbook to identify the boundary that failed and choose the narrowest
safe next action. It covers process startup, connector-host readiness, signed
Action handling, Chatwoot provider calls, webhook ingress, and durable Event
delivery.

Pause new mutations whenever provider outcome is uncertain. Preserve the
original Action ID, idempotency key, account ID, route assignment, remote
status, webhook delivery ID, and durable state before changing a deployment.

Identify the error surface

Start with the surface that produced the evidence. The same HTTP status can
represent different trust and recovery boundaries.

| Surface | Evidence shape | First trust check |

| Process startup | Error text and non-zero exit | Compare configuration, secret-file ownership, storage, and admission |
| Pre-envelope host rejection | Unsigned compact JSON | Trust only the controlled TLS path; no valid envelope exists to sign |
| Connector-host response | Signed AIP error envelope | Verify signature, sender, recipient, and correlation |
| Durable Action | ActionResult, state, and events | Read it even when the original HTTP exchange failed |
| Chatwoot webhook | Compact { "error": { "code": "..." } } | Correlate the authenticated ingress request and delivery ID |
| Event delivery | Local Event and outbox state | Separate accepted ingress from central publication |

Automate on the exact code, category, retry flag, uncertainoutcome, and
durable state. Message text is diagnostic and is not a stable interface.

Choose the first check

| Symptom | First check | Safe first action |

| Capability is absent | Authenticated tenant catalogue and enabled binding | Correct discovery or admission; do not call a host directly |
| Host returns 401 | Host trust code versus connector.chatwoot.authentication | Repair only the failed authentication boundary |
| Host returns 403 | Route, credential revision, approval, or provider code | Re-resolve the exact authority boundary; do not bypass it |
| /health works and /ready fails | Drain, lease, storage, and connector fields | Follow the failing readiness owner |
| Read returns a provider or transport failure | Published retry support and Action budget | Retry only the same eligible read through policy |
| Mutation has no definitive provider result | uncertainoutcome and existing provider state | Reconcile before another mutation |
| Same idempotency key is rejected | Existing key record and input hash | Reuse, wait, reconcile, or stop according to its state |
| Webhook returns 400 or 401 | Exact compact code and retained raw request | Correct that delivery without generating a replacement event |
| Webhook returns 503 | Replay, Event-log, and outbox storage | Restore storage, then redeliver the same request |
| Webhook succeeds but central Event is absent | Accepted local Event and outbox batch | Recover publication; do not replay the provider action |

Resolve startup failures

Startup failures occur before a protocol error can be returned. Compare the
first process error with the configured base URL, account ID, token file,
response limit, allowed-operation set, webhook secret, PostgreSQL state,
central Event endpoint, manifest, and immutable host identity.

Correct one owning boundary at a time. Require /ready and route a safe account
read through aipd after a repair. A direct Chatwoot request does not prove the
AIP tenant, route, approval, or signing boundary.

Follow readiness to its owner

/health is a process-liveness response. /ready succeeds only while the host
is not draining, its lease is valid, durable storage is ready, and the
authenticated Chatwoot account probe succeeds.

| Failing field or observation | Owner | Decision |

| draining | Host lifecycle | Let in-flight work settle and replace or restore the replica |
| leasevalid | Registry heartbeat and fencing | Recover the same admitted identity or replace it through admission |
| storageready | Host PostgreSQL state | Restore durable state before accepting Actions or webhooks |
| connectorready | Chatwoot account probe | Correct account, token, base URL, or provider availability |
| Ready host is not selected | Binding, route, or central registry | Re-resolve the admitted route; do not edit the assignment |
| connectorhost.capacity | Replica-local concurrency | Honor bounded backoff through the original Action policy |

Use the operations runbook for containment, heartbeat recovery, replacement,
and rollback. Readiness does not prove every Chatwoot operation or webhook
delivery path.

Look up Chatwoot connector codes

The table covers every explicit connector.chatwoot. code at the reviewed
revision. All connector failures set automatic retryable to false.

| Code | Meaning | Decision |

| connector.chatwoot.authentication | Chatwoot returned 401 or 403 | Correct account, token, scope, or provider access |
| connector.chatwoot.configuration | URL, client, metadata, or connector configuration is invalid | Correct the named configuration boundary |
| connector.chatwoot.invalidaction | Capability, operation, input, or composite rule is invalid | Correct the Action; do not retry unchanged |
| connector.chatwoot.invalidcredential | Token material is empty or structurally invalid | Rotate the owner-only token file and replace the process |
| connector.chatwoot.invalidwebhook | Webhook signature, timestamp, body, or mapping input is invalid | Preserve the raw delivery and correct authentication or format |
| connector.chatwoot.invalidwebhookdelivery | Typed ingestion payload cannot be decoded | Correct the host-to-connector delivery object |
| connector.chatwoot.missingidempotencykey | A catalogue mutation has no key | Reissue only as the same reviewed Action with a stable key |
| connector.chatwoot.remoterejected | Chatwoot returned another non-success status | Correct request or provider state; inspect mutation outcome |
| connector.chatwoot.remotetemporary | Chatwoot returned 429 or 5xx | An invocation can be uncertain; do not infer retry safety |
| connector.chatwoot.transport | Provider transport failed | An invocation can be uncertain; reconcile before replay |
| connector.chatwoot.responsetoolarge | Provider response exceeded the configured bound | Review the operation and limit; an invocation can be uncertain |
| connector.chatwoot.replaystore | Webhook replay storage failed | Restore storage and retain the original delivery |
| connector.chatwoot.webhookreplay | The typed replay policy rejected a delivery | Inspect the retained delivery and replay state |
| connector.chatwoot.cancelled | Invocation was cancelled before a result settled | Treat provider outcome as uncertain and read durable state |

connector.chatwoot.webhookreplay exists in the typed error mapping, but the
ordinary duplicate webhook path does not construct it. That path reproduces a
deterministic Event ID so durable Event storage can deduplicate the delivery.

Look up connector-host codes

The host can reject a request before Chatwoot is called. Product-neutral trust,
route, credential-revision, approval, replay, and execution codes are defined
in the global error reference. These compact or immediate host codes often
identify the first diagnostic branch.

| Code | HTTP | Trust and recovery decision |

| schema.envelope.invalid | 400 | Correct the JSON against the Envelope schema; the compact response is unsigned |
| envelope.decode.invalid | 400 | Correct typed Envelope decoding; the compact response is unsigned |
| connectorhost.notready | 503 | Wait for a ready, leased, non-draining host through normal routing |
| connectorhost.capacity | 429 | Honor the 100 ms hint only within Action retry policy |
| connectorhost.responsesigning | 500 | Isolate the response and read central durable state; authenticity is incomplete |

A signed host code such as connectorhost.authentication,
connectorhost.routemismatch, or
connectorhost.credentialrevisiondenied identifies a host policy boundary,
not a Chatwoot credential result. Keep the original signed envelope when
escalating.

Look up webhook HTTP codes

The standalone ingress route returns a compact error rather than an AIP error
envelope.

| Code | HTTP | Decision |

| webhook.missingheader | 400 | Restore the delivery, timestamp, and signature headers |
| webhook.invalidtimestamp | 400 | Supply a canonical signed decimal timestamp |
| webhook.invalidutf8 | 400 | Preserve exact bytes and correct the sender or proxy encoding |
| webhook.authenticationfailed | 401 | Verify secret selection, timestamp window, HMAC, raw body, and JSON delivery |
| webhook.invaliddelivery | 400 | Correct the typed delivery after preserving the original evidence |
| webhook.invalidmapping | 500 | Retain the accepted payload and escalate the impossible Event mapping |
| webhook.storageunavailable | 503 | Restore replay, Event-log, or outbox storage and redeliver the same request |

An HTTP 200 can report a locally accepted Event whose central publication is
still queued. Recover the durable outbox instead of asking Chatwoot to create a
new webhook or repeating the Action that caused it.

Decide whether to retry

Apply these checks in order and stop at the first unmet condition:
1. verify the signed response or read the durable Action result;
2. classify the operation as a read, mutation, or webhook delivery;
3. for a mutation, establish provider outcome from authoritative state;
4. require published implementation retry support;
5. require retryable not to be false;
6. keep the same Action, input, and idempotency key;
7. respect attempt, elapsed-time, and backoff limits.

At this revision, the conversation-status composite and the 62 catalogue reads
publish retry support. Connector-mapped failures still set retryable to
false. The host capacity and not-ready codes can carry retryable: true, but
that hint cannot override operation safety or the Action budget.

Treat uncertain outcomes as nonterminal

remotetemporary, transport, responsetoolarge, and cancelled can mark
an invocation outcome uncertain. Stop new mutations for the affected resource,
retain the original identifiers, read Chatwoot and AIP state, and decide whether
the intended effect is present before any compensation or replacement Action.

The connector does not publish transaction or reconciliation support. Recovery
therefore uses the canonical provider resource reads and the original durable
Action evidence. A temporary category or missing provider request ID is never
proof that a mutation did not happen.

Collect minimum evidence

Retain a bounded incident bundle before changing configuration or state:
• UTC interval, deployment revision, instance ID, replica ID, and account ID;
• Action ID, correlation ID, capability ID, route assignment, and key reference;
• exact code, category, retry flag, uncertainty flag, and remote status;
• readiness snapshot and the eight process-local host metrics;
• redacted provider resource state for an uncertain mutation;
• webhook delivery ID, timestamp, response code, Event ID, and outbox state;
• the first startup or heartbeat failure without credentials or raw payloads.

Route configuration defects to the connector owner and lease or capacity
defects to the fleet operator. Route provider-access defects to the Chatwoot
account owner, webhook authentication defects to the ingress owner, and
uncertain mutations to the Action owner. Escalate cross-tenant output,
signature failures, route misbinding, credential exposure, and impossible
Event mappings as security or integrity incidents.

Verify recovery

Close the incident only when the corrected path is proven through the normal
gateway route, readiness and lease agree, durable Action or Event state is
terminal, and no uncertain mutation remains unresolved. Record the evidence
and remove temporary containment explicitly.

Related documentation
• Health, observability, and recovery (../operations/health-observability-and-recovery.md)
• Configuration (../reference/configuration.md)
• Mutation approval and idempotency (../reference/mutation-approval-and-idempotency.md)
• Webhook security, loop prevention, and replay (../reference/webhook-security-loop-prevention-and-replay.md)
• Errors and retry decisions (../../../reference/errors.md)