Troubleshoot Cal.diy

Use this runbook to trace a Cal.diy failure to the signed host, pinned route, credential, provider request, durable mutation, or webhook boundary. The result should be one explicit decision: correct, wait, retry a read within policy, reconcile, or escalate.

Pause new mutations whenever provider outcome is uncertain. Preserve the original action ID, idempotency key reference, route assignment, provider operation ID, provider request ID, and durable state before changing the deployment.

Trust the right error shape

Determine where the failure appeared before interpreting its fields:

Automate on code, category, retryable, uncertain_outcome, and durable state. Do not match message text or trust an unsigned pre-envelope response as authenticated action state.

Choose the first check

Diagnose authentication in order

Several layers can return an authentication-looking error. Start at the first failed boundary:

1. connector_host.authentication means the request did not match the pinned gateway principal, DID, signature, or trust domain. Correct the signed gateway path.
2. connector_host.route_mismatch means the signed assignment does not match this type, version, instance, replica, endpoint, manifest, tenant, binding, or fence. Discard it and resolve a current route.
3. connector_host.credential_revision_denied or connector_host.credential_revision_unavailable means the pinned revision is denied or cannot be read. Restore the reviewed policy; do not select a different credential silently.
4. connector.cal_diy.invalid_credential means local credential material is empty or structurally invalid before a useful provider exchange.
5. connector.cal_diy.authentication means Cal.diy returned HTTP 401 or 403. Verify the intended account, credential mode, scope, expiry, and provider-side status.

The standalone aip-host-cal-diy pins one account and one deployment credential. The optional library credential-routing mode adds connector.cal_diy.credential_resolution. In that mode, inspect the verified actor, tenant membership, tenant-to-account binding, credential issuer, tenant, operation scope, expiry, resolved material, and account match in that order. Do not look for library routing configuration in a standalone host.

After any credential repair, require /ready and route a non-mutating cap:cal_diy:profile.get through aipd. A successful direct provider request does not prove the AIP tenant and route boundary.

Separate discovery from account routing

An absent capability and a wrong Cal.diy account are different failures:

The standalone host does not dynamically choose among accounts. Changing its account ID or provider credential is a deployment change, not request recovery.

Interpret provider responses by operation type

The client retains a redacted provider code, HTTP status, request ID, and parsed Retry-After. It does not expose the raw provider error body.

For a mutation, retryable: true on rate limiting does not override the connector’s published implementation support, which enables automatic retry only for reads. A definitive rejection can establish not_committed, but a new attempt still requires the same operation, idempotency key, and input plus the owning action or transaction policy.

Look up Cal.diy connector codes

The following groups cover every explicit connector.cal_diy.* code at the pinned revision. The generic connector.cal_diy fallback represents an unexpected non-typed connector error.

Input, policy, and planning

Credential and provider

Durable mutation and reconciliation

Webhook and fallback

Category and retry describe the emitted error. The mutation ledger remains the authority for replay safety.

Decide whether to retry

Apply this order. Stop at the first unmet condition:

1. Verify the signed response or read the durable action result.
2. Determine whether the operation is a read or mutation.
3. For a mutation, read its provider operation and reconciliation state. If the state is not terminal not_committed, do not retry.
4. Require capability and implementation retry support. At this revision, Cal.diy publishes retry implementation support only for reads.
5. Require retryable not to be false and respect the action’s attempt, elapsed-time, and backoff budgets.
6. Let the runtime retry the same read action according to policy; do not bypass the gateway with a provider call.

connector.cal_diy.in_flight means another attempt owns the key, not that a second call is needed. connector.cal_diy.idempotency_collision means the key was reused with different input and must not be repaired by changing stored state.

Resolve an unknown mutation outcome

When uncertain_outcome is true or the code is connector.cal_diy.outcome_unknown or connector.cal_diy.post_dispatch_state:

1. freeze new mutations for the affected resource;
2. retain the original action, transaction, idempotency, provider operation, request, and route identifiers;
3. read the existing action and transaction state;
4. invoke the normal reconciliation path with the exact provider operation ID;
5. correlate Cal.diy audit or resource state and any embedded AIP booking metadata;
6. accept committed or not_committed only from durable trusted evidence;
7. keep the result nonterminal when evidence remains ambiguous;
8. compensate only after the committed effect is identified and the capability contract permits compensation.

A stale pre-dispatch claimed record can become not_committed. A stale dispatching record becomes uncertain. Waiting 120 seconds is therefore not by itself permission to replay a dispatched mutation.

Troubleshoot webhook ingress

The standalone route returns compact HTTP errors:

A successful response can report durably_queued rather than central acknowledgement. Recover central publication from the retained outbox; do not ask Cal.diy to generate a new event. Follow Webhook security and replay for the exact HMAC, timestamp, duplicate, and delivery contract.

Resolve startup failures

Startup failures precede the AIP error surface. Inspect the first process error and compare it with Cal.diy configuration. Common causes include:

• an invalid base URL or account ID;
• a mixed or incomplete authentication mode;
• a secret file that is unreadable or exceeds its configured bound;
• an invalid webhook secret mapping;
• a missing central event endpoint for enabled ingress;
• a database or manifest-discovery failure; or
• a registry identity mismatch.

Correct one owning boundary at a time. Re-run readiness and the gateway-routed profile read after every change; do not combine credential, account, artifact, and registry changes into one diagnostic restart.

Verify the resolution

Close the incident only when:

• the signed error or startup failure no longer occurs for the corrected path;
• /ready, the lifecycle lease, and tenant discovery agree;
• a gateway-routed profile read reaches the intended account;
• the original mutation is terminal or remains explicitly paused as uncertain;
• the exact accepted webhook event appears centrally when ingress was affected;
• no credential, raw provider body, webhook secret, or personal data entered logs or the incident record.

Escalate with:

• the source revision plus artifact and manifest digests;
• the tenant-safe identity set and timestamps;
• the signed error plus action and transaction state;
• provider operation and request IDs;
• remote status, retry hint, and uncertainty flag;
• readiness fields and metric snapshots; and
• webhook event IDs and redacted logs.

Related documentation

• Errors and retry decisions
• Cal.diy authentication and tenant routing
• Cal.diy configuration
• Recover the Cal.diy connector
• Cal.diy webhook security and replay