{
  "schemaVersion": "1.0",
  "title": "Recover the Cal.diy connector",
  "description": "Use this runbook when a Cal.diy host is unavailable, not ready, rejecting traffic, losing its lease, returning uncertain mutation outcomes, or retaining provider events for later delivery. It helps you identify the failing boundary and reco",
  "canonical": "https://getaip.org/docs/connectors/cal-diy/operations/health-observability-and-recovery",
  "route": "/docs/connectors/cal-diy/operations/health-observability-and-recovery",
  "source": "docs/connectors/cal-diy/operations/health-observability-and-recovery.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/cal-diy/operations/health-observability-and-recovery.md",
    "txt": "/docs/download/connectors/cal-diy/operations/health-observability-and-recovery.txt",
    "json": "/docs/download/connectors/cal-diy/operations/health-observability-and-recovery.json",
    "pdf": "/docs/download/connectors/cal-diy/operations/health-observability-and-recovery.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Recover the Cal.diy connector\ndescription: >-\n  Diagnose Cal.diy host health and recover durable actions and webhook events\n  without repeating provider effects\nkind: runbook\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: cal-diy\n---\n\n# Recover the Cal.diy connector\n\nUse this runbook when a Cal.diy host is unavailable, not ready, rejecting\ntraffic, losing its lease, returning uncertain mutation outcomes, or retaining\nprovider events for later delivery. It helps you identify the failing boundary\nand recover the same durable work without creating a second provider effect.\n\nRun these checks through the operator network. Keep action submission and\nprovider webhook traffic on their authenticated routes. This runbook assumes\nthe one-account deployment described in\n[Deploy the Cal.diy connector](deployment.md).\n\n## Contain risk before diagnosis\n\nPause new Cal.diy mutations or disable the affected tenant binding when any\naccepted mutation has an unknown outcome, durable storage is unavailable, or\nthe replica cannot renew its lease. Leave read traffic enabled only when its\nprovider and tenant boundaries remain verified.\n\nDo not kill the process, clear PostgreSQL, rotate credentials, create a new\naction ID, or move a pinned action as an initial response. First retain:\n\n- UTC incident start and last known-ready time;\n- connector type, version, instance, replica, and artifact digest;\n- tenant, external account, configuration, and credential revisions;\n- lease sequence and expiry from the lifecycle view;\n- action ID, idempotency key reference, provider operation ID, and provider\n  request ID where present;\n- `/ready` response and two `/metrics` snapshots with timestamps;\n- gateway assignment, action result, events, receipts, and redacted logs;\n- Cal.diy audit evidence and central event observations where applicable.\n\nThese identities determine whether recovery may resume existing work. A new\nhealthy replica does not make an old uncertain mutation safe to repeat.\n\n## Read five views in order\n\nUse all five views. No single view proves the outcome of an action.\n\nThe read-only examples require `curl`, `jq`, and an operator-only observation\norigin in `CAL_HOST_OBSERVE_ORIGIN`.\n\n| View | What it answers | What it does not prove |\n|---|---|---|\n| `/health` | Can the host process answer its static AIP liveness route? | Lease, storage, provider, or tenant readiness |\n| `/ready` | Are drain, lease, storage, and the Cal.diy profile probe ready now? | Other capabilities or one action's outcome |\n| `/metrics` | Is local readiness, load, rejection, or lease-recovery state changing? | Durable action or webhook-batch identity |\n| Lifecycle and gateway | Is the replica eligible and where is accepted work pinned? | Whether Cal.diy committed an external effect |\n| Action and provider evidence | What happened to this action at AIP and Cal.diy? | General host readiness for new work |\n\nCapture the three host surfaces without sending an action:\n\n```sh\ncurl --silent --show-error \"$CAL_HOST_OBSERVE_ORIGIN/health\" | jq .\ncurl --silent --show-error \\\n  --write-out '\\nHTTP %{http_code}\\n' \\\n  \"$CAL_HOST_OBSERVE_ORIGIN/ready\"\ncurl --silent --show-error \"$CAL_HOST_OBSERVE_ORIGIN/metrics\"\n```\n\nThe `/ready` probe calls the authenticated Cal.diy profile endpoint every\ntime. Set probe frequency and timeout within the reviewed provider and network\nbudgets. Do not use aggressive polling as a substitute for metrics.\n\n## Interpret readiness precisely\n\n`/health` returns success while the common process router is alive. `/ready`\nreturns HTTP `200` only when all four predicates are true:\n\n| Readiness field | Ready value | Failure boundary |\n|---|---:|---|\n| `draining` | `false` | Planned or partial shutdown |\n| `lease_valid` | `true` | Lifecycle path, clock, registry, or expired lease |\n| `runtime.ready` | `true` | PostgreSQL read/write readiness |\n| `connector.ready` | `true` | Authenticated Cal.diy profile request |\n\nThe `runtime.durability` field identifies the configured store class. A\nproduction PostgreSQL host should report `durable_shared`. The nested recovery\nsummary is captured during startup:\n\n| Recovery field | Meaning |\n|---|---|\n| `recovered_at` | Time at which runtime recovery began |\n| `pending_approvals` | Existing approvals retained for continuation |\n| `recoverable_transactions` | Transactions retained for reconciliation |\n| `queued_results` | Queued actions completed during startup recovery |\n| `delegation_results` | Delegations completed during startup recovery |\n| `callback_deliveries` | Callback deliveries resumed during recovery |\n\nThe summary is startup evidence, not a live queue gauge. If queued-action or\ndelegation recovery reports an error, the process fails before registration\nrather than advertising damaged state as ready.\n\nCal.diy connector readiness performs `cap:cal_diy:profile.get` against the\nconfigured account. Success proves that the selected credential reached that\nprofile endpoint at that moment. It does not validate booking permissions,\nwebhook delivery, every operation-specific scope, or a mutation result.\n\n## Interpret the eight host metrics\n\nThe pinned host emits exactly these unlabeled Prometheus metrics:\n\n| Metric | Operational use | Important limit |\n|---|---|---|\n| `aip_connector_host_ready` | Combined local lease, drain, storage, and connector state | Reads cached flags; correlate with `/ready` and registry state |\n| `aip_connector_host_storage_ready` | Last durable-storage probe result | Does not identify a damaged record |\n| `aip_connector_host_in_flight` | Actions executing in this process | Does not include provider work after a lost response |\n| `aip_connector_host_requests_total` | Native messages observed by the host | Does not classify success by capability |\n| `aip_connector_host_rejected_total` | Native messages that ended in a host or connector rejection | Requires error-code correlation |\n| `aip_connector_host_heartbeat_failures_total` | Failed heartbeat renewal or lease-recovery cycles | One increase is not proof of lease expiry |\n| `aip_connector_host_lease_recovery_attempts_total` | Re-registration attempts after lease expiry | Includes attempts that do not restore readiness |\n| `aip_connector_host_lease_recoveries_total` | Successful expired-lease recoveries | Compare with attempts and the current lease |\n\nThe source emits no Cal.diy-specific latency, provider-status, credential,\ndispatch-ledger, webhook-replay, or event-outbox metric. Use AIP action and\nevent records, provider request IDs, redacted logs, and deployment-specific\nread-only diagnostics for those views. Do not infer a successful booking from\n`aip_connector_host_ready`.\n\nDefine alert duration and ratios from the deployment SLO. At minimum, alert on\nthese relationships rather than isolated samples:\n\n- readiness remains `0` outside a planned drain;\n- storage readiness is `0` or `/ready` reports a storage failure;\n- heartbeat failures grow as the current lease approaches expiry;\n- lease-recovery attempts grow without a successful recovery;\n- in-flight work remains at the configured concurrency bound;\n- rejection growth diverges from request growth;\n- provider readiness fails while process and storage health remain good;\n- an accepted webhook event is absent from central observation beyond its\n  delivery objective.\n\n## Classify the failing boundary\n\n| Observation | Boundary | Immediate decision |\n|---|---|---|\n| No process response | Process, node, ingress, or network | Preserve the endpoint and database; follow hard-crash recovery |\n| `/health` works and `draining` is true | Planned or interrupted shutdown | Stop new assignments and finish drain or investigate its failure |\n| `lease_valid` is false | Lifecycle network, TLS, DID, clock, or admission | Keep mutations paused and allow fenced lease recovery after repair |\n| `runtime.ready` is false | PostgreSQL or durable state | Isolate the replica; never replace the database with an empty store |\n| `connector.ready` is false | Provider origin, CA, credential, account, or Cal.diy availability | Repair the provider boundary and verify a profile read |\n| Rejections rise with high in-flight work | Local capacity | Preserve action IDs and let bounded backpressure work |\n| Rejections rise with low load | Schema, signature, route, credential revision, approval, or connector error | Group by exact error code before changing capacity |\n| Mutation returns `outcome_unknown` | Provider dispatch evidence | Reconcile; do not resubmit or compensate yet |\n| Webhook says `durably_queued` | Central event delivery | Restore central ingress and observe the retained batch |\n\n## Recover process or lease loss\n\nIf the process is still running, repair lifecycle DNS, TLS, CA trust, pinned\nDID, clock, or service availability before restarting it. The host retains its\nlast committed lease until expiry. Once the lease has expired and local storage\nand provider probes are ready, the heartbeat worker performs signed\nre-registration automatically.\n\nAn ambiguous lifecycle response is retried with the same request identity and\nregistration bytes. A definite admission or configuration rejection starts a\nnew request only after the mismatch is corrected. Do not extend the lease or\nset registry status by hand.\n\nAfter a hard crash:\n\n1. prove that no process still owns the replica key and endpoint;\n2. preserve the same PostgreSQL database, artifact, instance, and replica;\n3. wait for clean offline state or lease expiry before re-registering that\n   replica identity;\n4. start the reviewed process and let durable recovery finish;\n5. stop if recovery reports queued-action or delegation errors;\n6. compare the recovery summary, new lease sequence, retained assignments, and\n   original action IDs;\n7. verify a non-mutating profile read before restoring mutation traffic.\n\nUse a new replica identity for a real replacement. Never run two processes\nwith the same replica ID or signing key.\n\n## Recover storage or provider readiness\n\nWhen `runtime.ready` is false, pause the affected binding before the next\nheartbeat makes the replica ineligible. Restore connectivity and the exact\ndatabase. Check capacity, certificates, database role, schema, and read/write\nhealth. Do not initialize a new database to make readiness green; that removes\nthe state needed to fence retries and recover callbacks, actions, and events.\n\nWhen `connector.ready` is false, verify the configured provider origin, CA,\naccount ID, credential mode, credential file, and credential revision. The\nprofile health request is read-only, but repeated failure can cause an\nunhealthy heartbeat and loss of routing eligibility. Repair or rotate through\nthe reviewed credential procedure, then require `/ready`, a current registry\nlease, and one gateway-routed profile read.\n\nDo not change instance or account identity to mask a provider failure.\n\n## Recover capacity and rejected traffic\n\nAt the local concurrency bound, new actions receive retryable\n`connector_host.capacity` with HTTP `429` and a short retry hint. Let the\ngateway and client honor backpressure with the original action identity. Scale\nonly by admitting a distinct replica under the same instance and durable\ncoordination boundary.\n\n`aip_connector_host_rejected_total` also increases for invalid envelopes,\nsignature or route failures, not-ready state, credential-revision denial,\napproval import failure, and connector execution errors. Capacity is not the\nright response when in-flight work is low. Correlate the signed AIP error with\n[Errors and retry decisions](../../../reference/errors.md).\n\nA retryable error describes transport or scheduling policy. It does not prove\nthat a Cal.diy mutation was not committed.\n\n## Recover a Cal.diy mutation\n\nCal.diy mutations use a durable, account-scoped dispatch ledger. A claim lease\ndefaults to 120 seconds, but time alone does not authorize a retry after the\nprovider-dispatch boundary.\n\n| Dispatch state | Meaning | Safe decision |\n|---|---|---|\n| `claimed` | The idempotency key is owned before provider dispatch | Wait while current; after expiry reconciliation can prove `not_committed` |\n| `dispatching` | The provider boundary may have been crossed | Wait while current; after expiry treat it as `uncertain` |\n| `completed` | A successful provider response and output are stored | Reuse the stored result for the same operation, key, and input |\n| `not_committed` | Durable evidence says provider dispatch did not commit | The same operation, key, and input may acquire a new claim |\n| `uncertain` | Dispatch occurred or may have occurred without terminal proof | Reconcile with provider and transaction evidence; do not retry |\n\nThe connector also marks a mutation uncertain when cancellation happens after\ndispatch, a successful provider response fails output validation, or a\ntransport or remote outcome is ambiguous. A definitive rejection or failure\nthat guarantees no dispatch is recorded as `not_committed`.\n\nReconciliation returns the stored provider request and status evidence. A\ntrusted durable transaction outcome cursor can establish committed or\nnot-committed terminal state. Without that evidence, `uncertain` remains\nnonterminal and the same provider operation ID is returned as the cursor.\n\nInspect Cal.diy for the embedded AIP booking metadata or other operation-owned\nevidence where the capability contract provides it. Do not edit the ledger or\ninvent a terminal result.\n\n## Recover webhook event delivery\n\nAn authenticated webhook is normalized into deterministic AIP event IDs,\nstored in the durable event log, and then accepted into the connector event\noutbox. The acknowledgement reports either `centrally_acknowledged` or\n`durably_queued`.\n\nThe outbox retains at most 10,000 pending batches, with 1 to 100 events per\nbatch. A background worker claims batches for 30 seconds, scans up to 100 at a\ntime, and retries transient central-delivery failure with bounded exponential\nbackoff. It requires a ready replica and current lease before publication.\n\nAfter restart, the worker scans the same PostgreSQL-backed profile state and\npublishes the exact retained event IDs. A crash after central acknowledgement\nbut before local deletion can resend those IDs; central event storage\ndeduplicates them. Do not ask Cal.diy to recreate an event that the host already\naccepted.\n\nThere is no public pending-outbox route or metric in this revision. Use the\nwebhook acknowledgement, central event presence, retained logs, and controlled\nread-only storage diagnostics. Preserve webhook replay state with the outbox;\ndeleting either state can turn recovery into duplicate ingestion. See\n[Webhook security and replay](../reference/webhook-security-and-replay.md) for\nthe signature, replay, and retention boundary.\n\n## Verify recovery\n\nRestore mutation traffic only when every applicable check passes:\n\n1. `/health` and `/ready` succeed and all four readiness predicates agree;\n2. the lifecycle view shows the intended replica with a current lease;\n3. metric growth has stabilized and available capacity is positive;\n4. the same action ID reaches its justified durable state;\n5. an uncertain mutation has terminal provider evidence or remains paused;\n6. central AIP observes the exact accepted webhook event IDs;\n7. a gateway-routed profile read reaches the intended Cal.diy account;\n8. the incident record retains every identity, transition, error, and operator\n   decision.\n\nIf the views disagree, keep the mutation boundary paused. Escalate with the\ndatabase retained and include the source revision, artifact and manifest\ndigests, lease history, readiness body, metric snapshots, action and provider\nidentifiers, event IDs, and redacted logs.\n\n## Related documentation\n\n- [Deploy the Cal.diy connector](deployment.md)\n- [Operate the connector host lifecycle](../../../guides/connector-host-lifecycle.md)\n- [Observe and recover AIP](../../../guides/observe-and-recover.md)\n- [Metrics reference](../../../reference/metrics.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n",
    "text": "Recover the Cal.diy connector\n\nUse this runbook when a Cal.diy host is unavailable, not ready, rejecting\ntraffic, losing its lease, returning uncertain mutation outcomes, or retaining\nprovider events for later delivery. It helps you identify the failing boundary\nand recover the same durable work without creating a second provider effect.\n\nRun these checks through the operator network. Keep action submission and\nprovider webhook traffic on their authenticated routes. This runbook assumes\nthe one-account deployment described in\nDeploy the Cal.diy connector (deployment.md).\n\nContain risk before diagnosis\n\nPause new Cal.diy mutations or disable the affected tenant binding when any\naccepted mutation has an unknown outcome, durable storage is unavailable, or\nthe replica cannot renew its lease. Leave read traffic enabled only when its\nprovider and tenant boundaries remain verified.\n\nDo not kill the process, clear PostgreSQL, rotate credentials, create a new\naction ID, or move a pinned action as an initial response. First retain:\n• UTC incident start and last known-ready time;\n• connector type, version, instance, replica, and artifact digest;\n• tenant, external account, configuration, and credential revisions;\n• lease sequence and expiry from the lifecycle view;\n• action ID, idempotency key reference, provider operation ID, and provider\n  request ID where present;\n• /ready response and two /metrics snapshots with timestamps;\n• gateway assignment, action result, events, receipts, and redacted logs;\n• Cal.diy audit evidence and central event observations where applicable.\n\nThese identities determine whether recovery may resume existing work. A new\nhealthy replica does not make an old uncertain mutation safe to repeat.\n\nRead five views in order\n\nUse all five views. No single view proves the outcome of an action.\n\nThe read-only examples require curl, jq, and an operator-only observation\norigin in CALHOSTOBSERVEORIGIN.\n\n| View | What it answers | What it does not prove |\n\n| /health | Can the host process answer its static AIP liveness route? | Lease, storage, provider, or tenant readiness |\n| /ready | Are drain, lease, storage, and the Cal.diy profile probe ready now? | Other capabilities or one action's outcome |\n| /metrics | Is local readiness, load, rejection, or lease-recovery state changing? | Durable action or webhook-batch identity |\n| Lifecycle and gateway | Is the replica eligible and where is accepted work pinned? | Whether Cal.diy committed an external effect |\n| Action and provider evidence | What happened to this action at AIP and Cal.diy? | General host readiness for new work |\n\nCapture the three host surfaces without sending an action:\n\ncurl --silent --show-error \"$CALHOSTOBSERVEORIGIN/health\" | jq .\ncurl --silent --show-error \\\n  --write-out '\\nHTTP %{httpcode}\\n' \\\n  \"$CALHOSTOBSERVEORIGIN/ready\"\ncurl --silent --show-error \"$CALHOSTOBSERVEORIGIN/metrics\"\n\nThe /ready probe calls the authenticated Cal.diy profile endpoint every\ntime. Set probe frequency and timeout within the reviewed provider and network\nbudgets. Do not use aggressive polling as a substitute for metrics.\n\nInterpret readiness precisely\n\n/health returns success while the common process router is alive. /ready\nreturns HTTP 200 only when all four predicates are true:\n\n| Readiness field | Ready value | Failure boundary |\n\n| draining | false | Planned or partial shutdown |\n| leasevalid | true | Lifecycle path, clock, registry, or expired lease |\n| runtime.ready | true | PostgreSQL read/write readiness |\n| connector.ready | true | Authenticated Cal.diy profile request |\n\nThe runtime.durability field identifies the configured store class. A\nproduction PostgreSQL host should report durableshared. The nested recovery\nsummary is captured during startup:\n\n| Recovery field | Meaning |\n\n| recoveredat | Time at which runtime recovery began |\n| pendingapprovals | Existing approvals retained for continuation |\n| recoverabletransactions | Transactions retained for reconciliation |\n| queuedresults | Queued actions completed during startup recovery |\n| delegationresults | Delegations completed during startup recovery |\n| callbackdeliveries | Callback deliveries resumed during recovery |\n\nThe summary is startup evidence, not a live queue gauge. If queued-action or\ndelegation recovery reports an error, the process fails before registration\nrather than advertising damaged state as ready.\n\nCal.diy connector readiness performs cap:caldiy:profile.get against the\nconfigured account. Success proves that the selected credential reached that\nprofile endpoint at that moment. It does not validate booking permissions,\nwebhook delivery, every operation-specific scope, or a mutation result.\n\nInterpret the eight host metrics\n\nThe pinned host emits exactly these unlabeled Prometheus metrics:\n\n| Metric | Operational use | Important limit |\n\n| aipconnectorhostready | Combined local lease, drain, storage, and connector state | Reads cached flags; correlate with /ready and registry state |\n| aipconnectorhoststorageready | Last durable-storage probe result | Does not identify a damaged record |\n| aipconnectorhostinflight | Actions executing in this process | Does not include provider work after a lost response |\n| aipconnectorhostrequeststotal | Native messages observed by the host | Does not classify success by capability |\n| aipconnectorhostrejectedtotal | Native messages that ended in a host or connector rejection | Requires error-code correlation |\n| aipconnectorhostheartbeatfailurestotal | Failed heartbeat renewal or lease-recovery cycles | One increase is not proof of lease expiry |\n| aipconnectorhostleaserecoveryattemptstotal | Re-registration attempts after lease expiry | Includes attempts that do not restore readiness |\n| aipconnectorhostleaserecoveriestotal | Successful expired-lease recoveries | Compare with attempts and the current lease |\n\nThe source emits no Cal.diy-specific latency, provider-status, credential,\ndispatch-ledger, webhook-replay, or event-outbox metric. Use AIP action and\nevent records, provider request IDs, redacted logs, and deployment-specific\nread-only diagnostics for those views. Do not infer a successful booking from\naipconnectorhostready.\n\nDefine alert duration and ratios from the deployment SLO. At minimum, alert on\nthese relationships rather than isolated samples:\n• readiness remains 0 outside a planned drain;\n• storage readiness is 0 or /ready reports a storage failure;\n• heartbeat failures grow as the current lease approaches expiry;\n• lease-recovery attempts grow without a successful recovery;\n• in-flight work remains at the configured concurrency bound;\n• rejection growth diverges from request growth;\n• provider readiness fails while process and storage health remain good;\n• an accepted webhook event is absent from central observation beyond its\n  delivery objective.\n\nClassify the failing boundary\n\n| Observation | Boundary | Immediate decision |\n\n| No process response | Process, node, ingress, or network | Preserve the endpoint and database; follow hard-crash recovery |\n| /health works and draining is true | Planned or interrupted shutdown | Stop new assignments and finish drain or investigate its failure |\n| leasevalid is false | Lifecycle network, TLS, DID, clock, or admission | Keep mutations paused and allow fenced lease recovery after repair |\n| runtime.ready is false | PostgreSQL or durable state | Isolate the replica; never replace the database with an empty store |\n| connector.ready is false | Provider origin, CA, credential, account, or Cal.diy availability | Repair the provider boundary and verify a profile read |\n| Rejections rise with high in-flight work | Local capacity | Preserve action IDs and let bounded backpressure work |\n| Rejections rise with low load | Schema, signature, route, credential revision, approval, or connector error | Group by exact error code before changing capacity |\n| Mutation returns outcomeunknown | Provider dispatch evidence | Reconcile; do not resubmit or compensate yet |\n| Webhook says durablyqueued | Central event delivery | Restore central ingress and observe the retained batch |\n\nRecover process or lease loss\n\nIf the process is still running, repair lifecycle DNS, TLS, CA trust, pinned\nDID, clock, or service availability before restarting it. The host retains its\nlast committed lease until expiry. Once the lease has expired and local storage\nand provider probes are ready, the heartbeat worker performs signed\nre-registration automatically.\n\nAn ambiguous lifecycle response is retried with the same request identity and\nregistration bytes. A definite admission or configuration rejection starts a\nnew request only after the mismatch is corrected. Do not extend the lease or\nset registry status by hand.\n\nAfter a hard crash:\n1. prove that no process still owns the replica key and endpoint;\n2. preserve the same PostgreSQL database, artifact, instance, and replica;\n3. wait for clean offline state or lease expiry before re-registering that\n   replica identity;\n4. start the reviewed process and let durable recovery finish;\n5. stop if recovery reports queued-action or delegation errors;\n6. compare the recovery summary, new lease sequence, retained assignments, and\n   original action IDs;\n7. verify a non-mutating profile read before restoring mutation traffic.\n\nUse a new replica identity for a real replacement. Never run two processes\nwith the same replica ID or signing key.\n\nRecover storage or provider readiness\n\nWhen runtime.ready is false, pause the affected binding before the next\nheartbeat makes the replica ineligible. Restore connectivity and the exact\ndatabase. Check capacity, certificates, database role, schema, and read/write\nhealth. Do not initialize a new database to make readiness green; that removes\nthe state needed to fence retries and recover callbacks, actions, and events.\n\nWhen connector.ready is false, verify the configured provider origin, CA,\naccount ID, credential mode, credential file, and credential revision. The\nprofile health request is read-only, but repeated failure can cause an\nunhealthy heartbeat and loss of routing eligibility. Repair or rotate through\nthe reviewed credential procedure, then require /ready, a current registry\nlease, and one gateway-routed profile read.\n\nDo not change instance or account identity to mask a provider failure.\n\nRecover capacity and rejected traffic\n\nAt the local concurrency bound, new actions receive retryable\nconnectorhost.capacity with HTTP 429 and a short retry hint. Let the\ngateway and client honor backpressure with the original action identity. Scale\nonly by admitting a distinct replica under the same instance and durable\ncoordination boundary.\n\naipconnectorhostrejectedtotal also increases for invalid envelopes,\nsignature or route failures, not-ready state, credential-revision denial,\napproval import failure, and connector execution errors. Capacity is not the\nright response when in-flight work is low. Correlate the signed AIP error with\nErrors and retry decisions (../../../reference/errors.md).\n\nA retryable error describes transport or scheduling policy. It does not prove\nthat a Cal.diy mutation was not committed.\n\nRecover a Cal.diy mutation\n\nCal.diy mutations use a durable, account-scoped dispatch ledger. A claim lease\ndefaults to 120 seconds, but time alone does not authorize a retry after the\nprovider-dispatch boundary.\n\n| Dispatch state | Meaning | Safe decision |\n\n| claimed | The idempotency key is owned before provider dispatch | Wait while current; after expiry reconciliation can prove notcommitted |\n| dispatching | The provider boundary may have been crossed | Wait while current; after expiry treat it as uncertain |\n| completed | A successful provider response and output are stored | Reuse the stored result for the same operation, key, and input |\n| notcommitted | Durable evidence says provider dispatch did not commit | The same operation, key, and input may acquire a new claim |\n| uncertain | Dispatch occurred or may have occurred without terminal proof | Reconcile with provider and transaction evidence; do not retry |\n\nThe connector also marks a mutation uncertain when cancellation happens after\ndispatch, a successful provider response fails output validation, or a\ntransport or remote outcome is ambiguous. A definitive rejection or failure\nthat guarantees no dispatch is recorded as notcommitted.\n\nReconciliation returns the stored provider request and status evidence. A\ntrusted durable transaction outcome cursor can establish committed or\nnot-committed terminal state. Without that evidence, uncertain remains\nnonterminal and the same provider operation ID is returned as the cursor.\n\nInspect Cal.diy for the embedded AIP booking metadata or other operation-owned\nevidence where the capability contract provides it. Do not edit the ledger or\ninvent a terminal result.\n\nRecover webhook event delivery\n\nAn authenticated webhook is normalized into deterministic AIP event IDs,\nstored in the durable event log, and then accepted into the connector event\noutbox. The acknowledgement reports either centrallyacknowledged or\ndurablyqueued.\n\nThe outbox retains at most 10,000 pending batches, with 1 to 100 events per\nbatch. A background worker claims batches for 30 seconds, scans up to 100 at a\ntime, and retries transient central-delivery failure with bounded exponential\nbackoff. It requires a ready replica and current lease before publication.\n\nAfter restart, the worker scans the same PostgreSQL-backed profile state and\npublishes the exact retained event IDs. A crash after central acknowledgement\nbut before local deletion can resend those IDs; central event storage\ndeduplicates them. Do not ask Cal.diy to recreate an event that the host already\naccepted.\n\nThere is no public pending-outbox route or metric in this revision. Use the\nwebhook acknowledgement, central event presence, retained logs, and controlled\nread-only storage diagnostics. Preserve webhook replay state with the outbox;\ndeleting either state can turn recovery into duplicate ingestion. See\nWebhook security and replay (../reference/webhook-security-and-replay.md) for\nthe signature, replay, and retention boundary.\n\nVerify recovery\n\nRestore mutation traffic only when every applicable check passes:\n1. /health and /ready succeed and all four readiness predicates agree;\n2. the lifecycle view shows the intended replica with a current lease;\n3. metric growth has stabilized and available capacity is positive;\n4. the same action ID reaches its justified durable state;\n5. an uncertain mutation has terminal provider evidence or remains paused;\n6. central AIP observes the exact accepted webhook event IDs;\n7. a gateway-routed profile read reaches the intended Cal.diy account;\n8. the incident record retains every identity, transition, error, and operator\n   decision.\n\nIf the views disagree, keep the mutation boundary paused. Escalate with the\ndatabase retained and include the source revision, artifact and manifest\ndigests, lease history, readiness body, metric snapshots, action and provider\nidentifiers, event IDs, and redacted logs.\n\nRelated documentation\n• Deploy the Cal.diy connector (deployment.md)\n• Operate the connector host lifecycle (../../../guides/connector-host-lifecycle.md)\n• Observe and recover AIP (../../../guides/observe-and-recover.md)\n• Metrics reference (../../../reference/metrics.md)\n• Errors and retry decisions (../../../reference/errors.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "7610bcf0c06602bdb4bd7dbee71b9c6c33834adff682002a8015a94d9152ce27"
  }
}