{
  "schemaVersion": "1.0",
  "title": "AIP operator runbooks",
  "description": "Use these runbooks for the first response to a common getaip-server, connector-fleet, or connector-host incident. Select the symptom below, contain new effects, use the recorded checks to identify the failing owner, and stop when the eviden",
  "canonical": "https://getaip.org/docs/guides/operator-runbooks",
  "route": "/docs/guides/operator-runbooks",
  "source": "docs/guides/operator-runbooks.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Deploy and Operate",
  "documentType": "Guide",
  "language": "en",
  "revision": {
    "lastReviewedRevision": "d7cce13d1d555644d04a4d73c66c95b113737635",
    "documentationSourceRevision": "9192fef3695ad294994f2712f6d156241e5e92fb",
    "basis": "frontmatter"
  },
  "downloads": {
    "md": "/docs/download/guides/operator-runbooks.md",
    "txt": "/docs/download/guides/operator-runbooks.txt",
    "json": "/docs/download/guides/operator-runbooks.json",
    "pdf": "/docs/download/guides/operator-runbooks.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: AIP operator runbooks\ndescription: Contain and recover common AIP runtime and connector-fleet incidents\nkind: runbook\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"d7cce13d1d555644d04a4d73c66c95b113737635\"\n---\n\n# AIP operator runbooks\n\nUse these runbooks for the first response to a common `getaip-server`, connector-fleet,\nor connector-host incident. Select the symptom below, contain new effects, use\nthe recorded checks to identify the failing owner, and stop when the evidence\nno longer supports a safe automatic recovery.\n\nThe procedures apply to source revision\n`d7cce13d1d555644d04a4d73c66c95b113737635`. They do not define\ndeployment-specific alert thresholds, database failover commands, provider\noperations, or incident ownership. Use [Observe and recover AIP\nwork](observe-and-recover.md) for the exact query flow and evidence hierarchy.\n\n## Select the runbook\n\n| Symptom | Start here | Immediate containment |\n|---|---|---|\n| Core `/ready` returns `503` | Core daemon is not ready | Stop new traffic to the unready replica |\n| Worker cycles stop or action leases do not progress | Runtime worker or action lease is stalled | Pause affected mutations; preserve action IDs |\n| Runtime storage fails or a database failover begins | Runtime database is unavailable | Stop new mutations for that state owner |\n| Fleet snapshot fails or no route is available | Fleet maintenance or route resolution fails | Keep affected bindings disabled or traffic paused |\n| Host `/ready` reports an invalid lease | Connector-host lease is lost | Remove the replica from new traffic; keep its state |\n| Remote queue or event ingress reaches capacity | A bounded admission queue is saturated | Apply backpressure at the authenticated edge |\n| A transaction is `outcome_unknown` or a duplicate is suspected | Provider outcome is uncertain | Stop retries for the original mutation |\n| Callback or remote delegation is dead-lettered | Delivery retry budget is exhausted | Do not rerun the source action |\n| A new connector rollout regresses | Connector rollout must be rolled back | Stop new assignments to the affected revision |\n| Signature or replay validation alerts | Trust or replay boundary is failing | Reject the traffic; preserve redacted evidence |\n\nBefore any runbook, retain the action, transaction, approval, delegation,\ncallback, message, tenant, and connector-route identifiers that exist. Never\ncopy credentials or governed payloads into an incident record.\n\n## Core daemon is not ready\n\n**Trigger:** `/health` succeeds but core `/ready` returns `503`, or the\nload balancer has removed a `getaip-server` replica.\n\n**Checks:**\n\n1. Read `gateway.ready`, storage, and local connector details.\n2. Read `supervisor.runtime_worker_running`, `nats_required`,\n   `nats_listener_running`, `last_worker_error`, and the last worker success.\n3. Inspect every required local module. Do not treat an optional fleet snapshot\n   as part of the top-level readiness calculation.\n4. Confirm that the process still uses the intended runtime store, trust files,\n   and authenticated edge.\n\n**Contain:** keep the unready replica out of traffic. If all replicas share the\nsame failing state owner, pause new actions instead of sending them to a fresh\nempty runtime.\n\n**Recover:** restore read/write access to the configured store, the required\nNATS listener, or the required local module. The runtime worker supervisor\nrestarts a failed child after a bounded delay. Repeated restart with the same\nerror means the dependency remains broken.\n\n**Verify:** require core `/ready` to return `200`, observe a new worker\nsuccess and advancing worker cycles, then read one known action from the\ndurable store.\n\n**Do not:** switch a clustered runtime to memory or a new file directory merely\nto obtain a ready response. That creates a second action and idempotency owner.\n\n## Runtime worker or action lease is stalled\n\n**Trigger:** queued actions age, running actions stop updating, or\n`runtime_worker_running` is false.\n\n**Checks:**\n\n1. List the affected queued and running actions.\n2. For each action, compare `updated_at`, retry `next_attempt_at`, lease\n   owner, lease ID, and lease expiry.\n3. Distinguish a future retry window, pending approval, and provider\n   reconciliation from a worker stall.\n4. Compare worker cycles, claimed-action activity, last success, and last error.\n5. Check runtime-store latency and clock health before increasing concurrency.\n\n**Contain:** pause new mutations for the affected capability when lease\nownership or provider outcome is uncertain. Keep read paths available.\n\n**Recover:** restore the worker dependency and let the existing worker loop\nclaim only actions whose retry window has opened and whose lease is absent or\nexpired. The storage backend must fence the claim atomically.\n\n**Verify:** the original action receives a new durable update or reaches a\nterminal state. Confirm that no second action ID was introduced.\n\n**Do not:** clear a valid lease, edit its expiry, or copy the queued payload to a\nnew action. A stale worker may not settle work after it loses the lease.\n\n## Runtime database is unavailable\n\n**Trigger:** gateway storage readiness fails, durable reads fail, or the\nruntime PostgreSQL owner enters failover.\n\n**Checks:**\n\n1. Identify whether the deployment uses PostgreSQL or one dedicated local\n   storage directory.\n2. Confirm which replicas share the logical runtime and which database role\n   they use.\n3. Record active actions, leases, approvals, transactions, callbacks,\n   delegations, replay claims, and provider operation references from the last\n   trustworthy view.\n4. Determine whether the database authority has fenced the former primary.\n\n**Contain:** stop admitting new mutations for this logical runtime. Keep old\nreplicas from writing until the database failover procedure establishes one\nprimary owner.\n\n**Recover:** complete the deployment's database failover or restore the exact\nlocal state directory. Start one reviewed `getaip-server` replica against that state\nowner and allow runtime recovery to inspect queued actions, approvals,\ntransactions, callbacks, and delegations. Reconcile unknown provider outcomes\nbefore restoring full concurrency.\n\n**Verify:** core readiness passes, action and idempotency history remains\nqueryable, lease ownership is unique, and retained provider references still\nmatch their transactions.\n\n**Rollback:** if the new database owner is not authoritative, stop the new\nwriter before returning to a previously fenced primary. Never run two writable\nhistories and attempt to merge them through action replay.\n\n## Fleet maintenance or route resolution fails\n\n**Trigger:** the optional fleet snapshot reports repeated failures, tenant\ncapability discovery omits an expected route, or new actions return no usable\nconnector assignment.\n\n**Checks:**\n\n1. Inspect `fleet.worker_running`, consecutive failures, last error, last\n   success, and the last aggregate summary.\n2. Check the registry data-plane connection without substituting an\n   administrator credential.\n3. For new work, verify the tenant binding, active connector type and version,\n   ready non-expired replica, remaining capacity, and topology constraint.\n4. For existing work, read the durable assignment for the original action.\n   Current discovery describes only routes available for new assignments.\n\n**Contain:** keep the affected binding disabled or pause new actions. Existing\nassignments remain pinned and must not be moved to the replica that currently\nlooks healthiest.\n\n**Recover:** restore the registry data path. Let the fleet maintenance worker\nexpire stale replica leases in bounded cycles and refresh its summary. Restore\nor register replicas through their signed lifecycle path.\n\n**Verify:** the fleet snapshot records a later successful cycle, tenant\ndiscovery returns the intended capability, and a new non-mutating action obtains\none assignment. Verify old ambiguous actions on their original routes.\n\n**Do not:** write registry status or assignment rows manually. Fleet failures\ndo not make the top-level core readiness response fail at this revision.\n\n## Connector-host lease is lost\n\n**Trigger:** a standalone host is alive but `/ready` reports\n`lease_valid: false`, the registry marks the replica expired, or connector\nevent publication stops after lifecycle-control failure.\n\n**Checks:**\n\n1. Read host `draining`, lease validity, runtime storage readiness, connector\n   readiness, and recovery summary.\n2. Confirm whether the lifecycle control plane is reachable and still trusts\n   the admitted replica identity.\n3. Compare the last lease sequence and expiry with the registry view.\n4. Record active assignments and unresolved provider work.\n\n**Contain:** remove the host from new traffic. Do not discard its durable\nruntime state or provider operation references. An invalid lease also blocks\nthe common host's event-outbox publication.\n\n**Recover:** when storage and connector health are ready, the heartbeat worker\nattempts lease recovery by re-registering the same immutable identity. Restore\nthe lifecycle path and allow that idempotent recovery to finish. If admission\nrejects the identity, correct the package, configuration, or capacity conflict\ninstead of weakening the check.\n\n**Verify:** host `/ready` returns `200`, the registry shows a current lease\nand health revision, and retained work remains on the same logical replica.\n\n**Fail over:** use a distinct replica identity for planned replacement. Reuse\nthe failed identity only after a clean offline transition or lease expiry and\nonly with the same durable recovery boundary.\n\n## A bounded admission queue is saturated\n\n**Trigger:** remote dispatch reports admission-capacity errors, remote queued\nwork or bytes stay near their configured bounds, or connector-event ingress\nrejects work because no permit is available.\n\n**Checks:**\n\n1. Distinguish global remote queue, per-tenant queue, queued-byte, queue-age,\n   and active-dispatch limits.\n2. Compare active and queued tenants to identify whether one tenant is\n   consuming its own bound or the shared scheduler is saturated.\n3. Check provider latency, registry latency, host capacity, and external rate\n   limits.\n4. For event ingress, compare configured capacity with available permits and\n   receiver latency.\n\n**Contain:** honor the typed retry guidance at the authenticated edge and apply\ntenant-scoped backpressure. Do not allow an unbounded upstream queue to hide\nthe rejection.\n\n**Recover:** restore the slow dependency or reduce accepted traffic. Change a\nlimit only after measuring memory, request size, provider concurrency, tenant\nfairness, and downstream rate limits.\n\n**Verify:** queued work and bytes decrease, permits become available, the\noldest admitted request stays within the intended queue-age bound, and other\ntenants make progress.\n\n**Do not:** treat process-local remote queue metrics as the durable action\nqueue. A client retry still needs the original action and idempotency identity.\n\n## Provider outcome is uncertain\n\n**Trigger:** a transaction is `outcome_unknown`, the connection failed after\na provider request, or two external effects may represent one intended action.\n\n**Checks:**\n\n1. Freeze automatic and manual retry for the action and idempotency key.\n2. Read the transaction, provider operation reference, reconciliation cursor,\n   action input hash, tenant, external account, and connector assignment.\n3. Query provider facts through an authorized, provider-specific reconciliation\n   operation.\n4. Compare receipts and audit events with provider timestamps and identifiers.\n\n**Contain:** block another commit and keep the original connector route\navailable. An unavailable response is not evidence that the provider rejected\nthe mutation.\n\n**Recover:** record the authoritative provider result in the existing\ntransaction through reconciliation. Resume, fail, or start a separately\ngoverned compensation only after the outcome is known.\n\n**Verify:** the transaction reaches a reconciled terminal status and the\nprovider shows one intended effect. If the provider offers no reliable lookup,\nescalate for human resolution without guessing.\n\n**Do not:** compensate and retry in parallel. Compensation is not a substitute\nfor identifying the original outcome.\n\n## Delivery retry budget is exhausted\n\n**Trigger:** a callback delivery or remote delegation reaches\n`dead_lettered`.\n\n**Checks:**\n\n1. Confirm the source action's terminal state.\n2. For callbacks, inspect target, profile, attempts, last error,\n   dead-letter reason, and receipt chain.\n3. Check DNS, TLS, allowlist, receiver authentication, signature validation,\n   response status, and receiver idempotency.\n4. For remote delegation, correlate the parent action, child identity, peer\n   observations, action events, and audit records.\n\n**Contain:** keep the source action terminal and preserve the delivery record.\nDo not rerun the action to regenerate a message.\n\n**Recover:** restore the destination first. Pending callback deliveries and\nexpired callback leases recover through the worker. Dead-lettered callback and\ndelegation records are terminal for automatic recovery, and this revision\nexposes no public replay command for either outbox. Use an approved,\ndeployment-specific redelivery procedure or escalate.\n\n**Verify:** the receiver acknowledges exactly one message with the intended\ndelivery identity and the incident record links that evidence to the original\naction.\n\n## Connector rollout must be rolled back\n\n**Trigger:** a newly admitted connector version, configuration revision, or\nreplica causes health, correctness, or provider failures.\n\n**Checks:**\n\n1. Identify the exact package revision, artifact digest, connector version,\n   instance, binding policy revision, and affected replicas.\n2. Separate new unassigned work from actions already pinned to the revision.\n3. Record active assignments, outcome-unknown actions, delivery outboxes, and\n   provider jobs.\n4. Confirm that the previous artifact and admission package remain trusted.\n\n**Contain:** pause affected mutations and apply a signed binding revision that\nprevents new assignments to the failed path. Do not revoke a whole package\nunless the incident requires that wider security boundary. If the artifact is\nno longer trusted, stop further provider execution and reconcile retained work\nfrom independent provider evidence.\n\n**Recover:** express rollback as a newer desired-state generation. Start\ndistinct replicas for the known-good immutable version, verify their\nregistration and readiness, and enable new assignments through a reviewed\nbinding. Begin drain on the failed-version replicas and keep their processes and\ndurable state available while pinned work finishes or reconciles.\n\n**Verify:** discovery sends new work only to the intended version, old\nassignments are terminal or reconciled, and active assignment counts reach zero\nbefore the failed processes stop and their replicas go offline.\n\n**Rollback the rollback:** create another newer generation. Do not decrement a\ngeneration, reuse a replica identity for another digest, or erase the execution\njournal.\n\n## Trust or replay boundary is failing\n\n**Trigger:** signatures fail, an unexpected signer appears, replay claims\nincrease, or a valid-looking request is rejected for time or identity context.\n\n**Checks:**\n\n1. Preserve the message ID, signer DID, claimed sender, recipient, timestamp,\n   authenticated edge identity, tenant, and redacted typed error.\n2. Verify clock synchronization and the configured acceptance window.\n3. Read the durable replay claim before resending the same message.\n4. Compare the signer-to-principal binding, trusted identity revision, scopes,\n   and recent key rotation.\n5. Determine whether the message was altered, duplicated, delayed, or signed by\n   a key outside the current trust set.\n\n**Contain:** reject the affected traffic and isolate a suspected key or edge.\nKeep unrelated tenants inside their own verified trust boundaries.\n\n**Recover:** correct clock or trust data through the reviewed configuration\nowner. Rotate or revoke a compromised signer, then verify the boundary with a\nfresh signed, non-mutating request. Retry a rejected mutation only after its\ndurable action state proves that no accepted or ambiguous effect exists.\n\n**Verify:** signatures bind to the expected principal, replay claims remain\ndurable across restart, and no unsigned exception was introduced.\n\n**Do not:** disable signature, replay, tenant, or authorization checks to\nrestore availability.\n\n## Retain evidence and escalate\n\nRetain:\n\n- source revision, artifact digest, deployment revision, and configuration\n  class;\n- action, approval, transaction, delegation, callback, receipt, and route\n  identifiers;\n- redacted typed errors, provider correlation IDs, event cursors, and\n  timestamps;\n- readiness and metrics snapshots from before and after recovery;\n- the containment, recovery, verification, and rollback decisions;\n- the approving operator and the owner accepting any residual uncertainty.\n\nEscalate when the provider outcome cannot be established, the authoritative\nstate owner is disputed, a dead-lettered delivery needs a new effect, or a\ntrust key may be compromised. Also escalate if recovery would change an\nexisting action identity or route.\n\n## Related documentation\n\n- [Errors and retry decisions](../reference/errors.md)\n- [Metrics reference](../reference/metrics.md)\n- [Operate the connector-host lifecycle](connector-host-lifecycle.md)\n- [Orchestrate and roll back connector hosts](connector-orchestration.md)\n- [Deploy AIP in production](production-deployment.md)\n",
    "text": "AIP operator runbooks\n\nUse these runbooks for the first response to a common getaip-server, connector-fleet,\nor connector-host incident. Select the symptom below, contain new effects, use\nthe recorded checks to identify the failing owner, and stop when the evidence\nno longer supports a safe automatic recovery.\n\nThe procedures apply to source revision\nd7cce13d1d555644d04a4d73c66c95b113737635. They do not define\ndeployment-specific alert thresholds, database failover commands, provider\noperations, or incident ownership. Use Observe and recover AIP\nwork (observe-and-recover.md) for the exact query flow and evidence hierarchy.\n\nSelect the runbook\n\n| Symptom | Start here | Immediate containment |\n\n| Core /ready returns 503 | Core daemon is not ready | Stop new traffic to the unready replica |\n| Worker cycles stop or action leases do not progress | Runtime worker or action lease is stalled | Pause affected mutations; preserve action IDs |\n| Runtime storage fails or a database failover begins | Runtime database is unavailable | Stop new mutations for that state owner |\n| Fleet snapshot fails or no route is available | Fleet maintenance or route resolution fails | Keep affected bindings disabled or traffic paused |\n| Host /ready reports an invalid lease | Connector-host lease is lost | Remove the replica from new traffic; keep its state |\n| Remote queue or event ingress reaches capacity | A bounded admission queue is saturated | Apply backpressure at the authenticated edge |\n| A transaction is outcomeunknown or a duplicate is suspected | Provider outcome is uncertain | Stop retries for the original mutation |\n| Callback or remote delegation is dead-lettered | Delivery retry budget is exhausted | Do not rerun the source action |\n| A new connector rollout regresses | Connector rollout must be rolled back | Stop new assignments to the affected revision |\n| Signature or replay validation alerts | Trust or replay boundary is failing | Reject the traffic; preserve redacted evidence |\n\nBefore any runbook, retain the action, transaction, approval, delegation,\ncallback, message, tenant, and connector-route identifiers that exist. Never\ncopy credentials or governed payloads into an incident record.\n\nCore daemon is not ready\n\nTrigger: /health succeeds but core /ready returns 503, or the\nload balancer has removed a getaip-server replica.\n\nChecks:\n1. Read gateway.ready, storage, and local connector details.\n2. Read supervisor.runtimeworkerrunning, natsrequired,\n   natslistenerrunning, lastworkererror, and the last worker success.\n3. Inspect every required local module. Do not treat an optional fleet snapshot\n   as part of the top-level readiness calculation.\n4. Confirm that the process still uses the intended runtime store, trust files,\n   and authenticated edge.\n\nContain: keep the unready replica out of traffic. If all replicas share the\nsame failing state owner, pause new actions instead of sending them to a fresh\nempty runtime.\n\nRecover: restore read/write access to the configured store, the required\nNATS listener, or the required local module. The runtime worker supervisor\nrestarts a failed child after a bounded delay. Repeated restart with the same\nerror means the dependency remains broken.\n\nVerify: require core /ready to return 200, observe a new worker\nsuccess and advancing worker cycles, then read one known action from the\ndurable store.\n\nDo not: switch a clustered runtime to memory or a new file directory merely\nto obtain a ready response. That creates a second action and idempotency owner.\n\nRuntime worker or action lease is stalled\n\nTrigger: queued actions age, running actions stop updating, or\nruntimeworkerrunning is false.\n\nChecks:\n1. List the affected queued and running actions.\n2. For each action, compare updatedat, retry nextattemptat, lease\n   owner, lease ID, and lease expiry.\n3. Distinguish a future retry window, pending approval, and provider\n   reconciliation from a worker stall.\n4. Compare worker cycles, claimed-action activity, last success, and last error.\n5. Check runtime-store latency and clock health before increasing concurrency.\n\nContain: pause new mutations for the affected capability when lease\nownership or provider outcome is uncertain. Keep read paths available.\n\nRecover: restore the worker dependency and let the existing worker loop\nclaim only actions whose retry window has opened and whose lease is absent or\nexpired. The storage backend must fence the claim atomically.\n\nVerify: the original action receives a new durable update or reaches a\nterminal state. Confirm that no second action ID was introduced.\n\nDo not: clear a valid lease, edit its expiry, or copy the queued payload to a\nnew action. A stale worker may not settle work after it loses the lease.\n\nRuntime database is unavailable\n\nTrigger: gateway storage readiness fails, durable reads fail, or the\nruntime PostgreSQL owner enters failover.\n\nChecks:\n1. Identify whether the deployment uses PostgreSQL or one dedicated local\n   storage directory.\n2. Confirm which replicas share the logical runtime and which database role\n   they use.\n3. Record active actions, leases, approvals, transactions, callbacks,\n   delegations, replay claims, and provider operation references from the last\n   trustworthy view.\n4. Determine whether the database authority has fenced the former primary.\n\nContain: stop admitting new mutations for this logical runtime. Keep old\nreplicas from writing until the database failover procedure establishes one\nprimary owner.\n\nRecover: complete the deployment's database failover or restore the exact\nlocal state directory. Start one reviewed getaip-server replica against that state\nowner and allow runtime recovery to inspect queued actions, approvals,\ntransactions, callbacks, and delegations. Reconcile unknown provider outcomes\nbefore restoring full concurrency.\n\nVerify: core readiness passes, action and idempotency history remains\nqueryable, lease ownership is unique, and retained provider references still\nmatch their transactions.\n\nRollback: if the new database owner is not authoritative, stop the new\nwriter before returning to a previously fenced primary. Never run two writable\nhistories and attempt to merge them through action replay.\n\nFleet maintenance or route resolution fails\n\nTrigger: the optional fleet snapshot reports repeated failures, tenant\ncapability discovery omits an expected route, or new actions return no usable\nconnector assignment.\n\nChecks:\n1. Inspect fleet.workerrunning, consecutive failures, last error, last\n   success, and the last aggregate summary.\n2. Check the registry data-plane connection without substituting an\n   administrator credential.\n3. For new work, verify the tenant binding, active connector type and version,\n   ready non-expired replica, remaining capacity, and topology constraint.\n4. For existing work, read the durable assignment for the original action.\n   Current discovery describes only routes available for new assignments.\n\nContain: keep the affected binding disabled or pause new actions. Existing\nassignments remain pinned and must not be moved to the replica that currently\nlooks healthiest.\n\nRecover: restore the registry data path. Let the fleet maintenance worker\nexpire stale replica leases in bounded cycles and refresh its summary. Restore\nor register replicas through their signed lifecycle path.\n\nVerify: the fleet snapshot records a later successful cycle, tenant\ndiscovery returns the intended capability, and a new non-mutating action obtains\none assignment. Verify old ambiguous actions on their original routes.\n\nDo not: write registry status or assignment rows manually. Fleet failures\ndo not make the top-level core readiness response fail at this revision.\n\nConnector-host lease is lost\n\nTrigger: a standalone host is alive but /ready reports\nleasevalid: false, the registry marks the replica expired, or connector\nevent publication stops after lifecycle-control failure.\n\nChecks:\n1. Read host draining, lease validity, runtime storage readiness, connector\n   readiness, and recovery summary.\n2. Confirm whether the lifecycle control plane is reachable and still trusts\n   the admitted replica identity.\n3. Compare the last lease sequence and expiry with the registry view.\n4. Record active assignments and unresolved provider work.\n\nContain: remove the host from new traffic. Do not discard its durable\nruntime state or provider operation references. An invalid lease also blocks\nthe common host's event-outbox publication.\n\nRecover: when storage and connector health are ready, the heartbeat worker\nattempts lease recovery by re-registering the same immutable identity. Restore\nthe lifecycle path and allow that idempotent recovery to finish. If admission\nrejects the identity, correct the package, configuration, or capacity conflict\ninstead of weakening the check.\n\nVerify: host /ready returns 200, the registry shows a current lease\nand health revision, and retained work remains on the same logical replica.\n\nFail over: use a distinct replica identity for planned replacement. Reuse\nthe failed identity only after a clean offline transition or lease expiry and\nonly with the same durable recovery boundary.\n\nA bounded admission queue is saturated\n\nTrigger: remote dispatch reports admission-capacity errors, remote queued\nwork or bytes stay near their configured bounds, or connector-event ingress\nrejects work because no permit is available.\n\nChecks:\n1. Distinguish global remote queue, per-tenant queue, queued-byte, queue-age,\n   and active-dispatch limits.\n2. Compare active and queued tenants to identify whether one tenant is\n   consuming its own bound or the shared scheduler is saturated.\n3. Check provider latency, registry latency, host capacity, and external rate\n   limits.\n4. For event ingress, compare configured capacity with available permits and\n   receiver latency.\n\nContain: honor the typed retry guidance at the authenticated edge and apply\ntenant-scoped backpressure. Do not allow an unbounded upstream queue to hide\nthe rejection.\n\nRecover: restore the slow dependency or reduce accepted traffic. Change a\nlimit only after measuring memory, request size, provider concurrency, tenant\nfairness, and downstream rate limits.\n\nVerify: queued work and bytes decrease, permits become available, the\noldest admitted request stays within the intended queue-age bound, and other\ntenants make progress.\n\nDo not: treat process-local remote queue metrics as the durable action\nqueue. A client retry still needs the original action and idempotency identity.\n\nProvider outcome is uncertain\n\nTrigger: a transaction is outcomeunknown, the connection failed after\na provider request, or two external effects may represent one intended action.\n\nChecks:\n1. Freeze automatic and manual retry for the action and idempotency key.\n2. Read the transaction, provider operation reference, reconciliation cursor,\n   action input hash, tenant, external account, and connector assignment.\n3. Query provider facts through an authorized, provider-specific reconciliation\n   operation.\n4. Compare receipts and audit events with provider timestamps and identifiers.\n\nContain: block another commit and keep the original connector route\navailable. An unavailable response is not evidence that the provider rejected\nthe mutation.\n\nRecover: record the authoritative provider result in the existing\ntransaction through reconciliation. Resume, fail, or start a separately\ngoverned compensation only after the outcome is known.\n\nVerify: the transaction reaches a reconciled terminal status and the\nprovider shows one intended effect. If the provider offers no reliable lookup,\nescalate for human resolution without guessing.\n\nDo not: compensate and retry in parallel. Compensation is not a substitute\nfor identifying the original outcome.\n\nDelivery retry budget is exhausted\n\nTrigger: a callback delivery or remote delegation reaches\ndeadlettered.\n\nChecks:\n1. Confirm the source action's terminal state.\n2. For callbacks, inspect target, profile, attempts, last error,\n   dead-letter reason, and receipt chain.\n3. Check DNS, TLS, allowlist, receiver authentication, signature validation,\n   response status, and receiver idempotency.\n4. For remote delegation, correlate the parent action, child identity, peer\n   observations, action events, and audit records.\n\nContain: keep the source action terminal and preserve the delivery record.\nDo not rerun the action to regenerate a message.\n\nRecover: restore the destination first. Pending callback deliveries and\nexpired callback leases recover through the worker. Dead-lettered callback and\ndelegation records are terminal for automatic recovery, and this revision\nexposes no public replay command for either outbox. Use an approved,\ndeployment-specific redelivery procedure or escalate.\n\nVerify: the receiver acknowledges exactly one message with the intended\ndelivery identity and the incident record links that evidence to the original\naction.\n\nConnector rollout must be rolled back\n\nTrigger: a newly admitted connector version, configuration revision, or\nreplica causes health, correctness, or provider failures.\n\nChecks:\n1. Identify the exact package revision, artifact digest, connector version,\n   instance, binding policy revision, and affected replicas.\n2. Separate new unassigned work from actions already pinned to the revision.\n3. Record active assignments, outcome-unknown actions, delivery outboxes, and\n   provider jobs.\n4. Confirm that the previous artifact and admission package remain trusted.\n\nContain: pause affected mutations and apply a signed binding revision that\nprevents new assignments to the failed path. Do not revoke a whole package\nunless the incident requires that wider security boundary. If the artifact is\nno longer trusted, stop further provider execution and reconcile retained work\nfrom independent provider evidence.\n\nRecover: express rollback as a newer desired-state generation. Start\ndistinct replicas for the known-good immutable version, verify their\nregistration and readiness, and enable new assignments through a reviewed\nbinding. Begin drain on the failed-version replicas and keep their processes and\ndurable state available while pinned work finishes or reconciles.\n\nVerify: discovery sends new work only to the intended version, old\nassignments are terminal or reconciled, and active assignment counts reach zero\nbefore the failed processes stop and their replicas go offline.\n\nRollback the rollback: create another newer generation. Do not decrement a\ngeneration, reuse a replica identity for another digest, or erase the execution\njournal.\n\nTrust or replay boundary is failing\n\nTrigger: signatures fail, an unexpected signer appears, replay claims\nincrease, or a valid-looking request is rejected for time or identity context.\n\nChecks:\n1. Preserve the message ID, signer DID, claimed sender, recipient, timestamp,\n   authenticated edge identity, tenant, and redacted typed error.\n2. Verify clock synchronization and the configured acceptance window.\n3. Read the durable replay claim before resending the same message.\n4. Compare the signer-to-principal binding, trusted identity revision, scopes,\n   and recent key rotation.\n5. Determine whether the message was altered, duplicated, delayed, or signed by\n   a key outside the current trust set.\n\nContain: reject the affected traffic and isolate a suspected key or edge.\nKeep unrelated tenants inside their own verified trust boundaries.\n\nRecover: correct clock or trust data through the reviewed configuration\nowner. Rotate or revoke a compromised signer, then verify the boundary with a\nfresh signed, non-mutating request. Retry a rejected mutation only after its\ndurable action state proves that no accepted or ambiguous effect exists.\n\nVerify: signatures bind to the expected principal, replay claims remain\ndurable across restart, and no unsigned exception was introduced.\n\nDo not: disable signature, replay, tenant, or authorization checks to\nrestore availability.\n\nRetain evidence and escalate\n\nRetain:\n• source revision, artifact digest, deployment revision, and configuration\n  class;\n• action, approval, transaction, delegation, callback, receipt, and route\n  identifiers;\n• redacted typed errors, provider correlation IDs, event cursors, and\n  timestamps;\n• readiness and metrics snapshots from before and after recovery;\n• the containment, recovery, verification, and rollback decisions;\n• the approving operator and the owner accepting any residual uncertainty.\n\nEscalate when the provider outcome cannot be established, the authoritative\nstate owner is disputed, a dead-lettered delivery needs a new effect, or a\ntrust key may be compromised. Also escalate if recovery would change an\nexisting action identity or route.\n\nRelated documentation\n• Errors and retry decisions (../reference/errors.md)\n• Metrics reference (../reference/metrics.md)\n• Operate the connector-host lifecycle (connector-host-lifecycle.md)\n• Orchestrate and roll back connector hosts (connector-orchestration.md)\n• Deploy AIP in production (production-deployment.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "e0b270007ade5d095af282eaca6be22de46be3207b6b4b8f288e9db2d8c0c132"
  }
}