{
  "schemaVersion": "1.0",
  "title": "Observe and recover AIP work",
  "description": "Use this guide when an AIP action is delayed, interrupted, or has an uncertain external outcome. You will establish whether the process can make progress, read the durable action and delivery records, choose a recovery action, and verify th",
  "canonical": "https://getaip.org/docs/guides/observe-and-recover",
  "route": "/docs/guides/observe-and-recover",
  "source": "docs/guides/observe-and-recover.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Deploy and Operate",
  "documentType": "Guide",
  "language": "en",
  "revision": {
    "lastReviewedRevision": "d7cce13d1d555644d04a4d73c66c95b113737635",
    "documentationSourceRevision": "9192fef3695ad294994f2712f6d156241e5e92fb",
    "basis": "frontmatter"
  },
  "downloads": {
    "md": "/docs/download/guides/observe-and-recover.md",
    "txt": "/docs/download/guides/observe-and-recover.txt",
    "json": "/docs/download/guides/observe-and-recover.json",
    "pdf": "/docs/download/guides/observe-and-recover.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Observe and recover AIP work\ndescription: Diagnose degraded or ambiguous work without duplicating external effects\nkind: how-to\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"d7cce13d1d555644d04a4d73c66c95b113737635\"\n---\n\n# Observe and recover AIP work\n\nUse this guide when an AIP action is delayed, interrupted, or has an uncertain\nexternal outcome. You will establish whether the process can make progress,\nread the durable action and delivery records, choose a recovery action, and\nverify that recovery without submitting the same provider mutation twice.\n\nThis procedure applies to the product-neutral `getaip-server` runtime and its optional\nconnector fleet at source revision\n`d7cce13d1d555644d04a4d73c66c95b113737635`. Product-specific failures still\nrequire the runbook for the selected Cal.diy, Hermes Agent, Chatwoot, Dify,\nCrewAI, or Twenty connector.\n\n## Preserve the incident boundary\n\nBefore changing configuration or restarting a process, record:\n\n- the action, capability, session, correlation, approval, transaction, and\n  delegation identifiers that are available;\n- the authenticated principal and tenant used for the operational read;\n- the original idempotency key and any provider operation reference;\n- the first observed failure time and the last known successful state update;\n- the gateway, connector type, instance, version, and replica involved, when\n  the action used the fleet;\n- the current readiness response and a metrics snapshot.\n\nPause new mutations for the affected tenant or capability when the blast radius\nis not yet known. Keep reads and evidence collection available.\n\nDo not submit a replacement action, edit a lease, delete an outbox record, or\nchange an action's route assignment to make the incident disappear. Logs,\ntraces, and metrics explain behavior; durable protocol and runtime records own\nthe current action state.\n\n## 1. Check liveness and readiness\n\nQuery the deployment through its controlled observability boundary:\n\n```sh\nexport AIP_BASE_URL=https://aip.example.com\n\ncurl --fail --silent --show-error \"$AIP_BASE_URL/health\"\ncurl --fail --silent --show-error \"$AIP_BASE_URL/ready\"\ncurl --fail --silent --show-error \"$AIP_BASE_URL/metrics\" >aip-metrics.prom\n```\n\n`/health` establishes only that the HTTP process is alive. `/ready` returns\nHTTP `200` when the core daemon is ready and `503` otherwise. Interpret the\nreadiness body by owner:\n\n| Readiness area | What to inspect | Decision |\n|---|---|---|\n| `gateway` | `ready`, storage, and local connector checks | Restore storage read/write access or a required local connector before accepting traffic |\n| `supervisor` | `runtime_worker_running`, `worker_cycles`, `last_worker_error`, and `last_worker_success_at` | A live HTTP process with a stopped or failing worker cannot recover queued work |\n| NATS | `nats_required` and `nats_listener_running` together | A stopped listener blocks readiness only when NATS is required |\n| `modules` | Every module marked `required` | Resolve the required module failure; optional modules do not control top-level readiness |\n| `fleet` | Worker state, consecutive failures, last error, last success, and summary | Diagnose registry maintenance and route supply separately from core readiness |\n\nGateway readiness performs a runtime-storage read/write check and probes\nregistered local connectors. Remote fleet replicas are not local gateway\nconnectors. The optional `fleet` snapshot is included in the response, but it\ndoes not participate in the top-level `ready` calculation at this revision.\nAn HTTP `200` can therefore coexist with no ready remote replica for a\nparticular tenant capability.\n\nDo not interpret `nats_listener_running: false` as a fault when\n`nats_required` is false. Do not restart solely because one counter has\nstopped; compare the worker state, last error, last success time, and the\ndurable work described below.\n\n## 2. Read the durable action view\n\nUse an identity authorized for the same tenant and action owner. Prefer the\nclient token file rather than placing a bearer token in process arguments:\n\n```sh\nexport AIP_TOKEN_FILE=/run/secrets/aip-native-token\n\ngetaip \\\n  --native-bearer-token-file \"$AIP_TOKEN_FILE\" \\\n  action list \"$AIP_BASE_URL\" \\\n  --state queued \\\n  --limit 100\n```\n\nChange the state or identity filters only when they narrow the incident. Once\nyou have the action ID, request the complete operational view:\n\n```sh\nexport ACTION_ID=act_example\n\ngetaip \\\n  --native-bearer-token-file \"$AIP_TOKEN_FILE\" \\\n  action status \"$AIP_BASE_URL\" \"$ACTION_ID\" \\\n  --include-result \\\n  --include-chunks \\\n  --include-receipts\n```\n\nRead the fields in this order:\n\n1. confirm `action_id`, `capability_id`, and the tenant-visible ownership;\n2. compare the stable `state` with the backend `queued_state`;\n3. inspect `result_status`, `approval_id`, `transaction_id`, and\n   `delegation_id`;\n4. compare `started_at`, `updated_at`, and `completed_at`;\n5. inspect the redacted `retry` and `lease` metadata;\n6. use the result, chunks, receipts, and typed links only after the identity and\n   state match the incident.\n\nAn empty result does not mean the action was never accepted. A queue record,\napproval, transaction, event, or callback may still own durable progress.\n\n## 3. Interpret the lifecycle state\n\nThe stable action view defines these states:\n\n| State | Meaning for recovery | Safe next action |\n|---|---|---|\n| `unknown` | The authorized runtime has no matching view | Recheck the endpoint, tenant, owner, and exact ID before considering a new action |\n| `accepted` | The protocol state allows accepted work before queue entry | Inspect events and audit evidence; the reviewed queue projection does not normally produce this state |\n| `queued` | Work is waiting, delayed for retry, or available after a lease | Inspect retry time, lease expiry, worker progress, and last error |\n| `running` | A worker owns the current attempt | Preserve the lease boundary and wait or follow events unless the lease has expired |\n| `streaming` | Ordered chunks exist before the terminal result | Resume from the last processed cursor and continue to wait for a terminal view |\n| `pending_approval` | The same action is parked behind durable governance | Inspect the approval record; do not submit a replacement action |\n| `cancelling` | The protocol state allows cancellation in progress | Treat the external outcome as unresolved until a terminal record exists |\n| `cancelled` | Runtime work ended as cancelled | Verify whether any provider effect preceded cancellation |\n| `completed` | A completed result is durable | Verify receipts, delivery, and provider evidence; do not rerun for a missing callback |\n| `failed` | The action has a terminal failed result | Classify the typed error and provider outcome before deciding on new work |\n| `expired` | Retention or queue policy expired the action | Preserve remaining evidence and decide through the owning policy |\n| `dead_lettered` | Automatic action recovery has stopped | Preserve the original record and require an explicit, reviewed recovery decision |\n\n`accepted` and `cancelling` are part of the stable lifecycle contract. The\nreviewed runtime projection does not currently map its stored queue and result\nrecords to those two states. Do not create an alert that assumes every defined\nstate must be observed in this implementation.\n\n## 4. Follow retry, lease, and event evidence\n\nFor a queued or running action, use the read model rather than database rows:\n\n- `retry` identifies attempts, retry policy, the next attempt time, and the\n  last redacted error when those values exist;\n- `lease` identifies the current worker and lease expiry without exposing the\n  internal fencing operation;\n- `updated_at` shows the last durable state transition;\n- `queued_state` preserves the backend queue label when it differs from the\n  stable lifecycle view.\n\nAn unexpired lease belongs to its current worker. An expired recoverable lease\ncan be claimed by the runtime worker using a new fencing token. Never extend,\nrelease, or transfer a lease with a manual database update. A stale worker is\nnot allowed to settle an action after losing its lease.\n\nRead action-scoped events with an opaque cursor:\n\n```sh\ngetaip \\\n  --native-bearer-token-file \"$AIP_TOKEN_FILE\" \\\n  action events \"$AIP_BASE_URL\" \"$ACTION_ID\" \\\n  --limit 100 \\\n  --include-chunks\n```\n\nPersist the returned cursor after successfully processing the page. Supply it\nwith `--cursor` for the next request, or use `--follow` for the HTTP event\nstream. Consumers must tolerate replay after reconnect and deduplicate by the\nevent identity or ordered chunk sequence.\n\n## 5. Resolve approval and transaction branches\n\nWhen the action exposes `approval_id`, read the approval and its related\naction state:\n\n```sh\nexport APPROVAL_ID=appr_example\n\ngetaip \\\n  --native-bearer-token-file \"$AIP_TOKEN_FILE\" \\\n  approval get \"$AIP_BASE_URL\" \"$APPROVAL_ID\" \\\n  --include-action-status \\\n  --include-receipts\n```\n\n`pending` waits for an authorized decision. `approved` permits the frozen\naction to continue through the durable transition path. `denied`, `expired`,\nand `revoked` are terminal approval decisions. Do not copy an approval result\ninto a new action or alter the frozen payload to bypass its policy hash.\n\nWhen the action exposes `transaction_id`, read the transaction separately:\n\n```sh\nexport TRANSACTION_ID=txn_example\n\ngetaip \\\n  --native-bearer-token-file \"$AIP_TOKEN_FILE\" \\\n  transaction get \"$AIP_BASE_URL\" \\\n  --transaction-id \"$TRANSACTION_ID\" \\\n  --include-result \\\n  --include-receipts\n```\n\nFor `outcome_unknown`, preserve the provider operation reference and\nreconciliation cursor. Allow reconciliation to establish whether the provider\naccepted the original operation. Do not submit another commit. Compensation is\na separate governed action and is available only when the capability contract\ndeclares it.\n\n## 6. Separate execution from delivery\n\nA completed action and its callback delivery have independent durable states.\nUse an authorized native HTTP client to list\n`GET /aip/v1/callback-deliveries` by action, status, profile, target, or\ntenant. Read one delivery at\n`GET /aip/v1/callback-deliveries/{delivery_id}`.\n\nInspect the delivery policy, status, attempt history, `next_attempt_at`,\n`leased_by`, `lease_expires_at`, `last_error`,\n`dead_letter_reason`, and optional receipt chain. The stable callback states\nare `pending`, `running`, `delivered`, `failed`, and\n`dead_lettered`. The reviewed dispatcher returns failed attempts to\n`pending` while retry budget remains and records exhausted delivery as\n`dead_lettered`; it does not normally publish `failed` as its final stored\nstate.\n\nPending delivery or a running delivery with an expired lease is recoverable by\nthe runtime worker. A dead-lettered callback is terminal for automatic\ndelivery, and this revision exposes no public callback replay command. Restore\nthe target first, preserve the record, and escalate through a reviewed\ndelivery-recovery procedure. Never rerun the action merely to recreate its\ncallback.\n\nRemote delegation also uses a durable outbox with `pending`, `leased`,\n`delivered`, and `dead_lettered` states. At this revision, the action view\ncan expose `delegation_id`, but `getaip-server` has no public HTTP or `getaip`\nquery for the delegation outbox record itself. Use action events, audit\nevidence, and peer observations; do not edit the outbox database to force a\nretry.\n\n## 7. Diagnose a fleet route without repinning work\n\nWhen the action targets a standalone connector host, compare three views:\n\n1. the gateway's `fleet` readiness snapshot and maintenance error;\n2. tenant-scoped capability discovery for new route availability;\n3. the authorized registry view of the action's durable assignment, together\n   with action events and provider evidence.\n\nThe public `ActionStatus` view does not expose the complete route assignment at\nthis revision. Use a controlled registry diagnostic or retained signed\nconnector request to identify the pinned route. Current capability discovery\ndescribes routes for new work and cannot prove where an existing action ran.\n\nReady-replica and active-assignment metrics describe fleet supply and activity.\nThey do not identify the outcome of one action. An already accepted action can\nremain pinned to its original connector type, version, instance, and replica\nwhile discovery sends new actions elsewhere.\n\nKeep the original replica and its durable state available until every assigned\nmutation is terminal or reconciled. Do not change the route of an ambiguous\naction to a healthy replica; that can repeat an external effect. Follow the\n[connector-host lifecycle](connector-host-lifecycle.md) for drain, lease loss,\nand replacement.\n\n## 8. Choose the recovery action\n\n| Observation | Recovery | Do not |\n|---|---|---|\n| Runtime worker is down or repeatedly errors | Restore its dependency, then restart the same reviewed process against the same durable store | Start a clean runtime with empty state |\n| Queued action has no live lease | Let the worker claim the existing action after readiness returns | Create a new action ID |\n| Running action has an unexpired lease | Wait, follow events, and inspect the provider timeout boundary | Force a second worker to execute it |\n| Running action has an expired lease | Let durable recovery claim it with a new fenced lease | Rewrite the lease owner or expiry |\n| Retry is scheduled | Preserve the original action and wait until `next_attempt_at` | Bypass backoff with a duplicate submission |\n| Approval is pending | Complete or expire the existing approval through its authority path | Change the frozen input or fabricate approval |\n| Transaction is `outcome_unknown` | Reconcile by provider operation reference and cursor | Retry commit or compensate before outcome is known |\n| Callback remains pending after target recovery | Let the callback worker recover its existing delivery | Rerun the completed action |\n| Action or delivery is dead-lettered | Preserve evidence and require an explicit policy decision | Delete the record to clear an alert |\n| Fleet has no route for new work | Keep the binding disabled or pause affected traffic until a ready route exists | Repin an already accepted ambiguous action |\n\nThe daemon's worker recovery covers pending approvals, recoverable\ntransactions, queued or running asynchronous actions, running delegations, and\nrecoverable callback deliveries. Recovery reuses the existing durable\nidentities. It does not make an unknown provider outcome safe to execute again.\n\n## 9. Use metrics for health, not action truth\n\nThe reviewed source emits useful health and capacity signals, including:\n\n| Signal group | Examples | Use |\n|---|---|---|\n| Core progress | `aip_daemon_ready`, `aip_runtime_worker_up`, `aip_runtime_worker_cycles_total` | Detect a daemon or worker that cannot make progress |\n| Optional NATS | `aip_nats_listener_up` | Correlate required listener state with readiness |\n| Work budgets | `aip_runtime_callback_capacity`, `aip_runtime_callback_available`, `aip_runtime_reconciliation_capacity`, `aip_runtime_reconciliation_available` | Detect exhausted callback or reconciliation concurrency |\n| Remote dispatch | `aip_connector_remote_active`, `aip_connector_remote_queued`, `aip_connector_remote_queued_bytes` | Observe in-process remote scheduling pressure |\n| Fleet | `aip_connector_fleet_ready_replicas`, `aip_connector_fleet_active_assignments`, `aip_connector_fleet_maintenance_failures_total` | Observe route supply, pinned work, and registry maintenance failure |\n\nThis revision does not emit dedicated gauges for the durable action queue,\nindividual leases, the delegation outbox, or execution checkpoints.\nExecution-checkpoint observers are process-local notifications and default to a\nno-op observer; they are not AIP wire records. Use the action, transaction,\ncallback, event, receipt, and audit read models for durable state. The\n[metrics reference](../reference/metrics.md) owns the complete metric catalog\nand label contract.\n\n## 10. Verify recovery\n\nRecovery is complete only when all applicable evidence agrees:\n\n1. `/ready` returns `200`, the runtime worker advances, and the relevant\n   fleet or local connector view has no blocking error;\n2. the original action reaches the expected durable state without a replacement\n   action ID;\n3. approval, transaction, delegation, and callback records agree with the\n   action;\n4. an `outcome_unknown` transaction has a reconciled terminal outcome;\n5. stream or event consumption resumes from the stored cursor;\n6. provider-side evidence identifies one intended effect;\n7. receipts and authorized audit records preserve the incident and recovery\n   sequence.\n\nIf these views disagree, keep the mutation boundary paused and treat the\nincident as unresolved. Read [errors and retry\ndecisions](../reference/errors.md) before authorizing any new state-changing\naction.\n\n## Related documentation\n\n- [Operator runbooks](operator-runbooks.md)\n- [Transactions and compensation](../concepts/transactions-and-compensation.md)\n- [Production deployment](production-deployment.md)\n",
    "text": "Observe and recover AIP work\n\nUse this guide when an AIP action is delayed, interrupted, or has an uncertain\nexternal outcome. You will establish whether the process can make progress,\nread the durable action and delivery records, choose a recovery action, and\nverify that recovery without submitting the same provider mutation twice.\n\nThis procedure applies to the product-neutral getaip-server runtime and its optional\nconnector fleet at source revision\nd7cce13d1d555644d04a4d73c66c95b113737635. Product-specific failures still\nrequire the runbook for the selected Cal.diy, Hermes Agent, Chatwoot, Dify,\nCrewAI, or Twenty connector.\n\nPreserve the incident boundary\n\nBefore changing configuration or restarting a process, record:\n• the action, capability, session, correlation, approval, transaction, and\n  delegation identifiers that are available;\n• the authenticated principal and tenant used for the operational read;\n• the original idempotency key and any provider operation reference;\n• the first observed failure time and the last known successful state update;\n• the gateway, connector type, instance, version, and replica involved, when\n  the action used the fleet;\n• the current readiness response and a metrics snapshot.\n\nPause new mutations for the affected tenant or capability when the blast radius\nis not yet known. Keep reads and evidence collection available.\n\nDo not submit a replacement action, edit a lease, delete an outbox record, or\nchange an action's route assignment to make the incident disappear. Logs,\ntraces, and metrics explain behavior; durable protocol and runtime records own\nthe current action state.\n1. Check liveness and readiness\n\nQuery the deployment through its controlled observability boundary:\n\nexport AIPBASEURL=https://aip.example.com\n\ncurl --fail --silent --show-error \"$AIPBASEURL/health\"\ncurl --fail --silent --show-error \"$AIPBASEURL/ready\"\ncurl --fail --silent --show-error \"$AIPBASEURL/metrics\" >aip-metrics.prom\n\n/health establishes only that the HTTP process is alive. /ready returns\nHTTP 200 when the core daemon is ready and 503 otherwise. Interpret the\nreadiness body by owner:\n\n| Readiness area | What to inspect | Decision |\n\n| gateway | ready, storage, and local connector checks | Restore storage read/write access or a required local connector before accepting traffic |\n| supervisor | runtimeworkerrunning, workercycles, lastworkererror, and lastworkersuccessat | A live HTTP process with a stopped or failing worker cannot recover queued work |\n| NATS | natsrequired and natslistenerrunning together | A stopped listener blocks readiness only when NATS is required |\n| modules | Every module marked required | Resolve the required module failure; optional modules do not control top-level readiness |\n| fleet | Worker state, consecutive failures, last error, last success, and summary | Diagnose registry maintenance and route supply separately from core readiness |\n\nGateway readiness performs a runtime-storage read/write check and probes\nregistered local connectors. Remote fleet replicas are not local gateway\nconnectors. The optional fleet snapshot is included in the response, but it\ndoes not participate in the top-level ready calculation at this revision.\nAn HTTP 200 can therefore coexist with no ready remote replica for a\nparticular tenant capability.\n\nDo not interpret natslistenerrunning: false as a fault when\nnatsrequired is false. Do not restart solely because one counter has\nstopped; compare the worker state, last error, last success time, and the\ndurable work described below.\n2. Read the durable action view\n\nUse an identity authorized for the same tenant and action owner. Prefer the\nclient token file rather than placing a bearer token in process arguments:\n\nexport AIPTOKENFILE=/run/secrets/aip-native-token\n\ngetaip \\\n  --native-bearer-token-file \"$AIPTOKENFILE\" \\\n  action list \"$AIPBASEURL\" \\\n  --state queued \\\n  --limit 100\n\nChange the state or identity filters only when they narrow the incident. Once\nyou have the action ID, request the complete operational view:\n\nexport ACTIONID=actexample\n\ngetaip \\\n  --native-bearer-token-file \"$AIPTOKENFILE\" \\\n  action status \"$AIPBASEURL\" \"$ACTIONID\" \\\n  --include-result \\\n  --include-chunks \\\n  --include-receipts\n\nRead the fields in this order:\n1. confirm actionid, capabilityid, and the tenant-visible ownership;\n2. compare the stable state with the backend queuedstate;\n3. inspect resultstatus, approvalid, transactionid, and\n   delegationid;\n4. compare startedat, updatedat, and completedat;\n5. inspect the redacted retry and lease metadata;\n6. use the result, chunks, receipts, and typed links only after the identity and\n   state match the incident.\n\nAn empty result does not mean the action was never accepted. A queue record,\napproval, transaction, event, or callback may still own durable progress.\n3. Interpret the lifecycle state\n\nThe stable action view defines these states:\n\n| State | Meaning for recovery | Safe next action |\n\n| unknown | The authorized runtime has no matching view | Recheck the endpoint, tenant, owner, and exact ID before considering a new action |\n| accepted | The protocol state allows accepted work before queue entry | Inspect events and audit evidence; the reviewed queue projection does not normally produce this state |\n| queued | Work is waiting, delayed for retry, or available after a lease | Inspect retry time, lease expiry, worker progress, and last error |\n| running | A worker owns the current attempt | Preserve the lease boundary and wait or follow events unless the lease has expired |\n| streaming | Ordered chunks exist before the terminal result | Resume from the last processed cursor and continue to wait for a terminal view |\n| pendingapproval | The same action is parked behind durable governance | Inspect the approval record; do not submit a replacement action |\n| cancelling | The protocol state allows cancellation in progress | Treat the external outcome as unresolved until a terminal record exists |\n| cancelled | Runtime work ended as cancelled | Verify whether any provider effect preceded cancellation |\n| completed | A completed result is durable | Verify receipts, delivery, and provider evidence; do not rerun for a missing callback |\n| failed | The action has a terminal failed result | Classify the typed error and provider outcome before deciding on new work |\n| expired | Retention or queue policy expired the action | Preserve remaining evidence and decide through the owning policy |\n| deadlettered | Automatic action recovery has stopped | Preserve the original record and require an explicit, reviewed recovery decision |\n\naccepted and cancelling are part of the stable lifecycle contract. The\nreviewed runtime projection does not currently map its stored queue and result\nrecords to those two states. Do not create an alert that assumes every defined\nstate must be observed in this implementation.\n4. Follow retry, lease, and event evidence\n\nFor a queued or running action, use the read model rather than database rows:\n• retry identifies attempts, retry policy, the next attempt time, and the\n  last redacted error when those values exist;\n• lease identifies the current worker and lease expiry without exposing the\n  internal fencing operation;\n• updatedat shows the last durable state transition;\n• queuedstate preserves the backend queue label when it differs from the\n  stable lifecycle view.\n\nAn unexpired lease belongs to its current worker. An expired recoverable lease\ncan be claimed by the runtime worker using a new fencing token. Never extend,\nrelease, or transfer a lease with a manual database update. A stale worker is\nnot allowed to settle an action after losing its lease.\n\nRead action-scoped events with an opaque cursor:\n\ngetaip \\\n  --native-bearer-token-file \"$AIPTOKENFILE\" \\\n  action events \"$AIPBASEURL\" \"$ACTIONID\" \\\n  --limit 100 \\\n  --include-chunks\n\nPersist the returned cursor after successfully processing the page. Supply it\nwith --cursor for the next request, or use --follow for the HTTP event\nstream. Consumers must tolerate replay after reconnect and deduplicate by the\nevent identity or ordered chunk sequence.\n5. Resolve approval and transaction branches\n\nWhen the action exposes approvalid, read the approval and its related\naction state:\n\nexport APPROVALID=apprexample\n\ngetaip \\\n  --native-bearer-token-file \"$AIPTOKENFILE\" \\\n  approval get \"$AIPBASEURL\" \"$APPROVALID\" \\\n  --include-action-status \\\n  --include-receipts\n\npending waits for an authorized decision. approved permits the frozen\naction to continue through the durable transition path. denied, expired,\nand revoked are terminal approval decisions. Do not copy an approval result\ninto a new action or alter the frozen payload to bypass its policy hash.\n\nWhen the action exposes transactionid, read the transaction separately:\n\nexport TRANSACTIONID=txnexample\n\ngetaip \\\n  --native-bearer-token-file \"$AIPTOKENFILE\" \\\n  transaction get \"$AIPBASEURL\" \\\n  --transaction-id \"$TRANSACTIONID\" \\\n  --include-result \\\n  --include-receipts\n\nFor outcomeunknown, preserve the provider operation reference and\nreconciliation cursor. Allow reconciliation to establish whether the provider\naccepted the original operation. Do not submit another commit. Compensation is\na separate governed action and is available only when the capability contract\ndeclares it.\n6. Separate execution from delivery\n\nA completed action and its callback delivery have independent durable states.\nUse an authorized native HTTP client to list\nGET /aip/v1/callback-deliveries by action, status, profile, target, or\ntenant. Read one delivery at\nGET /aip/v1/callback-deliveries/{deliveryid}.\n\nInspect the delivery policy, status, attempt history, nextattemptat,\nleasedby, leaseexpiresat, lasterror,\ndeadletterreason, and optional receipt chain. The stable callback states\nare pending, running, delivered, failed, and\ndeadlettered. The reviewed dispatcher returns failed attempts to\npending while retry budget remains and records exhausted delivery as\ndeadlettered; it does not normally publish failed as its final stored\nstate.\n\nPending delivery or a running delivery with an expired lease is recoverable by\nthe runtime worker. A dead-lettered callback is terminal for automatic\ndelivery, and this revision exposes no public callback replay command. Restore\nthe target first, preserve the record, and escalate through a reviewed\ndelivery-recovery procedure. Never rerun the action merely to recreate its\ncallback.\n\nRemote delegation also uses a durable outbox with pending, leased,\ndelivered, and deadlettered states. At this revision, the action view\ncan expose delegationid, but getaip-server has no public HTTP or getaip\nquery for the delegation outbox record itself. Use action events, audit\nevidence, and peer observations; do not edit the outbox database to force a\nretry.\n7. Diagnose a fleet route without repinning work\n\nWhen the action targets a standalone connector host, compare three views:\n1. the gateway's fleet readiness snapshot and maintenance error;\n2. tenant-scoped capability discovery for new route availability;\n3. the authorized registry view of the action's durable assignment, together\n   with action events and provider evidence.\n\nThe public ActionStatus view does not expose the complete route assignment at\nthis revision. Use a controlled registry diagnostic or retained signed\nconnector request to identify the pinned route. Current capability discovery\ndescribes routes for new work and cannot prove where an existing action ran.\n\nReady-replica and active-assignment metrics describe fleet supply and activity.\nThey do not identify the outcome of one action. An already accepted action can\nremain pinned to its original connector type, version, instance, and replica\nwhile discovery sends new actions elsewhere.\n\nKeep the original replica and its durable state available until every assigned\nmutation is terminal or reconciled. Do not change the route of an ambiguous\naction to a healthy replica; that can repeat an external effect. Follow the\nconnector-host lifecycle (connector-host-lifecycle.md) for drain, lease loss,\nand replacement.\n8. Choose the recovery action\n\n| Observation | Recovery | Do not |\n\n| Runtime worker is down or repeatedly errors | Restore its dependency, then restart the same reviewed process against the same durable store | Start a clean runtime with empty state |\n| Queued action has no live lease | Let the worker claim the existing action after readiness returns | Create a new action ID |\n| Running action has an unexpired lease | Wait, follow events, and inspect the provider timeout boundary | Force a second worker to execute it |\n| Running action has an expired lease | Let durable recovery claim it with a new fenced lease | Rewrite the lease owner or expiry |\n| Retry is scheduled | Preserve the original action and wait until nextattemptat | Bypass backoff with a duplicate submission |\n| Approval is pending | Complete or expire the existing approval through its authority path | Change the frozen input or fabricate approval |\n| Transaction is outcomeunknown | Reconcile by provider operation reference and cursor | Retry commit or compensate before outcome is known |\n| Callback remains pending after target recovery | Let the callback worker recover its existing delivery | Rerun the completed action |\n| Action or delivery is dead-lettered | Preserve evidence and require an explicit policy decision | Delete the record to clear an alert |\n| Fleet has no route for new work | Keep the binding disabled or pause affected traffic until a ready route exists | Repin an already accepted ambiguous action |\n\nThe daemon's worker recovery covers pending approvals, recoverable\ntransactions, queued or running asynchronous actions, running delegations, and\nrecoverable callback deliveries. Recovery reuses the existing durable\nidentities. It does not make an unknown provider outcome safe to execute again.\n9. Use metrics for health, not action truth\n\nThe reviewed source emits useful health and capacity signals, including:\n\n| Signal group | Examples | Use |\n\n| Core progress | aipdaemonready, aipruntimeworkerup, aipruntimeworkercyclestotal | Detect a daemon or worker that cannot make progress |\n| Optional NATS | aipnatslistenerup | Correlate required listener state with readiness |\n| Work budgets | aipruntimecallbackcapacity, aipruntimecallbackavailable, aipruntimereconciliationcapacity, aipruntimereconciliationavailable | Detect exhausted callback or reconciliation concurrency |\n| Remote dispatch | aipconnectorremoteactive, aipconnectorremotequeued, aipconnectorremotequeuedbytes | Observe in-process remote scheduling pressure |\n| Fleet | aipconnectorfleetreadyreplicas, aipconnectorfleetactiveassignments, aipconnectorfleetmaintenancefailurestotal | Observe route supply, pinned work, and registry maintenance failure |\n\nThis revision does not emit dedicated gauges for the durable action queue,\nindividual leases, the delegation outbox, or execution checkpoints.\nExecution-checkpoint observers are process-local notifications and default to a\nno-op observer; they are not AIP wire records. Use the action, transaction,\ncallback, event, receipt, and audit read models for durable state. The\nmetrics reference (../reference/metrics.md) owns the complete metric catalog\nand label contract.\n10. Verify recovery\n\nRecovery is complete only when all applicable evidence agrees:\n1. /ready returns 200, the runtime worker advances, and the relevant\n   fleet or local connector view has no blocking error;\n2. the original action reaches the expected durable state without a replacement\n   action ID;\n3. approval, transaction, delegation, and callback records agree with the\n   action;\n4. an outcomeunknown transaction has a reconciled terminal outcome;\n5. stream or event consumption resumes from the stored cursor;\n6. provider-side evidence identifies one intended effect;\n7. receipts and authorized audit records preserve the incident and recovery\n   sequence.\n\nIf these views disagree, keep the mutation boundary paused and treat the\nincident as unresolved. Read errors and retry\ndecisions (../reference/errors.md) before authorizing any new state-changing\naction.\n\nRelated documentation\n• Operator runbooks (operator-runbooks.md)\n• Transactions and compensation (../concepts/transactions-and-compensation.md)\n• Production deployment (production-deployment.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "e97aea67a589d41e996b402da4ba04cf0e9d2dcf2367cfdf86984551c53b1c89"
  }
}