Observe and recover AIP work

Use this guide when an AIP action is delayed, interrupted, or has an uncertain
external outcome. You will establish whether the process can make progress,
read the durable action and delivery records, choose a recovery action, and
verify that recovery without submitting the same provider mutation twice.

This procedure applies to the product-neutral getaip-server runtime and its optional
connector fleet at source revision
d7cce13d1d555644d04a4d73c66c95b113737635. Product-specific failures still
require the runbook for the selected Cal.diy, Hermes Agent, Chatwoot, Dify,
CrewAI, or Twenty connector.

Preserve the incident boundary

Before changing configuration or restarting a process, record:
• the action, capability, session, correlation, approval, transaction, and
  delegation identifiers that are available;
• the authenticated principal and tenant used for the operational read;
• the original idempotency key and any provider operation reference;
• the first observed failure time and the last known successful state update;
• the gateway, connector type, instance, version, and replica involved, when
  the action used the fleet;
• the current readiness response and a metrics snapshot.

Pause new mutations for the affected tenant or capability when the blast radius
is not yet known. Keep reads and evidence collection available.

Do not submit a replacement action, edit a lease, delete an outbox record, or
change an action's route assignment to make the incident disappear. Logs,
traces, and metrics explain behavior; durable protocol and runtime records own
the current action state.
1. Check liveness and readiness

Query the deployment through its controlled observability boundary:

export AIPBASEURL=https://aip.example.com

curl --fail --silent --show-error "$AIPBASEURL/health"
curl --fail --silent --show-error "$AIPBASEURL/ready"
curl --fail --silent --show-error "$AIPBASEURL/metrics" >aip-metrics.prom

/health establishes only that the HTTP process is alive. /ready returns
HTTP 200 when the core daemon is ready and 503 otherwise. Interpret the
readiness body by owner:

| Readiness area | What to inspect | Decision |

| gateway | ready, storage, and local connector checks | Restore storage read/write access or a required local connector before accepting traffic |
| supervisor | runtimeworkerrunning, workercycles, lastworkererror, and lastworkersuccessat | A live HTTP process with a stopped or failing worker cannot recover queued work |
| NATS | natsrequired and natslistenerrunning together | A stopped listener blocks readiness only when NATS is required |
| modules | Every module marked required | Resolve the required module failure; optional modules do not control top-level readiness |
| fleet | Worker state, consecutive failures, last error, last success, and summary | Diagnose registry maintenance and route supply separately from core readiness |

Gateway readiness performs a runtime-storage read/write check and probes
registered local connectors. Remote fleet replicas are not local gateway
connectors. The optional fleet snapshot is included in the response, but it
does not participate in the top-level ready calculation at this revision.
An HTTP 200 can therefore coexist with no ready remote replica for a
particular tenant capability.

Do not interpret natslistenerrunning: false as a fault when
natsrequired is false. Do not restart solely because one counter has
stopped; compare the worker state, last error, last success time, and the
durable work described below.
2. Read the durable action view

Use an identity authorized for the same tenant and action owner. Prefer the
client token file rather than placing a bearer token in process arguments:

export AIPTOKENFILE=/run/secrets/aip-native-token

getaip \
  --native-bearer-token-file "$AIPTOKENFILE" \
  action list "$AIPBASEURL" \
  --state queued \
  --limit 100

Change the state or identity filters only when they narrow the incident. Once
you have the action ID, request the complete operational view:

export ACTIONID=actexample

getaip \
  --native-bearer-token-file "$AIPTOKENFILE" \
  action status "$AIPBASEURL" "$ACTIONID" \
  --include-result \
  --include-chunks \
  --include-receipts

Read the fields in this order:
1. confirm actionid, capabilityid, and the tenant-visible ownership;
2. compare the stable state with the backend queuedstate;
3. inspect resultstatus, approvalid, transactionid, and
   delegationid;
4. compare startedat, updatedat, and completedat;
5. inspect the redacted retry and lease metadata;
6. use the result, chunks, receipts, and typed links only after the identity and
   state match the incident.

An empty result does not mean the action was never accepted. A queue record,
approval, transaction, event, or callback may still own durable progress.
3. Interpret the lifecycle state

The stable action view defines these states:

| State | Meaning for recovery | Safe next action |

| unknown | The authorized runtime has no matching view | Recheck the endpoint, tenant, owner, and exact ID before considering a new action |
| accepted | The protocol state allows accepted work before queue entry | Inspect events and audit evidence; the reviewed queue projection does not normally produce this state |
| queued | Work is waiting, delayed for retry, or available after a lease | Inspect retry time, lease expiry, worker progress, and last error |
| running | A worker owns the current attempt | Preserve the lease boundary and wait or follow events unless the lease has expired |
| streaming | Ordered chunks exist before the terminal result | Resume from the last processed cursor and continue to wait for a terminal view |
| pendingapproval | The same action is parked behind durable governance | Inspect the approval record; do not submit a replacement action |
| cancelling | The protocol state allows cancellation in progress | Treat the external outcome as unresolved until a terminal record exists |
| cancelled | Runtime work ended as cancelled | Verify whether any provider effect preceded cancellation |
| completed | A completed result is durable | Verify receipts, delivery, and provider evidence; do not rerun for a missing callback |
| failed | The action has a terminal failed result | Classify the typed error and provider outcome before deciding on new work |
| expired | Retention or queue policy expired the action | Preserve remaining evidence and decide through the owning policy |
| deadlettered | Automatic action recovery has stopped | Preserve the original record and require an explicit, reviewed recovery decision |

accepted and cancelling are part of the stable lifecycle contract. The
reviewed runtime projection does not currently map its stored queue and result
records to those two states. Do not create an alert that assumes every defined
state must be observed in this implementation.
4. Follow retry, lease, and event evidence

For a queued or running action, use the read model rather than database rows:
• retry identifies attempts, retry policy, the next attempt time, and the
  last redacted error when those values exist;
• lease identifies the current worker and lease expiry without exposing the
  internal fencing operation;
• updatedat shows the last durable state transition;
• queuedstate preserves the backend queue label when it differs from the
  stable lifecycle view.

An unexpired lease belongs to its current worker. An expired recoverable lease
can be claimed by the runtime worker using a new fencing token. Never extend,
release, or transfer a lease with a manual database update. A stale worker is
not allowed to settle an action after losing its lease.

Read action-scoped events with an opaque cursor:

getaip \
  --native-bearer-token-file "$AIPTOKENFILE" \
  action events "$AIPBASEURL" "$ACTIONID" \
  --limit 100 \
  --include-chunks

Persist the returned cursor after successfully processing the page. Supply it
with --cursor for the next request, or use --follow for the HTTP event
stream. Consumers must tolerate replay after reconnect and deduplicate by the
event identity or ordered chunk sequence.
5. Resolve approval and transaction branches

When the action exposes approvalid, read the approval and its related
action state:

export APPROVALID=apprexample

getaip \
  --native-bearer-token-file "$AIPTOKENFILE" \
  approval get "$AIPBASEURL" "$APPROVALID" \
  --include-action-status \
  --include-receipts

pending waits for an authorized decision. approved permits the frozen
action to continue through the durable transition path. denied, expired,
and revoked are terminal approval decisions. Do not copy an approval result
into a new action or alter the frozen payload to bypass its policy hash.

When the action exposes transactionid, read the transaction separately:

export TRANSACTIONID=txnexample

getaip \
  --native-bearer-token-file "$AIPTOKENFILE" \
  transaction get "$AIPBASEURL" \
  --transaction-id "$TRANSACTIONID" \
  --include-result \
  --include-receipts

For outcomeunknown, preserve the provider operation reference and
reconciliation cursor. Allow reconciliation to establish whether the provider
accepted the original operation. Do not submit another commit. Compensation is
a separate governed action and is available only when the capability contract
declares it.
6. Separate execution from delivery

A completed action and its callback delivery have independent durable states.
Use an authorized native HTTP client to list
GET /aip/v1/callback-deliveries by action, status, profile, target, or
tenant. Read one delivery at
GET /aip/v1/callback-deliveries/{deliveryid}.

Inspect the delivery policy, status, attempt history, nextattemptat,
leasedby, leaseexpiresat, lasterror,
deadletterreason, and optional receipt chain. The stable callback states
are pending, running, delivered, failed, and
deadlettered. The reviewed dispatcher returns failed attempts to
pending while retry budget remains and records exhausted delivery as
deadlettered; it does not normally publish failed as its final stored
state.

Pending delivery or a running delivery with an expired lease is recoverable by
the runtime worker. A dead-lettered callback is terminal for automatic
delivery, and this revision exposes no public callback replay command. Restore
the target first, preserve the record, and escalate through a reviewed
delivery-recovery procedure. Never rerun the action merely to recreate its
callback.

Remote delegation also uses a durable outbox with pending, leased,
delivered, and deadlettered states. At this revision, the action view
can expose delegationid, but getaip-server has no public HTTP or getaip
query for the delegation outbox record itself. Use action events, audit
evidence, and peer observations; do not edit the outbox database to force a
retry.
7. Diagnose a fleet route without repinning work

When the action targets a standalone connector host, compare three views:
1. the gateway's fleet readiness snapshot and maintenance error;
2. tenant-scoped capability discovery for new route availability;
3. the authorized registry view of the action's durable assignment, together
   with action events and provider evidence.

The public ActionStatus view does not expose the complete route assignment at
this revision. Use a controlled registry diagnostic or retained signed
connector request to identify the pinned route. Current capability discovery
describes routes for new work and cannot prove where an existing action ran.

Ready-replica and active-assignment metrics describe fleet supply and activity.
They do not identify the outcome of one action. An already accepted action can
remain pinned to its original connector type, version, instance, and replica
while discovery sends new actions elsewhere.

Keep the original replica and its durable state available until every assigned
mutation is terminal or reconciled. Do not change the route of an ambiguous
action to a healthy replica; that can repeat an external effect. Follow the
connector-host lifecycle (connector-host-lifecycle.md) for drain, lease loss,
and replacement.
8. Choose the recovery action

| Observation | Recovery | Do not |

| Runtime worker is down or repeatedly errors | Restore its dependency, then restart the same reviewed process against the same durable store | Start a clean runtime with empty state |
| Queued action has no live lease | Let the worker claim the existing action after readiness returns | Create a new action ID |
| Running action has an unexpired lease | Wait, follow events, and inspect the provider timeout boundary | Force a second worker to execute it |
| Running action has an expired lease | Let durable recovery claim it with a new fenced lease | Rewrite the lease owner or expiry |
| Retry is scheduled | Preserve the original action and wait until nextattemptat | Bypass backoff with a duplicate submission |
| Approval is pending | Complete or expire the existing approval through its authority path | Change the frozen input or fabricate approval |
| Transaction is outcomeunknown | Reconcile by provider operation reference and cursor | Retry commit or compensate before outcome is known |
| Callback remains pending after target recovery | Let the callback worker recover its existing delivery | Rerun the completed action |
| Action or delivery is dead-lettered | Preserve evidence and require an explicit policy decision | Delete the record to clear an alert |
| Fleet has no route for new work | Keep the binding disabled or pause affected traffic until a ready route exists | Repin an already accepted ambiguous action |

The daemon's worker recovery covers pending approvals, recoverable
transactions, queued or running asynchronous actions, running delegations, and
recoverable callback deliveries. Recovery reuses the existing durable
identities. It does not make an unknown provider outcome safe to execute again.
9. Use metrics for health, not action truth

The reviewed source emits useful health and capacity signals, including:

| Signal group | Examples | Use |

| Core progress | aipdaemonready, aipruntimeworkerup, aipruntimeworkercyclestotal | Detect a daemon or worker that cannot make progress |
| Optional NATS | aipnatslistenerup | Correlate required listener state with readiness |
| Work budgets | aipruntimecallbackcapacity, aipruntimecallbackavailable, aipruntimereconciliationcapacity, aipruntimereconciliationavailable | Detect exhausted callback or reconciliation concurrency |
| Remote dispatch | aipconnectorremoteactive, aipconnectorremotequeued, aipconnectorremotequeuedbytes | Observe in-process remote scheduling pressure |
| Fleet | aipconnectorfleetreadyreplicas, aipconnectorfleetactiveassignments, aipconnectorfleetmaintenancefailurestotal | Observe route supply, pinned work, and registry maintenance failure |

This revision does not emit dedicated gauges for the durable action queue,
individual leases, the delegation outbox, or execution checkpoints.
Execution-checkpoint observers are process-local notifications and default to a
no-op observer; they are not AIP wire records. Use the action, transaction,
callback, event, receipt, and audit read models for durable state. The
metrics reference (../reference/metrics.md) owns the complete metric catalog
and label contract.
10. Verify recovery

Recovery is complete only when all applicable evidence agrees:
1. /ready returns 200, the runtime worker advances, and the relevant
   fleet or local connector view has no blocking error;
2. the original action reaches the expected durable state without a replacement
   action ID;
3. approval, transaction, delegation, and callback records agree with the
   action;
4. an outcomeunknown transaction has a reconciled terminal outcome;
5. stream or event consumption resumes from the stored cursor;
6. provider-side evidence identifies one intended effect;
7. receipts and authorized audit records preserve the incident and recovery
   sequence.

If these views disagree, keep the mutation boundary paused and treat the
incident as unresolved. Read errors and retry
decisions (../reference/errors.md) before authorizing any new state-changing
action.

Related documentation
• Operator runbooks (operator-runbooks.md)
• Transactions and compensation (../concepts/transactions-and-compensation.md)
• Production deployment (production-deployment.md)