Observe and Recover

AIP exposes an operational read model so an operator can answer what the system knows without reconstructing truth from logs.

Use logs and traces to debug execution. Use action, approval, transaction, callback, event, receipt, and audit records to determine protocol state.

Start with Readiness

curl --fail http://127.0.0.1:18080/health
curl --fail http://127.0.0.1:18080/ready
curl --fail http://127.0.0.1:18080/metrics

An alive process can still be unready because storage, a registered connector, the runtime worker, or the required NATS listener is unavailable. Invalid identity or authority configuration fails daemon startup instead of becoming a dynamic readiness component.

Find the Action

List recent actions or filter by capability, session, principal, approval, or transaction:

cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  action list https://aip.example.com \
  --state pending_approval \
  --limit 100

Read one action with result, stream chunks, and receipts:

cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  action status https://aip.example.com ACT_ID \
  --include-result \
  --include-chunks \
  --include-receipts

Follow Events

Read action-scoped events:

cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  action events https://aip.example.com ACT_ID \
  --include-chunks \
  --follow

Cursors are opaque. Persist the last successfully processed cursor and tolerate replayed events after reconnect.

Diagnose Common States

State	Meaning	Operator action
queued	Waiting for a worker lease	Check queue age, worker readiness, and lease metrics
running	A worker owns execution	Check deadline, heartbeat, provider request id, and lease renewal
streaming	Ordered chunks are being emitted	Check consumer cursor and terminal result
pending_approval	Governed work is parked	Inspect approval record, authority, TTL, and evidence hash
cancelling	Cancellation requested but not terminal	Check provider cancellation and outcome reconciliation
dead_lettered	Retry budget exhausted or recovery cannot proceed	Inspect typed error and decide replay, remediation, or compensation
failed	Terminal failure	Use error category and retry safety; do not retry blindly

Inspect Approval State

cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  approval get https://aip.example.com APPR_ID \
  --include-action-status \
  --include-receipts

Confirm that the decision is unexpired, the authority path is trusted, the policy hash matches, and separation-of-duties rules were satisfied.

Inspect Transaction State

cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  transaction get https://aip.example.com \
  --transaction-id TXN_ID \
  --include-result \
  --include-receipts

For outcome_unknown, find the provider operation id and allow the configured reconciliation path to establish the result. Do not submit a new commit.

Inspect Callback Delivery

Use GET /aip/v1/callback-deliveries or the message API to inspect pending, running, delivered, failed, and dead-lettered callback records. Check attempt history, next attempt time, lease owner, last error, and receipt chain.

A successful action and a failed callback are separate states. The action must not be re-executed merely to redeliver its result.

Verify Receipts

Fetch a chain:

cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  receipt get https://aip.example.com \
  --chain-id CHAIN_ID

Save the JSON response and verify its hash links:

cargo run -p aipctl -- receipt verify receipt-chain.json

Receipt integrity shows that the retained chain has not changed. It does not by itself prove the actor was correctly authenticated; signer and identity evidence must also be valid.

Query Audit Evidence

cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  audit events https://aip.example.com \
  --action-id ACT_ID \
  --include-receipts

Use --export only for an authorized evidence export. Normal browsing and evidence export have different scopes because exported data may leave the runtime trust boundary.

Recovery Rules

• Let expired leases be recovered by the durable worker contract.
• Preserve the original action and idempotency key.
• Reconcile uncertain provider outcomes before retry.
• Resume approved actions from their frozen payload.
• Redeliver callbacks from the durable outbox without rerunning the action.
• Move irrecoverable work to dead letter with a typed reason and audit record.
• Use compensation only when the transaction contract declares it.