# Observe and Recover

AIP exposes an operational read model so an operator can answer what the system
knows without reconstructing truth from logs.

Use logs and traces to debug execution. Use action, approval, transaction,
callback, event, receipt, and audit records to determine protocol state.

## Start with Readiness

```sh
curl --fail http://127.0.0.1:18080/health
curl --fail http://127.0.0.1:18080/ready
curl --fail http://127.0.0.1:18080/metrics
```

An alive process can still be unready because storage, a registered connector,
the runtime worker, or the required NATS listener is unavailable. Invalid
identity or authority configuration fails daemon startup instead of becoming a
dynamic readiness component.

## Find the Action

List recent actions or filter by capability, session, principal, approval, or
transaction:

```sh
cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  action list https://aip.example.com \
  --state pending_approval \
  --limit 100
```

Read one action with result, stream chunks, and receipts:

```sh
cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  action status https://aip.example.com ACT_ID \
  --include-result \
  --include-chunks \
  --include-receipts
```

## Follow Events

Read action-scoped events:

```sh
cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  action events https://aip.example.com ACT_ID \
  --include-chunks \
  --follow
```

Cursors are opaque. Persist the last successfully processed cursor and tolerate
replayed events after reconnect.

## Diagnose Common States

| State | Meaning | Operator action |
|---|---|---|
| `queued` | Waiting for a worker lease | Check queue age, worker readiness, and lease metrics |
| `running` | A worker owns execution | Check deadline, heartbeat, provider request id, and lease renewal |
| `streaming` | Ordered chunks are being emitted | Check consumer cursor and terminal result |
| `pending_approval` | Governed work is parked | Inspect approval record, authority, TTL, and evidence hash |
| `cancelling` | Cancellation requested but not terminal | Check provider cancellation and outcome reconciliation |
| `dead_lettered` | Retry budget exhausted or recovery cannot proceed | Inspect typed error and decide replay, remediation, or compensation |
| `failed` | Terminal failure | Use error category and retry safety; do not retry blindly |

## Inspect Approval State

```sh
cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  approval get https://aip.example.com APPR_ID \
  --include-action-status \
  --include-receipts
```

Confirm that the decision is unexpired, the authority path is trusted, the
policy hash matches, and separation-of-duties rules were satisfied.

## Inspect Transaction State

```sh
cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  transaction get https://aip.example.com \
  --transaction-id TXN_ID \
  --include-result \
  --include-receipts
```

For `outcome_unknown`, find the provider operation id and allow the configured
reconciliation path to establish the result. Do not submit a new commit.

## Inspect Callback Delivery

Use `GET /aip/v1/callback-deliveries` or the message API to inspect pending,
running, delivered, failed, and dead-lettered callback records. Check attempt
history, next attempt time, lease owner, last error, and receipt chain.

A successful action and a failed callback are separate states. The action must
not be re-executed merely to redeliver its result.

## Verify Receipts

Fetch a chain:

```sh
cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  receipt get https://aip.example.com \
  --chain-id CHAIN_ID
```

Save the JSON response and verify its hash links:

```sh
cargo run -p aipctl -- receipt verify receipt-chain.json
```

Receipt integrity shows that the retained chain has not changed. It does not by
itself prove the actor was correctly authenticated; signer and identity evidence
must also be valid.

## Query Audit Evidence

```sh
cargo run -p aipctl -- \
  --native-bearer-token "$AIP_TOKEN" \
  audit events https://aip.example.com \
  --action-id ACT_ID \
  --include-receipts
```

Use `--export` only for an authorized evidence export. Normal browsing and
evidence export have different scopes because exported data may leave the
runtime trust boundary.

## Recovery Rules

- Let expired leases be recovered by the durable worker contract.
- Preserve the original action and idempotency key.
- Reconcile uncertain provider outcomes before retry.
- Resume approved actions from their frozen payload.
- Redeliver callbacks from the durable outbox without rerunning the action.
- Move irrecoverable work to dead letter with a typed reason and audit record.
- Use compensation only when the transaction contract declares it.