Operator Runbooks

These runbooks define first-response steps for common AIP incidents. They are starting procedures, not substitutes for deployment-specific ownership, alerting thresholds, backup policy, or change control.

Preserve action ids, transaction ids, provider operation references, cursors, and timestamps. Never paste raw credentials or governed payloads into an incident ticket.

Daemon Is Not Ready

1. Check /health; if it fails, treat the process as unavailable.
2. Read /ready and identify the failing storage, connector, worker, or NATS component.
3. Check that the runtime worker is running and the configured NATS listener is active when NATS is required.
4. Verify storage connectivity without bypassing the configured identity.
5. Inspect connector readiness individually in the readiness response.
6. Restore the dependency or remove the connector only through an approved deployment change.

Current aipd treats every registered connector as readiness-critical. Isolate optional connectors in another deployment when their outage must not block unrelated capabilities.

Do not switch to in-memory or file storage to make a clustered deployment look ready. That creates a second state owner.

Queue Age or Lease Failures Increase

1. List queued and running actions and record the oldest updated_at value.
2. Confirm worker liveness, lease renewal, database latency, and clock health.
3. Check whether actions are waiting for retry windows, approval, or provider recovery rather than worker capacity.
4. Allow expired leases to be reclaimed by the runtime fencing contract.
5. Do not manually clear a valid lease or copy a queued action into a new id.
6. Investigate dead-lettered work after the retry budget is exhausted.

Scale workers only after storage contention and provider rate limits are understood. More workers can make a downstream outage worse.

Action Is Stuck in pending_approval

1. Read the approval record and linked action.
2. Confirm the approval has not expired or been revoked.
3. Verify the expected authority rule, tenant, quorum, and separation of duties.
4. Compare policy, governed-input, action, and capability hashes.
5. Confirm the decision reached the durable approval outbox.
6. Let the runtime resume the original frozen action.

Never create a replacement commit with copied input to bypass a stuck approval.

Transaction Is outcome_unknown

1. Stop automatic retries and manual re-submission.
2. Read the retained provider operation id and reconciliation cursor.
3. Query provider state through the connector’s reconciliation operation.
4. Record the definitive provider outcome in the existing transaction.
5. Resume, fail, or compensate only after reconciliation is authoritative.
6. Escalate to a human when the provider offers no reliable lookup.

An unavailable provider response does not prove that a mutation failed.

Callback Is Dead-Lettered

1. Verify that the source action is terminal; do not rerun it.
2. Read callback attempt history, target, profile, next-attempt time, and last typed error.
3. Check HTTPS, DNS, allowlist, signature, receiver authentication, and response status.
4. Repair the delivery boundary.
5. Redeliver from the durable outbox under the same action and idempotency identity.

If the destination changed, treat that as a governed configuration change, not an edit to retained callback history.

Suspected Duplicate Side Effect

1. Freeze further retry for the idempotency key.
2. Compare action input hash, capability, principal, tenant, and external account scope.
3. Inspect provider request and operation ids.
4. Query provider facts directly through an authorized connector operation.
5. Determine whether duplicate delivery, stale lease execution, or provider behavior created the mutation.
6. Use compensation only when the capability contract permits it.

Do not delete the idempotency record before the investigation is complete.

Replay or Signature Alert

1. Preserve the rejected message id, signer DID, timestamp, source boundary, and redacted error.
2. Confirm clock synchronization and the configured replay window.
3. Check whether the message id was already claimed by the durable replay registry.
4. Verify signer-to-principal bindings and recent key rotation.
5. Revoke a compromised signer through the trusted binding source.
6. Do not add an unsigned exception to restore traffic.

Storage Failover or Restart

1. Stop admitting new mutating work if durable ownership is uncertain.
2. Complete the database failover using the deployment’s fencing procedure.
3. Start one aipd replica and wait for runtime recovery to finish.
4. Inspect the recovery report for approvals, transactions, queued actions, callbacks, and delegations.
5. Reconcile uncertain provider operations before scaling out.
6. Restore remaining replicas and verify lease ownership remains unique.

Evidence to Retain

• exact source and image digest;
• deployment revision and configuration class;
• action, approval, transaction, delegation, callback, and receipt ids;
• typed errors and provider correlation ids;
• relevant event cursors and timestamps;
• remediation command and approving operator;
• confirmation that secrets and governed payloads were redacted.

Use Observe and Recover for query commands and Errors and Retry Decisions for retry classification.