{
  "schemaVersion": "1.0",
  "title": "CrewAI health, observability, and recovery",
  "description": "Use this runbook when a CrewAI replica is unhealthy, not ready, losing its lease, or holding an Action without trustworthy terminal evidence. Preserve the host runtime and sidecar journal before changing the deployment.",
  "canonical": "https://getaip.org/docs/connectors/crewai/operations/health-observability-and-recovery",
  "route": "/docs/connectors/crewai/operations/health-observability-and-recovery",
  "source": "docs/connectors/crewai/operations/health-observability-and-recovery.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/crewai/operations/health-observability-and-recovery.md",
    "txt": "/docs/download/connectors/crewai/operations/health-observability-and-recovery.txt",
    "json": "/docs/download/connectors/crewai/operations/health-observability-and-recovery.json",
    "pdf": "/docs/download/connectors/crewai/operations/health-observability-and-recovery.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: CrewAI health, observability, and recovery\ndescription: >-\n  Diagnose CrewAI host, sidecar, lease, storage, and Action state, then recover\n  without discarding durable identity\nkind: runbook\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: crewai\naliases:\n  - CrewAI connector health\n  - CrewAI connector observability\n  - Recover the CrewAI connector\n---\n\n# CrewAI health, observability, and recovery\n\nUse this runbook when a CrewAI replica is unhealthy, not ready, losing its\nlease, or holding an Action without trustworthy terminal evidence. Preserve the\nhost runtime and sidecar journal before changing the deployment.\n\nFour states matter independently: process liveness, fleet routing readiness,\nsidecar durability, and provider outcome. A green probe at one boundary does\nnot prove the others.\n\n## Start with the four boundaries\n\n| Boundary | Primary evidence | Healthy meaning | Excluded meaning |\n|---|---|---|---|\n| Rust process | Host `GET /health` | Router responds with AIP host identity | Storage, sidecar, or lease readiness |\n| Fleet replica | Host `GET /ready` | Not draining, valid lease, storage ready, connector ready | Successful CrewAI model or tool call |\n| Python sidecar | Authenticated sidecar `GET /health` | Runtime loaded, journal owned, registry and policy available | Provider credentials and external tools work |\n| Action outcome | Host checkpoints, sidecar record and events, provider evidence | Evidence agrees on a terminal result | Automatic safety of a replacement Action |\n\nCheck the boundaries in that order. A failed liveness probe makes later local\nprobes unavailable, but it does not erase durable evidence.\n\n## Read host liveness\n\nThe host `/health` response is static:\n\n```json\n{\n  \"status\": \"ok\",\n  \"protocol\": \"AIP\",\n  \"version\": \"1.0\",\n  \"service\": \"aip-connector-host\"\n}\n```\n\nUse it only to determine whether the process and HTTP router respond. If it\nfails, inspect process termination, listener binding, ingress routing, resource\nlimits, and container state.\n\nDo not restart repeatedly before preserving PostgreSQL checkpoints and the\nsidecar volume. An interrupted Action can retain external effects even when the\nhost is gone.\n\n## Read routing readiness\n\nThe host `/ready` endpoint returns HTTP `200` only when all four conditions are\ntrue:\n\n1. the replica is not draining;\n2. its registry lease remains valid;\n3. durable runtime storage is readable and writable;\n4. the CrewAI connector health probe succeeds within its deadline.\n\nOtherwise it returns HTTP `503` with structured state:\n\n```json\n{\n  \"status\": \"not_ready\",\n  \"lease_valid\": false,\n  \"draining\": false,\n  \"runtime\": {\n    \"durability\": \"durable\",\n    \"ready\": true,\n    \"detail\": \"durable runtime storage is readable and writable\"\n  },\n  \"connector\": {\n    \"ready\": true,\n    \"detail\": \"CrewAI sidecar is durable and exposes configured crews\"\n  }\n}\n```\n\nThe exact connector detail includes the configured crew count. Use the fields,\nnot the example string, as the stable diagnostic contract.\n\nRemove a not-ready replica from new assignments. Preserve existing Action\nidentity until its runtime and sidecar state are reconciled.\n\n## Read sidecar health\n\nThe Rust connector authenticates to sidecar `/health` and requires:\n\n- `status` equal to `ready`;\n- `durable_journal` equal to `true`;\n- every configured host crew ID in `crews`;\n- every descriptor-admitted operation in `operations`.\n\nThe sidecar response also reports:\n\n| Field | Operational use |\n|---|---|\n| `active_runs` | Current records still marked running |\n| `retained_runs` | Journal occupancy |\n| `max_retained_runs` | Admission capacity ceiling |\n| `stream_checkpoint_events` | Persistence interval for streamed chunks |\n| `training_directory` | Whether protected training storage is configured |\n\nThe host rejects a sidecar with a narrower operation policy than its admitted\ndescriptors. Extra sidecar operations do not become AIP capabilities unless a\ndescriptor admits them.\n\nSidecar health does not call a model, tool, knowledge store, evaluator, or crew\ntask. Use a bounded qualification Action for those claims.\n\n## Monitor the host metrics\n\nHost `/metrics` exposes eight Prometheus text metrics without labels:\n\n| Metric | Type | Interpretation |\n|---|---|---|\n| `aip_connector_host_ready` | Gauge | Cached conjunction of lease, storage, connector, and drain state |\n| `aip_connector_host_storage_ready` | Gauge | Cached durable runtime storage result |\n| `aip_connector_host_in_flight` | Gauge | Actions currently held by the host limiter |\n| `aip_connector_host_requests_total` | Counter | Native AIP requests observed |\n| `aip_connector_host_rejected_total` | Counter | Requests rejected before or during dispatch |\n| `aip_connector_host_heartbeat_failures_total` | Counter | Failed heartbeat or lease-recovery cycles |\n| `aip_connector_host_lease_recovery_attempts_total` | Counter | Re-registration attempts after lease expiry |\n| `aip_connector_host_lease_recoveries_total` | Counter | Successful lease recoveries |\n\nScrape each replica with deployment labels added outside the process. Alert on\nreadiness loss, storage loss, sustained in-flight saturation, rejection growth,\nheartbeat failures, and recovery attempts without matching recoveries.\n\nThe sidecar has no `/metrics` route at the reviewed revision. Collect its health\nfields, process resources, structured logs, journal occupancy, and Action-event\nevidence separately.\n\n## Correlate logs and durable evidence\n\nUse these identifiers on every query:\n\n- tenant, connector type, version, instance, and replica;\n- host artifact and sidecar image digests;\n- AIP Action ID and idempotency key digest;\n- crew ID and operation;\n- sidecar event sequence and host checkpoint;\n- provider, model, tool, and artifact request identifiers when available.\n\nThe sidecar logs admission, start, completion, cancellation, archive, and\nfailure with Action ID, crew ID, and operation. Failure logging includes a\nprovider exception traceback, which may contain governed data.\n\nRestrict sidecar logs and redact them before sharing. The durable failed event\ncontains only the exception class and a generic diagnostic message.\n\n## Recover a sidecar readiness failure\n\n1. stop new assignments to the host replica;\n2. read host `/ready` and authenticated sidecar `/health` once;\n3. record image digests, configuration revision, crew descriptors, and enabled\n   operations;\n4. preserve `runs.json`, its lock context, CrewAI home, and training artifacts;\n5. compare sidecar crews and operations with the host descriptor set;\n6. repair the missing registry, policy, secret, path, permission, or capacity\n   input;\n7. restart only one sidecar against the preserved volume;\n8. reconcile every restored uncertain record;\n9. verify sidecar health, host readiness, and registry lease before routing.\n\nDo not bypass the journal lock. A second writer can invalidate the durable\nrequest fence even if both processes appear healthy.\n\n## Recover a storage failure\n\nWhen host readiness reports runtime storage failure:\n\n1. remove the replica from new routing;\n2. keep the host and sidecar evidence available for inspection;\n3. verify PostgreSQL endpoint, credential file, TLS, permissions, capacity, and\n   schema availability;\n4. restore readable and writable durable storage without replacing Action\n   identities;\n5. confirm the runtime recovery report has no queued or delegation errors;\n6. require `/ready` to return `200` before reassignment.\n\nHost registration is blocked when durable recovery reports queued Action or\ndelegation errors. Investigate those records instead of forcing the replica\nonline.\n\n## Recover an expired lease\n\nThe heartbeat loop probes connector and storage health before renewal. An\nexpired lease is re-registered only while both are ready.\n\nFor an ambiguous control-plane failure, the host retains the exact recovery\nrequest identity and payload for idempotent replay. Admission or configuration\nrejection clears that pending identity because no lease committed.\n\nIf lease recovery fails:\n\n1. keep the replica out of new assignments;\n2. compare attempt, recovery, and heartbeat-failure counters;\n3. verify control-plane TLS, DID, endpoint, and registry identity;\n4. check for a live conflicting replica or immutable identity mismatch;\n5. repair the control-plane boundary;\n6. observe one successful recovery and a valid lease in `/ready`.\n\nDo not create a second process with the same replica ID while the first may\nstill hold a valid lease.\n\n## Recover an interrupted Action\n\nA restored sidecar record that was `running` becomes terminal `failed` with\n`uncertain_outcome: true`. The task is not resumed.\n\nUse this decision order:\n\n| Evidence | Action |\n|---|---|\n| Host and sidecar both show completed | Verify output and provider effects, then retain the result |\n| Sidecar shows running and process is healthy | Continue bounded status or event observation |\n| Restart failure reports uncertainty | Reconcile provider, tool, model, and artifact effects |\n| Sidecar reports cancelled for thread-backed work | Assume provider work may continue |\n| Journal record is absent but host shows dispatch | Escalate as lost sidecar identity; do not recreate blindly |\n| Exact repeat returns a collision | Preserve the original record and correct changed request material |\n\nNever archive a record to make a retry fit. Archive only a terminal record after\nthe replay-retention window and an explicit operator decision.\n\n## Drain and restore routing\n\nGraceful shutdown marks the replica draining, waits for in-flight work until the\nconfigured deadline, stops the listener, and attempts an offline transition.\n\nAfter repair, require:\n\n- host liveness and readiness;\n- authenticated durable sidecar health;\n- a valid registry lease and exact replica identity;\n- stable storage and heartbeat metrics;\n- reconciled non-terminal and uncertain Actions;\n- a bounded qualification result for affected provider behavior.\n\nRestore route weight gradually. Keep the previous artifact and state available\nuntil the observation window closes.\n\n## Related documentation\n\n- [Deploy the CrewAI connector](deployment.md)\n- [CrewAI connector configuration](../reference/configuration.md)\n- [CrewAI idempotency, cancellation, and process fencing](../reference/idempotency-cancellation-and-process-fencing.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n- [Metrics reference](../../../reference/metrics.md)\n",
    "text": "CrewAI health, observability, and recovery\n\nUse this runbook when a CrewAI replica is unhealthy, not ready, losing its\nlease, or holding an Action without trustworthy terminal evidence. Preserve the\nhost runtime and sidecar journal before changing the deployment.\n\nFour states matter independently: process liveness, fleet routing readiness,\nsidecar durability, and provider outcome. A green probe at one boundary does\nnot prove the others.\n\nStart with the four boundaries\n\n| Boundary | Primary evidence | Healthy meaning | Excluded meaning |\n\n| Rust process | Host GET /health | Router responds with AIP host identity | Storage, sidecar, or lease readiness |\n| Fleet replica | Host GET /ready | Not draining, valid lease, storage ready, connector ready | Successful CrewAI model or tool call |\n| Python sidecar | Authenticated sidecar GET /health | Runtime loaded, journal owned, registry and policy available | Provider credentials and external tools work |\n| Action outcome | Host checkpoints, sidecar record and events, provider evidence | Evidence agrees on a terminal result | Automatic safety of a replacement Action |\n\nCheck the boundaries in that order. A failed liveness probe makes later local\nprobes unavailable, but it does not erase durable evidence.\n\nRead host liveness\n\nThe host /health response is static:\n\n{\n  \"status\": \"ok\",\n  \"protocol\": \"AIP\",\n  \"version\": \"1.0\",\n  \"service\": \"aip-connector-host\"\n}\n\nUse it only to determine whether the process and HTTP router respond. If it\nfails, inspect process termination, listener binding, ingress routing, resource\nlimits, and container state.\n\nDo not restart repeatedly before preserving PostgreSQL checkpoints and the\nsidecar volume. An interrupted Action can retain external effects even when the\nhost is gone.\n\nRead routing readiness\n\nThe host /ready endpoint returns HTTP 200 only when all four conditions are\ntrue:\n1. the replica is not draining;\n2. its registry lease remains valid;\n3. durable runtime storage is readable and writable;\n4. the CrewAI connector health probe succeeds within its deadline.\n\nOtherwise it returns HTTP 503 with structured state:\n\n{\n  \"status\": \"notready\",\n  \"leasevalid\": false,\n  \"draining\": false,\n  \"runtime\": {\n    \"durability\": \"durable\",\n    \"ready\": true,\n    \"detail\": \"durable runtime storage is readable and writable\"\n  },\n  \"connector\": {\n    \"ready\": true,\n    \"detail\": \"CrewAI sidecar is durable and exposes configured crews\"\n  }\n}\n\nThe exact connector detail includes the configured crew count. Use the fields,\nnot the example string, as the stable diagnostic contract.\n\nRemove a not-ready replica from new assignments. Preserve existing Action\nidentity until its runtime and sidecar state are reconciled.\n\nRead sidecar health\n\nThe Rust connector authenticates to sidecar /health and requires:\n• status equal to ready;\n• durablejournal equal to true;\n• every configured host crew ID in crews;\n• every descriptor-admitted operation in operations.\n\nThe sidecar response also reports:\n\n| Field | Operational use |\n\n| activeruns | Current records still marked running |\n| retainedruns | Journal occupancy |\n| maxretainedruns | Admission capacity ceiling |\n| streamcheckpointevents | Persistence interval for streamed chunks |\n| trainingdirectory | Whether protected training storage is configured |\n\nThe host rejects a sidecar with a narrower operation policy than its admitted\ndescriptors. Extra sidecar operations do not become AIP capabilities unless a\ndescriptor admits them.\n\nSidecar health does not call a model, tool, knowledge store, evaluator, or crew\ntask. Use a bounded qualification Action for those claims.\n\nMonitor the host metrics\n\nHost /metrics exposes eight Prometheus text metrics without labels:\n\n| Metric | Type | Interpretation |\n\n| aipconnectorhostready | Gauge | Cached conjunction of lease, storage, connector, and drain state |\n| aipconnectorhoststorageready | Gauge | Cached durable runtime storage result |\n| aipconnectorhostinflight | Gauge | Actions currently held by the host limiter |\n| aipconnectorhostrequeststotal | Counter | Native AIP requests observed |\n| aipconnectorhostrejectedtotal | Counter | Requests rejected before or during dispatch |\n| aipconnectorhostheartbeatfailurestotal | Counter | Failed heartbeat or lease-recovery cycles |\n| aipconnectorhostleaserecoveryattemptstotal | Counter | Re-registration attempts after lease expiry |\n| aipconnectorhostleaserecoveriestotal | Counter | Successful lease recoveries |\n\nScrape each replica with deployment labels added outside the process. Alert on\nreadiness loss, storage loss, sustained in-flight saturation, rejection growth,\nheartbeat failures, and recovery attempts without matching recoveries.\n\nThe sidecar has no /metrics route at the reviewed revision. Collect its health\nfields, process resources, structured logs, journal occupancy, and Action-event\nevidence separately.\n\nCorrelate logs and durable evidence\n\nUse these identifiers on every query:\n• tenant, connector type, version, instance, and replica;\n• host artifact and sidecar image digests;\n• AIP Action ID and idempotency key digest;\n• crew ID and operation;\n• sidecar event sequence and host checkpoint;\n• provider, model, tool, and artifact request identifiers when available.\n\nThe sidecar logs admission, start, completion, cancellation, archive, and\nfailure with Action ID, crew ID, and operation. Failure logging includes a\nprovider exception traceback, which may contain governed data.\n\nRestrict sidecar logs and redact them before sharing. The durable failed event\ncontains only the exception class and a generic diagnostic message.\n\nRecover a sidecar readiness failure\n1. stop new assignments to the host replica;\n2. read host /ready and authenticated sidecar /health once;\n3. record image digests, configuration revision, crew descriptors, and enabled\n   operations;\n4. preserve runs.json, its lock context, CrewAI home, and training artifacts;\n5. compare sidecar crews and operations with the host descriptor set;\n6. repair the missing registry, policy, secret, path, permission, or capacity\n   input;\n7. restart only one sidecar against the preserved volume;\n8. reconcile every restored uncertain record;\n9. verify sidecar health, host readiness, and registry lease before routing.\n\nDo not bypass the journal lock. A second writer can invalidate the durable\nrequest fence even if both processes appear healthy.\n\nRecover a storage failure\n\nWhen host readiness reports runtime storage failure:\n1. remove the replica from new routing;\n2. keep the host and sidecar evidence available for inspection;\n3. verify PostgreSQL endpoint, credential file, TLS, permissions, capacity, and\n   schema availability;\n4. restore readable and writable durable storage without replacing Action\n   identities;\n5. confirm the runtime recovery report has no queued or delegation errors;\n6. require /ready to return 200 before reassignment.\n\nHost registration is blocked when durable recovery reports queued Action or\ndelegation errors. Investigate those records instead of forcing the replica\nonline.\n\nRecover an expired lease\n\nThe heartbeat loop probes connector and storage health before renewal. An\nexpired lease is re-registered only while both are ready.\n\nFor an ambiguous control-plane failure, the host retains the exact recovery\nrequest identity and payload for idempotent replay. Admission or configuration\nrejection clears that pending identity because no lease committed.\n\nIf lease recovery fails:\n1. keep the replica out of new assignments;\n2. compare attempt, recovery, and heartbeat-failure counters;\n3. verify control-plane TLS, DID, endpoint, and registry identity;\n4. check for a live conflicting replica or immutable identity mismatch;\n5. repair the control-plane boundary;\n6. observe one successful recovery and a valid lease in /ready.\n\nDo not create a second process with the same replica ID while the first may\nstill hold a valid lease.\n\nRecover an interrupted Action\n\nA restored sidecar record that was running becomes terminal failed with\nuncertainoutcome: true. The task is not resumed.\n\nUse this decision order:\n\n| Evidence | Action |\n\n| Host and sidecar both show completed | Verify output and provider effects, then retain the result |\n| Sidecar shows running and process is healthy | Continue bounded status or event observation |\n| Restart failure reports uncertainty | Reconcile provider, tool, model, and artifact effects |\n| Sidecar reports cancelled for thread-backed work | Assume provider work may continue |\n| Journal record is absent but host shows dispatch | Escalate as lost sidecar identity; do not recreate blindly |\n| Exact repeat returns a collision | Preserve the original record and correct changed request material |\n\nNever archive a record to make a retry fit. Archive only a terminal record after\nthe replay-retention window and an explicit operator decision.\n\nDrain and restore routing\n\nGraceful shutdown marks the replica draining, waits for in-flight work until the\nconfigured deadline, stops the listener, and attempts an offline transition.\n\nAfter repair, require:\n• host liveness and readiness;\n• authenticated durable sidecar health;\n• a valid registry lease and exact replica identity;\n• stable storage and heartbeat metrics;\n• reconciled non-terminal and uncertain Actions;\n• a bounded qualification result for affected provider behavior.\n\nRestore route weight gradually. Keep the previous artifact and state available\nuntil the observation window closes.\n\nRelated documentation\n• Deploy the CrewAI connector (deployment.md)\n• CrewAI connector configuration (../reference/configuration.md)\n• CrewAI idempotency, cancellation, and process fencing (../reference/idempotency-cancellation-and-process-fencing.md)\n• Errors and retry decisions (../../../reference/errors.md)\n• Metrics reference (../../../reference/metrics.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "0e6ecc8a22ee1d9134beae7b053b061b6384afa8cdb21815afe70af7891301b0"
  }
}