{
  "schemaVersion": "1.0",
  "title": "Recover the Hermes Agent connector",
  "description": "Use this runbook when a Hermes Agent host is unavailable, not ready, losing its lease, rejecting traffic, or retaining work whose provider outcome is uncertain. It helps you locate the failing boundary and recover the same AIP Action, opera",
  "canonical": "https://getaip.org/docs/connectors/hermes-agent/operations/health-observability-and-recovery",
  "route": "/docs/connectors/hermes-agent/operations/health-observability-and-recovery",
  "source": "docs/connectors/hermes-agent/operations/health-observability-and-recovery.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/hermes-agent/operations/health-observability-and-recovery.md",
    "txt": "/docs/download/connectors/hermes-agent/operations/health-observability-and-recovery.txt",
    "json": "/docs/download/connectors/hermes-agent/operations/health-observability-and-recovery.json",
    "pdf": "/docs/download/connectors/hermes-agent/operations/health-observability-and-recovery.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Recover the Hermes Agent connector\ndescription: >-\n  Diagnose Hermes host health and recover durable AIP and operator work\n  without replaying uncertain provider effects\nkind: runbook\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: hermes-agent\n---\n\n# Recover the Hermes Agent connector\n\nUse this runbook when a Hermes Agent host is unavailable, not ready, losing\nits lease, rejecting traffic, or retaining work whose provider outcome is\nuncertain. It helps you locate the failing boundary and recover the same AIP\nAction, operator binding, or delegated child without starting the work again.\n\nRun these checks through an operator-controlled network. The procedure assumes\nthe fixed endpoint set and shared PostgreSQL boundary established in\n[Deploy the Hermes Agent connector](deployment.md).\n\n## Contain risk before diagnosis\n\nPause new model, tool, session, job, operator, and delegation mutations for the\naffected binding when durable storage is unavailable, the host lease is\ninvalid, or an accepted operation has an unknown outcome. Keep bounded reads\navailable only when tenant and endpoint ownership remain verified.\n\nDo not restart Hermes, clear either database, rotate an endpoint, create a new\nAction ID, or move pinned work as the first response. Retain:\n\n- UTC incident start and the last known-ready time;\n- connector type, version, instance, replica, artifact, and manifest digest;\n- tenant, endpoint ID, provider origin, configuration, and credential revision;\n- lease sequence, lease expiry, and registry status;\n- Action, idempotency, correlation, session, approval, delegation, and run IDs;\n- `/ready` plus two timestamped `/metrics` snapshots;\n- AIP results, events, chunks, receipts, and redacted provider evidence;\n- operator binding status and provider job or session identity when relevant.\n\nA healthy replacement cannot make an earlier uncertain model or tool effect\nsafe to repeat.\n\n## Read six views in order\n\nNo single surface proves the outcome of an Action.\n\n| View | What it answers | What it does not prove |\n|---|---|---|\n| Host `/health` | Can the common host process answer its static route? | Lease, storage, endpoint, or provider readiness |\n| Host `/ready` | Are drain, lease, PostgreSQL, and all basic endpoint probes ready now? | Bearer auth, model credentials, MCP, tools, or one Action outcome |\n| Host `/metrics` | Are local load, rejection, heartbeat, and lease signals changing? | Per-endpoint, per-capability, or per-Action truth |\n| Registry and gateway | Is the replica eligible and where is accepted work pinned? | Whether Hermes or a tool committed an effect |\n| Protected Hermes read | Does the selected endpoint accept its Bearer token and expose the expected model or runtime view? | Chat, tool, mutation, or delegation success |\n| AIP and provider state | What evidence belongs to this exact Action and provider handle? | General readiness for new work |\n\nCapture the host surfaces without submitting an Action:\n\n```sh\nexport HERMES_HOST_OBSERVE_ORIGIN=https://hermes-connector.internal.example\n\ncurl --silent --show-error \"$HERMES_HOST_OBSERVE_ORIGIN/health\"\ncurl --silent --show-error \\\n  --write-out '\\nHTTP %{http_code}\\n' \\\n  \"$HERMES_HOST_OBSERVE_ORIGIN/ready\"\ncurl --silent --show-error \"$HERMES_HOST_OBSERVE_ORIGIN/metrics\"\n```\n\nDo not expose these unauthenticated host routes to an untrusted network.\nScraping `/metrics` does not exercise the endpoint Bearer token.\n\n## Interpret host readiness precisely\n\n`/health` returns static process metadata. `/ready` returns HTTP `200` only\nwhen all four gates pass and HTTP `503` otherwise:\n\n| Readiness field | Ready value | Failure boundary |\n|---|---:|---|\n| `draining` | `false` | Planned or interrupted shutdown |\n| `lease_valid` | `true` | Lifecycle network, registry, clock, or expired lease |\n| `runtime.ready` | `true` | PostgreSQL read/write readiness |\n| `connector.ready` | `true` | Basic health of every configured Hermes endpoint |\n\nThe response includes `runtime.durability` and a startup recovery summary:\n\n| Recovery field | Meaning |\n|---|---|\n| `recovered_at` | Time at which startup recovery began |\n| `pending_approvals` | Existing approvals retained for continuation |\n| `recoverable_transactions` | Transactions retained for reconciliation |\n| `queued_results` | Queued Actions completed during startup recovery |\n| `delegation_results` | Delegations completed during startup recovery |\n| `callback_deliveries` | Callback deliveries resumed during recovery |\n\nThis summary is startup evidence, not a live backlog. A queued-action or\ndelegation recovery error blocks registration. A broader storage or callback\nrecovery failure also stops host construction instead of producing readiness.\n\nConnector readiness visits every endpoint in normalized endpoint order and\nsends unauthenticated `GET /health`. Every endpoint must return parsable JSON\nwith a successful HTTP status. The readiness loop does not require the JSON\n`status` field to equal `ok`.\n\nThe configured host health timeout wraps the complete multi-endpoint loop. Its\ndefault is 2000 milliseconds, even though one direct endpoint health request\nhas a 30-second client timeout. A slow first endpoint can therefore end the\nwhole host probe before later endpoints are visited.\n\nBasic endpoint health does not prove Bearer authentication, model-provider\ncredentials, MCP connectivity, tool availability, or operator policy. Confirm\nthe intended endpoint with the authenticated `health_detailed` or `models`\ncapability after host readiness returns.\n\n## Interpret the eight host metrics\n\nThe host emits exactly eight unlabeled Prometheus metrics:\n\n| Metric | Operational use | Important limit |\n|---|---|---|\n| `aip_connector_host_ready` | Combined lease, drain, storage, and connector state | Uses cached probe flags; correlate with `/ready` and registry state |\n| `aip_connector_host_storage_ready` | Last durable-storage probe result | Does not identify a damaged record |\n| `aip_connector_host_in_flight` | Actions executing in this process | Does not include provider work after a lost response |\n| `aip_connector_host_requests_total` | Native messages observed by the host | Does not count every HTTP request or classify capability success |\n| `aip_connector_host_rejected_total` | Native messages rejected before successful handling | Requires signed error-code correlation |\n| `aip_connector_host_heartbeat_failures_total` | Failed renewal or lease-recovery cycles | One increase does not prove lease expiry |\n| `aip_connector_host_lease_recovery_attempts_total` | Re-registration attempts after lease expiry | Includes attempts that did not recover |\n| `aip_connector_host_lease_recoveries_total` | Successful expired-lease recoveries | Does not resolve Action or provider outcomes |\n\nThe metrics endpoint returns HTTP `200` even when the ready gauge is `0`.\nStorage and connector gauges are refreshed by readiness checks and heartbeat\nprobes, not by a new probe inside each metrics scrape. Counters and gauges are\nprocess-local and reset when the host restarts.\n\nThis revision exposes no Hermes endpoint label, provider status, latency,\nmodel, MCP, run, approval, session, job, operator, or delegated-result metric.\nUse durable AIP records and bounded provider reads for those views.\n\nDefine alert windows from the deployment SLO. At minimum, alert on these\nrelationships:\n\n- readiness remains `0` outside a planned drain;\n- storage readiness remains `0` or `/ready` reports storage failure;\n- heartbeat failures grow as the current lease approaches expiry;\n- lease-recovery attempts grow without a successful recovery;\n- in-flight work remains at the configured concurrency bound;\n- rejected messages grow faster than total native messages;\n- basic endpoint health passes while a protected read fails;\n- an accepted Action stops producing durable state within its contract bound.\n\n## Classify the failing boundary\n\n| Observation | Boundary | Immediate decision |\n|---|---|---|\n| No host response | Process, node, ingress, or network | Preserve both state stores and follow hard-crash recovery |\n| `/health` works and `draining` is true | Planned or partial shutdown | Stop new assignments and finish or investigate the drain |\n| `lease_valid` is false | Lifecycle TLS, DID, clock, registry, or admission | Keep mutations paused and allow fenced recovery after repair |\n| `runtime.ready` is false | Connector PostgreSQL | Isolate the replica; never substitute an empty database |\n| `connector.ready` is false | One or more endpoint origins or basic health routes | Restore every endpoint or revise the admitted endpoint set |\n| Host ready, protected read gets `401` or `403` | Endpoint token or endpoint selection | Replace the correct token and host process |\n| Host ready, models read fails | Model provider, route alias, quota, or credential | Keep model and tool work paused; basic health is insufficient |\n| Rejections rise with high in-flight work | Local capacity | Preserve Action IDs and let bounded backpressure work |\n| Rejections rise with low load | Schema, signature, route, policy, credential revision, or execution | Group by exact signed error code |\n| Bound operator run cannot be found | Hermes provider run state | Reconcile as volatile provider-state loss; do not start again |\n| Queue and lifecycle disagree on a delegated result | Durable integrity | Fail closed and preserve both records |\n\n## Recover process or lease loss\n\nIf the host still runs, repair lifecycle DNS, TLS, CA trust, pinned DID, clock,\nor service availability before replacing it. The host retains its last lease\nuntil expiry. While the lease is valid, each heartbeat reports the combined\nstorage and connector result. An unhealthy report moves the replica offline.\n\nAfter lease expiry, a healthy host uses signed re-registration instead of a\nheartbeat. An ambiguous lifecycle response is retried with the same request\nidentity and registration bytes. A definite admission or configuration\nrejection clears that request; correct the immutable mismatch before the next\nattempt. Never edit lease expiry, sequence, or registry status by hand.\n\nAfter a hard crash:\n\n1. prove that no process still owns the replica key and endpoint;\n2. preserve the same PostgreSQL database, artifact, instance, and replica;\n3. preserve each Hermes endpoint's local state and provider identity;\n4. wait for clean offline state or lease expiry;\n5. start the reviewed process and let recovery finish before registration;\n6. stop if recovery reports queued-action or delegation errors;\n7. compare recovery counts, retained assignments, and the new lease sequence;\n8. require host readiness and one protected read before restoring traffic.\n\nUse a new replica identity for an actual replacement. Never run two processes\nwith the same replica ID or signing key, and never point overlapping replicas\nat independent connector databases.\n\n## Recover storage, endpoints, and capacity\n\nWhen `runtime.ready` is false, restore connectivity to the exact PostgreSQL\ndatabase. Check capacity, certificates, role permissions, schema, and\nread/write health. Do not initialize a clean store; that removes the state\nneeded to fence Actions, approvals, callbacks, operator bindings, and child\nresults.\n\nWhen `connector.ready` is false, identify the failing endpoint from the\nbounded connector detail and endpoint evidence. Check its fixed origin,\nnetwork trust, Hermes process, and JSON health response. One failed endpoint\nkeeps the complete host unready. Restore it or change the endpoint set through\nthe reviewed deployment and admission process.\n\nA protected read failure requires separate checks for endpoint ID, Bearer\ntoken, Hermes API-server policy, model route, and upstream credentials.\nReplacing token bytes requires a host restart because endpoint keys are read\nat startup.\n\nAt local capacity, a new Action receives retryable\n`connector_host.capacity` with HTTP `429`. A draining, lease-invalid, or\nconnector-unready host returns retryable `connector_host.not_ready`. Preserve\nthe original Action contract and honor backoff. A retryable transport label\ndoes not prove that Hermes or a tool performed no work.\n\n## Recover by provider state owner\n\nFirst read the durable AIP Action, including its result, events, chunks,\napproval, and delegation links. Then use the operation's provider-owned state:\n\n| Work family | Provider state at the pinned revision | Recovery decision |\n|---|---|---|\n| Health and discovery reads | Point-in-time endpoint response | Repeat only as a new governed read when the original Action is settled |\n| Chat and Responses execution | Model and tool work may already have started | Read the original AIP result first; do not resubmit automatically after transport, decode, deadline, or server failure |\n| Stored Responses objects | Normally endpoint-local SQLite; may fall back to memory if that store cannot open | Read by the retained response ID on the owning endpoint; absence does not authorize model replay |\n| Structured runs | Status, stream queue, active task, and approval session are in process memory | A provider restart can lose the `run_id`; preserve AIP evidence and do not recreate the run |\n| Persistent sessions | Endpoint-local `state.db`; no in-memory session fallback | Read the session and messages on the owning endpoint before repeating create, patch, fork, chat, or delete |\n| Scheduled jobs | Profile-local persistent scheduler files and run claims | Read the current job and output evidence; an accepted mutation does not prove a due run or delivery |\n| Operator execution | Durable PostgreSQL binding plus a volatile provider run | Resume observation only when a bound `run_id` still exists; otherwise preserve the uncertain outcome |\n| Delegated child | Exact durable queue and lifecycle records | Accept only a matching terminal result for the expected child Action and principal |\n\nEndpoint identity includes its local Hermes home and runtime state. Do not\nredirect an existing endpoint ID to an empty Hermes instance and treat a\nmissing response, session, job, or run as proof that no effect occurred.\n\n## Recover an operator or delegated result\n\nThe standalone host stores operator bindings in the same PostgreSQL-backed\nprofile state used by its runtime. Each binding retains the source Action,\nendpoint, request fingerprint, provider run ID when known, cancellation\nintent, approval generation and commands, sequence, terminal output, and\ntyped error.\n\nUse this decision order:\n\n1. read the source Action and retained operator binding;\n2. reject any endpoint or request-fingerprint mismatch;\n3. return a retained terminal result without contacting Hermes again;\n4. when `run_id` is bound, poll that exact run and reconcile cancellation;\n5. when start ownership disappeared before `run_id` was stored, keep\n   `outcome_unknown` and do not replay `POST /v1/runs`;\n6. when an approval command is claimed without a durable outcome, preserve its\n   uncertain state instead of sending the choice again;\n7. require a Hermes terminal status before calling cancellation confirmed.\n\nA provider `404` for a bound run can mean provider restart or retention loss.\nIt does not prove that models or tools had no effect.\n\nFor first-class delegation, the resolver reads the exact child Action from the\ndurable queue and lifecycle stores. It verifies child identity, tenant,\ntransport-authenticated Hermes principal, and immutable execution contract.\nA terminal record without a result, conflicting queue and lifecycle results,\nor a missing expected child is an integrity or policy failure. Never substitute\nthe Hermes model's final prose for that result.\n\n## Preserve stream and event evidence\n\nAIP Action events and chunks use durable runtime positions. Persist the opaque\ncombined cursor only after processing its page. Provider SSE event IDs,\nprovider run IDs, chunk IDs, and AIP cursors are not interchangeable.\n\nA lost follow connection does not cancel an Action. Initial operator\nobservation can consume the volatile Hermes SSE queue; after reconnect or host\nrestart, a bound operator polls provider run status instead. Chunks already\npublished to AIP remain available, but unobserved provider events may be lost.\n\nDo not infer terminal execution from an ended stream. Require the durable AIP\nresult and, where applicable, the provider's terminal state.\n\n## Verify recovery\n\nRestore affected traffic only when every applicable check passes:\n\n1. `/health` and `/ready` agree on a live, non-draining host;\n2. storage and all endpoint probes are ready within the configured bound;\n3. the registry shows the intended replica with a current lease;\n4. metric growth has stabilized and usable capacity remains;\n5. one protected read reaches the intended endpoint and credential;\n6. each retained Action has a justified durable state under its original ID;\n7. provider-local sessions, jobs, responses, and runs are read from their\n   owning endpoint rather than inferred from host readiness;\n8. uncertain operator, approval, cancellation, or tool effects remain paused\n   until terminal evidence exists;\n9. the incident record retains identities, state transitions, errors, and\n   operator decisions.\n\nIf these views disagree, keep the mutation boundary paused. Escalate with the\nsource revisions, artifact and manifest digests, endpoint set, database\nidentities, lease history, readiness body, metric snapshots, Action and\nprovider identifiers, operator binding, and redacted evidence.\n\n## Related documentation\n\n- [Deploy the Hermes Agent connector](deployment.md)\n- [Operate the connector host lifecycle](../../../guides/connector-host-lifecycle.md)\n- [Observe and recover AIP work](../../../guides/observe-and-recover.md)\n- [Run a governed Hermes operator lifecycle](../guides/operator-lifecycle.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n",
    "text": "Recover the Hermes Agent connector\n\nUse this runbook when a Hermes Agent host is unavailable, not ready, losing\nits lease, rejecting traffic, or retaining work whose provider outcome is\nuncertain. It helps you locate the failing boundary and recover the same AIP\nAction, operator binding, or delegated child without starting the work again.\n\nRun these checks through an operator-controlled network. The procedure assumes\nthe fixed endpoint set and shared PostgreSQL boundary established in\nDeploy the Hermes Agent connector (deployment.md).\n\nContain risk before diagnosis\n\nPause new model, tool, session, job, operator, and delegation mutations for the\naffected binding when durable storage is unavailable, the host lease is\ninvalid, or an accepted operation has an unknown outcome. Keep bounded reads\navailable only when tenant and endpoint ownership remain verified.\n\nDo not restart Hermes, clear either database, rotate an endpoint, create a new\nAction ID, or move pinned work as the first response. Retain:\n• UTC incident start and the last known-ready time;\n• connector type, version, instance, replica, artifact, and manifest digest;\n• tenant, endpoint ID, provider origin, configuration, and credential revision;\n• lease sequence, lease expiry, and registry status;\n• Action, idempotency, correlation, session, approval, delegation, and run IDs;\n• /ready plus two timestamped /metrics snapshots;\n• AIP results, events, chunks, receipts, and redacted provider evidence;\n• operator binding status and provider job or session identity when relevant.\n\nA healthy replacement cannot make an earlier uncertain model or tool effect\nsafe to repeat.\n\nRead six views in order\n\nNo single surface proves the outcome of an Action.\n\n| View | What it answers | What it does not prove |\n\n| Host /health | Can the common host process answer its static route? | Lease, storage, endpoint, or provider readiness |\n| Host /ready | Are drain, lease, PostgreSQL, and all basic endpoint probes ready now? | Bearer auth, model credentials, MCP, tools, or one Action outcome |\n| Host /metrics | Are local load, rejection, heartbeat, and lease signals changing? | Per-endpoint, per-capability, or per-Action truth |\n| Registry and gateway | Is the replica eligible and where is accepted work pinned? | Whether Hermes or a tool committed an effect |\n| Protected Hermes read | Does the selected endpoint accept its Bearer token and expose the expected model or runtime view? | Chat, tool, mutation, or delegation success |\n| AIP and provider state | What evidence belongs to this exact Action and provider handle? | General readiness for new work |\n\nCapture the host surfaces without submitting an Action:\n\nexport HERMESHOSTOBSERVEORIGIN=https://hermes-connector.internal.example\n\ncurl --silent --show-error \"$HERMESHOSTOBSERVEORIGIN/health\"\ncurl --silent --show-error \\\n  --write-out '\\nHTTP %{httpcode}\\n' \\\n  \"$HERMESHOSTOBSERVEORIGIN/ready\"\ncurl --silent --show-error \"$HERMESHOSTOBSERVEORIGIN/metrics\"\n\nDo not expose these unauthenticated host routes to an untrusted network.\nScraping /metrics does not exercise the endpoint Bearer token.\n\nInterpret host readiness precisely\n\n/health returns static process metadata. /ready returns HTTP 200 only\nwhen all four gates pass and HTTP 503 otherwise:\n\n| Readiness field | Ready value | Failure boundary |\n\n| draining | false | Planned or interrupted shutdown |\n| leasevalid | true | Lifecycle network, registry, clock, or expired lease |\n| runtime.ready | true | PostgreSQL read/write readiness |\n| connector.ready | true | Basic health of every configured Hermes endpoint |\n\nThe response includes runtime.durability and a startup recovery summary:\n\n| Recovery field | Meaning |\n\n| recoveredat | Time at which startup recovery began |\n| pendingapprovals | Existing approvals retained for continuation |\n| recoverabletransactions | Transactions retained for reconciliation |\n| queuedresults | Queued Actions completed during startup recovery |\n| delegationresults | Delegations completed during startup recovery |\n| callbackdeliveries | Callback deliveries resumed during recovery |\n\nThis summary is startup evidence, not a live backlog. A queued-action or\ndelegation recovery error blocks registration. A broader storage or callback\nrecovery failure also stops host construction instead of producing readiness.\n\nConnector readiness visits every endpoint in normalized endpoint order and\nsends unauthenticated GET /health. Every endpoint must return parsable JSON\nwith a successful HTTP status. The readiness loop does not require the JSON\nstatus field to equal ok.\n\nThe configured host health timeout wraps the complete multi-endpoint loop. Its\ndefault is 2000 milliseconds, even though one direct endpoint health request\nhas a 30-second client timeout. A slow first endpoint can therefore end the\nwhole host probe before later endpoints are visited.\n\nBasic endpoint health does not prove Bearer authentication, model-provider\ncredentials, MCP connectivity, tool availability, or operator policy. Confirm\nthe intended endpoint with the authenticated healthdetailed or models\ncapability after host readiness returns.\n\nInterpret the eight host metrics\n\nThe host emits exactly eight unlabeled Prometheus metrics:\n\n| Metric | Operational use | Important limit |\n\n| aipconnectorhostready | Combined lease, drain, storage, and connector state | Uses cached probe flags; correlate with /ready and registry state |\n| aipconnectorhoststorageready | Last durable-storage probe result | Does not identify a damaged record |\n| aipconnectorhostinflight | Actions executing in this process | Does not include provider work after a lost response |\n| aipconnectorhostrequeststotal | Native messages observed by the host | Does not count every HTTP request or classify capability success |\n| aipconnectorhostrejectedtotal | Native messages rejected before successful handling | Requires signed error-code correlation |\n| aipconnectorhostheartbeatfailurestotal | Failed renewal or lease-recovery cycles | One increase does not prove lease expiry |\n| aipconnectorhostleaserecoveryattemptstotal | Re-registration attempts after lease expiry | Includes attempts that did not recover |\n| aipconnectorhostleaserecoveriestotal | Successful expired-lease recoveries | Does not resolve Action or provider outcomes |\n\nThe metrics endpoint returns HTTP 200 even when the ready gauge is 0.\nStorage and connector gauges are refreshed by readiness checks and heartbeat\nprobes, not by a new probe inside each metrics scrape. Counters and gauges are\nprocess-local and reset when the host restarts.\n\nThis revision exposes no Hermes endpoint label, provider status, latency,\nmodel, MCP, run, approval, session, job, operator, or delegated-result metric.\nUse durable AIP records and bounded provider reads for those views.\n\nDefine alert windows from the deployment SLO. At minimum, alert on these\nrelationships:\n• readiness remains 0 outside a planned drain;\n• storage readiness remains 0 or /ready reports storage failure;\n• heartbeat failures grow as the current lease approaches expiry;\n• lease-recovery attempts grow without a successful recovery;\n• in-flight work remains at the configured concurrency bound;\n• rejected messages grow faster than total native messages;\n• basic endpoint health passes while a protected read fails;\n• an accepted Action stops producing durable state within its contract bound.\n\nClassify the failing boundary\n\n| Observation | Boundary | Immediate decision |\n\n| No host response | Process, node, ingress, or network | Preserve both state stores and follow hard-crash recovery |\n| /health works and draining is true | Planned or partial shutdown | Stop new assignments and finish or investigate the drain |\n| leasevalid is false | Lifecycle TLS, DID, clock, registry, or admission | Keep mutations paused and allow fenced recovery after repair |\n| runtime.ready is false | Connector PostgreSQL | Isolate the replica; never substitute an empty database |\n| connector.ready is false | One or more endpoint origins or basic health routes | Restore every endpoint or revise the admitted endpoint set |\n| Host ready, protected read gets 401 or 403 | Endpoint token or endpoint selection | Replace the correct token and host process |\n| Host ready, models read fails | Model provider, route alias, quota, or credential | Keep model and tool work paused; basic health is insufficient |\n| Rejections rise with high in-flight work | Local capacity | Preserve Action IDs and let bounded backpressure work |\n| Rejections rise with low load | Schema, signature, route, policy, credential revision, or execution | Group by exact signed error code |\n| Bound operator run cannot be found | Hermes provider run state | Reconcile as volatile provider-state loss; do not start again |\n| Queue and lifecycle disagree on a delegated result | Durable integrity | Fail closed and preserve both records |\n\nRecover process or lease loss\n\nIf the host still runs, repair lifecycle DNS, TLS, CA trust, pinned DID, clock,\nor service availability before replacing it. The host retains its last lease\nuntil expiry. While the lease is valid, each heartbeat reports the combined\nstorage and connector result. An unhealthy report moves the replica offline.\n\nAfter lease expiry, a healthy host uses signed re-registration instead of a\nheartbeat. An ambiguous lifecycle response is retried with the same request\nidentity and registration bytes. A definite admission or configuration\nrejection clears that request; correct the immutable mismatch before the next\nattempt. Never edit lease expiry, sequence, or registry status by hand.\n\nAfter a hard crash:\n1. prove that no process still owns the replica key and endpoint;\n2. preserve the same PostgreSQL database, artifact, instance, and replica;\n3. preserve each Hermes endpoint's local state and provider identity;\n4. wait for clean offline state or lease expiry;\n5. start the reviewed process and let recovery finish before registration;\n6. stop if recovery reports queued-action or delegation errors;\n7. compare recovery counts, retained assignments, and the new lease sequence;\n8. require host readiness and one protected read before restoring traffic.\n\nUse a new replica identity for an actual replacement. Never run two processes\nwith the same replica ID or signing key, and never point overlapping replicas\nat independent connector databases.\n\nRecover storage, endpoints, and capacity\n\nWhen runtime.ready is false, restore connectivity to the exact PostgreSQL\ndatabase. Check capacity, certificates, role permissions, schema, and\nread/write health. Do not initialize a clean store; that removes the state\nneeded to fence Actions, approvals, callbacks, operator bindings, and child\nresults.\n\nWhen connector.ready is false, identify the failing endpoint from the\nbounded connector detail and endpoint evidence. Check its fixed origin,\nnetwork trust, Hermes process, and JSON health response. One failed endpoint\nkeeps the complete host unready. Restore it or change the endpoint set through\nthe reviewed deployment and admission process.\n\nA protected read failure requires separate checks for endpoint ID, Bearer\ntoken, Hermes API-server policy, model route, and upstream credentials.\nReplacing token bytes requires a host restart because endpoint keys are read\nat startup.\n\nAt local capacity, a new Action receives retryable\nconnectorhost.capacity with HTTP 429. A draining, lease-invalid, or\nconnector-unready host returns retryable connectorhost.notready. Preserve\nthe original Action contract and honor backoff. A retryable transport label\ndoes not prove that Hermes or a tool performed no work.\n\nRecover by provider state owner\n\nFirst read the durable AIP Action, including its result, events, chunks,\napproval, and delegation links. Then use the operation's provider-owned state:\n\n| Work family | Provider state at the pinned revision | Recovery decision |\n\n| Health and discovery reads | Point-in-time endpoint response | Repeat only as a new governed read when the original Action is settled |\n| Chat and Responses execution | Model and tool work may already have started | Read the original AIP result first; do not resubmit automatically after transport, decode, deadline, or server failure |\n| Stored Responses objects | Normally endpoint-local SQLite; may fall back to memory if that store cannot open | Read by the retained response ID on the owning endpoint; absence does not authorize model replay |\n| Structured runs | Status, stream queue, active task, and approval session are in process memory | A provider restart can lose the runid; preserve AIP evidence and do not recreate the run |\n| Persistent sessions | Endpoint-local state.db; no in-memory session fallback | Read the session and messages on the owning endpoint before repeating create, patch, fork, chat, or delete |\n| Scheduled jobs | Profile-local persistent scheduler files and run claims | Read the current job and output evidence; an accepted mutation does not prove a due run or delivery |\n| Operator execution | Durable PostgreSQL binding plus a volatile provider run | Resume observation only when a bound runid still exists; otherwise preserve the uncertain outcome |\n| Delegated child | Exact durable queue and lifecycle records | Accept only a matching terminal result for the expected child Action and principal |\n\nEndpoint identity includes its local Hermes home and runtime state. Do not\nredirect an existing endpoint ID to an empty Hermes instance and treat a\nmissing response, session, job, or run as proof that no effect occurred.\n\nRecover an operator or delegated result\n\nThe standalone host stores operator bindings in the same PostgreSQL-backed\nprofile state used by its runtime. Each binding retains the source Action,\nendpoint, request fingerprint, provider run ID when known, cancellation\nintent, approval generation and commands, sequence, terminal output, and\ntyped error.\n\nUse this decision order:\n1. read the source Action and retained operator binding;\n2. reject any endpoint or request-fingerprint mismatch;\n3. return a retained terminal result without contacting Hermes again;\n4. when runid is bound, poll that exact run and reconcile cancellation;\n5. when start ownership disappeared before runid was stored, keep\n   outcomeunknown and do not replay POST /v1/runs;\n6. when an approval command is claimed without a durable outcome, preserve its\n   uncertain state instead of sending the choice again;\n7. require a Hermes terminal status before calling cancellation confirmed.\n\nA provider 404 for a bound run can mean provider restart or retention loss.\nIt does not prove that models or tools had no effect.\n\nFor first-class delegation, the resolver reads the exact child Action from the\ndurable queue and lifecycle stores. It verifies child identity, tenant,\ntransport-authenticated Hermes principal, and immutable execution contract.\nA terminal record without a result, conflicting queue and lifecycle results,\nor a missing expected child is an integrity or policy failure. Never substitute\nthe Hermes model's final prose for that result.\n\nPreserve stream and event evidence\n\nAIP Action events and chunks use durable runtime positions. Persist the opaque\ncombined cursor only after processing its page. Provider SSE event IDs,\nprovider run IDs, chunk IDs, and AIP cursors are not interchangeable.\n\nA lost follow connection does not cancel an Action. Initial operator\nobservation can consume the volatile Hermes SSE queue; after reconnect or host\nrestart, a bound operator polls provider run status instead. Chunks already\npublished to AIP remain available, but unobserved provider events may be lost.\n\nDo not infer terminal execution from an ended stream. Require the durable AIP\nresult and, where applicable, the provider's terminal state.\n\nVerify recovery\n\nRestore affected traffic only when every applicable check passes:\n1. /health and /ready agree on a live, non-draining host;\n2. storage and all endpoint probes are ready within the configured bound;\n3. the registry shows the intended replica with a current lease;\n4. metric growth has stabilized and usable capacity remains;\n5. one protected read reaches the intended endpoint and credential;\n6. each retained Action has a justified durable state under its original ID;\n7. provider-local sessions, jobs, responses, and runs are read from their\n   owning endpoint rather than inferred from host readiness;\n8. uncertain operator, approval, cancellation, or tool effects remain paused\n   until terminal evidence exists;\n9. the incident record retains identities, state transitions, errors, and\n   operator decisions.\n\nIf these views disagree, keep the mutation boundary paused. Escalate with the\nsource revisions, artifact and manifest digests, endpoint set, database\nidentities, lease history, readiness body, metric snapshots, Action and\nprovider identifiers, operator binding, and redacted evidence.\n\nRelated documentation\n• Deploy the Hermes Agent connector (deployment.md)\n• Operate the connector host lifecycle (../../../guides/connector-host-lifecycle.md)\n• Observe and recover AIP work (../../../guides/observe-and-recover.md)\n• Run a governed Hermes operator lifecycle (../guides/operator-lifecycle.md)\n• Errors and retry decisions (../../../reference/errors.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "4fd436d51a30b788212adf5dc3e8bce91b803e6f6531d955223b6d526052b86c"
  }
}