Recover the Hermes Agent connector

Use this runbook when a Hermes Agent host is unavailable, not ready, losing its lease, rejecting traffic, or retaining work whose provider outcome is uncertain. It helps you locate the failing boundary and recover the same AIP Action, operator binding, or delegated child without starting the work again.

Run these checks through an operator-controlled network. The procedure assumes the fixed endpoint set and shared PostgreSQL boundary established in Deploy the Hermes Agent connector.

Contain risk before diagnosis

Pause new model, tool, session, job, operator, and delegation mutations for the affected binding when durable storage is unavailable, the host lease is invalid, or an accepted operation has an unknown outcome. Keep bounded reads available only when tenant and endpoint ownership remain verified.

Do not restart Hermes, clear either database, rotate an endpoint, create a new Action ID, or move pinned work as the first response. Retain:

• UTC incident start and the last known-ready time;
• connector type, version, instance, replica, artifact, and manifest digest;
• tenant, endpoint ID, provider origin, configuration, and credential revision;
• lease sequence, lease expiry, and registry status;
• Action, idempotency, correlation, session, approval, delegation, and run IDs;
• /ready plus two timestamped /metrics snapshots;
• AIP results, events, chunks, receipts, and redacted provider evidence;
• operator binding status and provider job or session identity when relevant.

A healthy replacement cannot make an earlier uncertain model or tool effect safe to repeat.

Read six views in order

No single surface proves the outcome of an Action.

Capture the host surfaces without submitting an Action:

export HERMES_HOST_OBSERVE_ORIGIN=https://hermes-connector.internal.example

curl --silent --show-error "$HERMES_HOST_OBSERVE_ORIGIN/health"
curl --silent --show-error \
  --write-out '\nHTTP %{http_code}\n' \
  "$HERMES_HOST_OBSERVE_ORIGIN/ready"
curl --silent --show-error "$HERMES_HOST_OBSERVE_ORIGIN/metrics"

Do not expose these unauthenticated host routes to an untrusted network. Scraping /metrics does not exercise the endpoint Bearer token.

Interpret host readiness precisely

/health returns static process metadata. /ready returns HTTP 200 only when all four gates pass and HTTP 503 otherwise:

The response includes runtime.durability and a startup recovery summary:

This summary is startup evidence, not a live backlog. A queued-action or delegation recovery error blocks registration. A broader storage or callback recovery failure also stops host construction instead of producing readiness.

Connector readiness visits every endpoint in normalized endpoint order and sends unauthenticated GET /health. Every endpoint must return parsable JSON with a successful HTTP status. The readiness loop does not require the JSON status field to equal ok.

The configured host health timeout wraps the complete multi-endpoint loop. Its default is 2000 milliseconds, even though one direct endpoint health request has a 30-second client timeout. A slow first endpoint can therefore end the whole host probe before later endpoints are visited.

Basic endpoint health does not prove Bearer authentication, model-provider credentials, MCP connectivity, tool availability, or operator policy. Confirm the intended endpoint with the authenticated health_detailed or models capability after host readiness returns.

Interpret the eight host metrics

The host emits exactly eight unlabeled Prometheus metrics:

The metrics endpoint returns HTTP 200 even when the ready gauge is 0. Storage and connector gauges are refreshed by readiness checks and heartbeat probes, not by a new probe inside each metrics scrape. Counters and gauges are process-local and reset when the host restarts.

This revision exposes no Hermes endpoint label, provider status, latency, model, MCP, run, approval, session, job, operator, or delegated-result metric. Use durable AIP records and bounded provider reads for those views.

Define alert windows from the deployment SLO. At minimum, alert on these relationships:

• readiness remains 0 outside a planned drain;
• storage readiness remains 0 or /ready reports storage failure;
• heartbeat failures grow as the current lease approaches expiry;
• lease-recovery attempts grow without a successful recovery;
• in-flight work remains at the configured concurrency bound;
• rejected messages grow faster than total native messages;
• basic endpoint health passes while a protected read fails;
• an accepted Action stops producing durable state within its contract bound.

Classify the failing boundary

Recover process or lease loss

If the host still runs, repair lifecycle DNS, TLS, CA trust, pinned DID, clock, or service availability before replacing it. The host retains its last lease until expiry. While the lease is valid, each heartbeat reports the combined storage and connector result. An unhealthy report moves the replica offline.

After lease expiry, a healthy host uses signed re-registration instead of a heartbeat. An ambiguous lifecycle response is retried with the same request identity and registration bytes. A definite admission or configuration rejection clears that request; correct the immutable mismatch before the next attempt. Never edit lease expiry, sequence, or registry status by hand.

After a hard crash:

1. prove that no process still owns the replica key and endpoint;
2. preserve the same PostgreSQL database, artifact, instance, and replica;
3. preserve each Hermes endpoint’s local state and provider identity;
4. wait for clean offline state or lease expiry;
5. start the reviewed process and let recovery finish before registration;
6. stop if recovery reports queued-action or delegation errors;
7. compare recovery counts, retained assignments, and the new lease sequence;
8. require host readiness and one protected read before restoring traffic.

Use a new replica identity for an actual replacement. Never run two processes with the same replica ID or signing key, and never point overlapping replicas at independent connector databases.

Recover storage, endpoints, and capacity

When runtime.ready is false, restore connectivity to the exact PostgreSQL database. Check capacity, certificates, role permissions, schema, and read/write health. Do not initialize a clean store; that removes the state needed to fence Actions, approvals, callbacks, operator bindings, and child results.

When connector.ready is false, identify the failing endpoint from the bounded connector detail and endpoint evidence. Check its fixed origin, network trust, Hermes process, and JSON health response. One failed endpoint keeps the complete host unready. Restore it or change the endpoint set through the reviewed deployment and admission process.

A protected read failure requires separate checks for endpoint ID, Bearer token, Hermes API-server policy, model route, and upstream credentials. Replacing token bytes requires a host restart because endpoint keys are read at startup.

At local capacity, a new Action receives retryable connector_host.capacity with HTTP 429. A draining, lease-invalid, or connector-unready host returns retryable connector_host.not_ready. Preserve the original Action contract and honor backoff. A retryable transport label does not prove that Hermes or a tool performed no work.

Recover by provider state owner

First read the durable AIP Action, including its result, events, chunks, approval, and delegation links. Then use the operation’s provider-owned state:

Endpoint identity includes its local Hermes home and runtime state. Do not redirect an existing endpoint ID to an empty Hermes instance and treat a missing response, session, job, or run as proof that no effect occurred.

Recover an operator or delegated result

The standalone host stores operator bindings in the same PostgreSQL-backed profile state used by its runtime. Each binding retains the source Action, endpoint, request fingerprint, provider run ID when known, cancellation intent, approval generation and commands, sequence, terminal output, and typed error.

Use this decision order:

1. read the source Action and retained operator binding;
2. reject any endpoint or request-fingerprint mismatch;
3. return a retained terminal result without contacting Hermes again;
4. when run_id is bound, poll that exact run and reconcile cancellation;
5. when start ownership disappeared before run_id was stored, keep outcome_unknown and do not replay POST /v1/runs;
6. when an approval command is claimed without a durable outcome, preserve its uncertain state instead of sending the choice again;
7. require a Hermes terminal status before calling cancellation confirmed.

A provider 404 for a bound run can mean provider restart or retention loss. It does not prove that models or tools had no effect.

For first-class delegation, the resolver reads the exact child Action from the durable queue and lifecycle stores. It verifies child identity, tenant, transport-authenticated Hermes principal, and immutable execution contract. A terminal record without a result, conflicting queue and lifecycle results, or a missing expected child is an integrity or policy failure. Never substitute the Hermes model’s final prose for that result.

Preserve stream and event evidence

AIP Action events and chunks use durable runtime positions. Persist the opaque combined cursor only after processing its page. Provider SSE event IDs, provider run IDs, chunk IDs, and AIP cursors are not interchangeable.

A lost follow connection does not cancel an Action. Initial operator observation can consume the volatile Hermes SSE queue; after reconnect or host restart, a bound operator polls provider run status instead. Chunks already published to AIP remain available, but unobserved provider events may be lost.

Do not infer terminal execution from an ended stream. Require the durable AIP result and, where applicable, the provider’s terminal state.

Verify recovery

Restore affected traffic only when every applicable check passes:

1. /health and /ready agree on a live, non-draining host;
2. storage and all endpoint probes are ready within the configured bound;
3. the registry shows the intended replica with a current lease;
4. metric growth has stabilized and usable capacity remains;
5. one protected read reaches the intended endpoint and credential;
6. each retained Action has a justified durable state under its original ID;
7. provider-local sessions, jobs, responses, and runs are read from their owning endpoint rather than inferred from host readiness;
8. uncertain operator, approval, cancellation, or tool effects remain paused until terminal evidence exists;
9. the incident record retains identities, state transitions, errors, and operator decisions.

If these views disagree, keep the mutation boundary paused. Escalate with the source revisions, artifact and manifest digests, endpoint set, database identities, lease history, readiness body, metric snapshots, Action and provider identifiers, operator binding, and redacted evidence.

Related documentation

• Deploy the Hermes Agent connector
• Operate the connector host lifecycle
• Observe and recover AIP work
• Run a governed Hermes operator lifecycle
• Errors and retry decisions