Troubleshoot the Hermes Agent connector

Use this runbook to trace a Hermes failure to the signed host, pinned route, endpoint credential, provider request, stream, operator binding, or delegated child. End with one explicit decision: correct, wait, retry a read within policy, reconcile the original work, or escalate.

Pause new model, tool, session, job, operator, and delegation mutations when the provider outcome is uncertain. Preserve the original Action ID, idempotency key, route assignment, endpoint ID, provider handle, and durable state before changing either deployment.

Trust the right error surface

Identify the surface before interpreting a code:

The host can also return unsigned connector_host.response_signing when it cannot sign a response. Treat that as an integrity failure and read the authoritative Action state through the central gateway.

Automate on code, category, retryable, uncertain_outcome, remote status, and durable state. Message text is diagnostic, not a stable interface. The Hermes connector does not populate provider request ID or provider operation fields at this revision.

Choose the first check

Diagnose trust and routing in order

Stop at the first failed layer:

1. connector_host.authentication means the request signer, configured gateway DID, sender, principal kind, or trust domain did not match. Repair the signed gateway path.
2. connector_host.replay means the message failed the host replay boundary. Preserve the rejected envelope and investigate duplicated or altered identity; do not weaken the replay policy.
3. connector_host.route_mismatch means the signed assignment does not match this connector type, version, instance, replica, endpoint, manifest, tenant, binding, or fence. Discard it and resolve a current route.
4. connector_host.credential_revision_denied or connector_host.credential_revision_unavailable means the route-pinned credential policy denied or could not read the revision. Restore that policy; do not select another credential silently.
5. connector_host.approval_authorization_conflict or connector_host.approval_authorization_unavailable means imported approval authority conflicts with durable state or cannot be verified. Preserve the Action and approval.
6. connector.hermes_agent.tenant_mismatch means the verified AIP tenant does not own the endpoint selected by the capability ID. Repair admission and binding; input cannot override the tenant.
7. connector.hermes_agent.missing_credential means a protected route has no endpoint API key. Replace the correct owner-only secret and host process.
8. connector.hermes_agent.authentication means Hermes returned 401 or 403. Verify the endpoint ID, token, API-server policy, and intended deployment.

The remaining common host codes identify availability or execution rather than another credential layer:

After a trust repair, require host readiness and route one protected models read through aipd. A direct Hermes request does not prove tenant, route, host-signature, or credential-revision enforcement.

Separate discovery, endpoint, and provider routing

The connector never takes a base URL, endpoint ID, or token from Action input. Changing the endpoint descriptor or pointing an existing ID at a different Hermes home is a deployment change, not request recovery.

Interpret provider failures by operation safety

The connector retains the endpoint ID in its diagnostic and exposes the remote HTTP status when one was received. It does not expose the provider response body as structured redacted details.

Only a read-only Hermes capability can make a temporary or transport invocation failure retryable. Unsafe invocations mark 429, server, transport, decode, and deadline failures as uncertain and not retryable.

A connector-level remote_rejected does not override operation-specific behavior. For example, a session fork can change source and child rows before returning a title-validation 400. Always consult the state owner before repeating a provider mutation.

Look up base connector codes

These codes are produced by the normal Hermes invocation path:

The category describes the emitted error. The Action and provider-state owner remain authoritative for replay safety.

Resolve stream failures

Parsed stream results have three additional non-retryable codes:

stream_publication differs from those result codes. It means a parsed provider frame could not be committed through AIP lifecycle storage. The provider stream may have continued while the durable AIP sequence stopped.

Use the original Action events and the last verified combined AIP cursor. Provider SSE IDs, event names, chunk IDs, and run_id values are not resume cursors. Closing or losing a follow connection does not cancel the Action.

For chat and Responses streams, do not start a second model turn merely to recover missing text. For structured run events, poll the retained provider run while it exists. For operator recovery, use the durable binding and bound run ID. Require a durable AIP result and provider terminal state where the operation defines one.

Resolve operator failures

The operator has durable result codes in addition to the base connector mapping:

operator_state covers PostgreSQL read, compare-and-set, serialization, or contention failure. operator_policy covers invalid policy, denied delegation, Action-ID collision, or missing binding. Correct the owning state or policy; do not bypass it with a different Action ID.

A bound run_id is still provider-local and volatile. Hermes restart or retention can make it return 404 run_not_found. That does not prove the model or a tool performed no effect. An unbound start is deliberately finalized as outcome_unknown and is never replayed automatically.

Follow the governed operator lifecycle for the source Action, two approval loops, restart, and cancellation decisions.

Resolve delegated-result failures

First-class delegation accepts only the exact durable child result. Use these codes to identify the failed invariant:

The retryable not-ready code means the same child has not settled. It never authorizes creation of another child. Model prose is not a delegated result.

Diagnose sessions, responses, runs, and jobs

Several provider resources are local to one Hermes endpoint:

A successful host readiness probe covers only basic endpoint health. It does not open state.db, prove response-store durability, inspect scheduler claims, or retain structured runs.

Decide whether to retry

Apply this order and stop at the first unmet condition:

1. verify the signed response or read the durable Action result;
2. confirm the exact endpoint-qualified capability and operation;
3. determine whether the capability is read-only;
4. for unsafe work, inspect its Action and provider-owned state; do not retry when the outcome is uncertain or no terminal no-effect evidence exists;
5. require connector implementation retry support and retryable: true;
6. respect attempt, elapsed-time, deadline, and backoff budgets;
7. let the runtime retry the same governed read rather than calling Hermes directly.

The connector has no generic provider transaction ledger that proves not_committed for every Hermes model, tool, session, or job operation. Absence of a provider handle is therefore not permission to repeat unsafe work.

Resolve startup failures

Startup failures occur before a signed AIP error. Inspect the first process error and compare it with the exact configuration. Common owners are:

• empty, oversized, malformed, or unknown-field endpoint JSON;
• invalid or duplicate normalized endpoint IDs;
• non-HTTP(S) endpoint URLs or tenant mismatch;
• missing, unsafe, empty, or oversized endpoint secret files;
• invalid operator-policy fields, bounds, or allowlist relationships;
• PostgreSQL connectivity, role, schema, or recovery failure;
• a manifest above the 512-capability host ceiling;
• host identity, digest, endpoint, revision, or registry admission mismatch.

Correct one owner at a time. After every change, require /ready, registry lease agreement, and one protected model read. Do not combine endpoint, credential, database, artifact, and registry changes in one diagnostic restart.

Verify the resolution

Close the incident only when:

• the signed error or startup failure no longer occurs on the corrected path;
• /ready, the lifecycle lease, and tenant discovery agree;
• a protected read reaches the intended endpoint and credential;
• the original Action is terminal or remains explicitly paused as uncertain;
• every stream, operator, approval, cancellation, and delegated child has a justified state under its original identity;
• session, response, run, and job evidence comes from the owning endpoint;
• no endpoint key, raw sensitive payload, model credential, or personal data entered the incident record.

Escalate with pinned source revisions, artifact and manifest digests, tenant-safe identities, timestamps, signed code and category, retry and uncertainty flags, and remote status. Include Action and provider state, stream cursor, operator binding, child integrity evidence, readiness fields, metric snapshots, and redacted diagnostics.

Related documentation

• Errors and retry decisions
• Hermes authentication and endpoints
• Hermes configuration
• Recover the Hermes Agent connector
• Hermes streaming cursors and cancellation