{
  "schemaVersion": "1.0",
  "title": "Troubleshoot the Hermes Agent connector",
  "description": "Use this runbook to trace a Hermes failure to the signed host, pinned route, endpoint credential, provider request, stream, operator binding, or delegated child. End with one explicit decision: correct, wait, retry a read within policy, rec",
  "canonical": "https://getaip.org/docs/connectors/hermes-agent/troubleshooting",
  "route": "/docs/connectors/hermes-agent/troubleshooting",
  "source": "docs/connectors/hermes-agent/troubleshooting/README.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/hermes-agent/troubleshooting.md",
    "txt": "/docs/download/connectors/hermes-agent/troubleshooting.txt",
    "json": "/docs/download/connectors/hermes-agent/troubleshooting.json",
    "pdf": "/docs/download/connectors/hermes-agent/troubleshooting.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Troubleshoot the Hermes Agent connector\ndescription: >-\n  Classify Hermes host, endpoint, stream, operator, and delegation failures\n  and choose a correction, bounded retry, reconciliation, or escalation\nkind: runbook\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: hermes-agent\n---\n\n# Troubleshoot the Hermes Agent connector\n\nUse this runbook to trace a Hermes failure to the signed host, pinned route,\nendpoint credential, provider request, stream, operator binding, or delegated\nchild. End with one explicit decision: correct, wait, retry a read within\npolicy, reconcile the original work, or escalate.\n\nPause new model, tool, session, job, operator, and delegation mutations when\nthe provider outcome is uncertain. Preserve the original Action ID,\nidempotency key, route assignment, endpoint ID, provider handle, and durable\nstate before changing either deployment.\n\n## Trust the right error surface\n\nIdentify the surface before interpreting a code:\n\n| Surface | Evidence shape | Trust decision |\n|---|---|---|\n| Pre-envelope host rejection | Unsigned `schema.envelope.invalid` or `envelope.decode.invalid` HTTP error | Trust only the controlled TLS path; no valid envelope exists to sign |\n| Connector-host response | Signed AIP error envelope with a `ProtocolError` | Verify host signature, sender, recipient, and correlation |\n| Durable Action lifecycle | `ActionResult` plus events, chunks, and typed links | Read this even when the original HTTP exchange was interrupted |\n| Process startup | Error text and non-zero exit | Compare exact configuration, files, storage, and admission; no protocol code exists yet |\n\nThe host can also return unsigned `connector_host.response_signing` when it\ncannot sign a response. Treat that as an integrity failure and read the\nauthoritative Action state through the central gateway.\n\nAutomate on `code`, category, `retryable`, `uncertain_outcome`, remote status,\nand durable state. Message text is diagnostic, not a stable interface. The\nHermes connector does not populate provider request ID or provider operation\nfields at this revision.\n\n## Choose the first check\n\n| Symptom | First check | Safe first action |\n|---|---|---|\n| Capability is absent | Authenticated tenant catalogue and enabled binding | Correct discovery or admission; do not call the host directly |\n| Host returns `401` | `connector_host.authentication` versus `connector.hermes_agent.authentication` | Repair only the failed trust layer |\n| Host returns `403` | Signed route, credential revision, approval, tenant, or provider code | Re-resolve the exact authority boundary; do not bypass it |\n| `/health` works and `/ready` fails | Drain, lease, runtime, and connector fields | Follow the failing readiness owner |\n| Basic health works but protected reads fail | Endpoint ID, token, Hermes policy, and model route | Keep model and tool traffic paused |\n| Read gets `429`, `5xx`, transport, or deadline failure | Read-only contract, retry flag, and budget | Retry only through the same governed Action policy |\n| Chat, mutation, or operator gets an ambiguous failure | `uncertain_outcome` plus AIP and provider state | Reconcile; do not submit a replacement |\n| Stream closes or stops publishing | AIP chunks, cursor, terminal flag, and provider state | Resume evidence collection; do not infer completion |\n| Operator returns `outcome_unknown` | Source Action and durable binding | Preserve the binding; never replay the start or command |\n| Delegation does not settle | Exact child Action, owner, tenant, queue, and lifecycle result | Wait on the same child or fail closed |\n| Session, response, run, or job is missing | Owning endpoint and its local state | Treat absence as a state-location problem, not no-effect proof |\n\n## Diagnose trust and routing in order\n\nStop at the first failed layer:\n\n1. `connector_host.authentication` means the request signer, configured\n   gateway DID, sender, principal kind, or trust domain did not match. Repair\n   the signed gateway path.\n2. `connector_host.replay` means the message failed the host replay boundary.\n   Preserve the rejected envelope and investigate duplicated or altered\n   identity; do not weaken the replay policy.\n3. `connector_host.route_mismatch` means the signed assignment does not match\n   this connector type, version, instance, replica, endpoint, manifest, tenant,\n   binding, or fence. Discard it and resolve a current route.\n4. `connector_host.credential_revision_denied` or\n   `connector_host.credential_revision_unavailable` means the route-pinned\n   credential policy denied or could not read the revision. Restore that\n   policy; do not select another credential silently.\n5. `connector_host.approval_authorization_conflict` or\n   `connector_host.approval_authorization_unavailable` means imported approval\n   authority conflicts with durable state or cannot be verified. Preserve the\n   Action and approval.\n6. `connector.hermes_agent.tenant_mismatch` means the verified AIP tenant does\n   not own the endpoint selected by the capability ID. Repair admission and\n   binding; input cannot override the tenant.\n7. `connector.hermes_agent.missing_credential` means a protected route has no\n   endpoint API key. Replace the correct owner-only secret and host process.\n8. `connector.hermes_agent.authentication` means Hermes returned `401` or\n   `403`. Verify the endpoint ID, token, API-server policy, and intended\n   deployment.\n\nThe remaining common host codes identify availability or execution rather\nthan another credential layer:\n\n| Code | Decision |\n|---|---|\n| `connector_host.not_ready` | Wait for drain, lease, and connector readiness under the original Action policy |\n| `connector_host.capacity` | Honor bounded backoff; do not bypass the admitted replica |\n| `connector_host.execution` | Read the durable Action and underlying typed connector error before deciding |\n| `connector_host.response_signing` | Isolate the host response and use central durable state; authenticity is incomplete |\n\nAfter a trust repair, require host readiness and route one protected\n`models` read through `aipd`. A direct Hermes request does not prove tenant,\nroute, host-signature, or credential-revision enforcement.\n\n## Separate discovery, endpoint, and provider routing\n\n| Observation | Owner | Decision |\n|---|---|---|\n| Tenant catalogue omits the capability | Admitted version, implementation support, or binding | Correct the signed catalogue or use a published capability |\n| Catalogue has it but no eligible replica exists | Lease, health, topology, capacity, or binding | Restore a ready admitted replica |\n| Host reports route mismatch | Assignment versus immutable host identity | Resolve again; never edit the signed route |\n| Capability selects the wrong endpoint ID | Application discovery choice | Stop and select the intended endpoint-qualified capability |\n| Provider output belongs to another Hermes home or model route | Endpoint origin and deployment state | Treat as an isolation or endpoint-replacement incident |\n| Basic health passes but protected model read fails | Provider auth or upstream model boundary | Repair the protected path before enabling model work |\n\nThe connector never takes a base URL, endpoint ID, or token from Action input.\nChanging the endpoint descriptor or pointing an existing ID at a different\nHermes home is a deployment change, not request recovery.\n\n## Interpret provider failures by operation safety\n\nThe connector retains the endpoint ID in its diagnostic and exposes the remote\nHTTP status when one was received. It does not expose the provider response\nbody as structured redacted details.\n\n| Provider observation | Connector code | Read-only decision | Unsafe-operation decision |\n|---|---|---|---|\n| HTTP `401` or `403` | `connector.hermes_agent.authentication` | Correct endpoint auth before a new read | Correct authority; do not replay accepted work |\n| HTTP `429` or `500`–`599` | `connector.hermes_agent.remote_temporary` | Runtime retry may be eligible within policy | Outcome is uncertain and automatic retry is disabled |\n| Other non-success status | `connector.hermes_agent.remote_rejected` | Correct request or provider state | Follow the operation owner; some provider operations are non-atomic |\n| Request, body, or SSE decode failure | `connector.hermes_agent.transport` | Runtime retry may be eligible within policy | Outcome is uncertain and automatic retry is disabled |\n| Trusted execution deadline | `connector.hermes_agent.deadline_exceeded` | Runtime retry may be eligible within policy | Provider work may continue; reconcile |\n| Runtime cancellation | `connector.hermes_agent.cancelled` | Read the original Action | Model, tool, message, or job effects can predate cancellation |\n| AIP chunk publication failure | `connector.hermes_agent.stream_publication` | Preserve published chunks and source state | Provider work and missing AIP chunks are uncertain |\n\nOnly a read-only Hermes capability can make a temporary or transport\ninvocation failure retryable. Unsafe invocations mark `429`, server,\ntransport, decode, and deadline failures as uncertain and not retryable.\n\nA connector-level `remote_rejected` does not override operation-specific\nbehavior. For example, a session fork can change source and child rows before\nreturning a title-validation `400`. Always consult the state owner before\nrepeating a provider mutation.\n\n## Look up base connector codes\n\nThese codes are produced by the normal Hermes invocation path:\n\n| Code | Category and automatic retry | Decision |\n|---|---|---|\n| `connector.hermes_agent.configuration_or_input` | Permanent, false | Correct endpoint, URL, capability, identifier, input, or local client configuration |\n| `connector.hermes_agent.tenant_mismatch` | Auth, false | Repair the verified tenant-to-endpoint binding |\n| `connector.hermes_agent.missing_credential` | Auth, false | Install the endpoint key and replace the host process |\n| `connector.hermes_agent.authentication` | Auth, false | Correct the selected endpoint token or provider access |\n| `connector.hermes_agent.remote_temporary` | Temporary, read-only conditional | Retry a read within policy; reconcile unsafe work |\n| `connector.hermes_agent.remote_rejected` | Permanent, false | Correct provider state, then apply the operation-specific recovery rule |\n| `connector.hermes_agent.transport` | Transport, read-only conditional | Retry a read within policy; reconcile unsafe work |\n| `connector.hermes_agent.stream_publication` | Connector, false, uncertain | Preserve provider and AIP stream evidence; do not restart the operation |\n| `connector.hermes_agent.cancelled` | Temporary, false | Inspect provider effects and the original Action |\n| `connector.hermes_agent.deadline_exceeded` | Temporary, read-only conditional | Retry only a read; reconcile unsafe work |\n| `connector.hermes_agent.operator_outcome_unknown` | Temporary, false, uncertain | Keep the source binding and do not replay the operator |\n| `connector.hermes_agent.operator_state` | Temporary, false | Restore the exact PostgreSQL state and resolve contention or corruption |\n| `connector.hermes_agent.operator_policy` | Permanent, false | Correct policy, approval, allowlist, binding, or idempotency collision |\n| `connector.hermes_agent.error` | Connector, false | Retain the wrapped error and escalate an unexpected connector path |\n\nThe category describes the emitted error. The Action and provider-state owner\nremain authoritative for replay safety.\n\n## Resolve stream failures\n\nParsed stream results have three additional non-retryable codes:\n\n| Code | Meaning | Decision |\n|---|---|---|\n| `connector.hermes_agent.stream_truncated` | The configured event limit ended consumption before terminal state | Read retained AIP chunks and provider state; do not rerun unsafe generation |\n| `connector.hermes_agent.stream_incomplete` | The provider stream closed without an explicit terminal event | Treat execution as nonterminal or uncertain until another state owner settles it |\n| `connector.hermes_agent.stream_failed` | Hermes sent a recognized terminal failure event | Retain the failure and inspect prior model, tool, message, and delivery effects |\n\n`stream_publication` differs from those result codes. It means a parsed\nprovider frame could not be committed through AIP lifecycle storage. The\nprovider stream may have continued while the durable AIP sequence stopped.\n\nUse the original Action events and the last verified combined AIP cursor.\nProvider SSE IDs, event names, chunk IDs, and `run_id` values are not resume\ncursors. Closing or losing a follow connection does not cancel the Action.\n\nFor chat and Responses streams, do not start a second model turn merely to\nrecover missing text. For structured run events, poll the retained provider\nrun while it exists. For operator recovery, use the durable binding and bound\nrun ID. Require a durable AIP result and provider terminal state where the\noperation defines one.\n\n## Resolve operator failures\n\nThe operator has durable result codes in addition to the base connector\nmapping:\n\n| Code | Meaning | Decision |\n|---|---|---|\n| `connector.hermes_agent.operator_start_rejected` | Hermes returned a definitive `4xx` before accepting the start | Correct the request and create new reviewed work only after the original Action is terminal |\n| `connector.hermes_agent.operator_failed` | The bound provider run reported terminal failure | Retain output and error; inspect tool and child effects |\n| `connector.hermes_agent.operator_outcome_unknown` | Start response, run binding, or governed timeout left provider outcome unknown | Never replay `POST /v1/runs`; reconcile the retained binding |\n| `connector.hermes_agent.approval_outcome_unknown` | Provider may have applied an approval response before its result was stored | Do not resend the choice; inspect the same run and approval generation |\n| `connector.hermes_agent.cancellation_outcome_unknown` | Stop did not reach a confirmed provider terminal state | Keep cancellation intent and poll or escalate the same run |\n\n`operator_state` covers PostgreSQL read, compare-and-set, serialization, or\ncontention failure. `operator_policy` covers invalid policy, denied delegation,\nAction-ID collision, or missing binding. Correct the owning state or policy;\ndo not bypass it with a different Action ID.\n\nA bound `run_id` is still provider-local and volatile. Hermes restart or\nretention can make it return `404 run_not_found`. That does not prove the model\nor a tool performed no effect. An unbound start is deliberately finalized as\n`outcome_unknown` and is never replayed automatically.\n\nFollow the [governed operator lifecycle](../guides/operator-lifecycle.md) for\nthe source Action, two approval loops, restart, and cancellation decisions.\n\n## Resolve delegated-result failures\n\nFirst-class delegation accepts only the exact durable child result. Use these\ncodes to identify the failed invariant:\n\n| Code | Category and retry | Decision |\n|---|---|---|\n| `connector.hermes_agent.delegated_result_not_ready` | Temporary, true | Continue waiting on the same child within the original deadline |\n| `connector.hermes_agent.delegated_execution_not_observed` | Policy, false | Hermes finished without creating the expected child; do not invent a result |\n| `connector.hermes_agent.delegated_action_contract_mismatch` | Policy, false | Isolate the route; the child contract differs from the recorded request |\n| `connector.hermes_agent.delegated_identity_unverified` | Auth, false | Restore transport-authenticated owner and `action:write` authority |\n| `connector.hermes_agent.delegated_identity_mismatch` | Auth, false | Isolate the child created by the wrong Hermes principal |\n| `connector.hermes_agent.delegated_identity_integrity` | Auth, false | Preserve the Action and trusted identity context; fail closed |\n| `connector.hermes_agent.delegated_tenant_mismatch` | Auth, false | Isolate cross-tenant execution and preserve evidence |\n| `connector.hermes_agent.delegated_tenant_expired` | Auth, false | Renew authority through policy; do not rewrite stored membership |\n| `connector.hermes_agent.delegated_result_integrity` | Connector, false | Preserve queue and lifecycle records that disagree or lack a terminal result |\n| `connector.hermes_agent.delegated_result_resolver_missing` | Connector, false | Restore the host's authoritative queue and lifecycle resolver |\n\nThe retryable not-ready code means the same child has not settled. It never\nauthorizes creation of another child. Model prose is not a delegated result.\n\n## Diagnose sessions, responses, runs, and jobs\n\nSeveral provider resources are local to one Hermes endpoint:\n\n| Symptom | State owner | Decision |\n|---|---|---|\n| Stored response returns `404` | Endpoint response SQLite or in-memory fallback | Verify the owning endpoint and original Action; do not repeat generation |\n| Session route returns `session_db_unavailable` through provider `503` | Endpoint `state.db` | Restore that database; mutation outcome still requires state inspection |\n| Session chat succeeded but messages are absent | Model result versus transcript persistence | Keep the result, inspect logs, and do not claim a durable append |\n| Session fork returns title-validation `400` | Source and child session rows | Inspect both rows because the provider fork is not atomic |\n| Structured run returns `run_not_found` | In-process run maps and retention | Preserve AIP evidence; do not recreate the run |\n| Job mutation times out or is cancelled | Profile-local scheduler store | Read the job before retrying; persistence may have won the race |\n| Job exists but no output was delivered | Scheduler claim, execution output, and delivery target | Separate stored schedule from run and delivery evidence |\n\nA successful host readiness probe covers only basic endpoint health. It does\nnot open `state.db`, prove response-store durability, inspect scheduler\nclaims, or retain structured runs.\n\n## Decide whether to retry\n\nApply this order and stop at the first unmet condition:\n\n1. verify the signed response or read the durable Action result;\n2. confirm the exact endpoint-qualified capability and operation;\n3. determine whether the capability is read-only;\n4. for unsafe work, inspect its Action and provider-owned state; do not retry\n   when the outcome is uncertain or no terminal no-effect evidence exists;\n5. require connector implementation retry support and `retryable: true`;\n6. respect attempt, elapsed-time, deadline, and backoff budgets;\n7. let the runtime retry the same governed read rather than calling Hermes\n   directly.\n\nThe connector has no generic provider transaction ledger that proves\n`not_committed` for every Hermes model, tool, session, or job operation.\nAbsence of a provider handle is therefore not permission to repeat unsafe work.\n\n## Resolve startup failures\n\nStartup failures occur before a signed AIP error. Inspect the first process\nerror and compare it with the exact configuration. Common owners are:\n\n- empty, oversized, malformed, or unknown-field endpoint JSON;\n- invalid or duplicate normalized endpoint IDs;\n- non-HTTP(S) endpoint URLs or tenant mismatch;\n- missing, unsafe, empty, or oversized endpoint secret files;\n- invalid operator-policy fields, bounds, or allowlist relationships;\n- PostgreSQL connectivity, role, schema, or recovery failure;\n- a manifest above the 512-capability host ceiling;\n- host identity, digest, endpoint, revision, or registry admission mismatch.\n\nCorrect one owner at a time. After every change, require `/ready`, registry\nlease agreement, and one protected model read. Do not combine endpoint,\ncredential, database, artifact, and registry changes in one diagnostic restart.\n\n## Verify the resolution\n\nClose the incident only when:\n\n- the signed error or startup failure no longer occurs on the corrected path;\n- `/ready`, the lifecycle lease, and tenant discovery agree;\n- a protected read reaches the intended endpoint and credential;\n- the original Action is terminal or remains explicitly paused as uncertain;\n- every stream, operator, approval, cancellation, and delegated child has a\n  justified state under its original identity;\n- session, response, run, and job evidence comes from the owning endpoint;\n- no endpoint key, raw sensitive payload, model credential, or personal data\n  entered the incident record.\n\nEscalate with pinned source revisions, artifact and manifest digests,\ntenant-safe identities, timestamps, signed code and category, retry and\nuncertainty flags, and remote status. Include Action and provider state,\nstream cursor, operator binding, child integrity evidence, readiness fields,\nmetric snapshots, and redacted diagnostics.\n\n## Related documentation\n\n- [Errors and retry decisions](../../../reference/errors.md)\n- [Hermes authentication and endpoints](../getting-started/authentication-and-endpoints.md)\n- [Hermes configuration](../reference/configuration.md)\n- [Recover the Hermes Agent connector](../operations/health-observability-and-recovery.md)\n- [Hermes streaming cursors and cancellation](../reference/streaming-cursors-and-cancellation.md)\n",
    "text": "Troubleshoot the Hermes Agent connector\n\nUse this runbook to trace a Hermes failure to the signed host, pinned route,\nendpoint credential, provider request, stream, operator binding, or delegated\nchild. End with one explicit decision: correct, wait, retry a read within\npolicy, reconcile the original work, or escalate.\n\nPause new model, tool, session, job, operator, and delegation mutations when\nthe provider outcome is uncertain. Preserve the original Action ID,\nidempotency key, route assignment, endpoint ID, provider handle, and durable\nstate before changing either deployment.\n\nTrust the right error surface\n\nIdentify the surface before interpreting a code:\n\n| Surface | Evidence shape | Trust decision |\n\n| Pre-envelope host rejection | Unsigned schema.envelope.invalid or envelope.decode.invalid HTTP error | Trust only the controlled TLS path; no valid envelope exists to sign |\n| Connector-host response | Signed AIP error envelope with a ProtocolError | Verify host signature, sender, recipient, and correlation |\n| Durable Action lifecycle | ActionResult plus events, chunks, and typed links | Read this even when the original HTTP exchange was interrupted |\n| Process startup | Error text and non-zero exit | Compare exact configuration, files, storage, and admission; no protocol code exists yet |\n\nThe host can also return unsigned connectorhost.responsesigning when it\ncannot sign a response. Treat that as an integrity failure and read the\nauthoritative Action state through the central gateway.\n\nAutomate on code, category, retryable, uncertainoutcome, remote status,\nand durable state. Message text is diagnostic, not a stable interface. The\nHermes connector does not populate provider request ID or provider operation\nfields at this revision.\n\nChoose the first check\n\n| Symptom | First check | Safe first action |\n\n| Capability is absent | Authenticated tenant catalogue and enabled binding | Correct discovery or admission; do not call the host directly |\n| Host returns 401 | connectorhost.authentication versus connector.hermesagent.authentication | Repair only the failed trust layer |\n| Host returns 403 | Signed route, credential revision, approval, tenant, or provider code | Re-resolve the exact authority boundary; do not bypass it |\n| /health works and /ready fails | Drain, lease, runtime, and connector fields | Follow the failing readiness owner |\n| Basic health works but protected reads fail | Endpoint ID, token, Hermes policy, and model route | Keep model and tool traffic paused |\n| Read gets 429, 5xx, transport, or deadline failure | Read-only contract, retry flag, and budget | Retry only through the same governed Action policy |\n| Chat, mutation, or operator gets an ambiguous failure | uncertainoutcome plus AIP and provider state | Reconcile; do not submit a replacement |\n| Stream closes or stops publishing | AIP chunks, cursor, terminal flag, and provider state | Resume evidence collection; do not infer completion |\n| Operator returns outcomeunknown | Source Action and durable binding | Preserve the binding; never replay the start or command |\n| Delegation does not settle | Exact child Action, owner, tenant, queue, and lifecycle result | Wait on the same child or fail closed |\n| Session, response, run, or job is missing | Owning endpoint and its local state | Treat absence as a state-location problem, not no-effect proof |\n\nDiagnose trust and routing in order\n\nStop at the first failed layer:\n1. connectorhost.authentication means the request signer, configured\n   gateway DID, sender, principal kind, or trust domain did not match. Repair\n   the signed gateway path.\n2. connectorhost.replay means the message failed the host replay boundary.\n   Preserve the rejected envelope and investigate duplicated or altered\n   identity; do not weaken the replay policy.\n3. connectorhost.routemismatch means the signed assignment does not match\n   this connector type, version, instance, replica, endpoint, manifest, tenant,\n   binding, or fence. Discard it and resolve a current route.\n4. connectorhost.credentialrevisiondenied or\n   connectorhost.credentialrevisionunavailable means the route-pinned\n   credential policy denied or could not read the revision. Restore that\n   policy; do not select another credential silently.\n5. connectorhost.approvalauthorizationconflict or\n   connectorhost.approvalauthorizationunavailable means imported approval\n   authority conflicts with durable state or cannot be verified. Preserve the\n   Action and approval.\n6. connector.hermesagent.tenantmismatch means the verified AIP tenant does\n   not own the endpoint selected by the capability ID. Repair admission and\n   binding; input cannot override the tenant.\n7. connector.hermesagent.missingcredential means a protected route has no\n   endpoint API key. Replace the correct owner-only secret and host process.\n8. connector.hermesagent.authentication means Hermes returned 401 or\n   403. Verify the endpoint ID, token, API-server policy, and intended\n   deployment.\n\nThe remaining common host codes identify availability or execution rather\nthan another credential layer:\n\n| Code | Decision |\n\n| connectorhost.notready | Wait for drain, lease, and connector readiness under the original Action policy |\n| connectorhost.capacity | Honor bounded backoff; do not bypass the admitted replica |\n| connectorhost.execution | Read the durable Action and underlying typed connector error before deciding |\n| connectorhost.responsesigning | Isolate the host response and use central durable state; authenticity is incomplete |\n\nAfter a trust repair, require host readiness and route one protected\nmodels read through aipd. A direct Hermes request does not prove tenant,\nroute, host-signature, or credential-revision enforcement.\n\nSeparate discovery, endpoint, and provider routing\n\n| Observation | Owner | Decision |\n\n| Tenant catalogue omits the capability | Admitted version, implementation support, or binding | Correct the signed catalogue or use a published capability |\n| Catalogue has it but no eligible replica exists | Lease, health, topology, capacity, or binding | Restore a ready admitted replica |\n| Host reports route mismatch | Assignment versus immutable host identity | Resolve again; never edit the signed route |\n| Capability selects the wrong endpoint ID | Application discovery choice | Stop and select the intended endpoint-qualified capability |\n| Provider output belongs to another Hermes home or model route | Endpoint origin and deployment state | Treat as an isolation or endpoint-replacement incident |\n| Basic health passes but protected model read fails | Provider auth or upstream model boundary | Repair the protected path before enabling model work |\n\nThe connector never takes a base URL, endpoint ID, or token from Action input.\nChanging the endpoint descriptor or pointing an existing ID at a different\nHermes home is a deployment change, not request recovery.\n\nInterpret provider failures by operation safety\n\nThe connector retains the endpoint ID in its diagnostic and exposes the remote\nHTTP status when one was received. It does not expose the provider response\nbody as structured redacted details.\n\n| Provider observation | Connector code | Read-only decision | Unsafe-operation decision |\n\n| HTTP 401 or 403 | connector.hermesagent.authentication | Correct endpoint auth before a new read | Correct authority; do not replay accepted work |\n| HTTP 429 or 500–599 | connector.hermesagent.remotetemporary | Runtime retry may be eligible within policy | Outcome is uncertain and automatic retry is disabled |\n| Other non-success status | connector.hermesagent.remoterejected | Correct request or provider state | Follow the operation owner; some provider operations are non-atomic |\n| Request, body, or SSE decode failure | connector.hermesagent.transport | Runtime retry may be eligible within policy | Outcome is uncertain and automatic retry is disabled |\n| Trusted execution deadline | connector.hermesagent.deadlineexceeded | Runtime retry may be eligible within policy | Provider work may continue; reconcile |\n| Runtime cancellation | connector.hermesagent.cancelled | Read the original Action | Model, tool, message, or job effects can predate cancellation |\n| AIP chunk publication failure | connector.hermesagent.streampublication | Preserve published chunks and source state | Provider work and missing AIP chunks are uncertain |\n\nOnly a read-only Hermes capability can make a temporary or transport\ninvocation failure retryable. Unsafe invocations mark 429, server,\ntransport, decode, and deadline failures as uncertain and not retryable.\n\nA connector-level remoterejected does not override operation-specific\nbehavior. For example, a session fork can change source and child rows before\nreturning a title-validation 400. Always consult the state owner before\nrepeating a provider mutation.\n\nLook up base connector codes\n\nThese codes are produced by the normal Hermes invocation path:\n\n| Code | Category and automatic retry | Decision |\n\n| connector.hermesagent.configurationorinput | Permanent, false | Correct endpoint, URL, capability, identifier, input, or local client configuration |\n| connector.hermesagent.tenantmismatch | Auth, false | Repair the verified tenant-to-endpoint binding |\n| connector.hermesagent.missingcredential | Auth, false | Install the endpoint key and replace the host process |\n| connector.hermesagent.authentication | Auth, false | Correct the selected endpoint token or provider access |\n| connector.hermesagent.remotetemporary | Temporary, read-only conditional | Retry a read within policy; reconcile unsafe work |\n| connector.hermesagent.remoterejected | Permanent, false | Correct provider state, then apply the operation-specific recovery rule |\n| connector.hermesagent.transport | Transport, read-only conditional | Retry a read within policy; reconcile unsafe work |\n| connector.hermesagent.streampublication | Connector, false, uncertain | Preserve provider and AIP stream evidence; do not restart the operation |\n| connector.hermesagent.cancelled | Temporary, false | Inspect provider effects and the original Action |\n| connector.hermesagent.deadlineexceeded | Temporary, read-only conditional | Retry only a read; reconcile unsafe work |\n| connector.hermesagent.operatoroutcomeunknown | Temporary, false, uncertain | Keep the source binding and do not replay the operator |\n| connector.hermesagent.operatorstate | Temporary, false | Restore the exact PostgreSQL state and resolve contention or corruption |\n| connector.hermesagent.operatorpolicy | Permanent, false | Correct policy, approval, allowlist, binding, or idempotency collision |\n| connector.hermesagent.error | Connector, false | Retain the wrapped error and escalate an unexpected connector path |\n\nThe category describes the emitted error. The Action and provider-state owner\nremain authoritative for replay safety.\n\nResolve stream failures\n\nParsed stream results have three additional non-retryable codes:\n\n| Code | Meaning | Decision |\n\n| connector.hermesagent.streamtruncated | The configured event limit ended consumption before terminal state | Read retained AIP chunks and provider state; do not rerun unsafe generation |\n| connector.hermesagent.streamincomplete | The provider stream closed without an explicit terminal event | Treat execution as nonterminal or uncertain until another state owner settles it |\n| connector.hermesagent.streamfailed | Hermes sent a recognized terminal failure event | Retain the failure and inspect prior model, tool, message, and delivery effects |\n\nstreampublication differs from those result codes. It means a parsed\nprovider frame could not be committed through AIP lifecycle storage. The\nprovider stream may have continued while the durable AIP sequence stopped.\n\nUse the original Action events and the last verified combined AIP cursor.\nProvider SSE IDs, event names, chunk IDs, and runid values are not resume\ncursors. Closing or losing a follow connection does not cancel the Action.\n\nFor chat and Responses streams, do not start a second model turn merely to\nrecover missing text. For structured run events, poll the retained provider\nrun while it exists. For operator recovery, use the durable binding and bound\nrun ID. Require a durable AIP result and provider terminal state where the\noperation defines one.\n\nResolve operator failures\n\nThe operator has durable result codes in addition to the base connector\nmapping:\n\n| Code | Meaning | Decision |\n\n| connector.hermesagent.operatorstartrejected | Hermes returned a definitive 4xx before accepting the start | Correct the request and create new reviewed work only after the original Action is terminal |\n| connector.hermesagent.operatorfailed | The bound provider run reported terminal failure | Retain output and error; inspect tool and child effects |\n| connector.hermesagent.operatoroutcomeunknown | Start response, run binding, or governed timeout left provider outcome unknown | Never replay POST /v1/runs; reconcile the retained binding |\n| connector.hermesagent.approvaloutcomeunknown | Provider may have applied an approval response before its result was stored | Do not resend the choice; inspect the same run and approval generation |\n| connector.hermesagent.cancellationoutcomeunknown | Stop did not reach a confirmed provider terminal state | Keep cancellation intent and poll or escalate the same run |\n\noperatorstate covers PostgreSQL read, compare-and-set, serialization, or\ncontention failure. operatorpolicy covers invalid policy, denied delegation,\nAction-ID collision, or missing binding. Correct the owning state or policy;\ndo not bypass it with a different Action ID.\n\nA bound runid is still provider-local and volatile. Hermes restart or\nretention can make it return 404 runnotfound. That does not prove the model\nor a tool performed no effect. An unbound start is deliberately finalized as\noutcomeunknown and is never replayed automatically.\n\nFollow the governed operator lifecycle (../guides/operator-lifecycle.md) for\nthe source Action, two approval loops, restart, and cancellation decisions.\n\nResolve delegated-result failures\n\nFirst-class delegation accepts only the exact durable child result. Use these\ncodes to identify the failed invariant:\n\n| Code | Category and retry | Decision |\n\n| connector.hermesagent.delegatedresultnotready | Temporary, true | Continue waiting on the same child within the original deadline |\n| connector.hermesagent.delegatedexecutionnotobserved | Policy, false | Hermes finished without creating the expected child; do not invent a result |\n| connector.hermesagent.delegatedactioncontractmismatch | Policy, false | Isolate the route; the child contract differs from the recorded request |\n| connector.hermesagent.delegatedidentityunverified | Auth, false | Restore transport-authenticated owner and action:write authority |\n| connector.hermesagent.delegatedidentitymismatch | Auth, false | Isolate the child created by the wrong Hermes principal |\n| connector.hermesagent.delegatedidentityintegrity | Auth, false | Preserve the Action and trusted identity context; fail closed |\n| connector.hermesagent.delegatedtenantmismatch | Auth, false | Isolate cross-tenant execution and preserve evidence |\n| connector.hermesagent.delegatedtenantexpired | Auth, false | Renew authority through policy; do not rewrite stored membership |\n| connector.hermesagent.delegatedresultintegrity | Connector, false | Preserve queue and lifecycle records that disagree or lack a terminal result |\n| connector.hermesagent.delegatedresultresolvermissing | Connector, false | Restore the host's authoritative queue and lifecycle resolver |\n\nThe retryable not-ready code means the same child has not settled. It never\nauthorizes creation of another child. Model prose is not a delegated result.\n\nDiagnose sessions, responses, runs, and jobs\n\nSeveral provider resources are local to one Hermes endpoint:\n\n| Symptom | State owner | Decision |\n\n| Stored response returns 404 | Endpoint response SQLite or in-memory fallback | Verify the owning endpoint and original Action; do not repeat generation |\n| Session route returns sessiondbunavailable through provider 503 | Endpoint state.db | Restore that database; mutation outcome still requires state inspection |\n| Session chat succeeded but messages are absent | Model result versus transcript persistence | Keep the result, inspect logs, and do not claim a durable append |\n| Session fork returns title-validation 400 | Source and child session rows | Inspect both rows because the provider fork is not atomic |\n| Structured run returns runnotfound | In-process run maps and retention | Preserve AIP evidence; do not recreate the run |\n| Job mutation times out or is cancelled | Profile-local scheduler store | Read the job before retrying; persistence may have won the race |\n| Job exists but no output was delivered | Scheduler claim, execution output, and delivery target | Separate stored schedule from run and delivery evidence |\n\nA successful host readiness probe covers only basic endpoint health. It does\nnot open state.db, prove response-store durability, inspect scheduler\nclaims, or retain structured runs.\n\nDecide whether to retry\n\nApply this order and stop at the first unmet condition:\n1. verify the signed response or read the durable Action result;\n2. confirm the exact endpoint-qualified capability and operation;\n3. determine whether the capability is read-only;\n4. for unsafe work, inspect its Action and provider-owned state; do not retry\n   when the outcome is uncertain or no terminal no-effect evidence exists;\n5. require connector implementation retry support and retryable: true;\n6. respect attempt, elapsed-time, deadline, and backoff budgets;\n7. let the runtime retry the same governed read rather than calling Hermes\n   directly.\n\nThe connector has no generic provider transaction ledger that proves\nnotcommitted for every Hermes model, tool, session, or job operation.\nAbsence of a provider handle is therefore not permission to repeat unsafe work.\n\nResolve startup failures\n\nStartup failures occur before a signed AIP error. Inspect the first process\nerror and compare it with the exact configuration. Common owners are:\n• empty, oversized, malformed, or unknown-field endpoint JSON;\n• invalid or duplicate normalized endpoint IDs;\n• non-HTTP(S) endpoint URLs or tenant mismatch;\n• missing, unsafe, empty, or oversized endpoint secret files;\n• invalid operator-policy fields, bounds, or allowlist relationships;\n• PostgreSQL connectivity, role, schema, or recovery failure;\n• a manifest above the 512-capability host ceiling;\n• host identity, digest, endpoint, revision, or registry admission mismatch.\n\nCorrect one owner at a time. After every change, require /ready, registry\nlease agreement, and one protected model read. Do not combine endpoint,\ncredential, database, artifact, and registry changes in one diagnostic restart.\n\nVerify the resolution\n\nClose the incident only when:\n• the signed error or startup failure no longer occurs on the corrected path;\n• /ready, the lifecycle lease, and tenant discovery agree;\n• a protected read reaches the intended endpoint and credential;\n• the original Action is terminal or remains explicitly paused as uncertain;\n• every stream, operator, approval, cancellation, and delegated child has a\n  justified state under its original identity;\n• session, response, run, and job evidence comes from the owning endpoint;\n• no endpoint key, raw sensitive payload, model credential, or personal data\n  entered the incident record.\n\nEscalate with pinned source revisions, artifact and manifest digests,\ntenant-safe identities, timestamps, signed code and category, retry and\nuncertainty flags, and remote status. Include Action and provider state,\nstream cursor, operator binding, child integrity evidence, readiness fields,\nmetric snapshots, and redacted diagnostics.\n\nRelated documentation\n• Errors and retry decisions (../../../reference/errors.md)\n• Hermes authentication and endpoints (../getting-started/authentication-and-endpoints.md)\n• Hermes configuration (../reference/configuration.md)\n• Recover the Hermes Agent connector (../operations/health-observability-and-recovery.md)\n• Hermes streaming cursors and cancellation (../reference/streaming-cursors-and-cancellation.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "fb6430e1258ff1f5826320b1fd9953dbc4115366af6d3e1da3d27ce5589a884f"
  }
}