Stream Hermes chat and Responses

Use this guide to submit one streaming Hermes action, render incremental output, and reach a verified AIP result. It covers chat_stream and responses_stream on an endpoint already admitted to the tenant catalogue.

Streaming can execute models, tools, memory writes, and message delivery. Partial text, a terminal-looking provider frame, and a final AIP result are different evidence. Never replay a stream merely because one delivery connection closed.

Understand the four views

One model turn appears in four nested views.

The outer follow stream can disconnect while the action and Hermes request continue. Conversely, an upstream Hermes stream can end while the AIP follow connection remains open long enough to deliver the final failure.

Use the AIP action ID as the recovery identity. Use provider IDs inside events to interpret Chat, Responses, tool, and continuity state. Do not substitute one for the other.

1. Choose Chat or Responses

Choose Responses with store: true when an application needs a provider record that can be reconciled after cancellation or upstream disconnect. Choose Chat when its message shape and explicit Hermes session continuity are the intended interface.

Both capabilities are stable, medium risk, retry-unsafe, and not subject to a connector retry. Both advertise synchronous, asynchronous, streaming, and cancellation support, but async_expected is false. The provider does not continue as a background delivery service after its HTTP stream is gone.

The published capability timeout is 300 seconds. The connector’s default HTTP client permits up to 900 seconds, but a trusted AIP deadline or explicit cancellation can stop the request earlier. Use the final AIP state as the outcome boundary; do not infer success from elapsed time.

2. Allocate stable AIP identity

Generate and retain the AIP action ID before submission. If the application uses the optional idempotency field, allocate one key for this exact request and retain it with the action ID.

An optional key does not make model or tool execution safe to retry. Provider streaming bypasses the in-memory idempotency cache used by non-streaming Chat and Responses calls.

Record at least:

• the endpoint-qualified capability ID;
• the AIP action ID and optional idempotency key;
• the tenant and authenticated principal used for submission;
• the chosen continuity fields;
• the durable follow cursor returned by aipd.

Do not generate a replacement action ID after an uncertain submission. Query the original ID first.

3. Prepare a bounded Chat stream

Use one Chat input branch and an explicit Hermes session ID:

{
  "messages": [
    {
      "role": "system",
      "content": "Return a concise operational summary."
    },
    {
      "role": "user",
      "content": "Summarize the current incident notes."
    }
  ],
  "model": "hermes-agent",
  "session_id": "incident-review-2030-01",
  "session_key": "support:incident-review",
  "max_events": 512
}

The connector forces stream to true. It removes continuity and trusted transport fields from provider JSON and sends them as headers.

Use an explicit session_id from the first turn. When it is absent, Hermes derives one and returns it only in an HTTP header. The current connector keeps provider SSE data but does not expose those response headers in its aggregate.

max_events limits parsed provider frames collected by the connector. It does not limit output tokens, tool calls, AIP follow pages, or model execution. The default is 4,096 frames. A low value is a safety cutoff, not normal completion.

4. Prepare a recoverable Responses stream

Keep the provider body separate from connector controls:

{
  "body": {
    "model": "hermes-agent",
    "input": "Summarize the current incident notes.",
    "instructions": "Return three concise findings.",
    "store": true,
    "truncation": "auto"
  },
  "session_key": "support:incident-review",
  "max_events": 512
}

The connector forces the provider body’s stream field to true. Put max_events, session_key, and the AIP idempotency value outside body.

With store: true, Hermes persists an in_progress snapshot immediately after response.created. It later replaces that snapshot with completed, failed, or incomplete provider state. Capture the response ID from the first event so response_get can reconcile the provider record.

For chaining, set previous_response_id, conversation, or explicit conversation_history according to the capability reference. Do not combine conversation and previous_response_id.

5. Submit and follow the same action

Place the chosen input in request.json, then start the stable action:

export AIP_CAPABILITY_ID='cap:hermes_agent:replace_endpoint:chat_stream'
export AIP_ACTION_ID='act_replace_with_stable_id_001'

cargo run --locked -p aipctl -- \
  --native-bearer-token-file "$AIP_TOKEN_FILE" \
  action call "$AIP_URL" \
  "$AIP_CAPABILITY_ID" \
  --action-id "$AIP_ACTION_ID" \
  --input @request.json

In a separate consumer, follow durable events and chunks for that same ID:

cargo run --locked -p aipctl -- \
  --native-bearer-token-file "$AIP_TOKEN_FILE" \
  action events "$AIP_URL" \
  "$AIP_ACTION_ID" \
  --include-chunks \
  --follow

An application can use the equivalent bearer-authenticated route:

GET /aip/v1/actions/{action_id}/events?include_chunks=true&follow=true

The follow loop polls the durable view every 500 milliseconds and emits an outer keepalive every 15 seconds. Hermes independently writes upstream keepalive comments after 30 seconds of inactivity. The connector ignores those comments; they do not consume max_events or create AIP chunks.

6. Decode each durable chunk

An outer aip.stream.chunk follow event contains an AIP chunk. Its structured payload has this shape:

{
  "action_id": "act_replace_with_stable_id_001",
  "sequence": 1,
  "kind": "data",
  "data": {
    "hermes_sse": {
      "event": null,
      "id": null,
      "data": {
        "choices": [
          {
            "delta": {
              "content": "The incident"
            },
            "finish_reason": null
          }
        ]
      },
      "raw_data": "{\"choices\":[{\"delta\":{\"content\":\"Hi\"}}]}",
      "terminal": false
    }
  }
}

The connector assigns StreamChunk.sequence from zero in provider-frame order. The embedded Responses sequence_number, when present, is a different provider sequence. Never compare or persist one as the other.

The runtime persists a chunk before publication returns. An identical repeat of the same AIP sequence is accepted as replay. Conflicting content, non-increasing new sequences, and chunks after an AIP terminal chunk are rejected.

Use data.hermes_sse.data as the provider event. Keep raw_data for controlled diagnostics only; it can contain prompts, output, tool arguments, tool results, and personal data.

7. Render Chat events

Handle Chat frames in this order.

Chat content deltas contribute to the final aggregate text, but the current connector does not copy them into StreamChunk.part. A renderer that reads only part will display no Chat text. Read the embedded choices delta.

hermes.tool.progress carries tool name, correlation ID, and running or completed state. Its provider event name does not match the connector’s generic tool.started, tool.completed, or tool.failed matcher. The AIP chunk is therefore normally kind: data, not kind: tool.

Never discard kind: data chunks when tools matter. De-duplicate tool updates by their provider correlation ID and AIP sequence.

8. Validate the Chat finish

The final Chat data chunk can carry one of three finish_reason values.

The provider writes [DONE] after all three. The connector treats [DONE] as an explicit terminal frame, but its final outcome classifier does not inspect Chat finish_reason. An ActionResult can therefore have status completed while the retained finish chunk says length or error.

Accept a Chat stream only when all of these are true:

1. the final AIP result is completed;
2. aggregate terminal is true and truncated is false;
3. a retained finish chunk has finish_reason: stop;
4. no retained provider error contradicts that finish;
5. any required tool effects have application-level confirmation.

Treat missing finish evidence as an incomplete outcome. Do not infer success from accumulated text.

9. Render Responses events

Responses uses named provider events with its own sequence_number.

Responses text deltas appear in both embedded data.delta and, when non-empty, StreamChunk.part. Prefer the structured provider identifiers so multiple output items cannot be merged accidentally.

Function calls and their outputs use response.output_item.* names. These events are retained as AIP data chunks rather than generic tool chunks. Pair them with call_id, item.id, and output_index from the provider payload.

The connector recognizes response.completed as Done and response.failed as Error. It also understands an upstream response.incomplete name, although the pinned Responses writer normally persists incomplete state on disconnect rather than delivering that terminal event over the broken stream.

10. Read the final AIP result

The connector returns this aggregate inside ActionResult.output:

{
  "endpoint_id": "replace_endpoint",
  "operation": "responses_stream",
  "http_status": 200,
  "events": [],
  "text": "Three findings are ready for review.",
  "terminal": true,
  "truncated": false
}

The actual events array contains every parsed provider frame, including its raw data. text is best-effort convenience output. It is not a substitute for the terminal provider response, finish reason, tool state, or final AIP status.

For Chat, apply the additional finish_reason rule from the preceding step. For Responses, require the terminal response object’s status to agree with the AIP result.

11. Choose the event limit safely

Omitting max_events uses the connector default of 4,096 parsed frames. The final aggregate retains all collected events, so choose a lower application bound only when it still leaves substantial headroom for text and tool frames.

The connector publishes and stores the frame that reaches the limit, then sets truncated true and terminal false. It checks the limit before accepting terminal completion. A terminal provider frame that lands exactly at the configured limit can therefore be published as Done while the final AIP result is Failed for truncation.

The final ActionResult wins over a conflicting last chunk. Do not retry with a larger limit after model or tool work may have occurred. Use the original action evidence to reconcile the effect.

An upstream close without an explicit terminal produces a Failed result. With runtime publication active, the connector also appends an AIP Error chunk with reason upstream_stream_closed_without_terminal_event. The truncation branch does not append that extra terminal Error chunk, so always read the result.

12. Recover a follow disconnect

Closing or losing the native aipd follow SSE does not cancel the action or the upstream Hermes request.

On reconnect:

1. keep the original action ID;
2. resume the same action route with its last retained route cursor;
3. tolerate a repeated outer event or chunk;
4. order chunks by AIP sequence and reject conflicting content;
5. read the final action status or result after the follow stream terminates.

Persist only a cursor returned by the same route and authorization context. The exact cursor formats, paging bounds, and expiry decisions belong to the streaming cursor and cancellation reference later in this documentation set.

Do not resume a provider SSE URL directly. Chat and Responses frames do not provide a usable replay cursor in this connector path.

13. Cancel the original action

Closing the renderer is not cancellation. Request cancellation explicitly:

cargo run --locked -p aipctl -- \
  --native-bearer-token-file "$AIP_TOKEN_FILE" \
  action cancel "$AIP_URL" \
  "$AIP_ACTION_ID" \
  --reason 'The requester no longer needs this stream'

The runtime records cancellation and wakes the action’s trusted cancellation token. The connector drops the active HTTP stream. When the Hermes writer observes the disconnect, it attempts agent.interrupt() and cancels its task.

This is cooperative stop, not effect rollback. A model request, tool call, message, memory write, or stored response may already exist. Read the original AIP result and provider state before deciding that another action is safe.

For responses_stream with store: true, cancellation can leave a retrievable incomplete response snapshot. For Chat, preserve explicit session identity and inspect the session according to the session capability reference; do not assume partial transcript persistence.

14. Reconcile a stored Responses stream

As soon as response.created arrives, persist its response.id. After an uncertain terminal outcome, call response_get once with that ID:

{
  "response_id": "resp_0123456789abcdef0123456789ab"
}

Interpret the stored status:

• completed supplies a terminal provider object for review;
• failed supplies retained failure state;
• incomplete preserves partial output after disconnect or cancellation;
• 404 means the response was not stored, was evicted or deleted, or the request reached another response store.

Do not submit a replacement stream merely because response_get returns incomplete. Tool or delivery effects can precede that snapshot.

15. Handle failures

Match protocol error codes, not free-form provider messages. Streaming calls are retry-unsafe even when the visible text is empty.

16. Verify the client

Before accepting the integration, use controlled evidence to verify that the client:

1. retains the action ID before submission;
2. follows the same ID with chunks included;
3. orders and de-duplicates by AIP chunk sequence;
4. renders Chat from choices rather than relying on part;
5. handles tool events carried as kind: data;
6. preserves Responses item and call identifiers;
7. saves response.id from response.created when storage is enabled;
8. requires explicit provider terminal evidence and final AIP state;
9. rejects Chat length and error finishes as complete success;
10. handles a terminal frame exactly on max_events as truncation;
11. resumes follow delivery without creating a new action;
12. sends explicit cancellation and then reconciles effects;
13. redacts raw frames, prompts, tools, outputs, IDs, and error data.

This is a source-aligned client checklist. It is not live model, latency, availability, cancellation, or production qualification evidence.

Related documentation

• Chat and Responses capabilities
• Hermes capability index
• Persistent sessions
• Use native AIP
• Transport bindings
• Errors and retry decisions