CrewAI idempotency, cancellation, and process fencing

This reference explains how the CrewAI connector fences duplicate work, persists job identity, and reports cancellation. Use it to recover an Action without treating a transport result as proof of provider behavior.

The reviewed implementation provides durable admission, not provider exactly-once execution. A terminal sidecar record can coexist with uncertain external effects when CrewAI work continues outside its asyncio task.

Keep three fences distinct

Preserve all three identities in evidence. Replacing an Action ID to bypass a failed record creates new provider authority and can duplicate an earlier effect.

Understand the durable request identity

Every startable operation requires an Idempotency-Key. The sidecar rejects a missing or blank key, a value over 512 UTF-8 bytes, or a code point below U+0020.

The sidecar calculates SHA-256 over canonical JSON containing:

• crew_id;
• action_id;
• operation;
• the complete operation input object;
• timeout_ms;
• the idempotency key.

The Action ID is the journal key across every admitted crew. An exact repeat returns the existing record. Reuse with a different request hash returns HTTP 409 and does not schedule new work.

The runtime inserts the record and persists the journal before creating the asyncio task. Persistence failure removes the in-memory record and returns HTTP 507, so provider work has not been scheduled through this path.

After successful persistence, later failures can leave the provider outcome unknown. The fence keeps the Action record available for status and event reconciliation.

Preserve the journal fence

Outside explicit loopback development, the sidecar requires a durable state file. Its parent must be a non-symlink directory without group or world write permission.

The runtime opens a companion .lock file as a regular file, forces mode 0600, and acquires an exclusive nonblocking flock. Startup fails when another process owns the same journal.

Use one state volume for exactly one active sidecar process. Horizontal scaling requires independent connector instances and independent state volumes. This lock is local to the mounted filesystem and is not a distributed lease.

The journal must be a non-symlink regular file with mode 0600. Restore also checks its byte bound, retained-record bound, unique Action IDs, event count, and per-event encoded size.

Persistence writes a mode-0600 temporary file in the journal directory, flushes and fsyncs it, replaces the journal atomically, then fsyncs the directory. This protects one writer from partial replacement, subject to the filesystem’s durability contract.

Interpret restart fencing

A restored running record cannot resume its Python task. The sidecar changes it to failed and appends a terminal event with uncertain_outcome: true.

Do not submit a replacement Action immediately. First reconcile the original Action with provider logs, tool effects, generated artifacts, and downstream systems.

The journal stores the request hash, events, status, and output. It does not store the original input or idempotency key, so retained dispatch evidence is also required for complete reconciliation.

Read each cancellation path precisely

Cancellation behavior depends on when the runtime observes cancellation and which connector method is active.

The non-stream select can return dispatched: false while its HTTP future has already sent bytes. Treat this field as local branch reporting, not network or provider evidence.

The explicit AIP cancel capability has its own Action identity and policy decision. The private sidecar route receives the target crew and Action ID but does not receive that AIP cancel Action’s idempotency key.

Separate sidecar cancellation from provider stop

The sidecar cancellation handler closes an attached streaming output when possible, cancels the owning asyncio task, waits for that task, and persists the record. The task records cancelled when it receives asyncio.CancelledError.

The connector advertises runtime cancellation for run, batch_run, replay, train, and test. The sidecar route itself accepts any running job, including operations without advertised cancellation support.

remote_stop_confirmed: true means only that the private cancel route returned success. It is not a provider receipt, rollback record, thread-termination proof, or confirmation that external tools stopped.

Interpret timeout as an uncertain boundary

The sidecar wraps operation execution in asyncio.wait_for using the request timeout. The blocking HTTP route separately waits for the same task with the same duration and returns HTTP 504 when its wait expires.

For thread-backed operations, cancelling the asyncio waiter does not terminate the Python worker thread. A timeout can therefore produce a failed or cancelled record while provider work continues.

An ordinary execution exception emits a generic terminal failed event with the exception class. It does not set uncertain_outcome; only restart recovery adds that field at this revision.

Interpret connector failures

The connector marks every reviewed ConnectorFailure as non-retryable. A caller must not convert a temporary category into an automatic retry without first reconciling the original Action.

The invocation-wide uncertainty rule is coarse. It can mark a read operation uncertain because reads and mutations share the same connector operation classification.

Choose a recovery action

Retain the original Action, request digest, journal record, event cursor, sidecar logs, provider request evidence, tool effects, and operator decision. Compensate only through a separately authorized Action.

Related documentation

• CrewAI connector overview
• CrewAI sidecar contract and lockfile
• CrewAI run lifecycle and replay
• CrewAI training and testing
• Errors and retry decisions