{
  "schemaVersion": "1.0",
  "title": "CrewAI idempotency, cancellation, and process fencing",
  "description": "This reference explains how the CrewAI connector fences duplicate work, persists job identity, and reports cancellation. Use it to recover an Action without treating a transport result as proof of provider behavior.",
  "canonical": "https://getaip.org/docs/connectors/crewai/reference/idempotency-cancellation-and-process-fencing",
  "route": "/docs/connectors/crewai/reference/idempotency-cancellation-and-process-fencing",
  "source": "docs/connectors/crewai/reference/idempotency-cancellation-and-process-fencing.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/crewai/reference/idempotency-cancellation-and-process-fencing.md",
    "txt": "/docs/download/connectors/crewai/reference/idempotency-cancellation-and-process-fencing.txt",
    "json": "/docs/download/connectors/crewai/reference/idempotency-cancellation-and-process-fencing.json",
    "pdf": "/docs/download/connectors/crewai/reference/idempotency-cancellation-and-process-fencing.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: CrewAI idempotency, cancellation, and process fencing\ndescription: >-\n  Preserve durable CrewAI Action identity, interpret cancellation evidence, and\n  recover without duplicating provider effects\nkind: reference\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: crewai\naliases:\n  - CrewAI idempotency\n  - CrewAI cancellation\n  - CrewAI journal fencing\n---\n\n# CrewAI idempotency, cancellation, and process fencing\n\nThis reference explains how the CrewAI connector fences duplicate work,\npersists job identity, and reports cancellation. Use it to recover an Action\nwithout treating a transport result as proof of provider behavior.\n\nThe reviewed implementation provides durable admission, not provider\nexactly-once execution. A terminal sidecar record can coexist with uncertain\nexternal effects when CrewAI work continues outside its asyncio task.\n\n## Keep three fences distinct\n\n| Fence | Identity or resource | What it prevents | What it does not prove |\n|---|---|---|---|\n| AIP Action | Immutable Action ID, input, tenant, target, and AIP idempotency key | Re-admission of changed AIP work under one identity | That CrewAI produced no external effect |\n| Sidecar request | Action ID plus canonical request hash | Reusing one sidecar job ID with different material | Provider exactly-once behavior |\n| Journal owner | Companion file lock for one state path | Two active sidecar processes writing one journal | Distributed or active-active ownership |\n\nPreserve all three identities in evidence. Replacing an Action ID to bypass a\nfailed record creates new provider authority and can duplicate an earlier\neffect.\n\n## Understand the durable request identity\n\nEvery startable operation requires an `Idempotency-Key`. The sidecar rejects a\nmissing or blank key, a value over 512 UTF-8 bytes, or a code point below\nU+0020.\n\nThe sidecar calculates SHA-256 over canonical JSON containing:\n\n- `crew_id`;\n- `action_id`;\n- `operation`;\n- the complete operation `input` object;\n- `timeout_ms`;\n- the idempotency key.\n\nThe Action ID is the journal key across every admitted crew. An exact repeat\nreturns the existing record. Reuse with a different request hash returns HTTP\n`409` and does not schedule new work.\n\nThe runtime inserts the record and persists the journal before creating the\nasyncio task. Persistence failure removes the in-memory record and returns\nHTTP `507`, so provider work has not been scheduled through this path.\n\nAfter successful persistence, later failures can leave the provider outcome\nunknown. The fence keeps the Action record available for status and event\nreconciliation.\n\n## Preserve the journal fence\n\nOutside explicit loopback development, the sidecar requires a durable state\nfile. Its parent must be a non-symlink directory without group or world write\npermission.\n\nThe runtime opens a companion `.lock` file as a regular file, forces mode\n`0600`, and acquires an exclusive nonblocking `flock`. Startup fails when\nanother process owns the same journal.\n\nUse one state volume for exactly one active sidecar process. Horizontal scaling\nrequires independent connector instances and independent state volumes. This\nlock is local to the mounted filesystem and is not a distributed lease.\n\nThe journal must be a non-symlink regular file with mode `0600`. Restore also\nchecks its byte bound, retained-record bound, unique Action IDs, event count,\nand per-event encoded size.\n\nPersistence writes a mode-`0600` temporary file in the journal directory,\nflushes and `fsync`s it, replaces the journal atomically, then `fsync`s the\ndirectory. This protects one writer from partial replacement, subject to the\nfilesystem's durability contract.\n\n## Interpret restart fencing\n\nA restored `running` record cannot resume its Python task. The sidecar changes\nit to `failed` and appends a terminal event with `uncertain_outcome: true`.\n\nDo not submit a replacement Action immediately. First reconcile the original\nAction with provider logs, tool effects, generated artifacts, and downstream\nsystems.\n\nThe journal stores the request hash, events, status, and output. It does not\nstore the original input or idempotency key, so retained dispatch evidence is\nalso required for complete reconciliation.\n\n## Read each cancellation path precisely\n\nCancellation behavior depends on when the runtime observes cancellation and\nwhich connector method is active.\n\n| Path | Connector result | Sidecar request | Safe interpretation |\n|---|---|---|---|\n| Before a streaming HTTP response | `cancelled_unconfirmed` failure | None proved | Dispatch may have raced with local cancellation |\n| During an events-read stream | Cancelled, `remote_stop_confirmed: false`, `dispatched: false` | None | The reader detached; the target job was not cancelled |\n| During a streamed start | Cancelled, `remote_stop_confirmed: true` after sidecar success | `POST /jobs/{id}/cancel` | The sidecar accepted cancellation; provider stop remains unproved |\n| During a non-stream invocation | Cancelled, `remote_stop_confirmed: false`, `dispatched: false` | Not issued by this select branch | The dispatch flag is not reliable across the race |\n| Explicit cancel capability | Result from the cancel route | `POST /jobs/{target}/cancel` | Terminal jobs are unchanged; running sidecar tasks are cancelled |\n| Stream validation failure after admission | Original stream error when cancel succeeds | Attempted for five cancellable operations | Provider effects may already exist |\n| Stream validation failure with failed cancel | `cancelled_unconfirmed` failure | Attempt failed | Reconcile before any replacement Action |\n\nThe non-stream select can return `dispatched: false` while its HTTP future has\nalready sent bytes. Treat this field as local branch reporting, not network or\nprovider evidence.\n\nThe explicit AIP cancel capability has its own Action identity and policy\ndecision. The private sidecar route receives the target crew and Action ID but\ndoes not receive that AIP cancel Action's idempotency key.\n\n## Separate sidecar cancellation from provider stop\n\nThe sidecar cancellation handler closes an attached streaming output when\npossible, cancels the owning asyncio task, waits for that task, and persists\nthe record. The task records `cancelled` when it receives\n`asyncio.CancelledError`.\n\n| Operation | Execution mechanism | Provider-stop boundary |\n|---|---|---|\n| `run` | Native async kickoff and optional async stream | Closing the stream and cancelling the task does not prove rollback of model or tool effects |\n| `batch_run` | Native async methods when present; synchronous fallback in a worker thread | The fallback thread can continue after its waiter is cancelled |\n| `replay` | Synchronous CrewAI call in a worker thread | The thread can continue |\n| `train` | Synchronous CrewAI call in a worker thread | Training and artifact writes can continue |\n| `test` | Synchronous CrewAI call in a worker thread | Model and evaluator calls can continue |\n| `knowledge_query` | Async method or worker-thread fallback | Cancellation is not advertised for this operation |\n| `memory_reset` | Synchronous CrewAI call in a worker thread | Cancellation is not advertised for this operation |\n\nThe connector advertises runtime cancellation for `run`, `batch_run`,\n`replay`, `train`, and `test`. The sidecar route itself accepts any running\njob, including operations without advertised cancellation support.\n\n`remote_stop_confirmed: true` means only that the private cancel route returned\nsuccess. It is not a provider receipt, rollback record, thread-termination\nproof, or confirmation that external tools stopped.\n\n## Interpret timeout as an uncertain boundary\n\nThe sidecar wraps operation execution in `asyncio.wait_for` using the request\ntimeout. The blocking HTTP route separately waits for the same task with the\nsame duration and returns HTTP `504` when its wait expires.\n\nFor thread-backed operations, cancelling the asyncio waiter does not terminate\nthe Python worker thread. A timeout can therefore produce a failed or cancelled\nrecord while provider work continues.\n\nAn ordinary execution exception emits a generic terminal `failed` event with\nthe exception class. It does not set `uncertain_outcome`; only restart recovery\nadds that field at this revision.\n\n## Interpret connector failures\n\n| Condition | Category | `uncertain_outcome` | `retryable` |\n|---|---|---:|---:|\n| Sidecar `401` or `403` | Auth | False | False |\n| Sidecar `429` or `5xx` | Temporary | True only for connector invocation | False |\n| Other sidecar status | Permanent | False | False |\n| HTTP transport failure | Transport | True only for connector invocation | False |\n| Invalid or incomplete stream | Connector | True | False |\n| Response limit exceeded | Connector | True only for connector invocation | False |\n| Cancellation not confirmed | Temporary | True | False |\n| Invalid credential, configuration, or input | Auth or permanent | False | False |\n\nThe connector marks every reviewed `ConnectorFailure` as non-retryable. A\ncaller must not convert a temporary category into an automatic retry without\nfirst reconciling the original Action.\n\nThe invocation-wide uncertainty rule is coarse. It can mark a read operation\nuncertain because reads and mutations share the same connector operation\nclassification.\n\n## Choose a recovery action\n\n| Evidence | Recovery decision |\n|---|---|\n| Exact repeat returns completed | Use the retained result; do not dispatch again |\n| Exact repeat returns running | Continue status or event observation within policy |\n| Restored record reports uncertain failure | Reconcile external effects before retry or compensation |\n| Cancel route succeeded for an async stream | Confirm provider and tool state independently |\n| Cancel route succeeded for thread-backed work | Assume work may continue; contain credentials and downstream authority if necessary |\n| Request changed under the same Action ID | Correct the caller; the `409` collision is protective |\n| Journal is full | Archive only terminal records after the AIP replay-retention window |\n| Journal owner lock fails | Stop the new replica; never bypass the lock or share the writable state path |\n\nRetain the original Action, request digest, journal record, event cursor,\nsidecar logs, provider request evidence, tool effects, and operator decision.\nCompensate only through a separately authorized Action.\n\n## Related documentation\n\n- [CrewAI connector overview](../README.md)\n- [CrewAI sidecar contract and lockfile](sidecar-contract-and-lockfile.md)\n- [CrewAI run lifecycle and replay](../capabilities/run-lifecycle-and-replay.md)\n- [CrewAI training and testing](../capabilities/training-and-testing.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n",
    "text": "CrewAI idempotency, cancellation, and process fencing\n\nThis reference explains how the CrewAI connector fences duplicate work,\npersists job identity, and reports cancellation. Use it to recover an Action\nwithout treating a transport result as proof of provider behavior.\n\nThe reviewed implementation provides durable admission, not provider\nexactly-once execution. A terminal sidecar record can coexist with uncertain\nexternal effects when CrewAI work continues outside its asyncio task.\n\nKeep three fences distinct\n\n| Fence | Identity or resource | What it prevents | What it does not prove |\n\n| AIP Action | Immutable Action ID, input, tenant, target, and AIP idempotency key | Re-admission of changed AIP work under one identity | That CrewAI produced no external effect |\n| Sidecar request | Action ID plus canonical request hash | Reusing one sidecar job ID with different material | Provider exactly-once behavior |\n| Journal owner | Companion file lock for one state path | Two active sidecar processes writing one journal | Distributed or active-active ownership |\n\nPreserve all three identities in evidence. Replacing an Action ID to bypass a\nfailed record creates new provider authority and can duplicate an earlier\neffect.\n\nUnderstand the durable request identity\n\nEvery startable operation requires an Idempotency-Key. The sidecar rejects a\nmissing or blank key, a value over 512 UTF-8 bytes, or a code point below\nU+0020.\n\nThe sidecar calculates SHA-256 over canonical JSON containing:\n• crewid;\n• actionid;\n• operation;\n• the complete operation input object;\n• timeoutms;\n• the idempotency key.\n\nThe Action ID is the journal key across every admitted crew. An exact repeat\nreturns the existing record. Reuse with a different request hash returns HTTP\n409 and does not schedule new work.\n\nThe runtime inserts the record and persists the journal before creating the\nasyncio task. Persistence failure removes the in-memory record and returns\nHTTP 507, so provider work has not been scheduled through this path.\n\nAfter successful persistence, later failures can leave the provider outcome\nunknown. The fence keeps the Action record available for status and event\nreconciliation.\n\nPreserve the journal fence\n\nOutside explicit loopback development, the sidecar requires a durable state\nfile. Its parent must be a non-symlink directory without group or world write\npermission.\n\nThe runtime opens a companion .lock file as a regular file, forces mode\n0600, and acquires an exclusive nonblocking flock. Startup fails when\nanother process owns the same journal.\n\nUse one state volume for exactly one active sidecar process. Horizontal scaling\nrequires independent connector instances and independent state volumes. This\nlock is local to the mounted filesystem and is not a distributed lease.\n\nThe journal must be a non-symlink regular file with mode 0600. Restore also\nchecks its byte bound, retained-record bound, unique Action IDs, event count,\nand per-event encoded size.\n\nPersistence writes a mode-0600 temporary file in the journal directory,\nflushes and fsyncs it, replaces the journal atomically, then fsyncs the\ndirectory. This protects one writer from partial replacement, subject to the\nfilesystem's durability contract.\n\nInterpret restart fencing\n\nA restored running record cannot resume its Python task. The sidecar changes\nit to failed and appends a terminal event with uncertainoutcome: true.\n\nDo not submit a replacement Action immediately. First reconcile the original\nAction with provider logs, tool effects, generated artifacts, and downstream\nsystems.\n\nThe journal stores the request hash, events, status, and output. It does not\nstore the original input or idempotency key, so retained dispatch evidence is\nalso required for complete reconciliation.\n\nRead each cancellation path precisely\n\nCancellation behavior depends on when the runtime observes cancellation and\nwhich connector method is active.\n\n| Path | Connector result | Sidecar request | Safe interpretation |\n\n| Before a streaming HTTP response | cancelledunconfirmed failure | None proved | Dispatch may have raced with local cancellation |\n| During an events-read stream | Cancelled, remotestopconfirmed: false, dispatched: false | None | The reader detached; the target job was not cancelled |\n| During a streamed start | Cancelled, remotestopconfirmed: true after sidecar success | POST /jobs/{id}/cancel | The sidecar accepted cancellation; provider stop remains unproved |\n| During a non-stream invocation | Cancelled, remotestopconfirmed: false, dispatched: false | Not issued by this select branch | The dispatch flag is not reliable across the race |\n| Explicit cancel capability | Result from the cancel route | POST /jobs/{target}/cancel | Terminal jobs are unchanged; running sidecar tasks are cancelled |\n| Stream validation failure after admission | Original stream error when cancel succeeds | Attempted for five cancellable operations | Provider effects may already exist |\n| Stream validation failure with failed cancel | cancelledunconfirmed failure | Attempt failed | Reconcile before any replacement Action |\n\nThe non-stream select can return dispatched: false while its HTTP future has\nalready sent bytes. Treat this field as local branch reporting, not network or\nprovider evidence.\n\nThe explicit AIP cancel capability has its own Action identity and policy\ndecision. The private sidecar route receives the target crew and Action ID but\ndoes not receive that AIP cancel Action's idempotency key.\n\nSeparate sidecar cancellation from provider stop\n\nThe sidecar cancellation handler closes an attached streaming output when\npossible, cancels the owning asyncio task, waits for that task, and persists\nthe record. The task records cancelled when it receives\nasyncio.CancelledError.\n\n| Operation | Execution mechanism | Provider-stop boundary |\n\n| run | Native async kickoff and optional async stream | Closing the stream and cancelling the task does not prove rollback of model or tool effects |\n| batchrun | Native async methods when present; synchronous fallback in a worker thread | The fallback thread can continue after its waiter is cancelled |\n| replay | Synchronous CrewAI call in a worker thread | The thread can continue |\n| train | Synchronous CrewAI call in a worker thread | Training and artifact writes can continue |\n| test | Synchronous CrewAI call in a worker thread | Model and evaluator calls can continue |\n| knowledgequery | Async method or worker-thread fallback | Cancellation is not advertised for this operation |\n| memoryreset | Synchronous CrewAI call in a worker thread | Cancellation is not advertised for this operation |\n\nThe connector advertises runtime cancellation for run, batchrun,\nreplay, train, and test. The sidecar route itself accepts any running\njob, including operations without advertised cancellation support.\n\nremotestopconfirmed: true means only that the private cancel route returned\nsuccess. It is not a provider receipt, rollback record, thread-termination\nproof, or confirmation that external tools stopped.\n\nInterpret timeout as an uncertain boundary\n\nThe sidecar wraps operation execution in asyncio.waitfor using the request\ntimeout. The blocking HTTP route separately waits for the same task with the\nsame duration and returns HTTP 504 when its wait expires.\n\nFor thread-backed operations, cancelling the asyncio waiter does not terminate\nthe Python worker thread. A timeout can therefore produce a failed or cancelled\nrecord while provider work continues.\n\nAn ordinary execution exception emits a generic terminal failed event with\nthe exception class. It does not set uncertainoutcome; only restart recovery\nadds that field at this revision.\n\nInterpret connector failures\n\n| Condition | Category | uncertainoutcome | retryable |\n\n| Sidecar 401 or 403 | Auth | False | False |\n| Sidecar 429 or 5xx | Temporary | True only for connector invocation | False |\n| Other sidecar status | Permanent | False | False |\n| HTTP transport failure | Transport | True only for connector invocation | False |\n| Invalid or incomplete stream | Connector | True | False |\n| Response limit exceeded | Connector | True only for connector invocation | False |\n| Cancellation not confirmed | Temporary | True | False |\n| Invalid credential, configuration, or input | Auth or permanent | False | False |\n\nThe connector marks every reviewed ConnectorFailure as non-retryable. A\ncaller must not convert a temporary category into an automatic retry without\nfirst reconciling the original Action.\n\nThe invocation-wide uncertainty rule is coarse. It can mark a read operation\nuncertain because reads and mutations share the same connector operation\nclassification.\n\nChoose a recovery action\n\n| Evidence | Recovery decision |\n\n| Exact repeat returns completed | Use the retained result; do not dispatch again |\n| Exact repeat returns running | Continue status or event observation within policy |\n| Restored record reports uncertain failure | Reconcile external effects before retry or compensation |\n| Cancel route succeeded for an async stream | Confirm provider and tool state independently |\n| Cancel route succeeded for thread-backed work | Assume work may continue; contain credentials and downstream authority if necessary |\n| Request changed under the same Action ID | Correct the caller; the 409 collision is protective |\n| Journal is full | Archive only terminal records after the AIP replay-retention window |\n| Journal owner lock fails | Stop the new replica; never bypass the lock or share the writable state path |\n\nRetain the original Action, request digest, journal record, event cursor,\nsidecar logs, provider request evidence, tool effects, and operator decision.\nCompensate only through a separately authorized Action.\n\nRelated documentation\n• CrewAI connector overview (../README.md)\n• CrewAI sidecar contract and lockfile (sidecar-contract-and-lockfile.md)\n• CrewAI run lifecycle and replay (../capabilities/run-lifecycle-and-replay.md)\n• CrewAI training and testing (../capabilities/training-and-testing.md)\n• Errors and retry decisions (../../../reference/errors.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "d3b05114eb667d4046c727031d610ea17a69241ec87712034d8304f75058628d"
  }
}