{
  "schemaVersion": "1.0",
  "title": "Train, test, and recover a CrewAI crew",
  "description": "Use this procedure to create one controlled CrewAI training artifact, evaluate the crew in a separate test Action, and retain enough evidence to accept or recover each outcome.",
  "canonical": "https://getaip.org/docs/connectors/crewai/guides/train-test-and-recover",
  "route": "/docs/connectors/crewai/guides/train-test-and-recover",
  "source": "docs/connectors/crewai/guides/train-test-and-recover.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/crewai/guides/train-test-and-recover.md",
    "txt": "/docs/download/connectors/crewai/guides/train-test-and-recover.txt",
    "json": "/docs/download/connectors/crewai/guides/train-test-and-recover.json",
    "pdf": "/docs/download/connectors/crewai/guides/train-test-and-recover.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Train, test, and recover a CrewAI crew\ndescription: >-\n  Run controlled training and evaluation, retain artifact and evaluator\n  evidence, and recover uncertain work without duplicate effects\nkind: how-to\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: crewai\ncapabilityIds:\n  - cap:crewai:<crew_id>:train\n  - cap:crewai:<crew_id>:test\n  - cap:crewai:<crew_id>:status\n  - cap:crewai:<crew_id>:events\n---\n\n# Train, test, and recover a CrewAI crew\n\nUse this procedure to create one controlled CrewAI training artifact, evaluate\nthe crew in a separate test Action, and retain enough evidence to accept or\nrecover each outcome.\n\nTraining and testing can invoke models, tools, and crew tasks many times. They\nare retry-unsafe provider operations even though their durable job events can\nbe streamed.\n\nThe connector advertises cancellation for both. Their synchronous CrewAI\nmethods remain non-cancellable once a worker thread starts.\n\n## Before you begin\n\nRequire:\n\n- a discovered crew that explicitly admits `train` and `test`;\n- exact Rust-host, sidecar, registry, CrewAI, crew-code, model, and tool\n  identities;\n- a protected durable training directory owned by one sidecar;\n- a reviewed training input set and independent evaluation input set;\n- approved model, evaluator, tool, network, storage, and telemetry credentials;\n- token, request, tool, rate, duration, and iteration budgets;\n- an evidence store separate from ordinary application logs;\n- authority to quarantine, preserve, promote, or remove generated artifacts.\n\nDo not enable `train` merely to repair a readiness failure. Review the high-risk\ncapability and its storage effects first.\n\n## 1. Define the acceptance boundary\n\nWrite the training and evaluation claim before dispatch. Include:\n\n| Identity | Required record |\n|---|---|\n| AIP | Source revision, connector profile, host artifact, tenant, instance, and replica |\n| Sidecar | Package, image digest, lockfile identity, state volume, and training directory |\n| CrewAI | Default package version, method-verification source commit, and any source-overlay identity |\n| Crew | Descriptor ID, registry revision, agents, tasks, process, models, and tools |\n| Data | Training and evaluation dataset digests, access policy, and redaction class |\n| Budget | Iterations, time, tokens, provider calls, tool calls, and monetary ceiling |\n| Result | Artifact digest, evaluation evidence, pass rule, reviewer, and timestamp |\n\nKeep training and evaluation datasets distinguishable. Reusing the same\nexamples can make a test look successful without measuring the intended\ngeneralization.\n\n## 2. Stage the training directory\n\nConfigure `AIP_CREWAI_TRAINING_DIR` before sidecar startup. The directory must\nbe a non-symlink and must not be group- or world-writable.\n\nUse a connector-owned durable volume. Give exactly one active sidecar process\nwrite authority to that volume and its journal.\n\nInventory existing artifact basenames before selecting a new one. The\nconnector validates a safe basename but does not prevent an intentional name\nfrom referring to an existing file.\n\nKeep CrewAI's durable home on the same controlled state boundary. The default\nproduct image places `HOME` under the sidecar volume for its local credential-\nencryption key.\n\n## 3. Freeze the training Action\n\nAllocate one Action ID, key, and unique artifact name:\n\n```text\nAction ID:       act_crewai_train_support_017\nIdempotency key: crewai-train-support-017-v1\nArtifact name:  support-training-017.json\n```\n\nBuild the input:\n\n```json\n{\n  \"n_iterations\": 4,\n  \"artifact_name\": \"support-training-017.json\",\n  \"inputs\": {\n    \"dataset_revision\": \"support-cases-2026-07-27\",\n    \"policy_revision\": \"support-policy-12\"\n  }\n}\n```\n\nIterations must be between 1 and 100. The artifact name permits 1 through 128\nletters, digits, dots, underscores, or hyphens and must begin with a letter or\ndigit.\n\nThe input object is passed to every pinned CrewAI training kickoff. Do not put\nsecret bytes or an untrusted filesystem path inside it.\n\n## 4. Approve training as a bounded mutation\n\nReview the immutable Action for:\n\n1. exact crew and training dataset;\n2. artifact name and overwrite disposition;\n3. iteration and budget ceilings;\n4. every model and tool authority reachable by crew tasks;\n5. human-input handling required by pinned CrewAI training;\n6. confidential-data and telemetry destinations;\n7. rollback limitations and uncertain-outcome procedure.\n\nApprove only that Action ID, key, and input hash. A prior test or ordinary run\ndoes not authorize training.\n\nThe connector persists its idempotency fence before scheduling CrewAI. A\njournal write failure therefore prevents provider admission; later failure can\nleave effects uncertain.\n\n## 5. Observe the training job\n\nFollow durable AIP chunks for the training Action. The sidecar normally emits\n`started` and a terminal event because the pinned training method runs in a\nworker thread with CrewAI streaming disabled.\n\nUse separate status and event read Actions targeting the training Action ID:\n\n```json\n{\n  \"run_action_id\": \"act_crewai_train_support_017\"\n}\n```\n\nDo not interpret the absence of token-level chunks as a stalled job. Use\nstatus, provider evidence, timeout policy, and resource observations together.\n\nDo not request runtime cancellation as a normal control. It cannot stop the\nsynchronous CrewAI training thread.\n\nThe sidecar can still persist `cancelled`, and Rust can return\n`remote_stop_confirmed: true`. That confirms only the sidecar route and must be\ntreated as an uncertain provider outcome.\n\n## 6. Inspect and seal the artifact\n\nA completed train result contains the artifact basename and iteration count.\nIt does not contain the file, a digest, or evaluation evidence.\n\nAfter terminal completion:\n\n1. verify that the resolved file remains under the configured training root;\n2. record its owner, mode, size, modification time, and cryptographic digest;\n3. scan content for credentials, prompts, personal data, and unintended\n   provider output;\n4. copy the accepted bytes into versioned integrity-protected storage;\n5. associate the copy with every identity and approval recorded in step 1;\n6. leave the sidecar-owned source file untouched until retention policy allows\n   removal.\n\nFile presence alone does not prove successful training. Require the completed\nAction, terminal sidecar event, reviewed artifact, and source identities.\n\nQuarantine an artifact when the Action failed, timed out, restarted, or lost\nits terminal evidence. Do not use it as an evaluator or deployment input until\nreconciled.\n\n## 7. Freeze a separate test Action\n\nTesting has its own Action ID, key, approval, inputs, and evaluator:\n\n```text\nAction ID:       act_crewai_test_support_017\nIdempotency key: crewai-test-support-017-v1\nEvaluator:       openrouter/reviewed-evaluator\n```\n\nBuild the test input:\n\n```json\n{\n  \"n_iterations\": 6,\n  \"eval_llm\": \"openrouter/reviewed-evaluator\",\n  \"inputs\": {\n    \"dataset_revision\": \"support-evaluation-2026-07-27\",\n    \"candidate_artifact\": \"support-training-017.json\"\n  }\n}\n```\n\n`eval_llm` is a bounded CrewAI model selector, not an AIP credential handle.\nThe deployment controls its provider credential and actual resolution.\n\nConfirm that the crew factory intentionally loads the candidate artifact. The\nconnector passes the test input to CrewAI but does not interpret a\n`candidate_artifact` convention.\n\n## 8. Approve and observe testing\n\nReview:\n\n- evaluator model and provider authority;\n- evaluation dataset separation and expected coverage;\n- six complete crew executions in this example;\n- all models and tools reachable by the evaluated crew;\n- pass criteria and where CrewAI evaluator output will be retained;\n- timeout and uncertain-outcome handling.\n\nFollow the test Action and read status or events by its own ID. Do not target\nthe training Action when checking test completion.\n\nThe completed connector result contains status, iteration count, and evaluator\nstring. It does not return a structured scorecard.\n\n## 9. Retain evaluator evidence\n\nThe pinned CrewAI method prints its evaluation result through CrewAI's output\npath. Configure an approved capture mechanism before starting the test.\n\nRetain:\n\n1. raw evaluator output in access-controlled evidence storage;\n2. a redacted reviewer-facing summary;\n3. evaluator model and credential revision;\n4. training and evaluation data digests;\n5. crew, registry, source, lockfile, and artifact identities;\n6. iteration-level pass, failure, timeout, and tool-effect observations;\n7. the declared acceptance rule and independent reviewer decision.\n\nDo not infer a passing evaluation from the connector's `status: completed`.\nThat value means the pinned CrewAI method returned without raising.\n\n## 10. Classify each terminal state\n\n| State | Evidence decision |\n|---|---|\n| `completed` with matching terminal event | Inspect artifact or evaluator evidence before acceptance |\n| `failed` before any `started` event | Verify whether provider admission occurred before classifying no effect |\n| `failed` after `started` | Treat model, tool, file, and evaluator effects as uncertain |\n| `cancelled` or local waiter ended | Treat underlying synchronous CrewAI work as potentially continuing |\n| Restored `running` record becomes `failed` | Preserve `uncertain_outcome: true` and reconcile external state |\n| No durable job under the expected ID | Check instance and journal ownership before another submission |\n| Same Action repeats exactly | Observe the existing record; do not interpret it as a new run |\n| Same Action conflicts | Restore the original input and key; never bypass the fence |\n\nOrdinary provider and timeout failures do not carry an `uncertain_outcome`\nfield at this revision. Apply the table from execution evidence; only a\nrestart-restored running record sets that field explicitly.\n\nProvider or tool logs can contain prompts, inputs, URLs, outputs, and\ncredentials. Use Action ID correlation and retain only approved redacted data.\n\n## 11. Recover uncertain training\n\n1. stop new training assignment to the affected crew instance;\n2. preserve the Action, approval, journal, sidecar, registry, and volume\n   identities;\n3. read durable status and all events for the original Action ID;\n4. inspect the intended artifact without modifying or replacing it;\n5. reconcile model, tool, storage, and human-input evidence;\n6. classify the artifact as absent, complete, partial, corrupt, or unknown;\n7. quarantine any bytes whose production path cannot be proved;\n8. create a new artifact name and Action only after the earlier effect is\n   settled and policy authorizes a distinct attempt.\n\nNever overwrite an uncertain artifact to make the directory look clean.\n\n## 12. Recover uncertain testing\n\n1. preserve the original test Action ID, key, evaluator, and input digest;\n2. read durable status and events;\n3. reconcile evaluator-provider requests and every crew model or tool effect;\n4. determine whether each planned iteration started and produced evidence;\n5. mark the campaign inconclusive when the required output is incomplete;\n6. retain partial evidence without counting it as a passing result;\n7. authorize a new test Action only under a new evidence identity.\n\nA new test can be appropriate after reconciliation because evaluation is a new\nmeasurement. It is never an automatic retry of an unknown Action.\n\n## Completion checklist\n\n- Training and evaluation claims were defined before dispatch.\n- Source, artifact, crew, model, tool, data, and credential identities were\n  retained.\n- One active sidecar owned the journal and training directory.\n- Training used a unique reviewed artifact basename.\n- Train and test used separate Action IDs, keys, approvals, and inputs.\n- A completed training artifact was inspected, hashed, and sealed.\n- Evaluator output was retained separately from connector completion metadata.\n- Timeout, cancellation, and restart states were treated as uncertain.\n- No new mutation began before earlier effects were reconciled.\n\n## Related documentation\n\n- [CrewAI training and testing reference](../capabilities/training-and-testing.md)\n- [Run a CrewAI crew and stream events](run-a-crew-and-stream-events.md)\n- [CrewAI connector configuration](../reference/configuration.md)\n- [Approvals and policy](../../../concepts/approvals-and-policy.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n",
    "text": "Train, test, and recover a CrewAI crew\n\nUse this procedure to create one controlled CrewAI training artifact, evaluate\nthe crew in a separate test Action, and retain enough evidence to accept or\nrecover each outcome.\n\nTraining and testing can invoke models, tools, and crew tasks many times. They\nare retry-unsafe provider operations even though their durable job events can\nbe streamed.\n\nThe connector advertises cancellation for both. Their synchronous CrewAI\nmethods remain non-cancellable once a worker thread starts.\n\nBefore you begin\n\nRequire:\n• a discovered crew that explicitly admits train and test;\n• exact Rust-host, sidecar, registry, CrewAI, crew-code, model, and tool\n  identities;\n• a protected durable training directory owned by one sidecar;\n• a reviewed training input set and independent evaluation input set;\n• approved model, evaluator, tool, network, storage, and telemetry credentials;\n• token, request, tool, rate, duration, and iteration budgets;\n• an evidence store separate from ordinary application logs;\n• authority to quarantine, preserve, promote, or remove generated artifacts.\n\nDo not enable train merely to repair a readiness failure. Review the high-risk\ncapability and its storage effects first.\n1. Define the acceptance boundary\n\nWrite the training and evaluation claim before dispatch. Include:\n\n| Identity | Required record |\n\n| AIP | Source revision, connector profile, host artifact, tenant, instance, and replica |\n| Sidecar | Package, image digest, lockfile identity, state volume, and training directory |\n| CrewAI | Default package version, method-verification source commit, and any source-overlay identity |\n| Crew | Descriptor ID, registry revision, agents, tasks, process, models, and tools |\n| Data | Training and evaluation dataset digests, access policy, and redaction class |\n| Budget | Iterations, time, tokens, provider calls, tool calls, and monetary ceiling |\n| Result | Artifact digest, evaluation evidence, pass rule, reviewer, and timestamp |\n\nKeep training and evaluation datasets distinguishable. Reusing the same\nexamples can make a test look successful without measuring the intended\ngeneralization.\n2. Stage the training directory\n\nConfigure AIPCREWAITRAININGDIR before sidecar startup. The directory must\nbe a non-symlink and must not be group- or world-writable.\n\nUse a connector-owned durable volume. Give exactly one active sidecar process\nwrite authority to that volume and its journal.\n\nInventory existing artifact basenames before selecting a new one. The\nconnector validates a safe basename but does not prevent an intentional name\nfrom referring to an existing file.\n\nKeep CrewAI's durable home on the same controlled state boundary. The default\nproduct image places HOME under the sidecar volume for its local credential-\nencryption key.\n3. Freeze the training Action\n\nAllocate one Action ID, key, and unique artifact name:\n\nAction ID:       actcrewaitrainsupport017\nIdempotency key: crewai-train-support-017-v1\nArtifact name:  support-training-017.json\n\nBuild the input:\n\n{\n  \"niterations\": 4,\n  \"artifactname\": \"support-training-017.json\",\n  \"inputs\": {\n    \"datasetrevision\": \"support-cases-2026-07-27\",\n    \"policyrevision\": \"support-policy-12\"\n  }\n}\n\nIterations must be between 1 and 100. The artifact name permits 1 through 128\nletters, digits, dots, underscores, or hyphens and must begin with a letter or\ndigit.\n\nThe input object is passed to every pinned CrewAI training kickoff. Do not put\nsecret bytes or an untrusted filesystem path inside it.\n4. Approve training as a bounded mutation\n\nReview the immutable Action for:\n1. exact crew and training dataset;\n2. artifact name and overwrite disposition;\n3. iteration and budget ceilings;\n4. every model and tool authority reachable by crew tasks;\n5. human-input handling required by pinned CrewAI training;\n6. confidential-data and telemetry destinations;\n7. rollback limitations and uncertain-outcome procedure.\n\nApprove only that Action ID, key, and input hash. A prior test or ordinary run\ndoes not authorize training.\n\nThe connector persists its idempotency fence before scheduling CrewAI. A\njournal write failure therefore prevents provider admission; later failure can\nleave effects uncertain.\n5. Observe the training job\n\nFollow durable AIP chunks for the training Action. The sidecar normally emits\nstarted and a terminal event because the pinned training method runs in a\nworker thread with CrewAI streaming disabled.\n\nUse separate status and event read Actions targeting the training Action ID:\n\n{\n  \"runactionid\": \"actcrewaitrainsupport017\"\n}\n\nDo not interpret the absence of token-level chunks as a stalled job. Use\nstatus, provider evidence, timeout policy, and resource observations together.\n\nDo not request runtime cancellation as a normal control. It cannot stop the\nsynchronous CrewAI training thread.\n\nThe sidecar can still persist cancelled, and Rust can return\nremotestopconfirmed: true. That confirms only the sidecar route and must be\ntreated as an uncertain provider outcome.\n6. Inspect and seal the artifact\n\nA completed train result contains the artifact basename and iteration count.\nIt does not contain the file, a digest, or evaluation evidence.\n\nAfter terminal completion:\n1. verify that the resolved file remains under the configured training root;\n2. record its owner, mode, size, modification time, and cryptographic digest;\n3. scan content for credentials, prompts, personal data, and unintended\n   provider output;\n4. copy the accepted bytes into versioned integrity-protected storage;\n5. associate the copy with every identity and approval recorded in step 1;\n6. leave the sidecar-owned source file untouched until retention policy allows\n   removal.\n\nFile presence alone does not prove successful training. Require the completed\nAction, terminal sidecar event, reviewed artifact, and source identities.\n\nQuarantine an artifact when the Action failed, timed out, restarted, or lost\nits terminal evidence. Do not use it as an evaluator or deployment input until\nreconciled.\n7. Freeze a separate test Action\n\nTesting has its own Action ID, key, approval, inputs, and evaluator:\n\nAction ID:       actcrewaitestsupport017\nIdempotency key: crewai-test-support-017-v1\nEvaluator:       openrouter/reviewed-evaluator\n\nBuild the test input:\n\n{\n  \"niterations\": 6,\n  \"evalllm\": \"openrouter/reviewed-evaluator\",\n  \"inputs\": {\n    \"datasetrevision\": \"support-evaluation-2026-07-27\",\n    \"candidateartifact\": \"support-training-017.json\"\n  }\n}\n\nevalllm is a bounded CrewAI model selector, not an AIP credential handle.\nThe deployment controls its provider credential and actual resolution.\n\nConfirm that the crew factory intentionally loads the candidate artifact. The\nconnector passes the test input to CrewAI but does not interpret a\ncandidateartifact convention.\n8. Approve and observe testing\n\nReview:\n• evaluator model and provider authority;\n• evaluation dataset separation and expected coverage;\n• six complete crew executions in this example;\n• all models and tools reachable by the evaluated crew;\n• pass criteria and where CrewAI evaluator output will be retained;\n• timeout and uncertain-outcome handling.\n\nFollow the test Action and read status or events by its own ID. Do not target\nthe training Action when checking test completion.\n\nThe completed connector result contains status, iteration count, and evaluator\nstring. It does not return a structured scorecard.\n9. Retain evaluator evidence\n\nThe pinned CrewAI method prints its evaluation result through CrewAI's output\npath. Configure an approved capture mechanism before starting the test.\n\nRetain:\n1. raw evaluator output in access-controlled evidence storage;\n2. a redacted reviewer-facing summary;\n3. evaluator model and credential revision;\n4. training and evaluation data digests;\n5. crew, registry, source, lockfile, and artifact identities;\n6. iteration-level pass, failure, timeout, and tool-effect observations;\n7. the declared acceptance rule and independent reviewer decision.\n\nDo not infer a passing evaluation from the connector's status: completed.\nThat value means the pinned CrewAI method returned without raising.\n10. Classify each terminal state\n\n| State | Evidence decision |\n\n| completed with matching terminal event | Inspect artifact or evaluator evidence before acceptance |\n| failed before any started event | Verify whether provider admission occurred before classifying no effect |\n| failed after started | Treat model, tool, file, and evaluator effects as uncertain |\n| cancelled or local waiter ended | Treat underlying synchronous CrewAI work as potentially continuing |\n| Restored running record becomes failed | Preserve uncertainoutcome: true and reconcile external state |\n| No durable job under the expected ID | Check instance and journal ownership before another submission |\n| Same Action repeats exactly | Observe the existing record; do not interpret it as a new run |\n| Same Action conflicts | Restore the original input and key; never bypass the fence |\n\nOrdinary provider and timeout failures do not carry an uncertainoutcome\nfield at this revision. Apply the table from execution evidence; only a\nrestart-restored running record sets that field explicitly.\n\nProvider or tool logs can contain prompts, inputs, URLs, outputs, and\ncredentials. Use Action ID correlation and retain only approved redacted data.\n11. Recover uncertain training\n1. stop new training assignment to the affected crew instance;\n2. preserve the Action, approval, journal, sidecar, registry, and volume\n   identities;\n3. read durable status and all events for the original Action ID;\n4. inspect the intended artifact without modifying or replacing it;\n5. reconcile model, tool, storage, and human-input evidence;\n6. classify the artifact as absent, complete, partial, corrupt, or unknown;\n7. quarantine any bytes whose production path cannot be proved;\n8. create a new artifact name and Action only after the earlier effect is\n   settled and policy authorizes a distinct attempt.\n\nNever overwrite an uncertain artifact to make the directory look clean.\n12. Recover uncertain testing\n1. preserve the original test Action ID, key, evaluator, and input digest;\n2. read durable status and events;\n3. reconcile evaluator-provider requests and every crew model or tool effect;\n4. determine whether each planned iteration started and produced evidence;\n5. mark the campaign inconclusive when the required output is incomplete;\n6. retain partial evidence without counting it as a passing result;\n7. authorize a new test Action only under a new evidence identity.\n\nA new test can be appropriate after reconciliation because evaluation is a new\nmeasurement. It is never an automatic retry of an unknown Action.\n\nCompletion checklist\n• Training and evaluation claims were defined before dispatch.\n• Source, artifact, crew, model, tool, data, and credential identities were\n  retained.\n• One active sidecar owned the journal and training directory.\n• Training used a unique reviewed artifact basename.\n• Train and test used separate Action IDs, keys, approvals, and inputs.\n• A completed training artifact was inspected, hashed, and sealed.\n• Evaluator output was retained separately from connector completion metadata.\n• Timeout, cancellation, and restart states were treated as uncertain.\n• No new mutation began before earlier effects were reconciled.\n\nRelated documentation\n• CrewAI training and testing reference (../capabilities/training-and-testing.md)\n• Run a CrewAI crew and stream events (run-a-crew-and-stream-events.md)\n• CrewAI connector configuration (../reference/configuration.md)\n• Approvals and policy (../../../concepts/approvals-and-policy.md)\n• Errors and retry decisions (../../../reference/errors.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "5567dff5b4cc4b579e969afa47770bc61964a1c199232d9024a32003089f7124"
  }
}