{
  "schemaVersion": "1.0",
  "title": "CrewAI training and testing capabilities",
  "description": "The train and test operations run bounded CrewAI evaluation workflows through the durable sidecar job boundary. Both can invoke models, tools, and crew tasks repeatedly, so iteration count is part of approval and cost review.",
  "canonical": "https://getaip.org/docs/connectors/crewai/capabilities/training-and-testing",
  "route": "/docs/connectors/crewai/capabilities/training-and-testing",
  "source": "docs/connectors/crewai/capabilities/training-and-testing.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/crewai/capabilities/training-and-testing.md",
    "txt": "/docs/download/connectors/crewai/capabilities/training-and-testing.txt",
    "json": "/docs/download/connectors/crewai/capabilities/training-and-testing.json",
    "pdf": "/docs/download/connectors/crewai/capabilities/training-and-testing.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: CrewAI training and testing capabilities\ndescription: >-\n  Use train and test with bounded iterations, protected artifacts, explicit\n  evaluators, approval, and truthful recovery semantics\nkind: capability-reference\naudience: application-developer\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: crewai\ncapabilityIds:\n  - cap:crewai:<crew_id>:train\n  - cap:crewai:<crew_id>:test\n---\n\n# CrewAI training and testing capabilities\n\nThe `train` and `test` operations run bounded CrewAI evaluation workflows\nthrough the durable sidecar job boundary. Both can invoke models, tools, and\ncrew tasks repeatedly, so iteration count is part of approval and cost review.\n\nTraining writes an operator-owned artifact. Testing selects an evaluator model\nand returns completion metadata, not a connector-normalized score report.\n\n## Compare train and test\n\n| Property | `train` | `test` |\n|---|---|---|\n| Capability | `cap:crewai:<crew_id>:train` | `cap:crewai:<crew_id>:test` |\n| Kind | Workflow | Workflow |\n| Risk | High | Medium |\n| Provider mutation | Yes | Yes |\n| Approval | Required | Required |\n| Idempotency key | Required | Required |\n| Streaming contract | Yes | Yes |\n| Advertised cancellation | Yes | Yes |\n| Connector retry | No | No |\n| Required identity | Safe artifact name | Evaluator model string |\n\nBoth operations create one durable sidecar job under their AIP Action ID. They\nexecute the pinned synchronous CrewAI method in a worker thread.\n\n## Train a crew\n\nThe exact input has two required fields and one optional object:\n\n```json\n{\n  \"n_iterations\": 3,\n  \"artifact_name\": \"support-training-2026-07.json\",\n  \"inputs\": {\n    \"scenario_set\": \"reviewed-support-cases\"\n  }\n}\n```\n\n| Field | Constraint | Meaning |\n|---|---|---|\n| `n_iterations` | Integer from 1 through 100 | Number of CrewAI training iterations |\n| `artifact_name` | `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$` | Basename inside the configured training directory |\n| `inputs` | Optional object, at most 512 top-level properties | Inputs supplied to each training kickoff |\n\nUnknown fields are rejected. The complete encoded input also shares the 1 MiB\nsidecar limit.\n\nThe sidecar resolves the artifact as:\n\n```text\n<AIP_CREWAI_TRAINING_DIR>/<artifact_name>\n```\n\nThe basename pattern excludes path separators and traversal segments. It does\nnot create artifact versions, checksums, retention policy, or write-once\nsemantics.\n\n## Understand the reviewed training behavior\n\nThe sidecar calls:\n\n```text\ncrew.train(n_iterations, artifact_path, inputs)\n```\n\nThe operation catalogue records CrewAI source `bfa652a7`, whose package version\nis `1.15.2`, as its method-verification baseline. The default product image\ninstead installs the frozen `1.15.5` package set.\n\nThe reviewed source method copies the crew, marks its tasks for human input,\ndisables agent delegation in the training copy, and runs every iteration. It\nevaluates collected data per agent and writes to the selected artifact.\n\nThose internal steps describe the source-verification baseline. The sidecar\ncall signature is connector code, but a built `1.15.5` artifact needs its own\nqualification evidence before those internals are attributed to it.\n\nA completed sidecar result has this shape:\n\n```json\n{\n  \"status\": \"completed\",\n  \"artifact_name\": \"support-training-2026-07.json\",\n  \"n_iterations\": 3\n}\n```\n\nThe result confirms that the pinned method returned. It does not include\nartifact bytes, a digest, evaluation details, or proof that another process\ncannot later change the file.\n\n## Protect the training directory\n\n`AIP_CREWAI_TRAINING_DIR` is required when the sidecar enables `train`. The\npath is created when absent and must resolve to a non-symlink directory.\n\nAny group or world write bit causes startup to fail. The controlled deployment\nplaces the directory on the connector-owned durable sidecar volume.\n\nApply these controls outside the capability input:\n\n- dedicate the directory to one active sidecar instance;\n- assign owner-only write access to the sidecar user;\n- inventory an artifact name before approving a replacement;\n- copy accepted artifacts into versioned, integrity-protected storage;\n- retain source, crew, model, tool, input-set, iteration, and approval identity;\n- scan artifacts for confidential data before broader distribution.\n\nAn AIP caller selects only the safe basename. It cannot change the training\nroot.\n\n## Test a crew\n\nThe exact input is:\n\n```json\n{\n  \"n_iterations\": 5,\n  \"eval_llm\": \"openrouter/reviewed-evaluator\",\n  \"inputs\": {\n    \"scenario_set\": \"release-candidate-17\"\n  }\n}\n```\n\n| Field | Constraint | Meaning |\n|---|---|---|\n| `n_iterations` | Integer from 1 through 100 | Number of evaluated crew executions |\n| `eval_llm` | Non-empty string, at most 256 bytes, no control characters | Evaluator passed to CrewAI model construction |\n| `inputs` | Optional object, at most 512 top-level properties | Inputs supplied to evaluated kickoffs |\n\nUnknown fields are rejected. The connector does not map `eval_llm` to a\ncredential or allowlist. The deployment's CrewAI configuration and provider\nenvironment determine whether that string resolves and which secret it uses.\n\nAt the pinned revision, the sidecar calls:\n\n```text\ncrew.test(n_iterations, eval_llm, inputs)\n```\n\nCrewAI constructs the evaluator LLM, copies the crew, runs each iteration, and\nprints its evaluation result through CrewAI's own output path.\n\nThe connector returns only bounded completion metadata:\n\n```json\n{\n  \"status\": \"completed\",\n  \"n_iterations\": 5,\n  \"eval_llm\": \"openrouter/reviewed-evaluator\"\n}\n```\n\nDo not treat this object as a scorecard. Retain the approved evaluator output\nthrough a deployment-owned evidence pipeline when scores are release evidence.\n\n## Review approval and cost\n\nApproval uses the complete input snapshot. Review at least:\n\n| Decision | `train` | `test` |\n|---|---|---|\n| Crew identity and admitted artifact | Required | Required |\n| Iteration count | Required | Required |\n| Input dataset authority and sensitivity | Required | Required |\n| Models and tools invoked by crew tasks | Required | Required |\n| Provider budgets and rate limits | Required | Required |\n| Artifact target and overwrite policy | Required | Not applicable |\n| Evaluator identity and credential scope | Not applicable | Required |\n| Telemetry and evidence destinations | Required | Required |\n\nThe AIP risk label does not estimate token consumption. One approved Action can\nrun the crew up to 100 times and can trigger additional evaluator calls.\n\nUse separate Action IDs and keys for training and testing. A successful test\ndoes not authorize a later training mutation.\n\n## Interpret streaming\n\nBoth capabilities advertise streaming because their durable jobs publish\nordered `started` and terminal events. The connector can return those events\nthrough the streaming AIP path.\n\nThe pinned sidecar sets `crew.stream = false` for both methods. It does not\nproject per-token CrewAI chunks from their synchronous worker threads.\n\nStreaming transport therefore improves lifecycle observation. It does not turn\ntraining or testing into an interruptible provider call.\n\n## Handle cancellation and timeout honestly\n\nBoth operations advertise cancellation. The sidecar accepts the request,\ncancels the asyncio task, persists `cancelled`, and the Rust connector can\nreturn `remote_stop_confirmed: true`.\n\nThe synchronous provider method still runs in a worker thread. The reported\nconfirmation acknowledges the sidecar route; it does not establish that\ntraining, evaluation, tools, or provider calls stopped.\n\nThe configured operation timeout bounds how long the coordinator waits. It\ndoes not terminate the underlying thread or undo model, tool, or file effects.\n\nOn a provider exception or timeout, the sidecar records `failed` with an\nexception class and generic message. That ordinary terminal event has no\n`uncertain_outcome` field, so operators must classify effects from operation\nsemantics and external evidence.\n\nThe pinned sidecar logs the failure with `LOGGER.exception`. Provider exception\ntext and traceback can therefore reach restricted logs even though the durable\nevent omits them.\n\nAfter a sidecar restart, a restored `running` record becomes failed with an\nuncertain outcome. The sidecar does not silently re-run it.\n\n## Recover without duplicating work\n\n1. preserve the original Action ID, key, input hash, crew, and artifact or\n   evaluator identity;\n2. read durable status and events for that Action;\n3. inspect the artifact path for training without executing or overwriting it;\n4. inspect provider, tool, and evaluator records correlated to the Action;\n5. classify completion, failure, cancellation, or continuing uncertainty;\n6. accept, quarantine, or remove an artifact under operator policy;\n7. create a new Action only after reconciling every possible prior effect.\n\nDo not repeat an Action with a different input under the same ID. The durable\nrequest hash rejects that collision.\n\n## Diagnose capability failures\n\n| Symptom | Decision |\n|---|---|\n| `train` is absent from discovery | Review descriptor and sidecar allowlists before changing either |\n| Sidecar requires a training directory | Configure a protected durable path before enabling `train` |\n| Artifact name is rejected | Use a safe basename of at most 128 characters |\n| Artifact already exists | Apply deployment overwrite and version policy before approval |\n| Evaluator is rejected | Use a non-empty bounded model string without control characters |\n| Evaluator cannot authenticate | Repair deployment-owned provider credentials, not Action input |\n| Action times out | Treat model, tool, and artifact effects as uncertain |\n| Job reports `cancelled` | Treat the worker thread and external effects as potentially continuing |\n| Completion lacks scores | Capture CrewAI evaluator output through the approved evidence path |\n| Sidecar restarted during work | Use the restored failed record and reconcile before any new Action |\n\n## Related documentation\n\n- [CrewAI capability index](README.md)\n- [CrewAI connector configuration](../reference/configuration.md)\n- [Authentication and admitted crews](../getting-started/authentication-and-admitted-crews.md)\n- [Approvals and policy](../../../concepts/approvals-and-policy.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n",
    "text": "CrewAI training and testing capabilities\n\nThe train and test operations run bounded CrewAI evaluation workflows\nthrough the durable sidecar job boundary. Both can invoke models, tools, and\ncrew tasks repeatedly, so iteration count is part of approval and cost review.\n\nTraining writes an operator-owned artifact. Testing selects an evaluator model\nand returns completion metadata, not a connector-normalized score report.\n\nCompare train and test\n\n| Property | train | test |\n\n| Capability | cap:crewai::train | cap:crewai::test |\n| Kind | Workflow | Workflow |\n| Risk | High | Medium |\n| Provider mutation | Yes | Yes |\n| Approval | Required | Required |\n| Idempotency key | Required | Required |\n| Streaming contract | Yes | Yes |\n| Advertised cancellation | Yes | Yes |\n| Connector retry | No | No |\n| Required identity | Safe artifact name | Evaluator model string |\n\nBoth operations create one durable sidecar job under their AIP Action ID. They\nexecute the pinned synchronous CrewAI method in a worker thread.\n\nTrain a crew\n\nThe exact input has two required fields and one optional object:\n\n{\n  \"niterations\": 3,\n  \"artifactname\": \"support-training-2026-07.json\",\n  \"inputs\": {\n    \"scenarioset\": \"reviewed-support-cases\"\n  }\n}\n\n| Field | Constraint | Meaning |\n\n| niterations | Integer from 1 through 100 | Number of CrewAI training iterations |\n| artifactname | ^[A-Za-z0-9][A-Za-z0-9.-]{0,127}$ | Basename inside the configured training directory |\n| inputs | Optional object, at most 512 top-level properties | Inputs supplied to each training kickoff |\n\nUnknown fields are rejected. The complete encoded input also shares the 1 MiB\nsidecar limit.\n\nThe sidecar resolves the artifact as:\n\n/\n\nThe basename pattern excludes path separators and traversal segments. It does\nnot create artifact versions, checksums, retention policy, or write-once\nsemantics.\n\nUnderstand the reviewed training behavior\n\nThe sidecar calls:\n\ncrew.train(niterations, artifactpath, inputs)\n\nThe operation catalogue records CrewAI source bfa652a7, whose package version\nis 1.15.2, as its method-verification baseline. The default product image\ninstead installs the frozen 1.15.5 package set.\n\nThe reviewed source method copies the crew, marks its tasks for human input,\ndisables agent delegation in the training copy, and runs every iteration. It\nevaluates collected data per agent and writes to the selected artifact.\n\nThose internal steps describe the source-verification baseline. The sidecar\ncall signature is connector code, but a built 1.15.5 artifact needs its own\nqualification evidence before those internals are attributed to it.\n\nA completed sidecar result has this shape:\n\n{\n  \"status\": \"completed\",\n  \"artifactname\": \"support-training-2026-07.json\",\n  \"niterations\": 3\n}\n\nThe result confirms that the pinned method returned. It does not include\nartifact bytes, a digest, evaluation details, or proof that another process\ncannot later change the file.\n\nProtect the training directory\n\nAIPCREWAITRAININGDIR is required when the sidecar enables train. The\npath is created when absent and must resolve to a non-symlink directory.\n\nAny group or world write bit causes startup to fail. The controlled deployment\nplaces the directory on the connector-owned durable sidecar volume.\n\nApply these controls outside the capability input:\n• dedicate the directory to one active sidecar instance;\n• assign owner-only write access to the sidecar user;\n• inventory an artifact name before approving a replacement;\n• copy accepted artifacts into versioned, integrity-protected storage;\n• retain source, crew, model, tool, input-set, iteration, and approval identity;\n• scan artifacts for confidential data before broader distribution.\n\nAn AIP caller selects only the safe basename. It cannot change the training\nroot.\n\nTest a crew\n\nThe exact input is:\n\n{\n  \"niterations\": 5,\n  \"evalllm\": \"openrouter/reviewed-evaluator\",\n  \"inputs\": {\n    \"scenarioset\": \"release-candidate-17\"\n  }\n}\n\n| Field | Constraint | Meaning |\n\n| niterations | Integer from 1 through 100 | Number of evaluated crew executions |\n| evalllm | Non-empty string, at most 256 bytes, no control characters | Evaluator passed to CrewAI model construction |\n| inputs | Optional object, at most 512 top-level properties | Inputs supplied to evaluated kickoffs |\n\nUnknown fields are rejected. The connector does not map evalllm to a\ncredential or allowlist. The deployment's CrewAI configuration and provider\nenvironment determine whether that string resolves and which secret it uses.\n\nAt the pinned revision, the sidecar calls:\n\ncrew.test(niterations, evalllm, inputs)\n\nCrewAI constructs the evaluator LLM, copies the crew, runs each iteration, and\nprints its evaluation result through CrewAI's own output path.\n\nThe connector returns only bounded completion metadata:\n\n{\n  \"status\": \"completed\",\n  \"niterations\": 5,\n  \"evalllm\": \"openrouter/reviewed-evaluator\"\n}\n\nDo not treat this object as a scorecard. Retain the approved evaluator output\nthrough a deployment-owned evidence pipeline when scores are release evidence.\n\nReview approval and cost\n\nApproval uses the complete input snapshot. Review at least:\n\n| Decision | train | test |\n\n| Crew identity and admitted artifact | Required | Required |\n| Iteration count | Required | Required |\n| Input dataset authority and sensitivity | Required | Required |\n| Models and tools invoked by crew tasks | Required | Required |\n| Provider budgets and rate limits | Required | Required |\n| Artifact target and overwrite policy | Required | Not applicable |\n| Evaluator identity and credential scope | Not applicable | Required |\n| Telemetry and evidence destinations | Required | Required |\n\nThe AIP risk label does not estimate token consumption. One approved Action can\nrun the crew up to 100 times and can trigger additional evaluator calls.\n\nUse separate Action IDs and keys for training and testing. A successful test\ndoes not authorize a later training mutation.\n\nInterpret streaming\n\nBoth capabilities advertise streaming because their durable jobs publish\nordered started and terminal events. The connector can return those events\nthrough the streaming AIP path.\n\nThe pinned sidecar sets crew.stream = false for both methods. It does not\nproject per-token CrewAI chunks from their synchronous worker threads.\n\nStreaming transport therefore improves lifecycle observation. It does not turn\ntraining or testing into an interruptible provider call.\n\nHandle cancellation and timeout honestly\n\nBoth operations advertise cancellation. The sidecar accepts the request,\ncancels the asyncio task, persists cancelled, and the Rust connector can\nreturn remotestopconfirmed: true.\n\nThe synchronous provider method still runs in a worker thread. The reported\nconfirmation acknowledges the sidecar route; it does not establish that\ntraining, evaluation, tools, or provider calls stopped.\n\nThe configured operation timeout bounds how long the coordinator waits. It\ndoes not terminate the underlying thread or undo model, tool, or file effects.\n\nOn a provider exception or timeout, the sidecar records failed with an\nexception class and generic message. That ordinary terminal event has no\nuncertainoutcome field, so operators must classify effects from operation\nsemantics and external evidence.\n\nThe pinned sidecar logs the failure with LOGGER.exception. Provider exception\ntext and traceback can therefore reach restricted logs even though the durable\nevent omits them.\n\nAfter a sidecar restart, a restored running record becomes failed with an\nuncertain outcome. The sidecar does not silently re-run it.\n\nRecover without duplicating work\n1. preserve the original Action ID, key, input hash, crew, and artifact or\n   evaluator identity;\n2. read durable status and events for that Action;\n3. inspect the artifact path for training without executing or overwriting it;\n4. inspect provider, tool, and evaluator records correlated to the Action;\n5. classify completion, failure, cancellation, or continuing uncertainty;\n6. accept, quarantine, or remove an artifact under operator policy;\n7. create a new Action only after reconciling every possible prior effect.\n\nDo not repeat an Action with a different input under the same ID. The durable\nrequest hash rejects that collision.\n\nDiagnose capability failures\n\n| Symptom | Decision |\n\n| train is absent from discovery | Review descriptor and sidecar allowlists before changing either |\n| Sidecar requires a training directory | Configure a protected durable path before enabling train |\n| Artifact name is rejected | Use a safe basename of at most 128 characters |\n| Artifact already exists | Apply deployment overwrite and version policy before approval |\n| Evaluator is rejected | Use a non-empty bounded model string without control characters |\n| Evaluator cannot authenticate | Repair deployment-owned provider credentials, not Action input |\n| Action times out | Treat model, tool, and artifact effects as uncertain |\n| Job reports cancelled | Treat the worker thread and external effects as potentially continuing |\n| Completion lacks scores | Capture CrewAI evaluator output through the approved evidence path |\n| Sidecar restarted during work | Use the restored failed record and reconcile before any new Action |\n\nRelated documentation\n• CrewAI capability index (README.md)\n• CrewAI connector configuration (../reference/configuration.md)\n• Authentication and admitted crews (../getting-started/authentication-and-admitted-crews.md)\n• Approvals and policy (../../../concepts/approvals-and-policy.md)\n• Errors and retry decisions (../../../reference/errors.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "bf077fd0189607b3e9b8541ab5de4e56b8bec64604d39fbcad3ba216acd3dff2"
  }
}