{
  "schemaVersion": "1.0",
  "title": "Testing and evidence index",
  "description": "Use this index to choose the smallest procedure that can support your claim and to find the evidence you must retain. Start with the claim, not with whichever test command is easiest to run.",
  "canonical": "https://getaip.org/docs/testing",
  "route": "/docs/testing",
  "source": "docs/testing/README.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Qualification and Evidence",
  "documentType": "Documentation",
  "language": "en",
  "revision": {
    "lastReviewedRevision": "d7cce13d1d555644d04a4d73c66c95b113737635",
    "documentationSourceRevision": "9192fef3695ad294994f2712f6d156241e5e92fb",
    "basis": "frontmatter"
  },
  "downloads": {
    "md": "/docs/download/testing.md",
    "txt": "/docs/download/testing.txt",
    "json": "/docs/download/testing.json",
    "pdf": "/docs/download/testing.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Testing and evidence index\ndescription: Select the AIP validation or qualification procedure that matches the claim you need to make\nkind: index\naudience: evaluator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"d7cce13d1d555644d04a4d73c66c95b113737635\"\n---\n\n# Testing and evidence index\n\nUse this index to choose the smallest procedure that can support your claim and\nto find the evidence you must retain. Start with the claim, not with whichever\ntest command is easiest to run.\n\nThe pinned GetAIP `2.0.0` source contains tests, conformance suites,\nqualification scripts, and CI definitions. The retained release record shows\nthat the six protected contexts and their named controlled gates passed for\nrevision `d7cce13d1d555644d04a4d73c66c95b113737635`. It does not extend that\nresult to a rebuilt image, external provider, or target deployment.\n\n## Choose evidence by the claim\n\n| Question you need to answer | Start here | Evidence boundary |\n|---|---|---|\n| What does the pinned source implement? | [Implementation status](../reference/implementation-status.md) | Code presence is not an executed result |\n| Does one schema instance, envelope, manifest, connector, or MCP surface satisfy its contract? | [Conformance and qualification](../reference/conformance.md) | Run the complete applicable matrix and assert its expected checks |\n| Do registry, routing, host, lease, revocation, outage, and recovery paths work together? | [Connector fleet qualification](connector-fleet-qualification.md) | Controlled multiprocess evidence is not product-provider evidence |\n| Does the registry meet its declared fleet-size thresholds? | [Fleet scale SLO](fleet-scale-slo.md) | Results apply only to the recorded environment and dataset |\n| Which upstream product revision must a connector campaign use? | [Pinned upstream baselines](../connectors/upstream-baselines.md) | A source pin does not qualify a built provider deployment |\n| How should an external product campaign be prepared and retained? | [Live product E2E](live-product-e2e.md) | Each product, credential policy, topology, and image needs its own evidence |\n| What did the older Cal.diy campaign prove? | [Cal.diy isolated-live evidence](cal-diy-isolated-live.md) | The result covers only the artifacts named in that report |\n| What did the older Hermes Agent campaign prove? | [Hermes Agent isolated-live evidence](hermes-agent-isolated-live.md) | The result covers only the two named endpoints and their dependencies |\n| Was a release built, signed, attested, and verified? | [Release artifacts](../reference/release-artifacts.md) | Workflow source is not publication evidence |\n\nIf the claim spans several rows, retain evidence for every row. A product\ncampaign does not replace protocol conformance, and a release signature does\nnot replace deployment recovery or capacity testing.\n\n## Follow the evidence progression\n\nUse the layers in this order so a later failure does not hide a cheaper, more\nlocal defect:\n\n1. validate schemas and typed invariants;\n2. run deterministic unit, property, fixture, and recovery tests;\n3. run every applicable core, profile, and connector conformance suite;\n4. exercise process, storage, routing, restart, and failure boundaries;\n5. run isolated-live campaigns against pinned external products or peers;\n6. build and verify immutable release artifacts;\n7. qualify the target deployment's security, recovery, fault, and capacity\n   requirements.\n\nThe layers are cumulative only when they refer to the same source and build\nlineage. If an image is rebuilt after conformance, identify the new digest and\nrerun every affected layer.\n\n## Use the source-owned procedures deliberately\n\n| Procedure family | Source-owned entry point | What its source is designed to cover |\n|---|---|---|\n| Source release gate | `cargo run -p xtask -- release-check` | Format, boundaries, semantic version checks, schemas, lint, tests, dependencies, fuzz compilation, and API docs |\n| Core diagnostic | `getaip conformance run` | One built-in message body or one supplied native envelope |\n| MCP client diagnostic | `getaip mcp conformance` | Four external-client checks only |\n| Connector harness | `run_connector_conformance` | Atomic manifest admission and twelve conditional behavior families |\n| PostgreSQL recovery | `aip-storage-postgres` recovery test | Durable runtime recovery behavior |\n| Registry and control plane | PostgreSQL registry and control-plane tests | Schema, roles, routing, lifecycle, concurrency, and reconnect behavior |\n| Fleet failure matrix | `verify-failure-matrix.sh` | Multiprocess trust, routing, failover, revocation, database outage, and durable execution boundaries |\n| Image boundaries | Migration and product image verification scripts | Labels, users, entrypoints, component separation, image IDs, histories, and SBOMs |\n| Controlled product fleet | `verify-product-fleet.sh` | A 45-case configured product-host matrix with controlled upstream behavior, restart, failover, and database isolation |\n| Scale gate | Registry scale test using `aip-fleet-slo-v1` | Fixed numeric seeding, latency, throughput, memory, and database-growth thresholds |\n| Cal.diy isolated-live | `examples/cal-diy-qualification/qualify.sh` | One pinned Cal.diy deployment and reversible booking lifecycle |\n\nRead the detailed page before running a destructive or external procedure.\nSome scripts create and remove dedicated containers and volumes, require\nowner-only secret files, or mutate an isolated product account.\n\nThe controlled product-fleet script ran in protected Gitea Actions run `4242`\nfor the `v2.0.0` release and its 45-case matrix passed. It remains controlled\nintegration evidence; every product, including Twenty, requires separate\nexternal-provider evidence for a live-provider claim.\n\n## Retain a complete evidence package\n\nAn acceptable package identifies both the subject and the observation.\nRetain at least:\n\n- full source commit, tree state, and build or image digest;\n- protocol, profile, manifest, schema, connector, and upstream revisions in\n  scope;\n- exact commands, arguments, tool versions, and expected check IDs;\n- configuration class, topology, trust boundaries, and credential revision\n  references without secret values;\n- UTC start and finish times plus the final status;\n- raw reports, logs, query results, request traces, and lifecycle identifiers\n  required to reproduce each assertion;\n- a digest and size for every retained artifact;\n- failed, skipped, blocked, and not-applicable checks, not only successes;\n- limitations, environmental conditions, and required follow-up work.\n\nUse explicit states such as `PASS`, `FAIL`, `BLOCKED`, `NOT_RUN`, and\n`NOT_APPLICABLE`. Record why a check is not applicable and tie that reason to\nthe published contract. Never use an empty report as proof; both shared report\naggregators consider an empty check list successful.\n\n## Protect evidence without weakening it\n\nEvidence must be useful to an evaluator and safe to retain:\n\n- replace credentials with opaque revision or fingerprint references;\n- redact tokens, cookies, authorization headers, webhook secrets, personal\n  data, prompts, and provider payload fields outside the assertion;\n- preserve hashes and correlation IDs when they are needed to prove identity\n  or ordering;\n- keep raw artifacts access-controlled and publish only the bounded redacted\n  view;\n- record the redaction method and verify that it did not change the asserted\n  fact.\n\nDo not paste customer data or secret material into a Markdown report. A useful\nreport points to retained evidence by immutable identifier.\n\n## Apply retention and expiry rules\n\nThe reviewed GitHub CI definition retains fleet-scale logs for 30 days and\nfleet failure and image evidence for 14 days. Local fleet scripts write to\nbounded `.getaip-server-*` directories; the controlled product-fleet script creates a\nnew UTC-named run directory. These are operational defaults, not a sufficient\nrelease-retention policy.\n\nCopy evidence needed for a release decision into the protected release record\nbefore CI expiry. Keep immutable release digests, signature verification,\nattestations, and required qualification reports for at least the supported\nlifetime of that release.\n\nRerun affected evidence after a change to source, dependencies, upstream\nrevision, image, credentials policy, schema, topology, trust configuration,\nstorage, network boundary, or capacity target. Mark the older result historical\ninstead of silently editing its artifact identity.\n\n## Review an evidence package\n\nBefore accepting a claim, verify all of the following:\n\n- the artifact under review is the artifact named in the report;\n- the expected suites and check IDs are complete and nonempty;\n- every applicable failure or omission is visible;\n- every referenced artifact resolves and matches its recorded digest;\n- controlled fixtures are not described as external-provider runs;\n- historical results are not assigned to a later build;\n- the claim uses `implemented`, `conforming`, `qualified`, or\n  `production-ready` only at the evidence level actually reached.\n\nIf any identity or required artifact is missing, the strongest defensible\nresult is `not established`, not an inferred pass.\n\n## Related documentation\n\n- [Implementation status](../reference/implementation-status.md)\n- [Conformance and qualification](../reference/conformance.md)\n- [Connector documentation](../connectors/README.md)\n- [Release artifacts](../reference/release-artifacts.md)\n",
    "text": "Testing and evidence index\n\nUse this index to choose the smallest procedure that can support your claim and\nto find the evidence you must retain. Start with the claim, not with whichever\ntest command is easiest to run.\n\nThe pinned GetAIP 2.0.0 source contains tests, conformance suites,\nqualification scripts, and CI definitions. The retained release record shows\nthat the six protected contexts and their named controlled gates passed for\nrevision d7cce13d1d555644d04a4d73c66c95b113737635. It does not extend that\nresult to a rebuilt image, external provider, or target deployment.\n\nChoose evidence by the claim\n\n| Question you need to answer | Start here | Evidence boundary |\n\n| What does the pinned source implement? | Implementation status (../reference/implementation-status.md) | Code presence is not an executed result |\n| Does one schema instance, envelope, manifest, connector, or MCP surface satisfy its contract? | Conformance and qualification (../reference/conformance.md) | Run the complete applicable matrix and assert its expected checks |\n| Do registry, routing, host, lease, revocation, outage, and recovery paths work together? | Connector fleet qualification (connector-fleet-qualification.md) | Controlled multiprocess evidence is not product-provider evidence |\n| Does the registry meet its declared fleet-size thresholds? | Fleet scale SLO (fleet-scale-slo.md) | Results apply only to the recorded environment and dataset |\n| Which upstream product revision must a connector campaign use? | Pinned upstream baselines (../connectors/upstream-baselines.md) | A source pin does not qualify a built provider deployment |\n| How should an external product campaign be prepared and retained? | Live product E2E (live-product-e2e.md) | Each product, credential policy, topology, and image needs its own evidence |\n| What did the older Cal.diy campaign prove? | Cal.diy isolated-live evidence (cal-diy-isolated-live.md) | The result covers only the artifacts named in that report |\n| What did the older Hermes Agent campaign prove? | Hermes Agent isolated-live evidence (hermes-agent-isolated-live.md) | The result covers only the two named endpoints and their dependencies |\n| Was a release built, signed, attested, and verified? | Release artifacts (../reference/release-artifacts.md) | Workflow source is not publication evidence |\n\nIf the claim spans several rows, retain evidence for every row. A product\ncampaign does not replace protocol conformance, and a release signature does\nnot replace deployment recovery or capacity testing.\n\nFollow the evidence progression\n\nUse the layers in this order so a later failure does not hide a cheaper, more\nlocal defect:\n1. validate schemas and typed invariants;\n2. run deterministic unit, property, fixture, and recovery tests;\n3. run every applicable core, profile, and connector conformance suite;\n4. exercise process, storage, routing, restart, and failure boundaries;\n5. run isolated-live campaigns against pinned external products or peers;\n6. build and verify immutable release artifacts;\n7. qualify the target deployment's security, recovery, fault, and capacity\n   requirements.\n\nThe layers are cumulative only when they refer to the same source and build\nlineage. If an image is rebuilt after conformance, identify the new digest and\nrerun every affected layer.\n\nUse the source-owned procedures deliberately\n\n| Procedure family | Source-owned entry point | What its source is designed to cover |\n\n| Source release gate | cargo run -p xtask -- release-check | Format, boundaries, semantic version checks, schemas, lint, tests, dependencies, fuzz compilation, and API docs |\n| Core diagnostic | getaip conformance run | One built-in message body or one supplied native envelope |\n| MCP client diagnostic | getaip mcp conformance | Four external-client checks only |\n| Connector harness | runconnectorconformance | Atomic manifest admission and twelve conditional behavior families |\n| PostgreSQL recovery | aip-storage-postgres recovery test | Durable runtime recovery behavior |\n| Registry and control plane | PostgreSQL registry and control-plane tests | Schema, roles, routing, lifecycle, concurrency, and reconnect behavior |\n| Fleet failure matrix | verify-failure-matrix.sh | Multiprocess trust, routing, failover, revocation, database outage, and durable execution boundaries |\n| Image boundaries | Migration and product image verification scripts | Labels, users, entrypoints, component separation, image IDs, histories, and SBOMs |\n| Controlled product fleet | verify-product-fleet.sh | A 45-case configured product-host matrix with controlled upstream behavior, restart, failover, and database isolation |\n| Scale gate | Registry scale test using aip-fleet-slo-v1 | Fixed numeric seeding, latency, throughput, memory, and database-growth thresholds |\n| Cal.diy isolated-live | examples/cal-diy-qualification/qualify.sh | One pinned Cal.diy deployment and reversible booking lifecycle |\n\nRead the detailed page before running a destructive or external procedure.\nSome scripts create and remove dedicated containers and volumes, require\nowner-only secret files, or mutate an isolated product account.\n\nThe controlled product-fleet script ran in protected Gitea Actions run 4242\nfor the v2.0.0 release and its 45-case matrix passed. It remains controlled\nintegration evidence; every product, including Twenty, requires separate\nexternal-provider evidence for a live-provider claim.\n\nRetain a complete evidence package\n\nAn acceptable package identifies both the subject and the observation.\nRetain at least:\n• full source commit, tree state, and build or image digest;\n• protocol, profile, manifest, schema, connector, and upstream revisions in\n  scope;\n• exact commands, arguments, tool versions, and expected check IDs;\n• configuration class, topology, trust boundaries, and credential revision\n  references without secret values;\n• UTC start and finish times plus the final status;\n• raw reports, logs, query results, request traces, and lifecycle identifiers\n  required to reproduce each assertion;\n• a digest and size for every retained artifact;\n• failed, skipped, blocked, and not-applicable checks, not only successes;\n• limitations, environmental conditions, and required follow-up work.\n\nUse explicit states such as PASS, FAIL, BLOCKED, NOTRUN, and\nNOTAPPLICABLE. Record why a check is not applicable and tie that reason to\nthe published contract. Never use an empty report as proof; both shared report\naggregators consider an empty check list successful.\n\nProtect evidence without weakening it\n\nEvidence must be useful to an evaluator and safe to retain:\n• replace credentials with opaque revision or fingerprint references;\n• redact tokens, cookies, authorization headers, webhook secrets, personal\n  data, prompts, and provider payload fields outside the assertion;\n• preserve hashes and correlation IDs when they are needed to prove identity\n  or ordering;\n• keep raw artifacts access-controlled and publish only the bounded redacted\n  view;\n• record the redaction method and verify that it did not change the asserted\n  fact.\n\nDo not paste customer data or secret material into a Markdown report. A useful\nreport points to retained evidence by immutable identifier.\n\nApply retention and expiry rules\n\nThe reviewed GitHub CI definition retains fleet-scale logs for 30 days and\nfleet failure and image evidence for 14 days. Local fleet scripts write to\nbounded .getaip-server- directories; the controlled product-fleet script creates a\nnew UTC-named run directory. These are operational defaults, not a sufficient\nrelease-retention policy.\n\nCopy evidence needed for a release decision into the protected release record\nbefore CI expiry. Keep immutable release digests, signature verification,\nattestations, and required qualification reports for at least the supported\nlifetime of that release.\n\nRerun affected evidence after a change to source, dependencies, upstream\nrevision, image, credentials policy, schema, topology, trust configuration,\nstorage, network boundary, or capacity target. Mark the older result historical\ninstead of silently editing its artifact identity.\n\nReview an evidence package\n\nBefore accepting a claim, verify all of the following:\n• the artifact under review is the artifact named in the report;\n• the expected suites and check IDs are complete and nonempty;\n• every applicable failure or omission is visible;\n• every referenced artifact resolves and matches its recorded digest;\n• controlled fixtures are not described as external-provider runs;\n• historical results are not assigned to a later build;\n• the claim uses implemented, conforming, qualified, or\n  production-ready only at the evidence level actually reached.\n\nIf any identity or required artifact is missing, the strongest defensible\nresult is not established, not an inferred pass.\n\nRelated documentation\n• Implementation status (../reference/implementation-status.md)\n• Conformance and qualification (../reference/conformance.md)\n• Connector documentation (../connectors/README.md)\n• Release artifacts (../reference/release-artifacts.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "aa1485aedeae0b4bc527d10754812946774c758349c071ffad433349b18aae3e"
  }
}