Testing and evidence index

Use this index to choose the smallest procedure that can support your claim and to find the evidence you must retain. Start with the claim, not with whichever test command is easiest to run.

The pinned GetAIP 2.0.0 source contains tests, conformance suites, qualification scripts, and CI definitions. The retained release record shows that the six protected contexts and their named controlled gates passed for revision d7cce13d1d555644d04a4d73c66c95b113737635. It does not extend that result to a rebuilt image, external provider, or target deployment.

Choose evidence by the claim

If the claim spans several rows, retain evidence for every row. A product campaign does not replace protocol conformance, and a release signature does not replace deployment recovery or capacity testing.

Follow the evidence progression

Use the layers in this order so a later failure does not hide a cheaper, more local defect:

1. validate schemas and typed invariants;
2. run deterministic unit, property, fixture, and recovery tests;
3. run every applicable core, profile, and connector conformance suite;
4. exercise process, storage, routing, restart, and failure boundaries;
5. run isolated-live campaigns against pinned external products or peers;
6. build and verify immutable release artifacts;
7. qualify the target deployment’s security, recovery, fault, and capacity requirements.

The layers are cumulative only when they refer to the same source and build lineage. If an image is rebuilt after conformance, identify the new digest and rerun every affected layer.

Use the source-owned procedures deliberately

Read the detailed page before running a destructive or external procedure. Some scripts create and remove dedicated containers and volumes, require owner-only secret files, or mutate an isolated product account.

The controlled product-fleet script ran in protected Gitea Actions run 4242 for the v2.0.0 release and its 45-case matrix passed. It remains controlled integration evidence; every product, including Twenty, requires separate external-provider evidence for a live-provider claim.

Retain a complete evidence package

An acceptable package identifies both the subject and the observation. Retain at least:

• full source commit, tree state, and build or image digest;
• protocol, profile, manifest, schema, connector, and upstream revisions in scope;
• exact commands, arguments, tool versions, and expected check IDs;
• configuration class, topology, trust boundaries, and credential revision references without secret values;
• UTC start and finish times plus the final status;
• raw reports, logs, query results, request traces, and lifecycle identifiers required to reproduce each assertion;
• a digest and size for every retained artifact;
• failed, skipped, blocked, and not-applicable checks, not only successes;
• limitations, environmental conditions, and required follow-up work.

Use explicit states such as PASS, FAIL, BLOCKED, NOT_RUN, and NOT_APPLICABLE. Record why a check is not applicable and tie that reason to the published contract. Never use an empty report as proof; both shared report aggregators consider an empty check list successful.

Protect evidence without weakening it

Evidence must be useful to an evaluator and safe to retain:

• replace credentials with opaque revision or fingerprint references;
• redact tokens, cookies, authorization headers, webhook secrets, personal data, prompts, and provider payload fields outside the assertion;
• preserve hashes and correlation IDs when they are needed to prove identity or ordering;
• keep raw artifacts access-controlled and publish only the bounded redacted view;
• record the redaction method and verify that it did not change the asserted fact.

Do not paste customer data or secret material into a Markdown report. A useful report points to retained evidence by immutable identifier.

Apply retention and expiry rules

The reviewed GitHub CI definition retains fleet-scale logs for 30 days and fleet failure and image evidence for 14 days. Local fleet scripts write to bounded .getaip-server-* directories; the controlled product-fleet script creates a new UTC-named run directory. These are operational defaults, not a sufficient release-retention policy.

Copy evidence needed for a release decision into the protected release record before CI expiry. Keep immutable release digests, signature verification, attestations, and required qualification reports for at least the supported lifetime of that release.

Rerun affected evidence after a change to source, dependencies, upstream revision, image, credentials policy, schema, topology, trust configuration, storage, network boundary, or capacity target. Mark the older result historical instead of silently editing its artifact identity.

Review an evidence package

Before accepting a claim, verify all of the following:

• the artifact under review is the artifact named in the report;
• the expected suites and check IDs are complete and nonempty;
• every applicable failure or omission is visible;
• every referenced artifact resolves and matches its recorded digest;
• controlled fixtures are not described as external-provider runs;
• historical results are not assigned to a later build;
• the claim uses implemented, conforming, qualified, or production-ready only at the evidence level actually reached.

If any identity or required artifact is missing, the strongest defensible result is not established, not an inferred pass.

Related documentation

• Implementation status
• Conformance and qualification
• Connector documentation
• Release artifacts