{
  "schemaVersion": "1.0",
  "title": "Connector fleet qualification and failure matrix",
  "description": "Use this procedure to evaluate routing, lifecycle, trust, persistence, and failure recovery across a complete isolated connector fleet. The source-owned matrix exercises real AIP processes and PostgreSQL state against controlled provider be",
  "canonical": "https://getaip.org/docs/testing/connector-fleet-qualification",
  "route": "/docs/testing/connector-fleet-qualification",
  "source": "docs/testing/connector-fleet-qualification.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Qualification and Evidence",
  "documentType": "Qualification evidence",
  "language": "en",
  "downloads": {
    "md": "/docs/download/testing/connector-fleet-qualification.md",
    "txt": "/docs/download/testing/connector-fleet-qualification.txt",
    "json": "/docs/download/testing/connector-fleet-qualification.json",
    "pdf": "/docs/download/testing/connector-fleet-qualification.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Connector fleet qualification and failure matrix\ndescription: Run and evaluate the isolated multiprocess connector fleet without confusing controlled failures with product qualification\nkind: procedure\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\n---\n\n# Connector fleet qualification and failure matrix\n\nUse this procedure to evaluate routing, lifecycle, trust, persistence, and\nfailure recovery across a complete isolated connector fleet. The source-owned\nmatrix exercises real AIP processes and PostgreSQL state against controlled\nprovider behavior. It does not contact or qualify any of the six public\nproducts.\n\nThis page describes the procedure at source revision\n`97be86e9efedf07ecf1783b03800f683f107fb04`. It does not record a run for that\nrevision. A result becomes evidence only after the exact source and generated\nartifacts are identified and the retained report is reviewed.\n\n## Understand the destructive scope\n\nThe script owns a fixed Compose project and removes that project's containers,\nnetworks, and volumes before and after the run. It also removes and recreates\nthe selected evidence directory.\n\nBefore continuing:\n\n- use a dedicated Docker host or confirm that no other run uses the Compose\n  project `aip-connector-fleet-qualification`;\n- choose a new, dedicated evidence path that contains no valuable files;\n- reserve enough time, CPU, memory, and disk for local Rust image builds and\n  several PostgreSQL-backed processes;\n- record the full Git commit, tree state, Docker and Compose versions, host\n  architecture, and UTC start time;\n- do not place provider credentials or production data in this topology.\n\nThe preparation step writes `deploy/connector-fleet/.env.qualification` with\nmode `0600`. It derives the source identity from tracked and unignored files,\nso a dirty worktree produces a different `workspace-sha256` identity. Preserve\nboth the Git state and derived identity in the report.\n\n## Know the isolated topology\n\n| Component | Role in the matrix |\n|---|---|\n| TLS edge | Terminates the internal HTTPS routes used by the qualification client |\n| Two AIP gateways | Prove routing through either data-plane replica |\n| Lifecycle service | Handles signed connector registration, heartbeat, drain, and offline transitions |\n| PostgreSQL | Holds the registry plus separately owned runtime and controlled-provider databases |\n| Three connector instances | Represent two controlled connector types across two tenants |\n| Four connector replicas | Provide one two-replica instance and two single-replica instances |\n| Qualification client | Sends signed native calls and catalog queries with tenant-specific identities |\n| Controlled providers | Expose deterministic reads, mutations, approvals, and audit facts |\n\nThe Compose network is internal. Runtime services use distinct identities,\nsigning material, credential policies, and PostgreSQL roles. Connector hosts do\nnot receive the catalog-administrator credential.\n\n## Review the baseline\n\nThe baseline runner must complete eight operations before the failure matrix\nstarts:\n\n| Baseline operation | Required observation |\n|---|---|\n| Four signed native calls | Both tenants, both controlled connector types, and both gateways return completed actions |\n| Tenant capability query | The permitted capability appears in the bounded catalog |\n| Signed catalog query for the allowed tenant | The second controlled capability is visible |\n| Signed catalog query for the isolated tenant | The same capability is absent |\n| Signed orchestration plan | Verification returns `verified` with snapshot fencing enabled |\n\nIts machine-readable summary must state `passed`, `cases: 8`, `gateways: 2`,\n`connector_instances: 3`, and `connector_replicas: 4`. These values establish\nthe expected topology; they are not a capacity or product result.\n\n## Review every failure family\n\nThe outer matrix performs the following cases in order. A later success does\nnot erase an earlier missing or failed case.\n\n| Case | Fault or transition | Acceptance invariant |\n|---|---|---|\n| Baseline and registry topology | Start the isolated stack and inspect live leases | The eight baseline operations pass and exactly four replicas are ready with unexpired leases |\n| Credential revision revocation | Revoke the connector's current credential revision | The call fails closed with the expected credential-revision error |\n| Credential policy restore | Atomically restore the permitted revision | The same bounded read completes |\n| Tenant-provider isolation | Ask one tenant for another tenant's provider record | The request fails closed |\n| Gateway replica failover | Stop the first gateway | The same signed read completes through the second gateway |\n| Graceful connector failover | Stop one connector replica cleanly | The registry marks it offline and routing continues through its peer |\n| Lease-expiry fencing | Hard-kill the remaining replica and disable automatic restart | After lease expiry, routing fails closed instead of using the dead replica |\n| Host recovery | Restart the hard-killed replica | Health and signed routing recover |\n| Idempotent mutation replay | Complete one governed mutation, hard-kill its host, then replay the same key | The provider retains one effect, one approval request, and one creation audit event |\n| Synchronous crash windows | Kill the selected host at each of five durable boundaries, then retry the same identifiers | Every retry completes with exactly one provider effect and one durable result chain |\n| Shared database outage | Stop PostgreSQL while registry and runtime paths are active | Calls fail closed |\n| Database recovery | Restart PostgreSQL without replacing the AIP processes | Gateways, lifecycle service, hosts, and signed routing recover |\n| Admission revocation | Revoke the signed connector admission package | Routing through the revoked package fails closed |\n\n### Verify the five crash windows\n\nEach crash case reuses one stable action ID, transaction ID, idempotency key,\ninstance, and replica. The controller waits for a durable observation marker,\nsends `SIGKILL`, restarts the process, and retries the same logical operation.\n\n| Crash boundary | What has happened before the kill |\n|---|---|\n| `before_intent_persistence` | No durable execution intent is promised |\n| `intent_persisted` | The runtime intent is durable |\n| `provider_effect_committed` | The controlled provider has committed its effect |\n| `provider_response_received` | The provider response has reached the connector |\n| `result_persisted` | The terminal AIP result is durable |\n\nAfter every recovery, independent provider and runtime queries must find\nexactly one of each required record:\n\n- provider effect;\n- approval request;\n- provider creation audit event;\n- settled idempotency result;\n- terminal action result;\n- transaction receipt chain.\n\nThis is the decisive exactly-once-effect assertion. A completed HTTP response\nwithout those independent counts is insufficient.\n\n## Run the source-owned matrix\n\nFrom the exact repository root, assign a fresh evidence directory and run:\n\n```sh\nexport AIP_FLEET_EVIDENCE_DIR=\"$PWD/.aipd-fleet-qualification/run-manual\"\ndeploy/connector-fleet/verify-failure-matrix.sh\n```\n\nThe script calls `prepare.sh` and builds locally identified images. It then\ninitializes the isolated trust and database state, executes the baseline and\nfailure matrix, captures diagnostics on both success and failure, and tears\ndown the dedicated Compose project with its volumes.\n\nDo not point `AIP_FLEET_EVIDENCE_DIR` at the repository root, a shared artifact\ndirectory, or any path that must survive `rm -rf`. Do not run two matrices\nconcurrently because the Compose project and generated environment file are\nshared.\n\n## Inspect the retained evidence\n\nThe selected directory should contain:\n\n| Artifact | Review purpose |\n|---|---|\n| `summary.json` | Final script status and completion time |\n| `failure-matrix.jsonl` | Ordered case, status, detail, and UTC record for the matrix |\n| `qualification/summary.json` | Baseline topology and case totals |\n| `qualification/operations.jsonl` | Per-operation tenant, gateway, time, status, and response reference |\n| Per-case JSON and stderr files | Native result or fail-closed observation for each probe |\n| Crash checkpoint and retry files | Durable stage observation, interrupted outcome, attempts, and recovered result |\n| Route snapshots | Assignment, lease, capacity, active work, health, and circuit state around recovery |\n| `registry-final-state.json` | Final replicas, routes, and admission counters |\n| `docker-events.jsonl` | Container lifecycle ordering |\n| `compose-ps.jsonl` and `compose.log` | Final process state and timestamped service diagnostics |\n| `compose-down.log` | Dedicated-project teardown result |\n\nThe script writes `summary.json` even when it fails. A file's presence is not a\npass. Accept the run only when all of these conditions hold:\n\n1. the final summary says `passed`;\n2. the ordered matrix contains every expected case and the final `complete`\n   record;\n3. the baseline contains all eight expected operations;\n4. every crash window has its checkpoint, retry history, recovered result, and\n   independent one-record assertions;\n5. all referenced artifacts exist, are redacted, and have retained digests;\n6. the source, local image IDs, environment, and time interval are recorded;\n7. no earlier failure was hidden by a later recovery.\n\n## Preserve CI evidence before expiry\n\nThe reviewed GitHub CI workflow runs this matrix together with migration-image\nand product-image verification and retains the three evidence directories for\n14 days. The reviewed Gitea workflow declares 30 days. Copy evidence required\nfor a release decision into a protected, immutable release record before the\nshorter retention window expires.\n\nCI artifacts still require review. Confirm that the uploaded source commit is\nthe commit under evaluation and that the matrix did not produce an incomplete\nbut superficially green report.\n\n## State the result narrowly\n\nA complete run can support this form of claim:\n\n> The identified AIP artifacts passed the controlled connector-fleet failure\n> matrix in the recorded isolated topology at the stated time.\n\nIt cannot establish external product behavior, the six public connector\ncatalogues, fleet-scale SLOs, independent protocol interoperability, published\nimage signatures, arbitrary infrastructure, or production readiness. Use\n[live product E2E](live-product-e2e.md), the\n[fleet scale SLO](fleet-scale-slo.md), and deployment-specific campaigns for\nthose claims.\n\n## Related documentation\n\n- [Testing and evidence index](README.md)\n- [Conformance and qualification](../reference/conformance.md)\n- [Connector fleet architecture](../architecture/connector-fleet.md)\n- [Registry and routing](../architecture/connector-registry-and-routing.md)\n- [Live product E2E](live-product-e2e.md)\n",
    "text": "Connector fleet qualification and failure matrix\n\nUse this procedure to evaluate routing, lifecycle, trust, persistence, and\nfailure recovery across a complete isolated connector fleet. The source-owned\nmatrix exercises real AIP processes and PostgreSQL state against controlled\nprovider behavior. It does not contact or qualify any of the six public\nproducts.\n\nThis page describes the procedure at source revision\n97be86e9efedf07ecf1783b03800f683f107fb04. It does not record a run for that\nrevision. A result becomes evidence only after the exact source and generated\nartifacts are identified and the retained report is reviewed.\n\nUnderstand the destructive scope\n\nThe script owns a fixed Compose project and removes that project's containers,\nnetworks, and volumes before and after the run. It also removes and recreates\nthe selected evidence directory.\n\nBefore continuing:\n• use a dedicated Docker host or confirm that no other run uses the Compose\n  project aip-connector-fleet-qualification;\n• choose a new, dedicated evidence path that contains no valuable files;\n• reserve enough time, CPU, memory, and disk for local Rust image builds and\n  several PostgreSQL-backed processes;\n• record the full Git commit, tree state, Docker and Compose versions, host\n  architecture, and UTC start time;\n• do not place provider credentials or production data in this topology.\n\nThe preparation step writes deploy/connector-fleet/.env.qualification with\nmode 0600. It derives the source identity from tracked and unignored files,\nso a dirty worktree produces a different workspace-sha256 identity. Preserve\nboth the Git state and derived identity in the report.\n\nKnow the isolated topology\n\n| Component | Role in the matrix |\n\n| TLS edge | Terminates the internal HTTPS routes used by the qualification client |\n| Two AIP gateways | Prove routing through either data-plane replica |\n| Lifecycle service | Handles signed connector registration, heartbeat, drain, and offline transitions |\n| PostgreSQL | Holds the registry plus separately owned runtime and controlled-provider databases |\n| Three connector instances | Represent two controlled connector types across two tenants |\n| Four connector replicas | Provide one two-replica instance and two single-replica instances |\n| Qualification client | Sends signed native calls and catalog queries with tenant-specific identities |\n| Controlled providers | Expose deterministic reads, mutations, approvals, and audit facts |\n\nThe Compose network is internal. Runtime services use distinct identities,\nsigning material, credential policies, and PostgreSQL roles. Connector hosts do\nnot receive the catalog-administrator credential.\n\nReview the baseline\n\nThe baseline runner must complete eight operations before the failure matrix\nstarts:\n\n| Baseline operation | Required observation |\n\n| Four signed native calls | Both tenants, both controlled connector types, and both gateways return completed actions |\n| Tenant capability query | The permitted capability appears in the bounded catalog |\n| Signed catalog query for the allowed tenant | The second controlled capability is visible |\n| Signed catalog query for the isolated tenant | The same capability is absent |\n| Signed orchestration plan | Verification returns verified with snapshot fencing enabled |\n\nIts machine-readable summary must state passed, cases: 8, gateways: 2,\nconnectorinstances: 3, and connectorreplicas: 4. These values establish\nthe expected topology; they are not a capacity or product result.\n\nReview every failure family\n\nThe outer matrix performs the following cases in order. A later success does\nnot erase an earlier missing or failed case.\n\n| Case | Fault or transition | Acceptance invariant |\n\n| Baseline and registry topology | Start the isolated stack and inspect live leases | The eight baseline operations pass and exactly four replicas are ready with unexpired leases |\n| Credential revision revocation | Revoke the connector's current credential revision | The call fails closed with the expected credential-revision error |\n| Credential policy restore | Atomically restore the permitted revision | The same bounded read completes |\n| Tenant-provider isolation | Ask one tenant for another tenant's provider record | The request fails closed |\n| Gateway replica failover | Stop the first gateway | The same signed read completes through the second gateway |\n| Graceful connector failover | Stop one connector replica cleanly | The registry marks it offline and routing continues through its peer |\n| Lease-expiry fencing | Hard-kill the remaining replica and disable automatic restart | After lease expiry, routing fails closed instead of using the dead replica |\n| Host recovery | Restart the hard-killed replica | Health and signed routing recover |\n| Idempotent mutation replay | Complete one governed mutation, hard-kill its host, then replay the same key | The provider retains one effect, one approval request, and one creation audit event |\n| Synchronous crash windows | Kill the selected host at each of five durable boundaries, then retry the same identifiers | Every retry completes with exactly one provider effect and one durable result chain |\n| Shared database outage | Stop PostgreSQL while registry and runtime paths are active | Calls fail closed |\n| Database recovery | Restart PostgreSQL without replacing the AIP processes | Gateways, lifecycle service, hosts, and signed routing recover |\n| Admission revocation | Revoke the signed connector admission package | Routing through the revoked package fails closed |\n\nVerify the five crash windows\n\nEach crash case reuses one stable action ID, transaction ID, idempotency key,\ninstance, and replica. The controller waits for a durable observation marker,\nsends SIGKILL, restarts the process, and retries the same logical operation.\n\n| Crash boundary | What has happened before the kill |\n\n| beforeintentpersistence | No durable execution intent is promised |\n| intentpersisted | The runtime intent is durable |\n| providereffectcommitted | The controlled provider has committed its effect |\n| providerresponsereceived | The provider response has reached the connector |\n| resultpersisted | The terminal AIP result is durable |\n\nAfter every recovery, independent provider and runtime queries must find\nexactly one of each required record:\n• provider effect;\n• approval request;\n• provider creation audit event;\n• settled idempotency result;\n• terminal action result;\n• transaction receipt chain.\n\nThis is the decisive exactly-once-effect assertion. A completed HTTP response\nwithout those independent counts is insufficient.\n\nRun the source-owned matrix\n\nFrom the exact repository root, assign a fresh evidence directory and run:\n\nexport AIPFLEETEVIDENCEDIR=\"$PWD/.aipd-fleet-qualification/run-manual\"\ndeploy/connector-fleet/verify-failure-matrix.sh\n\nThe script calls prepare.sh and builds locally identified images. It then\ninitializes the isolated trust and database state, executes the baseline and\nfailure matrix, captures diagnostics on both success and failure, and tears\ndown the dedicated Compose project with its volumes.\n\nDo not point AIPFLEETEVIDENCEDIR at the repository root, a shared artifact\ndirectory, or any path that must survive rm -rf. Do not run two matrices\nconcurrently because the Compose project and generated environment file are\nshared.\n\nInspect the retained evidence\n\nThe selected directory should contain:\n\n| Artifact | Review purpose |\n\n| summary.json | Final script status and completion time |\n| failure-matrix.jsonl | Ordered case, status, detail, and UTC record for the matrix |\n| qualification/summary.json | Baseline topology and case totals |\n| qualification/operations.jsonl | Per-operation tenant, gateway, time, status, and response reference |\n| Per-case JSON and stderr files | Native result or fail-closed observation for each probe |\n| Crash checkpoint and retry files | Durable stage observation, interrupted outcome, attempts, and recovered result |\n| Route snapshots | Assignment, lease, capacity, active work, health, and circuit state around recovery |\n| registry-final-state.json | Final replicas, routes, and admission counters |\n| docker-events.jsonl | Container lifecycle ordering |\n| compose-ps.jsonl and compose.log | Final process state and timestamped service diagnostics |\n| compose-down.log | Dedicated-project teardown result |\n\nThe script writes summary.json even when it fails. A file's presence is not a\npass. Accept the run only when all of these conditions hold:\n1. the final summary says passed;\n2. the ordered matrix contains every expected case and the final complete\n   record;\n3. the baseline contains all eight expected operations;\n4. every crash window has its checkpoint, retry history, recovered result, and\n   independent one-record assertions;\n5. all referenced artifacts exist, are redacted, and have retained digests;\n6. the source, local image IDs, environment, and time interval are recorded;\n7. no earlier failure was hidden by a later recovery.\n\nPreserve CI evidence before expiry\n\nThe reviewed GitHub CI workflow runs this matrix together with migration-image\nand product-image verification and retains the three evidence directories for\n14 days. The reviewed Gitea workflow declares 30 days. Copy evidence required\nfor a release decision into a protected, immutable release record before the\nshorter retention window expires.\n\nCI artifacts still require review. Confirm that the uploaded source commit is\nthe commit under evaluation and that the matrix did not produce an incomplete\nbut superficially green report.\n\nState the result narrowly\n\nA complete run can support this form of claim:\n\nThe identified AIP artifacts passed the controlled connector-fleet failure\nmatrix in the recorded isolated topology at the stated time.\n\nIt cannot establish external product behavior, the six public connector\ncatalogues, fleet-scale SLOs, independent protocol interoperability, published\nimage signatures, arbitrary infrastructure, or production readiness. Use\nlive product E2E (live-product-e2e.md), the\nfleet scale SLO (fleet-scale-slo.md), and deployment-specific campaigns for\nthose claims.\n\nRelated documentation\n• Testing and evidence index (README.md)\n• Conformance and qualification (../reference/conformance.md)\n• Connector fleet architecture (../architecture/connector-fleet.md)\n• Registry and routing (../architecture/connector-registry-and-routing.md)\n• Live product E2E (live-product-e2e.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "3c83c8f10679ad4eb4cd268ee042d5b4c6ae25db3f1bff63c8b436f2276f426a"
  }
}