{
  "schemaVersion": "1.0",
  "title": "Fleet scale SLO",
  "description": "Use this procedure to check that connector catalog and route resolution remain within the source-owned aip-fleet-slo-v1 regression bounds. The gate uses a real PostgreSQL database and synthetic registry records. It measures registry behavio",
  "canonical": "https://getaip.org/docs/testing/fleet-scale-slo",
  "route": "/docs/testing/fleet-scale-slo",
  "source": "docs/testing/fleet-scale-slo.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Qualification and Evidence",
  "documentType": "Qualification evidence",
  "language": "en",
  "downloads": {
    "md": "/docs/download/testing/fleet-scale-slo.md",
    "txt": "/docs/download/testing/fleet-scale-slo.txt",
    "json": "/docs/download/testing/fleet-scale-slo.json",
    "pdf": "/docs/download/testing/fleet-scale-slo.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Fleet scale SLO\ndescription: Run and interpret the PostgreSQL connector-registry regression gate without treating one environment as a universal capacity promise\nkind: procedure\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\n---\n\n# Fleet scale SLO\n\nUse this procedure to check that connector catalog and route resolution remain\nwithin the source-owned `aip-fleet-slo-v1` regression bounds. The gate uses a\nreal PostgreSQL database and synthetic registry records. It measures registry\nbehavior, not connector execution or end-to-end application latency.\n\nThis page describes the test at source revision\n`97be86e9efedf07ecf1783b03800f683f107fb04`. No scale run was performed while\npreparing the page. A result applies only to the recorded source, database,\nhost, dataset, configuration, and time interval.\n\n## Prevent a silent skip\n\nThe test is opt-in. If `AIP_POSTGRES_SCALE_TEST_URL` is missing or empty, it\nprints a skip message and returns success. A green `cargo test` exit status is\ntherefore not sufficient evidence.\n\nAccept a run only when its combined output contains exactly one final line\nbeginning with:\n\n```text\nAIP_SCALE_RESULT=\n```\n\nThe JSON after that prefix must identify `aip-fleet-slo-v1` and contain\n`\"passed\": true`. CI and local wrappers must assert the result line rather than\nrelying only on the test process exit code.\n\n## Use the default dataset for comparable results\n\n| Dataset dimension | Default | Definition |\n|---|---:|---|\n| Connector types | 1,000 | One active version is initially created for each type |\n| Global capabilities | 1,000 | Unique definitions reused across versions and tenants |\n| Capability providers | 100,000 | Each version supplies 100 capabilities |\n| Connector instances | 10,000 | Distributed across 100 tenants and the connector types |\n| Ready replicas | 10,000 | One replica with capacity 100 per instance |\n| Tenant bindings | 100,000 | Ten capability bindings per instance |\n| Tenants | 100 | Fixed by the test |\n| Parallel catalog reads | 32 | One first-page read for each of the first 32 tenants |\n| Concurrent route resolutions | 500 | Timed route assignments for one tenant and capability |\n| Lease churn | 1,000 replicas | The smaller of 1,000 and the configured instance count |\n\nThe configurable counts must be positive. Capabilities must be at least 100,\nand instances must be at least the number of connector types.\n\nOverride these values only for a named custom campaign:\n\n| Environment variable | Default |\n|---|---:|\n| `AIP_SCALE_CONNECTOR_TYPES` | `1000` |\n| `AIP_SCALE_CAPABILITIES` | `1000` |\n| `AIP_SCALE_INSTANCES` | `10000` |\n| `AIP_SCALE_CONCURRENT_ROUTES` | `500` |\n\nThe thresholds do not change when the dataset is overridden. Record every\noverride and do not compare a custom dataset directly with the default CI\nbaseline.\n\n## Apply the fixed thresholds\n\n| Metric | Gate | Exact source value |\n|---|---:|---:|\n| Seed and lifecycle setup | At most 60 seconds | `60000 ms` |\n| Parallel catalog p99 | At most 2 seconds | `2000000 us` |\n| Concurrent route p99 | At most 3 seconds | `3000000 us` |\n| Route throughput | At least 100 assignments per second | `100.0/s` |\n| Rust test process high-water RSS | At most 512 MiB | `524288 KiB` |\n| PostgreSQL database growth | At most 1 GiB | `1073741824 bytes` |\n\nThe setup timer includes seeding plus the version rollout, noisy-tenant case,\nand retry-storm case. It is broader than raw insert time.\n\nCatalog p99 is computed from 32 concurrent first-page reads at the default\ndataset. Route p99 and throughput use the 500 concurrent route tasks. The test\nuses its own integer percentile function over the collected samples; do not\nsubstitute a monitoring system's percentile method when comparing results.\n\nProcess RSS is the Rust test process high-water mark. It does not include\nPostgreSQL memory. Database growth is the change in `pg_database_size` for the\ncurrent database and can be distorted by unrelated concurrent work. Use an\nisolated database.\n\nIf the platform-specific RSS probe fails, the source records zero instead of\nfailing the test. Reject a memory-bound claim when\n`process_rss_kib_observed_max` is zero or otherwise implausible.\n\n## Verify the functional invariants\n\nThe numeric thresholds are only part of the gate. The same run must also prove:\n\n| Invariant | Required behavior |\n|---|---|\n| Version rollout | One instance moves to a new active version, its old replica drains, and a new ready replica serves the new version |\n| Noisy-tenant isolation | Of 64 simultaneous requests for a tenant with budget four, four are admitted and 60 are rejected while four healthy-tenant requests succeed |\n| Retry-storm isolation | Of 64 simultaneous retries with budget eight, eight are admitted and 56 are rejected while eight healthy requests succeed |\n| Catalog pagination | Page size never exceeds 50, revision and total remain stable, and every returned capability is unique |\n| Route distribution | Every eligible replica is used when the sample is large enough, and the largest and smallest assignment counts differ by at most three |\n| Reservation release | Settling all timed routes leaves zero active assignments in the run's replicas |\n| Retry fencing | Up to 100 repeated action IDs resolve to their original assignments |\n| Credential rotation | An existing route remains pinned to credential revision one while a new action uses revision two |\n| Indexed routing | The plan does not sequentially scan tenant bindings or replicas and uses an accepted binding index |\n| Lease churn | Up to 1,000 leases renew, expire, and transition through bounded batches |\n\nThese assertions make the test a mixed functional and performance regression\ngate. A threshold pass with a missing lifecycle assertion is not a valid\nresult.\n\n## Prepare a disposable PostgreSQL database\n\nUse a database dedicated to this campaign. The test creates a unique UUIDv7\nrun identifier and prefixes its rows, then removes those rows after a successful\nresult. If the test fails or is interrupted before the final cleanup call, its\nrun-scoped rows may remain.\n\nFor that reason:\n\n- do not point the test at a production or shared development database;\n- take the database baseline before unrelated jobs start;\n- ensure the database user can install and use the registry schema;\n- record PostgreSQL version, `max_connections`, `shared_buffers`, storage\n  class, host resources, and connection pool settings;\n- discard or restore the dedicated database after a failed run instead of\n  guessing which partial rows are safe to delete.\n\nSet a descriptive environment label. It must contain 1 to 128 ASCII letters,\ndigits, `-`, `_`, `.`, or `/`. If omitted, the result says\n`local-unspecified`, which is too weak for comparison evidence.\n\n## Run the gate and retain its output\n\nFrom the exact source root, use a shell with pipeline failure propagation:\n\n```bash\nexport AIP_POSTGRES_SCALE_TEST_URL='postgresql://user:password@host/database'\nexport AIP_SCALE_ENVIRONMENT_LABEL='dedicated-linux-amd64-postgres-17.6'\n\nset -o pipefail\ncargo test -p aip-connector-registry-postgres --test scale \\\n  indexed_catalog_and_routing_remain_bounded_at_fleet_scale \\\n  -- --nocapture 2>&1 | tee .aipd-scale-qualification.log\n```\n\nUse a secret-file or CI secret mechanism for the real database URL. Do not\ncommit the command history, environment, or log if it contains credentials.\n\nExtract and validate the final result:\n\n```bash\ntest \"$(grep -c '^AIP_SCALE_RESULT=' .aipd-scale-qualification.log)\" -eq 1\n\nresult=$(sed -n 's/^AIP_SCALE_RESULT=//p' \\\n  .aipd-scale-qualification.log | tail -n 1)\n\ntest -n \"$result\"\nprintf '%s\\n' \"$result\" | jq -e \\\n  '.slo.profile == \"aip-fleet-slo-v1\" and .slo.passed == true'\n```\n\nReject the evidence if the result line is missing, duplicated, malformed, or\ndoes not match the expected source profile.\n\nRequire both a successful pipeline exit and the validated JSON. The source\nprints `AIP_SCALE_RESULT` before deleting its run-scoped rows, so cleanup can\nstill fail after a JSON object says `passed: true`. The `pipefail` setting\npreserves that late test failure through `tee`.\n\n## Inspect the result fields\n\nThe final JSON records:\n\n- environment label, operating system, architecture, PostgreSQL version and\n  selected settings;\n- exact dataset counts;\n- threshold values embedded by the source;\n- catalog and routing p50, p95, and p99 samples;\n- routing throughput, errors, eligible replicas, and action count;\n- version rollout, tenant isolation, retry isolation, credential rotation, and\n  lease-churn outcomes;\n- observed process RSS, database sizes and growth, available parallelism, and\n  connection pool snapshot.\n\nRetain the full combined log, parsed JSON, source commit, dependency lockfile,\ndatabase image or package identity, database configuration, host inventory,\nstart and finish times, and cryptographic digests for the evidence files.\n\nThe log also contains the routing query plan. Review it even when the summary\npasses, because it explains which index path produced the measured result.\n\n## Compare runs honestly\n\nCompare two results only when their source, dataset, PostgreSQL major version,\ndatabase settings, host resources, architecture, storage class, and contention\nare equivalent. Otherwise describe the difference as an environment\nobservation rather than a regression.\n\nUse repeated runs and distribution-aware production telemetry for capacity\nplanning. One 32-sample catalog percentile and one 500-route burst are useful\nCI guards, not a full latency study.\n\nThe reviewed GitHub and Gitea CI definitions use PostgreSQL 17.6, provide an\nexplicit environment label, retain the combined log, and declare a 30-day\nartifact lifetime. Copy evidence required for a release decision before it\nexpires.\n\n## State the result narrowly\n\nA valid claim is:\n\n> The identified source passed `aip-fleet-slo-v1` with the recorded dataset and\n> PostgreSQL environment at the stated time.\n\nThe result does not qualify connector processes, external products, network\ntransports, multi-host behavior, application execution, arbitrary fleet sizes,\nor production capacity. Combine it with the\n[connector fleet failure matrix](connector-fleet-qualification.md) and\ndeployment-specific load, recovery, and observability campaigns.\n\n## Related documentation\n\n- [Testing and evidence index](README.md)\n- [Registry and routing architecture](../architecture/connector-registry-and-routing.md)\n- [Metrics reference](../reference/metrics.md)\n- [Connector fleet qualification](connector-fleet-qualification.md)\n- [Conformance and qualification](../reference/conformance.md)\n",
    "text": "Fleet scale SLO\n\nUse this procedure to check that connector catalog and route resolution remain\nwithin the source-owned aip-fleet-slo-v1 regression bounds. The gate uses a\nreal PostgreSQL database and synthetic registry records. It measures registry\nbehavior, not connector execution or end-to-end application latency.\n\nThis page describes the test at source revision\n97be86e9efedf07ecf1783b03800f683f107fb04. No scale run was performed while\npreparing the page. A result applies only to the recorded source, database,\nhost, dataset, configuration, and time interval.\n\nPrevent a silent skip\n\nThe test is opt-in. If AIPPOSTGRESSCALETESTURL is missing or empty, it\nprints a skip message and returns success. A green cargo test exit status is\ntherefore not sufficient evidence.\n\nAccept a run only when its combined output contains exactly one final line\nbeginning with:\n\nAIPSCALERESULT=\n\nThe JSON after that prefix must identify aip-fleet-slo-v1 and contain\n\"passed\": true. CI and local wrappers must assert the result line rather than\nrelying only on the test process exit code.\n\nUse the default dataset for comparable results\n\n| Dataset dimension | Default | Definition |\n\n| Connector types | 1,000 | One active version is initially created for each type |\n| Global capabilities | 1,000 | Unique definitions reused across versions and tenants |\n| Capability providers | 100,000 | Each version supplies 100 capabilities |\n| Connector instances | 10,000 | Distributed across 100 tenants and the connector types |\n| Ready replicas | 10,000 | One replica with capacity 100 per instance |\n| Tenant bindings | 100,000 | Ten capability bindings per instance |\n| Tenants | 100 | Fixed by the test |\n| Parallel catalog reads | 32 | One first-page read for each of the first 32 tenants |\n| Concurrent route resolutions | 500 | Timed route assignments for one tenant and capability |\n| Lease churn | 1,000 replicas | The smaller of 1,000 and the configured instance count |\n\nThe configurable counts must be positive. Capabilities must be at least 100,\nand instances must be at least the number of connector types.\n\nOverride these values only for a named custom campaign:\n\n| Environment variable | Default |\n\n| AIPSCALECONNECTORTYPES | 1000 |\n| AIPSCALECAPABILITIES | 1000 |\n| AIPSCALEINSTANCES | 10000 |\n| AIPSCALECONCURRENTROUTES | 500 |\n\nThe thresholds do not change when the dataset is overridden. Record every\noverride and do not compare a custom dataset directly with the default CI\nbaseline.\n\nApply the fixed thresholds\n\n| Metric | Gate | Exact source value |\n\n| Seed and lifecycle setup | At most 60 seconds | 60000 ms |\n| Parallel catalog p99 | At most 2 seconds | 2000000 us |\n| Concurrent route p99 | At most 3 seconds | 3000000 us |\n| Route throughput | At least 100 assignments per second | 100.0/s |\n| Rust test process high-water RSS | At most 512 MiB | 524288 KiB |\n| PostgreSQL database growth | At most 1 GiB | 1073741824 bytes |\n\nThe setup timer includes seeding plus the version rollout, noisy-tenant case,\nand retry-storm case. It is broader than raw insert time.\n\nCatalog p99 is computed from 32 concurrent first-page reads at the default\ndataset. Route p99 and throughput use the 500 concurrent route tasks. The test\nuses its own integer percentile function over the collected samples; do not\nsubstitute a monitoring system's percentile method when comparing results.\n\nProcess RSS is the Rust test process high-water mark. It does not include\nPostgreSQL memory. Database growth is the change in pgdatabasesize for the\ncurrent database and can be distorted by unrelated concurrent work. Use an\nisolated database.\n\nIf the platform-specific RSS probe fails, the source records zero instead of\nfailing the test. Reject a memory-bound claim when\nprocessrsskibobservedmax is zero or otherwise implausible.\n\nVerify the functional invariants\n\nThe numeric thresholds are only part of the gate. The same run must also prove:\n\n| Invariant | Required behavior |\n\n| Version rollout | One instance moves to a new active version, its old replica drains, and a new ready replica serves the new version |\n| Noisy-tenant isolation | Of 64 simultaneous requests for a tenant with budget four, four are admitted and 60 are rejected while four healthy-tenant requests succeed |\n| Retry-storm isolation | Of 64 simultaneous retries with budget eight, eight are admitted and 56 are rejected while eight healthy requests succeed |\n| Catalog pagination | Page size never exceeds 50, revision and total remain stable, and every returned capability is unique |\n| Route distribution | Every eligible replica is used when the sample is large enough, and the largest and smallest assignment counts differ by at most three |\n| Reservation release | Settling all timed routes leaves zero active assignments in the run's replicas |\n| Retry fencing | Up to 100 repeated action IDs resolve to their original assignments |\n| Credential rotation | An existing route remains pinned to credential revision one while a new action uses revision two |\n| Indexed routing | The plan does not sequentially scan tenant bindings or replicas and uses an accepted binding index |\n| Lease churn | Up to 1,000 leases renew, expire, and transition through bounded batches |\n\nThese assertions make the test a mixed functional and performance regression\ngate. A threshold pass with a missing lifecycle assertion is not a valid\nresult.\n\nPrepare a disposable PostgreSQL database\n\nUse a database dedicated to this campaign. The test creates a unique UUIDv7\nrun identifier and prefixes its rows, then removes those rows after a successful\nresult. If the test fails or is interrupted before the final cleanup call, its\nrun-scoped rows may remain.\n\nFor that reason:\n• do not point the test at a production or shared development database;\n• take the database baseline before unrelated jobs start;\n• ensure the database user can install and use the registry schema;\n• record PostgreSQL version, maxconnections, sharedbuffers, storage\n  class, host resources, and connection pool settings;\n• discard or restore the dedicated database after a failed run instead of\n  guessing which partial rows are safe to delete.\n\nSet a descriptive environment label. It must contain 1 to 128 ASCII letters,\ndigits, -, , ., or /. If omitted, the result says\nlocal-unspecified, which is too weak for comparison evidence.\n\nRun the gate and retain its output\n\nFrom the exact source root, use a shell with pipeline failure propagation:\n\nexport AIPPOSTGRESSCALETESTURL='postgresql://user:password@host/database'\nexport AIPSCALEENVIRONMENTLABEL='dedicated-linux-amd64-postgres-17.6'\n\nset -o pipefail\ncargo test -p aip-connector-registry-postgres --test scale \\\n  indexedcatalogandroutingremainboundedatfleetscale \\\n  -- --nocapture 2>&1 | tee .aipd-scale-qualification.log\n\nUse a secret-file or CI secret mechanism for the real database URL. Do not\ncommit the command history, environment, or log if it contains credentials.\n\nExtract and validate the final result:\n\ntest \"$(grep -c '^AIPSCALERESULT=' .aipd-scale-qualification.log)\" -eq 1\n\nresult=$(sed -n 's/^AIPSCALERESULT=//p' \\\n  .aipd-scale-qualification.log | tail -n 1)\n\ntest -n \"$result\"\nprintf '%s\\n' \"$result\" | jq -e \\\n  '.slo.profile == \"aip-fleet-slo-v1\" and .slo.passed == true'\n\nReject the evidence if the result line is missing, duplicated, malformed, or\ndoes not match the expected source profile.\n\nRequire both a successful pipeline exit and the validated JSON. The source\nprints AIPSCALERESULT before deleting its run-scoped rows, so cleanup can\nstill fail after a JSON object says passed: true. The pipefail setting\npreserves that late test failure through tee.\n\nInspect the result fields\n\nThe final JSON records:\n• environment label, operating system, architecture, PostgreSQL version and\n  selected settings;\n• exact dataset counts;\n• threshold values embedded by the source;\n• catalog and routing p50, p95, and p99 samples;\n• routing throughput, errors, eligible replicas, and action count;\n• version rollout, tenant isolation, retry isolation, credential rotation, and\n  lease-churn outcomes;\n• observed process RSS, database sizes and growth, available parallelism, and\n  connection pool snapshot.\n\nRetain the full combined log, parsed JSON, source commit, dependency lockfile,\ndatabase image or package identity, database configuration, host inventory,\nstart and finish times, and cryptographic digests for the evidence files.\n\nThe log also contains the routing query plan. Review it even when the summary\npasses, because it explains which index path produced the measured result.\n\nCompare runs honestly\n\nCompare two results only when their source, dataset, PostgreSQL major version,\ndatabase settings, host resources, architecture, storage class, and contention\nare equivalent. Otherwise describe the difference as an environment\nobservation rather than a regression.\n\nUse repeated runs and distribution-aware production telemetry for capacity\nplanning. One 32-sample catalog percentile and one 500-route burst are useful\nCI guards, not a full latency study.\n\nThe reviewed GitHub and Gitea CI definitions use PostgreSQL 17.6, provide an\nexplicit environment label, retain the combined log, and declare a 30-day\nartifact lifetime. Copy evidence required for a release decision before it\nexpires.\n\nState the result narrowly\n\nA valid claim is:\n\nThe identified source passed aip-fleet-slo-v1 with the recorded dataset and\nPostgreSQL environment at the stated time.\n\nThe result does not qualify connector processes, external products, network\ntransports, multi-host behavior, application execution, arbitrary fleet sizes,\nor production capacity. Combine it with the\nconnector fleet failure matrix (connector-fleet-qualification.md) and\ndeployment-specific load, recovery, and observability campaigns.\n\nRelated documentation\n• Testing and evidence index (README.md)\n• Registry and routing architecture (../architecture/connector-registry-and-routing.md)\n• Metrics reference (../reference/metrics.md)\n• Connector fleet qualification (connector-fleet-qualification.md)\n• Conformance and qualification (../reference/conformance.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "addd5727125f86b535692f968c8f007c57b0dde9ca6749024467c65db2724b3b"
  }
}