Metrics reference
AIP exposes process health, work capacity, fleet state, and bounded activity counters through three different metric surfaces. This reference is for operators who need to select the correct endpoint, recognize every current product-neutral family, and interpret an absent or reset series safely.
This inventory describes source revision
97be86e9efedf07ecf1783b03800f683f107fb04. It covers aipd, the shared host
used by Cal.diy, Hermes Agent, Chatwoot, Dify, CrewAI, and Twenty, and the
connector lifecycle control plane. It does not invent provider-specific
business metrics.
Surface selector
| Process | Path | Format | Current inventory |
|---|---|---|---|
aipd |
GET /metrics |
Prometheus text, version 0.0.4 | 37 possible families: 34 direct daemon plus 3 embedded gateway |
| Connector host | GET /metrics |
Prometheus text, version 0.0.4 | 8 fixed families |
| Connector control plane | GET /metrics |
JSON | 6 connection-pool fields |
The three paths have the same name but not the same representation. Do not configure a Prometheus text parser against the control-plane JSON endpoint.
Stable names versus current emission
The observability crate declares 11 stable metric names for AIP implementations. A declaration fixes a shared name and meaning; it does not register or emit a series.
| Stable name | Declared meaning | Emitted at this revision |
|---|---|---|
aip_envelopes_received_total |
Envelopes received by a gateway | Yes, by the embedded gateway |
aip_envelopes_sent_total |
Envelopes sent by a gateway | No |
aip_actions_total |
Action invocations | Yes, by the embedded gateway |
aip_delegations_total |
Delegation requests | No |
aip_delegation_duration_ms |
Delegation latency in milliseconds | No |
aip_action_duration_ms |
Action latency in milliseconds | No |
aip_connector_invocations_total |
Connector invocations | No |
aip_connector_duration_ms |
Connector latency in milliseconds | No |
aip_errors_total |
Protocol errors | No |
aip_sessions_active |
Active sessions | No |
aip_stream_chunks_total |
Emitted stream chunks | No |
Only the two rows marked Yes are emitted by the pinned product-neutral
processes. Do not create alerts or dashboards for the other nine until an
implementation registers them. No product-neutral histogram is emitted at
this revision.
aipd Prometheus families
Readiness and native traffic
| Metric | Type | Updated by | Meaning |
|---|---|---|---|
aip_daemon_ready |
Gauge | Daemon /ready request |
Composite daemon readiness: gateway, worker, required NATS listener, and required modules |
aip_gateway_ready |
Gauge | Gateway readiness evaluation | Gateway storage and registered connector readiness |
aip_runtime_worker_up |
Gauge | Worker lifecycle and daemon /ready |
Whether the durable background worker is running |
aip_nats_listener_up |
Gauge | NATS listener lifecycle and daemon /ready |
Whether the native NATS listener is running; readiness requires it only when configured as required |
aip_envelopes_received_total |
Counter | Gateway envelope entry | Envelope attempts received before policy completion, labeled by message_type |
aip_actions_total |
Counter | Gateway envelope entry | Received envelopes whose body is an action, before policy completion |
aip_actions_total is not a success counter. Pair it with durable action state
and error evidence when calculating outcomes.
Fleet maintenance and registry pools
| Metric | Type | Updated by | Meaning |
|---|---|---|---|
aip_connector_fleet_ready_replicas |
Gauge | Successful fleet maintenance cycle | Ready replicas in the current fleet summary |
aip_connector_fleet_active_assignments |
Gauge | Successful fleet maintenance cycle | Current active route assignments |
aip_connector_fleet_expired_replicas_total |
Counter | Successful fleet maintenance cycle | Replicas expired by maintenance since this process started |
aip_connector_fleet_maintenance_failures_total |
Counter | Failed fleet maintenance cycle | Expiry or fleet-status cycle failures since process start |
aip_connector_registry_control_pool_connections |
Gauge | Successful cycle with a pool snapshot | Open control-plane database connections |
aip_connector_registry_control_pool_idle_connections |
Gauge | Successful cycle with a pool snapshot | Idle control-plane database connections |
aip_connector_registry_control_pool_max_connections |
Gauge | Successful cycle with a pool snapshot | Configured control-plane connection ceiling |
aip_connector_registry_data_pool_connections |
Gauge | Successful cycle with a pool snapshot | Open data-plane database connections |
aip_connector_registry_data_pool_idle_connections |
Gauge | Successful cycle with a pool snapshot | Idle data-plane database connections |
aip_connector_registry_data_pool_max_connections |
Gauge | Successful cycle with a pool snapshot | Configured data-plane connection ceiling |
The six pool gauges describe the registry owned by aipd; they do not describe
the separate lifecycle control-plane process.
Remote admission and event ingress
The remote admission gauges are refreshed immediately before aipd renders a
metrics scrape, but only when the scheduler is configured.
| Metric | Type | Meaning |
|---|---|---|
aip_connector_remote_active |
Gauge | Active admitted remote actions |
aip_connector_remote_queued |
Gauge | Queued remote actions |
aip_connector_remote_queued_bytes |
Gauge | Serialized bytes retained in the admission queue |
aip_connector_remote_active_tenants |
Gauge | Tenant partitions with active remote work |
aip_connector_remote_queued_tenants |
Gauge | Tenant partitions with queued remote work |
Event-ingress capacity gauges are also refreshed at scrape time when central connector event ingress is configured.
| Metric | Type | Meaning |
|---|---|---|
aip_connector_event_ingress_capacity |
Gauge | Configured maximum concurrent ingress requests |
aip_connector_event_ingress_available |
Gauge | Currently available ingress permits |
aip_connector_event_ingress_received_total |
Counter | Requests reaching either connector-events or connector-callbacks handler |
aip_connector_event_ingress_accepted_total |
Counter | Ingress envelopes accepted by the shared handler |
aip_connector_event_ingress_rejected_total |
Counter | Requests rejected for missing configuration, schema, decode, trust, route, capacity, or durable handling failure |
Both ingress URL paths use the same counters. The metrics do not separate connector events from stream callbacks and do not carry connector, replica, or tenant labels.
Runtime work budgets and retention
| Metric | Type | Updated by | Meaning |
|---|---|---|---|
aip_runtime_callback_capacity |
Gauge | Worker cycle | Maximum concurrent callback-delivery work |
aip_runtime_callback_available |
Gauge | Worker cycle | Available callback-delivery permits |
aip_runtime_reconciliation_capacity |
Gauge | Worker cycle | Maximum concurrent reconciliation work |
aip_runtime_reconciliation_available |
Gauge | Worker cycle | Available reconciliation permits |
aip_runtime_worker_cycles_total |
Counter | Completed worker cycle | Worker cycles completed since process start |
aip_runtime_worker_claimed_actions_total |
Counter | Completed worker cycle | Durable queued actions claimed by this process |
aip_retention_events_deleted_total |
Counter | Hourly retention pass | Event records deleted since process start |
aip_retention_replay_claims_deleted_total |
Counter | Hourly retention pass | Replay claims deleted since process start |
aip_retention_dead_letters_deleted_total |
Counter | Hourly retention pass | Dead-letter records deleted since process start |
aip_retention_callback_deliveries_deleted_total |
Counter | Hourly retention pass | Callback-delivery records deleted since process start |
aip_retention_outbox_records_deleted_total |
Counter | Hourly retention pass | Outbox records deleted since process start |
The capacity gauges express concurrency permits, not queue depth. A value of zero available with nonzero capacity means the corresponding work class is saturated at that sample.
Connector-host Prometheus families
Every one of the six standalone connector hosts renders all eight families on each scrape. The endpoint uses local atomics and has no metric labels.
| Metric | Type | Meaning |
|---|---|---|
aip_connector_host_ready |
Gauge | 1 only when the host is not draining, its lease is valid, storage is ready, and the connector is ready |
aip_connector_host_storage_ready |
Gauge | Last known durable runtime-storage readiness |
aip_connector_host_in_flight |
Gauge | Action requests currently executing inside this host |
aip_connector_host_requests_total |
Counter | Native message requests received by this process |
aip_connector_host_rejected_total |
Counter | Requests rejected by the instrumented schema, trust, route, readiness, capacity, credential, approval, or gateway paths |
aip_connector_host_heartbeat_failures_total |
Counter | Lease renewal or recovery attempts that failed from the heartbeat loop |
aip_connector_host_lease_recovery_attempts_total |
Counter | Attempts to re-register after a lease is no longer valid |
aip_connector_host_lease_recoveries_total |
Counter | Lease recoveries that successfully applied a new lease |
requests_total - rejected_total is not a supported success calculation.
Response-signing and process failures can occur outside a counter update, and
accepted action state is owned by the durable lifecycle rather than this
arithmetic.
Control-plane JSON snapshot
The connector lifecycle control plane returns a ConnectorRegistryPoolSnapshot
as JSON:
{
"control_size": 1,
"control_idle": 1,
"control_max": 4,
"data_size": 1,
"data_idle": 1,
"data_max": 4
}
| Field | Meaning |
|---|---|
control_size |
Open registry administrator/control connections |
control_idle |
Idle control connections |
control_max |
Configured control connection ceiling |
data_size |
Open lifecycle data connections |
data_idle |
Idle lifecycle data connections |
data_max |
Configured data connection ceiling |
These values are a point-in-time JSON diagnostic. A Prometheus collector must transform them explicitly or use a separate JSON exporter; they are not native Prometheus series.
Labels and cardinality
The current product-neutral Prometheus emitters use one metric label:
message_type on aip_envelopes_received_total. Its value comes from the
bounded AIP message-type enum. Every other current daemon and host family is
unlabeled.
Stable tracing fields such as action ID, message ID, session ID, correlation ID, delegation ID, connector ID, and error code are not metric labels. Keep those values in traces and structured logs. Do not add tenant, action, resource, provider request, free-form message, or replica identifiers as Prometheus labels.
Use target metadata outside the application to distinguish processes, regions, connector types, and immutable deployment artifacts. Keep the target-label set bounded and controlled by the monitoring system.
Series activation and reset behavior
Prometheus recorder families are registered lazily when code updates them. Consequently:
aip_daemon_readyandaip_gateway_readymay be absent until readiness has been evaluated;- remote admission and ingress gauges are absent when their optional subsystem is not configured;
- fleet and pool gauges are absent until a successful maintenance cycle, and pool gauges also require a pool snapshot;
- traffic, failure, and retention counters may be absent until their first update;
- the nine stable names with no emitter remain absent for the entire process.
Absence is not zero. Monitor the scrape target and the corresponding health or configuration boundary separately. All counters and gauges are process-local; they reset when the process restarts. Use rate or increase semantics for counters and tolerate a reset.
Operational interpretation
| Signal | Combine with | Interpretation |
|---|---|---|
Daemon ready is 0 |
/ready body and required-module state |
One or more readiness prerequisites failed |
Runtime worker up is 0 |
Worker error and durable queue state | Background recovery, callbacks, retention, and queued execution may not progress |
NATS listener up is 0 |
Deployment configuration | Incident only when NATS is required for this deployment |
| Ready replicas fall | Fleet maintenance status, leases, and host readiness | Routing capacity is shrinking; do not infer a provider outage from this gauge alone |
| Active assignments remain high while ready replicas fall | Drain and lease state | Existing pinned work may be preventing safe removal |
Available admission or ingress permits reach 0 |
Queue gauges and rejection rate | A bounded work class is saturated |
| Event rejected counter rises | Structured error code and ingress logs | Diagnose schema, trust, route, capacity, or storage; the counter has no reason label |
| Heartbeat failures rise | Host readiness, lease expiry, and control-plane health | Lease renewal or recovery is failing |
| Recovery attempts rise without recoveries | Control-plane error and immutable registration | Recovery cannot obtain and apply a valid lease |
| Retention counters stop changing | Worker cycles and current retained volume | May be normal when nothing is eligible; inspect durable state before alerting |
Deployment-owned service objectives determine alert thresholds and windows. This source defines signals and semantics, not universal thresholds.
Endpoint security
At this revision, each metrics path is mounted outside the process’s native application-auth middleware. Keep it on a private operations network or place it behind authenticated, authorized, and encrypted monitoring ingress.
The current families deliberately avoid tenant and resource identifiers, but they still reveal capacity, readiness, connection-pool, traffic, and failure information. Do not expose them as public diagnostics.