Live Product End-to-End Tests

Contract tests and live product tests provide different evidence and are kept separate:

• connector unit tests validate HTTP payloads, authentication headers, cancellation, streaming, webhook verification, and retry declarations against deterministic protocol peers;
• orchestration contract tests validate native AIP lifecycle behavior without claiming that an external product was deployed;
• the ignored live suite invokes real Dify, CrewAI sidecar, and Chatwoot endpoints through an AIP Gateway and performs customer-visible writes.

Required Environment

The live suite requires dedicated non-production tenants and these variables:

Variable	Purpose
AIP_E2E_DIFY_URL	Dify application API base URL
AIP_E2E_DIFY_API_KEY	Dify application API credential
AIP_E2E_DIFY_APP_ID	Dify application identifier exposed by the connector
AIP_E2E_DIFY_APP_MODE	Dify mode, such as workflow or chat
AIP_E2E_DIFY_INPUT_JSON	Optional complete Dify test input
AIP_E2E_CREWAI_URL	CrewAI sidecar base URL
AIP_E2E_CREWAI_CREW_ID	Crew identifier configured in the sidecar
AIP_E2E_CREWAI_BEARER_TOKEN	Optional sidecar bearer token; required when sidecar auth is enabled
AIP_E2E_CHATWOOT_URL	Chatwoot base URL
AIP_E2E_CHATWOOT_ACCOUNT_ID	Dedicated Chatwoot test account
AIP_E2E_CHATWOOT_TOKEN	Chatwoot API access token
AIP_E2E_CHATWOOT_CONVERSATION_ID	Disposable conversation receiving the private note
AIP_E2E_CAL_DIY_URL	Isolated Cal.diy API base URL, including /api when required
AIP_E2E_CAL_DIY_API_KEY	Dedicated Cal.diy API key or access token
AIP_E2E_CAL_DIY_EVENT_TYPE_ID	Event type used for reversible slot reservation
AIP_E2E_CAL_DIY_SLOT_START	Currently available future slot in ISO 8601 format
AIP_E2E_CAL_DIY_SLOT_DURATION_MINUTES	Optional slot duration, default 30 minutes

Run only the ignored live test:

cargo test -p aip-tests --test e2e_chatwoot_dify_crewai \
  live_chatwoot_dify_crewai_workflow_runs_through_aip_gateway \
  -- --ignored --exact --nocapture

The test fails when a required variable is absent, a product health probe is not ready, approval state is not persisted, or any product operation does not complete. A normal workspace test run reports this test as ignored; it must not be cited as live-product evidence unless the explicit command above completed against identified product instances.

Safety

Use disposable data. The suite invokes a Dify application, executes a CrewAI crew, and posts a private Chatwoot message. Credentials must come from the environment or a deployment secret store and must never be committed to the repository or emitted in test logs.

Exact-Upstream Cal.diy Qualification

The release qualification is a repository-owned, isolated Docker campaign. It admits the exact clean Cal.diy source revision and matching image, creates fresh dedicated AIP and Cal.diy databases, and exercises both native AIP and MCP Streamable HTTP through aipd:

examples/cal-diy-qualification/qualify.sh

The driver fails unless it proves all of these boundaries: verified tenant/account and opaque credential-handle routing; 81 admitted capabilities; real slot discovery; plan validation without mutation; an independent authenticated approval decision; exactly one booking after approval resume; exact idempotent replay; signed webhook acceptance, duplicate suppression, invalid-HMAC rejection, and timestamp-skew rejection; durable action, transaction, event, and provider facts after aipd restart.

Runtime secrets and raw evidence are written only under the ignored examples/cal-diy-qualification/.runtime/ directory. The redacted retained record is cal-diy-isolated-live.md, with its machine-readable artifact linked from that report. A passing deterministic provider fixture or a skipped ignored test is not live Cal.diy evidence.

The older environment-driven ignored test remains available as a narrow manual probe for an operator-owned Cal.diy deployment. It verifies authenticated reservation/replay/release behavior but does not replace the complete release qualification:

AIP_E2E_CAL_DIY_URL=http://127.0.0.1:5555/api \
AIP_E2E_CAL_DIY_API_KEY="$CAL_DIY_TEST_API_KEY" \
AIP_E2E_CAL_DIY_EVENT_TYPE_ID=42 \
AIP_E2E_CAL_DIY_SLOT_START=2050-01-01T10:00:00Z \
cargo test -p aip-connector-cal-diy \
  live_cal_diy_reservation_round_trip_uses_the_aip_connector \
  -- --ignored --exact --nocapture

Exact-Upstream CrewAI Qualification

The CrewAI connector can be qualified independently before the three-product workflow is available. Build or install CrewAI from the exact revision in docs/connectors/upstream-baselines.md, start the qualification sidecar, and run:

AIP_E2E_CREWAI_URL=http://127.0.0.1:8090 \
AIP_E2E_CREWAI_CREW_ID=support-qualification \
AIP_E2E_CREWAI_BEARER_TOKEN="$AIP_CREWAI_BEARER_TOKEN" \
cargo test -p aip-connector-crewai \
  exact_upstream_sidecar_executes_a_real_crewai_crew \
  -- --ignored --nocapture

This test is product evidence only when the sidecar was built from the recorded upstream revision. It asserts a real terminal CrewOutput and token usage over the Rust connector’s authenticated HTTP boundary. Unit tests remain the authoritative deterministic checks for stream ordering, cancellation, and restart fencing.