{
  "schemaVersion": "1.0",
  "title": "Operate and recover the Dify connector",
  "description": "Use this runbook to locate a Dify connector failure before changing state. Inspect process, readiness, lease, storage, credential, Action, and provider evidence in that order.",
  "canonical": "https://getaip.org/docs/connectors/dify/operations/health-observability-and-recovery",
  "route": "/docs/connectors/dify/operations/health-observability-and-recovery",
  "source": "docs/connectors/dify/operations/health-observability-and-recovery.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/dify/operations/health-observability-and-recovery.md",
    "txt": "/docs/download/connectors/dify/operations/health-observability-and-recovery.txt",
    "json": "/docs/download/connectors/dify/operations/health-observability-and-recovery.json",
    "pdf": "/docs/download/connectors/dify/operations/health-observability-and-recovery.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Operate and recover the Dify connector\ndescription: >-\n  Diagnose Dify health, leases, storage, credentials, streams, task state, and\n  uncertain provider outcomes without unsafe replay\nkind: runbook\naudience: operator\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: dify\naliases:\n  - Dify connector health\n  - Dify connector recovery\n---\n\n# Operate and recover the Dify connector\n\nUse this runbook to locate a Dify connector failure before changing state.\nInspect process, readiness, lease, storage, credential, Action, and provider\nevidence in that order.\n\nWhen a mutation has an uncertain provider outcome, stop related mutations.\nReconcile durable Action and provider state before any retry or corrective\naction.\n\n## Read signals in order\n\n| Order | Surface | Primary fact |\n|---:|---|---|\n| 1 | `GET /health` | The host HTTP process responds |\n| 2 | `GET /ready` | This replica can accept routed Actions now |\n| 3 | `GET /metrics` | Pressure, rejection, or lease recovery is changing |\n| 4 | Connector registry | Version, route, replica, and lease are authoritative |\n| 5 | Durable Action and profile state | Work and remote task identity survive locally |\n| 6 | Dify provider reads | The configured app or workspace state exists |\n| 7 | Stream or cancellation evidence | Completion or remote-stop certainty is known |\n\nDo not use a later signal to overwrite an earlier fact. A live process can be\nnot ready, and a ready replica cannot determine an older mutation outcome.\n\n## Treat liveness as process-only\n\n`GET /health` returns HTTP `200` while the host router can serve:\n\n```json\n{\n  \"status\": \"ok\",\n  \"protocol\": \"AIP\",\n  \"version\": \"1.0\",\n  \"service\": \"aip-connector-host\"\n}\n```\n\nThis route does not probe PostgreSQL, Dify, credentials, admission, lease,\ntask correlation, or stream publication.\n\n## Read composite readiness\n\n`GET /ready` runs a bounded storage check and live connector health probe. It\nalso reads the drain flag and current lease expiry.\n\nHTTP `200` requires:\n\n- the replica is not draining;\n- the lifecycle lease is valid;\n- durable runtime storage is readable and writable;\n- every configured Dify credential passes its provider probe.\n\nAny false condition returns HTTP `503`. Inspect `draining`, `lease_valid`,\n`runtime.ready`, and `connector.ready` independently.\n\nThe `runtime.recovery` object describes bounded startup recovery. It is not a\nlive task-store, approval, callback, or provider-work backlog.\n\n## Interpret Dify product health\n\nThe connector probes every configured credential:\n\n| Credential | Provider request |\n|---|---|\n| App | `GET /v1/parameters` |\n| Knowledge | `GET /v1/datasets?limit=1` |\n\nChecks run with concurrency bounded at 32. One failed credential makes the\nwhole connector health probe fail.\n\nThe common host bounds the complete probe by its configured health timeout,\nwhich defaults to 2,000 milliseconds.\n\nA ready result proves route reachability and bearer-key acceptance for all\nconfigured descriptors at that moment. It does not verify app mode, mutation\npermissions, streaming, cancellation, indexing, or pipeline execution.\n\nThe healthy detail reports app and Knowledge credential counts. A failure does\nnot expose a structured per-credential status list.\n\n## Use host metrics\n\n`GET /metrics` renders eight Prometheus families:\n\n| Metric | Type | Operational meaning |\n|---|---|---|\n| `aip_connector_host_ready` | Gauge | Cached provider and storage flags plus live lease and drain state |\n| `aip_connector_host_storage_ready` | Gauge | Last stored durable-storage probe result |\n| `aip_connector_host_in_flight` | Gauge | Actions executing in this process |\n| `aip_connector_host_requests_total` | Counter | Native message requests received |\n| `aip_connector_host_rejected_total` | Counter | Instrumented host rejections |\n| `aip_connector_host_heartbeat_failures_total` | Counter | Failed lease renewal or recovery attempts |\n| `aip_connector_host_lease_recovery_attempts_total` | Counter | Re-registration attempts after expiry |\n| `aip_connector_host_lease_recoveries_total` | Counter | Re-registrations that applied a new lease |\n\nThe metrics have no application labels and reset on process restart. Add\nbounded scrape-target metadata for replica, version, zone, and connector type.\n\nThe ready gauge does not run a provider probe. `/ready` and the heartbeat loop\nupdate cached flags.\n\n## Account for observability gaps\n\nThe reviewed host has no public Dify-specific metric for:\n\n- provider route, status, or latency distribution;\n- individual credential health;\n- response-bound or invalid-stream failures;\n- emitted SSE events or callback backlog;\n- active durable task mappings;\n- cancellation attempts or remote-stop confirmations;\n- Dify indexing, workflow, model, or pipeline state.\n\nSome successful operation envelopes include Dify `x-request-id`. Typed\nconnector failures currently leave `provider_request_id` empty.\n\nRetain the AIP Action ID and any output request ID before redaction. Do not\ninfer an empty task store or healthy provider route from a normal scrape.\n\n## Understand lease behavior\n\nThe heartbeat loop probes Dify and durable storage, records readiness flags,\nreports in-flight work, and renews or recovers the replica lease.\n\nAn unhealthy heartbeat moves the registry replica offline. A later healthy\ncycle can restore it under a valid lease or re-register after expiry.\n\nLease recovery preserves one request ID across ambiguous control-plane\nfailures. Definitive admission or configuration rejection clears that pending\nrequest before a corrected attempt.\n\nRising heartbeat failures without recovery indicate control-plane, identity,\nsequence, admission, storage, provider, or network failure. They do not isolate\nthe Dify boundary by themselves.\n\n## Interpret Dify failure codes\n\n| Code | Category | Retry rule | Uncertain mutation condition |\n|---|---|---|---|\n| `connector.dify.authentication` | Auth | Never | No |\n| `connector.dify.remote_temporary` | Temporary | Safe reads only | Invocation mutation |\n| `connector.dify.remote_rejected` | Permanent | Never | No |\n| `connector.dify.transport` | Transport | Safe reads only | Mutation after connection began |\n| `connector.dify.invalid_stream` | Connector | Safe reads only | Invocation mutation |\n| `connector.dify.response_too_large` | Permanent | Never | Invocation mutation |\n| `connector.dify.cancelled` | Temporary | Never | Mutation without confirmed remote stop |\n| `connector.dify.task_store` | Temporary | Never | Invocation mutation |\n| `connector.dify.invalid_credential` | Auth | Never | No |\n| `connector.dify.configuration_or_input` | Permanent | Never | No |\n\nTemporary classification does not make a mutation replay-safe. The catalogue's\nread safety controls the `retryable` field for temporary, transport, and\ninvalid-stream failures.\n\n## Contain and capture evidence\n\nDrain or remove the route when identity, storage, provider origin, credential,\nmanifest, or task-state ownership is uncertain.\n\nRetain:\n\n- host, version, instance, replica, tenant, and external-account identity;\n- timestamped `/health`, `/ready`, and `/metrics` observations;\n- registry route, status, lease sequence, and expiry;\n- configuration and credential revisions without secret bytes;\n- Action ID, capability, input hash, approval, and idempotency-key hash;\n- Dify status, event name, task ID, and request ID when available;\n- final provider reads used for reconciliation.\n\nNever retain API keys, signing seeds, database URLs, or unrestricted customer\npayloads in incident records.\n\n## Classify the failing boundary\n\n| Observation | Likely boundary | Next evidence |\n|---|---|---|\n| `/health` unavailable | Process, listener, edge, or network | Supervisor and bounded startup error |\n| `/health` is `200`, `/ready` is `503` | Drain, lease, storage, or Dify health | Individual readiness fields |\n| `runtime.ready` is false | PostgreSQL, schema, role, or timeout | Durable storage probe and migration state |\n| `connector.ready` is false | Origin, TLS, credentials, or Dify | Descriptor set and bounded health error |\n| `lease_valid` is false | Heartbeat, control plane, sequence, or admission | Registry and recovery counters |\n| In-flight equals capacity | Local saturation | Action duration and route pressure |\n| Rejections increase | Host validation, trust, readiness, approval, or capacity | Signed protocol error code |\n| `invalid_stream` appears | Provider SSE, bounds, or callback emission | Last valid event and cleanup certainty |\n| `task_store` appears | Profile-state durability | Storage health and action mapping ownership |\n| Cancellation is uncertain | Missing task identity or failed stop | Durable mapping and provider task state |\n\n## Recover storage and task state\n\n1. Keep the replica out of routing.\n2. Verify the configured database URL file, role, schema, and reachability.\n3. Preserve durable Actions, idempotency records, and profile state.\n4. Restore one coherent storage authority for all instance replicas.\n5. Inspect bounded startup recovery and task-store access.\n6. Require `runtime.ready: true` before registration or routing.\n7. Reconcile every mutation whose dispatch boundary is unknown.\n\nDo not delete task mappings or idempotency records to force readiness.\n\n## Recover provider health or credentials\n\n1. Confirm the fixed Dify origin and descriptor hashes.\n2. Identify the failing credential revision without printing its key.\n3. Distinguish TLS, DNS, timeout, `401`, `403`, rate limit, and server failure.\n4. Rotate the owner-only key file through a replacement replica when required.\n5. Re-run readiness and the matching read-only canary.\n6. Reconcile earlier mutations before enabling writes.\n\nProduct keys and descriptors are startup-owned. Replacing file bytes does not\nprove that a running process adopted them.\n\n## Recover lease or control-plane state\n\n1. Compare local instance, replica, version, manifest, artifact, and sequence\n   with the registry.\n2. Verify the fixed control endpoint, trusted DID, TLS, and registry state.\n3. Preserve a pending recovery request across ambiguous transport failure.\n4. Correct definitive admission or configuration rejection first.\n5. Require a valid ready lease and correct route before traffic resumes.\n\nNever start a second live process with the same replica ID while lease or\npinned Action ownership remains unresolved.\n\n## Recover capacity pressure\n\n`connector_host.capacity` is returned before provider dispatch and is marked\nretryable by the host. Confirm durable Action state before repeating the same\ncorrelated request.\n\nReduce pressure or add an admitted replica before raising the in-flight limit.\nA larger limit does not increase Dify capacity or preserve latency.\n\n## Recover stream failure\n\nPreserve the last valid event, task mapping, response-byte count, and cleanup\nresult. A malformed or oversized stream can occur after Dify admitted a task.\n\nIf a durable task ID exists, verify the mode-specific stop result. If no task\nID was observed, retain an uncertain outcome and reconcile provider state.\n\nDo not convert premature close into completion or replay the model execution\nunder a new Action ID.\n\n## Recover an uncertain mutation\n\n1. Preserve the original Action ID, key, approval, and canonical input.\n2. Read durable Action and task-correlation state.\n3. Query the authoritative provider resource when an operation exposes one.\n4. Correlate Dify request, task, workflow-run, document, or batch identity.\n5. Accept matching state, retain ambiguity, or issue a separately approved\n   corrective intent.\n\nAn idempotency key fences AIP replay identity. It does not prove provider-side\ndeduplication or absence of effects.\n\n## Verify recovery\n\nRequire:\n\n- `/health` is reachable;\n- `/ready` returns `200` with valid lease, no drain, and both probes ready;\n- registry identity and route match the intended replica;\n- ready and storage gauges agree with the latest readiness probe;\n- recovery counters stop increasing unexpectedly;\n- exact manifest counts and read-only canaries pass;\n- task-store access and streaming callback ownership are intact;\n- every uncertain mutation has an explicit reconciliation decision;\n- no durable state or secret was deleted to obtain recovery.\n\nObserve at least one lease renewal and a normal provider window. These checks\nestablish operational recovery, not complete capability qualification.\n\n## Related documentation\n\n- [Deploy the Dify connector](deployment.md)\n- [Dify connector configuration](../reference/configuration.md)\n- [Dify streaming, cancellation, and binary responses](../reference/streaming-cancellation-and-binary-responses.md)\n- [Observe and recover](../../../guides/observe-and-recover.md)\n- [Metrics reference](../../../reference/metrics.md)\n",
    "text": "Operate and recover the Dify connector\n\nUse this runbook to locate a Dify connector failure before changing state.\nInspect process, readiness, lease, storage, credential, Action, and provider\nevidence in that order.\n\nWhen a mutation has an uncertain provider outcome, stop related mutations.\nReconcile durable Action and provider state before any retry or corrective\naction.\n\nRead signals in order\n\n| Order | Surface | Primary fact |\n\n| 1 | GET /health | The host HTTP process responds |\n| 2 | GET /ready | This replica can accept routed Actions now |\n| 3 | GET /metrics | Pressure, rejection, or lease recovery is changing |\n| 4 | Connector registry | Version, route, replica, and lease are authoritative |\n| 5 | Durable Action and profile state | Work and remote task identity survive locally |\n| 6 | Dify provider reads | The configured app or workspace state exists |\n| 7 | Stream or cancellation evidence | Completion or remote-stop certainty is known |\n\nDo not use a later signal to overwrite an earlier fact. A live process can be\nnot ready, and a ready replica cannot determine an older mutation outcome.\n\nTreat liveness as process-only\n\nGET /health returns HTTP 200 while the host router can serve:\n\n{\n  \"status\": \"ok\",\n  \"protocol\": \"AIP\",\n  \"version\": \"1.0\",\n  \"service\": \"aip-connector-host\"\n}\n\nThis route does not probe PostgreSQL, Dify, credentials, admission, lease,\ntask correlation, or stream publication.\n\nRead composite readiness\n\nGET /ready runs a bounded storage check and live connector health probe. It\nalso reads the drain flag and current lease expiry.\n\nHTTP 200 requires:\n• the replica is not draining;\n• the lifecycle lease is valid;\n• durable runtime storage is readable and writable;\n• every configured Dify credential passes its provider probe.\n\nAny false condition returns HTTP 503. Inspect draining, leasevalid,\nruntime.ready, and connector.ready independently.\n\nThe runtime.recovery object describes bounded startup recovery. It is not a\nlive task-store, approval, callback, or provider-work backlog.\n\nInterpret Dify product health\n\nThe connector probes every configured credential:\n\n| Credential | Provider request |\n\n| App | GET /v1/parameters |\n| Knowledge | GET /v1/datasets?limit=1 |\n\nChecks run with concurrency bounded at 32. One failed credential makes the\nwhole connector health probe fail.\n\nThe common host bounds the complete probe by its configured health timeout,\nwhich defaults to 2,000 milliseconds.\n\nA ready result proves route reachability and bearer-key acceptance for all\nconfigured descriptors at that moment. It does not verify app mode, mutation\npermissions, streaming, cancellation, indexing, or pipeline execution.\n\nThe healthy detail reports app and Knowledge credential counts. A failure does\nnot expose a structured per-credential status list.\n\nUse host metrics\n\nGET /metrics renders eight Prometheus families:\n\n| Metric | Type | Operational meaning |\n\n| aipconnectorhostready | Gauge | Cached provider and storage flags plus live lease and drain state |\n| aipconnectorhoststorageready | Gauge | Last stored durable-storage probe result |\n| aipconnectorhostinflight | Gauge | Actions executing in this process |\n| aipconnectorhostrequeststotal | Counter | Native message requests received |\n| aipconnectorhostrejectedtotal | Counter | Instrumented host rejections |\n| aipconnectorhostheartbeatfailurestotal | Counter | Failed lease renewal or recovery attempts |\n| aipconnectorhostleaserecoveryattemptstotal | Counter | Re-registration attempts after expiry |\n| aipconnectorhostleaserecoveriestotal | Counter | Re-registrations that applied a new lease |\n\nThe metrics have no application labels and reset on process restart. Add\nbounded scrape-target metadata for replica, version, zone, and connector type.\n\nThe ready gauge does not run a provider probe. /ready and the heartbeat loop\nupdate cached flags.\n\nAccount for observability gaps\n\nThe reviewed host has no public Dify-specific metric for:\n• provider route, status, or latency distribution;\n• individual credential health;\n• response-bound or invalid-stream failures;\n• emitted SSE events or callback backlog;\n• active durable task mappings;\n• cancellation attempts or remote-stop confirmations;\n• Dify indexing, workflow, model, or pipeline state.\n\nSome successful operation envelopes include Dify x-request-id. Typed\nconnector failures currently leave providerrequestid empty.\n\nRetain the AIP Action ID and any output request ID before redaction. Do not\ninfer an empty task store or healthy provider route from a normal scrape.\n\nUnderstand lease behavior\n\nThe heartbeat loop probes Dify and durable storage, records readiness flags,\nreports in-flight work, and renews or recovers the replica lease.\n\nAn unhealthy heartbeat moves the registry replica offline. A later healthy\ncycle can restore it under a valid lease or re-register after expiry.\n\nLease recovery preserves one request ID across ambiguous control-plane\nfailures. Definitive admission or configuration rejection clears that pending\nrequest before a corrected attempt.\n\nRising heartbeat failures without recovery indicate control-plane, identity,\nsequence, admission, storage, provider, or network failure. They do not isolate\nthe Dify boundary by themselves.\n\nInterpret Dify failure codes\n\n| Code | Category | Retry rule | Uncertain mutation condition |\n\n| connector.dify.authentication | Auth | Never | No |\n| connector.dify.remotetemporary | Temporary | Safe reads only | Invocation mutation |\n| connector.dify.remoterejected | Permanent | Never | No |\n| connector.dify.transport | Transport | Safe reads only | Mutation after connection began |\n| connector.dify.invalidstream | Connector | Safe reads only | Invocation mutation |\n| connector.dify.responsetoolarge | Permanent | Never | Invocation mutation |\n| connector.dify.cancelled | Temporary | Never | Mutation without confirmed remote stop |\n| connector.dify.taskstore | Temporary | Never | Invocation mutation |\n| connector.dify.invalidcredential | Auth | Never | No |\n| connector.dify.configurationorinput | Permanent | Never | No |\n\nTemporary classification does not make a mutation replay-safe. The catalogue's\nread safety controls the retryable field for temporary, transport, and\ninvalid-stream failures.\n\nContain and capture evidence\n\nDrain or remove the route when identity, storage, provider origin, credential,\nmanifest, or task-state ownership is uncertain.\n\nRetain:\n• host, version, instance, replica, tenant, and external-account identity;\n• timestamped /health, /ready, and /metrics observations;\n• registry route, status, lease sequence, and expiry;\n• configuration and credential revisions without secret bytes;\n• Action ID, capability, input hash, approval, and idempotency-key hash;\n• Dify status, event name, task ID, and request ID when available;\n• final provider reads used for reconciliation.\n\nNever retain API keys, signing seeds, database URLs, or unrestricted customer\npayloads in incident records.\n\nClassify the failing boundary\n\n| Observation | Likely boundary | Next evidence |\n\n| /health unavailable | Process, listener, edge, or network | Supervisor and bounded startup error |\n| /health is 200, /ready is 503 | Drain, lease, storage, or Dify health | Individual readiness fields |\n| runtime.ready is false | PostgreSQL, schema, role, or timeout | Durable storage probe and migration state |\n| connector.ready is false | Origin, TLS, credentials, or Dify | Descriptor set and bounded health error |\n| leasevalid is false | Heartbeat, control plane, sequence, or admission | Registry and recovery counters |\n| In-flight equals capacity | Local saturation | Action duration and route pressure |\n| Rejections increase | Host validation, trust, readiness, approval, or capacity | Signed protocol error code |\n| invalidstream appears | Provider SSE, bounds, or callback emission | Last valid event and cleanup certainty |\n| taskstore appears | Profile-state durability | Storage health and action mapping ownership |\n| Cancellation is uncertain | Missing task identity or failed stop | Durable mapping and provider task state |\n\nRecover storage and task state\n1. Keep the replica out of routing.\n2. Verify the configured database URL file, role, schema, and reachability.\n3. Preserve durable Actions, idempotency records, and profile state.\n4. Restore one coherent storage authority for all instance replicas.\n5. Inspect bounded startup recovery and task-store access.\n6. Require runtime.ready: true before registration or routing.\n7. Reconcile every mutation whose dispatch boundary is unknown.\n\nDo not delete task mappings or idempotency records to force readiness.\n\nRecover provider health or credentials\n1. Confirm the fixed Dify origin and descriptor hashes.\n2. Identify the failing credential revision without printing its key.\n3. Distinguish TLS, DNS, timeout, 401, 403, rate limit, and server failure.\n4. Rotate the owner-only key file through a replacement replica when required.\n5. Re-run readiness and the matching read-only canary.\n6. Reconcile earlier mutations before enabling writes.\n\nProduct keys and descriptors are startup-owned. Replacing file bytes does not\nprove that a running process adopted them.\n\nRecover lease or control-plane state\n1. Compare local instance, replica, version, manifest, artifact, and sequence\n   with the registry.\n2. Verify the fixed control endpoint, trusted DID, TLS, and registry state.\n3. Preserve a pending recovery request across ambiguous transport failure.\n4. Correct definitive admission or configuration rejection first.\n5. Require a valid ready lease and correct route before traffic resumes.\n\nNever start a second live process with the same replica ID while lease or\npinned Action ownership remains unresolved.\n\nRecover capacity pressure\n\nconnectorhost.capacity is returned before provider dispatch and is marked\nretryable by the host. Confirm durable Action state before repeating the same\ncorrelated request.\n\nReduce pressure or add an admitted replica before raising the in-flight limit.\nA larger limit does not increase Dify capacity or preserve latency.\n\nRecover stream failure\n\nPreserve the last valid event, task mapping, response-byte count, and cleanup\nresult. A malformed or oversized stream can occur after Dify admitted a task.\n\nIf a durable task ID exists, verify the mode-specific stop result. If no task\nID was observed, retain an uncertain outcome and reconcile provider state.\n\nDo not convert premature close into completion or replay the model execution\nunder a new Action ID.\n\nRecover an uncertain mutation\n1. Preserve the original Action ID, key, approval, and canonical input.\n2. Read durable Action and task-correlation state.\n3. Query the authoritative provider resource when an operation exposes one.\n4. Correlate Dify request, task, workflow-run, document, or batch identity.\n5. Accept matching state, retain ambiguity, or issue a separately approved\n   corrective intent.\n\nAn idempotency key fences AIP replay identity. It does not prove provider-side\ndeduplication or absence of effects.\n\nVerify recovery\n\nRequire:\n• /health is reachable;\n• /ready returns 200 with valid lease, no drain, and both probes ready;\n• registry identity and route match the intended replica;\n• ready and storage gauges agree with the latest readiness probe;\n• recovery counters stop increasing unexpectedly;\n• exact manifest counts and read-only canaries pass;\n• task-store access and streaming callback ownership are intact;\n• every uncertain mutation has an explicit reconciliation decision;\n• no durable state or secret was deleted to obtain recovery.\n\nObserve at least one lease renewal and a normal provider window. These checks\nestablish operational recovery, not complete capability qualification.\n\nRelated documentation\n• Deploy the Dify connector (deployment.md)\n• Dify connector configuration (../reference/configuration.md)\n• Dify streaming, cancellation, and binary responses (../reference/streaming-cancellation-and-binary-responses.md)\n• Observe and recover (../../../guides/observe-and-recover.md)\n• Metrics reference (../../../reference/metrics.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "6c261bf700e1ccad71a721a393d79d2b86c34a41a31b6f9af8af6fa24a57e65a"
  }
}