{
  "schemaVersion": "1.0",
  "title": "Dify knowledge pipelines and uploads",
  "description": "Use this family to discover datasource nodes, upload a local source, run one datasource node, or run a full Dify knowledge pipeline. The four operations use one workspace-scoped knowledge credential.",
  "canonical": "https://getaip.org/docs/connectors/dify/capabilities/knowledge-pipelines-and-uploads",
  "route": "/docs/connectors/dify/capabilities/knowledge-pipelines-and-uploads",
  "source": "docs/connectors/dify/capabilities/knowledge-pipelines-and-uploads.md",
  "protocol": "Agent Interoperability Protocol",
  "protocolVersion": "1.0",
  "section": "Connectors",
  "documentType": "Connector",
  "language": "en",
  "downloads": {
    "md": "/docs/download/connectors/dify/capabilities/knowledge-pipelines-and-uploads.md",
    "txt": "/docs/download/connectors/dify/capabilities/knowledge-pipelines-and-uploads.txt",
    "json": "/docs/download/connectors/dify/capabilities/knowledge-pipelines-and-uploads.json",
    "pdf": "/docs/download/connectors/dify/capabilities/knowledge-pipelines-and-uploads.pdf"
  },
  "content": {
    "format": "text/markdown",
    "markdown": "---\ntitle: Dify knowledge pipelines and uploads\ndescription: >-\n  Discover and run knowledge pipeline sources through the frozen Dify\n  connector and its bounded response model\nkind: capability-reference\naudience: application-developer\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: dify\n---\n\n# Dify knowledge pipelines and uploads\n\nUse this family to discover datasource nodes, upload a local source, run one\ndatasource node, or run a full Dify knowledge pipeline. The four operations use\none workspace-scoped knowledge credential.\n\nPipeline inputs can fetch external content, invoke plugins, and create provider\nstate. Approval must bind the dataset, pipeline version, source identities,\ncredentials, inputs, and expected result handling.\n\n## Choose an operation\n\n| Operation suffix | Method and route | Provider output | Policy |\n|---|---|---|---|\n| `pipeline.datasource_plugin.list` | `GET /v1/datasets/{dataset_id}/pipeline/datasource-plugins` | JSON list | Low-risk read; retry supported |\n| `pipeline.datasource_node.run` | `POST /v1/datasets/{dataset_id}/pipeline/datasource/nodes/{node_id}/run` | Provider SSE, buffered by connector | Medium mutation; approval and key required |\n| `pipeline.run` | `POST /v1/datasets/{dataset_id}/pipeline/run` | Blocking JSON or provider SSE | Medium mutation; approval and key required |\n| `pipeline.file.upload` | `POST /v1/datasets/pipeline/file-upload` | JSON upload record | Medium mutation; approval and key required |\n\nThese Tool operations have knowledge scope and no app-mode projection. The\ncapability template is:\n\n```text\ncap:dify:knowledge:<credential_id>:<operation>\n```\n\nThe first three routes require a provider UUID dataset owned by the selected\nworkspace. Pipeline file upload is workspace-scoped and has no dataset route\nvalue.\n\n## Discover datasource nodes\n\n`pipeline.datasource_plugin.list` accepts optional query `is_published`. It\ndefaults to true; false selects the current draft pipeline.\n\n```json\n{\n  \"dataset_id\": \"0190c42f-2d5a-7000-8000-000000000801\",\n  \"query\": {\n    \"is_published\": true\n  }\n}\n```\n\nEach returned entry can include `node_id`, plugin and provider identity,\n`datasource_type`, title, user input variables, and available datasource\ncredentials. A credential entry can mark the provider default.\n\nDiscovery and execution must use the same published or draft selection.\nPersist the selected node, variable contract, datasource type, and credential\nidentity before a run.\n\n## Upload one local pipeline file\n\n`pipeline.file.upload` accepts one Action `file` and no `body`:\n\n```json\n{\n  \"file\": {\n    \"filename\": \"approved-source.pdf\",\n    \"content_type\": \"application/pdf\",\n    \"content_base64\": \"JVBERi0=\"\n  }\n}\n```\n\nThe connector requires a safe basename, decodes the base64, and sends one\nmultipart file. Its decoded upload limit is 32 MiB. Provider file type, tenant\nquota, and configured upload size can be stricter.\n\nDify returns status 201 with `id`, `name`, `size`, `extension`, `mime_type`,\ncreator, and creation time. Use that upload ID as the local-file reference in\na later pipeline run.\n\nUploading does not attach the file to a dataset or prove that a pipeline has\nprocessed it. Retain the approved upload ID and apply the workspace's orphaned\nfile policy after uncertain outcomes.\n\n## Run one datasource node\n\n`pipeline.datasource_node.run` requires `node_id` and these body fields:\n\n- `inputs`, containing the node's discovered variables\n- `datasource_type`, selecting the source family\n- `is_published`, selecting published or draft state\n- optional `credential_id`, otherwise the provider default is used\n\nDatasource type accepts `local_file`, `online_document`, `website_crawl`, or\n`online_drive`.\n\n```json\n{\n  \"dataset_id\": \"0190c42f-2d5a-7000-8000-000000000801\",\n  \"node_id\": \"datasource-node-1\",\n  \"body\": {\n    \"inputs\": {\n      \"source_id\": \"upload-0190\"\n    },\n    \"datasource_type\": \"local_file\",\n    \"credential_id\": \"credential-0190\",\n    \"is_published\": true\n  }\n}\n```\n\nThe provider always returns an event stream for this route. The connector\ndeclares the operation as synchronous JSON, reads the complete response, and\nplaces unparsed SSE text in the generic result `body`.\n\nThis operation does not emit AIP stream chunks and has no cancellation\ncontract. The response-size limit applies to the entire buffered stream.\n\n## Run the full pipeline\n\n`pipeline.run` requires `inputs`, `datasource_type`, `datasource_info_list`,\n`start_node_id`, `is_published`, and `response_mode`.\n\n```json\n{\n  \"dataset_id\": \"0190c42f-2d5a-7000-8000-000000000801\",\n  \"body\": {\n    \"inputs\": {},\n    \"datasource_type\": \"local_file\",\n    \"datasource_info_list\": [\n      {\n        \"reference\": \"upload-0190\",\n        \"name\": \"approved-source.pdf\"\n      }\n    ],\n    \"start_node_id\": \"datasource-node-1\",\n    \"is_published\": true,\n    \"response_mode\": \"blocking\"\n  }\n}\n```\n\nUse `blocking` for a structured JSON result through this connector version.\nIf `streaming` is selected, the provider returns SSE, but the connector buffers\nit and exposes the complete event text as one generic result body.\n\nDatasource items follow the selected family:\n\n| Type | Required item identity |\n|---|---|\n| `local_file` | `reference`; `related_id` is a provider alias |\n| `online_document` | `workspace_id` and `page.page_id` plus `page.type` |\n| `website_crawl` | `url` |\n| `online_drive` | `id` and `type`, where type is `file` or `folder` |\n\nLocal file can add `name`. Online document can add page name and credential.\nWebsite crawl can add title. Online drive can add bucket and name.\n\nTreat every remote URL, workspace, page, file, folder, bucket, plugin, and\ncredential as untrusted until policy verifies its source and scope. The\nconnector does not add datasource-specific allowlists.\n\n## Interpret the connector result\n\nBlocking JSON returns through the generic envelope:\n\n```json\n{\n  \"http_status\": 200,\n  \"body\": {\n    \"event\": \"workflow_finished\",\n    \"data\": {}\n  },\n  \"content_type\": \"application/json\",\n  \"provider_request_id\": null\n}\n```\n\nFor provider SSE, `body` is a string containing the buffered event-stream\ntext. Parse it only with a bounded SSE parser after verifying content type.\nDo not reinterpret that string as an AIP streaming execution.\n\nThe connector response limit defaults to 8 MiB and allows 1 to 64 MiB. An\noversized or non-terminating provider stream fails the operation without a\nremote cancellation guarantee.\n\n## Apply mutation governance\n\nAll three mutations require a non-blank Action idempotency key of at most 512\nbytes. AIP retains its tenant-scoped reservation for 24 hours and revalidates\nthe input hash on collision.\n\nThe key does not prove provider deduplication. These operations publish no\nautomatic retry, transaction, reconciliation, cancellation, or compensation\nsupport.\n\nTreat uploads, external source references, credentials, node inputs, provider\nevents, generated documents, and errors as confidential PII-bearing data.\nRedact them from general logs.\n\n## Recover safely\n\n| Observation | Decision |\n|---|---|\n| Dataset, node, input, or file is rejected before dispatch | Correct the bounded request without claiming a provider effect |\n| Upload response is lost | Reconcile provider upload records before another upload |\n| Node or credential differs from discovery | Refresh the same published or draft node list before execution |\n| Buffered SSE exceeds the response bound | Treat the outcome as unknown; do not replay automatically |\n| Pipeline streaming connection does not terminate | Preserve timing and provider identity, then inspect dataset state |\n| Blocking run loses its response | Inspect provider pipeline and generated document state before another run |\n| Remote source fails policy | Reject the run before dispatch and preserve the authorization decision |\n| Draft execution reaches a production workflow | Stop traffic and treat it as a release-boundary incident |\n| Returned content crosses a workspace or dataset boundary | Stop traffic and treat it as an isolation incident |\n\nPreserve the Action ID, digest, approval, idempotency key, credential, dataset,\npipeline version, node, source identities, upload ID, and provider evidence\nneeded to settle an ambiguous outcome.\n\n## Related documentation\n\n- [Dify capability index](README.md)\n- [Dify knowledge documents and indexing](knowledge-documents-and-indexing.md)\n- [Dify connector configuration](../reference/configuration.md)\n- [Authentication and credential scopes](../getting-started/authentication-and-credential-scopes.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n",
    "text": "Dify knowledge pipelines and uploads\n\nUse this family to discover datasource nodes, upload a local source, run one\ndatasource node, or run a full Dify knowledge pipeline. The four operations use\none workspace-scoped knowledge credential.\n\nPipeline inputs can fetch external content, invoke plugins, and create provider\nstate. Approval must bind the dataset, pipeline version, source identities,\ncredentials, inputs, and expected result handling.\n\nChoose an operation\n\n| Operation suffix | Method and route | Provider output | Policy |\n\n| pipeline.datasourceplugin.list | GET /v1/datasets/{datasetid}/pipeline/datasource-plugins | JSON list | Low-risk read; retry supported |\n| pipeline.datasourcenode.run | POST /v1/datasets/{datasetid}/pipeline/datasource/nodes/{nodeid}/run | Provider SSE, buffered by connector | Medium mutation; approval and key required |\n| pipeline.run | POST /v1/datasets/{datasetid}/pipeline/run | Blocking JSON or provider SSE | Medium mutation; approval and key required |\n| pipeline.file.upload | POST /v1/datasets/pipeline/file-upload | JSON upload record | Medium mutation; approval and key required |\n\nThese Tool operations have knowledge scope and no app-mode projection. The\ncapability template is:\n\ncap:dify:knowledge::\n\nThe first three routes require a provider UUID dataset owned by the selected\nworkspace. Pipeline file upload is workspace-scoped and has no dataset route\nvalue.\n\nDiscover datasource nodes\n\npipeline.datasourceplugin.list accepts optional query ispublished. It\ndefaults to true; false selects the current draft pipeline.\n\n{\n  \"datasetid\": \"0190c42f-2d5a-7000-8000-000000000801\",\n  \"query\": {\n    \"ispublished\": true\n  }\n}\n\nEach returned entry can include nodeid, plugin and provider identity,\ndatasourcetype, title, user input variables, and available datasource\ncredentials. A credential entry can mark the provider default.\n\nDiscovery and execution must use the same published or draft selection.\nPersist the selected node, variable contract, datasource type, and credential\nidentity before a run.\n\nUpload one local pipeline file\n\npipeline.file.upload accepts one Action file and no body:\n\n{\n  \"file\": {\n    \"filename\": \"approved-source.pdf\",\n    \"contenttype\": \"application/pdf\",\n    \"contentbase64\": \"JVBERi0=\"\n  }\n}\n\nThe connector requires a safe basename, decodes the base64, and sends one\nmultipart file. Its decoded upload limit is 32 MiB. Provider file type, tenant\nquota, and configured upload size can be stricter.\n\nDify returns status 201 with id, name, size, extension, mimetype,\ncreator, and creation time. Use that upload ID as the local-file reference in\na later pipeline run.\n\nUploading does not attach the file to a dataset or prove that a pipeline has\nprocessed it. Retain the approved upload ID and apply the workspace's orphaned\nfile policy after uncertain outcomes.\n\nRun one datasource node\n\npipeline.datasourcenode.run requires nodeid and these body fields:\n• inputs, containing the node's discovered variables\n• datasourcetype, selecting the source family\n• ispublished, selecting published or draft state\n• optional credentialid, otherwise the provider default is used\n\nDatasource type accepts localfile, onlinedocument, websitecrawl, or\nonlinedrive.\n\n{\n  \"datasetid\": \"0190c42f-2d5a-7000-8000-000000000801\",\n  \"nodeid\": \"datasource-node-1\",\n  \"body\": {\n    \"inputs\": {\n      \"sourceid\": \"upload-0190\"\n    },\n    \"datasourcetype\": \"localfile\",\n    \"credentialid\": \"credential-0190\",\n    \"ispublished\": true\n  }\n}\n\nThe provider always returns an event stream for this route. The connector\ndeclares the operation as synchronous JSON, reads the complete response, and\nplaces unparsed SSE text in the generic result body.\n\nThis operation does not emit AIP stream chunks and has no cancellation\ncontract. The response-size limit applies to the entire buffered stream.\n\nRun the full pipeline\n\npipeline.run requires inputs, datasourcetype, datasourceinfolist,\nstartnodeid, ispublished, and responsemode.\n\n{\n  \"datasetid\": \"0190c42f-2d5a-7000-8000-000000000801\",\n  \"body\": {\n    \"inputs\": {},\n    \"datasourcetype\": \"localfile\",\n    \"datasourceinfolist\": [\n      {\n        \"reference\": \"upload-0190\",\n        \"name\": \"approved-source.pdf\"\n      }\n    ],\n    \"startnodeid\": \"datasource-node-1\",\n    \"ispublished\": true,\n    \"responsemode\": \"blocking\"\n  }\n}\n\nUse blocking for a structured JSON result through this connector version.\nIf streaming is selected, the provider returns SSE, but the connector buffers\nit and exposes the complete event text as one generic result body.\n\nDatasource items follow the selected family:\n\n| Type | Required item identity |\n\n| localfile | reference; relatedid is a provider alias |\n| onlinedocument | workspaceid and page.pageid plus page.type |\n| websitecrawl | url |\n| onlinedrive | id and type, where type is file or folder |\n\nLocal file can add name. Online document can add page name and credential.\nWebsite crawl can add title. Online drive can add bucket and name.\n\nTreat every remote URL, workspace, page, file, folder, bucket, plugin, and\ncredential as untrusted until policy verifies its source and scope. The\nconnector does not add datasource-specific allowlists.\n\nInterpret the connector result\n\nBlocking JSON returns through the generic envelope:\n\n{\n  \"httpstatus\": 200,\n  \"body\": {\n    \"event\": \"workflowfinished\",\n    \"data\": {}\n  },\n  \"contenttype\": \"application/json\",\n  \"providerrequestid\": null\n}\n\nFor provider SSE, body is a string containing the buffered event-stream\ntext. Parse it only with a bounded SSE parser after verifying content type.\nDo not reinterpret that string as an AIP streaming execution.\n\nThe connector response limit defaults to 8 MiB and allows 1 to 64 MiB. An\noversized or non-terminating provider stream fails the operation without a\nremote cancellation guarantee.\n\nApply mutation governance\n\nAll three mutations require a non-blank Action idempotency key of at most 512\nbytes. AIP retains its tenant-scoped reservation for 24 hours and revalidates\nthe input hash on collision.\n\nThe key does not prove provider deduplication. These operations publish no\nautomatic retry, transaction, reconciliation, cancellation, or compensation\nsupport.\n\nTreat uploads, external source references, credentials, node inputs, provider\nevents, generated documents, and errors as confidential PII-bearing data.\nRedact them from general logs.\n\nRecover safely\n\n| Observation | Decision |\n\n| Dataset, node, input, or file is rejected before dispatch | Correct the bounded request without claiming a provider effect |\n| Upload response is lost | Reconcile provider upload records before another upload |\n| Node or credential differs from discovery | Refresh the same published or draft node list before execution |\n| Buffered SSE exceeds the response bound | Treat the outcome as unknown; do not replay automatically |\n| Pipeline streaming connection does not terminate | Preserve timing and provider identity, then inspect dataset state |\n| Blocking run loses its response | Inspect provider pipeline and generated document state before another run |\n| Remote source fails policy | Reject the run before dispatch and preserve the authorization decision |\n| Draft execution reaches a production workflow | Stop traffic and treat it as a release-boundary incident |\n| Returned content crosses a workspace or dataset boundary | Stop traffic and treat it as an isolation incident |\n\nPreserve the Action ID, digest, approval, idempotency key, credential, dataset,\npipeline version, node, source identities, upload ID, and provider evidence\nneeded to settle an ambiguous outcome.\n\nRelated documentation\n• Dify capability index (README.md)\n• Dify knowledge documents and indexing (knowledge-documents-and-indexing.md)\n• Dify connector configuration (../reference/configuration.md)\n• Authentication and credential scopes (../getting-started/authentication-and-credential-scopes.md)\n• Errors and retry decisions (../../../reference/errors.md)\n"
  },
  "integrity": {
    "algorithm": "sha256",
    "sourceDigest": "8497c9636991708b659b3d5a409c86c44f146481c3ccc78b292707e27e0f4707"
  }
}