{ "schemaVersion": "1.0", "title": "Dify audio transcription and synthesis", "description": "Use this family to convert one authorized audio file to text or convert authorized text to speech. Both operations use the selected app credential and a stable Dify end-user identity.", "canonical": "https://getaip.org/docs/connectors/dify/capabilities/audio", "route": "/docs/connectors/dify/capabilities/audio", "source": "docs/connectors/dify/capabilities/audio.md", "protocol": "Agent Interoperability Protocol", "protocolVersion": "1.0", "section": "Connectors", "documentType": "Connector", "language": "en", "downloads": { "md": "/docs/download/connectors/dify/capabilities/audio.md", "txt": "/docs/download/connectors/dify/capabilities/audio.txt", "json": "/docs/download/connectors/dify/capabilities/audio.json", "pdf": "/docs/download/connectors/dify/capabilities/audio.pdf" }, "content": { "format": "text/markdown", "markdown": "---\ntitle: Dify audio transcription and synthesis\ndescription: >-\n Transcribe one bounded audio upload or synthesize bounded speech through the\n frozen Dify connector\nkind: capability-reference\naudience: application-developer\nappliesTo: \"1.x\"\nwritingStandard: \"aip-docs/1.0\"\nlastReviewedRevision: \"97be86e9efedf07ecf1783b03800f683f107fb04\"\nconnector: dify\n---\n\n# Dify audio transcription and synthesis\n\nUse this family to convert one authorized audio file to text or convert\nauthorized text to speech. Both operations use the selected app credential and\na stable Dify end-user identity.\n\nAudio content can contain sensitive personal data. Approval must cover the\nsource material, selected app, end user, model invocation, and intended result\nhandling before either request runs.\n\n## Choose an operation\n\n| Operation suffix | Method and route | Request | Result | Connector projection |\n|---|---|---|---|---|\n| `audio.transcribe` | `POST /v1/audio-to-text` | Multipart `file` and `user` | JSON transcript | All six app modes |\n| `audio.synthesize` | `POST /v1/text-to-audio` | JSON body and `user` | Bounded binary | All six app modes |\n\nBoth operations are medium-risk Tool mutations. Each requires tenant-policy\napproval and one Action idempotency key, publishes no safe retry or\ncancellation, and declares rollback unsupported.\n\nThe capability template is:\n\n```text\ncap:dify::\n```\n\nUse tenant discovery to select the app capability. The Action cannot replace\nthe configured provider origin or Bearer key.\n\n## Transcribe one audio file\n\nBefore dispatch, verify that the audio belongs to the selected end user and\nthat the chosen Dify app enables speech-to-text. Supply a safe filename, exact\nMIME type, and base64 content:\n\n```json\n{\n \"user\": \"tenant-user-0190\",\n \"file\": {\n \"filename\": \"request.wav\",\n \"content_type\": \"audio/wav\",\n \"content_base64\": \"UklGRg==\"\n }\n}\n```\n\nThe connector decodes the content and sends multipart fields named `file` and\n`user`. It does not send Action `body` as a multipart field for this operation.\n\nThe pinned provider accepts these MIME types:\n\n- `audio/mp3`\n- `audio/mpga`\n- `audio/m4a`\n- `audio/wav`\n- `audio/amr`\n\nThe provider limit is 30 MiB. The connector permits at most 32 MiB of decoded\nupload data, while its base64 schema and the native Action envelope can impose\na lower effective ceiling. The smallest applicable limit governs the request.\n\nA successful result uses the generic JSON envelope:\n\n```json\n{\n \"http_status\": 200,\n \"body\": {\n \"text\": \"The transcribed speech appears here.\"\n },\n \"content_type\": \"application/json\",\n \"provider_request_id\": null\n}\n```\n\nThe connector does not normalize the language, confidence, speaker identity,\nor punctuation. At the pinned provider revision, the public response contains\none required `text` value.\n\n## Synthesize bounded speech\n\nThe provider accepts `message_id`, `voice`, `text`, and `streaming` in the\noperation body. The connector inserts the validated `user` value.\n\nUse direct text when the caller owns the content:\n\n```json\n{\n \"user\": \"tenant-user-0190\",\n \"body\": {\n \"text\": \"Your scheduled review begins at fourteen thirty.\",\n \"voice\": \"provider-voice-id\"\n }\n}\n```\n\nWhen `message_id` is present, Dify gives that message priority over `text`.\nThe message lookup is constrained to the selected app and end user. Without a\nmessage ID, `text` is required.\n\n`voice` is optional. Dify uses the app's configured voice when available, or a\nvoice returned by its model provider. The connector does not publish a voice\ncatalog and does not validate provider-specific voice identifiers.\n\nThe `streaming` field is reserved by Dify for compatibility. Provider output\ncan arrive as streamed `audio/mpeg`, but this AIP operation buffers the bytes\nand returns one bounded binary result.\n\n```json\n{\n \"http_status\": 200,\n \"content_base64\": \"SUQz\",\n \"content_type\": \"audio/mpeg\",\n \"size_bytes\": 3,\n \"provider_request_id\": null\n}\n```\n\nThe configured connector response limit applies to all returned audio. Its\ndefault is 8 MiB, with an allowed configuration range from 1 to 64 MiB.\n\nDecode the result only after authorization, content-type policy, malware\ncontrols, and output-size checks. The connector does not write audio to a\ncaller-selected filesystem path or preserve it as durable provider state.\n\n## Apply mutation governance\n\nEach operation requires a non-blank Action idempotency key of at most 512\nbytes. AIP retains the tenant-scoped reservation for 24 hours and revalidates\nthe input hash on collision.\n\nThe key does not prove that Dify or the selected model provider deduplicates\nthe invocation. Neither operation publishes automatic retry, transaction,\nreconciliation, cancellation, or compensation support.\n\nTreat the uploaded audio, source text, referenced message, transcript, and\ngenerated bytes as confidential PII-bearing data. Retain only the evidence\nneeded to establish authorization and outcome.\n\n## Recover safely\n\n| Observation | Decision |\n|---|---|\n| Filename, base64, user, or body is rejected before dispatch | Correct the same request without claiming a provider effect |\n| Provider rejects the audio MIME type | Convert or select an allowed source under a new reviewed input |\n| Audio exceeds an effective request limit | Reduce the authorized source; do not raise response limits |\n| Speech-to-text or text-to-speech is disabled | Correct the selected app configuration before a new Action |\n| Message lookup returns no usable audio | Verify the same app, end user, message ID, and message state |\n| Voice is absent or unsupported | Select a provider-advertised voice or repair app configuration |\n| Binary output exceeds the response bound | Treat the result as failed and review expected output size |\n| Response is lost after model invocation | Treat the external outcome as uncertain; do not replay automatically |\n| Returned transcript or audio crosses an app or user boundary | Stop traffic and treat the event as an isolation incident |\n\nPreserve the Action ID, input digest, approval, idempotency key, provider\nrequest identity, and any bounded result used to settle an ambiguous outcome.\n\n## Related documentation\n\n- [Dify capability index](README.md)\n- [Dify app metadata and files](app-metadata-and-files.md)\n- [Dify connector configuration](../reference/configuration.md)\n- [Authentication and credential scopes](../getting-started/authentication-and-credential-scopes.md)\n- [Errors and retry decisions](../../../reference/errors.md)\n", "text": "Dify audio transcription and synthesis\n\nUse this family to convert one authorized audio file to text or convert\nauthorized text to speech. Both operations use the selected app credential and\na stable Dify end-user identity.\n\nAudio content can contain sensitive personal data. Approval must cover the\nsource material, selected app, end user, model invocation, and intended result\nhandling before either request runs.\n\nChoose an operation\n\n| Operation suffix | Method and route | Request | Result | Connector projection |\n\n| audio.transcribe | POST /v1/audio-to-text | Multipart file and user | JSON transcript | All six app modes |\n| audio.synthesize | POST /v1/text-to-audio | JSON body and user | Bounded binary | All six app modes |\n\nBoth operations are medium-risk Tool mutations. Each requires tenant-policy\napproval and one Action idempotency key, publishes no safe retry or\ncancellation, and declares rollback unsupported.\n\nThe capability template is:\n\ncap:dify::\n\nUse tenant discovery to select the app capability. The Action cannot replace\nthe configured provider origin or Bearer key.\n\nTranscribe one audio file\n\nBefore dispatch, verify that the audio belongs to the selected end user and\nthat the chosen Dify app enables speech-to-text. Supply a safe filename, exact\nMIME type, and base64 content:\n\n{\n \"user\": \"tenant-user-0190\",\n \"file\": {\n \"filename\": \"request.wav\",\n \"contenttype\": \"audio/wav\",\n \"contentbase64\": \"UklGRg==\"\n }\n}\n\nThe connector decodes the content and sends multipart fields named file and\nuser. It does not send Action body as a multipart field for this operation.\n\nThe pinned provider accepts these MIME types:\n• audio/mp3\n• audio/mpga\n• audio/m4a\n• audio/wav\n• audio/amr\n\nThe provider limit is 30 MiB. The connector permits at most 32 MiB of decoded\nupload data, while its base64 schema and the native Action envelope can impose\na lower effective ceiling. The smallest applicable limit governs the request.\n\nA successful result uses the generic JSON envelope:\n\n{\n \"httpstatus\": 200,\n \"body\": {\n \"text\": \"The transcribed speech appears here.\"\n },\n \"contenttype\": \"application/json\",\n \"providerrequestid\": null\n}\n\nThe connector does not normalize the language, confidence, speaker identity,\nor punctuation. At the pinned provider revision, the public response contains\none required text value.\n\nSynthesize bounded speech\n\nThe provider accepts messageid, voice, text, and streaming in the\noperation body. The connector inserts the validated user value.\n\nUse direct text when the caller owns the content:\n\n{\n \"user\": \"tenant-user-0190\",\n \"body\": {\n \"text\": \"Your scheduled review begins at fourteen thirty.\",\n \"voice\": \"provider-voice-id\"\n }\n}\n\nWhen messageid is present, Dify gives that message priority over text.\nThe message lookup is constrained to the selected app and end user. Without a\nmessage ID, text is required.\n\nvoice is optional. Dify uses the app's configured voice when available, or a\nvoice returned by its model provider. The connector does not publish a voice\ncatalog and does not validate provider-specific voice identifiers.\n\nThe streaming field is reserved by Dify for compatibility. Provider output\ncan arrive as streamed audio/mpeg, but this AIP operation buffers the bytes\nand returns one bounded binary result.\n\n{\n \"httpstatus\": 200,\n \"contentbase64\": \"SUQz\",\n \"contenttype\": \"audio/mpeg\",\n \"sizebytes\": 3,\n \"providerrequestid\": null\n}\n\nThe configured connector response limit applies to all returned audio. Its\ndefault is 8 MiB, with an allowed configuration range from 1 to 64 MiB.\n\nDecode the result only after authorization, content-type policy, malware\ncontrols, and output-size checks. The connector does not write audio to a\ncaller-selected filesystem path or preserve it as durable provider state.\n\nApply mutation governance\n\nEach operation requires a non-blank Action idempotency key of at most 512\nbytes. AIP retains the tenant-scoped reservation for 24 hours and revalidates\nthe input hash on collision.\n\nThe key does not prove that Dify or the selected model provider deduplicates\nthe invocation. Neither operation publishes automatic retry, transaction,\nreconciliation, cancellation, or compensation support.\n\nTreat the uploaded audio, source text, referenced message, transcript, and\ngenerated bytes as confidential PII-bearing data. Retain only the evidence\nneeded to establish authorization and outcome.\n\nRecover safely\n\n| Observation | Decision |\n\n| Filename, base64, user, or body is rejected before dispatch | Correct the same request without claiming a provider effect |\n| Provider rejects the audio MIME type | Convert or select an allowed source under a new reviewed input |\n| Audio exceeds an effective request limit | Reduce the authorized source; do not raise response limits |\n| Speech-to-text or text-to-speech is disabled | Correct the selected app configuration before a new Action |\n| Message lookup returns no usable audio | Verify the same app, end user, message ID, and message state |\n| Voice is absent or unsupported | Select a provider-advertised voice or repair app configuration |\n| Binary output exceeds the response bound | Treat the result as failed and review expected output size |\n| Response is lost after model invocation | Treat the external outcome as uncertain; do not replay automatically |\n| Returned transcript or audio crosses an app or user boundary | Stop traffic and treat the event as an isolation incident |\n\nPreserve the Action ID, input digest, approval, idempotency key, provider\nrequest identity, and any bounded result used to settle an ambiguous outcome.\n\nRelated documentation\n• Dify capability index (README.md)\n• Dify app metadata and files (app-metadata-and-files.md)\n• Dify connector configuration (../reference/configuration.md)\n• Authentication and credential scopes (../getting-started/authentication-and-credential-scopes.md)\n• Errors and retry decisions (../../../reference/errors.md)\n" }, "integrity": { "algorithm": "sha256", "sourceDigest": "e7e53e0c7a23f1a2277cd8e55e7173104c32df04b896d649f1fc5f60bd36555e" } }