AI Dubbing โ
This document refers to the official Tencent Cloud documentation AI Dubbing Integration.
It provides AI speech synthesis (TTS) and voice cloning, and is used for audiobooks, podcasts, video dubbing, and more, supporting 40 or more languages. The synchronous API(SyncDubbing) API processes cloning and short-text synthesis immediately, while the asynchronous API(ProcessMedia) API processes long-text synthesis (TextToSpeech) and speaker replacement (SpeechToSpeech).
API Information โ
| Item | Value |
|---|---|
| Service | MPS |
| API Action | SyncDubbing(clone/tts), ProcessMedia(async-tts/async-sts) |
| Endpoint | mps.tencentcloudapi.com |
| API version | 2019-06-12 |
| Processing mode | Asynchronous (query the result after a TaskId is issued) |
For authentication, all API use the same TC3-HMAC-SHA256 (SecretId/SecretKey) signature. For how to generate a signature, refer to the TokenHub document in Quick Start.
You can obtain the result through DescribeTaskDetail or a completion callback.
SyncDubbing (clone / tts) โ
| Parameter | Type | Required | Example | Description |
|---|---|---|---|---|
| Text | String | Required for tts | Hello, welcome! | Text to synthesize. In synchronous mode, 2000 characters or fewer |
| TextLang | String | Optional | en | Text language. Defaults to zh. See the language table below |
| VoiceId | String | Required for tts | v1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/... | Voice ID. A system voice or the VoiceId returned by clone mode |
| AudioData | String | Required for clone (choose one) | base64 string | Base64 of the cloning audio. WAV/MP3/MP4 and more are supported |
| AudioUrl | String | Required for clone (choose one) | https://example.com/voice.wav | URL of the cloning audio. Use either this or AudioData |
| AudioLang | String | Optional | zh | Language of the cloning audio. Defaults to zh |
| ExtParam | String | Optional | {"synExt":{"sampleRate":44100,"pitch":2},"cloneExt":{"timeRanges":[[5.2,20]]}} | Extended parameter (JSON string). synExt: sampleRate / pitch / duration, cloneExt: timeRanges (list of [start, end] in seconds) |
| Output.OutputType | String | Optional | URL | When URL is specified, a valid audio URL of 24 hours is returned instead of base64 |
| ResourceId | String | Optional | - | Resource ID. The default value is the primary resource ID of the account |
ProcessMedia (async-tts / async-sts) โ
| Parameter | Type | Required | Example | Description |
|---|---|---|---|---|
| InputInfo | Object | Required | {"Type":"URL","UrlInputInfo":{"Url":"https://example.com/video.mp4"}} | For async-sts, the actual input media. For async-tts, any accessible URL can be used as a placeholder |
| OutputStorage | Object | Required | {"Type":"COS","CosOutputStorage":{"Bucket":"mybucket-125xxx","Region":"ap-guangzhou"}} | Output COS storage. In asynchronous mode, COS is required |
| OutputDir | String | Required | /output/dubbing/ | Output directory. Must start with / and end with / |
| AiAnalysisTask.Definition | Integer | Required | 36 | Dubbing task template ID. Fixed value 36 |
| AiAnalysisTask.ExtendedParameter | String | Required | {"dubbing":{"dubbingType":"TextToSpeech","text":"...","voiceId":"v1_..."}} | Dubbing configuration (JSON string). See the table below |
| TaskNotifyConfig.NotifyUrl | String | Optional | https://example.com/callback | Task completion callback URL. NotifyType is URL |
The dubbing object fields of ExtendedParameter are as follows.
| Parameter | Type | Required | Example | Description |
|---|---|---|---|---|
| dubbingType | String | Required | TextToSpeech | TextToSpeech (long-text TTS) or SpeechToSpeech (speaker replacement) |
| text | String | Required for TextToSpeech | long text... | Text to synthesize |
| textLang | String | Optional | zh | Text language |
| voiceId | String | Conditionally required | v1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/... | Voice ID. For SpeechToSpeech, either voiceId or cloneVideoUrl is required |
| cloneVideoUrl | String | Conditionally required | https://example.com/train.mp4 | URL of the video/audio to clone. 5 seconds or longer, single speaker |
| cloneVideoLang | String | Optional | zh | Language of the video/audio to clone |
| srcLang | String | Optional | zh | Language of the source video/audio (SpeechToSpeech) |
| outputPattern | String | Optional | {taskType}-{timestamp} | Prefix of the output file name. The {taskType} and {timestamp} placeholders are supported |
| extraPara.synExt | Object | Optional | {"pitch":2,"sampleRate":44100} | Audio quality parameters. pitch ranges from -12 to 12, and sampleRate can be 8000 / 16000 (default) / 22050 / 32000 / 44100. sampleRate is used for TTS |
Supported language codes (partial): zh / en / ja / ko / de / fr / es / it / ru / pt / ar / hi / th / vi / id / ms / tr / nl / pl / sv / fi / yue / he / fa, and more, 40 or more in total.
SyncDubbing โ
| Parameter | Type | Required | Example | Description |
|---|---|---|---|---|
| ErrorCode | Integer | Always | 0 | Processing result code. 0 indicates success |
| Msg | String | Always | success | Result message |
| VoiceId | String | For clone | v1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/... | Issued voice ID. Used later in tts/async mode |
| AudioData | String | For tts (choose one) | base64 string | Base64 of the synthesized audio. Returned when OutputType is not specified |
| AudioUrl | String | For tts (choose one) | https://... | URL of the synthesized audio. Returned when OutputType=URL is specified, valid for 24 hours |
| RequestId | String | Always | 3c140219-cfe9-470e-b241-907877d6fb03 | Request identifier |
ProcessMedia โ
| Parameter | Type | Required | Example | Description |
|---|---|---|---|---|
| TaskId | String | Always | 2600011633-WorkflowTask-xxxxx | Issued task ID. Used to query the result |
| RequestId | String | Always | 3c140219-cfe9-470e-b241-907877d6fb03 | Request identifier |
Request Example โ
import json
from tencentcloud.common import credential
from tencentcloud.common.profile.client_profile import ClientProfile
from tencentcloud.common.profile.http_profile import HttpProfile
from tencentcloud.mps.v20190612 import mps_client, models
cred = credential.Credential("<SecretId>", "<SecretKey>")
http_profile = HttpProfile
http_profile.endpoint = "mps.tencentcloudapi.com"
client = mps_client.MpsClient(cred, "ap-guangzhou", ClientProfile(httpProfile=http_profile))
# tts mode: synchronous synthesis with text + VoiceId
params = {
"Text": "Hello, welcome to Tencent Cloud voice synthesis!",
"TextLang": "en",
"VoiceId": "v1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/...",
"ExtParam": json.dumps({"synExt": {"sampleRate": 44100, "pitch": 0}})
}
req = models.SyncDubbingRequest
req.from_json_string(json.dumps(params))
resp = client.SyncDubbing(req)
print(resp.to_json_string)# async-sts mode: replace the video speaker (ProcessMedia, Definition=36)
params = {
"InputInfo": {
"Type": "URL",
"UrlInputInfo": {"Url": "https://example.com/video.mp4"}
},
"OutputStorage": {
"Type": "COS",
"CosOutputStorage": {"Bucket": "mybucket-125xxx", "Region": "ap-guangzhou"}
},
"OutputDir": "/output/sts/",
"AiAnalysisTask": {
"Definition": 36,
"ExtendedParameter": json.dumps({
"dubbing": {
"dubbingType": "SpeechToSpeech",
"voiceId": "v1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/...",
"srcLang": "zh"
}
})
}
}
req = models.ProcessMediaRequest
req.from_json_string(json.dumps(params))
resp = client.ProcessMedia(req)
print(resp.to_json_string)Response Example โ
{
"ErrorCode": 0,
"Msg": "success",
"AudioData": "UklGRiQAAABXQVZFZm10...",
"RequestId": "3c140219-cfe9-470e-b241-907877d6fb03"
}Usage Notes โ
Use the complete VoiceId returned by clone mode. It is an encrypted base64 value that starts with the v1_ prefix and is 80 characters or longer.
For async-tts, specify an accessible URL in InputInfo. Specify OutputDir so that it starts and ends with /, and configure the output COS Bucket.
Mode selection criteria: use clone if a VoiceId needs to be issued (audio required); use tts for short-text synthesis of 2000 characters or fewer (VoiceId required); use async-tts for long-text synthesis; and use async-sts for speaker replacement of existing audio/video (the actual input plus either voiceId or cloneVideoUrl is required).