Skip to content

AI Dubbing โ€‹

Updated

This document refers to the official Tencent Cloud documentation AI Dubbing Integration.

It provides AI speech synthesis (TTS) and voice cloning, and is used for audiobooks, podcasts, video dubbing, and more, supporting 40 or more languages. The synchronous API(SyncDubbing) API processes cloning and short-text synthesis immediately, while the asynchronous API(ProcessMedia) API processes long-text synthesis (TextToSpeech) and speaker replacement (SpeechToSpeech).

API Information โ€‹

ItemValue
ServiceMPS
API ActionSyncDubbing(clone/tts), ProcessMedia(async-tts/async-sts)
Endpointmps.tencentcloudapi.com
API version2019-06-12
Processing modeAsynchronous (query the result after a TaskId is issued)

For authentication, all API use the same TC3-HMAC-SHA256 (SecretId/SecretKey) signature. For how to generate a signature, refer to the TokenHub document in Quick Start.

You can obtain the result through DescribeTaskDetail or a completion callback.

SyncDubbing (clone / tts) โ€‹

ParameterTypeRequiredExampleDescription
TextStringRequired for ttsHello, welcome!Text to synthesize. In synchronous mode, 2000 characters or fewer
TextLangStringOptionalenText language. Defaults to zh. See the language table below
VoiceIdStringRequired for ttsv1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/...Voice ID. A system voice or the VoiceId returned by clone mode
AudioDataStringRequired for clone (choose one)base64 stringBase64 of the cloning audio. WAV/MP3/MP4 and more are supported
AudioUrlStringRequired for clone (choose one)https://example.com/voice.wavURL of the cloning audio. Use either this or AudioData
AudioLangStringOptionalzhLanguage of the cloning audio. Defaults to zh
ExtParamStringOptional{"synExt":{"sampleRate":44100,"pitch":2},"cloneExt":{"timeRanges":[[5.2,20]]}}Extended parameter (JSON string). synExt: sampleRate / pitch / duration, cloneExt: timeRanges (list of [start, end] in seconds)
Output.OutputTypeStringOptionalURLWhen URL is specified, a valid audio URL of 24 hours is returned instead of base64
ResourceIdStringOptional-Resource ID. The default value is the primary resource ID of the account

ProcessMedia (async-tts / async-sts) โ€‹

ParameterTypeRequiredExampleDescription
InputInfoObjectRequired{"Type":"URL","UrlInputInfo":{"Url":"https://example.com/video.mp4"}}For async-sts, the actual input media. For async-tts, any accessible URL can be used as a placeholder
OutputStorageObjectRequired{"Type":"COS","CosOutputStorage":{"Bucket":"mybucket-125xxx","Region":"ap-guangzhou"}}Output COS storage. In asynchronous mode, COS is required
OutputDirStringRequired/output/dubbing/Output directory. Must start with / and end with /
AiAnalysisTask.DefinitionIntegerRequired36Dubbing task template ID. Fixed value 36
AiAnalysisTask.ExtendedParameterStringRequired{"dubbing":{"dubbingType":"TextToSpeech","text":"...","voiceId":"v1_..."}}Dubbing configuration (JSON string). See the table below
TaskNotifyConfig.NotifyUrlStringOptionalhttps://example.com/callbackTask completion callback URL. NotifyType is URL

The dubbing object fields of ExtendedParameter are as follows.

ParameterTypeRequiredExampleDescription
dubbingTypeStringRequiredTextToSpeechTextToSpeech (long-text TTS) or SpeechToSpeech (speaker replacement)
textStringRequired for TextToSpeechlong text...Text to synthesize
textLangStringOptionalzhText language
voiceIdStringConditionally requiredv1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/...Voice ID. For SpeechToSpeech, either voiceId or cloneVideoUrl is required
cloneVideoUrlStringConditionally requiredhttps://example.com/train.mp4URL of the video/audio to clone. 5 seconds or longer, single speaker
cloneVideoLangStringOptionalzhLanguage of the video/audio to clone
srcLangStringOptionalzhLanguage of the source video/audio (SpeechToSpeech)
outputPatternStringOptional{taskType}-{timestamp}Prefix of the output file name. The {taskType} and {timestamp} placeholders are supported
extraPara.synExtObjectOptional{"pitch":2,"sampleRate":44100}Audio quality parameters. pitch ranges from -12 to 12, and sampleRate can be 8000 / 16000 (default) / 22050 / 32000 / 44100. sampleRate is used for TTS

Supported language codes (partial): zh / en / ja / ko / de / fr / es / it / ru / pt / ar / hi / th / vi / id / ms / tr / nl / pl / sv / fi / yue / he / fa, and more, 40 or more in total.

SyncDubbing โ€‹

ParameterTypeRequiredExampleDescription
ErrorCodeIntegerAlways0Processing result code. 0 indicates success
MsgStringAlwayssuccessResult message
VoiceIdStringFor clonev1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/...Issued voice ID. Used later in tts/async mode
AudioDataStringFor tts (choose one)base64 stringBase64 of the synthesized audio. Returned when OutputType is not specified
AudioUrlStringFor tts (choose one)https://...URL of the synthesized audio. Returned when OutputType=URL is specified, valid for 24 hours
RequestIdStringAlways3c140219-cfe9-470e-b241-907877d6fb03Request identifier

ProcessMedia โ€‹

ParameterTypeRequiredExampleDescription
TaskIdStringAlways2600011633-WorkflowTask-xxxxxIssued task ID. Used to query the result
RequestIdStringAlways3c140219-cfe9-470e-b241-907877d6fb03Request identifier

Request Example โ€‹

python
import json
from tencentcloud.common import credential
from tencentcloud.common.profile.client_profile import ClientProfile
from tencentcloud.common.profile.http_profile import HttpProfile
from tencentcloud.mps.v20190612 import mps_client, models

cred = credential.Credential("<SecretId>", "<SecretKey>")
http_profile = HttpProfile
http_profile.endpoint = "mps.tencentcloudapi.com"
client = mps_client.MpsClient(cred, "ap-guangzhou", ClientProfile(httpProfile=http_profile))

# tts mode: synchronous synthesis with text + VoiceId
params = {
    "Text": "Hello, welcome to Tencent Cloud voice synthesis!",
    "TextLang": "en",
    "VoiceId": "v1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/...",
    "ExtParam": json.dumps({"synExt": {"sampleRate": 44100, "pitch": 0}})
}

req = models.SyncDubbingRequest
req.from_json_string(json.dumps(params))
resp = client.SyncDubbing(req)
print(resp.to_json_string)
python
# async-sts mode: replace the video speaker (ProcessMedia, Definition=36)
params = {
    "InputInfo": {
        "Type": "URL",
        "UrlInputInfo": {"Url": "https://example.com/video.mp4"}
    },
    "OutputStorage": {
        "Type": "COS",
        "CosOutputStorage": {"Bucket": "mybucket-125xxx", "Region": "ap-guangzhou"}
    },
    "OutputDir": "/output/sts/",
    "AiAnalysisTask": {
        "Definition": 36,
        "ExtendedParameter": json.dumps({
            "dubbing": {
                "dubbingType": "SpeechToSpeech",
                "voiceId": "v1_Pi1pR9Q9UHqVOrQ0YpZFwL+Q/...",
                "srcLang": "zh"
            }
        })
    }
}

req = models.ProcessMediaRequest
req.from_json_string(json.dumps(params))
resp = client.ProcessMedia(req)
print(resp.to_json_string)

Response Example โ€‹

json
{
  "ErrorCode": 0,
  "Msg": "success",
  "AudioData": "UklGRiQAAABXQVZFZm10...",
  "RequestId": "3c140219-cfe9-470e-b241-907877d6fb03"
}

Usage Notes โ€‹

Use the complete VoiceId returned by clone mode. It is an encrypted base64 value that starts with the v1_ prefix and is 80 characters or longer.

For async-tts, specify an accessible URL in InputInfo. Specify OutputDir so that it starts and ends with /, and configure the output COS Bucket.

Mode selection criteria: use clone if a VoiceId needs to be issued (audio required); use tts for short-text synthesis of 2000 characters or fewer (VoiceId required); use async-tts for long-text synthesis; and use async-sts for speaker replacement of existing audio/video (the actual input plus either voiceId or cloneVideoUrl is required).

Built with VitePress ยท Deployed on Tencent EdgeOne Pages