Dub- S 2 ST : Textless Speech-to-Speech Translation for Seamless Dubbing (2025)

Abstract: This paper introduces a cross-lingual dubbing system that translates speech from one language to another while preserving key characteristics such as duration, speaker identity, and speaking speed. Despite the strong translation quality of existing speech translation approaches, they often overlook the transfer of speech patterns, leading to mismatches with source speech and limiting their suitability for dubbing applications. To address this, we propose a discrete diffusion-based speech-to-unit translation model with explicit duration control, enabling time-aligned translation. We then synthesize speech based on the translated units and source speaker’s identity using a conditional flow matching model. Additionally, we introduce a unit-based speed adaptation mechanism that guides the translation model to produce speech at a rate consistent with the source, without relying on any text. Extensive experiments demonstrate that our framework generates natural and fluent translations that align with the original speech’s duration and speaking pace, while achieving competitive translation performance.

## introduction recent advancements in translation systems and speech technologies have enabled a vast amount of multimedia content to support multiple languages through automated speech dubbing. cross-lingual dubbing @xcite , which replaces speech audio of one language with that of another, allows global audiences to consume content in their native languages. while this significantly reduces language barriers, ensuring effective dubbing requires meeting several specific criteria: maintaining duration, speaker identity, and speaking speed @xcite . traditional dubbing systems typically employ a cascade of automated speech recognition * equal contribution. to address these limitations, speech-to-speech translation (s2st) systems @xcite @xcite have emerged and evolved into textless s2st @xcite @xcite , aiming to translate speech directly without intermediate text. while recent approaches have achieved translation quality comparable to cascaded systems, most of them lack the capability to control speech duration during translation. as a result, the output typically requires postprocessing like manual stretching or contracting to match the original speech duration for dubbing purposes. however, this process can degrade speech quality and lead to unnatural prosody and speaking pace. a key challenge underlying this issue lies in the limitations of existing training datasets. highquality dubbing demands not only accurate translation but also faithful preservation of the source speech's voice characteristics and speaking speed. however, it is inherently difficult to construct largescale datasets containing the same speaker uttering aligned content in multiple languages. conse-quently, most s2st datasets rely on synthesized target speech @xcite or web-crawled data @xcite , which prioritize linguistic fidelity over speechrelated information consistency. these datasets tend to exhibit discrepancies in speaker identity and speaking speed, making it challenging for models to learn how to preserve such attributes. despite progress in translation accuracy, existing s2st models trained on these datasets remain suboptimal for seamless dubbing. in response, we propose dub-s2st, a novel textless s2st framework specifically designed for dubbing applications, effectively leveraging existing datasets. to mitigate the ambiguity caused by discrepancies between source and target speech, we eliminate variations in the target that deviate from the paired source. specifically, we first convert continuous speech to discrete units that retain rich semantic features and minimal acoustic variations @xcite . we then apply our unit-based speed adaptation strategy to adjust the target's speaking rate to the source. using the processed data, we develop a speechto-unit translation model trained with a discrete diffusion objective @xcite . we also incorporate diffusion transformer @xcite , which allows the model to accurately predict speech units conditioned on diffusion timestep. moreover, our model inherently supports duration control by using predetermined lengths based on the duration of source speech. finally, we incorporate a conditional flow matching (cfm) @xcite -based synthesizer that generates high-quality speech conditioned on the translated speech units and the original source speech, closely resembling the original speaker's identity. through extensive evaluation, our proposed framework demonstrates superior preservation of duration, speaker identity, and speaking speed, while maintaining competitive translation accuracy. ablation studies further validate the effectiveness of each component in improving dubbing quality. to the best of our knowledge, dub-s2st is the first textless s2st framework tailored for seamless automatic dubbing that preserves both speaker identity and speaking speed. ## method in this section, we explain the data preparation, model architecture and training objective, and final synthesis of the translated speech. the overall architecture of our model is illustrated in fig 1 . ## conclusion in this paper, we introduce an s2st system suitable for dubbing applications. the proposed model generates translated speech with accurate content while preserving the duration and speaking speed of the source. this is achieved by the unique design that can generate speech with arbitrary duration, and speed adaptation that mitigates discrepancies between source and target speech. extensive experiments with systematic ablations demonstrate that dub-s2st outperforms the existing baselines and verify its applicability to cross-lingual dubbing.