Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning (2024)

Abstract: Traditional Automatic Video Dubbing (AVD) pipeline consists of three key modules, namely, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). Within AVD pipelines, isometric-NMT algorithms are employed to regulate the length of the synthesized output text. This is done to guarantee synchronization with respect to the alignment of video and audio subsequent to the dubbing process. Previous approaches have focused on aligning the number of characters and words in the source and target language texts of Machine Translation models. However, our approach aims to align the number of phonemes instead, as they are closely associated with speech duration. In this paper, we present the development of an isometric NMT system using Reinforcement Learning (RL), with a focus on optimizing the alignment of phoneme counts in the source and target language sentence pairs. To evaluate our models, we propose the Phoneme Count Compliance (PCC) score, which is a measure of length compliance. Our approach demonstrates a substantial improvement of approximately 36% in the PCC score compared to the state-of-the-art models when applied to English-Hindi language pairs. Moreover, we propose a student-teacher architecture within the framework of our RL approach to maintain a trade-off between the phoneme count and translation quality.

## introduction automatic video dubbing (avd) technologies have become popular in recent times with the advent of generative ai technologies. avd technology automatically converts a video from one language to another language in three steps, (i) automatic speech recognition (asr) (ii) neural machine translation (nmt), and (iii) text-to-speech (tts). this task has become crucial especially in content creation as it helps to break down language barriers and reach a wider audience. a crucial factor underlying the quality and effectiveness of an avd system is the synchronization of the audio and video post-dubbing. for seamless and consistent synchronization, the duration of the target language speech generated by tts in the avd system must match with the duration of the source language speech. if the duration is not matched, various signal processing techniques can be applied to a certain extent to manipulate the duration of the final audio. however, this process introduces artifacts and degrades the quality of tts output. hence, a major focus of the research community has shifted towards controlling the length of the text output after nmt, such that there is much less mismatch in duration after dubbing. in this paper, we strive to enhance the performance of the isometric nmt model, introduced in @xcite , which is tasked with controlling the length of generated texts. traditionally machine translation for avd has been done as a two-step process @xcite , where for every input sentence, various output sentences are generated and then re-ranked according to length-matching. @xcite marked the advent of self-learning methods for the nmt task for avd. further works aimed to produce output texts with the duration compliance directly @xcite . however, these models rely on training a separate duration generation model for the length compliance, which is computationally too expensive. furthermore, works like @xcite use the matching of the number of characters or words between the source and target language sentences. however, in this work, we model this problem as matching the number of phonemes between the source and target language sentences because phonemes have a closer association with the speech duration @xcite . we model this matching as a reward indicator which simplifies and speeds up the training process in contrast with some previous works @xcite where the duration of translated texts was controlled using estimates of phoneme lengths, which is time-consuming @xcite . in addition, we propose a reinforcement learning (rl) based training strategy to achieve the task of isometric nmt in the context of generating translation outputs such that the phoneme counts of the source and target language sentences are as close as possible. we first translate the source language sentences using a pre-trained transformerbased nmt model (generation step), which we treat as an rl agent. then, we compute the ratio between the phoneme counts of the source and the generated target language sentences. after this, we filter out sentences where the phoneme count ratio (pcr) deviates from a pre-defined threshold determined empirically. we then use the filtered data for finetuning the agent model. we perform multiple iterations of generation using the rl agent and subsequent finetuning using the duration-based positively rewarded dataset. with each finetuning step, we make the pcr criteria stricter by increasing the threshold value for reward strategies, which positively reflects in the results we obtain (see sec.4.5), by achieving higher pcc scores. however, this adversely affects the translation quality. to address this issue, we modify the rl-agent (i.e., a fine-tuned (ft) model) with the help of knowledge distillation step via a studentteacher architecture (see figure 1 ) @xcite . we use knowledge-distillation step and student-teacher interchangeably throughout the paper. here, the teacher is the sota nmt model (i.e., the model with the best bleu score, but, possibly a poor pcc score). this further finetuning step helps the student (the current ft-model) to learn to produce good-quality as well as phoneme count compliant output. the effectiveness of our proposed model is demonstrated for the english-hindi language pair (hindi is spoken by more than 500 million people). for the training and validation, we used the bpcc corpus @xcite , and for testing, we used i) held-out bpcc test corpus, ii) flores, and iii) a movie database (see section 4.1). we significantly improved the performance of the english-hindi nmt with respect to various metrics like bleu, bleurt, comet, chrf, and a novel metric, namely, pcc which measures the length compliance between the source and translated sentence. we summarize our contributions as: ## related work neural machine translation models @xcite @xcite have majorly improved the performance in the machine translation task. transformer @xcite architecture is widely used in state-of-the-art nmt models. automatic video dubbing pipeline requires the use of nmt models which produce outputs such that the corresponding speech duration of the target language sentence matches the speech duration of the source language sentence. lakew et al. (2019) formulated this problem as matching the number of characters in the source and target language sentences. they injected the information regarding the number of characters in the positional embeddings with the help of tags appended to the source language sentence. in the work, lakew et al. (2022) introduced a self-training approach, both offline and online, and implemented the tagging of source sentences with the length ratio between the source and target language sentences, calculated based on the number of characters. @xcite formulated the problem as matching the duration in terms of the number of mel-frames of the source and target language sentences. they incorporated the number of mel-frames in positional embeddings of the transformer architecture. ## methodology in this section, we first discuss the problem setup of formulating the mt task in the rl framework. next, we propose our rl-based training approach for isometric nmt for achieving phoneme count compliant translation. finally, we conclude by proposing the student-teacher architecture by modifying the agent in the rl training to mitigate the problem of quality degradation. ## summary and conclusion in this paper, we proposed a reinforcement learningbased training strategy for isometric nmt. we proposed to match the count of phonemes for this task, as phonemes have a strong correlation with speech duration. further, we enhanced our agent in the rl-based training strategy with a student-teacher architecture to circumvent the problem of quality degradation that arises from optimizing the model for generating phoneme count compliant sentences. we also proposed the phoneme count compliance score to evaluate the performance of isometric nmt models. experimental results showed that our approach gives significant performance improvements in terms of phoneme compliance scores over various state-of-the-art nmt models including llms.