Towards Speech to Speech Machine Translation focusing on I ndian Languages (2023)

Abstract: We introduce an SSMT (Speech to Speech Machine Translation, aka Speech to Speech Video Translation) Pipeline( https://ssmt.iiit.ac.in/ssmtiiith ), as web application for translating videos from one language to another by cascading multiple language modules. Our speech translation system combines highly accurate speech to text (ASR) for Indian English, pre-possessing modules to bridge ASR-MT gaps such as spoken disfluency and punctuation, robust machine translation (MT) systems for multiple language pairs, SRT module for translated text, text to speech (TTS) module and a module to render translated synthesized audio on the original video. It is user-friendly, flexible, and easily accessible system. We aim to provide a complete configurable speech translation experience to users and researchers with this system. It also supports human intervention where users can edit outputs of different modules and the edited output can then be used for subsequent processing to improve overall output quality. By adopting a human-in-the-loop approach, the aim is to configure technology in such a way where it can assist humans and help to reduce the involved human efforts in speech translation involving English and Indian languages. As per our understanding, this is the first fully integrated system for English to Indian languages (Hindi, Telugu, Gujarati, Marathi and Punjabi) video translation. Our evaluation shows that one can get 3.5+ MOS score using the developed pipeline with human intervention for English to Hindi. A short video demonstrating our system is available at https://youtu.be/MVftzoeRg48 .

## introduction india writes in many languages and speaks in many more tongues 2 . it is a geographically vast multilingual society with 22 recognized languages. the languages constitute 1.17+ billion speakers across 1 https://ssmt.iiit.ac.in/ssmtiiith 2 shorturl.at/dnsv8 28 states and 7 union territories. according to the 2011 census, while 129 million (10.6%) indians speak english, only 259,678 (0.02%) indians speak it as their first language. and only 8% indians read newspapers in english while others prefer news in their local languages. as stated in a report from karnataka (gfgc.kar.nic.in, 2014), about 40% of all enrolled students from non-metropolitan regions fail to achieve their educational goals because they are unable to cope with english and very few study materials are available in native indian languages foot_0 . same is true for medical and health awareness related content. there is a huge void of content in indian languages that necessitates urgent action. one solution for this problem is to use translation. such translations can help different language speakers to seamlessly communicate with each other. with this work, we aim to ease this language barrier through speech to speech machine translation (ssmt) by providing a baseline system for video translation. generally, there are two ways to implement speech-to-speech translation (ssmt): the first approach is to cascade systems of automatic speech recognition (asr), machine translation (mt), and text-to-speech (tts); second, direct end to end speech translation. cascaded ssmt systems have been successfully demonstrated for english and european languages, but one finds minimal work done for indian languages. recent work in direct end to end speech translation (translatotron) @xcite attempts to directly translate speech from one language into speech in another language with the source speaker's voice in the translated speech. it achieves high translation quality on two spanish to english datasets, although the reported performance is poorer than a baseline cascade of speech translation and tts models @xcite . we are not aware of any cascaded or direct speech-to-speech translation work involving indian languages. for the purpose of our work, we decided to implement a cascaded ssmt system. we also analyse gaps between the automatic modules and address them with pre-processing and post-processing tools. these gaps are :speech disfluencies, domain processing, and target language subtitling. our system takes an english video as an input and outputs the same video in the chosen indian language. our proposed video translation pipeline is user-friendly, flexible, and easily accessible with following key modules: ## approaches the canonical approach as shown in figure -1 includes automatic speech recognition (asr) to transcribe source language speech to text and then, machine translation (mt) to translate transcribed text into target language, and at the end, text to speech synthesis (tts) to generate speech in target language from the translated text @xcite @xcite . in this work, we aim to develop a system which can translate speech or video in english to selected indian languages. while following a cascaded approach, one can not directly chain modules such as asr, mt, and tts as it is a well known fact that spoken language has various idiosyncrasies. these include lack of well-formed sentences and disfluencies @xcite . traditional machine translation systems are trained on well formed, written, and grammatical pairs of sentences. therefore, it is crucial to address these aspects before directly translating transcribed text using machine translation. similarly, to sync original video with the translated text, time-stamping translated text is an essential step before text to speech synthesis. also, video content syncing (speaker lip, video content) with the generated speech is another important factor in making system complete. in recent times, technological advancements have enabled asr, mt, and tts to make quantum leaps. today, computers are capable of doing these with greater accuracy and efficiency than ever before. but they cannot be expected to be 100% accurate. human effort is still required to correct or edit these outputs. therefore, in our pipeline, we also include steps where a human can intervene after each automatic module which eventually reduces the overall human effort for the task. for the translation of technical lectures, domain processing is one of the important steps before translating transcribed text into the target language. domain processing is included as a pre-processing step, where identification of domain and domain terms are carried out before machine translation. in this pipeline, we also aim to develop an interface for state-of-the-art video to video machine translation between english to 5 indian languages (hindi, telugu, gujarati, marathi, and punjabi) along with pre-processing of asr output to make it translatable as well as post-processing of machine translation output to make it suitable for dubbing and video syncing. the subsequent section explains the ssmt pipeline, the interface, and the process in it. in section 4, we discuss the pipeline performance and conclude in section 5. ## process here, our task is to combine technologies to make video to video translation possible for english to indian languages as shown in figure-2. as discussed earlier, there are gaps between the components that need special processing. in order to fill these gaps, we link the major components with pre/post processing support tools with the provision of human interventions. these are before and after each major language component. therefore, as visible in different colors in figure -2 , we categorize the overall process into 4 major parts: input/output, core technology, pre/post processing support tools, and pre/post editing as human intervention. ## performance we evaluate the performance of the developed pipeline with two different metrics. they are: time taken to execute the pipeline and performance of major modules on their known evaluation metrics. one can access and execute presented ssmt pipeline using internet without installing specialized tools. the execution time for pipeline depends on the video length. on an average, it takes 1/3 of the video time for execution on a single gpu (nvidia-3080ti) system. due to resource constraints, for now we have set the input video length limit to 1 min in the interface, but this can be increased based on availability of compute infrastructure. we created an evaluation dataset of 2 small english technical videos of computer science domain for english-hindi translation direction. we hired experienced language professionals to carry out manual transcription and hindi translation for these videos. we used wer @xcite to evaluate asr verbatim (wer) and asrtomt gaps (wer) (that is verbatim + correct punctuation and without spoken disfluency). we used bleu @xcite to evaluate english to hindi machine translation eng-hin mt (bleu) performance without and with domain pre-processing eng-hin mt + domain (bleu) to the mt. mean opinion score (mos) is used to evaluate generated hindi speech and synced video. table-1 shows the evaluation results for ssmt pipeline for 2 videos. 1 st rows "direct + no punct" of video 1 and video 2 show the results when asr verbatim (without punctuation), mt, tts, and video syncing modules are used. the 2 nd rows "+ fix len punct" of video 1 & 2 show the results when a punctuation sym-bol is placed after every 20 tokens on the direct asr verbatim. the 3 rd rows "+ punct by asr" of video 1 & 2 show the results when punctuations are given by asr along with asr verbatim. the 4 th ## conclusion and future work in this paper, we introduce an ssmt pipeline, an intelligent speech to speech video translation interface for english to hindi, telugu, gujarati, marathi, and punjabi. this work demonstrates that speech to speech (video to video) translation is possible with a cascaded pipeline and support tools. we believe that large scale deployment of this can help lower the language barrier. the results also point out that human intervention is necessary to get high quality video translation output. in the near future, we aim to come up with benchmark corpora for speech to speech machine translation and evaluation involving english and multiple indian languages. we also plan to further improve the developed pipeline and its components to reduce involved human effort over a period of time. we will also plan to add multiple language directions to the pipeline in future.