Prabhupadavani: A Code-mixed Speech Translation Data for 25 Languages (2022)

Abstract: Nowadays, the interest in code-mixing has become ubiquitous in Natural Language Processing (NLP); however, not much attention has been given to address this phenomenon for Speech Translation (ST) task. This can be solely attributed to the lack of code-mixed ST task labelled data. Thus, we introduce Prabhupadavani, which is a multilingual code-mixed ST dataset for 25 languages. It is multi-domain, covers ten language families, containing 94 hours of speech by 130+ speakers, manually aligned with corresponding text in the target language. The Prabhupadavani is about Vedic culture and heritage from Indic literature, where code-switching in the case of quotation from literature is important in the context of humanities teaching. To the best of our knowledge, Prabhupadvani is the first multi-lingual code-mixed ST dataset available in the ST literature. This data also can be used for a code-mixed machine translation task. All the dataset can be accessed at: https://github.com/frozentoad9/CMST .

## introduction speech translation (st) is a task in which speech is simultaneously translated from source language to a different target language. 1 it aids to overcome the language barriers across different communities for various applications such as social media, education, tourism, medical etc. earlier attempts to build a robust speech translation system mainly focused on a cascaded approach where two separate architectures for automatic speech recognition (asr) and machine translation (mt) are used in pipeline mode @xcite @xcite @xcite . however, these approaches mainly suffer from cascading effect of error propagation. thus, attention shifted to end-to-end approaches (bérard et al., 2018; duong et al., 2016; weiss et al., 1 we refer to speech translation as a speech-to-text task. 2017; @xcite due to their ability to obliviate error propagation and ease of maintaining a single architecture. however, these end-to-end approaches could not match the performance of cascaded systems due to the lack of sufficiently large data @xcite . notably, with the recent upsurge in st datasets @xcite @xcite , this gap has been closed @xcite . nowadays, most users prefer to communicate using a mixture of two or many languages on platforms such as social media, online blogs, chatbots, etc. thus, code-mixing has become ubiquitous in all kinds of natural language processing (nlp) resources/tasks @xcite @xcite . however, the existing nlp tools may not be robust enough to address this phenomenon of code-mixing for various downstream nlp applications (srivastava and singh, 2021). therefore, there has been a surge in creating code-mixed datasets: (1) to understand reasons for the failure of existing models, and (2) to empower existing models for overcoming this phenomenon. nevertheless, it is challenging to find natural resources that essentially capture different aspects of code-mixing for creating datasets for a wide range of nlp tasks. although there has been considerable research in generating copious data and novel architectures for the st task, we find that not much attention has been given to address the code-mixing phenomenon on the st task. possibly, this can be justified due to the lack of a code-mixed st dataset. to the best of our knowledge, no such sufficiently large, multi-lingual, naturally occurring, code-mixed dataset is available for the st. thus, in this work, we introduce prabhupadavani, a multi-lingual, multi-domain, speech translation dataset for 25 languages containing 94 hours of speech by 130 speakers. the prabhupadavani is about vedic culture and heritage from indic litera-ture, where code-switching in the case of quotation from literature is important in the context of humanities teaching. the multiple domains cover utterances from public lectures, conversations, debates, and interviews on various social issues. this is the first code-mixed data for speech translation to the best of our knowledge. it is code-mixed with english, bengali and sanskrit. from the typological point of view, the languages covered vary over ten language families. all the audios files have been manually aligned and translated. we believe that our work will ignite research in this direction to understand-(1) how to make existing systems robust for handling this phenomenon effectively? (2) can multi-lingual training help to improve performance on code-mixed speech translation? (3) will the gap between the cascade and end-to-end systems be closed? (4) can we train a single model for all languages using parallel nature of the dataset? foot_0 2 related work speech translation: recently, there have been increased efforts for creating large speech translation data sets for many languages. nevertheless, the available datasets are limited to certain languages or only underpaid licenses for non-english languages. most of the relatively larger datasets are english-centric @xcite @xcite or with limited speech hours @xcite . recently, there has been upsurge in st datasets in the literature @xcite @xcite . to the best of our knowledge, there is no such naturally occurring, sufficiently large and multi-lingual st dataset that contains a code-switching phenomenon. we fill this gap by contributing a code-mixed speech translation dataset for 25 languages. code-mixing: code-mixing is ubiquitous and well addressed on variety of downstream nlp tasks. however, majority of code-mixed datasets are synthetically generated @xcite ; therefore, they may not be able to capture the different aspects of code-mixing. the code-mixed datasets for an asr task are either limited to only one/two languages or contain only a few hours of speech data @xcite . synthetic code-mixed datasets may not capture the different aspects of code-mixing. to the best of our knowledge, prabhupadavani is the first code-mixed dataset available for 25 languages on speech translation task. ## data description resource: vanimedia's multi-language subtitle project foot_1 has created 1,080 audio mini-clips of śr īla prabhupāda's lectures, conversations, debates, interviews and is now transcribing them in multiple languages. foot_2 700+ translators participate in creating subtitles for all 1,080 mini-clips for 108+ languages. currently, this work has been completed for 25 languages. to procure subtitles for each clip in multiple languages, translators are provided with mini-clips and english subtitles that are manually aligned with each utterance. they use a third-party software named dotsub.com foot_3 and the task of translators is to provide translation for the corresponding utterance with the help of given transcription. on average, there are 3-4 translators for each language, and each clip takes more or less one hour for translation. each translator has invested an average of 6 hours every day in translating these clips. collectively, the time taken to translate 1,080 clips into 25 languages is over 45 weeks. current release of the dataset contains 25 languages (including transcription) for which transcription/translations are available. for these languages, we have over 53k utterances of transcription and translations. preprocessing: in this section, we describe preprocessing steps followed to arrive at the final version of the data. first, we scrape transcription and their translations in 25 languages from dotsub and extracted the corresponding youtube links of all audio clips. we use selenium foot_4 web crawler to automatize the process of downloading. we convert those videos into mp3 audio clips using a thirdparty application foot_5 . we chop the converted audio files based on the timestamps provided in the subtitle (.srt) files. 8 this process boils down to 53,000 code-mixing: prabhupadavani is code-mixed across three languages: english, sanskrit, and bengali. language diversity: from typological point of view, prabhupadavani covers 25 languages (inlcluding asr) from 10 language families. they are listed as follows: (1) indo-european:-(a) romance: italian, portuguese, spanish, french (b) germanic: german, swedish, danish, dutch, english (c) baltic: lithuanian, latvian (d) slavic: applications of dataset: in this section, we throw some light on possible applications of prabhupadavani: (1) the full dataset of prabhupadavani contains 2,400 hours of speech. the current release of prabhupadavani dataset can be utilized to facilitate automatic subtitling of the remaining part of data in the different languages. in this way, it will also be helpful to generate relatively larger dataset. (2) this dataset can provide a fertile soil to investigate on-how to make existing systems robust to code-mixing phenomenon? will the gap between cascade and end-to-end to approaches be closed? can multi-lingual training help to address code-mixing phenomenon? can we train single st system for all languages? how robust will be these models on another domain? ## conclusion and discussion in this work, we focused on a code-mixed speech translation dataset. although code-mixing is a spoken language phenomenon, not much attention has been given to address this phenomenon due to the unavailability of such a dataset. thus, we released prabhupadavani, a high-quality multilingual multidomain code-mixed st dataset containing 94 hours of speech data, 130+ speakers, for 25 languages covering 10 language families. the dataset is codemixed with three languages: english, bengali, and sanskrit. in order to reduce the efforts needed for pre-processing, we provide stratified data splits for the dataset. the same dataset can be utilized for the code-mixed machine translation task. we believe that these efforts will (1) set a fertile soil for investigating the applicability of existing solutions, (2) help to analyze the kind of errors existing systems are making, and (3) facilitate researchers to propose a novel solution to make existing systems robust for code-mixing. we plan to extend this dataset for 108+ languages. in code-switching conversations, speakers prefer a specific communication in a specific language of choice. in that context, the interesting factor is often when the boundary between languages becomes fuzzy, as in cases where an english verb stem is used with spanish morphology. in the case of prabhupadavani, the audience does not necessarily speak sanskrit or bengali, and the code-switching is primarily quotation or explanation. that is interesting as a phenomenon, but it is not the same as dual bilingual code-switching. we believe future work may help to get deep insights into this phenomenon.