An Experiment on Speech-to-Text Translation Systems for M anipuri to E nglish on Low Resource Setting (2021)

Abstract: In this paper, we report the experimental findings of building Speech-to-Text translation systems for Manipuri-English on low resource setting which is first of its kind in this language pair. For this purpose, a new dataset consisting of a Manipuri-English parallel corpus along with the corresponding audio version of the Manipuri text is built. Based on this dataset, a benchmark evaluation is reported for the Manipuri-English Speech-to-Text translation using two approaches: 1) a pipeline model consisting of ASR (Automatic Speech Recognition) and Machine translation, and 2) an end-to-end Speech-to-Text translation. Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) and Time delay neural network (TDNN) Acoustic models are used to build two different pipeline systems using a shared MT system. Experimental result shows that the TDNN model outperforms GMM-HMM model significantly by a margin of 2.53% WER. However, their evaluation of Speech-to-Text translation differs by a small margin of 0.1 BLEU. Both the pipeline translation models outperform the end-to-end translation model by a margin of 2.6 BLEU score.

## introduction in recent times, the advance in machine transla tion (mt) systems research jumped from textual modality to multi modality. the success of the several machine translation system for major lan guages based on statistical and neural approaches shed light towards building better translations sys tems for low resource languages as well. of these, the statistical machine translation (smt) @xcite and neural machine translation (nmt) models @xcite started its journey from the traditional texttotext translation which fur ther expanded to the use of multiple modalities @xcite in the translation task. the usage of multiple modalities in mt un covers new avenues for mt researchers. mt tasks where multiple modalities are utilized include us ing multipleinput modalities, for example, incor porating visual and text modalities @xcite , translation between dif ferent input and output modalities such as speech totext translation @xcite , etc. with these various methodologies of mt, the main goal is to obtain the most key information in a modality in generating the optimal sentence trans lation. the speechtotext (s2t) translation is the translation of a speech in a source language to a target language text. the speechtotext transla tion task can be broadly addressed using two ap proaches: 1) with a pipeline strategy, which sep arates the different modalities into modality con version, i.e., asr, followed by texttotext mt. 2) endtoend (e2e) translation where the target text is directly generated from the speech in the source language. the speechtotext (s2t) can find its application in our daily life by creating an ease form of communication for individuals with phys ical disabilities. it can also be used in reducing the turnaround of quick documentation, generating subtitles, etc. despite the fact that researchers are pushing the frontiers in machine translation and improv ing their capabilities, most of the work is focused on wellstudied languages while work on low re source languages such as manipuri is falling be hind. manipuri (also known as meiteilon) is the official language of manipur, a northeastern state of india. manipuri is an extremely low resource language with a limited dataset available for the nlp (natural language processing) tasks which is one of the primary reasons that hindered the de velopment of nlp systems for the language. this work aims to promote speechtotext translation of an extremely low resource language by presenting a benchmark evaluation on a manu ally collected speech dataset. this work makes the following contributions: the rest of this paper is presented as follows: the prior relevant research is discussed in section 2, followed by the framework of our model in sec tion 3. section 4 and section 5 explain the setup of our system and analysis of our results. the con clusion and future work are summarized in section 6. ## related works early attempts to address s2t translation follows a pipeline approach of two independent models: asr and mt systems @xcite . the approach utilized the hypothesis of asr as an input to the mt model to generate the targetlanguage text. initial work on direct speech totext translation includes @xcite . using a small frenchenglish synthetic dataset from 7 speakers, @xcite carried out an end toend s2t translation. the author reported that their system to be capable of generalizing to a new speaker effectively. bansal et al. (2018 works on mt for manipurienglish language pair are reported using various techniques such as examplebased mt (singh and bandyopadhyay, 2010a), smt (singh and bandyopadhyay, 2010b; singh, 2013), and unsupervised nmt (singh and singh, 2020). in a comparative study of smt and nmt systems on the manipurienglish language pair, the authors @xcite reported nmt system to perform bet ter than the smt system. to date, there is no work in s2t translation for manipurienglish language pair. in order to fill this gap, a manipurienglish s2t translation is developed using a small dataset in our work. ## methodologies figure 1 illustrates the methodology of our work. as the initial step of our work, english text dataset is collected from news articles, which is translated to manipuri language. in the next step, speech is recorded for the manipuri text. the overall col lected dataset is then used to train the pipeline and endtoend s2t translation models. ## experimental setup in this section, we present the different speechto text translation experiments conducted, including the dataset and experimental setup. the training, development, and test data sets for @xmath0. ## results and analysis in this section, we illustrate the results of our @xmath1. along with the automatic met ric evaluation, we carried out an indepth qualita tive analysis and human evaluation of our transla tion systems. ## conclusion and future work in this work, a comparative study of the con ventional pipeline model and endtoend model of s2t translation on an extremely lowresource manipurienglish language pair is presented. we also made a comparison of two acoustic models: gmmhmm and tdnn, for the asr module. an improvement of 2.53% wer is observed in the asr model with tdnn compared to gmm hmm. the tdnn asr model is observed to be more robust than the gmmhmm model in terms of ngram match. the asr output is fed to a shared nmt system (trained with the indomain or the additional out of domain dataset) in our pipeline model. in comparison, the translation hy pothesis of the pipeline models are comparable in terms of the bleu score. however, using an nmt system trained with a dataset from mixed domain results to the decrease in the automatic evaluation score. though the endtoend s2t translation has various advantages over traditional pipeline mod els, the limited size of our dataset led to the endto end s2t model's low performance compared to the pipeline model. an extensive collection of parallel s2t translation training data is generally required to train such an endtoend s2t translation model. in future, we plan to increase the size of the dataset along with the collection of other forms of modalities such as images. we also plan to explore various speechtotext machine translation mod els to enhance the performance.