End-to-End Speech Translation of A rabic to E nglish Broadcast News (2022)

Abstract: Speech translation (ST) is the task of directly translating acoustic speech signals in a source language into text in a foreign language. ST task has been addressed, for a long time, using a pipeline approach with two modules : first an Automatic Speech Recognition (ASR) in the source language followed by a text-to-text Machine translation (MT). In the past few years, we have seen a paradigm shift towards the end-to-end approaches using sequence-to-sequence deep neural network models. This paper presents our efforts towards the development of the first Broadcast News end-to-end Arabic to English speech translation system. Starting from independent ASR and MT LDC releases, we were able to identify about 92 hours of Arabic audio recordings for which the manual transcription was also translated into English at the segment level. These data was used to train and compare pipeline and end-to-end speech translation systems under multiple scenarios including transfer learning and data augmentation techniques.

## introduction end-to-end approach to speech translation is gradually replacing the cascaded approach which consists of transcribing the speech inputs with an asr system, and translating the obtained transcription using a text-to-text mt system. for instance, and for the first time, the winning system in the iwslt 2020 ted english-to-german speech translation shared task was an end-to-end system @xcite . despite this positive result, the endto-end approach is used on a limited scale due to the lack of labeled data. indeed, data scarcity is today the major blocker for the widespread adoption of the end-to-end models. taking this into consideration, recent works have focused on developing speech translation corpora. joint efforts in this direction have allowed us to collect a significant quantity and good quality of speech transla-tion corpora. not surprisingly, speech translation corpus development has started for well-resourced languages including english and some european languages. in @xcite , the 236 hours @xmath0. shortly after, @xcite released the must-c corpus including few hundreds (385 to 504 hours) of parallel st data of ted talks translations from english to eight european languages. at the same time, the europarl-st @xcite was released with translations between 6 european languages, with a total of 30 translation directions. while all the previous resource development effort has focused on well-resourced languages, the most recent published corpora covost @xcite and covost2 @xcite . these latter works released a large-scale multilingual speech-to-text translation corpus covering translations from 21 languages to english and from english to 15 languages. although the arabic-english is one of the language pairs covered by the covost2 corpus, the authors consider it as a low-resource pair. in fact, covost2 corpus contained only 6 hours of prepared speech uniformly splitted between train, dev and test sets. in this paper, we conduct a series of experiments to present the first results of arabic to english endto-end broadcast news speech translation. this paper is organized as follows: section 2 presents arabic-to-english speech translation related works. section 3 gives details about the source of our training data and the method we have used to extract theses data. in section 4, we present our experimental setup as well as the used toolkits to train ou models. sections 5 and 6 provides details about the pipeline and end-to-end speech translation systems, respectively. section 7, gives a brief discussion and results analysis. finally, section 8 concludes the findings of this study and discusses future work. ## related works arabic-english (ar-en) is one of the most studied language pair in the context of speech recognition and machine translation. for instance, this language pair was integrated in several evaluation campaigns and projects including the international workshop on spoken language translation (iwslt) and darpa's babylon project. these earlier projects have built on the traditional pipelined architecture integrating speech recognition system in the source language followed by machine translation from the transcript to the target language. in iwslt, the speech translation task was introduced for the first time in 2006. the iwslt06 @xcite translation campaign was carried using either the manual or the automatic transcription of speech input in the travel domain. this translation task was renewed for several years using always the pipeline approach. pipeline architecture was also used by bbn in the context babylon project @xcite . they developed the transtalk system including a pipeline of asr and mt systems in both directions (ar↔en). more recently, but still with the same approach, qcri presented their live arabic-to-english speech translation system in @xcite . the system is a pipeline of a kaldi-based speech recognition followed by a phrase-based/neural mt system. recently, there has been a shift to the most recent end-to-end approach without the intermediate step of transcribing the source language. indeed, iwslt 2018 was the first time where organizers drooped the asr task and participants needed to develop an end-to-end speech translation systems. end-toend speech translation has shown its effectiveness for multiple languages and in multiple scenarios. it becomes now a well-established task in iwslt evaluation campaign were multiple shared taks are proposed to assess spoken language translation (slt) systems for many language pair in several settings. despite the great interest being shown to the end-to-end approach for speech translation task, we were able to identify only one recent work by @xcite including arabic-english language pair limited to a corpus of 6 hours. we are also aware of the iwslt 2022 dialectal speech translation foot_0 task which, unlike 3 training data whatever the chosen architecture for speech translation system (pipeline or end-to-end), it requires a large amount of manually annotated training data that might be hard to obtain for multiple language pairs. for the arabic-english language pair, a large amount of training data for asr and mt was manually annotated in the framework of the darpa's global autonomous language exploitation (gale) project @xcite . this huge amount of work was done for the purpose of making foreign languages speech and text accessible to english-only speakers through the development of automatic speech recognition and machine translation systems. in this respect, arabic broadcast news and conversation speech were collected from multiple sources, then annotated under the supervision of linguistic data consortium (ldc). audio corpora and their transcripts are separately released in three phases : gale phase 2, 3 and 4. in addition to the speech audio corpora and transcripts released to train arabic asr systems, ldc also made available multiple arabic to english parallel corpora.the latter are intended to be used for training and evaluating arabic to english mt systems. they have been developed by manually translating from a number of different sources including arabic news-wire, discussion groups and broadcast news and conversation. upon closer inspection of these parallel corpora, we have found that part of the broadcast news and conversation parallel corpora were built by translating the manual transcripts released for the asr task. following the discovery we dug deep in the gale speech recognition and machine translation ldc related releases and, as illustrated in figure 1 , we parsed all gale speech recognition and machine translation corpora in order to extract a 3-way parallel corpus consisting of arabic audio along with their arabic transcriptions and english translation. as shown in figure 1 , for each transcribed audio file part of the gale corpus, we extract only segments for which we were able to find an exact match between the manual transcription, from asr training data, and the source side of parallel corpora. all the extracted segments were afterwards aligned using timestamps information from asr transcript and translation from mt target side. as ## experimental setup all our experiments are built using open source toolkits with the following settings: asr models were built using the end-to-end speech processing toolkit espnet @xcite . we trained an attention-based encoder-decoder architecture with an encoder of 4 vggblstm layers including 1024 cells in each layer. the second and third bottom blstm layers of the encoder reduced the utterance length by a factor of two. we used a decoder of one 1024-dimensional blstm layer. for both asr and st speech utterance, we extracted 40 melfilterbank energy features with a step size of 10ms and a window size of 25ms. the extracted features, we applied mean and variance normalization. mt models were built using the fairseq package @xcite . we trained end-to-end word and bpe-based translation systems using the "lstm_luong_wmt_en_de" model template. this template is a standard lstm encoder-decoder architecture composed of 4 stacked blstm layers, each with 1000 cells, and 1000-dimensional embeddings @xcite . translation tasks (ast and mt) evaluation was carried out using case-sensitive bleu metric @xcite . scores are calculated using one human reference with moses'mteval-v14.pl script foot_2 applied to de-tokenized and punctuated translation output. as for asr, systems were evaluated using word error rate (wer). ## pipeline speech translation in this section, we evaluate the pipeline approach for speech translation in two different scenarios, plausible for many language pairs, depending on the amount and the type of training data used for the development of the speech translation task. 1. constrained scenario : under this scenario we have access to a 3-way limited training data. this data includes speech audio files in source language their transcriptions in the source language and translations to the target language. as to the first scenario of the pipeline approach we only used the 3-way parallel data reported in as previously stated, within the pipeline st framework, the output of the asr module is automatically translated to the target language using the mt module. the mt module is also an end-to-end system trained using fairseq toolkit @xcite under both constrained (mt_const) and unconstrained (mt_uncons) scenarios. as we have seen above, translation quality is further improved when the input to the translation module is of a higher quality generated by the unconstrained asr system (row mt_unconst_asr_unconst). this allows to reach a dev and test bleu scores of 36.48 and 25.80 respectively. as expected, the bleu score is even better when it comes to translate the reference transcription (mt_unconst_ref_transc) as shown in the last row of 6 end-to-end speech translation in this section, we present and evaluate the end-to-end approach for arabic to english speech translation task. the end-to-end system is built using the espnet toolkit @xcite . we used an attention-based encoder-decoder architecture. the encoder has two vgg-like cnn blocks followed by five stacked 1024-dimensional blstm layers. the decoder is composed of two 1024-dimensional lstm layers. each vgg block contains two 2d-convolution layers followed by a 2d-maxpooling layer whose aim is to reduce both time and frequency dimension of the input speech features by a factor of 2. all our experiments are conducted using characters as target tokens. the first row from from this initial baseline and with the aim of improving the end-to-end system translation quality, we employed the well established transfer learning technique @xcite commonly referred as encoder pre-training. indeed, using the asr encoder of the unconstrained asr system (row asr_unconst in 3) to initialize the parameters of the st encoder greatly improves the performance of end-to-end st networks. the results of the encoder pre-training are shown in the second row ( (1) + enc. init) of just like the transfer learning via encoder pretraining approach, data augmentation is proven to enhance end-to-end speech translation quality. it is carried out using synthetic data which is generated by automatically translating the transcripts of an asr corpora in the source language. herein, we used the unconstrained nmt system (mt_unconst) of ## discussion and analysis despite the improvements brought by transfer learning and data augmentation technics, the best results are still obtained using cascade architecture. we believe that this performance gap can be partly explained by the fact that end-to-end system was trained using only a small amount (∼ 84 hours) of real speech translation corpus. based on the results of previous works from @xcite , the end-to-end st models are known as an effective means of circumventing the error-propagation problem faced by the conventional pipeline system. indeed, every involved component in the traditional pipeline approach produces errors, which are propagated through the cascade and lead to compounding follow-up errors. in order to assess the ability of our endto-end system to overcome this error-propagation pattern, we selected some translation examples where pipeline system fails due to this problem and we checked the translation output of the end-to-end system. example from in addition to this error-propagation problem we have found that end-to-end system is sometimes penalized although its translation is correct. e2e mt and now i said i said that there are some interpretations of the institutions of the fatah movement. mt reference i said, i clearly said that there are, there are some extremist currents within the fatah movement. ## conclusions in this paper, we presented the first results on arabic-to-english end-to-end automatic speech translation system. arabic-english language pairs is one of the well-studied language pair in natural language processing. therefore, large quantities of data are made available in a wide variety of domains including asr and mt. starting from independent ldc releases for mt and asr systems, we were able to extract around 92 hours of speech translation corpus composed of arabic audio and their source transcriptions and english translation. we used this corpus to conduct speech translation experiments using a pipeline and an end-to-end speech translation architecture. both methods were tested under a constrained and an unconstrained conditions. we showed that the performance gap, which is too big between the two considered approaches under the constrained condition, can be narrowed under unconstrained condition through the use of transfer learning and data augmentation techniques. in spite of considerable improvement obtained by applying these techniques, the gap remains important and we plan to reduce it in several ways including decoder pre-training, spectrogram augmentation and self-supervised learning.