VUS at IWSLT 2021: A Finetuned Pipeline for Offline Speech Translation (2021)

Abstract: In this technical report, we describe the fine-tuned ASR-MT pipeline used for the IWSLT shared task. We remove less useful speech samples by checking WER with an ASR model, and further train a wav2vec and Transformers-based ASR module based on the filtered data. In addition, we cleanse the errata that can interfere with the machine translation process and use it for Transformer-based MT module training. Finally, in the actual inference phase, we use a sentence boundary detection model trained with constrained data to properly merge fragment ASR outputs into full sentences. The merged sentences are post-processed using part of speech. The final result is yielded by the trained MT module. The performance using the dev set displays BLEU 20.37, and this model records the performance of BLEU 20.9 with the test set.

## introduction offline speech translation is a task that infers the text of a target language by using speech as input. a pipeline system is used as a representative method, which converts source speech into the source text via automatic speech recognition and machine translates it. recently, many speech corpora have been disclosed, and studies are being conducted on an end-to-end method, namely directly decoding speech input into the text of a target language @xcite . in this iwslt shared offline task, we implement an english-german speech translation system in a pipeline format. the advantage of pipeline architecture is that it can explain whether the given speech translation is challenging in view of the acoustic domain or the translation perspective, considering the whole process of converting source speech to the target text. this makes it easier for us to discern difficult or erroneous parts in speech and text processing. in general, a limitation of a pipeline system compared to an end-to-end system is that the quality of the final result is largely influenced by the intermediate text representation, which is usually obtained in an explicit format @xcite . therefore, we primarily remove training samples that can lower the asr performance, following the method used in @xcite . thereafter, based on the trained asr module, the output of test speech samples is transformed into the text and fed to the machine translation system to produce a final output. in this process, we conduct post-processing to obtain an accurate sentencelevel output, such as setting the sentence boundary between the fragment texts and re-aggregating some wrongly merged sentences. the performance is checked mainly with bleu score @xcite . through the system construction, we obtained a bleu score of 20.9 in en-de speech translation. in detail, the performance of the asr module reaches wer 28.3% based on 2015 test set, and the mt module records a bleu score of 32.2 based on the wmt dataset @xcite . in addition, we have observed that various pre-and post-processings lead to meaningful performance gains. in this paper, we first skim the related works on speech translation, automatic speech recognition, and machine translation, focusing on the publicly available datasets. then we describe how we obtained the asr and mt module used for the campaign. next, we demonstrate how we finally reach the translation for the dev and test set, along with some pre-and post-processing techniques. the results are provided with the analysis. ## related work various datasets exist for speech translation using english as the source language, being utilized in the training and evaluation in a wide range of studies. the representative one is @xcite , which provides english speech of ted talks, its transcript, and the translation to other indo-european languages, including german, where we exploit en-de in this study. in addition, covost enables multilingual speech translation based on common voice (cv) data @xcite , of which the wikipedia articles are the source text. europarl-st @xcite also provides various translations, for the debates in european parliament. data used for speech translation can also be used for automatic speech recognition and machine translation, but there are also corpora built for asr and mt only, on a large scale. librispeech @xcite , which is used for evaluation of asr models, is the most famous example, and tedlium is also the case foot_1 . they consist of the speech of the source language (english) and latin alphabet-based transcription. in contrast, since only text data is used in mt, the scale is much larger. typically used sources are wmt datasets @xcite @xcite ) and open subtitles. 3 all of the above datasets can be usefully used in speech translation, so they have been actively utilized in the previous iwslt campaigns @xcite . ## model we chose the cascading scheme to leverage the high performance of asr and mt modules. thus, we exploit a large variety of corpora mentioned above to train each module. ## inference we infer the final output with the speech instances of the dev set using the trained asr and mt modules. after the inference, we submit the inference of the test set using the model that yields the best results with the dev set. in the inference process of the dev and test set, a proper sentence split is additionally required. for the dev and test set, we separated the utterances from silence using the given segmentation information. the segmented audio files were transcribed with the asr module. in the post-processing of the transcribed speech, we use the following strategies. in detail, we set two cases of prohibit as first and prohibit as final as follows: -prohibit as first: ['md', 'to', 'rp', 'vb', 'vbn', 'vbd'] -prohibit as final: ['cc', 'dt', 'ex', 'md', 'pdt', 'pos', 'wdt', 'wp', 'wp$', 'wrb'] whenever the segmented sentence regards either case, it is concatenated with the previous sentence or the following sentence. prohibit as final was primarily applied. the list of sentences obtained from the above process is translated by the trained mt module. ## experiment overall, our speech translation pipeline has the following procedure. we achieved the performance of bleu 20.37 with the official dev set. we finally obtained the performance of bleu 20.9 with the test set using given segmentation. ## conclusion in this paper, we report the vus asr-mt pipeline system for en-de speech translation. the featured engineering schemes are wav2vec-based asr module, transformer-based mt, speech segmentation and post-processing, and various cleansing for the enhancement. we obtained similar performance with both dev and test set, the bleu score of 20.37 and 20.9 respectively. our model is explainable and partially improvable, given the transparent description of our pipeline system.