On the Impact of Noises in Crowd-Sourced Data for Speech Translation (2022)

Abstract: Training speech translation (ST) models requires large and high-quality datasets. MuST-C is one of the most widely used ST benchmark datasets. It contains around 400 hours of speech-transcript-translation data for each of the eight translation directions. This dataset passes several quality-control filters during creation. However, we find that MuST-C still suffers from three major quality issues: audiotext misalignment, inaccurate translation, and unnecessary speaker’s name. What are the impacts of these data quality issues for model development and evaluation? In this paper, we propose an automatic method to fix or filter the above quality issues, using English-German (En-De) translation as an example. Our experiments show that ST models perform better on clean test sets, and the rank of proposed models remains consistent across different test sets. Besides, simply removing misaligned data points from the training set does not lead to a better ST model.

## introduction speech-to-text translation (st) aims to translate a speech of a certain language into a text translation of another language. recent advances of end-to-end st models have been largely boosted by the release of large high-quality parallel datasets @xcite @xcite . a clean test set is essential to evaluate the effectiveness of proposed models, and a sizeable well-aligned training set is important to train powerful st models @xcite . currently, the most widely-used st benchmark dataset is @xcite . it consists of around 400 hours of speech-transcripttranslation data from english into eight languages (german, spanish, french, italian, dutch, portuguese, romanian, and russian). must-c was built upon english ted talks, which are often transcribed and translated by voluntary human annotators. a bilingual sentence-level text corpus is firstly constructed based on sentence segmentation and gargantua alignment tool @xcite . then, the transcription is aligned to the corresponding audio tracks using gentle forced aligner foot_1 built on kaldi asr toolkit @xcite . during alignment, entire talks are discarded if greater than 15% of words cannot be recognized, and sentences are removed if none of the words was aligned. though must-c passed through several qualitycontrol filters, this dataset is still not perfect. through manual checking, we find three major quality issues in the dataset -inaccurate translation, audio-text misalignment, and unnecessary speaker's name. along with the three issues identified, more importantly, we are interested in the following questions: do they affect the robustness of end-to-end speech translation models trained on this corpus? can we trust the results from existing works using this data? in order to answer the above questions, we propose an automatic method to filter or fix the aforementioned errors in both the training and test sets. and based on the original and the fixed datasets, we evaluate many popular st systems including codebases such as espnet @xcite and published models such as xstnet @xcite . our experiments have shown that the performance of models we test is actually better than we thought, and their rank remains consistent across test sets. besides, simply removing those data points with audio-text misalignment from the training set cannot significantly improve st models. ## examining the impact of quality issues in this section we examine the impact of discovered quality issues on both training and test set of must-c en-de dataset. we first fix errors for training and test sets. then we train models on both original and clean training sets and evaluate their empirical performances on test sets with and without errors. ## related works the quality control of st datasets is an essential but hard to solve task for dataset creators. @xcite was built upon ted talks, which naturally comes with the question of inaccurate audio segmentation and audio-text alignment. other datasets like covost 2 @xcite , which was built by reading given sentences, do not possess this kind of problems. besides, must-c used gentle to conduct the forced alignment and there are other newly developed forced aligners we can use such as the one we developed in this paper and montreal forced aligner @xcite which both take advantage of deep transformer model and large audio datasets. ## conclusion in this paper, we first identify three types of error in must-c en-de dataset: inaccurate trans-#case transcript reference translation translation w/ misalignment w/o misalignment i who are they actually supposed to be informing? wen wollen sie eigentlich damit informieren? angenommen, wer sind sie eigentlich? ca: wer sollen sie eigentlich informieren? ii and so if we think about that, we have an interesting situation in hands. und deshalb, falls wir darüber nachdenken haben wir eine interessante situation vor uns. wenn wir also darüber nachdenken, haben wir eine interessante situation. wenn wir also darüber nachdenken, haben wir eine interessante situation in unseren händen.