IESTAC : E nglish- I talian Parallel Corpus for End-to-End Speech-to-Text Machine Translation (2020)

Abstract: We discuss a set of methods for the creation of IESTAC: a English-Italian speech and text parallel corpus designed for the training of end-to-end speech-to-text machine translation models and publicly released as part of this work. We first mapped English LibriVox audiobooks and their corresponding English Gutenberg Project e-books to Italian e-books with a set of three complementary methods. Then we aligned the English and the Italian texts using both traditional Gale-Church based alignment methods and a recently proposed tool to perform bilingual sentences alignment computing the cosine similarity of multilingual sentence embeddings. Finally, we forced the alignment between the English audiobooks and the English side of our textual parallel corpus with a text-to-speech and dynamic time warping based forced alignment tool. For each step, we provide the reader with a critical discussion based on detailed evaluation and comparison of the results of the different methods.

## introduction traditionally, most research on machine translation has been concerned with text-to-text systems. however, there is an increasing interest in speech translation. speech translation usually refers to the task of translating source language audio signals into a text spoken in a target language. traditionally, it has been tackled by cascaded st (speech translation) models that concatenates three technologies: asr (automatic speech recognition), mt (machine translation), and tts (text-to-speech). latency and error propagation are two intrinsic drawbacks of cascaded st models @xcite . end-to-end speech-to-text machine translation, usually also referred to as direct speech translation, avoid error propagation and reduce latency by directly translating source language audio signals into target language texts. a variety of end-to-end machine and deep learning architectures @xcite @xcite @xcite have been proposed to infer patterns from a first sequence (source language audio utterances) and a second sequence (target language textual translations). training these models require a large amount of source language audio utterances paired up with their textual translations. furthermore, since pre-training the encoder on asr seems to improve the quality of the st results @xcite , the training data should preferably also include source language audio utterances paired up with their transcriptions. we discuss the creation of iestac (italian-english speech and text audiobooks corpus), designed for training english-to-italian speech-to-text machine translation models. it is publicly available foot_0 and composed of around 130 hours of english speech aligned with its transcription and italian textual translation at a sentence level. for a more detailed description of the corpus, see @xcite . our objective is to provide the readers with a methodological contribution for the creation of corpora designed for end-to-end speech-to-text machine translation training. we describe a pipeline to semi-automatically collect audio-textual data from the web and automatically align them. alignment is performed as a two step process. we first perform bilingual sentences alignment between the english text and its italian textual translation. then we force the alignment between the english audio and the english text already aligned with its italian textual translation. for each step we discuss different possible tools, and evaluate the results, allowing us to give recommendations for tools to use for creating new corpora for other languages. ## related work there have been some previous work on creating resources for end-to-end speech-to-text machine translation @xcite @xcite . there are also other related efforts, including computational language documentation for low-resource languages @xcite , multilingual speech corpora creation @xcite , and multi-modal corpora creation @xcite . @xcite created a speechto-text corpus of 5 thousands triplets of mboshi speech, mboshi transcription, and french textual translations. speech elicitation from text was done manually by three qualified speakers. @xcite created a large corpus of aligned text, speech, and pronounciation for 700 languages, with texts from the bible. the average duration for each language is 2 hours. @xcite created a multimodal corpus by aligning at a word-level 2000 hours of english instructional youtube videos with their subtitles. portuguese textual translations were added by paying bilingual english-portuguese speakers. augmented librispeech @xcite seem to be the first corpus designed for training end-to-end english-to-french speech-totext machine translation systems. it was created by collecting public domain audiobooks and e-books from the web and automatically align them. a similar approach was used by @xcite to create librivoxdeen, a corpus for german-to-english speech translation. most recent works have been focused on multilingual corpora creation for speech-to-text machine translation: must-c @xcite , europarl-st, covost @xcite and covost2 @xcite ## corpus creation due to the fact that we are interested in releasing a freely available corpus with a permissive licence, we mainly follow the approach proposed by @xcite : ## text collection our first challenge was to identify freely available italian e-books corresponding to available english e-books. as a starting point, we used part of the librispeech metadata, more specifically the list of the gutenberg project english e-books titles. sometimes it might happen that a single book was published several times with different titles. therefore, we pre-processed the english titles list using regular expressions to increase the number of possible titles. we manually found patterns that indicate the presence of alternative titles, subtitles, or publication specific information. these patterns were used to augment the possible titles. for instance, many gutenberg project english e-book titles contained two or more possible titles separated by the sub-string , or, (e.g. "tom swift and his sky racer, or, the quickest flight on record"). at the end of the pre-processing step, the list of english titles was augmented with the inclusion of "tom swift and his sky racer" and "the quickest flight on record" as two individual list elements. ## forced alignment forced alignment is the process to return time intervals matching word or sentence utterances in a audio file with their corresponding strings in a parallel text file. we forced the alignment between 373 librivox wav files and the english side of our textual parallel corpus, using aeneas foot_12 rather than gentle (as in kocabiyikoglu et al. ( 2018 )). ## corpus statistics and structure iestac 17 contains 60561 triplets of english audio, english source text, and italian textual translation. statistics are given in in each chapter folder, there are several triplets of files. the alignment is preserved by the basename notation: each element of the triplet has a base-name composed of three concatenated integers (e-bookid, chapterid, segmentid). the file-17 https://github.com/ giuseppe-della-corte/iestac 18 parallel.it and parallel.en contain respectively: total tokens (1425072 -1577118), unique tokens @xcite name extension is used to disambiguate between the audio, the italian textual translation, and the english source text (e.g. 83 07 33.wav, 83 07 33.it, 83 07 33.en). we also provide the users with a sql database to allow them to query the corpus according to their needs. ## future work in future work, we want first of all to perform an extrinsic evaluation of our corpus on both automatic speech recognition and end-to-end speech-to-text machine translation tasks. a possible tool for that might be fbk-fairseq-st foot_14 . we also want to further improve the quality of the alignments by filtering out low-quality ones. for such task we might investigate the use of alignment scores, character error rate, language model perplexity, sentence embeddings similarity scores, and length ratio heuristic. another direction of interest is to work towards a full taxonomy for collecting speech-to-text datasets for machine translation. ## conclusion we have explored methods for creating a bilingual corpus with english speech and text, and italian text. the corpus collection is based on available english speech and text in the librivox and the project gutenberg collection of audiobooks and e-books, which we mapped to italian texts in this work. we explored and evaluated a number of methods for the different steps needed in the corpus creation, which might guide future work in creating corpora for other language pairs. for the text collection, we needed to map english books to their equivalent italian translations. we proposed three methods based on wikidata matching, wikipedia matching, and mt of book titles. we found that the three methods were complementary, each contributing some unique titles. the next step was bilingual sentence alignment, for which we found that vecalign, a method based on computing the cosine similarity of consecutive sentence embeddings, outperformed the traditional gale-church method when dealing with difficult alignments. finally we argued for the use of a tts dynamic time warping system for forcing the alignment between english speech and text, and showed that the results were of high quality. as part of this work we release iestac