Robust Translation of F rench Live Speech Transcripts (2022)

Abstract: Despite a narrowed performance gap with direct approaches, cascade solutions, involving automatic speech recognition (ASR) and machine translation (MT) are still largely employed in speech translation (ST). Direct approaches employing a single model to translate the input speech signal suffer from the critical bottleneck of data scarcity. In addition, multiple industry applications display speech transcripts alongside translations, making cascade approaches more realistic and practical. In the context of cascaded simultaneous ST, we propose several solutions to adapt a neural MT network to take as input the transcripts output by an ASR system. Adaptation is achieved by enriching speech transcripts and MT data sets so that they more closely resemble each other, thereby improving the system robustness to error propagation and enhancing result legibility for humans. We address aspects such as sentence boundaries, capitalisation, punctuation, hesitations, repetitions, homophones, etc. while taking into account the low latency requirement of simultaneous ST systems.

## introduction speech translation is the task of converting speech utterances given in a source language into text written in a different, target language. conventional st systems employ a two-step cascaded pipeline composed of asr and mt modules @xcite ; @xcite . one of the main drawbacks of these systems is error propagation, a problem that has received considerable attention in the last years @xcite ; @xcite . multiple research efforts have tried to tightly integrate both modules by using n-best lists or word lattices @xcite ; @xcite ; @xcite . these systems are nowadays strongly challenged by direct approaches employing a single model to translate the input speech signal, where all network components are jointly trained to maximize translation performance without the need for an intermediate readable representation @xcite ; @xcite ; @xcite . despite their architectural simplicity, reduced information loss and minimal error propagation of direct systems, cascaded solutions are still not widely used, mainly because of the data scarcity problem. moreover, industry applications usually display speech transcripts alongside translations, making cascade approaches more realistic and practical. within the standard cascaded framework, researchers have encountered many challenges, mainly based on the fact that asr transcripts exhibit very different features from those of the texts used to train neural machine translation (nmt) networks. while nmt models are often trained with clean and well-structured text, spoken utterances contain multiple disfluencies and recognition errors which are not well modeled by nmt systems. in addition, asr systems do not usually predict sentence boundaries or capital letters correctly, as they are not reliably accessible as acoustic cues @xcite ; @xcite . while asr output is sufficient for many applications, where speech segments are usually short, it is difficult to use in applications that transcribe long speech segments @xcite . typical asr systems segment the input speech using only acoustic information, i.e., pauses in speaking, which greatly differ from the units expected by conventional mt systems. at the other end of the spectrum, systems using longer segments may span multiple sentences. this causes important translation delays, which harms the reading experience. limited translation delays are typically achieved via starting translation before the entire audio input is received, a practice that introduces important challenges @xcite ; @xcite ; @xcite . in this work, we consider live speech-to-text translation, a task closely resembling simultaneous interpreting, that performs multilingual translations in real time and that has recently been in increasing demand in a variety of settings (radio and television broadcasts, movies, podcasts, online meetings, conferences and lectures, live events, etc.). we propose a simple but efficient st system following a cascaded asr-mt pipeline for live translation of french speeches into english with focus on the political discourse domain. figure 1 shows a screenshot of our live st system interface. inspired by @xcite ; @xcite , we propose several data augmentation techniques to simulate errors generated by an asr system, thus allowing the mt system to recover from asr errors. our contributions are summarised as follows: after introducing and presenting related work, we outline the particularities of the used speech transcripts in section 2. details of the presented framework for live multilingual st are given in section 3. section 4 describes our experimental framework. results are presented in section 5. finally, section 6 concludes this work. ## speech transcripts a vast amount of audio sources are nowadays being produced on a daily basis. asr systems enable such speech content to be used in multiple applications (i.e. indexing, cataloging, subtitling, translation, multimedia content production, etc). details depend of individual asr systems but their output, commonly called transcripts, typically consist of plain text enriched with time codes. figure 2 (top) illustrates the transcript resulting from a french utterance. notice time codes and confidence scores for each record. latency records indicate pauses. punctuation partially due to absence of sentence boundaries, no punctuation marks are produced by asr systems in real time mode, a key feature for the legibility of speech transcriptions. ## live speech translation our st system is a standard cascading asr-mt pipeline, where asr outputs a french singlebest hypothesis without punctuation, lower-cased, non segmented and containing multiple recognition disfluencies. to alleviate the asr-mt mismatch we employ neural models that: repetitions inserts 1 to 3 repetitions of a word with probability inversely proportional to word length, and decreasing probability according to the number of repetition (84% chance to repeat once, 13% to repeat twice, 3% to repeat 3 times). deletions deletes a word with probability inversely proportional to word length. homophones replaces a word with a word of different orthography but similar pronunciation, according to this homophone frequency in the language, with tolerance for frequent variation in french pronunciation ([e]/[ ]). numbers replaces a string representing a number with a phonetically plausible decomposition of it (e.g. 2001 → 2 1000 1). speechify lower-cases words and strips punctuation. other options teach the model the ability to handle special tokens representing information available in asr transcripts: oovs in train replaces random words with (oov), in inference corresponds to genuine out of vocabulary words for the model. pauses in train replaces random punctuation signs with (pause), in inference corresponds to pauses detected by the asr. ## experimental setup pronounced ces alignements-là pour que le système d'intelligence artificielle fonctionne, il faut le faire sur beaucoup beaucoup de données. transcript ces alignements la pour que le système d'intelligence artificielle fonctionne il faut le faire sur beaucoup beaucoup de données fr2fr ces alignements là pour que le système d'intelligence artificielle fonctionne, il faut le faire sur beaucoup de données. fr2fr+fr2en these alignments there for the artificial intelligence system to work, it must be done on a lot of data. pronounced j'en, on voit quand même qu'il y avait des choses qui fonctionnent pas mal. transcript jean on voit quand même qu'il y avait des choses qui fonctionne pas mal fr2fr jean, on voit quand même qu'il y avait des choses qui ne fonctionnent pas mal. fr2fr+fr2en jean, we can still see that there were some things that did not work badly. figure 5 : examples where fr2fr model segments and punctuates the asr output, correcting homophones, repetition and missing words. we observe that further work could tackle multiword homophones or quasi-homophones and written rewording of speech-specific structures. transcript et c'est ici que s'est produite la faillite fondamental de l' homme (pause) si fondamental que toutes les autres en découle merci fr2en and here's the fundamental bankruptcy of the human, if all the others are thank. fr2fr et c'est ici que s'est produite la faillite fondamentale de l'homme, si fondamentale que toutes les autres en découlent. merci fr2fr+fr2en and here's the fundamental bankruptcy of man, so fundamental that all others derive from it. thank you. fr2en and this is where the fundamental human failure has taken place. so fundamental. thank you for all the other things. fr2fr:en and this is where the fundamental human failure has taken place. so fundamental. thank you for all the other things. reference [...] and here occurred man's fundamental failure, so fundamental that all other failures ensue it ..." thank you. figure 6 : example where fr2fr+fr2en achieves the best translation by correcting homophones and meaningfully segmenting (in red incorrect segmentations incurred by other models). ## experimental results as it can be seen, the fr2en model, trained on clean parallel data, exhibits the worst results. differences in training and inference data sets significantly impact performance. concerning models learned using noisy source data, best bleu performance is achieved by the fr2en model. we hypothesize that fr2fr+fr2en suffers from error propagation, which means errors introduced in the first module fr2fr can not be recovered by the fr2en module. results by fr2fr:en are very similar to those obtained by fr2fr+fr2en. despite its lower bleu score, the model fr2fr+fr2en outputs clean fr transcripts, which is an important asset for some industry applications, and can impact segmentation and translation, as can be seen in figure 6 . the presented framework delivers translations with very low delay rates. each new word supplied by the asr produces a new translation hypothesis which is immediately displayed to the user. even though the segment being decoded can fluctuate (translation changes when including additional words), as soon as an end of segment (eos) followed by a fixed number of words is predicted, the segment remains unchanged. we encountered very limited fluctuations, impacting the last words of the hypotheses being decoded. ## conclusions we presented a framework for live speech translation based on the cascaded approach. we proposed several techniques to automatically enrich clean parallel corpora with several noise types typically present in speech transcripts, thereby improving the system robustness to error propagation. we pay special attention to translation delay rates to enhance legibility for humans. results indicate the suitability of the framework presented showing important accuracy gains when compared to a baseline system and attaining very low delay rates. we plan to extend this work using a transformer with dual decoder: a system that uses a single encoder for the asr transcripts and two parallel decoders to produce a clean version of the transcript in the same language and its corresponding translation, with the ability to attend to each other. this way, we expect to obtain similar delay rates with improved translation accuracy than our best performing model, and to additionally produce clean transcripts.