Is “moby dick” a Whale or a Bird? Named Entities and Terminology in Speech Translation (2021)

Abstract: Automatic translation systems are known to struggle with rare words. Among these, named entities (NEs) and domain-specific terms are crucial, since errors in their translation can lead to severe meaning distortions. Despite their importance, previous speech translation (ST) studies have neglected them, also due to the dearth of publicly available resources tailored to their specific evaluation. To fill this gap, we i) present the first systematic analysis of the behavior of state-of-the-art ST systems in translating NEs and terminology, and ii) release NEuRoparl-ST, a novel benchmark built from European Parliament speeches annotated with NEs and terminology. Our experiments on the three language directions covered by our benchmark (en→es/fr/it) show that ST systems correctly translate 75–80% of terms and 65–70% of NEs, with very low performance (37–40%) on person names.

## introduction the translation of rare words is one of the main challenges for neural machine translation (nmt) models @xcite . among rare words, named entities (nes) and terminology are particularly critical: not only are they important to understand the meaning of a sentence @xcite , but they are also difficult to handle due to the small number of valid translation options. while common words can be rendered in the target language with synonyms or paraphrases, nes and terminology offer less expressive freedom, which is typically limited to one valid option. under these conditions, translation errors often result in blatant (meaningless, hilarious, or even offensive) errors, which jeopardize users' trust in the translation system. one example is "moby dick" (in lower case, as in the typical output of a speech recognition system): google translate 1 returns mazikó poulí (massive bird) for 1 accessed on the 27th april 2021. greek, while the translation contains profanities for other languages like hungarian or italian. previous works in nmt tried to mitigate the problem by: i) integrating knowledge graphs (kg) into the models @xcite @xcite , ii) exploiting dedicated modules for ne translation @xcite , or iii) adding ne tags to the source @xcite . for terminology, the so far proposed dictionary-based approaches @xcite @xcite @xcite share the idea of enriching the source sentence with terms' translations found in a dictionary (e.g. "this is a term"→"this is a #term#término#"). the problem is even more challenging in automatic speech recognition (asr) and speech-to-text translation (st), where a lookup into kgs or dictionaries is not feasible due to the different modality of the input (an audio signal rather than text). as regards nes, the few existing studies @xcite are all limited to asr, for which two benchmarks are available @xcite , while suitable benchmarks do not even exist for st. the situation is similar for terminology: few annotated test sets exist for mt @xcite @xcite , but none for st, which so far has remained unexplored. in light of the above, the contribution of this work is twofold: (1) we present the first investigation on the behavior of state-of-the-art st systems in translating nes and terms, discussing their weaknesses and providing baseline results for future comparisons; (2) we release the annotated data that made our study possible. our test set -neuroparl-st -is derived from europarl-st @xcite and covers three language pairs: @xmath0. it relies on the europarl-st (audio, transcript, translation) triplets and en-riches their textual portions with ne and terminology annotation. besides being the first benchmark of this type for st, it can also be used for the evaluation of ne/terminology recognition (asr) and translation (mt). the dataset is available at: ict.fbk.eu/neuroparl-st/. ## speech translation models our goal is to assess the capability of state-of-theart st systems to properly translate nes and terminology present in an utterance. to this aim, we compare instances of the two main approaches. one is the traditional cascade approach @xcite , which consists of a pipeline where an asr model produces a transcript of the input audio and an mt model generates its translation. the other is the so-called direct approach @xcite , which relies on a single neural network that maps the audio into target language text bypassing any intermediate symbolic representation. the two approaches have inherent strengths and weaknesses @xcite . cascade solutions can exploit sizeable datasets for the asr and mt subcomponents, but rely on a complex architecture prone to error propagation. direct models suffer from the paucity of training data, but avoid error propagation and can take advantage of unmediated access to audio information (e.g. prosody) during the translation phase. in recent years, after a long dominance of the cascade paradigm, the initially huge performance gap between the two approaches has gradually closed @xcite . our cascade system integrates competitive transformer-based @xcite asr and mt components built from large training corpora. specifically, the asr model is trained on librispeech @xcite , tedlium v3 @xcite and mozilla common voice, foot_0 together with (utterance, transcript) pairs extracted from must-c @xcite , europarl-st (iranzo-sánchez et al., 2020), and covost 2 @xcite st corpora. asr outputs are post-processed to add true-casing and punctuation. the mt model is trained on data collected from the opus repository, foot_1 amounting to about 19m, 28m, and 45m parallel sentence pairs respectively for en-es, en-fr, and en-it. our direct model has the same transformer-based architecture of the asr component used in the cascade system. it exploits data augmentation and knowledge transfer techniques successfully applied by participants in the iwslt-2020 evaluation campaign @xcite @xcite and it is trained on must-c, europarl-st and synthetic data (∼1.5m pairs for each language direction). systems' performance is shown in ## evaluation data: neuroparl-st to the best of our knowledge, freely available ne/term-labelled st benchmarks suitable for our analysis do not exist. the required resource should contain i) the audio corresponding to an utterance, ii) its transcript, iii) its translation in multiple target languages (three in our case), and iv) ne/term annotation in both transcripts and target texts. currently available mt, st, asr, ne and terminology datasets lack at least one of these key components. for example, most mt corpora (e.g. europarl) lack both the audio sources and ne/terminology annotations. the very few available mt corpora annotated with ne/terminology still lack the audio portion, and extending them to st would require generating synthetic audio, which is known to be problematic for models' performance. for these reasons, we preferred to create a the @xmath1, which are mainly derived from the same original speeches. the result is a multilingual benchmark featuring very high content overlap, thus enabling cross-lingual comparisons. ne annotation. we used the 18 tags and the annotation scheme defined by the guidelines ("ontonotes named entity guidelines -version 14.0") used to annotate the ontonotes5 corpus @xcite . the annotation was carried out manually by a professional interpreter with a multi-year experience in translating from english, french and italian into spanish the verbatim reports of the european parliament plenary meetings. this guarantees the high level of language knowledge and domain expertise required to ensure maximum quality and precision. to ease the task, the annotator was provided with transcripts and translations automatically pre-annotated with the bert-based ner model foot_2 available in deep-pavlov @xcite . human annotation was then conducted in parallel on the three test sets by labelling, for each audio segment, the english transcript and the three corresponding translations. to check annotations' reliability, all the english transcripts were also independently labelled by a second annotator with a background in linguistics and excellent english knowledge. inter-annotator agreement was calculated in terms of complete agreement, i.e. the exact match of the whole ne in the two annotations. the resulting dice coefficient foot_3 @xcite amounts to 93.87% and can be considered highly satisfactory. for the subset of nes for which complete agreement was found (1,409 in total), we also computed the agreement on labels' assignment with the kappa coefficient (in scott's π formulation) @xcite . the resulting value is 0.94, which corresponds to "almost perfect" agreement according to its standard interpretation (landis and @xcite . terminology annotation. similar to @xcite , terminology was automatically extracted by exploiting the iate term base. 6 each entry in iate has an identifier and a language code. entries with the same identifier and different language codes represent the translations of a term in the corresponding languages. to annotate our parallel texts, we first removed stop-words and lemmatized the remaining words and iate entries. 7 then, for each parallel sentence, we marked as terms only those words in the source and the target side that were present in iate with the same identifier. this source/target match is essential to avoid the annotation of words that are used with a generic, common meaning but, being polysemic, can be technical terms in different contexts (e.g. the word "board" can refer to a tool or to a committee). checking the presence of the corresponding translation in the target language disambiguates these cases, leading to a more accurate annotation. ne and term annotations were merged into a single test set using bio @xcite as span labeling format. had a word been tagged both as term and ne, the latter was chosen favoring the more reliable manual annotation. ## results we use our benchmark to measure systems' ability to handle nes and terminology. besides comparing the two st models described in section 2, we extend our evaluation to the asr and mt sub-components (the latter being fed with human transcripts) of the cascade system. as shown in for the sake of completeness, case-sensitive ne/term accuracy is also given in ## conclusions while previous st research has focused on improving overall systems' performance, little has been done to evaluate the existing paradigms in relation to well known specific problems in automatic translation at large. translating rare words is no exception, also due to the dearth of suitable labelled benchmarks. to fill this gap, we focused on named entities and terminology, which combine the problems inherent to low frequency in the training data with the difficulty of recognizing them in the audio and mapping their meaning into few valid options. we created neuroparl-st, an annotated benchmark covering three language directions, and used it for the first comparison of state-of-the-art cascade and direct st systems on ne and term translation. our results show that nes, especially person names, are in general more difficult to handle than terminology.