Who Are We Talking About? Handling Person Names in Speech Translation (2022)

Abstract: Recent work has shown that systems for speech translation (ST) – similarly to automatic speech recognition (ASR) – poorly handle person names. This shortcoming does not only lead to errors that can seriously distort the meaning of the input, but also hinders the adoption of such systems in application scenarios (like computer-assisted interpreting) where the translation of named entities, like person names, is crucial. In this paper, we first analyse the outputs of ASR/ST systems to identify the reasons of failures in person name transcription/translation. Besides the frequency in the training data, we pinpoint the nationality of the referred person as a key factor. We then mitigate the problem by creating multilingual models, and further improve our ST systems by forcing them to jointly generate transcripts and translations, prioritising the former over the latter. Overall, our solutions result in a relative improvement in token-level person name accuracy by 47.8% on average for three language pairs (en->es,fr,it).

## introduction automatic speech translation (st) is the task of generating the textual translation of utterances. research on st @xcite has so far focused on comparing the cascade (a pipeline of an automatic speech recognition -asr -and a machine translation -mt -model) and direct paradigms @xcite , or on improving either of them in terms of overall quality. quality is usually measured with automatic metrics such as bleu @xcite and ter @xcite , possibly corroborated by manual analyses. these metrics -as well as neural-based ones like comet @xcite ) -are relatively insensitive to errors on named entities (nes) and numbers @xcite , which instead are of paramount importance for human readers @xcite . as such, the blind pursue of higher scores can lead to systems biased toward the metrics and not targeted on real users. in addition, there are cases in which users are interested only in nes. for instance, interpreters easily craft more fluent and intelligible translations than machines @xcite , but during simultaneous sessions suffer from a high cognitive workload @xcite , to which nes and specific terminology significantly contribute @xcite @xcite . indeed, these elements i) are hard to remember @xcite , ii) can be unknown to interpreters and difficult to recognize @xcite , and iii) differently from other types of words, usually have one or few correct translations. for this reason, modern computer-assisted interpreting (cai -fantinuoli 2017) tools aim at automatically recognizing, displaying, and translating nes and terms. however, current solutions rely on predefined dictionaries to identify and translate the elements of interest @xcite , preventing them to both generalize and disambiguate homophones/homonyms. this would be instead possible using st system, but they need to reliably recognize and translate nes and terms, without generating wrong suggestions that are even harmful @xcite . in contrast with these needs, @xcite recently showed on their newly created benchmark -neuroparl-st -that both asr models (and thus cascade st systems) and direct st systems perform poorly on person names, with transcription/translation accuracy of ~40%. hence, as a first step toward st systems more targeted for human needs, and in particular toward the long-term goal of integrating st models in assistant tools for live interpreting, this work focuses on i) identifying the factors that lead to the wrong transcription and translation of person names, and ii) proposing dedicated solutions to mitigate the problem. to achieve these objectives, our first contribution ( §3.1) is the annotation foot_0 of each person name occurring in neuroparl-st with information about their nationality and the nationality of the speaker (as a proxy of the native language) -e.g. if a german person says "macron is the french president", the speaker nationality is german, while the referent nationality is french. drawing on this additional information, our second contribution ( §3.2-3.3) is the analysis of the concurring factors involved in the correct recognition of person names. besides their frequency, we identify as key discriminating factor the presence in the training data of speech uttered in the referent's native language (e.g. french in the above example). this finding, together with an observed accuracy gap between person name transcription (asr) and translation (st), leads to our third contribution ( §4): a multilingual st system that jointly transcribes and translates the input audio, giving higher importance to the transcription task in favour of a more accurate translation of names. our model shows relative gains in person name translation by 48% on average on three language pairs (@xmath0,fr,it), producing useful translations for interpreters in 66% of the cases. ## related work when the source modality is text, person names can often be "copied", i.e. replicated unchanged, into the output. this task has been shown to be well accomplished by both statistical and neural translation systems @xcite . on the contrary, when the source modality is speech (as in asr and st), systems struggle due to the impossibility to copy the audio source. the recognition of person names from speech is a complex task that has mostly been studied in the context of recognizing a name from a pre-defined list, such as phone contacts @xcite @xcite . the scenario of an open or undefined set of possible names is instead under-explored. few studies @xcite focus on comparing endto-end and cascade approaches in the transcription and recognition of nes from speech. they do not directly investigate person names though, as they do not disaggregate their results by ne category. similarly, @xcite explore ne recognition from speech in low-resource languages, and propose two end-to-end methods: one adds a tag after each word in the generated text to define whether it is a ne or not, and one uses a dedicated decoder. however, they do not provide specific insights on the system ability to correctly generate person names and limit their study to asr, without investigating st. closer to our work, @xcite highlight the difficulty of asr/st neural models to transcribe/translate nes and terminology. although they identify person names as the hardest ne category by far, they neither analyse the root causes nor propose mitigating solutions. ## factors influencing name recognition as shown in @xcite , the translation of person names is difficult both for direct and cascade st systems, which achieve similar accuracy scores (~40%). the low performance of cascade solutions is largely due to errors made by the asr component, while the mt model usually achieves nearly perfect scores. for this reason, henceforth we will focus on identifying the main issues related to the transcription and translation of person names, respectively in asr and direct st. we hypothesize that three main factors influence the ability of a system to transcribe/translate a person name: i) its frequency in the training data, as neural models are known to poorly handle rare words, ii) the nationality of the referent, as different languages may involve different phonemeto-grapheme mappings and may contain different sounds, and iii) the nationality of the speaker, as non-native speakers typically have different accents and hence different pronunciations of the same name. to validate these hypotheses, we inspect the outputs of transformer-based @xcite asr and st models trained with the configuration defined in @xcite . for the sake of reproducibility, complete details on our experimental settings are provided in the appendix. foot_1 ## improving person name translation the previous section has uncovered that only two of the three considered factors actually have a tangible impact: the frequency in the training set, and the referent nationality. the first issue can be tackled either by collecting more data, or by generating synthetic instances @xcite . fine-tuning the model on additional material is usually a viable solution in the use case of assisting interpreters since, during their preparation phase, they have access to various sources of information @xcite , including recordings of previous related sessions. focusing on the second issue, we hereby explore i) the creation of models that are more robust to a wider range of phonetic features and hence to names of different nationalities ( §4.1), and ii) the design of solutions to close the gap between asr and st systems attested by previous work @xcite and confirmed by our preliminary results shown in ## error analysis while the goal is the correct rendering of person names, not all the errors have the same weight. for interpreters, for instance, minor misspellings of a name may not be problematic, an omission can be seen as a lack of help, but the generation of a wrong name is harmful, as potentially distracting and/or confusing. to delve into these aspects, we first carried out a manual analysis on the asr outputs ( §5.1) and then compared the findings with the same analysis on st outputs ( §5.2). ## conclusions humans and machines have different strengths and weaknesses. nonetheless, we have shown that when it comes to person names in speech, they both struggle in handling names in languages they do not know and names that they are not used to hear. this finding seems to insinuate that humans cannot expect help from machines in this regard, but we demonstrated that there is hope, moving the first steps toward st systems that can better handle person names. indeed, since machines are faster learners than humans, we can train them on more data and more languages. moreover, we can design dedicated architectural solutions aimed to add an inductive bias and to improve the ability to handle specific elements. along this line of research, we have shown that a multilingual st model, which jointly predicts the transcript and conditions the translation on it, has relative improvements in person name accuracy by 48% on average. we also acknowledge that much work is still needed in this area, with large margin of improvements available, especially to avoid the two most common type of errors pointed out by our analysis: omissions and replacements with different person names. using mini-batches containing 5,000 tokens, and accumulating the gradient for 16 mini-batches. we average 5 checkpoints around the best on the validation loss. all trainings last ∼4 days for the multilingual systems, and ∼3 days for the base system.