C harles Locock, Lowcock or Lockhart? Offline Speech Translation: Test Suite for Named Entities (2024)

Abstract: Generating rare words is a challenging task for natural language processing in general and in speech translation (ST) specifically. This paper introduces a test suite prepared for the Offline ST shared task at IWSLT. In the test suite, corresponding rare words (i.e. named entities) were annotated on TED-Talks for English and German and the English side was made available to the participants together with some distractors (irrelevant named entities). Our evaluation checks the capabilities of ST systems to leverage the information in the contextual list of named entities and improve translation quality. Systems are ranked based on the recall and precision of named entities (separately on person, location, and organization names) in the translated texts. Our evaluation shows that using contextual information improves translation quality as well as the recall and precision of NEs. The recall of organization names in all submissions is the lowest of all categories with a maximum of 87.5 % confirming the difficulties of ST systems in dealing with names.

## introduction generating rare words is a big challenge for several natural language processing (nlp) tasks such as machine and speech translation and speech recognition. rare words are those terms that have a low frequency in the training data and include, among others, named entities (ne), i.e. names of persons, organizations, and locations, acronyms and abbreviations, and domain-specific terms. these words carry a huge amount of the information of a sentence @xcite and their wrong realization in a text can significantly impact the user's understanding and experience. in machine translation (mt), there has been a significant effort in making the translation system able to translate better the rare words @xcite @xcite . this becomes crucial when trans-lations serve as a base for upstream tasks like summarization, errors in those named entities can introduce wrong attributions or overall misleading information. to improve the accuracy of translating named entities correctly one could either integrate a knowledge graph @xcite or use ne tags in the source sentence to make the nmt system aware of the nes @xcite @xcite . for automatic speech recognition (asr) the problem with rare words and nes is even harder since with speech the system has to handle an additional modality. similar to nmt there is a lack of training data for those entities and in addition to that, the pronunciation of named entities is often different compared to other words. current state-ofthe-art approaches tackle this problem using contextual biasing @xcite where the asr system is provided with contextual information which can be a list of named entities. the work is usually distinguished in a shallow-fusion, where the actual asr model is untouched and only modifications are added at inference time @xcite and a deep-fusion approach, where a context mechanism is trained and later used as a black-box @xcite @xcite @xcite . in speech translation (st), addressing the modality problems encountered in asr and the lack of alternative translations for nes in neural machine translation simultaneously increases the complexity of the problem. there is already existing work exploring the capability of st system handling nes @xcite . similar to their work also this test suite concentrates on evaluating the accuracy of translating named entities for person, organization and location names. additional to gaido et al. also the precision in translating named entities of the systems is evaluated. furthermore contextual information is given per talk as a list of named entities to evaluate if a system can utilize this information for the translation task. it has been shown that the main factors for a cascaded system might be the frequency of words occurring in the training and foreign words with different pronunciations @xcite . they suggest tackling the first factor by using more data, synthetic data, or fine-tuning on in-domain data. the second factor is tackled by using multilingual speech data to increase the variety seen for phoneme-to-grapheme mappings during training. additionally, there has been work incorporating a list of named entities into a direct st model to improve the accuracy for ne translation @xcite based on the clas approach for contextual asr @xcite . we proposed a test suite for the offline st shared task at iwslt to draw attention to the problem of ne translation in speech translation. the test suite was used to evaluate the ability of st systems to translate nes in the english-german ted test set accurately. the test suite provides contextual information in the form of a list of source language nes that may or may not be present in the source spoken audio. the aim is to assist the st system in improving translation quality. this paper introduces the test suite and examines the performance of different submitted st systems on our test. our findings indicate that st systems encounter difficulties when translating nes, but the list of nes can help enhance the performance when utilized. for this reason, in addition to the classic test audio for the english into german translation direction, contextual information is available in textual form. this information might be used to mitigate translation errors on these contextual terms. the context information was given as a list of entities per english audio file. to emulate real scenarios, where large lists can be used without any adaptation to specific audio, some entities that were not present in that audio were added as distractors. the goal of each participant and system is to distill the correct information from the list and use it to improve translation quality. ## st models all tested systems are cascaded systems that first transcribe the audio by an asr system and trans-late the transcript with an nmt system. that might be due to the fact that cascaded systems performed better than end-to-end systems for offline st in the last years' evaluations @xcite @xcite . there exist three different data conditions foot_2 : firstly constrained, where the systems are only allowed to be trained on a fixed amount of data, secondly constrained + llm where in addition a list of allowed large language model (llm) can be used and thirdly unconstrained to allow training the system and a large amount of training data. the only system incorporating the contextual information is the submission of the karlsruhe institute of technology (kit). their cascaded system uses a lora (hu et al., 2021) fine-tuned llm to 1) post-edit the asr transcript incorporating the n-best list and 2) to post-edit the mt output on document-level. only their primary (prm) submission injects contextual information in the second step by including the words into their llm prompt. the first contrastive submission (ctr1) only applies the asr post-edit step and for ctr2 both llm corrections are used but without injecting the contextual information. all unconstrained systems use a multilingual asr model -namely whipser-large-v3 @xcite ) -for transcription. as stated above also the huawei translation service center (hw-tsc) and carnegie mellon university (cmu) submitted a cascaded approach. ## results all systems' results are reported in looking closer at the unconstrained submissions one can observe that cmu's primary submission is the best-performing submission for comet, but nya's contrastive submissions achieve a better ne-recall as well as ne-precision. comparing hw-tsc's primary submission on the constrained data to the condition with llm, it achieves the highest precision for named entities in general and also has a competitive performance for the recall. from the results for kit's primary (prm) and second contrastive (ctr2) submission, it can be seen that the overall recall and precision of nes as well as the scores for person and organization names increased. this indicates that the provided context information can be useful to not only increase the general comet score but also the translation for nes. additionally, the number of appearing distractors (dt) in the translations was measured. only kit's primary submission used the provided context information and is therefore prone to copying a wrong-named entity from the provided list. nevertheless, 0 distractors were copied from the provided context list. additionally translations of an abbreviation resp. an organizational named entity, namely warif which is short for women at risk international foundation, are reported in ## conclusions in our test suite, we explored the translation of named entities for english-german st. named entities are translated correctly with a recall of approx. 92% and a precision of approx. 75% in an unconstrained, approx. 88% resp. 74 % in a constrained data condition without llms and ap-prox. 90% resp. 81% in a constrained data condition with using a llm. firstly this indicates that llms comprise contextual knowledge about named entities which is useful to translate named entities. but secondly that also suggests that there is still a gap in translating named entities correctly, especially looking at the category of organization names where when additionally using a llm the precision and recall was not improved. furthermore that might indicate that the capabilities of llm of improving the quality of named entity translation is limited due to the fact that some misreconized named entities can not be corrected without the access to audio information in a cascaded system. the given contextual information (list of named entities) improved the overall comet score as well as the recall and precision of ne translation. we are looking forward having more systems using a context list for st to see more benefits from using provided contextual information or llms using audio information for translation directly.