Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach (2024)

Abstract: Direct speech translation (ST) models often struggle with rare words. Incorrect translation of these words can have severe consequences, impacting translation quality and user trust. While rare word translation is inherently challenging for neural models due to sparse learning signals, real-world scenarios often allow access to translations of past recordings on similar topics. To leverage these valuable resources, we propose a retrieval-and-demonstration approach to enhance rare word translation accuracy in direct ST models. First, we adapt existing ST models to incorporate retrieved examples for rare word translation, which allows the model to benefit from prepended examples, similar to in-context learning. We then develop a cross-modal (speech-to-speech, speech-to-text, text-to-text) retriever to locate suitable examples. We demonstrate that standard ST models can be effectively adapted to leverage examples for rare word translation, improving rare word translation accuracy over the baseline by 17.6% with gold examples and 8.5% with retrieved examples. Moreover, our speech-to-speech retrieval approach outperforms other modalities and exhibits higher robustness to unseen speakers. Our code is publicly available.

## introduction speech translation (st) traditionally involves cascading automatic speech recognition (asr) and machine translation (mt) @xcite to convert spoken language into text in a different language. however, recent years have witnessed rapid progress in direct st models @xcite @xcite that bypass intermediate text representations for lower inference latency and reduced error propagation @xcite ). 1 : siqilii/retrieve-and-demonstration-st * equal contribution; siqi's work done while at kit despite the advancements, accurately translating rare words like person names @xcite remains a significant challenge for st systems. while infrequent, incorrect translations of rare words can severely degrade overall translation quality and even users' trust in the deployed models. rare word translation is inherently difficult for st models due to limited or absent learning signals. practically, however, valuable external resources hold the potential to address this issue. real-world scenarios often allow access to translations from past recordings on similar topics, sometimes even from the same speaker. similarly, human translators often leverage existing translations @xcite , especially for special terminologies @xcite . inspired by these observations, we ask the question: how can we improve the rare word translation performance of direct st models by leveraging an example pool that contains similar translations? the envisioned approach faces challenges in both the retrieval and translation components. first, the retrieval task is complicated by the variability of speech and the locality of rare words. as the speaking condition for the same rare word differs in every utterance, source-side feature matching as often done in text translation @xcite @xcite @xcite is not sufficient to handle the pronunciation variations. moreover, as rare words only constitute a small portion of the query and candidate utterances, the retriever must be able to locate the relevant information in long speech utterances. for the translation model, integrating retrieved utterance-translation pairs is also non-trivial. standard models trained on sentencelevel data require adaptation to ingest the examples. besides processing longer inputs, they also need to pinpoint both the acoustic features and corresponding textual translations of rare words. addressing the above challenges, we introduce a retrieval-and-demonstration framework (figure 1 ) effective for improving rare word translation accuracy of st models. specifically, we adapt standard st models to benefit from prepended examples in a way similar to in-context learning @xcite , and then build a retriever to find suitable examples. building on recent multi-modal encoders @xcite , the retriever supports multiple modalities @xcite . second, we propose an evaluation methodology to adapt standard st corpora, @xcite in this case, for targeted assessment of rare words translation ( §3.1). our main findings are: ## proposed framework our retrieval-and-demonstration framework is illustrated in figure 1 . first, a trained direct st model is finetuned to ingest examples ( §2.1), which serve as demonstrations of correctly translating the rare words in question. during inference, given an utterance containing rare words, we retrieve ( §2.2) a relevant utterance and its translation as a demonstration to guide the inference ( §2.3). ## conclusion we introduced a retrieval-and-demonstration approach to improve rare word translation accuracy in direct st. for real-world applications, e.g., translating scientific talks, we recommend adding utterances from the same speaker to the example pool and using speech-to-speech retrieval to identify examples. when feasible, one should consider incorporating an additional verification step to ensure the relevance of the retrieved sentences, by human-in-the-loop or automated techniques.