G en T ranslate: Large Language Models are Generative Multilingual Speech and Machine Translators (2024)

Abstract: Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by its reduced representation errors and incorporated external knowledge. However, both translation tasks typically utilize beam search decoding and top-1 hypothesis selection for inference. These techniques struggle to fully exploit the rich information in the diverse N-best hypotheses, making them less optimal for translation tasks that require a single, high-quality output sequence. In this paper, we propose a new generative paradigm for translation tasks, namely GenTranslate, which builds upon LLMs to generate better results from the diverse translation versions in N-best list. Leveraging the rich linguistic knowledge and strong reasoning abilities of LLMs, our new paradigm can integrate the diverse N-best candidates to generate a higher-quality translation result. Furthermore, to support LLM finetuning, we build and release a HypoTranslate dataset that contains over 592K hypotheses-translation pairs in 11 languages. Experiments on various speech and machine translation benchmarks (e.g., FLEURS, CoVoST-2, WMT) demonstrate that our GenTranslate significantly outperforms the state-of-the-art model.

## introduction recent advances in large language models (llms) have attracted a surge of research interest due to their strong abilities in logical reasoning and language generation @xcite @xcite . these models have achieved surprisingly wide-ranging success across various natural language processing (nlp) tasks @xcite @xcite . in the realm of nlp, the translation tasks, which encompasses speech and machine translation (st & mt), hold significant practical importance for global communication. similar to other nlp tasks, translation tasks also gain a notable progress thanks to the recent advancement of llms @xcite . in the domain of speech translation, whisper @xcite demonstrates superior performance by collecting 680k-hour data for web-scale model training. @xcite integrates both text-and speech-based language models into a unified architecture to process and generate text and speech, thereby augmenting speech translation performance to a great extent. on the other hand, llms also show remarkable ability in machine translation. nllb @xcite is the first to extend llms' linguistic capability to over 200 languages. bigtranslate @xcite ) is finetuned on llama @xcite with multilingual instruction tuning, which achieves comparable performance to @xcite and google translate. most recent work proposes seamlessm4t @xcite , a foundational multilingual and multitask model that can translate across speech and text, which achieves the state-of-the-art on both st and mt tasks on various public datasets. despite the superior performance, most existing translation models employ the typical beam search algorithm for inference and select the top-1 hypothesis as final output (see fig. 1 (a) ), following that in automatic speech recognition (asr) @xcite . however, this strategy discards the 2 to n -best hypotheses that could be advantageous to the generation of ground-truth translation. as illustrated in fig. 2 , the discarded 2 to n -best hypotheses contain abundant semantic information that is the key to composite the ground-truth utterance, while the 1-best hypothesis lacks this part of information. as a result, the typical top-1 hypothesis selection is sub-optimal to the translation tasks that require a single informative and high-quality output sequence @xcite . inspired by the recent works on llms-enhanced asr @xcite @xcite , we propose a new generative paradigm for translation tasks, namely gentranslate (see fig. 1 (b) ). leveraging the rich linguistic knowledge and strong reasoning ability of llms, our paradigm integrates the diverse translation versions in the n -best list from foundation model to generate a higher-quality translation result. furthermore, in order to support llm finetuning, we also build and release a hypo-translate dataset that contains over 592k pairs of n -best hypotheses and ground-truth translation in 11 languages. experimental evidence on various st and mt benchmarks (e.g., fleurs, covost-2, wmt) demonstrate that our proposed gentranslate significantly outperforms the state-of-the-art model with efficient llm finetuning. our contributions are summarized as follows: ## methodology in this section, we introduce the proposed method. first, we describe the latest foundational translation model, seamlessm4t, which we employ for beam search decoding and hypotheses generation ( §3.1). then, we introduce our llms-based gentranslate paradigm by n -best hypotheses integration ( §3.2). finally, we present the details of our released hypo-translate dataset for gentranslate training ( §3.3). ## conclusion in this paper, we propose a generative paradigm for translation tasks, namely gentranslate, which leverages llms to integrate the diverse candidates in the decoded n -best list and generate a higherquality translation result. furthermore, we release a hypotranslate dataset to support llm finetuning, which contains over 592k hypotheses-translation pairs in 11 languages. experimental evidence on various speech and machine translation benchmarks shows that our gentranslate significantly outperforms the state-of-the-art model.