Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications? (2024)

Abstract: Large language models hold significant promise in multilingual applications. However, inherent biases stemming from predominantly English-centric pre-training have led to the widespread practice of pre-translation, i.e., translating non-English inputs to English before inference, leading to complexity and information loss. This study re-evaluates the need for pre-translation in the context of PaLM2 models, which have been established as highly performant in multilingual tasks. We offer a comprehensive investigation across 108 languages and 6 diverse benchmarks, including open-end generative tasks, which were excluded from previous similar studies. Our findings challenge the pre-translation paradigm established in prior research, highlighting the advantages of direct inference in PaLM2. Specifically, PaLM2-L consistently outperforms pre-translation in 94 out of 108 languages. These findings pave the way for more efficient and effective multilingual applications, alleviating the limitations associated with pre-translation and unlocking linguistic authenticity.

## introduction large language models (llms) have become increasingly powerful, leading to their widespread application across various multilingual tasks. however, inherent biases in pre-training data, often heavily skewed towards english, have limited the performance of llms on non-english tasks @xcite @xcite . this limitation fueled the standard practice of pre-translation, where inputs are translated into english before llm inference. while bypassing the bias issue, this approach introduces complexities and risks information loss @xcite , impacting both efficiency and effectiveness. while recent studies have extensively explored the impact of pre-translation, their focus has primarily been on discriminative tasks, assessing language understanding @xcite . the influence of pre-translation on generative capabilities of llms, has been largely unexplored. furthermore, commonly employed performance metrics, when aggregated across languages can be misleading, as outlier results for individual languages can skew the averages and mask the optimal approach for most languages. this study conducts a comprehensive investigation into the effectiveness of direct inference in the source language compared to pre-translation when utilizing palm2 models, addressing the aforementioned limitations within existing research. we analyze 108 languages across six different benchmarks, including open-ended tasks to assess the impact of pre-translation on generative abilities. our findings challenge the established pre-translation paradigm @xcite , highlighting the advantages of direct inference with palm2 models. specifically, palm2-l consistently achieves superior performance with direct inference in 94 out of 108 evaluated languages. by revealing palm2's superiority with direct inference and offering robust evaluation tools, we aim to inspire further llm development that transcends pre-translation, paving the way for seamless multilingual communication. ## method our primary objective is to determine whether pretranslation remains a necessary practice for optimal llm performance, specifically focusing on palm2, which have been established as highly performant in multilingual tasks @xcite . in contrast to prior studies, we employ a novel methodology specifically tailored to capture both the nuances of performance across different languages, and asses impact of generative capabilities. to that end, the evaluation incorporates a mixture of closed and open-ended datasets. while the initial step of pre-translation involves a straightforward translation of the source language to english, evaluating the inference output requires careful consideration. in closed-ended tasks, such as multiple choice question answering (qa) @xcite , the language of the answer is mostly inconsequential. however, open-ended tasks, which assess generation, require the desired answer in the source language, i.e. evaluation in source language. this entails an additional postinference translation step to facilitate evaluation, as illustrated in figure 1 . evaluating pre-translation performance for openended tasks (text generation), presents unique challenges @xcite stemming from the reliance on lexical evaluation metrics like f1 (rajpurkar et al., 2016) and rouge @xcite . addressing these challenges becomes crucial for text summarization and attributive qa datasets utilized in our evaluation. attributive qa is a question answering task where the answer relies primarily on information present within the provided context. here, the ground truth (gt) is a substring of the context, with evaluation metrics measuring the lexical overlap between the predicted answer and the gt. this presents a potential disadvantage for pre-translation, which lacks access to the original context from which the gt was extracted. we address this with a dedicated filtering scheme prescribed in section 3. furthermore, when aggregating and comparing lexical metrics like f1 and rouge across languages, inconsistencies arise due to their sensitivity to language morphology. to address this, we propose a complementary evaluation in english. we translate both gt and direct inference results to english and calculate lexical metrics in english, regardless of the source language. this way we establish a consistent basis for comparison, eliminating the impact of language-specific variations in the metrics. finally, to facilitate a more granular analysis of multiple language performance and address potential bias introduced by a few languages dominating the average, we propose the "language-ratio" measure. this measure reports the proportion of languages where one method outperforms the other, offering a more nuanced understanding of relative strengths and weaknesses across languages. ## experimental setup this study utilizes six publicly available multilingual benchmarks for evaluating both discriminative and generative capabilities. for assessing discriminative capabilities, we employ belebele @xcite , xcopa @xcite , and xstorycloze @xcite . generative capabilities are evaluated using xlsum @xcite , tydiqa-gp @xcite , and xquad @xcite . task types, language counts, and evaluated metrics for each dataset are available in appendix 7.1. two palm2 variants are evaluated, palm2-s (bison), and palm2-l (unicorn). google trans-late api foot_0 is employed for pre-translation. refer to appendix 7.2 for detailed prompts used in each dataset. to achieve a balanced assessment in attributive qa tasks (xquad and tydiqa-gp) we employ filtering. we translate gt from the source language to english and then back to the source language, subsequently discarding any evaluation samples where the translated gt is no longer a substring of the context (details in appendix 7.5). this approach helps address the inherent bias against pretranslation, discussed in section 2 by accounting for potential discrepancies in lexical alignment introduced by translation. ## results and analysis while differences in average accuracy might appear subtle, as exemplified by the 1.1% advantage of palm2-s with direct inference on belebele, the language-ratio highlights a more decisive advantage. in this instance, 74% of languages exhibit superior performance under direct inference. this observation emphasizes the importance of looking beyond average accuracy for a nuanced understanding of model performance across languages. with this in mind and considering the apparent contradiction with recent findings on the same benchmarks @xcite , we re-examine previously reported results. in @xcite we find that while authors generally concluded that pre-translation yielded superior results across all models, adding language-ratio to the results reported on gpt4 reveals a different narrative. although average accuracy favors pre-translation, in the majority of evaluated languages better performance is achieved with direct inference (see appendix 7.3 ## conclusions we performed a comprehensive comparative analysis of direct inference and pre-translation using palm2 models on a variety of discriminative and generative tasks across multiple languages. contrary to prior research, our findings indicate that pre-translation is not universally required and, in fact, direct inference demonstrably improves performance. in our study we highlight the importance of extending the evaluation scope when assessing the impact of pre-translation. to that end, we propose a set of methods for fair comparison and aggregation of quality metrics across languages. additionally, we demonstrate the feasibility of evaluation on generative tasks, despite inherent challenges. finally, our per-language analysis reveals potential gaps and opportunities for future development in african languages, including those with relatively large speaker populations. these languages appear to be at a disadvantage compared to other low-resource languages, warranting further investigation and targeted efforts to bridge the performance gap.