Evaluating Multilingual Long-Context Models for Retrieval and Reasoning (2024)

Abstract: Recent large language models (LLMs) demonstrate impressive capabilities in handling long contexts, some exhibiting near-perfect recall on synthetic retrieval tasks. However, these evaluations have mainly focused on English text and involved a single target sentence within lengthy contexts. Our work investigates how LLM performance generalizes to multilingual settings with multiple hidden target sentences. We create a new dataset – mLongRR – to comprehensively evaluate several multilingual long-context LLMs on retrieval and reasoning tasks across five languages: English, Vietnamese, Indonesian, Swahili, and Somali. These languages share the Latin script but belong to distinct language families and resource levels. Our analysis reveals a significant performance gap between languages. The best-performing models such as Gemini-1.5 and GPT-4o, achieve around 96% accuracy in English to around 36% in Somali with a single target sentence. However, this accuracy drops to 40% in English and 0% in Somali when dealing with three target sentences. Our findings highlight the challenges long-context LLMs face when processing longer contexts, an increase in the number of target sentences, or languages of lower resource levels.

## introduction the ability to model long context sequences spanning tens of thousands of tokens is crucial for tasks such as summarization and question answering based on long documents such as books and reports, and code generation at the repository level. recent advancements in large language models (llms) have focused on improving their capabilities in processing long context information @xcite @xcite . long-context language models, particularly multilingual ones, have the potential to enable remarkable progress in various applications by under-standing lengthy textual data across different languages. an example of this potential was recently demonstrated by the newly introduced gemini-1.5 pro model @xcite which leveraged its long-context window for in-context learning. by including a grammar manual in its context window, the model was able to learn to translate from english to kalamang, an extremely low-resource language with fewer than 200 speakers @xcite . such examples highlight the potential of long-context models in tackling challenging tasks in low-resource languages, where data scarcity has traditionally been a barrier. current methods for evaluating long-context llms primarily focus on english text. this has led to a severe lack of insights into their performance across diverse languages. evaluating multilingual performance is crucial, not only for informing the development of effective models that serve diverse communities @xcite , but also for developing safer models as research suggests that llms tend to generate more unsafe and irrelevant responses to malicious prompts in lower-resource languages @xcite . however, there is a notable lack of multilingual benchmarks hindering our understanding of how longcontext llms perform across different linguistic contexts. to address this gap, we present the first comprehensive study of long-context llms in multilingual settings leveraging evaluation frameworks relying on synthetic tasks @xcite @xcite @xcite . although the task is partially synthetic, we create a new dataset -mlongrr foot_0 -consisting of naturally occurring text and human translated data, making the setup as close to real-world setting while creating a controlled environment for comparing model performance across languages. in addition to retrieval tasks, we introduce a new reasoning task where the models not only need to retrieve relevant items but also compare them with each other. for this, the models must keep track of these items in a longcontext scenario, allowing us to analyze the models' reasoning capabilities. we conduct a systematic evaluation of six different llms across five languages of varying resource levels. our research aims to answer the following two questions: (1) how do the long context capabilities of llms compare in retrieval and reasoning tasks in multilingual contexts? (2) are there significant performance differences between llms in multilingual contexts? some of our key findings are summarized as follows: we hope that the findings of our study will contribute to a deeper understanding of current longcontext evaluation in multilingual contexts and encourage the development of more effective longcontext multilingual models. ## related work recent advancements in language models have focused on improving their ability to recall and reason over fine-grained information from tens of thousands of tokens of context @xcite . due to shortage of really long-context benchmarks, evaluation is typically focused on synthetic tasks such as passkey retrieval @xcite @xcite @xcite or needle in a haystack @xcite @xcite ) which measure a model's ability to accurately recall information from a vast corpus of data. recently, gemini 1.5 @xcite and claude-3 (anthropic, 2024) models reported nearperfect recall on the needle in a haystack task. prior work has also studied perplexity but a low perplexity score has shown to not necessarily indicate proficiency in handling long contexts or reflect the model's performance on sequence-level tasks in real applications @xcite @xcite . furthermore, most of these studies have been limited to english only texts. although some long-context real-world benchmarks have been recently introduced, they are also limited to english @xcite , and while some bilingual english/chinese @xcite @xcite datasets offer a slight improvement, due to the effort-intensive nature of dataset creation, they are limited to a very small number of languages. ## results and discussion this section presents our results of four main experiments: (1) performance of different models, (2) performance with respect to varying needle depths and haystack lengths, (3) performance across five different languages, and (4) reasoning over magic numbers and world cities. for the first three experiments, we analyze the models' responses when asked to retrieve and reason about the magic number. in the last experiment, we compare the models' performance when asked about the magic number or the city. ## conclusion we introduce a new dataset designed to study longcontext retrieval and reasoning tasks across multiple languages. by evaluating six llms on their ability to process text in five languages with varying resource levels, using naturally occurring text and a needle-in-a-haystack paradigm with different numbers of needles, we discovered key insights. notably, we observed a significant decline in performance, particularly when dealing with longer contexts, an increased number of needles, or lower resource levels. even seemingly simple synthetic tasks like needle-in-a-haystack revealed substantial performance disparities. our findings highlight the need to develop not only more effective long-context models but also improved tokenization schemes for the effective processing of lowresource languages.