Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models (2025)

Abstract: While recent large language models (LLMs) demonstrate remarkable abilities in responding to queries in diverse languages, their ability to handle long multilingual contexts is unexplored. As such, a systematic evaluation of the long-context capabilities of LLMs in multilingual settings is crucial, specifically in the context of information retrieval. To address this gap, we introduce the MultiLingual Needle-in-a-Haystack (MLNeedle) test, designed to assess a model’s ability to retrieve relevant information (the needle) from a collection of multilingual distractor texts (the haystack). This test serves as an extension of the multilingual question-answering task, encompassing both monolingual and cross-lingual retrieval. We evaluate four state-of-the-art LLMs on MLNeedle. Our findings reveal that model performance can vary significantly with language and needle position. Specifically, we observe that model performance is the lowest when the needle is (i) in a language outside the English language family, and (ii) located in the middle of the input context. Furthermore, although some models claim a context size of 8k tokens or greater, none demonstrate satisfactory cross-lingual retrieval performance as the context length increases. Our analysis provides key insights into the long-context behavior of LLMs in multilingual settings to guide future evaluation protocols. To our knowledge, this is the first study to investigate the multilingual long-context behavior of LLMs.

## introduction in recent years, large language models (llms) have demonstrated remarkable capabilities across a wide range of natural language processing tasks, including text generation, translation, and questionanswering. a critical aspect of these models figure 1 : monolingual long-context performance (accuracy in radial axis) for various llms averaged across different context sizes (4k, 8k, 16k, and 32k). we observe a considerable drop in performance for all languages except english, suggesting that multilingual llms struggle to process non-english (or non-latin) long input contexts. a recent study @xcite ) has shed light on the potential and limitations of llms in handling extended sequences. they evaluate various attention mechanisms and model architectures, revealing that while some models can technically manage long contexts, their effective utilization of this capacity is often suboptimal. furthermore, @xcite highlighted a major problem faced by llms while handling long contexts: a marked decline in performance when the relevant information is located in the middle of a long input context. the performance curve is characteristically "u-shaped", where models exhibit better accuracy when the relevant information is at the beginning or end of the context, underscoring the challenges llms face in maintaining attention and relevance throughout an entire input sequence. this phenomenon, dubbed the lost-in-the-middle problem, suggests that current llms are not yet fully equipped to handle long contexts robustly and reliably, particularly in tasks that require retrieval of dispersed information @xcite @xcite . while these papers shed some light on the long-context capabilities of llms, a significant research gap remains: most existing benchmarks and evaluations have focused exclusively on monolingual english settings. this leaves a critical question unanswered: how do llms perform when the long input contexts are multilingual, particularly when the context is in a low-resource, non-latin language? as shown in figure 1 , llms usually perform poorly while handling non-english long contexts. furthermore, multilingual and cross-lingual contexts introduce additional complexities, such as varied syntax, grammar, and semantic nuances, which can significantly affect a model's retrieval and processing capabilities. in this paper, we address this gap by analyzing how llms process and retrieve information from long multilingual contexts. specifically, we introduce the multilingual needle-in-a-haystack (mlneedle) test, which extends the multilingual question-answering task to assess the llm's ability to locate and extract relevant information (the needle) from a large collection of multilingual distractor texts (the haystack). our experiments systematically vary the language and position of the needle within the haystack to evaluate the robustness of several state-of-the-art llms in handling multilingual long contexts. we make the following contributions foot_0 : our findings reveal that llm performance is highly sensitive to both the language and position of the needle in the haystack. ## multilingual needle in a haystack in this section, we introduce our multilingual needle in a haystack (mlneedle) benchmark. as our goal is to better understand how llms process multilingual input contexts, we analyze the performance of the model on a multilingual question answering task, which requires a model to find relevant information (the needle) from the input context (the haystack) to answer the given question. specifically, we conduct experiments where we systematically change (i) the position of the needle, (ii) the language of the needle, and (iii) the language of the haystack, and study the effect on performance. if llms are able to use information from long multilingual contexts, their performance should remain relatively stable regardless of changes in language or needle position. ## experimental results we run experiments with context sizes ranging from 4k to 32k tokens to compare the performance of various models on mlneedle foot_2 . ## ablation studies effect of temperature sampling. here, we investigate whether the choice of generation strategy significantly influences the model's performance. ## discussion why are language models sensitive to language of relevant information? recent findings by zhao et al. (2024) reveal that llms process multilingual content in three stages: first, they convert input into an english-centric representation, then process it during task-solving, and finally, they generate output in the original language. this conversion step, particularly for non-latin languages like hindi or arabic, can lead to significant information loss due to structural and semantic differences from english. during the task-solving phase, the model relies on this english-centric representation, but the lost or distorted information may not be fully reconstructed, resulting in inaccurate or incomplete internal representations. consequently, the model struggles more with retrieving correct information when dealing with non-latin languages, leading to a more pronounced performance drop, as seen in figure 3 . in contrast, languages like german or spanish, which share more similarities with english, tend to retain more information during the conversion process, resulting in better performance and less information loss. this is potentially why llms perform unevenly across different language families and we aim to explore this further in future work. why are language models sensitive to position of relevant information? @xcite identify an intrinsic positional attention bias in transformer-based architectures @xcite , such as llms. this bias manifests as a u-shaped attention distribution, where models disproportionately focus on tokens at the beginning and end of a sequence, over those in the middle. the authors show that this skewed attention distribution is not merely a byproduct of positional encodings but an inherent characteristic of the models' attention mechanisms @xcite . this explains the u-shaped performance curve observed in figure 4 . as shown in the figure, models consistently performed best when the needle was positioned at the start or end of the input context, with a noticeable drop in accuracy when it is placed in the middle. we aim to study this in the multilingual context in depth in future work. are language models answering from parametric knowledge? we construct our mlneedle test using the publicly available mlqa dataset @xcite . since mlqa is curated from wikipedia there is potential for leakage into llms' pretraining data @xcite @xcite , which may interfere with the llm's performance on mlneedle. however, several aspects of our experimental design and findings suggest that llms are actively processing the provided context rather than relying on memorisation . firstly, as shown in figure 4 , llms struggle with mid-context retrieval ("lost in the middle"). if llms relied purely on parametric knowledge, performance would remain stable across positions since the query and needle content stay unchanged. the baseline scores in ## conclusion in this study, we introduced the multilingual needle in a haystack (mlneedle) test, designed to systematically evaluate the long-context capabilities of multilingual llms. through a series of controlled experiments, we investigated how changes in the language and position of relevant information in a long-context affects the llms' retrieval performance. our findings revealed that llms exhibit significant sensitivity to both the language and position of the relevant information, particularly struggling when the relevant content is in non-latin languages or positioned in the middle of a long context. conversely, the models demonstrated relative robustness to variations in the language of distractor passages, suggesting that the primary challenges lie in how llms process and retrieve the needle from diverse linguistic contexts. these findings underscore the need for further research to enhance the multilingual capabilities of llms, particularly in handling scenarios where relevant information may be dispersed across different languages and positions within the input. our work represents a first step towards systematically evaluating the long-context behavior of multilingual llms.