LC -Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding (2025)

Abstract: Recent advancements in Large Language Models (LLMs) have demonstrated sophisticated capabilities, including the ability to process and comprehend extended contexts. These emergent capabilities necessitate rigorous evaluation methods to effectively assess their performance in long-context understanding. In this paper, we present LC-Eval, a bilingual, multi-task evaluation benchmark designed to evaluate long-context understanding in English and Arabic, targeting context lengths ranging from 4k to over 128k tokens. LC-Eval introduces four novel and challenging tasks: multi-document question answering, bilingual question answering, claim verification within a paragraph, and multiple-choice questions based on long contexts. These tasks are designed to assess LLMs’ abilities in deep reasoning, document comprehension, information tracing, and bilingual information extraction and understanding. The benchmark includes datasets in both Arabic and English for each task, allowing for a comparative analysis of their performance across different text genres. Evaluations were conducted on both open-weight and closed LLMs, with results indicating that LC-Eval presents significant challenges. Even high-performing models, such as GPT-4o, struggled with certain tasks, highlighting the complexity and rigor of the benchmark.

## introduction context length of large language models (llms) typically indicates how many tokens a language model can process as an input. although early models can process up to 4k tokens, more recent models have the context length varying from 8k to 128k even to 1m @xcite @xcite @xcite @xcite . these long context language models (lclms) are extremely helpful for understanding long documents, minimizing hallucinations and retrieval augmented generation @xcite . since arabic is one of the major languages spoken by more than 400 million people as their mother tongue (worlddata.info, 2024), a number of arabic large language models that understand both arabic and english have been released @xcite @xcite , . however, these models are often evaluated using english benchmarks or proprietary datasets, making it challenging to publicly benchmark their performance in arabic or to assess their capabilities across various tasks. additionally, although benchmark datasets in english cover aspects such as reasoning, document summarization, and document understanding, most of these benchmarks lack a focus on deep reasoning. moreover, most of the existing datasets evaluate lclms on multiple-choice questions (mcqs) @xcite or very short generation of text @xcite . furthermore, many task-specific datasets fail to fully evaluate lclms across their entire context length, leading to the need for new benchmark datasets both for arabic and english @xcite . another significant challenge is evaluating the responses of lclms to open-ended questions. most existing evaluation methods, such as bleu @xcite and rouge @xcite , rely on exact word matching. since a differently formed sentence with different words can carry the same meaning of the compared sentence and even when queried with the same question twice, the same lclm can generate responses that are semantically equivalent but phrased differently, this makes an exact word matching an unreliable evaluation criterion. another alternative approach is to use similarity-based method which addresses some issues of word matching. however, it can also lead to incorrect evaluations because similarity and semantic meaning are different from each other. for example, the sentences "the capital of france is paris" and "the capital of france is rome" could have a high similarity score, yet their semantic meanings are entirely different. to address these issues, we introduce lc-eval, a bilingual multi-task evaluation benchmark for english and arabic long-context understanding, covering context lengths from 4k to more than 128k. lc-eval comprises four challenging tasks: our dataset was initially generated using gpt-4 (with task-specific prompts), followed by multistage refinements to increase complexity. to ensure accuracy, three human annotators validated all data, with majority agreement determining the final verdict. the validation criteria were task-specific, and human validators received specialized training on the respective tasks before beginning the validation process. evaluation results show that lc-eval poses significant challenges for lclms. our key contributions are: the rest of the paper is organized as follows: related work is presented in section a.1. in section 2, we present an overview of all the tasks that are being evaluated, while in section 3 we give the details of how the datasets were curated. section 4 highlights the data validation process. in section 5, the experiment setup is presented and the actual evaluation and results are presented in section 6. then we conclude in section 7. ## data curation our data collection process drew from both arabic and english corpora, leveraging multiple publicly available datasets to ensure broad coverage. we utilized the 2024 wikipedia dumps, wikinews, wikihow, and wikibooks for both languages, providing a rich mix of encyclopedic, instructional, and news content. additionally, we incorporated english books from project gutenberg @xcite and arabic books from the hindawi organization @xcite to ensure a well-balanced representation of formal and literary language. for timely and relevant news content in both languages, we included articles from the saudi press agency 1 . while these datasets are non-parallel, they provide valuable coverage across diverse domains such as economy, biology, and many more. we discuss the license of the data in appendix a.2. since these datasets may contain harmful content, such as hate speech, we employed a custom word-based dictionary filtering method to remove potentially harmful content. for dynamic tasks (e.g., multi-document qa), we sampled according to domain, word count, and source, distributing samples uniformly across sources (with complete inclusion of sources below 100 samples). for fixed tasks, the sampling aimed for a similar distribution across varying context lengths. in total we obtained 7,903 evaluation samples, ## data validation to ensure high-quality data, we engaged human annotators to review and validate the datasets. in all tasks, each data sample was reviewed by three annotators. we only accept samples that were accepted by at least two annotators. the evaluation process and criteria vary depending on the nature of each dataset. in the following, we outline the process in detail. more details about the annotation process and guidelines are presented in appendix a.9. ## evaluation 6.1 multi-document question answering the low average of entity relationship recall and recall can be attributed to the significant performance decline observed in most models as the number of words increases (see appendix a.5), highlighting the limited capability of lclms when handling increased context lengths or larger numbers of documents. for arabic, the standard deviation of entity relationship recall across word counts ranges from 2.29% to 11.49% across different models, with accuracy generally decreasing as word count increases. a similar trend is observed in english, where standard deviations for entity relationship recall across word counts range from 4.72% to 19.36%. notably, the standard deviations for recall across word count bins are typically much higher than those for entity relationship recall, further emphasizing the overall limitations of lclms in document tracing, particularly as context length increases. compared to rouge-l, entity relationship recall and f1-score are higher. this is because entity relationship scores are based on semantic meaning, which provides a more relaxed evaluation criterion than bleu and rouge. nevertheless, we observe a strong correlation between entity relationship recall and rouge-l, with a pearson correlation coefficient of 0.77 for arabic and 0.94 for english. ## conclusion our work introduces a new benchmark dataset for long-context english and arabic, designed to evaluate lclms' capabilities in deep reasoning, information extraction, and tracing. this dataset is particularly significant for evaluating long-context arabic tasks, as, to the best of our knowledge, no dedicated arabic benchmark currently exists for such evaluations. since the dataset is humanvalidated, it ensures high quality and serves as a valuable resource for advancing progress toward artificial general intelligence (agi) in both arabic and english. the evaluation results across four distinct tasks demonstrate that, although the initial data was generated using gpt-4o, our data creation methodology introduces sufficient complexity to challenge even gpt-4o, preventing it from achieving exceptionally high scores. notably, in certain tasks in the benchmark, other models outperformed gpt-4o. moreover, although we evaluated most of the baseline models up to 64k words for arabic (approximately 128k tokens), lc-eval is capable of evaluating context lengths of up to 256k tokens. this is because it includes data points more than 128k words, and the token fertility rate of arabic @xmath0. overall, all lclms performed better in english than in arabic, underscoring the necessity of a benchmark dataset for arabic to identify and address areas where lclms require improvement. lc-eval also uncovers multi-document reasoning flaws: models can generate correct-seeming answers yet fail to cite correct sources. bi-lingual qa shows further challenges beyond translation, with performance varying by model and language pair and declining at longer contexts. finally, our entity relationship recall method for open-ended questions considers semantic meaning, offering a more robust evaluation than existing methods.