ML issard: Multilingual Long and Simple Sequential Reasoning Benchmarks (2024)

Abstract: Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce MLissard, a multilingual benchmark designed to evaluate models’ abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity. Our evaluation of open-source and proprietary models show a consistent decline in performance across all models and languages as the complexity of the sequence increases. Surprisingly, the use of in-context examples in languages other than English helps increase extrapolation performance significantly.

## introduction figure 1 : performance of gpt-4 on the mlissard benchmark. see the efficacy of language models, particularly in reasoning tasks, is significantly impacted by longer text lengths than those seen in training @xcite @xcite . this phenomenon, referred to as "length generalization" or "length extrapolation" in the literature @xcite , is also common in models based on the transformer architecture @xcite @xcite . notably, even large language models (llms), known for their strong performance in a wide range of tasks and domains, are not immune to this problem @xcite . recent research tried to address this challenge by modifications to the positional embeddings @xcite @xcite @xcite or by using prompting strategies such as scratchpad @xcite and chainof-thought reasoning @xcite . nevertheless, there remains a lack of datasets specifically designed for the systematic evaluation of the problem. while benchmarks such as zeroscrolls @xcite and infinitebench @xcite were designed to evaluate models in natural language tasks that involve long sequences, its effectiveness in monitoring model performance degradation within the context of length generalization may be limited by lack of explicit control of task complexity with respect to sequence length. for example, when using natural language texts there is no guarantee that answering a question about a longer text is harder than responding to one about a shorter text. this limitation highlights the need for benchmarks that can explicitly manipulate and test the impact of sequence length on model performance. in benchmarks pertaining to dialogues @xcite and multi-document question answering @xcite , techniques like retrieval-augmented generation (rag) are preva-lent, and therefore explicitly isolating the length extrapolation issue poses a challenge. to address these aforementioned problems, we present mlissard, a multilingual benchmark that offer support for 6 languages (english, german, portuguese, russian, spanish and ukrainian) designed to evaluate the ability of models on tasks that require the use of repetitive simple rules, whose difficulty increases with respect to the sequence length. by incorporating varying degrees of difficulty within the same tasks, mlissard facilitates the identification of a models' breaking points. given the syntactic nature of the datasets, researchers have the capability to generate new examples and increase the task difficulty, thus making it more challenging for newer and more capable models to be evaluated effectively. this flexibility also mitigates the contamination problem -where models may inadvertently be exposed to test datasets during their training @xcite ) -since synthetic datasets can be generated as needed, a advantage over traditional, manually curated datasets. at the time of this research, this is the first multilingual dataset designed to evaluate the quality of models in extrapolation via length. our analysis, which includes evaluations on proprietary models such as @xcite , as well as open-source ones like llama-3 @xcite , reveals a common trend among them. as illustrated in figure 1 , our findings underscore that irrespective of their architectures and parameter counts, all examined models demonstrate a performance degradation with increasing length, controlled by the number of key entities (see their definition in our findings further demonstrated that the effect of extrapolation is not isolated; variables such as language and model size significantly influence the outcomes. for instance, despite english being a high-resource language, its performance was only average and was surpassed by other languages such as german. moreover, ablation tests revealed improvements in extrapolation performance when in-context examples comprised a mixture of languages. this underscores the influence of language selection on the extrapolation capabilities of lan-guage models. ## related work the challenge of length extrapolation in the domain of natural language processing has been a persistent and long-standing issue. an array of studies has demonstrated that neural architectures encounter difficulties when confronted with sequences of longer than those they encountered during their training @xcite @xcite @xcite @xcite @xcite . despite efforts to expand the context window in llms, this issue persists, particularly when tackling tasks involving complex reasoning @xcite . recent endeavors have been undertaken to enhance the general performance of llms by employing prompt engineering techniques and by developing novel decoding methods aimed at expanding their capacity to extrapolate effectively over lengthy sequences of tokens. for instance, nye et al. introduced the concept of a "scratchpad" that enables the model to generate draft responses in natural language before producing the final output. to assess the performance of this method, a range of tasks were employed, including math and coding tasks. moreover, studies by @xcite improvements by configuring the model to generate explanations for problemsolving and breaking down tasks into multiple interactive steps. these enhancements were particularly noticeable in tasks requiring the ability to extrapolate, such as scan @xcite (compositional generalization), and mathematical reasoning. additionally, bueno et al. showed that utilizing markups tokens as position representations help the model to generalize to longer sequences in tasks related to mathematical addition and compositional generalization. han et al. devised a decoding method to improve generalization over extended sequences. in addition to techniques for customizing prompts, recent research has explored modifying the position encoding function of the original transformer architecture to enhance its extrapolation capabilities @xcite @xcite @xcite . for instance, @xcite conducted an evaluation of commonly used positional encoding methods, finding that omitting positional encoding altogether yielded superior results in downstream tasks. the studies cited above illustrate multiple methods designed to address the challenge of extrapolation. nevertheless, there is a notable gap in research concerning the development of diverse and standardized datasets specifically for assessing the generation and synthesis of extended text sequences by neural models. this gap is particularly notable given that many of the traditional datasets may already have been employed in the training of large language models. ## datasets description our benchmark incorporates a combination of existing tasks, such as those from big-bench (bench authors, 2023), as well as newly developed ones. the criteria for selecting tasks were based on their ease of solution, the ability to expand new examples of varying lengths via scripting, and their effectiveness in exercising reasoning and memorization. we intentionally excluded classical datasets (e.g., scan) from the analysis since their test sets are publicly available and many solutions have been extensively detailed in scientific literature, potentially making them familiar to large language models (llms). in addition to english (en), the language set includes german (de), spanish (es), portuguese (pt), russian (ru), and ukrainian (ua). we achieved this expansion by integrating automatic translation systems and using python scripts to generate synthetic data. the following sections describe the idea of key entities, tasks, and how evaluation was performed. ## baseline methods the evaluation of each task involved analyzing responses from gpt-4 (gpt4-0613) and llama-3 (llama-3.1-405b-instruct and llama-3-instruction-70b) using greedy decoding. we observed no repetition issues. each task was preceded by a predefined instruction (description of the task) with in-context examples: four for "object counting," "find intersection," and "last letter concat," and one for "repeat copy logic" because inputs already provided sufficient information to perform the task. both the instructions and examples were in the target language of the evaluation. for instance, english tasks used english instructions and examples (see figure 2 (a) ). for the in-context ex-amples used during model evaluation, we selected samples contained in the first bin, as these contain the smallest lengths. we utilized the exact match as the primary metric. this methodology is further modified in section 5.2, where we discuss the impact of crosslanguage inputs on model performance. ## results figure 3 presents the results obtained via gpt-4 in the target tasks and languages. overall, there is a gradual decline in the performance of language models across tasks as complexity increases, as measured by the number of key entities in the input sequence. for instance, in the "object counting" task, when presented with inputs containing 1 to 7 objects, the model achieve approximately 100% accuracy. however, their accuracy drops below 50% when confronted with sequences with 12 to 17 objects. this behavior is reflected in the target languages as well, all of which present a loss of more than 50% when dealing with more complex input sequences. we also observed considerable variability in performance between languages depending on the specific task. for instance, differences ranging from 2.4 to 42 points are observed in the intermediate bins for tasks such as "last letter concatenation" and "repeat copy logic". these variations are intriguing as there doesn't appear to be a general language preference. for example, in the "last letter concatenation" task, german, portuguese, and spanish outperform russian by a margin of 42.6 points in the 15-22 bin. conversely, in the "repeat copy logic" task, russian outperforms portuguese by 42.5 points. contrary to the general trend observed in studies of multilingual models, english did not exhibit exceptional performance when compared to other languages. except for the "list intersection" task, english consistently remained at an average or lower accuracy level across bins. generalization performance also varies between tasks; as demonstrated in regarding the performance of open-source models in the mlissard benchmark, figure 4 illustrates that both models performed similarly in bin 1, with accuracy points ranging between 70 and 100. however, as task complexity increased from bin 2 onwards, differences in performance stood out. except for the "repeat copy logic" task, gpt-4 outperformed llama-3.1-405b by 5 to 60 accuracy points (see on the other hand, in the "repeat copy logic" task, there is a reverse comparison, where llama-3.1-405b outperforms gpt-4 in all bins, with the difference ranging from 9 points to 16 points of accuracy. in relation to language preference behavior, both the llama-3.1-405b and gpt-4 models exhibit similar task-dependent variations. llama-3.1-405b demonstrates more consistent performance across portuguese, german, and english. ## conclusion we presented a multilingual benchmark to evaluate the ability of language models to deal with long texts across languages. our approach distinguishes itself from existing benchmarks through the introduction of a control mechanism, which we refer to as "key entities." this mechanism enables us to systematically increase task complexity in tandem with sequence length. furthermore, the ability to solve these tasks is predicated on the repeated application of simple rules, providing more control and enabling a detailed analysis of model performance in relation to the frequency of rule application. this contrasts with benchmarks that rely on lengthy natural language texts, where the relationship between text length and task difficulty may become obscured. despite the apparent simplicity of these tasks, they reveal significant limitations in state-of-the-art llms concerning the processing and generation of text as lengths increase. our findings indicate that language and model size significantly affect extrapolation results. moreover, including in-context examples in multiple languages improves mlissard's generalization performance.