Paper: M$^4$LE: A Multi-Ability Multi-Range Long Context Evaluation Benchmark for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
It is better to include some comparison between M4LE and other benchmarks in Table 2. It is also good to include the max. length number in it.

Author response:
**Q1: It is better to include some comparison between M4LE and other benchmarks in Table 2. It is also good to include the max. length number in it.**

We appreciate your suggestion and have accordingly included detailed statistics in the revised paper under Appendix A.1.

Reviewer comment (Evaluation):
The observations from the evaluation in the paper are weak and uninformative, except that the existing LLMs struggle to process long inputs, and ChatGPT performs the strongest among all. These can be shown from other existing benchmarks as well.

Author response:
**Q2: The evaluation in the paper are weak and uninformative, except that the existing LLMs struggle to process long inputs, and ChatGPT performs the strongest among all. These can be shown from other existing benchmarks as well.
The categorization and organization of different ranges of sentence lengths can also be done in other similar benchmarks.**

Our paper introduces the first comprehensive benchmark for long-context evaluation, systematically designed around five key abilities that encompass a broad spectrum of real-world scenarios, with contexts extending up to 32K words. Therefore, it is important to examine the performance of different LLMs on this comprehensive benchmark. Additionally, our analysis delves into performance across different languages and investigates the 'lost-in-the-middle' phenomenon, as reported by Liu et al. (2023). We also present further analysis of performance across various task types, as mentioned below.

Reviewer comment (Novelty):
The categorization and organization of different ranges of sentence lengths can also be done in other similar benchmarks.

Author response:
**Q2: The evaluation in the paper are weak and uninformative, except that the existing LLMs struggle to process long inputs, and ChatGPT performs the strongest among all. These can be shown from other existing benchmarks as well.
The categorization and organization of different ranges of sentence lengths can also be done in other similar benchmarks.**

Our paper introduces the first comprehensive benchmark for long-context evaluation, systematically designed around five key abilities that encompass a broad spectrum of real-world scenarios, with contexts extending up to 32K words. Therefore, it is important to examine the performance of different LLMs on this comprehensive benchmark. Additionally, our analysis delves into performance across different languages and investigates the 'lost-in-the-middle' phenomenon, as reported by Liu et al. (2023). We also present further analysis of performance across various task types, as mentioned below.

Reviewer comment (Evaluation):
M4LE spans different domains and tasks, while the evaluation does not consider these points, e.g. specific analysis on the different impacts of QA, translation.

Author response:
**Q3: M4LE spans different domains and tasks, while the evaluation does not consider these points**

Thank you for your valuable feedback. In response, we have added a new section, "Appendix A.6: Performance in Various Task Types", to our revised paper. We also want to highlight that our benchmark, designed to encompass a broad spectrum of real-world scenarios with long inputs, is based on five key long-context abilities. This benchmark, which considers various domains and tasks, ensures comprehensive coverage of different use cases. Therefore, our main paper focuses primarily on analyses related to these five abilities.

Reviewer comment (Experiments):
It shows that the situations from 3 types of tasks are different. There can be more sophisticated explanations and experiments to furnish further ideas.

Author response:
**Q4: it shows the situations from 3 types of tasks are different. There can be more sophisticated explanations and experiments to furnish further ideas.**


We seek clarification on whether you are referring to Figure 3, which shows the effect of placing the supporting document at different locations. While we acknowledge the subtle differences highlighted in Figure 3, our paper's primary focus is on a methodology for assessing the performance variations with context length. Therefore, we think it would be best for future work to perform more extensive experiments (More task types, more datasets per task type, different setup, etc.).  As for the performance variations in different tasks of our main experiments, we have included them in the revised paper as stated in the previous question.

Reviewer comment (Evaluation):
In Figure 2, why do most models perform worse on explicit multi-span understanding compared to both semantic multi-span and global context understanding?

Author response:
**Q5: why most models perform worse on explicit multi-span understanding compared to both semantic multi-span and global context understanding**

Thank you for raising this interesting question!  We briefly discussed this phenomenon in Section 4.3 under "Multiple-span understanding is more difficult, and semantic retrieval is even harder for competent models." While we think that more work is needed to investigate this, we hypothesize that explicit multi-span understanding requires the model to generate hexadecimal identifiers is an extremely unfamiliar task for LLMs. It is very different from most pre-training tasks, and therefore only the most capable LLMs that were fine-tuned on massively diverse tasks and aligned well with human preference can generalize and demonstrate adequate performance at this task.

Reviewer comment (Experiments):
The paper seems to have no detailed report about how to construct the benchmark. For example, which instances are considered to be combined together, or are they chosen randomly? How is the number of instances N decided? This gives the impression that the M4LE just simply combines small datasets together to build long-context datasets, where most of them have no relevance between each other. It means that M4LE is a benchmark that mainly tests the retrieval ability over long-context but considers less the comprehensive understanding over the global context.

Author response:
**Q7: The paper seems to have no detailed report about how to construct the benchmark**

Thank you for your suggestions. We have included more details in "Section 3.3 Task Construction". Specifically, we determine the number of instances by dividing the desired context length with the median length of the instances in the testing set. Then,  we sample these instances randomly from the testing set.

**Q8: I will think that the M4LE just simply combine the small datasets together to build long-context datasets, where most of them have no relevance between each other. It means that M4LE is a benchmark mainly tests the retrieval ability over long-context but less consider the comprehensive understanding over the global context.**

Thank you for your comment regarding the M4LE. I would like to take this opportunity to clarify the design of M4LE. M4LE does not just test the retrieval ability over long-context.  It contains 22 tasks that target the semantic and global understanding ability of the models.
These tasks require LLM to understand the full context thoroughly in order to retrieve relevant information to complete the task.
For tasks targeting semantic understanding, the model needs to identify a few key text spans after it comprehends the whole context to finish the task. Tasks focused on global understanding require the model to integrate and interpret information from the full context to complete the task.
Therefore, by targeting the five different abilities, M4LE is a comprehensive benchmark that evaluates a wide range of long-context understanding scenarios.

Reviewer comment (Evaluation):
In the experiments on different languages, the authors claim that Vicuna and Long-chat exhibit a more pronounced performance drop in Chinese. However, in Figure 4, the drops of these models between the two languages have no obvious difference. For example, Vicuna-13B-1.5-16k declines greatly both in English and Chinese after 4k length.

Author response:
**Q9: The authors claim that Vicuna and Long-chat exhibit a more pronounced performance drop in Chinese. However, in Figure 4, the drops of these models between these two languages have no obvious difference.**

Thank you for your observation and comment! We originally computed the slope of the best-fit lines for both languages and found that Vicuna and LongChat models indeed have steeper slope in Chinese. However, after the updates on the results as mentioned in the global response, we observed similar slopes for both languages. Therefore, we updated the paper to reflect that.

Reviewer comment (Evaluation):
There is confusion regarding M4LE seemingly being built from existing web sources, which LLMs are exposed to during training. How does it alleviate the data leakage problem, which also exists for other benchmarks?

Author response:
**Q10 : How does it alleviate the data leakage problem, which also exists for other benchmarks?**

Thank you for your question. Indeed, it is an important problem for benchmarking, as also mentioned by Reviewer "sezj".

Our methodology combines the shorter texts which actually present an instance that is sufficiently distinct to the original ones. For instance, in the WikiText-103 (NLI) task, we observed that even though Wikipedia articles are commonly included in language model training datasets, the performance of most models is poor even when the input contexts are limited to 1k tokens. This suggests that the benchmark effectively challenges the models beyond their training conditions.

Besides, changing the question type and format, for example by asking the models to return the hexadecimal identifier of all the articles that belong to a certain class presents a different challenge for different LLMs, as evidenced by the poor performance of most models in MNDS News (Explicit Multiple) and THUCNews (Explicit Multiple).

We hope these revisions and clarifications address your concerns adequately. Thank you for your insightful feedback, which has significantly contributed to the improvement of our paper.

Reviewer comment (Experiments):
Additional statistical information of the proposed dataset could be provided, such as the distribution of sentence lengths and the proportion of samples in different languages.

Author response:
We have now included detailed statistics in the revised paper under Appendix A.1. We believe these additional details will provide a clearer understanding of the dataset's composition and diversity.

Reviewer comment (Experiments):
Do the authors have a data cleaning process?

Author response:
Regarding our data cleaning process, we adhered to the preprocessing process of the original datasets. Additionally, we exclude instances that were either excessively long or short for our benchmark. The precise methodology and criteria used for this filtering process have been outlined in detail in Appendix A.1 of our original submission.

Reviewer comment (Experiments):
How is the data cleaning process done?

Author response:
Regarding our data cleaning process, we adhered to the preprocessing process of the original datasets. Additionally, we exclude instances that were either excessively long or short for our benchmark. The precise methodology and criteria used for this filtering process have been outlined in detail in Appendix A.1 of our original submission.

Reviewer comment (Experiments):
My primary concern pertains to the strategy of constructing long sequences by simply converting short sequences into longer ones. It is clear that the model's performance deteriorates significantly as the length of the context increases, which raises doubts about the effectiveness and appropriateness of such data formation. The random selection of original short sequences appears to negatively impact the performance. Further investigation is needed to determine how to control the variance of data distributions/domains within a long sequence of text and assess the validity and reasonability of this data formation approach.

Author response:
You raised a critical point regarding the construction of long sequences from shorter ones, and the potential impact of this methodology on model performance. We agree that the performance degradation observed in models as context length increases highlights the challenges LLMs face in processing long sequences. However, we argue that this actually supports that our approach is effective rather than a flaw in our data formation approach.

Our method of extending short sequences into longer contexts is designed to rigorously test the five long-context understanding abilities of LLMs without building new datasets from scratch. The performance deterioration observed in some models, especially at longer context lengths, is indicative of their current limitations in handling extended sequences. This is an important aspect of our benchmark, as it successfully distinguishes between models based on their ability to process long contexts. For instance, as illustrated in Figure 2, more competent models like GPT-3.5-Turbo-16K and ChatGLM2-6B-32K exhibit a relatively smaller drop in performance, demonstrating their stronger long-context capabilities.

Additionally, we acknowledge the concern about the random selection of original short sequences. We want to clarify that our approach also aims to mitigate potential pre-training biases inherent in LLMs. Therefore, the difficulty models face with instances formed with randomly selected sequences underscores the effectiveness of our approach in providing a more unbiased assessment of their long-context understanding abilities.

Reviewer comment (Experiments):
Additionally, I kindly request a performance comparison with recent state-of-the-art models, such as Llama2-4k.

Author response:
Regarding your request for a performance comparison with the latest state-of-the-art models, such as Llama2-4k, we appreciate the importance of including such comparisons in our study. In our initial submission, we evaluated the Llama2-4k models, specifically the 7B and 13B variants, alongside their chat versions.

For Llama2-70B models, we are currently conducting experiments with these models. However, due to the extensive computational resources required for these experiments, we anticipate some delay in obtaining and analyzing the results. We are committed to including this data in our study as soon as it is available and will provide updates accordingly.

Reviewer comment (Experiments):
The construction of the dataset is detached from realistic scenarios. This is because the dataset is mainly composed of short texts synthesized into longer documents, which is not typically how users interact with long-context language models. They are unlikely to use such synthesized long documents as inputs. Moreover, this dataset is evidently more beneficial to models trained on these synthesized short texts, therefore introducing a potential bias.

Author response:
We acknowledge your concern regarding the realistic applicability of our dataset.
It is important to note that M4LE includes 10 tasks that directly target global context understanding abilities pertinent to real-world scenarios, such as summarization, translation, question answering, and classification tasks. These tasks represent a significant portion of our benchmark and are designed to reflect common use cases of LLMs in practical applications.

Furthermore, while some tasks in M4LE may seem detached from typical user interactions with LLMs, they are intentionally designed to evaluate LLM across five key long-context understanding abilities as mentioned in the paper. These five abilities encompass a broad spectrum of long-sequence scenarios and challenges that a language model may face in real-world applications. Therefore, by constructing a diverse benchmark that evaluates these five abilities, we can obtain a measure that closely approximates the performance of models in realistic, long-context scenarios.

Reviewer comment (Evaluation):
Unfair metrics may be a potential reason for the questionable experimental results. This paper mainly uses n-gram matching metrics like ROUGE and F-1, which may not correlate with human evaluation results.

Author response:
Regarding the evaluation metrics, we use accuracy as the metric for QA, consistent with previous work (Liu et al., 2023). For classification tasks, we use F1 scores, which do not rely on n-gram matching. N-gram metrics are applied in only 15 of the 36 tasks, primarily in summarization and translation. We acknowledge the limitations of these metrics but have focused on performance trends over context lengths to mitigate potential biases. Besides, these N-gram metrics are still commonly reported in much recent literature for summarization and translation tasks.

Reviewer comment (Experiments):
Position bias in the dataset construction may also be a potential reason for the questionable experimental results. Most answers are at the beginning of the document, and additional input is noise.

Author response:
To address potential position biases, we have carefully constructed instances with relevant supporting paragraphs uniformly distributed within the input context, as outlined in Section 3.1 of our paper.

Reviewer comment (Experiments):
The current long context models indeed cannot handle many input tokens, but how is the Llama2-4k? This remains unanswered.

Author response:
Finally, regarding Llama2-4k, we used dynamic NTK scaling to extend the effective window size of LLama2 models, which we found to be an effective approach without further fine-tuning as mentioned in the paper.