Paper: L-Eval: Instituting Standardized Evaluation for Long Context Language Models (ICLR-2024)

Reviewer comment (Experiments):
Uses Claude-100k as the data filter. This detects unanswerable questions, but careful human review would be better, rather than steering scores towards closed LLMs.

Author response:
Thank you for your valuable advice. We use Claude-100k to detect unanswerable questions because it has the longest context window and is powerful in our preliminary experiments. We plan to re-detect the unanswerable questions using another long context model, gpt3.5-turbo-16k, and review the questions to further ensure the quality.

Reviewer comment (Novelty):
Length Instruction Enhanced (LIE) is arguably not a new or significant contribution, so the authors should be careful not to frame it as such. The analysis of it is what is interesting.

Author response:
We concur that the enhancement of length instruction is not a significant contribution of this work. Rather, our primary objective is to draw researchers' attention to the length bias in evaluation metrics. The evaluation metrics for LCLMs should favor accurate content over precise length. We will further clarify this point in the introduction section of our paper.

Reviewer comment (Experiments):
Did you test performance of models with the question but without the context? I.e. you make the argument that the questions are designed not to be easy just from using parametric knowledge – can we validate this?

Author response:
Thank you for a very interesting question. We test 3 models on the 4 new datasets annotated by us: SFiction (factuality+loyalty) and Coursera (accuracy), CodeU (accuracy), and LongFQA (F-1). Results are as follows:

|                   | SFiction | SFiction(no ctx) | Coursera | Coursera(no ctx) |
|-------------------|---------:|-----------------:|---------:|-----------------:|
| GPT3.5-turbo-16k  |    64.84 |            50.78 |    63.51 |            44.47 |
| Llama2-13b-4k     |    54.68 |            42.18 |    35.75 |            16.27 |
| Vicuna1.5-13b-16k |    61.71 |            46.09 |    40.69 |            30.75 |

|                   | CodeU | CodeU(no ctx) | LongFQA | LongFQA(no ctx) |
|-------------------|------:|--------------:|--------:|---------------:|
| GPT3.5-turbo-16k  | 12.22 |           0.00|    45.36|           34.89|
| Llama2-13b-4k     |  1.11 |           0.00|    38.07|           20.42 |
| Vicuna1.5-13b-16k |  0.00 |           0.00|    45.57|           18.90 |

We will include more models in our camera-ready version.

Reviewer comment (Experiments):
Confused about the 96 subset. Are there many evaluation questions in your benchmark ignored / not used?

Author response:
We selected the 96-question subset, which is similar in scale to the popular Vicuna 80-question test set.  We split the subset mainly due to the high cost of GPT-4 evaluation. We have also included the results of SPACE, QMSum, NQ, and NarrativeQA with GPT-4 evaluation in Table 2, but testing the 4 datasets has already exceeded a cost of **$100**. Therefore, we only reported n-gram metrics results in Tables 6, 7, and 8 for the remaining open-ended tasks. Researchers can conduct experiments on some sub-tasks from L-Eval or directly on the 96-question subset based on their settings.

Reviewer comment (Novelty):
The findings of the paper are rather intuitive and not enough insightful. For example, the authors mention in the penultimate paragraph of the Introduction, "There is still a significant gap between open-source LCLMs and commercial models, for both closed-ended tasks (Table 3) and open-ended tasks evaluated by LLMs and humans (Table 4, 5). However, this gap is not accurately reflected by n-gram metrics."

Author response:
Although the serious issues with n-gram metrics have been revealed by previous research, the fact remains that almost all current long-context models still use ROUGE and F-1 as evaluation metrics for open-ended tasks. Many LCLMs claim to outperform commercial models on various tasks, but our metric experiments suggest this could be due to biased metrics. It's still challenging for open-source LCLMs to achieve a high win-rate compared to GPT3.5-16k on L-Eval.

*Other interesting findings*
* Reference-based metrics are very sensitive to length.  By simply informing the expected length, we improve Claude by **50** F-1 score on NQ. 
* We observe the performance disparity between LCLMs on open-ended and closed-ended tasks
* The performance differences on retrieval tasks and reasoning tasks when increasing the RoPE base frequency. 

These findings are different from previous (or concurrent) works.

Reviewer comment (Novelty):
Could the authors provide further insights into the shortcomings of current open-sourced long-context models, and how the dataset provided in this paper could facilitate enhancements in these problems?

Author response:
*Shortcomings*:

According to our experiments with L-Eval, the most significant issue with open-source LCLMs is that as the number of input tokens increases, the performance on open-ended tasks decreases. However, this issue is not easily noticeable from n-gram metrics where the ROUGE scores are still competitive. Our human eval shows one of the reasons is the weakened instruction-following ability. 

*How L-Eval contributes to the research of LCLMs*:

1. 	Our work can help researchers re-think previous methods.  It shows that scaled position embedding and further training enhance the model's usage of more input tokens (or information)  for closed-ended tasks. However, open-source LCLMs struggle with open-ended tasks, highlighting a need for better instruction understanding in tasks with varied question styles.

2. 	We are calling for the use of more fair evaluation metrics instead of solely relying on metrics like ROUGE, which rewards models simply for generating an output similar in length to the standard answer.

3. 	L-Eval integrates more closed-ended tasks, devoid of evaluation metric biases.  Notably, open-source LCLMs still significantly lag behind commercial models, and the low testing cost of closed-ended tasks can provide researchers with rapid and accurate feedback.

Reviewer comment (Writing):
The paper has multiple typos and grammar errors. E.g.  
- “We inject come new”  
- “NTK-ware”  
- “directly exposure to LCLMs”  
I suggest the authors run the text by simple automatic grammar correction since I’m pretty sure it would catch most of these.

Author response:
Thank you so much for pointing out the grammar errors. We have polished this paper and used some automatic grammar correction tools based on your suggestions.

Reviewer comment (Novelty):
Abstract: “GPT-4 and Claude can largely preserve the reasoning ability in an extended context” - this statement is misleading, there is a significant body of work showing how the reasoning abilities of these models is limited particularly for long contexts / multi-hop reasoning tasks.

Author response:
> **W1**: “GPT-4 and Claude can largely preserve the reasoning ability…” is misleading.

This sentence is mainly based on the comparison between GPT-4/Claude and open-source models. We will add “compared with open-source models” at the end of this sentence.

Reviewer comment (Novelty):
There is no clear highlight of differences between the currently proposed benchmark and existing long sequence benchmarks in the literature (which the authors describe in Section 2.2). Why is L-eval superior to existing benchmarks is not clarified in this paper.

Author response:
>**Q4**: How is L-Eval different from existing long sequence benchmarks?

To the best of our knowledge, the most related evaluation suites for testing long context language models include ZeroScrolls (Findings of EMNLP 2023)[1] and Longbench (Arxiv)[2]. In accordance with the ICLR 2024 review policy, these can be classified as concurrent work.

*Differences*:
* In contrast to previous work, all samples in L-Eval are manually selected. Furthermore, we filter out both unanswerable and mistaken questions to ensure high-quality data.
* We refrain from relying on n-gram metrics for open-ended tasks, which leads to different conclusions.
*  We annotate a larger number of closed-ended tasks, yielding results that are less likely to be influenced by metric biases.
More details can be found in Section 2.2.

Reviewer comment (Novelty):
The paper should motivate why researchers would use this benchmark instead of other existing long-context benchmarks, especially given its potential to impact the research community.

Author response:
> **W4**: The blunt recommendation of LLM-based metrics
1. *LLM judges have been widely used in modern open-ended benchmarks*

Firstly, almost all modern open-ended benchmarks like AlpacaEval[3], MT-bench[4] Vicuna Bench[5] use LLM judges, discarding ROUGE and F-1. However, long context LLM still insists on using ROUGE or F-1. Compared with ROUGE and F-1, we highly recommend using LLM judges instead.  In long-context settings, we cannot provide the entire input to the judge; this forms a key difference. Consequently, we have designed a new judge prompt within this setting to better adapt to long-context situations.

2. *Our experiments have confirmed that the LLM judges can achieve better consistency with human evaluation.*

We agree that automatic metrics for open-ended tasks including LLM judges have many drawbacks but we strongly disagree with the assertion that the recommendation of LLM-based judges is blunt. We have experimentally confirmed that LLM judges outperform n-gram metrics in the long-context setting by ranking generation results from seven models.  We adopt the widely used Kendall-Tau correlation coefficient [6][7] to measure the correlation between these automatic metrics and human assessments.  Specifically, we rank these model outputs (7 models x 85 questions) using various metrics (e.g., F1, ROUGE, human score, and GPT4) and compute the Kendall-Tau correlation coefficient with human judgment. Results indicate that the Kendall-Tau correlation coefficient of ROUGE is only **0.52**, while the GPT-4 judge achieves a significantly higher score of **0.95**, outperforming other metrics. Please refer to Appendix A2 for the details of the human evaluation setup.

Reviewer comment (Evaluation):
It would be good to include length statistics for both inputs and references across the tasks in the benchmark, and maybe correlate this with the difficulty of solving a particular task. The paper only limits to briefly mentioning “The length of reference in L-Eval also varies significantly across tasks.”

Author response:
>**W3**: Include length statistics for both inputs and references across the tasks.

The length statistics for input can be found in Table 1. We list the average reference length here:

| Dataset       | Avg reference len |
| ------------- | ------------------------ |
| NQ            | 4                        |
| narrativeQA   | 9                        |
| Qasper        | 12                       |
| multidoc2dial | 21                       |
| QMSum         | 60                       |
| LongFQA       | 61                       |
| SPACE         | 97                       |
| SummScreen    | 98                       |
| bigpatent     | 124                      |
| GovReport     | 261                      |
| Openreview    | 273                      |
| multi-news    | 274                      |
We will add this information in Table 1.

Reviewer comment (Evaluation):
The authors conduct evaluation of reference-based metrics on their proposed benchmark, and conclude that n-gram based metrics are inferior to LLM-based metrics for long-text generation. However, the main takeaway from the evaluation analysis is that the authors recommend the use of LLM-based metrics without discussing problems associated with them.

Author response:
> **W4**: The blunt recommendation of LLM-based metrics
1. *LLM judges have been widely used in modern open-ended benchmarks*

Firstly, almost all modern open-ended benchmarks like AlpacaEval[3], MT-bench[4] Vicuna Bench[5] use LLM judges, discarding ROUGE and F-1. However, long context LLM still insists on using ROUGE or F-1. Compared with ROUGE and F-1, we highly recommend using LLM judges instead.  In long-context settings, we cannot provide the entire input to the judge; this forms a key difference. Consequently, we have designed a new judge prompt within this setting to better adapt to long-context situations.

2. *Our experiments have confirmed that the LLM judges can achieve better consistency with human evaluation.*

We agree that automatic metrics for open-ended tasks including LLM judges have many drawbacks but we strongly disagree with the assertion that the recommendation of LLM-based judges is blunt. We have experimentally confirmed that LLM judges outperform n-gram metrics in the long-context setting by ranking generation results from seven models.  We adopt the widely used Kendall-Tau correlation coefficient [6][7] to measure the correlation between these automatic metrics and human assessments.  Specifically, we rank these model outputs (7 models x 85 questions) using various metrics (e.g., F1, ROUGE, human score, and GPT4) and compute the Kendall-Tau correlation coefficient with human judgment. Results indicate that the Kendall-Tau correlation coefficient of ROUGE is only **0.52**, while the GPT-4 judge achieves a significantly higher score of **0.95**, outperforming other metrics. Please refer to Appendix A2 for the details of the human evaluation setup.

Reviewer comment (Evaluation):
The recommendation of LLM-based metrics is a strong limitation of this paper because there is no thorough discussion of their biases and failure cases.

Author response:
> **W5**: Potential biases and failure rate in LLM judges

Thank you for your valuable advice. We have conducted a study on the biases and failure rate of LLM judges and will include these results in our camera-ready version. Results from the 85-question subset are shown in the following table. In this table, 'fail-rate' represents the cases where judgments from human judges and LLM judges are contradictory. 'b-first' represents the percentage of cases when a judge favors the first answer, and 'b-details' is the percentage of cases when a judge favors the more detailed answer.
 |               | b-first | b-details | fail-rate | Error-samples |
| ------------- | ------- | --------- | --------- | ------------- |
| GPT-3.5 judge | 56%     | 68%       | 49/154    | 16            |
| GPT-4 judge   | 31%     | 37%       | 22/170    | 0             |

Reviewer comment (Reproducibility):
The paper does not present details of how the manual annotation was conducted for human evaluation, which could impact the results reported and the conclusions of the paper.

Author response:
> **W6**: human evaluation is not flawless and the details of human evaluation.

The human annotation process can be found in Appendix A2. All three annotators are PhD students. Annotating the 85 questions from 7 models (7x85 outputs) took approximately 4 days. Although we agree human evaluation is also biased, it is still generally believed to be more accurate than current automatic metrics.