Paper: Style Over Substance: Evaluation Biases for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The study is small scale: it only analyzes forty questions, which makes it hard to make systematic conclusions.

Author response:
The reviewer highlights concerns regarding the small scale of our study, which involved only 40 questions. However, it is important to note that our analysis extended far beyond the raw number of questions. The evaluation encompassed these questions across eight different models, resulting in **5280 individual comparisons**. The total resource and financial expenditure for this scale of analysis amounted to **thousands of USD.**

**Moreover, our analysis demonstrates a consistent trend in the Elo rankings when sampling from these 40 questions.** This consistency indicates that, despite the seemingly small sample size, the dataset is robust enough to yield reliable insights and trends. Therefore, we believe that the chosen sample size and methodology strike a delicate yet effective balance between comprehensive analysis and practical constraints such as cost and resource allocation.

Reviewer comment (Evaluation):
The study only considers three different attributes to evaluate, and only a few types of generated answers. It’s possible these three attributes are useful for identifying problems with these types of answers, but not others.

Author response:
Our study focused on three attributes – accuracy, helpfulness, and response length – based on their relevance and importance in the context of LLM evaluation. We agree that exploring additional attributes could provide a more comprehensive understanding of LLM capabilities. **This choice was a strategic decision to delve deeper into a few significant aspects rather than superficially covering a broader range.** Future research could incorporate more diverse attributes, such as creativity, engagement, or cultural sensitivity, to enrich the evaluation framework.

Reviewer comment (Experiments):
The characteristics rely on GPT-4 faithfully following the prompt, which doesn’t work, so the authors manually edit the text. This adds a human bias to the study, which could impact the results.

Author response:
**This step was necessary to ensure that the responses adhered to the study's requirements**, as GPT-4 did not always generate outputs that aligned with the intended error types or language proficiency levels. We would like to highlight that only a few examples required post-editing.

Reviewer comment (Evaluation):
Do you expect the results to change with a larger study, or with different attributes (e.g., beyond "helpfulness") that you measure?

Author response:
Regarding the question about the potential impact of a larger study or different attributes, we believe that expanding the study scale and incorporating varied attributes would likely yield more nuanced insights and possibly reveal additional biases or evaluation trends. However, **the core findings regarding the preference for longer, factually incorrect responses over shorter, accurate ones still hold,** as these reflect deeper biases in human and LLM evaluation processes. A larger-scale study would help in validating these findings and understanding their generalizability.

Reviewer comment (Writing):
Several important parts are not clearly presented or defined.

Author response:
You pointed out that several parts are not clearly presented or defined. We acknowledge this concern and will revise the manuscript to enhance clarity including the placement of the related work, the human annotation guidelines, and so on. We will also clearly define the terms, like “game” and “hesitancy”, in the future revisions of our manuscript

Reviewer comment (Presentation):
Putting related work at the end makes the reading harder — for example, it was not clear until the end if Elo rating is a widely used approach.

Author response:
Elo ratings are a method for calculating the relative skill levels of players in two-player games such as chess. In each game, the winner gains points from the loser. **Elo ratings have been adopted to evaluate Language Learning Models (LLMs) in recent publications (Bai et al., 2022; Dettmers et al., 2023; Zheng et al., 2023) and a popular LLM leaderboard (https://arena.lmsys.org/).**

Reviewer comment (Novelty):
It is not clear whether the mentioned criteria are usually used for evaluating question answering, or if it is something proposed in this work; it seems that evaluating on different criteria separately is new.

Author response:
**Our Multi-Elo Rating System is inspired by MQM, though it is not identical to MQM.** The concept of breaking down the evaluation into several categories effectively improves the interpretation and transparency of the evaluation results.

Reviewer comment (Experiments):
It is not clear why only GPT-4 is used for the evaluation on different criteria, and not the other LLM (Claude-1) — this is important, especially because of the bias toward own outputs (mentioned by authors, too).

Author response:
**Due to cost constraints and the results in Table 1, we did not include Claude-1 in Table 4.** As demonstrated in Table 1, Claude-1 shows similar results to GPT-4, therefore, we used only GPT-4 in subsequent experiments.

Reviewer comment (Writing):
Some terms should be revised for the sake of clarity (e.g., replace "games" by "pairwise comparisons", "answer order" by "order of outputs").

Author response:
We will also clearly define the terms, like “game” and “hesitancy”, in the future revisions of our manuscript

Reviewer comment (Writing):
In the Introduction, the phrase "we observe hesitancy among human" is unclear — what is "hesitancy" in this context?

Author response:
We will also clearly define the terms, like “game” and “hesitancy”, in the future revisions of our manuscript

Reviewer comment (Evaluation):
It is not clear what the widely used evaluation methods and criteria are — for example, is calculating Elo rating from pairwise rankings widely used or only in some publications?

Author response:
Elo ratings are a method for calculating the relative skill levels of players in two-player games such as chess. In each game, the winner gains points from the loser. **Elo ratings have been adopted to evaluate Language Learning Models (LLMs) in recent publications (Bai et al., 2022; Dettmers et al., 2023; Zheng et al., 2023) and a popular LLM leaderboard (https://arena.lmsys.org/).**

Reviewer comment (Reproducibility):
Is the set of questions publicly available? If yes, on which link?

Author response:
The questions used in this work are available at https://github.com/lm-sys/vicuna-blog-eval/blob/main/eval/table/question.jsonl

Reviewer comment (Presentation):
In Section 2.3, the guidelines for the annotators are very important and should be part of the main sections, not only in the Appendix.

Author response:
We acknowledge this concern and will revise the manuscript to enhance clarity including the placement of the related work, the human annotation guidelines, and so on.

Reviewer comment (Writing):
Footnote 2: It seems that "games" refer to pairwise comparisons; the term "games" is confusing and should be replaced by "comparisons".

Author response:
We will also clearly define the terms, like “game” and “hesitancy”, in the future revisions of our manuscript

Reviewer comment (Presentation):
In Figure 2, what are "Assistant 1" and "Assistant 2"? There is no explanation in the table caption.

Author response:
Assistant 1 and Assistant 2 are the two models used in each comparison.

Reviewer comment (Presentation):
"The order of answers" — is this the order of generated outputs? What is "Assistant 1" and "Assistant 2"? What is exactly the order of the answers? Which is the first output and which is the second? Should they be presented at the same time in order to be ranked? Is the first on the left and the second on the right? Or the first is above the second? Or something else?

Author response:
Assistant 1 and Assistant 2 are the two models used in each comparison.

Reviewer comment (Experiments):
In Table 4, why is Claude-1 not used for the experiment of evaluating separately according to three aspects?

Author response:
**Due to cost constraints and the results in Table 1, we did not include Claude-1 in Table 4.** As demonstrated in Table 1, Claude-1 shows similar results to GPT-4, therefore, we used only GPT-4 in subsequent experiments.

Reviewer comment (Presentation):
Related work should follow the Introduction, not be placed almost at the end of the paper.

Author response:
We acknowledge this concern and will revise the manuscript to enhance clarity including the placement of the related work, the human annotation guidelines, and so on.

Reviewer comment (Experiments):
The expert evaluation section (2.4) discusses games, but these games are not introduced/presented ahead of this section: “we sample 200 games out of a total pool of 5280 games, which is statistically sufficient to yield meaningful insights into the model’s performance”.

Author response:
**Our use of the term "games" refers to pairwise comparisons of model outputs**, similar to chess matches in the Elo rating system. Each expert annotator evaluated 10 such comparisons. We will update our manuscript as you suggested.

Reviewer comment (Experiments):
“By incorporating multiple LLM judges, we can establish a more comprehensive and unbiased assessment of the generated answers in our study.” - Generally agree with this statement, however in the experiments only GPT-4 and Claude (Antrophic) are used, how is this in line with your claims? Only two LLM-based models is not reflective of the goals of your statement. In addition, Claude is not even mentioned in the Introduction or anytime before Section 2.5, leaving the reader the impression you are only using GPT4.

Author response:
**By definition, “multiple” means “more than one”, and our claim remains consistent with our methodology.** We incorporated Claude-1 alongside GPT-4 to counter potential biases, especially relevant since our answers were refined post-generation by GPT-4. The inclusion of more LLMs was considered; however, we had concerns about their capabilities in accurately evaluating model outputs, coupled with limited resources for this study. This decision was a strategic choice to maintain focus and ensure quality in our evaluation process.

Reviewer comment (Evaluation):
The findings of the user study conducted by the authors are unsurprising. The length of the text is a well-known confounder in the evaluation of generated texts. Similarly, texts which contain grammatical errors can be quickly identified as having poor quality. Moreover, it is unsurprising that humans tend to be biased towards factually incorrect answers that look coherent and are grammatically correct (especially true if they lack in-domain knowledge).

Author response:
We believe that the value of our study lies not only in uncovering new insights but also in empirically validating existing assumptions within the field. While some of our findings may align with general expectations, their empirical substantiation is a crucial contribution. This validation is particularly significant in a rapidly evolving field like natural language processing, where many widely-held beliefs have yet to be rigorously tested. Our study offers concrete evidence to support or question these beliefs, thus advancing the overall understanding of natural language generation evaluation. **We believe that unsurprising findings should not be a reason for rejection, as the single-score Elo rating is still widely adopted, and general expectations require rigorous scientific confirmation.**

Furthermore, the study's design and the nuances of our findings offer deeper insights into the complexities of human and machine evaluations, going beyond mere confirmation of expected trends. For example, there are few grammatical errors in your review which are hard to be identified without the assistance of the grammar-checker. This actually confirms our claim that humans are not able to effectively identify the errors in the text. In summary, while some findings might seem predictable, their empirical confirmation and the detailed analysis provided are valuable contributions to the field.

Reviewer comment (Evaluation):
“Regarding the “Helpfulness” dimension, both expert annotators and GPT-4 consistently consider longer answers to be more helpful.” - of course, since they contain more information! Nevertheless, the authors only make vague assumptions and fail to quantify the degree of factuality to support their statements.

Author response:
As shown in Figure 11, we require the human annotators, including the crowd-sourced annotators and experts, to **consider the relevance of the information, whether it addresses the given question, and to take into account the depth of the response.**

Reviewer comment (Evaluation):
The setup is not consistent: it evaluates i) language proficiency, ii) factual accuracy, and iii) response length, and then proposes a rating system that accounts for i) language, ii) accuracy, and iii) helpfulness. Helpfulness criteria is not clearly motivated, and its inclusion is inconsistent with the initial evaluation criteria. Besides, the paper does not clearly describe how this aspect is being evaluated, to whom it is helpful and in which contexts.

Author response:
The initial phase of our evaluation concentrated on three key aspects: language proficiency, factual accuracy, and response length. These were selected to rigorously assess the biases of the LLM evaluation process. In transitioning to our Multi-Elo Rating System (MERS), we selected aspects i) language, ii) accuracy, and iii) helpfulness. This selection was made for demonstrative purposes, to showcase the system's capability in evaluating diverse and crucial elements of language generation. **It is important to note that our proposed MERS is adaptable and not limited to these three aspects.** The system is designed to be flexible, allowing for the inclusion of additional evaluation criteria as needed.

Reviewer comment (Evaluation):
There is no clear motivation in the paper on why the authors chose to focus on these particular three aspects (language proficiency, factual accuracy, and response length) of machine-generated texts. There are so many aspects of the generated texts that can be considered, and this paper fails to make a convincing argument on the selection of their evaluation criteria. While the authors acknowledge in the Discussion section that this may not be the perfect setup, they fail to motivate their experimental choice.

Author response:
The initial phase of our evaluation concentrated on three key aspects: language proficiency, factual accuracy, and response length. These were selected to rigorously assess the biases of the LLM evaluation process. In transitioning to our Multi-Elo Rating System (MERS), we selected aspects i) language, ii) accuracy, and iii) helpfulness. This selection was made for demonstrative purposes, to showcase the system's capability in evaluating diverse and crucial elements of language generation. **It is important to note that our proposed MERS is adaptable and not limited to these three aspects.** The system is designed to be flexible, allowing for the inclusion of additional evaluation criteria as needed.

Reviewer comment (Novelty):
The Related Work section is not very informative for the topic of the paper, particularly the paragraph focused on enumerating existing LLMs is not discussing evaluation at all (which is the main focus of the current paper). In addition, the authors fail to cite and refer to important bodies of work in the literature focused on both human and machine evaluation. I recommend a complete rewrite of this section and instead of presenting generalities, discuss papers that have addressed the same topic and are particularly relevant to your current work.

Author response:
In the related work, we present numerous researches that are related to our study, **covering the literature of both large language models and evaluation methods.** We will include more specific references to works directly addressing NLG evaluation in our future revisions.

Reviewer comment (Experiments):
In general, the experiment setup decisions made in this paper seem rather arbitrary and not well documented/supported by the arguments presented. The authors fail to discuss and include significant papers focused on the evaluation of natural language generation (NLG) models, and do not place their work in the context of recent literature on NLG evaluation.

Author response:
In the related work, we present numerous researches that are related to our study, **covering the literature of both large language models and evaluation methods.** We will include more specific references to works directly addressing NLG evaluation in our future revisions.