LINKAGE : Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLM s (2024)

Abstract: Non-Factoid (NF) Question Answering (QA) is challenging to evaluate due to diverse potential answers and no objective criterion. The commonly used automatic evaluation metrics like ROUGE or BERTScore cannot accurately measure semantic similarities or answers from different perspectives. Recently, Large Language Models (LLMs) have been resorted to for NFQA evaluation due to their compelling performance on various NLP tasks. Common approaches include pointwise scoring of each candidate answer and pairwise comparisons between answers. Inspired by the evolution from pointwise to pairwise to listwise in learning-to-rank methods, we propose a novel listwise NFQA evaluation approach, that utilizes LLMs to rank candidate answers in a list of reference answers sorted by descending quality. Moreover, for NF questions that do not have multi-grade or any golden answers, we leverage LLMs to generate the reference answer list of various quality to facilitate the listwise evaluation. Extensive experimental results on three NFQA datasets, i.e., ANTIQUE, the TREC-DL-NF, and WebGLM show that our method has significantly higher correlations with human annotations compared to automatic scores and common pointwise and pairwise approaches.

## introduction in recent years, studies on various aspects of large language models (llms) have been drawing significant attention, a majority of which are based on the task of factoid question answering (qa) @xcite @xcite . new evaluation metrics and benchmarks have also been proposed for assessing the factuality of llms @xcite . however, * corresponding author. evaluating nfqa is challenging since nonfactoid questions often involve subjective interpretations and the potential answers can be diverse instead of a definite fact. most prior work used automatic evaluation metrics such as measuring word overlaps (e.g., rouge @xcite and bleu @xcite ) and semantic similarities (e.g., bertscore @xcite ) with the ground truth answers. to ensure the evaluation reliability, a small amount of manual annotations are also incorporated to compare the nfqa performance. however, both of them have some limitations: automatic metrics like rouge, bleu, and bertscore cannot accurately measure the responses with semantically similar expressions or from a different but reasonable perspective respectively; human evaluations, although more accurate in measuring various aspects of the long-form answers, often require annotators to have related knowledge to be reliable and are too expensive to apply on a large scale. @xcite . moreover, even for humans, evaluation of nfqa can still be challenging due to the requirement of domain knowledge as well as subjective interpretations of the questions and judgment criterions. by ingesting large-scale data from multi-tasks, llms, such as the gpt series, have achieved compelling performance on numerous natural language processing (nlp) tasks, and sometimes even outperform humans @xcite . increasing attention has been drawn to leveraging llms as surrogates for large-scale evaluation on modelgenerated responses @xcite @xcite . following the routines of human evaluation, approaches that leverage llms as judges often adopt the ways of pointwise scoring that grades each candidate answer individually and pairwise comparisons that compare pairs of answers @xcite . the pair for comparison can be two candidate answers or a candidate answer and a ground truth answer. figure 1 shows a concrete example of these two approaches. pointwise grading is hard since the accurate perception of differences between each grade can be difficult. subtle differences between candidates may not be discerned and reflected in the final score. pairwise comparison is relatively easier and can be more accurate but it is not scalable to the large number of candidates when the comparisons are between candidates. in contrast, there is no such issue when comparing the pair of a candidate and a ground truth answer. however, it is not feasible when the ground truth is unavailable. moreover, when only a single ground truth exists, the evaluation may not be accurate to cover various aspects. inspired by the evolution of learning to rank in information retrieval, i.e., from pointwise to pairwise to listwise @xcite , we propose a listwise nfqa evaluation approach that leverages llms to conduct listwise ranking among varied-quality references, abbreviated as linkage. specifically, we use llms to assess a candidate answer by its rank in a list of reference answers sorted by quality descendingly. when there are ground truth answers of multiple grades, they can be used as the varied-quality references. when there is only one or no golden answer, we will construct some examples of multi-grade answers and utilize the in-context learning ability of llms to generate more reference answers of different quality. compared to the pointwise and pairwise approach, listwise ranking can yield more accurate assessment since the llm judge can take reference answers of various quality into consideration simultaneously. when only one reference answer is used, our method degenerates to pairwise comparisons with a ground truth answer. additionally, given an ordered reference answer list, llms only ingest the reference list and candidate answer once, which costs much less than comparing each reference answer with the candidate pairwise and aggregate the score. we conduct extensive experiments on three nfqa datasets: antique @xcite , the non-factoid portion of trec dl @xcite , and webglm @xcite . antique and trec dl have multi-grade manual annotations on the candidate answers while webglm is a non-factoid qa dataset based on retrieval augmented generation (rag) that provides retrieval passages and a single ground truth answer. under the settings where there are multiple, single, or none ground truth answers, our method outperforms the automatic similarity scores, as well as pointwise, and pairwise llm evaluation methods significantly in terms of the correlation with human judgments. by offering more accurate nfqa evaluation, our work can pave the way for future studies on improving nfqa performance, especially promoting llms to become more capable of answering complex questions. ## method in this section, we propose a listwise ranking among varied-quality references method (link-age) for evaluating nfqa. we formally define the task of nfqa evaluation and introduce some basic evaluation approaches, then introduce the details of our linkage. ## case study we conduct case studies to qualitatively compare the results of different methods. as shown in the figure 3 , because candidate answer 1 contains many matching keywords, even though it does not effectively answer the question, pointwise method and pairwise method both assign it a high score. as a result, the two candidate answers cannot be effectively distinguished. in contrast, our linkage can better distinguish the fine-grained quality differences between candidate answers and obtain results that are more consistent with humans. ## conclusion in this paper, we propose a listwise nfqa evaluation approach (linkage), which leverages llms to assess a candidate answer by its rank in a list of sorted reference answers. our approach is capable of considering reference answers of various quality simultaneously. therefore, it can enable llms to establish a better evaluation system and yield more accurate assessments. extensive experiments on three datasets, i.e., antique, trec-dl-nf, and webglm, demonstrate the effectiveness of our method, whether it is in situations with multi-grade ground truth answers, single-grade ground truth answers, or no ground truth. hoping this more accurate evaluation method can promote future research on nfqa.