PEDANTS : Cheap but Effective and Interpretable Answer Equivalence (2024)

Abstract: Question answering (QA) can only make progress if we know if an answer is correct, but current answer correctness (AC) metrics struggle with verbose, free-form answers from large language models (LLMs). There are two challenges with current short-form QA evaluations: a lack of diverse styles of evaluation data and an over-reliance on expensive and slow LLMs. LLM-based scorers correlate better with humans, but this expensive task has only been tested on limited QA datasets. We rectify these issues by providing rubrics and datasets for evaluating machine QA adopted from the Trivia community. We also propose an efficient, and interpretable QA evaluation that is more stable than an exact match and neural methods (BERTScore).

## introduction qa evaluation is a necessary pipeline to train and optimize qa models @xcite . from a model selection perspective, section 5.3 shows that a weak evaluation can lead to incorrect conclusions about the ranking of models. from a training perspective, having the right objective to train a model is an essential step to guide the model to the right direction @xcite shows that keeping models, data, and test evaluation fixed, if we improve automatic evaluation during training time, the trained model improves on test set accuracy than using a more rigorous training evaluation-exact match (em). however, one of the reasons em is popular is because it is efficient and interpretable compared to other evaluations @xcite . twelve out of fourteen papers @xcite @xcite in standard (em) and neural evaluations (bem) are unstable on different styles of qa datasets, where em is only good for evaluating factoid answers and bem is mostly good at answers that require reasoning. section 4 shows that only black-box llm evaluators (gpt-4) are strong short-form ac judges but they require much more cost and runtime than standard evaluations; open-sourced llms struggle at judging incorrect answers, generating many false positives. to come up with a consistent and structured short-form and factoid qa evaluation, we revise existing rules from the trivia community, integrating standardized ac rubrics from @xcite and the efficient qa competitions @xcite . the rules help us define machine ac, especially for the challenging long tail scenarios (section 3). we also incorporate difficult qa examples from the jeopardy! community to evaluate current qa evaluation metrics (appendix g). we then use the our ac framework to prompt gpt-4 @xcite and distil ac knowledge to train a two-level classifier that goes beyond em and provides a more fine-grained, interpretable and efficient ac evaluation. this broader view of qa evaluation is necessary because modern llm generate more verbose answers @xcite , making em less effective. out of the examples we analyzed for em, 90% of answers have correct candidate answers missing from reference answer sets, which humans consider correct but em does not (section 5). our contributions to automatic ac evaluation are: ## learning evaluation from the trivia community using llms to evaluate qa is increasingly popular @xcite . however, existing llm evaluations have inconsistent definitions, rubrics, and prompts to evaluate specific benchmark datasets @xcite . since llms are sensitive to change of prompts/definitions, comparing results among various llm evaluation papers becomes hard. on the other hand, although em is still the most popular standard evaluation, its lack of formal correctness definitions makes it unstable and less effective on more verbose qa models (section 5). the imperfections of current evaluations urges for starting rubrics and frameworks to evaluate models more fairly. luckily, we are not starting from scratch! because many of the qa data and evaluations-from watson on jeopardy! to triviaqa-come from the trivia community, @xcite argues that researchers should not just take their data but also examine their rules to bridge the gap between nlp and human community needs. moreover, the trivia community reflects decades of evaluating ac in high-stakes human competitions that need comprehensive and uniform rules to ensure fairness. because the subject areas are meant to cover the standard school knowledge, the curriculum breadth reflects many of the evaluation challenges for llms. while they are not perfect, they are more comprehensive than any existing automatic qa evaluation framework @xcite . we propose ac rules from gold standard human qa evaluations: naqt (national academic quiz tournaments, 2024), jeopardy! @xcite , and efficientqa @xcite . ## efficient qa evaluation evaluation is not a task like qa itself where the size of the model is a secondary consideration; an effective method (i.e., one that could be adopted) needs to be accurate, fast, and low-cost @xcite . this section's goal is to match the effectiveness of the llm evaluation while minimizing computation, latency, and disk space. em and token f 1 evaluations are fast and require no storage but are superficial. llm evaluations are trainable but require substantial disk space, cost, computational time, and latency. in aggregate, they might be too expensive for universal adoption in standard qa training and evaluation pipelines. by revising ac for text-based qa, we can construct an efficient automatic evaluation that not only aligns with our ac decision-making process but also goes beyond token-level matching that follows the human mental judgment process. ## evaluation setup an effective evaluation of evaluation methods requires validations on diverse data and models. we want to test limitations automatic evaluations on many datasets and qa models to 1: verify their effectiveness on different styles of datasets, evaluating its strengths over other metrics and weaknesses regarding different the styles of answers; 2: evaluate their stabilization across answers generated by various qa models (different sizes, different pretraining data, black-box and open-source models...) on the same dataset, which is important to provide a more fair comparison to qa models on a benchmark; 3: ensure the test data is not restricted by the patterns of the training data to prevent overfitting on learned metrics. ## error analysis with available datasets with human judgments, this section analyzes the human agreement, time and monetary cost, and model ranking accuracy. then we dive into examples that are challenging to the standard evaluation methods (token matching and neural evaluation) but can be handled by pedants. ## related work drawing on community expertise there is a recent trend of using data from well-defined communities ethically @xcite , and the trivia community is no different. while their data have been used for nlp tasks-from watson's tour-de-force on jeopardy! to triviaqa-the data have been used without attending to the norms of how the trivia community uses it. we should listen to their evaluations if we adopt their data. machine qa evaluations qa evaluations span a continuum-easy to assess using em for short 7 see examples in appendix j with the continual development of machine qa models' ability to answer questions, the area of qa evaluation still falls behind, where most research papers still use standard evaluation metrics-em or f 1 ( ## conclusion automated qa evaluation is an important pipeline for developing better models. however, evaluation metrics like em are popular for their efficiency, but the lack of ac rubrics and diverse evaluation datasets still hold us from exploring and building more robust and stable qa metrics competitive with llm evaluations, which is not always available. a basic beginning framework derived from qa human experts can enhance and generalize automated evaluation methods' agreement more with human judgments. though not perfect, pedants requires few computational resources and time but more robust across benchmark datasets. future work can integrate and refine these rubrics into long-form qa and novel qa, where we can collect better evaluation data and improve fact-checking and commonsense reasoning of ac; future work can also combine efficient metric with llm to improve runtime evaluation efficiency and reduce cost.