CHIE : Generative MRC Evaluation for in-context QA with Correctness, Helpfulness, Irrelevancy, and Extraneousness Aspects (2024)

Abstract: The evaluation of generative models in Machine Reading Comprehension (MRC) presents distinct difficulties, as traditional metrics like BLEU, ROUGE, METEOR, Exact Match, and F1 score often struggle to capture the nuanced and diverse responses. While embedding-based metrics such as BERTScore and BARTScore focus on semantic similarity, they still fail to fully address aspects such as recognizing additional helpful information and rewarding contextual faithfulness. Recent advances in large language model (LLM) based metrics offer more fine-grained evaluations, but challenges such as score clustering remain. This paper introduces a multi-aspect evaluation framework, CHIE,incorporating aspects of C orrectness, H elpfulness, I rrelevance, and E xtraneousness. Our approach, which uses binary categorical values rather than continuous rating scales, aligns well with human judgments, indicating its potential as a comprehensive and effective evaluation method.

## introduction evaluating generative models in machine reading comprehension (mrc) presents distinct challenges, as traditional n-gram-based metrics like bleu @xcite , rouge @xcite , exact match (em), and f1 score often prove inadequate. these metrics are typically limited in their ability to assess the generalization capabilities of generative models, which are characterized by their production of diverse and nuanced responses. to address the n-gram matching problem, embedding-based metrics, i.e., bertscore @xcite and bartscore @xcite , focus on semantic similarity assessments of the ground truth and prediction. moreover, various evaluation criteria have been proposed to assess model outputs for generalized evaluation methods. for instance, usr @xcite evaluates dialogue responses based on fluency, relevance, and knowledge conditioning using a roberta-base model @xcite . similarly, unieval @xcite employs t5 @xcite to assess qa tasks from multiple perspectives, encoding texts as questions and answers and scoring them across various dimensions. however, these methods typically require datasets for fine-tuning, which can limit their applicability. recent advances in large language models (llms) based metrics, such as a gpt-based metric for translation @xcite , summarization @xcite , and dialogue @xcite ) tasks, offers more fine-grain evaluations using continuous rating scales. despite these improvements, challenges like score clustering remain. this is because generative answers produced by llms require more generalized measurements than extractive ones. moreover, these evaluation methods have mainly been used for high-resource languages like english. for lowresource languages, current research has not been thoroughly tested, creating a gap in understanding their performance in these languages. in this paper, we propose a multi-aspect evaluation framework to assess the generalization of in-context learning called chie comprising four aspects: correctness, helpfulness, irrelevance, and extraneousness. our work distinguishes itself from existing evaluation metrics, such as f1 and bertscore, as illustrated in figure 1 . in particular, we introduce a multi-aspect evaluation scheme that delivers a more comprehensive and detailed analysis of a model's ability to present information. unlike other llm-based evaluations, such as llm-eval, which provides a single numeric output (e.g., three on a 1-5 scale) that lacks explainability and can be challenging for human interpretation, our approach uses binary categorical values with objectively defined classes. our method ensures explainability and human-interpretable scoring and is specifically designed for the mrc task. to demonstrate the generalization of our evaluation method, we evaluate machine reading comprehension (mrc) capabilities in a multilingual environment. in particular, we evaluate six models on three languages compared to two evaluation metrics using xquad @xcite . our findings reveal that commonly used metrics, such as f1, em, and bertscore, lack generalizability and do not accurately reflect the robustness of the evaluated models. in contrast, our experiments show that chie consistently aligns with human judgments, indicating its potential as a more reliable alternative for evaluating model responses. furthermore, models evaluated using our proposed metric exhibit improved generalization compared to previous methods, suggesting that chie is more effective at capturing performance nuances across diverse scenarios. this is particularly significant in complex and ambiguous cases where traditional metrics fall short, underscoring the need for more sophisticated evaluation frameworks. in summary, our main contributions are as follows: ## proposed method in this section, we discuss an llm-based evaluator covering multiple aspects of mrc called chie. we first describe the desired features in section 3.1. second, we provide evaluation criteria for mrc evaluations in section 3.2. last but not least, section 3.3 explains the components of chie-based prompting. ## experimental results in this section, we report experimental results from three studies. section 5.1 compares our multiaspect approach, chie, with two single-aspect metrics, f1 and bertscore. section 5.1 explores the possibility of automating multi-aspect evaluations using an llm. section 5.3 provides a componentwise analysis of chie through a/b preference evaluation using humans and an llm. ## conclusion we present chie, a novel automatic evaluation framework using gpt-4 for assessing mrc model responses. in comparison to single-aspect measures such as bertscore, chie provides a more holistic means of assessing mrc responses by assessing the helpfulness of the answer and screening for irrelevancy and out-of-context information in addition to correctness. we also explore the possibility of using llms as evaluators. the results demonstrate potential for further development for using an llm in a completely automated evaluation process or as an evaluator to reduce the human evaluation workload.