QR el S core: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance (2022)

Abstract: Existing metrics for assessing question generation not only require costly human reference but also fail to take into account the input context of generation, rendering the lack of deep understanding of the relevance between the generated questions and input contexts. As a result, they may wrongly penalize a legitimate and reasonable candidate question when it (1) involves complicated reasoning with the context or (2) can be grounded by multiple evidences in the context.In this paper, we propose QRelScore, a context-aware Relevance evaluation metric for Question Generation.Based on off-the-shelf language models such as BERT and GPT2, QRelScore employs both word-level hierarchical matching and sentence-level prompt-based generation to cope with the complicated reasoning and diverse generation from multiple evidences, respectively.Compared with existing metrics, our experiments demonstrate that QRelScore is able to achieve a higher correlation with human judgments while being much more robust to adversarial samples.

## introduction question generation (qg) systems aim to generate natural language questions that are relevant to and usually can be answered by a given piece of input text @xcite @xcite . qg can be used to improve various applications, such as question answering (qa) @xcite @xcite , conversational systems @xcite , and information retrieval (ir) @xcite . meanwhile, it has long been criticized that qg models usually suffer from the semantic drift problem owing to the widely adopted likelihood-based training, i.e. the models ask questions that are not relevant to and can not be supported by the context @xcite . thus, how to accurately evaluate the relevance between generated questions and the context is attracting more and more attention. one of the most accurate evaluation methods is human evaluation. however, human evaluation is expensive, time-consuming, and non-reproducible. therefore, it is necessary to develop automatic evaluation metrics for question generation systems. traditional automatic metrics (e.g. bleu @xcite , rouge @xcite and me-teor @xcite ) measure the n-gram overlap between the candidate and corresponding reference question, but they often fail to robustly match paraphrases. more recently, q-bleu @xcite and bert-based metrics such as bertscore @xcite , moverscore @xcite and ls_score @xcite were proposed to evaluate the answerability and semantic similarity of a candidate question, achieving better correlation with human judgments. however, on the one hand, they compute the similarity between the system output and the reference without considering the crucial input context of generation. therefore, they cannot properly capture the reasoning relationship between the generated output and input context. on the other hand, comparing with a reference question omits the incompleteness of the reference: we can ask different questions based on the same context by paying attention to different information (or evidence) in it, while the reference question only represents one possible output. as a result, existing qg or text generation metrics struggle in evaluating the quality of candidate questions that (i) involve complicated reasoning with the context, or (ii) are generated from the evidence in the context that differs from in this paper, we present qrelscore , an automatic reference-free evaluation metric for question generation @xcite . qrelscore addresses the weaknesses above by considering the context-aware relevance in a word-and sentence-level manner. on the one hand, inspired by the hierarchical procedure taken by masked language models such as bert to understand a question @xcite , qrelscore understands the word-level relevance by explicitly capturing the reasoning relationship between the candidate tokens and the context tokens. on the other hand, based on the benefit of intra-sentence coherence in the autoregressive language models such as gpt2 that originates from the word-by-word nature of human language production, the sentence-level relevance is measured by the overall factual consistency between the candidate and all the possible evidences in the context. we verify the effectiveness and efficiency of qrelscore through various experiments. first, we demonstrate that qrelscore can improve the performance of question answering: by serving as a reward to train a qg model with reinforce-ment learning and then use it to augment a qa dataset (e.g. the squad dataset @xcite ), the performance of a qa model can be improved by fine-tuning on the augmented dataset. second, qrelscore achieves a state-of-the-art correlation with human judgments on the candidates generated by the existing qg models. furthermore, when considering the available human reference of the dataset in qrelscore, we present a reference-augmented version, ref-qrelscore, which achieves an even higher correlation. last, extensive experiments on the robustness test also demonstrate that qrelscore has a stronger ability to discriminate against adversarial samples when compared to existing metrics. ## qrelscore metric in this section, we formulate our reference-free evaluation metric qrelscore based on the offthe-shelf pre-trained language models. specifically, qrelscore consists of two scoring components: the local relevance matching (qrel lrm ) component and the global relevance generation (qrel grg ) component. the former is used to handle the candidates involving complicated reasoning with the contexts by computing word-level similarity using layer-wise embeddings and cross attention, while the latter is responsible for measuring the factual consistency between the candidate and all evidences of a given answer by comparing the difference in the confidence of generating the context with or without a prompt. based on the local and global relevance measurement, qrelscore can not only handle the candidate involving complicated reasoning with the context but also pay equal attention to all evidences of a given answer in the context and ensure the fluency of generation. figure 1 illustrates the computation of qrel lrm and qrel grg . ## conclusion existing evaluation metrics for question generation are still reference-based and ignore the crucial input context of generation, lacking a deep understanding of the relevance between the generated questions and context. to address these issues, we propose qrelscore, which measures the word-and sentence-level relevance through the off-the-shelf language models. extensive experiments demonstrate that qrelscore achieves start-of-the-art correlation with human judgments and makes up for the shortcomings of existing reference-based metrics.