KPQA : A Metric for Generative Question Answering Using Keyphrase Weights (2021)

Abstract: In the automatic evaluation of generative question answering (GenQA) systems, it is difficult to assess the correctness of generated answers due to the free-form of the answer. Especially, widely used n-gram similarity metrics often fail to discriminate the incorrect answers since they equally consider all of the tokens. To alleviate this problem, we propose KPQA metric, a new metric for evaluating the correctness of GenQA. Specifically, our new metric assigns different weights to each token via keyphrase prediction, thereby judging whether a generated answer sentence captures the key meaning of the reference answer. To evaluate our metric, we create high-quality human judgments of correctness on two GenQA datasets. Using our human-evaluation datasets, we show that our proposed metric has a significantly higher correlation with human judgments than existing metrics in various datasets. Code for KPQA-metric will be available at https://github.com/hwanheelee1993/KPQA .

## introduction question answering (qa) has received consistent attention from the natural language processing community. recently, research on qa systems has reached the stage of generating free-form answers, called genqa, beyond extracting the answer to a given question from the context @xcite @xcite @xcite . however, as a bottleneck in developing genqa models, there are no proper automatic metrics to evaluate generated answers @xcite . in evaluating a genqa model, it is essential to consider whether a generated response correctly contains vital information to answer the question. there exist several n-gram similarity metrics such * this research was done while the author was affiliated with adobe research. context : ... , this process, called hypothesis testing, consists of four steps. , ... question : how many steps are involved in a hypothesis test? reference answer : four steps are involved in a hypothesis test. generated answer : there are seven steps involved in a hypothesis test . human judgment : 0.063 bleu-1 : 0.778 bleu-1-kpqa : 0.057 rouge-l : 0.713 rouge-l-kpqa : 0.127 figure 1 : an example from ms-marco @xcite where widely used n-gram similarity metrics does not align with human judgments of correctness. on the other hand, our kpqa-metrics focus on the key information and give low scores to incorrect answers similar to humans. as bleu @xcite and rouge-l @xcite , that measure the word overlaps between the generated response and the reference answer; however, these metrics are insufficient to evaluate a genqa system @xcite . for instance, in the example in figure 1 from the ms-marco @xcite , the generated answer receives a high score on bleu-1 (0.778) and rouge-l (0.713) due to the many overlaps of words with those in the reference. however, humans assign a low score of 0.063 on the scale from 0 to 1 due to the mismatch of critical information. as in this example, we find that existing metrics often fail to capture the correctness of the generated answer that considers the key information for the question. to overcome this shortcoming of the existing metrics, we propose a new metric called kpqametric for evaluating genqa systems. to derive the metric, we first develop keyphrase predictor for question answering (kpqa). kpqa computes the importance weight of each word in both the generated answer and the reference answer by considering the question. by integrating the output from the kpqa, we compute the kpqa-metric in two steps: (1) given a {question, generated answer, reference answer}, we compute importance weights for each question-answer pair {question, generated answer} and {question, reference answer} using a kpqa; (2) we then compute a weighted similarity score by integrating the importance weights into existing metrics. our approach can be easily integrated into most existing metrics, including ngram similarity metrics and the recently proposed bertscore @xcite . additionally, we newly create two datasets for assessing automatic evaluation metrics with regard to the correctness in the genqa domain. we first generate answers using state-of-the-art genqa models on ms-marco and avsd @xcite where the target answers are natural sentences rather than short phrases. we then collect human judgements of correctness over the 1k generated answers for each dataset. in experiments on the human-evaluation datasets, we show that our kpqa-metrics have significantly higher correlations with human judgments than the previous metrics. for example, bertscore-kpqa, one of our kpqa-integrated metrics, obtains pearson correlation coefficients of 0.673 on ms-marco whereas the original bertscore obtains 0.463. further analyses demonstrate that our kpqa-metrics are robust to the question type and domain shift. overall, our main contributions can be summarized as follows: ## preliminaries: automated text evaluation metrics bertscore is a recently proposed text evaluation metric that use pre-trained representations from bert @xcite . bertscore first computes the contextual embeddings for given references and candidates independently with bert, and then computes pairwise cosine similarity scores. when computing similarity, bertscore adopts inverse document frequency (idf) to apply importance weighting. ## proposed metric for evaluating genqa to build a better metric for genqa, we first propose kpqa. by considering the question, the kpqa assigns different weights to each token in the answer sentence such that salient tokens receive a high value. we then integrate the kpqa into existing metrics to make them evaluate correctness as well. ## related work one important next step for current qa systems is to generate answers in natural language for a given question and context. following this interest, several generative (abstractive) qa datasets @xcite @xcite , where the answer is not necessarily in the passage, have recently been released. since the task is to generate natural language for the given question, the qa system is often trained with seq2seq @xcite objective simi-larly to other natural generation tasks such as neural machine translation. hence, researchers often use n-gram based similarity metrics such as bleu to evaluate the genqa systems, following other natural language generation tasks. however, most of these n-gram metrics including bleu were originally developed to evaluate machine translation and previous works @xcite @xcite have shown that these metrics have poor correlations with human judgments in other language generation tasks such as dialogue systems. as with other text generation systems, for genqa, it is difficult to assess the performance through ngram metrics. especially, n-gram similarity metrics can give a high score to a generated answer that is incorrect but shares many unnecessary words with the reference answer. previous works @xcite @xcite have pointed out the difficulty of similar problems and studied automated metrics for evaluating qa systems. inspired by these works, we focus on studying and developing evaluation metrics for genqa datasets that have more abstractive and diverse answers. we analyze the problem of using existing n-gram similarity metrics across multiple genqa datasets and propose alternative metrics for genqa. ## conclusion in this paper, we create high-quality human judgments on two genqa datasets, ms-marco and avsd, and show that previous evaluation metrics are poorly correlated with human judgments in terms of the correctness of an answer. we propose kpqa-metric, which uses the pre-trained model that can predict the importance weights of words in answers to a given question to be integrated with existing metrics. our approach has a dramatically higher correlation with human judgments than existing metrics, showing that our model-based importance weighting is critical to measure the correctness of a generated answer in genqa.