Unsupervised Evaluation for Question Answering with Transformers (2020)

Abstract: It is challenging to automatically evaluate the answer of a QA model at inference time. Although many models provide confidence scores, and simple heuristics can go a long way towards indicating answer correctness, such measures are heavily dataset-dependent and are unlikely to generalise. In this work, we begin by investigating the hidden representations of questions, answers, and contexts in transformer-based QA architectures. We observe a consistent pattern in the answer representations, which we show can be used to automatically evaluate whether or not a predicted answer span is correct. Our method does not require any labelled data and outperforms strong heuristic baselines, across 2 datasets and 7 domains. We are able to predict whether or not a model’s answer is correct with 91.37% accuracy on SQuAD, and 80.7% accuracy on SubjQA. We expect that this method will have broad applications, e.g., in semi-automatic development of QA datasets.

## introduction evaluation of a qa model usually requires humanannotated answer spans to compare a model's output with. at inference time, however, it is hard to automatically estimate whether an extracted answer span is correct. while many models can provide confidence scores, and other heuristics might be used to deduce whether a prediction is correct, such measures are heavily dataset-dependent and are not likely to generalise. hence, given a new domain, a costly procedure of human annotation needs to be initiated in order to provide an estimate of the model's accuracy. however, this approach naturally does not scale well to new unlabelled sequences. in this work, we investigate transformer-based qa models. we hypothesise that hidden representations of later layers in such models contain information related to correctness of answers. indeed, we observe a consistent pattern of closely clustered answer token representations in the top three layers, whenever bert correctly predicts an answer span (see figure 2 ). conversely, both true and predicted answer spans are clustered together with the remainder of the context, when an answer prediction is erroneous. with clustering, we refer to the transformation of high-dimensional token representations into 2-dimensional vector space through the employment of pca @xcite , followed by t-sne (van der @xcite . we furthermore see that correctly identified answer spans show a high mean cosine similarity across final layers (figure 1 ). before computing the cosine similarity between token representations, we apply pca to remove noise, and preserve 95% of the variance in the low-rank, orthogonal representation (see 2.4 for detailed information). we demonstrate how this insight can be used to predict whether or not a prediction from a transformer-based qa model is correct. we evaluate our method across two distinct qa datasets in figure 2 : hidden representations of answers are clustered separately from the remaining context for correct answer span predictions. this clustering is obtained by applying pca followed by t-sne (to save computational time), projecting the hidden representations for each token in a randomly chosen input sequence into r 2 . blue diamonds: question. red stars: answer. grey dots: context. english, covering 7 domains. we observe that the pattern can be used for automatic evaluation in both squad v2.0 @xcite and subjqa @xcite , a recently released dataset containing subjective questions and answers across several domains. we show that we can evaluate such models without any labelled test data, with an accuracy of 91.37% on squad, and 80.7% accuracy on the more challenging and diverse subjqa. contributions (i) we investigate how a transformer-based model encodes correct and incorrect answer spans in its hidden representations; @xcite we propose a method to leverage the information contained in these representations to predict whether a given answer span prediction is correct or not; (iii) we demonstrate that a combination of our method with simple heuristics yields near-perfect predictions of answer correctness. ## related work since automatic evaluation is considered an important topic in other areas of nlp, e.g. mt @xcite and summarisation @xcite , we want to draw attention to such techniques for qa. to the best of our knowledge, ours is the first proposal unsupervised qa evaluation method. one recent study which we take inspiration from present a layer-wise analysis of bert's transformer layers to investigate how bert answers questions @xcite . for each transformer layer, they project the model's hidden representations into r 2 to illustrate how bert clusters source squad subjqa question "what term did eisenhower use to describe the character of communism?" "are there any reviews on bath options at this hotel?" answer "atheistic" "great bathroom" bert qa incorrect correct heuristic correct incorrect heuristic ⊕ cosw incorrect heuristic ⊕ cosc correct cos l h(a) ∀@xmath0.07, 0.16, 0.26, 0.27, 0.31, 0.20] [0.06, 0.14, 0.29, 0.62, 0.63, 0.55] 2019 ), the study focuses on a single implementation of bert and exclusively exploited squad @xcite without inspecting bert's behaviour with respect to other, more challenging qa datasets where contexts belong to different domains. the latter is particularly important for real-world settings, which is why we also evaluate on subjqa. ## discussion the heuristic method we investigate in this work, based on features such as n-gram overlap between question and answer, yielded surprisingly high results on squad. on the other hand, the results for subjqa were quite low when using the heuristic only. this shows that, although a simple heuristic might be sufficient for a single dataset, it does not necessarily generalise across datasets and domains. conversely, our proposed method, which takes answer span similarities into account, was highly successful on subjqa without the need for any heuristic features, but only outperformed the squad baseline by 15-20% macro f1 score. combining the two methods yielded the best results across both data sets and all domains. this demonstrates that the information contained in the heuristic approach and in our proposed method are complementary. ## conclusion we have shown that the hidden representations of answers in transformer-based models can be used to predict whether or not that answer is correct. in combination with heuristic methods, we are able to predict the correctness of answers with a macro f1 score of 88.38% for squad and 76.42% for subjqa. apart from the applications in unsupervised evaluation of qa, we expect that this method can be applied to semi-automatic generation of qa datasets.