What’s in a Name? Answer Equivalence For Open-Domain Question Answering (2021)

Abstract: A flaw in QA evaluation is that annotations often only provide one gold answer. Thus, model predictions semantically equivalent to the answer but superficially different are considered incorrect. This work explores mining alias entities from knowledge bases and using them as additional gold answers (i.e., equivalent answers). We incorporate answers for two settings: evaluation with additional answers and model training with equivalent answers. We analyse three QA benchmarks: Natural Questions, TriviaQA, and SQuAD. Answer expansion increases the exact match score on all datasets for evaluation, while incorporating it helps model training over real-world datasets. We ensure the additional answers are valid through a human post hoc evaluation.

## method: an entity by any other name this section reviews the open-domain qa (odqa) pipeline and introduces how we expand gold answer sets for both training and evaulation. ## experiment: just as sweet we present results on three qa datasets-nq, triviaqa and squad-on how including aliases as alternative answers impacts evaluation and training. since the official test sets are not released, we use the original dev sets as the test sets, and randomly split 10% training data as the held-out dev sets. all of these datasets are extractive qa datasets where answers are spans in wikipedia articles. ## analysis: does qa retain that dear perfection with another name? a sceptical reader would rightly suspect that accuracy is only going up because we have added more correct answers. clearly this can go too far. . . if we enumerate all finite length strings we could get perfect accuracy. this section addresses this criticism by examining whether the new answers found with augmented training and evaluation would still satisfy user information-seeking needs @xcite for both the training and test sets. ## conclusion: wherefore art thou single answer? our approach for matching entities in a kb is a simple approach to improve qa accuracy. we expect future improvements-e.g.,, entity linking source passages would likely improve precision at the cost of recall. future work should also investigate the role of context in deciding the correctness of predicted answers. beyond entities, future work should also consider other types of answers such as non-entity phrases and free-form expressions. as the qa community moves to odqa and multilingual qa, robust approaches will need to holistically account for unexpected but valid answers. this will better help users, use training data more efficiently, and fairly compare models.