Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy (2021)

Abstract: The goal of question answering (QA) is to answer _any_ question. However, major QA datasets have skewed distributions over gender, profession, and nationality. Despite that skew, an analysis of model accuracy reveals little evidence that accuracy is lower for people based on gender or nationality; instead, there is more variation on professions (question topic) and question ambiguity. But QA’s lack of representation could itself hide evidence of bias, necessitating QA datasets that better represent global diversity.

## introduction question answering (qa) systems have impressive recent victories-beating trivia masters @xcite and superhuman reading @xcite -but these triumphs hold only if they generalize; qa systems should be able to answer questions even if they do not look like training examples. while other work (section 4) focuses on demographic representation in nlp resources, our focus is how well qa models generalize across demographic subsets. after mapping mentions to a knowledge base (section 2), we show existing qa datasets lack diversity in the gender and national origin of the people mentioned: english-language qa datasets mostly ask about us men from a few professions (section 2.2). this is problematic because most english speakers (and users of english qa systems) are not from the us or uk. moreover, multilingual qa datasets are often translated from english datasets @xcite . however, no work has verified that qa systems generalize to infrequent demographic groups. section 3 investigates whether statistical tests reveal patterns on demographic subgroups. despite skewed distributions, accuracy is not correlated with gender or nationality, though it is with professional field. for instance, natural questions @xcite systems do well with entertainers but poorly with scientists, which are handled well in triviaqa. however, absence of evidence is not evidence of absence (section 5), and existing qa datasets are not yet diverse enough to vet qa's generalization. ## mapping questions to entities we analyze four qa tasks: nq, 1 squad @xcite , qb @xcite and triviaqa @xcite . google cloud-nl 2 finds and links entity mentions in qa examples. 3 ## related work language is a reflection of culture. like other cultural artifacts-encyclopedias @xcite , and films @xcite -qa has more men than women. other artifacts like children's books have more gender balance but reflect other aspects of culture @xcite . the nlp literature is also grappling with demographic discrepancies. standard coreference systems falter on gender-balanced corpora @xcite , and @xcite create synthetic training data to reduce bias. similar coreference issues plague machine translation systems @xcite , and @xcite use qa to probe biases of nlp systems. @xcite show that there are shortcomings in qa datasets and evaluations by analysing their out-of-domain generalization capabilities and ability to handle question variation. joint models of vision and language suggest that biases come from language, rather than from vision @xcite . however, despite a range of mitigation techniques @xcite none, to our knowledge, have been successfully applied to qa, especially from the demographic viewpoint. ## discussion and conclusion this paper delivers both good news and bad news. while datasets remain imperfect and reflect societal imperfections, for many demographic properties, we do not find strong evidence that qa suffers from this skew. however, this is an absence of evidence rather 1 ). highly influential features (p-@xmath0.1), both positive (blue) and negative (red). higher number of ⋆'s signals higher significance. a broader analysis beyond person entities would indeed be a natural extension of this work. label propagation can expand the analysis beyond people: the hershey-chase experiment is associated with alfred hershey and martha chase, so it would-given the neighboring entities in the wikipedia link graph-be 100% american, 50% male, and 50% female. another direction for future work is accuracy under counterfactual perturbation: swapping real-world entities (in contrast with nonce entities in @xcite ) with different demographic values. nonetheless, particularly for professional fields, imbalances remain. the lack of representation in qa could cause us to think that things are better than they are because of simpson's paradox @xcite : gender and profession are not independent! for example, in nq, our accuracy on women is higher in part because of its tilt toward entertainment, and we cannot say much about women scientists. we therefore caution against interpreting strong model performance on existing qa datasets as evidence that the task is 'solved'. instead, future work must consider better dataset construction strategies and robustness of accuracy metrics to different subsets of available data, as well as unseen examples.