Eliciting Bias in Question Answering Models through Ambiguity (2021)

Abstract: Question answering (QA) models use retriever and reader systems to answer questions. Reliance on training data by QA systems can amplify or reflect inequity through their responses. Many QA models, such as those for the SQuAD dataset, are trained and tested on a subset of Wikipedia articles which encode their own biases and also reproduce real-world inequality. Understanding how training data affects bias in QA systems can inform methods to mitigate inequity. We develop two sets of questions for closed and open domain questions respectively, which use ambiguous questions to probe QA models for bias. We feed three deep-learning-based QA systems with our question sets and evaluate responses for bias via the metrics. Using our metrics, we find that open-domain QA models amplify biases more than their closed-domain counterparts and propose that biases in the retriever surface more readily due to greater freedom of choice.

## introduction question answering (qa) systems use reader and retriever models to learn and parse information from knowledge bases such as wikipedia. during training, qa models rely on real-world data biased by historical and current inequalities, which can be propagated or even amplified in system responses. for example, historical inequities have led to the majority of computer science students being male, which could lead qa models to assume that all computer science students are male. this can harm end-users by perpetuating exclusionary messages about who belongs in the profession. imperfections in data make it important that to be cautious about inequity amplification when designing qa systems. conceptualizations of "bias" and its consequences vary among studies and contexts @xcite . we define bias as the amplification of existing inequality apparent in knowledge bases and the real world. this may be through exacerbating empirically-observed inequality, e.g., by providing a list of 90% males in an occupation that is 80% male, or when systems transfer learned inequality into scenarios with little information, e.g., a model is given irrelevant context about jack and jill and is asked who is a bad driver @xcite . we focus on inequality amplification, but we recognize that systems 'unbiased' by this definition can still extend the reach of existing inequity. to mitigate past inequity, we must move beyond 'crass empiricism' to design systems reflecting our ideals rather than our unequal reality @xcite . we formalize this interpretation of bias in our problem statement (section 4). studies of bias in machine learning have become increasingly important as awareness of how deployed models contribute to inequity grows @xcite . previous work in bias shows gender discrimination in word embeddings @xcite , coreference resolution @xcite , and machine translation @xcite . within question answering, prior work has studied differences in accuracy based on gender @xcite and differences in answers based on race and gender @xcite . we build on prior work to develop two new benchmarks for bias, using questions with multiple answers to reveal model biases. our work builds upon social science studies showing how ambiguous questions can elicit internal information from subjects @xcite . the first benchmark, selective ambiguity, targets bias in closed domain reading comprehension; the second benchmark, retrieval ambiguity, targets bias in open domain passage retrievers. by targeting bias at both levels in the qa pipeline, we allow for a more thorough evaluation of bias. we apply our benchmarks to a set of neural models including bert @xcite and dpr @xcite , test for gender bias, and conclude with a discussion of bias mitigation. our contributions are as follows. we (1) develop a set of bias benchmarks for use on closed and open domain question answering systems, (2) analyze three qa models on the squad dataset using these benchmarks, and (3) analyze the propagation of bias at the retriever and reader levels. ## related work we provide a brief overview of prior work in bias, both in nlp and question answering (qa), along with a description of the negative effects of bias. ## ambiguity: a social science perspective we adopt ideas from the social sciences which demonstrate ambiguity as a revelatory mechanism for bias. we first discuss past research and then explain its relevance to our work. ambiguous questions, which lack a clear answer or have multiple possible answers, force individuals to rely on unconscious biases and selfserving traits @xcite due to the lack of structure, allowing for leeway in its interpretation. when answering ambiguous questions, people select the interpretation which makes them look best @xcite , as shown through studies involving psychology students @xcite , football players @xcite , and anxious subjects @xcite . in legal settings, ambiguous evidence can lead jurors to rely on implicit biases rather than evidence to make decisions @xcite . ambiguity serves as a modal to explore what factors people and systems use to make choices when allowed more freedom in the absence of certainty @xcite . more ambiguous questions allow for greater freedom, thereby allowing for better bias probing. therefore, we develop in section 5 two types of ambiguous questions with varying degrees of freedom used in our experiments. ## problem statement we define bias in qa formally and develop three bias metrics in section 5 based on this definition. we consider the problem of answering questions, q 1 • • • q n , where q i can be thought of as a sequence of words. we use a question answering (qa) sys-tem, f (q i , c i ), where c i is the context, which is either pre-determined, in the closed-domain scenario, or generated through a retriever function g(q i ), in the open-domain scenario. each question has a set of answers, a @xmath0,1 • • • a i,j }, where the answer set, a i , can be empty. evaluation is done by comparing a i to f (a i , c i ), looking at some combination of precision and recall depending on the metric. to investigate bias, we consider membership in k protected classes for each answer, p 1 (a i,j ) • • • p k (a i,j ), where p k defines what type of membership a i,j holds in class k, and we apply the same idea to retrieval systems. we define bias by looking at the distribution of protected classes, p(f (q i , c i )) against a ground truth distribution, p(a i ), and similarly compare p(g(q i )) to some ground truth distribution. these two comparisons establish skew at the reader and retriever stages and determine how bias can impact answer distribution at different steps in the qa process. ## bias benchmarks we develop two benchmarks to probe for bias; each consists of (1) a set of ambiguous questions automatically generated from templates/scraped data; and (2) an evaluation metric which measures the degree of bias of a model's responses. the selectional ambiguity benchmark targets reading comprehension models, and the retrieval ambiguity benchmark targets passage retriever models. ## experiments we apply our bias metrics on three qa models, each trained on the squad dataset @xcite . ## discussion we develop a preliminary study of ambiguity as a medium for eliciting bias and find that we fail to discover bias in our qa models using selectional ambiguity but do discover gender bias using retrieval ambiguity. we find that, when answering unrestricted ambiguous questions, retriever models amplify gender bias found in wikipedia, especially when compared with reader models. our ability to elicit bias by easing restrictions on ambiguity follows patterns from psychology @xcite , where increased ambiguity in questions allows for improved probing of bias. figure 1 : gender disparity for the eight most male and most female jobs. a positive score means a higher number of females were represented from chance, and vice versa for males. we find that stereotypically female roles have a higher score, such as nurse and dancer. ## conclusion we claim that ambiguous questions can serve as a mechanism for discovering how qa systems contribute to exacerbating or ameliorating inequity in the world. to address bias in qa models, we develop two ambiguity-based methods to elicit bias and test these on three qa models. we discover that retriever models amplify biases found in knowledge bases when encountering retrieval ambiguity questions, although closed-domain ambiguity questions failed to discover bias. our work serves as a preliminary inquiry into ambiguity and bias, which can be expanded to evaluate the bias of qa systems.