Mitigating Bias for Question Answering Models by Tracking Bias Influence (2024)

Abstract: Models of various NLP tasks have been shown to exhibit stereotypes, and the bias in the question answering (QA) models is especially harmful as the output answers might be directly consumed by the end users. There have been datasets to evaluate bias in QA models, while bias mitigation technique for the QA models is still under-explored. In this work, we propose BMBI, an approach to mitigate the bias of multiple-choice QA models. Based on the intuition that a model would lean to be more biased if it learns from a biased example, we measure the bias level of a query instance by observing its influence on another instance. If the influenced instance is more biased, we derive that the query instance is biased. We then use the bias level detected as an optimization objective to form a multi-task learning setting in addition to the original QA task. We further introduce a new bias evaluation metric to quantify bias in a comprehensive and sensitive way. We show that our method could be applied to multiple QA formulations across multiple bias categories. It can significantly reduce the bias level in all 9 bias categories in the BBQ dataset while maintaining comparable QA accuracy.

## introduction large language models (lms) have been found to produce harmful output reflecting social stereotypes @xcite inherited from pretraining @xcite and fine-tuning corpus for many nlp tasks such as relation extraction @xcite , textual entailment @xcite and coreference resolution @xcite . existing literature observe bias contained in question answering (qa) models @xcite . building on the definition of bias in qa introduced in @xcite ; @xcite , we specifically focus on stereotyping behavior that the qa model's predictions reflect positive or negative associations with specific demographic groups. deploying a stereotyping qa model could lead to negative representational impacts by propagating stereotypes or denigration of demographics, and negative allocational impacts by introducing technology barriers for discriminated social groups @xcite @xcite . recent works have collected human-written evaluation datasets for the qa task to quantify the bias @xcite . however, bias mitigation methods for qa models are still under-explored due to several non-trivial challenges. first, existing bias mitigation works heavily rely on manually defined bias attribute words (e.g. pronouns for gender bias) @xcite @xcite or only support mitigating bias of a single category @xcite . an ideal method should be able to mitigate bias of different categories, especially the ones expressing stereotypes without explicit textual cues. second, identifying bias in qa is difficult as it requires commonsense reasoning of the content and interaction among context, question and predicted answer. third, limited supervision resources are available and there is no instance-level bias annotation. thus, bias mitigation methods relying on a supervised trained bias detector for decoding reweighting @xcite or reinforcement learning rewarding @xcite do not work. the bias of humans or models is formed mostly as a result of digesting incoming information. existing works show that people tend to hold similar harmful stereotypes after reading content expressing such stereotypes @xcite @xcite . @xcite further show that the unfair context example can amplify model biases under an in-context learning setting. motivated by the phenomenon of bias propagation and amplification, we propose bmbi, a bias mitigation method for qa models by tracking bias influence. we first gauge the bias level of negative towards dorothy (female) figure 1: model design of bmbi. the example illustrates the bias mitigation process of a query instance in terms of the gender identity bias category. the output space of the ruler instance defines the bias axis. since the common societal bias about emotional closedness is negative towards males (represented by the answer candidate "kenneth"), a positive bias level indicates the query instance contains a negative bias towards the protected group of males. the output of the qa task and bias detection module will be used to calculate losses respectively. a certain query qa instance by observing its influence on another ruler instance. the ruler instance contains bias labels for its answer candidates (i.e. stereotyped group if a candidate is chosen). we apply the influence by concatenating verbalized query instance with the input of ruler instance following the in-context learning paradigm, where the model learns from analogy and replicates the behavior of the examples provided in the context @xcite . then we obtain the predictions of the ruler instance while giving two separate parallel queries: one with a neutral instance as context, and the other one with the query instance as context. we could obtain the bias level of the query qa instance by tracing the difference in the predictions. with a small number of neutral and ruler instances used as additional resources, the bias detection module enables us to obtain bias level estimation without the need for instance-level bias annotation in an unsupervised manner. to perform bias mitigation, we use the detected bias level to form an additional learning objective to let the model perform multi-task learning of the bias mitigation task along with the original qa task. bmbi tackles the challenges mentioned above as 1) bias of different categories can be mitigated by using different ruler instances without relying on explicit textual cue; 2) instead of identifying bias from model predictions directly, we trace the prediction distribution shift of a ruler instance with known bias labels influenced by the query instance to utilize the lm's learning-from-analogy capabil-ity and decompose the bias detection task to an influence tracing problem; 3) there is no need to use instance-level bias annotations, making the bias mitigation possible with limited resources. since the evaluation metric for bias in qa models proposed in previous works is not sensitive, we propose a new metric that is more consistent and sensitive to reflect the subtle bias level difference. we evaluate bmbi on the improved evaluation mechanism based on the bbq evaluation dataset @xcite . the experimental results show that bmbi is able to significantly reduce the bias magnitude for at most 8.28 points foot_0 across 9 bias categories while keeping the model's qa performance to correctly answer questions. our contributions are summarized as follows: 1) we propose a bias detection module that is generalizable to various bias categories without the need of instance-level training data; 2) we develop a bias mitigation method for the qa task supporting multiple qa formulations and bias categories; 3) we propose an improved bias evaluation metric for multiple-choice qa; 4) the experimental results show that the bias mitigation module could mitigate various categories of bias while keeping the model's capability to answer questions correctly. existing bias mitigation methods either constrain their applications to diverse bias categories by depending on specific information that only available for certain categories (e.g. bias attribute words and heuristics), or requiring instance-level annotations (e.g. whether the instance is related to a specific protected group, whether certain bias is contained in the instance). in our work, we propose to use much less supervision signal and ensure the method can be applied to mitigate different categories of biases. social biases in question answering. existing works investigate how to quantify social biases contained in the qa models since general extrinsic bias metrics fail to capture the interaction among context, question and predicted answer @xcite . @xcite propose the first dataset for this purpose by using underspecified questions to assess model biases from gendered name-occupation association, nationality, ethnicity, and religion. @xcite investigate whether linguistic ethical interventions can amend a qa model's unethical behavior based on @xcite . @xcite show gender and demographic biases in qa models measured by accuracy of qa data subsets splited by the appearance of gender or demographic entities. @xcite further extend types of ambiguity and study bias for both closed and open-domain qa models. recently, @xcite developed the bbq evaluation dataset which covers more bias categories, and disambiguated questions besides the underspecified ones. existing works focus on analyzing and quantifying social biases in qa. to our knowledge, bias mitigation techniques for qa models are still under-explored. ## the bmbi bias mitigation method we propose bmbi to mitigate bias by optimizing towards a weaker bias magnitude. bmbi contains two components: 1) a bias detection module (introduced in section 4.1) that takes in the (ct, q, a; a ′ ) of a qa instance and produces the bias level if the predicted answer is a ′ ∈ a. 2) a bias mitigation method (introduced in section 4.2) on top of the base qa model, where we use the detected bias level to create an additional optimization objective to decrease the bias contained in the qa model. figure 1 demonstrates our proposed framework. ## bias evaluation mechanism we introduce the dataset and bias score definition used to quantify the bias exhibited by qa models. ## conclusion and future work we propose bmbi, a bias mitigation method for classification-based or generation-based qa models across various bias categories. the bias detection component identifies bias by tracing the bias influence of the query instance, and the bias mitigation component uses an additional loss to minimize the detected bias magnitude. we also introduce a new bias score metric for a more sensitive and fair evaluation. our method is shown to be effective by significantly reducing the bias magnitude while keeping its qa performance. we plan to apply the idea of mitigating bias via tracing influence on other tasks.