Mind Your Bias: A Critical Review of Bias Detection Methods for Contextual Language Models (2022)

Abstract: The awareness and mitigation of biases are of fundamental importance for the fair and transparent use of contextual language models, yet they crucially depend on the accurate detection of biases as a precursor. Consequently, numerous bias detection methods have been proposed, which vary in their approach, the considered type of bias, and the data used for evaluation. However, while most detection methods are derived from the word embedding association test for static word embeddings, the reported results are heterogeneous, inconsistent, and ultimately inconclusive. To address this issue, we conduct a rigorous analysis and comparison of bias detection methods for contextual language models. Our results show that minor design and implementation decisions (or errors) have a substantial and often significant impact on the derived bias scores. Overall, we find the state of the field to be both worse than previously acknowledged due to systematic and propagated errors in implementations, yet better than anticipated since divergent results in the literature homogenize after accounting for implementation errors. Based on our findings, we conclude with a discussion of paths towards more robust and consistent bias detection methods.

## introduction humans are intrinsically biased, yet we desire our machines to be objective and make fair decisions. however, language models (lms) that empower much of the web as we know it are well known to contain biases that promote structural discrimination in downstream tasks against minorities and larger social groups alike @xcite . the word representations that are derived from these models (so-called word embeddings) also retain potentially harmful biases contained in the data that are used in the training process @xcite . to identify and ultimately address these biases, numerous techniques have been proposed for the detection of biases in lms. however, given the heterogeneity of published bias detection methods, which rely on a multitude of assumptions and use diverging definitions of bias, a thorough comparison is challenging @xcite . in practice, inconsistencies are observed even within the results of single methods @xcite . consequentially, a comprehensive overview of biases in lms remains elusive, while findings are inconsistent, inconclusive, and not suitable for determining approaches to debiasing. in this work, we aim to address these issues by reproducing and rigorously comparing recent stateof-the-art (sota) bias detection methods for contextualized word embeddings (cwes). we focus on four parameters for this comparison, namely the descriptors that are used for targets of bias, the mode of word contextualization for the extraction of cwes, the encoding levels that are used as output of the lms, and the rationale behind the evaluation metric. for each parameter choice, we investigate its respective influence on the resulting bias scores in an intra-method comparison. where feasible, we also conduct inter-method comparisons. based on our findings, we are able to trace some inconsistencies in published results to implementation errors and design choices (and remediate them), and provide recommendations and requirements for the future design of improved bias detection methods. contributions. we provide a comprehensive comparison of sota bias detection methods for cwes by extending method-specific design choices of individual methods to all compatible methods, based on extensive adaptation, reimplementation, and the refinement of test sets. we alleviate inconsistencies in bias detection methods, increase the comparability between methods, and identify approaches for future developments. our code and data are available at https://github. com/silkehusse/re-evaluating-bias . ## related work related work can be split into two categories, namely foundational work into bias detection in static lms, and bias detection in contextual lms. ## experimental setup we review and compare bias detection methods that are derived from weat. the rationale behind this selection is threefold. first, in contrast to subspacebased methods, weat is a supervised test that is backed by data and insights from the iat in psychology. second, weat-based tests enable us to compare bias in lms solely on the basis of embeddings and predictions. finally, weat-based tests have seen the most research contributions and are in need of subsumption. in the following, we discuss the experimental setup for this comparison. ## experimental results we first report the results of our replication experiments in comparison to results from the literature in sec. 4.1, before presenting the results of the extended experiments in sec. 4.2 and 4.3. ## discussion in our investigation, we uncovered several consistencies and inconsistencies in prior work. consistent with previous research, we obtain the highest and lowest number of significant bias scores for elmo and gpt-2, respectively. when including the results we obtain for opt and bloom, no correlation to the models' contextualization capacity is apparent. as a major inconsistency, we find that existing bias detection methods are not robust and minor differences in design choices yield divergent bias scores. while we can make some recommendations based on our findings, such as the use of group terms as stimuli for detecting gender bias, or the use of word-level encodings (instead of sentence representations), our results for contextualization and evaluation metric choices are inconclusive and point at a fundamental disagreement between methods. overall, we find cosine-based methods to be more robust, yet empathize that there is but one probability-based method in our comparison. furthermore, we trace some of the previously reported inconsistencies to erroneous implementations and the haphazard simplification of word sets, which constitute major sources of discrepancies between methods. nevertheless, after accounting for these issues, we find that the results homogenize in comparison to the disparate results that had been previously reported. at the very least, we hope that our findings serve as a guidebook for practitioners seeking to apply bias detection methods and help in identifying toeholds for debiasing lms while more sophisticated methods are developed. in conclusion, upon examining the descriptions, implementations, and relations of bias detection methods for clms, one is reminded of an anecdote attributed to john von neumann, who -upon being presented with a model that was over-reliant on pa-rameters -reportedly exclaimed in frustration that with just four parameters he could define a function that draws an elephant, and have it wiggle its trunk with five. of course, von neuman was referring to explicit parameters, while many of the design decisions underlying current bias detection methods are made implicitly or hidden away -at best in supplementaries, and at worst in code and test data. if bias in language models is the elephant in the room, then as a community we are currently not dissimilar to the blind men in the indian parable, who are learning about the elephant by touching different parts of its body and sharing their interpretations. given our blindness to the full picture, we would therefore do well to not also be mute and fail in clearly communicating our approaches. concretely, we should strive to establish robust estimators of bias, clean and curated test sets, and guidelines for their rigorous applications within the (often restrictive) confines of language model apis to avoid measuring the biases that we introduce in the process of detecting them. outlook and future work. we see no shortage of opportunities for future work as outlined above, and we would like to think of this paper as a call to action. however, in addition to the need for robust bias detection methods and suitable data sets, our own work also leaves room for further investigation, as we discuss subsequently. ## limitations in the following, we discuss the limitations of our study and -where applicable -how they could be addressed in future work. qualitative performance differences. in our findings, we highlight the differences in performance that arise between bias detection methods when varying the experimental parameters by quantifying the changes in observed bias scores. however, this does not necessarily point towards the qualitative reasons for these changes. while we investigated these where possible and provided explanations and interpretations, a thorough investigation of causal links between experimental parameters and detected biases would likely help in the development of more robust detection methods. in particular, a methodically sound interpretation of negative bias scores would be of substantial benefit. ## ethical statement no sensitive data were used in our experiments. the impact of bias in language models on the development of fair, accountable and transparent algorithms is substantial and stands to affect numerous groups and social minorities, which directly entails the importance of accurate and reliable bias detection methods that can be applied to a variety of biases. in this work, we demonstrate the lack of comprehensive methods and aim to identify common problems in existing methods to provide directions for future research that can address their shortcomings. we provide insights into how and when existing methods can be used in the meantime. at the same time, we argue that addressing biases in language models requires and deserves a concerted community effort (including domain expertise, data curation, and method development) instead of the current reliance on a patchwork of locally optimal detection methods that may ultimately end up hiding biases globally when an unsuitable method is deployed.