Debiasing Methods in Natural Language Understanding Make Bias More Accessible (2021)

Abstract: Model robustness to bias is often determined by the generalization on carefully designed out-of-distribution datasets. Recent debiasing methods in natural language understanding (NLU) improve performance on such datasets by pressuring models into making unbiased predictions. An underlying assumption behind such methods is that this also leads to the discovery of more robust features in the model’s inner representations. We propose a general probing-based framework that allows for post-hoc interpretation of biases in language models, and use an information-theoretic approach to measure the extractability of certain biases from the model’s representations. We experiment with several NLU datasets and known biases, and show that, counter-intuitively, the more a language model is pushed towards a debiased regime, the more bias is actually encoded in its inner representations.

## introduction state of the art neural language models such as bert @xcite usually work by pretraining an encoder to learn universal word representations, and then fine-tuning it on some classification or regression task. from a robustness point of view, such pretrain-and-fine-tune pipelines are known to be prone to biases that are present in data @xcite @xcite . various methods were proposed to mitigate such biases in a form of robust training, where a bias model is trained to capture the bias and then used to relax the predictions of a main model, so that it can focus less on biased examples and more on the "hard", more challenging examples @xcite @xcite 0 10 20 30 40 50 ∆ robustness 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ∆ bias extractability ρ = 0.626 figure 1 : amount of subsequence bias extracted from different language models vs. the robustness of models to the bias. robustness is measured as improvement of the model on out-of-distribution examples, while extractability is measured as the improvement of the probe's ability to extract the bias from a debiased model, compared to the baseline. we conduct experiments on two nli datasets and one fact verification dataset across a variety of debiasing methods and bias types, and test whether the bias removal is as successful as o.o.d evaluation suggests. surprisingly, we discover that making models robust from the perspective of the downstream task, causes the inner representations to encode more of the information about the specific bias in question. figure 1 shows an example of this trend in nli, where as robustness of the fine-tuned model to biased predictions increases, so does the ability of the probing classifier to extract bias. to summarize, we make the following contributions: ## methods we lay down a general framework for interpreting bias in inner model representations. given a model f θ : @xmath0, we assume that it can be decoupled into two stages: for example, in nli we assume that data samples are given as sentence pairs @xmath1, h) where p is a premise and h is a hypothesis. r (p, h) is the joint representation of the two, and this representation is then used by f to produce a prediction. in this work, we compare baseline models finetuned on some down-stream task to models debiased during the fine-tuning step. we produce representations from both types of models and measure the extractability of bias using a probing classifier. our probing tasks are defined in terms of "biasrevealing" properties, which are based on a-priori knowledge of the bias in question, and are able to distinguish between biased and unbiased samples from the original dataset. we next describe how to construct such probing tasks and appropriate datasets. ## results in this section, we first report our main findingthe correlation between the robustness of models. we then analyze each bias type and dataset in a more fine-grained manner. the only exception is negwords bias on mnli, where we report a negative correlation. as we analyze below, in this case some models do not improve on o.o.d data, but their compression still increases. this suggests that even though various debiasing methods are not always successful on different datasets and bias types, they still make bias more accessible in the representations. ## discussion and conclusion all of our experiments tested model-based debiasing, where a weak learner is used to capture biased features and discourage their use in model predictions. we discover that for both explicit and implicit modeling of the bias, this method exposes the biased features in the representation. when we fix the model and change the effect of debiasing (through the "focusing parameter" of dfl), we observe the same trend, where stronger bias mitigation leads to higher extractability of the modelled bias. based on our results, we stipulate that while current debiasing methods are good at making model predictions less biased, they are a bad proxy for learning unbiased text representations. the increased extractability of bias from the representations is not necessarily a bad trait: for example, the negwords task does not reveal more granular semantics of negation, which may be useful for the generalization of the model. by probing for linguistic properties using the senteval tasks, we also observe that debiased models do not make linguistic information less extractable, which can also contribute to their improvement in performance. we argue that future research should look for more interpretable methods for debiasing language mod-els, and consider the problem of finding robust, bias-free feature detectors. another domain where this finding may be alarming is social bias. previous studies show that word vectors contain social bias @xcite , and that debiasing them does not necessarily remove this information @xcite . our work shows that debiasing sometimes increases the information available about bias in the representations, albeit in the context of dataset bias rather than social bias. our work shows that unbiased @xmath3. we speculate that there exists a proxy for the language model that removes bias information from the representations and consequently improves the generalization of predictions out of distribution. future work could focus on methods that are both representation-robust and prediction-robust w.r.t various biases. finding such methods can help alleviate leakage of bias from data to the model's representations, without sacrificing the in-distribution performance.