Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2020)

Abstract: Advances in language modeling architectures and the availability of large text corpora have driven progress in automatic text generation. While this results in models capable of generating coherent texts, it also prompts models to internalize social biases present in the training corpus. This paper aims to quantify and reduce a particular type of bias exhibited by language models: bias in the sentiment of generated text. Given a conditioning context (e.g., a writing prompt) and a language model, we analyze if (and how) the sentiment of the generated text is affected by changes in values of sensitive attributes (e.g., country names, occupations, genders) in the conditioning context using a form of counterfactual evaluation. We quantify sentiment bias by adopting individual and group fairness metrics from the fair machine learning literature, and demonstrate that large-scale models trained on two different corpora (news articles, and Wikipedia) exhibit considerable levels of bias. We then propose embedding and sentiment prediction-derived regularization on the language model’s latent representations. The regularizations improve fairness metrics while retaining comparable levels of perplexity and semantic similarity.

## introduction language modeling has advanced rapidly due to efficient model architectures @xcite and the availability of large-scale datasets @xcite . large-scale language models have been applied not only for representation extraction to support downstream tasks @xcite , but are also used for many natural language generation applications @xcite @xcite ; zhang ♦ denotes equal contribution. ♥ work done during an internship at deepmind. ♠ corresponding author: posenhuang@google.com. had a grand time organising...(0.97) hear from her all the time all the problems...(0.17) 're working on a prototype for her banana bread recipe...(0.51) ## counterfactual evaluation of sentiment bias fairness specification. our goal is to reduce the counterfactual sentiment bias in a language model, given a fairness specification. in our specification, we consider a set of sensitive attribute values (e.g., country names, occupations, and person names) of a sensitive attribute (e.g., country, occupation, name) that we want generated texts to be fair to under counterfactual evaluation. formally, considering for example the sensitive attribute gender, we use @xmath0, male} to denote the set of values considered, and use @xmath1. for each input sequence x containing sensitive tokens φ(a) (which are given in the specification, e.g., φ(a)={he, his, him, husband, paul} for @xmath2, we choose another value ã of the sensitive attribute from the set a \ {a}, and define the counterfactual input @xmath3, a, ã) by replacing all occurrences of each sensitive token in φ(a) with the corresponding token in φ(ã), and leaving all other non-sensitive tokens of x unchanged. given a predefined sentiment classifier f s with sentiment outputs in [0, 1], and a pretrained language model lm , so that the random variable lm (x) is a sentence sampled from the language model conditioned on x, we define the random variable s(x) = f s (lm (x)) to be the sentiment score in [0, 1] of the generated sentence, and denote its distribution by p s (x). next, for counterfactual evaluation, we measure the difference between p s (x) and p s ( x) as follows. when quantifying the difference between two output distributions for a binary classification problem -such as sentiment prediction -we typically consider predictions formulated as ŷ = 1(@xmath4, given a decision threshold τ . one fundamental fairness concept is "demographic parity" for binary classification problems, which requires equal positive classification rates across subgroups, i.e., p(ŷ = 1 | @xmath5, ã ∈ a. we can measure deviation from it, i.e. "demographic disparity" using the differences between the subgroup positive rates: @xcite ). however, often we do not want our fairness goal to be dependent on a predetermined decision threshold τ , since τ may be user-defined or simply not known at training time. this consideration leads us to match output distributions, which is called "strong demographic parity" @xcite . concretely applied in our lm context, these distributions are p s (x|@xmath6. extending this definition to measure unfairness between counterfactual pairs of subgroups, demographic disparity is the difference between positive sentiment rates of s(x) and s( x): |p(s(x) > τ )-p(s( x) > τ )|. we can then measure the deviation by computing the statistical disparity averaged over uniformly random choices of τ ∈ [0, 1], that is, e τ ∼u [0,1] |p(s(x) > τ ) -p(s( x) > τ )| where u denotes the random uniform distribution. this quantity is equal to the wasserstein-1 distance between p s (x) and p s ( x) @xcite : for each sensitive attribute value @xmath7, ã ∈ a \ {a}, and a chosen @xmath8. this fairness formulation also expresses individual fairness which requires similar individuals to be treated similarly @xcite , where similar individuals share similar non-sensitive words in a sentence. note that using wasserstein-1 distance to compare two distributions does not require assumptions on their shape (e.g., symmetry). fairness evaluation. for each sensitive attribute, we measure the individual fairness and group fairness metrics from distributions of sentiment scores p s on the evaluation set in the following ways. ## language models with fair sentiment distribution in this section, we introduce two approaches for reducing counterfactual sentiment bias in language models, which will be subsequently evaluated with the above described fairness metrics. given an input prefix x 1:i with i tokens, x 1:@xmath9, • • • , x i ), where the last token x @xmath10, we construct a perturbed prefix by replacing x i with a token @xmath11, where fairness between the two subgroups should be maintained. we obtain a perturbed prefix x1:@xmath12:i-1 , xi ). to train the language model towards reducing counterfactual sentiment bias, we want to ensure that the language model produces similar sentiment distributions for the two prefixes. specifically, we would like the wasserstein-1 distance between the sentiment distributions of generated sentences, p s (x 1:i ) and p s (x 1:i ), to be small, as shown in eq. 2. but in practice, it is prohibitively expensive to sample a distribution of generated sequences for every x 1:i and x1:i during training. instead, we use hidden features from the language model as a proxy to represent the distribution of future generated sequences, since p(x i+1 , x i+2 , • • • |x 1:i ) and p(x i+1 , x i+2 , • • • |x 1:i ) depend on the hidden states of the language model conditioned on x 1:i and x1:i , respectively. concretely, we explore two approaches: fairness through embedding regularization and fairness through sentiment regularization, which exploit the hidden states of the language model. given an l-layer transformer based language model with an input x 1:i , we let h(x 1:i ) = h (1) (x 1:i ), • • • , h (l) (x 1:i ) denote the hidden features (or contextual embeddings) obtained by its hidden layers. fairness through embedding regularization. in this approach, we desire that the embeddings h (j) (x 1:i ) and h (j) (x 1:i ) are close, since the joint distributions p(x i+1 , x i+2 , • • • |x 1:i ) and p(x i+1 , x i+2 , • • • |x 1:i ) are determined by these embeddings. we call it the "embedding regularization" approach, and define the fairness loss as a distance between the embeddings, denoted as d(h(x 1:i ), h(x 1:i )). we use the cosine distance: since the embedding regularization method enforces the model's predictions to be similar for the original input x 1:i and the perturbed input x1:i without specification measure information, a potential drawback of this method is that the regularization can be too strong. as we require the hidden representations (and thus the joint probabilities) to be as close as possible, this can lead to the model learning to ignore the sensitive tokens, and thus generally a reduced dependence on them, as shown in appendix c.6. despite being completely fair in this extreme case, model performance may suffer since the generated texts should ideally be contextually conditioned on x i or xi . ## experiments we now evaluate our proposed sentiment regularization and embedding regularization methods via both automatic scores and human evaluations. ## conclusion as large-scale language models are increasingly deployed for real-world applications, developing methods for assessing and mitigating bias with respect to sensitive attributes is an important area of inquiry to enable pro-social outcomes. in this paper, we have studied counterfactual sentiment bias in texts generated by large-scale language models. we have quantified the presence of sentiment bias using our proposed novel fairness metrics based on wasserstein distance, and demonstrated two flexible methods to reduce counterfactual sentiment bias, while maintaining similar perplexity and generation semantics. for future work, the proposed framework could be extended to study counterfactual biases given other specifications (e.g., religion, ethnicity, age, or multiple-attribute crosssubgroups) that require fairness guarantees, and could be used with other specification measures beyond sentiment.