Using Gender- and Polarity-Informed Models to Investigate Bias (2021)

Abstract: In this work we explore the effect of incorporating demographic metadata in a text classifier trained on top of a pre-trained transformer language model. More specifically, we add information about the gender of critics and book authors when classifying the polarity of book reviews, and the polarity of the reviews when classifying the genders of authors and critics. We use an existing data set of Norwegian book reviews with ratings by professional critics, which has also been augmented with gender information, and train a document-level sentiment classifier on top of a recently released Norwegian BERT-model. We show that gender-informed models obtain substantially higher accuracy, and that polarity-informed models obtain higher accuracy when classifying the genders of book authors. For this particular data set, we take this result as a confirmation of the gender bias in the underlying label distribution, but in other settings we believe a similar approach can be used for mitigating bias in the model.

## introduction as is well established, training data for nlp tasks may contain various types of bias that can be inherited by the models we train, and that may potentially lead to unintended and undesired effects when deployed @xcite . the bias can stem from the unlabeled texts used for pretraining of language models (lms), or from the language or the label distribution used for tuning a downstream classifier. typically, when a classifier is fitted on top of a pre-trained lm for a given task, only textual data is considered by the learned representations. in this work we investigate the effect of adding metadata information about demographic variables that are known to be associated with bias in the training data. specifically, we focus on the task of binary sentiment classification based on data where gender has previously been shown to be correlated with the label distribution. the data we use are norwegian book reviews, where the gender of both critics and book authors have previously been annotated @xcite . when considering all pairs of male/female critics/authors, @xcite showed that female critics tended to assign lower ratings to female authors, relative to other gender pairs. in this work we explore the effect of adding information about gender to a document-level polarity classifier trained on top of a pre-trained bert model for norwegian, showing that the model is able to take this metadata into account when making predictions. through experiments with gender classification on the same data set, we also demonstrate that the language of the reviews is itself indeed gendered. we believe that adding this type of metadata about e.g., demographic information when available can in many cases be used to mitigate bias in models. consider the case of a model for toxic language classification; it seems intuitively plausible that incorporating information about users could help reducing the risk of false positives for selfreferential mentions by marginalized groups. however, we have a different focus for the particular experiments reported here: we show how adding information about gender in a polarity classifier confirms gender bias, by showing how a genderinformed model obtains substantially higher accuracy when evaluated on a biased label distribution. in what follows, we start in section 3 with an overview of related work, after providing a brief bias statement in section 2. in section 4 we present our dataset, and give a detailed description of our experiments in section 5. we present and analyse our results in section 6, followed by an error analysis in section 7. finally, we summarize our findings and discuss future works in section 8. ## bias statement this work focuses on gender bias, which we identify as the differences in language use between persons, on the unique basis of their genders. the concrete task that we deal with in the current paper is that of polarity classification of book reviews, using labels derived from the numerical ratings assigned by professional critics. we use an existing dataset of book reviews dubbed norec gender @xcite , which is a subset of the norwegian review corpus @xcite , a dataset primarily used for document-level sentiment analysis. the subset norec gender has previously been augmented with information about the gender of both critics and book authors. through experiments with gender predictions of both critics and book authors, we demonstrate the presence of gendered language in these reviews. previous work has also shown that the distribution of ratings in the dataset to some degree is correlated with the gender of the critics and the authors. consequently, work on sentiment classification on the basis of the dataset could risk inheriting aspects of gender bias unknowingly, either in the model predictions themselves or in how these are evaluated, or both. one of our motivations in this work is exactly to assess whether the predictions of sentiment classifiers trained on review data may to some degree depend on gender, by explicitly incorporating this as a variable in the model. note that there are also issues of what could be argued to be representational harm @xcite associated with the underlying encoding of gender itself, since only the binary gender categories of male/female are present in the data. while the dataset we use only reflects binary gender categories, we acknowledge the fact that gender as an identity spans a wider spectrum than this. ## related work state-of-the-art results for various nlp tasks nowadays typically build on some pre-trained transformer language models like bert @xcite . despite their great achievements, these models have been shown to include various types of bias @xcite @xcite @xcite . recent works have shown the advantage of adding extra information to pre-trained language models for numerous tasks, e.g., dialog systems @xcite , natural language inference @xcite , and machine translation @xcite . knowledge graphs have also been used to enrich embedding information. @xcite use entries from wikidata, as well as their relation to each others, to represent and inject structural knowledge aggregates to a collection of large-scale corpora. they show that their approach reduces noisy data and improves bert fine-tuning on limited datasets. @xcite enrich a german bert model with linguistic knowledge represented as a lexicon as well as manually generated syntactic features. peinelt et al. (2020) enrich a bert with lda topics, and show that this combination improves performance of semantic similarity. @xcite use a combination of metadata about books to enrich a bertbased multi-class classification model. they train a bert model on the title and the texts of each book, and concatenate the output with metadata information and author embeddings from wikipedia, and feed them into a multilayer perceptron (mlp). when it comes to gender and gender bias, previous research has been devoted to the identification of bias in textual content and models @xcite @xcite , and in input representations as static and contextualised embeddings @xcite @xcite @xcite @xcite . a considerable amount of previous work has also gone into either mitigating existing bias in embeddings @xcite @xcite , making them gender neutral @xcite , or using debiased embeddings @xcite . instead of debiasing and mitigating bias in embeddings, some work has focused on creating gender balanced corpora @xcite . several previous studies have focused on gender and gender bias in sentiment analysis, both from data and model perspectives. to name a few: @xcite propose an evaluation corpus (equity evaluation corpus) that can be used to mitigate biases towards a selection of genders and races. occupational gender stereotypes exist in sentiment analysis models @xcite , both in training data and in pre-trained contextualized models. models have also been proposed to uncover gender biases @xcite . incorporating extra demographic information into sentiment classification models have also been successful. @xcite has shown that incorporation gender information (as embeddings) in models can improve sentiment classification. they show that such an approach can reduce the bias towards minorities, as for example females, who tend to communicate differently from the norm. in this paper, we do not focus on biases present in existing systems , nor do we try to mitigate them in a traditional way. we use a dataset of norwegian book reviews for which a previous study has indicated some degree of gender bias in the label distribution of review ratings @xcite . here, we investigate whether this bias is reflected in the text, as measured by classification scores on two tasks, namely binary sentiment and gender classification, and whether adding metadata information explicitly providing the gender of the authors and critics of the reviews, or the sentiment score of the review increases classification performance. similarly to @xcite , we explore the effects of adding this metadata information to document classification tasks using a bertbased model, in this case the norwegian norbert @xcite . ## dataset in this work, we focus on gender effects in reviews written by male or female critics, which in turn rates the works of male and female authors. the dataset we use is the norec gender 1 @xcite subset of the norwegian review corpus (norec @xcite ). norec gender is a corpus of 4,313 professional book reviews from several of the major norwegian news sources. each review is rated with a numerical score on a scale from 1 to 6 (represented by the number of dots on a die), assigned by a professional critic. the reviews also contain additional metadata information like the name of the critics, name of the book authors, and their respective genders. the numerical ratings and name of the critics were already provided in the metadata data of norec @xcite , while the name of the authors and the information about the genders were manually annotated with the release of 1 https://github.com/ltgoslo/norec_ gender m f total unique critics 125 74 199 unique authors 1,435 882 2,317 norec gender @xcite . as pointed out by @xcite , some of the reviews were written by children, unknown authors/critics, or by editors, these were not assigned genders and were therefore not included in our work. this results in a set of 4,083 documents. each review in norec gender comes with a numerical dice score from 1 to 6. similarly to @xcite , we choose to focus on clear positive and negative reviews and therefore only use reviews with negative ratings representing dice scores 1, 2, and 3, and reviews with positive ratings representing scores 5 and 6. however, in order to control for the distribution of positive and negative labels, we have selected a subset of reviews with rating 5 to have a balanced distribution of positive and negative reviews in the train set. this results in a subset of 683 negative and 708 positive reviews for norec gender . a distribution of these across the train, dev, and test splits can be seen in the dataset norec gender also contains a bias in the distribution of labels, based on the gender of the critics and the authors @xcite . figure 1 shows the total number of ratings in our dataset, where the first letter (m/f) indicates the gender of the critic and the second letter indicates that of the author. for example, mf represents reviews written by male critics reviewing the works of female authors. here we observe a clear difference in the ratings given by female critics to female authors (ff). while most reviews seem to have a certain amount of balance between positive and negative polarities with slightly more positive than negative reviews, for ff it is the opposite. this, in addition to the unbalance between the total number of reviews based on gender, represent the bias present in norec gender 's label distribution. ## experiments we use the norwegian bert model norbert foot_0 @xcite . the model uses the same architecture as bert base cased @xcite , and uses a 28,600 entry norwegian-specific sentence piece vocabulary. it was jointly trained on both official norwegian written forms bokmål and nynorsk, on 200m sentences (around 2 billion tokens) from wikipedia articles and news articles from the norwegian news corpus. 3 we use a similar architecture to @xcite as shown in figure 2 . we feed our review texts to a norbert architecture of 12 hidden layers consisting of 768 units each. these representations and the metadata are subsequently concatenated and passed to a two-layer multilayer perceptron (mlp), using relu as activation function. the output layer (softmax) gives for each task its binary output, i.e., either binary sentiment classification labels, or binary gender classification labels. we set the learning rate for adamw @xcite to 5e -5, and batch size to 32. we train the model for 5 epochs, and keep the best model on the dev set with regards to f 1 . we have experimented with various input sizes (first 300 tokens, first 512 tokens, and first 128 + last 383 tokens) both with tokenized and untokenized texts. the best results were achieved using untokenized texts, and using the first 128 and last 383 tokens, as pointed out by @xcite . these are the input sizes used in the models we report in this work. our metadata is one-hot encoded, and has a dimension of two for gender (female and male), and two for polarity (positive and negative). in the case where we combine information about the genders of both authors and critics, the dimension is four (i.e., two gender dimensions each). for the task of binary gender classification, we perform a set of four experiments: for each of the binary classification of genders of authors or critics, we perform the following two experiments: in all of our experiments, we use the task specific norbert-none as baselines. ## results we observe that the norbert-ga model, which incorporate information about the gender of the authors is the most accurate model on the test set, with an f 1 score of 84.21, while it is the third most accurate on the dev split with an f 1 score of 84.51. norbert-gc, which adds information about the gender of the critics, also yields better results than the baseline with an f 1 score of 84.92 on dev, and 82.33 on test. the best performing model on the dev set is norbert-gac, with added information about the genders of both authors and critics. this model is also the second best model on test with a f 1 score of 82.92. the scores of our gender classification tasks are presented in for the classification of the authors' genders, the baseline classifier norbert-none performs quite good with a f 1 score of 89.57 and 90.12 on dev and test respectively. however, adding the metadata about the polarity of the review (if it's positive or negative) influences the classification task by 5.36 and 4.48 points on dev and test respectively. interestingly, we observe the opposite situation for the classification of the gender of critics. here, the baseline model norbert-none outperforms the norbert-polarity model by 5.41 and 6.08 f 1 score points on respectively dev and test splits. for the task of author gender classification, knowing the polarity of the review clearly influences the classification. again, this indicates that gender and polarity are correlated in our data. the results also point to a difference between the gender of authors and critics. however, additional information about the polarity of the review, seems to hurt the classification of the genders of critics. ## error analysis in order to gain further insight into the differences between the models we are comparing and in particular, the classification differences caused by the addition of information on gender/polarity, we perform an error analysis by comparing, for each task, how our models perform compared to the taskspecific baselines. figure 3 shows how the three models norbertga, norbert-gc, and norbert-gac have different predictions than their baseline norbert-none for binary sentiment classification. we show the relative differences of true positives as a heatmap. these are made on the test predictions of each model over all five runs. positive numbers (dark purple) specify that the model made more correct predictions than the baseline norbert-none, while negative numbers (white) indicate it made fewer correct predictions. the abbreviations ff, fm, mf, and mm represent the gender of the critic reviewing the work of an author of a given gender. ff refers to female critic and female author, fm female critic and male author, mf male critic and female author, and mm for male author and male critic. it is clear that all three gender-informed models become more accurate in the classification of reviews written by female critics and reviewing the works of female authors (ff). as previously mentioned, and as pointed out by @xcite , female critics tend to be more negative towards female authors, and therefore there are few reviews that fall within this category with positive polarity. adding information about the gender of the authors and the critics, seems to help the model identify some of the ff reviews that norbertnone was not able to classify correctly. this information seems to be particularly important for norbert-ga, which was the best model on the test set achieving 12 f 1 points more than the baseline on ff. this model also seems slightly better at identifying reviews for mm. a closer analysis differentiating the positive and negative polarities also shows that the three models are more accurate precisely in identifying the positive reviews in the ff subset. the same applies to a lesser degree for fm. knowing the gender of the authors and the critics, separately, enables the models to correctly classify more reviews than norbert-none. in contrary, for mf, only knowing the gender of both the critics and authors seems to slightly improve classification. for the mm reviews, the norbert-ga model is better at identifying the positive reviews, while norbert-gac is better at identifying the negative reviews. figure 4 shows the breakdown of the relative differences of true positives. here again, the relative differences are made on the test predictions of each model over all five runs. positive numbers (dark blue) represent the cases where the model made more correct predictions than the baseline norbert-none, while negative numbers (white) indicates the opposite. for clarity, we add a prefix to each model in the figure to specify the task. ga-norbert-pn represent the model norbert-pn for the task of author gender classification, while gc-norbert-pn represents the task of critic gender