Gender and sentiment, critics and authors: a dataset of N orwegian book reviews (2020)

Abstract: Gender bias in models and datasets is widely studied in NLP. The focus has usually been on analysing how females and males express themselves, or how females and males are described. However, a less studied aspect is the combination of these two perspectives, how female and male describe the same or opposite gender. In this paper, we present a new gender annotated sentiment dataset of critics reviewing the works of female and male authors. We investigate if this newly annotated dataset contains differences in how the works of male and female authors are critiqued, in particular in terms of positive and negative sentiment. We also explore the differences in how this is done by male and female critics. We show that there are differences in how critics assess the works of authors of the same or opposite gender. For example, male critics rate crime novels written by females, and romantic and sentimental works written by males, more negatively.

## introduction gender is a widely studied source of bias in textual content @xcite @xcite . there has been considerable previous work analyzing gender bias in nlp models and in particular, in input representations such as static and contextualized word embeddings @xcite @xcite @xcite . gender-annotated datasets largely focus on the gender of the author of a specific piece of text, such as a blog @xcite or a tweet @xcite and has given rise to considerable research focused on author gender identification @xcite . datasets which enable the study of response to gender in text, however, are considerably fewer @xcite . with a few noteworthy exceptions @xcite , a majority of previous work has focused on gender modeling and the study of gender bias in english. social psychological research on gender bias in language has shown that there are sociocultural stereotypes inherent in the language used to describe females and males @xcite . while the descriptions of females tend to focus on their communal traits, males are described for their agentic traits @xcite . @xcite show that the gender of the writer can also influence how females and males are described. they show that gender stereotypes can discriminate female applicants in an academic setting, due to their recommendation letters which tend to contain more communal-related words, in contrary to letters written for males which focus more on their agentic abilities. also, males in their recommendation letters, tend to describe the agentic traits of females more often than females do @xcite . this makes explicit the need to investigate the gender of both sides: the writer, and the person being written about. this paper introduces a dataset of norwegian book reviews with information about the gender of both the (professional) critic and the book author. each review comes with a rating on a scale of 1-6, which can be used as a supervision signal for overall positive/negative sentiment of the text. as a part of describing the provided dataset, we include an exploratory analysis of the data through a series of empirical experiments on gender-and sentiment classification. the combination of gender information on two sides in addition to ratings allows for investigating several interesting research questions. mainly, we here seek to address the following two closely related questions, mostly differing with respect to perspective: as a simplifying assumption, we only consider gender as a binary category (male and female) in this work. we acknowledge the fact that gender as an identity spans a wider spectrum than this, but this simplification was here deemed necessary to enable our annotation of the reviews. it is worth noting that during our manual annotation we did not come across any mention of known (to us) non-binary or transgender authors or critics. however, it is quite possible that some of the gender labels assigned would have been different had we been able to rely on self-identification. bias statement: this work mainly attempts to shed light on whether there are latent biases inherent in the data directly, focusing on book reviews. as part of this, we investigate whether polarities associated with certain words to some degree are correlated with the gender of either the critic or the author (or both, in combination). one of our motivations is to assess whether the predictions of sentiment classifiers trained on review data -as is commonly the case -may to some degree depend on the gender of either the critics, the creator of the work being reviewed (the author), or both. by extension, and in terms of possible harms, the dataset we present here suffers from representational harms @xcite . the book reviews present in the norec corpus, which are written by professional norwegian critics, seem to contain gender stereotypes when describing the works of female and male authors. the societal lexical asymmetries in how females and males are portrayed is present in the language use. for example, words related to emotions and feelings are used negatively to describe the works of female authors, but positively when the works described are written by males. also, words related to achievements with regards to literary genre or the process of publishing in general are positively used when describing the work of males, and negatively for the works of females. these observations seem to maintain the existing social hierarchies that tend to focus on emotional traits when describing females, while focusing on competence traits when describing males @xcite . ## related work much of the previous work on bias in ml models within nlp has focused on identifying biases in word embeddings and how to mitigate them @xcite @xcite @xcite , or even make them gender neutral @xcite . however, such efforts have received criticism by @xcite who argue that the biases have not been removed, but only "hidden" and kept at a deeper level in the embedding space. bias has also been investigated in several other settings, like multilingual embeddings @xcite , deep contextual representations @xcite , language models @xcite , coreference resolution (cao and @xcite @xcite , and machine translation @xcite , just to name a few of the more recent efforts. another line of work has focused on investigating gender representations in corpora and models, and release gender-neutral corpora (corpora in which either the distribution of genders is balanced, or where gender stereotypes and gendered words are removed). @xcite use film scripts to analyse the linguistic and structure variations in dialogues and how these differ based on gender. @xcite investigate gender biases and discrimination in blogposts. they use a metric to compute the salience of word classes combined with semantic and psycholinguistic resources to identify dominant word classes. these latter are used to uncover the underlying differences in the choice of word classes and concept usage between man and women. they show that the gender of a blog author can be identified using gender-based word disambiguation techniques, and that changes in word frequencies and contexts contribute to the differences between genders. costa-jussà et al. (2020) present a tool gebiotoolkit that automatically extracts multilingual parallel sentences using wikipedia biographies from several languages, which also relies on gender information to create a gender-balanced corpus. they also introduce the multilingual, parallel and gender-balanced corpus gebiocorpus, a corpus for machine translation applications covering the three languages catalan, english, and spanish. several studies have also focused on gender and gender bias in sentiment analysis. @xcite present the equity evaluation corpus (eec) that contains a set of manually crafted english sentences, with the sole purpose to mitigate biases towards certain races and genders. they also show that using the eec corpus helped uncover the existing biases in over two hundred sentiment analysis systems, which seemed to give higher sentiment predictions for sentences associated with one given race or gender. @xcite use a generative latent-variable model to represent collocations of positive and negative adjective and verb choices, given a gendered head noun. their analyses goes beyond qualitative analysis, and shed light on the differences on how men and women are described differently. they use a corpus of books spanning various genres, and show for example that positive adjectives used to describe women are related to their bodies more often than is the case for men. @xcite analyse the existence of occupational gender stereotypes in sentiment analysis models. they show that all their tested models (bow+logistic regression, bilstm, bert @xcite ) contain occupational gender stereotypes to some extent. they also show that simple models seem to show biases in training data, while contextual models might reflect biases introduced while pretraining. @xcite present an annotated corpus for the gender of the addressee and the sentiment and relevance of comments. the corpus comprised comments from responses to facebook and reddit comments, ted talks, and posts on fitocracy. this work has similarities to ours, since they look at the responses to gender e.g. how the content can differ based on the gender of the person being addressed. however, in our work we also add the dimension of the gender of the critic, and how it can positively or negatively affect the description of the book authors' gender. in this paper, we do not focus on the differences in gender representations and biases present in existing systems, nor do we try to mitigate them. we rather investigate the differences in gender descriptions in norwegian book reviews, and if this affects the ratings of the reviews. to this end we introduce a new dataset of rated reviews with meta-information about the gender of both critics and authors of the work under review. we focus on how positive and negative words can be informative for a simple machine learning model, and how these differ between genders. ## gender-coded book reviews the underlying source data in this study is the norwegian review corpus (norec) comprising professional reviews across a wide variety of domains, collected from several of the major norwegian news sources @xcite . each review is rated with a numerical dice score on a scale from 1 to 6. in the current work, we only deal with the subset of 4,313 book reviews, for which we have extended the meta-information for each review to include manually coded information about gender -both of the critics and book authors. in what follows, we give an overview of our manual and semi-automatic annotation efforts, and provide the resulting corpus statistics. ## gender classification regardless of sentiment, our first experiments were based on the language use in gender classification. we used our corpus for binary gender classification using logistic regression and cross validation, from both the authors' and the critics' perspectives. we opted for this simple classifier because it allows us to easily access the most informative features (in our case words) that guide the classification during training. we manually analysed the top 200 most informative words for each gender, and we were able to see that there were differences in the use of language in relation to gender, regardless of sentiment. however, in an effort to see if there are indeed differences in the language with regard to sentiment, we tested our gender classifiers on two different subsets: (i) a positive test set containing reviews with ratings 5 and 6, (ii) a negative test set comprising reviews with ratings 1, 2, and 3. our experiments as presented in ## sentiment classification based on the observations presented in section 4, our main interest in the current work is therefore to investigate if there are any differences in how the literary works of female and male authors are described in positive or negative reviews by female and male critics. in particular, we want to investigate whether the gender of the author or the critic affects the language use of the review and its rating. in order to do so, we train a number of models on differing critic-author combinations to predict the rating of a review, e.g. male critics reviewing female authors, male critics reviewing male authors, etc. we then go on to analyze the most informative words of these models using clustering over their word embeddings. for transparency and ease of interpretability, we first make use of machine learning models based on traditional approaches (i.e. with discrete and count-based features) that allows for straightforward extraction of the most informative features. we thereafter use word embeddings to cluster these features and identify representations of the content. to this end, we focus solely on the use of content words, i.e. adjectives, nouns, and verbs. these are already available in norec which is annotated with pos tags using udpipe @xcite . to train our models, we create two subsets of our gender-annotated dataset: (i) a subset containing reviews reviewing female authors (r f ), and (ii) a subset of reviews reviewing male authors (r m ). instead of looking at the full range of ratings as given in norec, we focus on the lowest and highest values of the rating range. the reason for this is that we want to analyze cases of clear positive or negative sentiments. we select all reviews with rating 1, 2, 3, and 6, and randomly select reviews with rating 5 to balance the distribution between the lower and higher ranges. ratings 1, 2, and 3 represent negative reviews, while 5 and 6 are positive. these categories are consecutively used for binary sentiment classification using logistic regression and cross validation. in order to obtain a richer picture of the important features for classification, and how these differ between genders, we investigate the results of two different strategies for training using our genderannotated data: authors: we combine the train and dev splits within each of r f and r m for cross validation. also, as previously mentioned, we balance the data within the splits such that the positive and negative classes are equally distributed. we thereafter analyse the results of four testing strategies: (1) train on r f train+dev, test on r f test, (2) train on r f train+dev, test on r m test, (3) train on r m train+dev, test on r m test, and (4) train on r m train+dev, test on r f test. critics: we follow the same steps as above, but run different models based on the gender of the critic. we add an additional dimension to the previous analysis by comparing the author and critic aspects. more concretely, we analyse results of four training combinations: (1) r f f : female critics, female authors, (2) r m f : male critics, female authors, (3) r f m : female critics, male authors, and (4) r m m : male critics, male authors. for each of these strategies, we have manually analysed the 200 most informative words, and looked at the overlap between them. we provide additional details in section 5.1 and section 5.2. ## non-professional reviews -the bokelskere corpus our gender-annotated literature subset of the norec corpus reflects how works by female and male authors are positively and negatively described. however, since these reviews are written by professionals, the language can be expected to be of a more formal and possibly less affective style. in order to investigate to what extent this proves to be correct, we carried out the same analysis done on norec on a corpus of non-professional reviews. we use a corpus comprising user-generated book reviews from bokelskere.no compiled by the national library of norway (språkbanken) foot_5 . we will refer to this corpus as the bokelskere (book lovers) corpus in what follows. the bokelskere corpus contains the raw texts from discussions and book reviews written by users of the bokelskere.no web community. the ratings follow the same scheme as in norec, with numerical scores ranging from 1 to 6. the reviews are structured as both reviews and comments on reviews. the corpus is in json format and contains a total of 219,000 review comments. for each of these, the corpus provides (amongst others) information about the book being reviewed (title and author), and the rating. we annotated the bokelskere corpus with pos tags using the same version of udpipe that was used to annotate norec @xcite . moreover, neither the gender of the users (i.e. critics), nor the gender of the book authors are provided in the bokelskere corpus. we therefore used our annotations from the norec corpus to automatically annotate bokelskere. we only annotate the authors from our gender-annotated corpus for whom we know the gender. we were able to identify the gender of 9,833 female authors, and 15,544 male authors. however, 1,691 and 2,815 reviews of female and male authors respectively did not contain ratings and were therefore disregarded. figure 4 gives an overview of the rating distributions of the remaining 8,142 female and 12,729 male works and negatively describe male works, while the words datter 'daughter' and søster 'sister' are negative in female description but positive for male descriptions. nouns related to crimes seems also to be positive for male, while negative for females (e.g. gjerningsmann 'perpetrator'). most verbs used to positively describe females' while negatively describe males' works are development (sammenligne 'compare', presentere 'present'), and evaluation (forestille 'imagine', oppfatte 'perceive'). conversely, verbs used negatively when describing works of females but positively for males seem to be related to completion (ende 'end', gjennomføre 'conduct'), development (lage 'make'), evaluation (forstå 'understand'), life-cycle and relationships (føde 'give birth', gifte 'marry'), and passion (forelske 'fall in love'). ## conclusion and limitations we present a gender-annotated dataset of professional book reviews, where both the gender of critics and the book authors are annotated. we also present a corpus of user reviews annotated for the gender of the book authors. we make all annotations and reviews publicly available. we have shown that there are differences in how female and male book authors are positively or negatively described, and that the gender of the critics influences the differences. for example, male critics deem female crime novels and male romantic and sentimental books as negative. this shows that book reviews contain the social hierarchies that tend to focus on emotional traits to describe females as in @xcite . there are several ways in which the preliminary analysis of the current work can be improved and extended. first, the annotations of the book authors are based on which book is being reviewed, and not if the author is being mentioned in the review. this can lead to issues during classification, since it might be possible that the review in itself contains references to the characters of the book, which might or might not be of the same gender as the author. therefore, the word usage might not actually reflect the book author, but rather the fictional characters of the book. secondly, we were able to identify differences in how female and male critics describe the works of female and male authors, but we did not quantify to which degree this is true. the distribution of ratings gives an indication of this, but more extensive analysis is necessary. in future works, we aim to explore how to quantify the amount of bias, but also identify if a review is discussing the quality of the book (as in the work of the author), or if it only focuses on the characters and the storyline.