“Hold on honey, men at work”: A semi-supervised approach to detecting sexism in sitcoms (2021)

Abstract: Television shows play an important role inpropagating societal norms. Owing to the popularity of the situational comedy (sitcom) genre, it contributes significantly to the over-all development of society. In an effort to analyze the content of television shows belong-ing to this genre, we present a dataset of dialogue turns from popular sitcoms annotated for the presence of sexist remarks. We train a text classification model to detect sexism using domain adaptive learning. We apply the model to our dataset to analyze the evolution of sexist content over the years. We propose a domain-specific semi-supervised architecture for the aforementioned detection of sexism. Through extensive experiments, we show that our model often yields better classification performance over generic deep learn-ing based sentence classification that does not employ domain-specific training. We find that while sexism decreases over time on average,the proportion of sexist dialogue for the most sexist sitcom actually increases. A quantitative analysis along with a detailed error analysis presents the case for our proposed methodology

## introduction apart from being one of the most popular genres on television foot_0 , sitcoms also attract the adolescent viewership foot_1 and thus play a vital role in the formation of their thought process @xcite . @xcite argue that documenting the prevalence and quality of television representations of women is a valuable endeavor as television depictions of women is known to influence attitudes and beliefs towards gender. therefore, these shows would ideally contain a minimal amount of sexist content. however, according to @xcite and @xcite , this may not be the case. for this reason, we present a dataset consisting of dialogue turns labeled as either 'sexist' or 'neutral'. we also build a system that automatically detects instances of sexism present in the dialogue of popular sitcoms. thus, we attempt to use machine learning to document the gap between activism and social change. often, a lack of labeled data can present a considerable challenge for text classification systems. manual annotation often requires domain knowledge and may be expensive and time-consuming for large datasets. manual annotation also carries the risk of introducing new annotator biases, privacybreaches, discrimination, and misunderstanding @xcite . although dialogue is not the only way that sexism is constructed in tv shows @xcite , the more subtle signs of discrimination can be more difficult to detect and analyze. our work addresses issues of manual annotation by using semi-supervised learning to generate a dataset in a new domain of pseudo-labels from unlabelled data to detect sexism in tv dialogue. this minimizes the need for a manual annotation process while creating large datasets. we make use of a previously published dataset @xcite to create a semisupervised domain adapted classifier. in general, domain adaptation uses labeled data in one or more source domains to solve new tasks in a target domain. it is a sub-category of transfer learning. since there is a lack of television show scripts annotated for sexism, we attempt a semi-supervised approach to develop our dataset. here, our source domain consists of tweets from waseem and hovy's (2016)'s 'hate speech twitter annotations' dataset and our target domain is the dialogue in popular sitcoms. these two domains are quite different. tweets are usually short, full of abbreviations, urban slang and grammatical errors. on the other hand, sitcom dialogue turns are descriptive, long, grammatically correct and contextually dependent on the dialogue turns that precede them. these differences warrant the need for a semi-supervised approach in our methodology. ## related work in the growing body of literature on the automatic detection of sexism in text on social media, twitter, in particular, has been the object of study and dataset creation. waseem and hovy (2016) created a dataset containing racist and sexist tweets. following this, there have been various efforts towards detecting sexism in english tweets @xcite , @xcite . @xcite . recently, @xcite developed a dataset for sexism detection in french tweets. while the study of sexism in tv shows has received little attention in natural language processing @xcite , @xcite , @xcite , it has received significant attention in the field of gender studies @xcite . in gender studies, @xcite conducted a quantitative analysis to document portrayals of women and men on prime-time television and @xcite examines the perception of gender roles on network prime-time television programming. to the best of our knowledge, no previous work has presented a comprehensive dataset for the presence of sexism in tv shows has been created. while efforts have been made to analyse the presence of sexism in tv shows @xcite , the question of developing a machine learning based detection system for identifying sexism in scripted tv dialogue remains under-explored. however, semi-supervised learning has received a lot of attention from the nlp community @xcite @xcite . our method most closely resembles unsupervised data augmentation @xcite , which uses labeled data to annotate unlabeled samples under low resource settings. ## experiment setup we begin by training a set of models on d train (section 3.1) to find the best performing model. we make use of a support vector machine (svm), a logistic regression classifier (lr), a random forest ensemble (rf), a naive bayes classifier (nb) , fine-tuned bert, and a bi-directional lstm (bi-lstm). we find that the bi-lstm outperforms the other models by 3.4%, with an accuracy of 76.03% on the held-out test set, d test . thus, we make use of the bi-lstm in our proposed semi-supervised approach. out of the 20 sitcom show scripts we collect, we use four, namely 'friends', 'the big bang theory', 'how i met your mother' and 'seinfeld' for manual annotation (see section 3.1 for more detail). next, we use the baseline bi-lstm to make predictions on the other 16 show scripts. out of these, eight are new shows and the other eight are old shows. the model classifies 1, 639 dialogue turns as sexist. to form d semisupervised , we add all dialogue turns identified as sexist by the baseline model and randomly sample 31, 944 dialogue turns from the 242, 108 dialogue turns identified as neutral. we combine d train and d semisupervised to form d f inal foot_5 . finally, we train a bi-lstm on d f inal . we make use of the softmax activation function and the categorical cross entropy loss function while training this bi-lstm. it consists of an embedding layer, a spatial dropout layer and makes use of the adam optimizer, with a dropout equal to 0.2. this bi-lstm attains an accuracy of 83.0% on d test . to offer a fair comparison, we also train other competitive models on d f inal . ## conclusion we generate a labeled, real-world dataset and build a classifier using a combination of transfer learning and semi-supervised learning to classify dialogues in sitcoms as sexist or neutral for the purpose of tracking the status of social discrimination. an analysis of the recent content reveals an overall decrease in sexist content over time but an increase in the amount of sexist content in the worst offending tv shows in the recent years.