Measuring Presence of Women and Men as Information Sources in News (2022)

Abstract: In the news, statements from information sources are often quoted, made by individuals who interact in the news. Detecting those quotes and the gender of their sources is a key task when it comes to media analysis from a gender perspective. It is a challenging task: the structure of the quotes is variable, gender marks are not present in many languages, and quote authors are often omitted due to frequent use of coreferences. This paper proposes a strategy to measure the presence of women and men as information sources in news. We approach the problem of detecting sentences including quotes and the gender of the speaker as a joint task, by means of a supervised multiclass classifier of sentences. We have created the first datasets for Spanish and Basque by manually annotating quotes and the gender of the associated sources in news items. The results obtained show that BERT based approaches are significantly better than bag-of-words based classical ones, achieving accuracies close to 90%. We also analyse a bilingual learning strategy and generating additional training examples synthetically; both provide improvements up to 3.4% and 5.6%, respectively.

## introduction text mining in general, and natural language understanding (nlu) in particular, are being successfully used as support tools on various areas of the humanities. in many studies, evidence is encoded in natural language, either in text or audio collections, and nlu techniques help significantly in the task of finding such evidence. within the humanities, one of the fields that has gained momentum in recent years is gender studies, which has come to cover a wide range of topics. this paper focuses on gender studies aimed at analysing the presence of women in the narratives constructed by the press. the objective of this kind of research is to quantify the presence of women compared to men in the news according to various indicators. the global media monitoring project (gmmp) is a long running international project carrying out such studies. it has a consolidated methodology that evaluates a wide number of indicators @xcite , such as: all of those indicators are analysed manually, which implies a great effort that limits the frequency and the size of the sample on which the studies are carried out. it is therefore appropriate to research whether such indicators can be measured using artificial intelligence. this paper focuses on those that require language comprehension. specifically, we tackle the indicator dealing with the presence of women and men as information sources in the news, by means of nlu techniques. we approach the task using a binary gender identification schema, due to technical reasons. this decision should not be interpreted as a denial of a more complex reality. this paper presents an attempt to measure the presence of women as sources of information for news stories, in the line of work started by @xcite . we approach the problem by using a simple strategy whose core is a multiclass supervised classifier. the main task consists on identifying sentences including quotes and detecting the gender of the sources of those quotes. this is no trivial task. the structure of the quotes is variable, in some languages there is no gender marking, and, moreover coreferences are often used and thus the source of the quote is not explicit. example 1 [eu] sagarduik foot_0 ohartarazi du «kostatuko» dela kutsatze-tasa 60 puntura jaistea. a manual analysis of a sample of news items (see section 3.1) showed that it is not necessary to solve all cases of coreference in order to measure the presence of women and men as sources of information. this fact allows us to tackle the task with a relatively simple pipeline. the pipeline consists of two steps; first lexical substitutions involving surnames (see example 1) are resolved, and afterwards a multiclass classifier detects the sentences in the news that correspond to utterances as well as their corresponding gender. to address the multiclass classification task, different strategies based on fine-tuning neural language models have been compared to bag-of-word paradigm based approaches. the contributions of this work are the following: from here on, the paper is organized as follows: the next section reviews the state of the art. in section 3 we present an analysis about the measurement of the presence of women and men as information sources, how datasets were prepared and what criteria we followed in the annotation. section 4 presents the approaches analysed to tackle the task as well as the results obtained. finally, some conclusions are drawn. ## state of the art when developing policies to address social challenges, evidence-based diagnostics are necessary, and the digital press is a source of such evidence, or more specifically, the narratives of reality that they offer. this type of analysis based on nlp techniques has already been carried out for several social challenges. @xcite applies several text mining techniques such as collocations, word co-occurrences and topic-modeling (lda) for extracting information about immigrant workers in korea. @xcite study the narratives in the press about the refugees during the 2015 refugee crisis across eight european countries. @xcite focus on gender equality, using an approach based on counting n-gram frequencies and extracting named entities on a large british news corpus. @xcite present a comprehensive survey of the challenges fake news detection in social media pose, including nlp approaches. in the same field, @xcite analyse fake news narratives generated about the ukrainian conflict and twitter users' reactions to those stories. regarding gender related research, we can find numerous works that make use of nlp. @xcite apply sentiment analysis techniques to identify sexist attitudes in chinese social media. @xcite study gender stereotypes in male-oriented magazines and female-oriented magazines. they use topic modelling techniques to analyse the topics associated with each gender and their evolution over time. @xcite present a large scale analysis of the gender disparity in literature published in the pre-modern period by means of nerc and ned techniques. @xcite use similar techniques, namely the booknlp pipeline @xcite , to research the evolution of the meaning of gender in works of fiction published between the 18th and 21st century. @xcite deals with the task of modelling the propagation of information in literature by paying attention to gender dynamics and their relationship to the representation of men and women in novels. to do so, they use, among others, coreference resolution and speaker attribution techniques. other authors @xcite @xcite study the presence of gender on audio content in order to analyse the presence of men and women in audiovisual media. quote extraction and speaker attribution are tasks that have been addressed in the literature. three types of quotes are distinguished in the literature: direct quotes, indirect quotes, and mixed quotes. early approaches to citation extraction focused on direct quotes and made use of symbolic strategies @xcite . @xcite also focus on direct quotes and propose a more complex strategy combining a nerc process, linguistic rules based on syntactic information and a supervised classifier. @xcite presents the first large-scale experiments on direct quotes, indirect quotes, and mixed quotes extraction. they deal with the task with a supervised approach based on conditional random field (crf) models and maximum entropy classifiers. the work closest to ours is @xcite . as we have already mentioned, they deal with the measurement of women as source of information as well, but unlike ours, it is based on a symbolic approach that also requires a more complex pipeline than the one we propose. on the other hand, there have also been published works @xcite @xcite on quote extraction, which are somehow related to the measurement of this indicator. these approaches, supervised in some cases, also present pipelines with a complexity that exceeds what is necessary to address the task we propose. ## presence of women as source information in news the measurement of the indicator of presence of women and men as sources of information in the news can be approached as an aggregation of the measurement of that indicator on the sentences that make up a news article. therefore, the nlu tasks to be addressed would be to identify the sentences that correspond to quotes and then classify the gender of the authors of these quotes. we can define the whole challenge as follows: given a document @xmath0, ..s i , ..s n } detect the sentences {s i } that correspond to quotes and assign the corresponding gender to the source of the quote gen(s i ) = {m, f }. unfortunately, the presence of coreferences, as well as the lack of explicit gender information in some languages, makes this task very difficult to solve. however, since the ultimate goal is the measurement of a macro indicator, it may not be necessary to resolve all quotes. in order to clarify these two points, we present a study carried out on a manually annotated sample in section 3.1. on the one hand, we analyse the complexity of the task in spanish and basque, basque being a language without gender marks, and on the other hand, whether resolving all quotes is necessary to measure the presence of women and men as sources of information. once the analysis was done, we further annotated a large number of examples in order to construct the datasets for training and evaluating the supervised approaches proposed in this work. details about annotation guidelines and datasets preparation are given in section. 3.2. ## identification of quote and source's gender we propose to measure the indicator of presence of women as a source in news by dealing with the task at sentence level. once solved at the sentence level, we can aggregate sentence results to compute the indicator at news article level, and subsequently at the collection level. we have shown in section 3.1 that the task at the sentence level can be simplified and not all types of coreference need to be taken into account. it is enough to consider only those citations that include gender marks and/or surnametype coreferences. the proposed approach to address the task has two steps: (i) lexical substitutions (surnames) are resolved at the news item level, and (ii) sentences from the news item are processed by a multiclass classifier that determines whether the sentence contains a quote and the gender of the source of the quote. we approach the problem of identifying quotes and the gender of their sources as a single sentence classification task. each sentence of the news article is classified based on three categories: to implement the multiclass classifier, two approaches have been compared: a) bag-of-words representation and svm (support vector machine) and lr (logistic regression) classifiers, and b) dense fine-tuned representation approach using a pretrained bert neural models. to implement the first approach we used the vocabulary with minimum absolute document frequency of 4 and maximum relative document frequency of 0.6. we used the tfidf statistic as the weight in the vector representation. to implement the neural approach, we adopted the fine-tuning strategy proposed by @xcite , using various bert models and fine-tuning them over the datasets presented in sections 3.1 and 3.2. we have analysed the following bert models: all the fine-tuning experiments were carried out using an nvidia titan rtx3090 gpu card. initial learning rate was set to 3e-5 and the best model was chosen over the results obtained in the development set, after fine-tuning up to ten 10 epochs. we report the best result out of 5 random initializations. the transformers library @xcite was used. ## conclusion this work addresses the task of automatically measuring the presence of women and men as sources of information in the news. this is a standard indicator in media monitoring processes for gender balance. we have shown that the large-scale measurement of this indicator can be automated using nlu techniques. to the best of our knowledge, this is the first work proposing a supervised approach to tackle this problem. the experimentation has been validated on two languages with different characteristics, spanish and basque. according to the analysis of our datasets, in order to estimate the presence indicator at the collection level it is not necessary to solve all the cases of coreference associated with the quotes, which simplifies the pipeline required for the measurement of the indicator. experiments show that the tasks of citation detection and author gender classification can be tackled jointly by means of a supervised multiclass classi-fier based on neural language models. fine-tuning a pretrained neural model provides significantly better results than supervised approaches based on bag-of-words paradigm. the supervised approach based on neural language models can achieve better results if they are trained with examples from both languages and a multilingual pretrained model is used. further improvement can also be achieved by adding synthetic examples to the training set, generated for person names not included in the training.