Are Fairy Tales Fair? Analyzing Gender Bias in Temporal Narrative Event Chains of Children’s Fairy Tales (2023)

Abstract: Social biases and stereotypes are embedded in our culture in part through their presence in our stories, as evidenced by the rich history of humanities and social science literature analyzing such biases in children stories. Because these analyses are often conducted manually and at a small scale, such investigations can benefit from the use of more recent natural language processing (NLP) methods that examine social bias in models and data corpora. Our work joins this interdisciplinary effort and makes a unique contribution by taking into account the event narrative structures when analyzing the social bias of stories. We propose a computational pipeline that automatically extracts a story’s temporal narrative verb-based event chain for each of its characters as well as character attributes such as gender. We also present a verb-based event annotation scheme that can facilitate bias analysis by including categories such as those that align with traditional stereotypes. Through a case study analyzing gender bias in fairy tales, we demonstrate that our framework can reveal bias in not only the unigram verb-based events in which female and male characters participate but also in the temporal narrative order of such event participation.

## introduction social biases and stereotypes are embedded in our culture in part through their presence in our narratives @xcite . despite the focus on documenting and mitigating the social bias that arises from the pre-trained embeddings used in natural language processing (nlp) @xcite @xcite , these methods also lend themselves to analyzing the biases within existing texts @xcite . meanwhile, the humanities and social sciences have a rich history of analyzing social bias in texts such as literary works, news reports, and fairy tales @xcite . however, these analyses are often conducted manually and at a small scale. advances in natural language processing now allow for in-depth, large scale analyses of social biases within narrative texts. as storybooks, especially fairy tales, are particularly important to children's mental, emotional, and social development @xcite , we use fairy tales as our genre of analysis. in this paper, we analyze the gender bias in children's fairy tales by comparing the event chains of female versus male characters. bias within the field of nlp can take on many different meanings @xcite . we adopt blodgett et al.'s definition of social bias as representational harm through social group stereotypes. these groups can be based on social attributes such as gender, race, economic class, and so on. we focus on gender bias as it is a crucial axis of social bias and has extensive work in the nlp literature, including the comparison of word embedding directions @xcite and the analysis of the gender representation in literary characters @xcite . few studies have considered gender differences in terms of narrative events such as @xcite who demonstrated gender differences in celebrity wikipedia pages by extracting action event triggers. we build upon this work by considering not just event triggers, but chains of event triggers in temporal order. a narrative can be simplified into a sequence of events in which a character participates as an agent (the entity which carries out the event) or as a patient (the entity onto which the event is done) @xcite . by considering the sequence, or chain, of events of characters, we can analyze the story narrative in greater detail. to accomplish this task, we develop a data processing pipeline which automatically extracts the temporal narrative event chains of characters, the characters' gender, and the characters' thematic roles in the event. we group events into event types to simplify analysis and focus on categories of interest which follow historical gender stereotypes. in summary, our paper presents three main contributions : ## event type annotation scheme there has been substantial previous work in annotating and clustering verbs. booknlp @xcite clusters event entities into nine supersense categories such as body, communication, competition, emotion, and possession based on wordnet's lexicographer files (princeton @xcite . verbnet @xcite clusters events into many of the same categories but includes more fine-grained groups to cover a total of 101 types and 270 classes. however, the categories from these two sources are not immediately useful for our analysis as the categories tend to include both synonyms and antonyms. for example, the event "harm" is categorized in the subclass "amuse" in verbnet along with events such as "please", "comfort", "delight", and "encourage". given the subject of our analysis, there were also some important missing categories related to common male and female stereotypes such as a grouping of domestic tasks or actions common in battle. to address these limitations, we used a mix of automated and manual methods to annotate the event types. ## analysis methods our primary numerical measure of bias is the odds ratio as used in @xcite . while typically used in fields such as medicine, it can be easily adapted and interpreted in the context of narrative bias. for example, in a given story, the occurrence of the event "kill" has an odds ratio of four from male to female characters. this means that male characters are four times more likely than female characters to be involved in an event regarding killing. we apply a common correction, haldane-anscombe, to account for cases in which one group has no observed counts of the event @xcite . to estimate the significance of biases' odds ratios, we calculate 95% confidence intervals using 1,000 bootstrap samples. we randomly sample, with replacement, 1,000 sets of the 278 stories from the fairytaleqa corpus. odds ratios are calculated for each event type for each bootstrap sample. if the confidence interval of an event type does not contain 1.0, it suggests that the bias towards that particular gender is statistically significant. we are also interested in whether a character is the agent or patient of an event. a character is considered the agent (the entity doing or instigating the event) if the semantic role labeling model identified them as the subject of the verb event. likewise, a character is considered a patient (the entity onto which the event is done), if the semantic role labeling model identified them as a direct object of the verb event. comparing the event chains of characters is nontrivial. a diverse set of verbs can cover the same event or type of event. the fairytaleqa corpus contains 1,431 unique events, many of which only occur a few times. this scarcity is compounded when considering the chains in which an event occurs as well as whether the character was involved as the agent or patient. additionally, characters have event chains of different lengths which correlate with character importance to the story. the bias towards male characters appearing more often in fairy tales also means that male characters will tend to have longer event chains. to facilitate analysis, event chains were broken down into segments or normalized. we always calculate separate odds ratios for events in which characters were agents or patients. in order to ensure a sufficient sample size, we only considered analysis units (unigrams, bigrams, etc.) that occurred at least five times in the corpus. in summary, we perform three types of analysis: for an illustrative example of how an event chain is broken up into the above analysis units, please see figure 6 in appendix a.3. ## analysis results the fairytaleqa corpus contained 33,577 events involving male and female characters of which 69% were attributed to male characters and 31% to female characters. these events were categorized into 172 event types including a type 'other' for events that do not fit in any other class. we focused on the event types related to common gender stereotypes shown in figure 2 : proportion of significant gender bias by analysis unit. when not considering temporal order, 1 in 4 event types are gender biased. temporal differences are represented by the "bigrams" and "event chain section" groupings. when looking at event bigrams, 1 in 5 show statistically significant bias. when looking at the location of an event within a character's narrative arc, female characters have more biased events in the beginning of their arcs while the bias for male characters is fairly consistent throughout all three sections of their arcs. ## conclusion and future work our character event chain extraction pipeline and odds ratio analysis was able to demonstrate that there are significant differences in not just the events that male and female fairy tale characters participate in, but also gendered differences in the temporal narrative order of such participation. in total, one-fourth of all event types showed significant gender bias no matter the temporal order, one-fifth when considering temporal order of bigram events, and one-fourth when dividing event chains into three equal parts (figure 2 ). this method of analysis offers a more nuanced look at differences in narrative text beyond simply counting the number or appearances of characters by gender or the rate of certain events. the method is supplemented by a more refined event-type annotation schema that separates antonyms and creates new classes that align with traditional gender stereotypes. there is ample room to build upon this analysis with a few distinct possibilities planned for future work. for example, there are numerous alternatives to compare event chains such as expanding the n-gram window or focusing on primary versus secondary characters. the method can be used to compare biases within and across cultural groups and genre. the social biases examined can also be extended by including other social group attributes in the extraction of character attributes such as race and ethnicity, age, and economic class. the results of this work further emphasize the urgency that future and "kill") show statistically significant differences towards common gender stereotypes. other event types (such as "help" and "obstinate-authority") show statistically significant differences against such stereotypes. this suggests that gender bias exists in how a character's narrative arc is structured and not just what occurs in such an arc. children-oriented nlp applications such as storybuddy @xcite should pay extra caution to the potential social biases and stereotypes issues embedded in the data and machine learning models.