A Diachronic Analysis of Human and Model Predictions on Audience Gender in How-to Guides (2025)

Abstract: We examine audience-specific how-to guides on wikiHow, in English, diachronically by comparing predictions from fine-tuned language models and human judgments. Using both early and revised versions, we quantitatively and qualitatively study how gender-specific features are identified over time. While language model performance remains relatively stable in terms of macro F _1 -scores, we observe an increased reliance on stereotypical tokens. Notably, both models and human raters tend to overpredict women as an audience, raising questions about bias in the evaluation of educational systems and resources.

## introduction marginalization and discrimination are central topics of recent advances in computational research on educational resources, like school textbooks, which contribute to shaping the sociocultural knowledge of learners @xcite . as an example, @xcite study sexism in textbooks, reporting gender bias on various dimensions across the over 30 countries examined. besides textbooks, also children's stories as sources of educational data drew the attention of the recent advances. @xcite focus on gender roles: their research shows women to be, despite progress in terms of representation, still subjected to different treatments. later work on children's books accounts for intersectional perspectives as well, comprising in the analyses not only texts but also images @xcite . another form of learning material are the collaboratively edited how-to guides from the online platform wikihow 1 . with this paper, we present a diachronic analysis of texts from wikihow guides that explicitly indicate a target audience based on gender in the guide's title (like in the example of in sociology, the concept of audience is not new. erving goffman's theory about the presentation of self designs the elements of interaction (offline) and conceptualizes the audience as part of the (social) performance @xcite . the audience is featured as the entity according to whom the performers act. however, different audiences might lead different performers to act differently. in the case of instructions, the performers are the writers, who might design and revise their texts with having in mind a specific target audience group. as instructional texts are resources meant to guide people in conducting activities, it becomes crucial to assess the variations according to the different intended audiences. the risk of leveraging stereotypes concerning the addressed audience groups might eventually lead to, for example, unfair treatments (cf. @xcite . sociolinguistic research also focuses on audience adaptation. this can be traced back at least to the 80s, with early studies on radio speakers @xcite , but previous research accounting for phenomena related to different audiences in instructional texts is just recent. for example, fanton et al. (2023) inspect audience-specific guides from wikihow qualitatively and quantitatively, revealing superficial differences in writing as well as gender-specific standards. this work extends previous work by including a diachronic perspective as well as by comparing model predictions with human judgments, following two main research questions: rq1. have the patterns learned by the training data changed over time in the task of distinguishing the gendered audience-specific instructional texts? rq2. how do model and human predictions in the task of distinguishing the (gendered) audiences of instructional texts differ over time? answers contribute to both the computational social science and the natural language processing sub-communities with a focus on genderrelated topics. furthermore, researchers in psychology or marketing domains, especially on the (perceived-)personalization of advertisement @xcite could benefit from our research as well. by detailing the gender biases that exist in audience-specific texts and investigating how people and language models use such biases, our work informs efforts to debias instructional text generation and system evaluation. ## related work in this section, we review gender bias in nlp ( §2.1) and connect the present work to previous literature concerning audiences ( §2.2). ## data this section presents the data employed for the two studies we conduct. we base our studies on wikihowaudiences (wha-en) @xcite , comprising guides tailored for specific audience groups, over gender and age. by opting for this data, we build upon previous findings concerning audience-specific instructional texts. we focus on the gender dimension only. briefly, each guide comprises title and howto instructional text. however, pursuing diachronic analyses require further data points, which wha-en does not offer. we detail how we proceed in the following. ## rq1: influence of training data over time in this section, we inspect the effects of using different data in finetuning lms. we address the following research question (rq1): have the patterns learned by the training data changed over time in the task of distinguishing the gendered audience-specific instructional texts? ## rq2: model vs. human predictions in this section, we study in how far the lms vs. the participants to the human-subjects experiment correctly predict the gender of the audience groups of the instructional texts. we address the following research question (rq2): how do model and human predictions 1. bert finetuned with early @xmath0. bert finetuned with revised @xmath1. roberta finetuned with early data 4. roberta finetuned with revised data 5. deberta finetuned with early data 6. deberta finetuned with revised data ## error analysis in a qualitative analysis, we found two sets of interesting errors: "hard cases", namely errors by both lms and human participants; and human errors, made by human participants only ( among the human errors, one instance may have been misclassified due to the emotional term 'cry', which could have biased perception toward women. in two cases, the phrase "(you want to be a) woman" may have led annotators to incorrectly infer the intended audience as women, despite the guide being directed at men. another misprediction may stem from ambiguous or misleading use of pronouns, further complicating accurate audience prediction. ## conclusion we studied audience-specific how-to guides from a diachronic perspective and compared human judgments and predictions by fine-tuned language models. our findings indicate that language models over time increasingly rely on stereotypical tokens, with earlier models additionally being heavily biased towards predicting women as a target audience. in our second study, we found that such bias decreases for more highly edited texts (from 72% to 58%) but that even humans are biased in their judgments, favoring women as a target audience in 59%. our error analysis revealed that stereotypical beliefs, such that only women would want to be (like) woman, could be a potential source for misjudgments. audience-unspecific, as in instructional text tailored for a general audience, can inform future analyses, especially with regard to the human ratings and their gender leaning. more data, also beyond the instructional domain, is necessary for broader generalizations. we encourage future work concerning this, and we point to movies' transcripts as well as to advertisement texts as starting points for further research. for more robust gender bias analyses, systematic approaches are needed. a more comprehensive understanding of texts that are written for specific audience groups will definitely benefit from this kind of assessments over different data source. our findings underscore the complexity of evaluating educational material and call attention to underlying challenges. future work should also explore broader demographic attributes and develop methodologies for mitigating representational biases in educational content.