On Shortcuts and Biases: How Finetuned Language Models Distinguish Audience-Specific Instructions in I talian and E nglish (2024)

Abstract: Instructional texts for different audience groups can help to address specific needs, but at the same time run the risk of perpetrating biases. In this paper, we extend previous findings on disparate social norms and subtle stereotypes in wikiHow in two directions: We explore the use of fine-tuned language models to determine how audience-specific instructional texts can be distinguished and we transfer the methodology to another language, Italian, to identify cross-linguistic patterns. We find that language models mostly rely on group terms, gender markings, and attributes reinforcing stereotypes.

## introduction instructional texts aim to convey the necessary knowledge for readers to accomplish specific tasks. on the collaboratively edited online platform wiki-how, hundreds of thousands of instructional texts are available on a variety of topics and in multiple languages. the goal, or mission, of this vast repository is to democratize access to knowledge and skills across diverse subject matters. 1 among other works on wikihow, prior research has explored in how far texts are formulated in linguistically inclusive terms and which adjustments are made for specific target audiences @xcite . however, these previous studies primarily relied on simple classifiers and focused exclusively on english texts, leaving a gap in understanding multilingual phenomena and if fine-tuning language models might exacerbate biases (see §2). acknowledging the limitations of prior research to english, we first compile a new dataset in a less resourced language, specifically italian (see §3). our initial research question investigates how texts for different target audiences in english and italian vary in terms of the topics they address (see §4). this exploration directly contributes to the analysis of social biases in the data (see in short, we make the following contributions: ## related work in this section we briefly review three related areas: our work continues a series of recent contributions dealing with the collection of data sets for italian. while there exists little work on instructional texts for italian, data in english has been examined and tested from different angles and perspectives in the nlp community. finally, work on model-based data interpretation has received increasing attention, but almost no work studied biases in audiencespecific instructional texts. italian nlp datasets. recent data collections for the italian language include diatopit (ramponi and casula, 2023a), a dataset representative in time and space on variations of non-standard italian. a new shared task for geo-locating the linguistic variation in italy @xcite ) is based on this data collection. another recent effort for the italian language is irma, a data collection for studying misinformation @xcite . in their paper, the authors curated a dataset from untrustworthy websites, and emphasized its significance for the less-represented language studied. @xcite advance the task of responsibility perspective transfer, in the context of studying gender-based violence, and a dataset of sentences for italian news about femicides. to the best of our knowledge, there are no previous studies on how-to guides in the italian language. instructional texts. @xcite introduced wikihowtoimprove, a data collection of original and revised sentences based on wikihow articles and their revision histories. @xcite contribute with a continual approach for instruction generation. @xcite examine audience-specific wikihow guides in english. they find traces of subtle biases, using shallow classifiers and qualitative analyses. in this work, we extend their findings to fine-tuned language models in two languages. ## data we first build a data collection to investigate our first research question, namely how texts for different target audiences in italian vary in terms of the topics they address. as a starting point, we use how-to guides from publicly available wikihow dumps foot_1 for italian. out of 34,801 guides, 1,031 feature an indicator between round parentheses at the end of the title (see ## data analysis we address our first research question, namely how texts for different target audiences in italian vary in terms of the topics, by clustering articles according to their content. we describe the approach in §4.1 and findings in §4.2. for this part of our work, we ## experimental setup given the data and analyses of the previous sections, we next investigate what features and biases computational models learn when they are trained to distinguish articles for different audiences. ## results we first discuss results in terms of model performance for the classification task itself ( §6.1) and then analyze the attributions of the models that perform best at distinguishing audiences ( §6.2). ## conclusion we introduced wikihowaudit, a dataset of instructional texts from wikihow for different audiences in italian. our data analysis has shown that wikihowaudit contains different topics across audiences, which makes computational modeling difficult. in order to still learn what biases can be found in texts for different audiences, we fine-tuned language models and investigated which attributes rank highest for each target group. as a result, we found that models perform very well even with training on only 100 data points and that they capture more fine-grained differences in english than simpler models from previous work. however, our analysis of the attributes also confirmed trends already observed with simpler methods: regardless of language, models consistently learn that texts for different audiences can be distinguished with high effectiveness based on group terms, grammatical gender, negations and stereotype reinforcing references. several of these points may represent critical issues, particularly given that wikihow is one of the most visited websites on the internet. 8 our results further support existing findings on gender roles in other domains, such as in stories for children and educational resources for young age groups, where females are also associated with gender stereotypes @xcite . one reason for us to analyze texts regarding biases is that we want to understand assumptions structurally made about the readers and to what extent these potentially reflect actual characteristics. future work should accordingly focus on how to identify and remove those biases that are inadequate (e.g. stereotypes) while maintaining adaptations that appeal to an audience (e.g. group terms). future work could also include different languages and varieties in order to provide a wider understanding of the shortcuts and biases hereby highlighted. for deeper insights on the biases, we encourage future research that could, for example, mask shortcuts by lms as identified in our study. we believe that wikihow is an ideal resource for such work because its collaborative nature makes it possible to put changes directly into practice and instructional texts in general would strongly benefit being easier accessible and more inclusive.