BiaSWE : An Expert Annotated Dataset for Misogyny Detection in Swedish (2025)

Abstract: In this study, we introduce the process for creating BiaSWE, an expert-annotated dataset tailored for misogyny detection in the Swedish language. To address the cultural and linguistic specificity of misogyny in Swedish, we collaborated with experts from the social sciences and humanities. Our interdisciplinary team developed a rigorous annotation process, incorporating both domain knowledge and language expertise, to capture the nuances of misogyny in a Swedish context. This methodology ensures that the dataset is not only culturally relevant but also aligned with broader efforts in bias detection for low-resource languages. The dataset, along with the annotation guidelines, is publicly available for further research.

## introduction large language models (llms) have experienced immense growth over the past years due to being capable of solving diverse tasks that previously required a separate model for each specific task @xcite . despite their apparent benefits, it is known that the characteristics of the dataset used to train a language model play a fundamental role in determining the model's behavior @xcite . llms are typically trained on large amounts of data from the internet and thus inevitably reflect the opinions and biases of its users. for example, a 2018 survey showed that about 85% of english wikipedia contributors identified as male @xcite . as llms' behavior "reflects the collective intelligence of western society", llms can perpetuate and even amplify biases and stereotypes of social minorities @xcite . the widespread presence of misogyny online is illustrated by a study from 2020 where 65% of women reported knowing another woman that had been the target of online violence (the economist intelligence unit, 2020). the way to avoid harmful machine learning models is to ensure that the datasets used for training are responsibly curated, involving diverse stakeholders @xcite . however, dataset creation alone is not sufficient, and additional approaches, such as alignment, play a role in guiding model outputs towards human values. in the context of bias detection, misogyny varies by language and culture @xcite . therefore, we consider creating expert-annotated, language-specific datasets crucial for detecting biases, helping to identify areas where models may risk perpetuating harmful stereotypes or undesirable attitudes. to address these challenges, we make two key contributions 1 : 1. we present biaswe, a small annotated dataset for misogyny detection in swedish, annotated for hate speech, misogyny, misogyny type categories and severity. 2. we share the creation process of the biaswe dataset and our annotation guidelines. by doing this, we show how an existing experiment can be adapted to the swedish needs and cultural context. ## related work in recent years, work has been done in the field of dataset creation for bias and hate speech in general, paying great attention to data coming from online sources, especially social media, such as twitter, facebook, reddit, or blogs. this kind of work has been carried out in a multitude of languages, across several cultural contexts, and tends to cover various forms of sexism as it presents in written language. this is the case of @xcite , who present a corpus for detecting sexism in french tweets. another example is the work of @xcite , who sample their bajer dataset from twitter, facebook and reddit posts in danish. however, research is also carried out with the target of more subtle, less explicit misogyny in mind; this is the case of the biasly dataset by @xcite , who gathered their data from scripts from north american movies, in english. the most common method for data collection among the different existing datasets is using keywords @xcite @xcite . the degree of detail or the number of keywords varies from words that do not necessarily imply misogyny (e.g. "she") to ambiguous keywords, to keywords that are very highly related to misogyny and sexism (e.g. "#metoo"). most of the existing misogyny detection datasets provide a taxonomy for different categories of misogyny in addition to the binary classification. many regard the addition of a multi-label classification layer as necessary, given that "binary detection [. . . ] disregards the diversity of sexist content, and fails to provide clear explanations for why something is sexist" @xcite . there is no clear consensus regarding the types of misogyny to classify the sentences into, or even on the optimal level of detail regarding the categories. to the best of our knowledge, this work is the first attempt to create resources for misogyny detection for the swedish language. ## method this section provides an overview of our data preparation process, introduces the team of expert annotators and details the annotation workflow. ## results this section provides an overview of the annotation results. hate speech each of the 450 posts was annotated by two to four experts and in almost twothirds of the cases all experts agreed on whether hate speech was present. in the 334 cases where all experts agreed, slightly more posts were annotated as containing hate speech compared to not containing hate speech but the difference was marginal. however, there was a large difference between the number of posts that were annotated as hate speech by all experts (172) and posts that were considered to be hate speech by at least one expert (288). misogyny figure 1 shows the annotation results for the misogyny classification task. a negative label in the previous task is considered to be a negative label in the misogyny classification task as well, in which case, all 450 posts were again annotated by two to four experts. in slightly more than two-thirds of the examples, all experts agreed on the label. however, in this case, there was a larger imbalance between the two possible labels: 207 posts were considered to be non-misogynistic and 139 misogynistic by all experts. the number of posts considered to be misogynistic by at least one expert was however larger at 243 posts. category there was more disagreement in choosing the category of misogyny. there were between zero and four category annotations for each of the 450 posts in the dataset. out of the 185 posts in the final dataset with more than one category annotation, in 93 cases all experts chose the same category. figure 2 gives an overview of all 546 category annotations in the dataset, comparing the number of times each of the five possible categories was chosen. severity the last annotation task asked the experts to estimate the severity of the misogyny in the post. similarly to the previous task, 185 posts had at least two severity annotations and a closer analysis of those revealed that although the experts seldom selected the same rating, in 91% of cases the difference between the minimum and the maximum rating was not larger than 3. ## discussion and conclusion this project's primary contribution lies in its interdisciplinary approach to misogyny detection in swedish rather than the dataset itself, which remains small. collaborating with experts from diverse fields, we developed an annotation process that captures the complexity of misogyny as it manifests in swedish online discourse. this experiment provides a valuable framework for future studies focused on bias detection in underresourced languages. in the context of misogyny detection, defining what constitutes misogynistic language is inherently challenging. attempting to capture a wider range of potentially harmful expressions risks being too broad, while using a stricter approach might fail to recognize subtler forms of misogyny. the challenge lies in determining who defines misogyny, as cultural, linguistic and societal factors have an influence over the definition, making it a complex decision. the feedback from the experts highlighted the need for clearer operational definitions and stronger contextual support. misogyny detection, particularly in a complex environment like flashback, would benefit from additional discussion on cultural nuances and interdisciplinary perspectives. additionally, better alignment between academic rigor and practical applicability is crucial to ensuring that interdisciplinary projects like this one fully realize their potential. we also took into account the experts' perspective in section 6. in conclusion, while the dataset is limited, the interdisciplinary approach and methodology offer a valuable starting point for future research. refining the annotation process and expanding the dataset could further improve the effectiveness of misogyny detection tools, especially for lowerresourced languages like swedish.