Hotter and Colder: A New Approach to Annotating Sentiment, Emotions, and Bias in Icelandic Blog Comments (2025)

Abstract: This paper presents Hotter and Colder, a dataset designed to analyze various types of online behavior in Icelandic blog comments. Building on previous work, we used GPT-4o mini to annotate approximately 800,000 comments for 25 tasks, including sentiment analysis, emotion detection, hate speech, and group generalizations. Each comment was automatically labeled on a 5-point Likert scale. In a second annotation stage, comments with high or low probabilities of containing each examined behavior were subjected to manual revision. By leveraging crowdworkers to refine these automatically labeled comments, we ensure the quality and accuracy of our dataset resulting in 12,232 uniquely annotated comments and 19,301 annotations. Hotter and Colder provides an essential resource for advancing research in content moderation and automatically detectiong harmful online behaviors in Icelandic. We release both the dataset and annotation interface.

## introduction the rapid growth of online communication platforms has led to an increase in harmful behaviors and, subsequently, an increased need for content moderation @xcite . inappropriate comments targeted at specific individuals or groups of people can even go so far as qualifying as hate speech, but more subtle ways of spreading these prejudiced ideas may, for instance, include fear speech, where attempts are made to incite fear about a target community @xcite 1 https://repository.clarin.is/ repository/xmlui/handle/20.500.12537/352 2 https://github.com/icelandic-lt/ annotation_if_sentiment 2023). recent work has focused on detecting these toxic behaviors automatically, thereby lessening the cost and workload for human moderators (see @xcite , @xcite and @xcite for instance). this paper addresses limitations in previous work on sentiment analysis in icelandic @xcite , using a new methodology to improve class imbalance and low annotator agreement in some tasks. our approach first uses gpt-4o mini to analyze approximately 800,000 icelandic blog comments across 25 tasks, including sentiment analysis, emotion detection, hate speech detection, and group generalizations. for most tasks, we employ focused binary annotation, targeting only the extreme cases (highly likely or highly unlikely to exhibit the behavior), rather than using rating scales which have been shown to present challenges in maintaining consistent annotation quality @xcite . the exception is sentiment analysis, where we maintain the standard negative, neutral, and positive categories. this targeted approach allows us to efficiently identify rare but important cases (the proverbial needles-in-a-haystack) such as hate speech comments, which would be resource-intensive to locate through random sampling as used in previous work. to ensure dataset quality, we then employ crowd workers to manually verify the model's predictions, focusing particularly on comments flagged as highly likely or highly unlikely to contain problematic content. this human verification step is crucial for maintaining accuracy and creating a high-consensus dataset. our contributions are as follows: our approach combines ai and human efforts in a two-phase annotation process designed to create a high-quality dataset for tasks where the phenomena of interest are often rare. this scarcity poses a significant challenge for dataset creation -random sampling would require extensive human annotation effort to find sufficient positive examples while focusing only on suspected positive cases could bias the dataset. our methodology aims to balance these concerns by using ai to efficiently identify potential cases across the full spectrum, followed by targeted human verification. in the first phase (silver labeling), an llm analyzes a large dataset of comments. for this initial screening, we use gpt-4o mini with a prompt designed for structured output (see section 2.1). while the model was instructed to consider itself an expert in icelandic blog analysis to maintain consistent task framing across annotations, we acknowledge this is a common but debatable prompting practice that warrants further investigation. for all tasks except sentiment analysis, the llm uses a 5-point scale for labeling to capture nuanced assessments. in the second phase (gold labeling), human annotators review selected comments, focusing primarily on those the llm rated at the extremes of the scale (1 or 5). this design choice reflects our priority of establishing a foundational dataset with clear, agreed-upon examples of each phenomenon. while this approach may not capture all nuanced edge cases, it serves several important purposes: (1) it enables efficient identification of clear positive examples for rare phenomena, (2) it helps establish reliable baseline annotations for model evaluation, and (3) it aligns with findings that human annotators achieve higher agreement on clear cases @xcite . we acknowledge this as a limitation -fu-3 [links redacted] ture work should explicitly target borderline cases to improve model robustness. human annotators perform binary (yes/no) annotations foot_0 for a single task at a time to reduce task switching fatigue. the simplified binary choice for humans, compared to the llm's 5-point scale, reflects our focus on identifying clear instances while acknowledging that intermediate cases may require more nuanced future investigation. this method of using a language model to identify potential candidates for gold labeling builds on established practices. for instance, when compiling their goemotions dataset, @xcite used a bert-based model to filter out comments that contained high levels of neutrality, leaving the more emotional comments for humans to annotate. ## discussion the gold standard, human annotated hotter and colder dataset is relatively small. while its main purpose is to serve as validation for the ai-labeled silver dataset, it can also be used as training data for few-shot learning models. the silver dataset offers considerable flexibility, supporting the training of models for individual tasks, such as the automated detection of hate speech. however, the utility of both datasets extends beyond singletask applications. multi-task learning (mtl) allows a model to tackle multiple tasks simultaneously, drawing on shared representations and insights across tasks to improve overall performance. in sentiment analysis, for example, an mtl framework enables a more nuanced understanding of human communication. @xcite demonstrate how sarcasm detection can significantly enhance the performance of sentiment analysis models, particularly in identifying negative sentiment in sarcastic contexts. our results indicate that sarcasm detection remains a challenge, likely contributing to the suboptimal performance of the model in the sentiment analysis task. given that icelandic humor often relies on sarcasm, this cultural factor may explain some of the difficulties the model encounters in this task. consequently, it is plausible that an icelandic sentiment analysis model would benefit from an mtl approach, particularly one that integrates sarcasm detection as a complementary task. when working with multilingual llms, cultural norms exhibited by the model might not always match those of the country in question @xcite . rather, these models reflect the cultural, legal, and ideological values of their creators. @xcite showcased that gpt-4o mini generally mirrors values that are commonly found in english-speaking and protestant european countries. while this cultural bias may not be inherently problematic for our purposes, it could lead to reduced agreement between human annotators and ai models in culturespecific annotations. for instance, ethical alignment performed during model training may influence the model's ability to judge appropriateness in social contexts. a model might consistently classify toxic or hateful comments as unacceptable, even when human annotators might tolerate such comments in specific contexts, such as in private conversations among friends or informal parliamentary discourse. these nuances in cultural and ethical standards may hinder the model's performance in tasks requiring a deep understanding of social norms and context. on the flip side of the coin, hotter and colder additionally offers invaluable insight into the sociolinguistic patterns of a small online community. future research will i.a. include an analysis of how discourse changes in liaison with current events, which communities are most affected by toxic behaviors and hate speech, and the characteristics of toxic users. ## conclusion this study presents hotter and colder, a dataset annotated for 25 tasks that examine various types of online behaviors. by leveraging both ai-based silver labeling and human-in-the-loop gold labeling, we ensure a comprehensive approach to annotating toxic behaviors, emotions, sentiments, and more in icelandic blog comments. this dual-phase annotation methodology enabled the identification of rare but critical instances of harmful speech while maintaining high annotator agreement across a variety of tasks. the introduction of a multi-task learning framework as a future direction holds promise for improving the detection of complex phenomena, such as sarcasm, which remains a challenge for both ai models and human annotators, particularly in culturally specific contexts. by integrating tasks such as sarcasm detection with sentiment analysis, future models may achieve greater accuracy and nuanced understanding in detecting various forms of harmful and toxic speech. hotter and colder lays the foundation for future work on mitigating bias and improving ethical alignment in ai models for icelandic, hopefully fostering safer and more inclusive online environments.