Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and LLM s (2025)

Abstract: Arguments evoke emotions, influencing the effect of the argument itself. Not only the emotional intensity but also the category influences the argument’s effects, for instance, the willingness to adapt stances. While binary emotionality has been studied in argumentative texts, there is no work on discrete emotion categories (e.g., ‘anger’) in such data. To fill this gap, we crowdsource subjective annotations of emotion categories in a German argument corpus and evaluate automatic LLM-based labeling methods. Specifically, we compare three prompting strategies (zero-shot, one-shot, chain-of-thought) on three large instruction-tuned language models (Falcon-7b-instruct, Llama-3.1-8B-instruct, GPT-4o-mini). We further vary the definition of the output space to be binary (is there emotionality in the argument?), closed-domain (which emotion from a given label set is in the argument?), or open-domain (which emotion is in the argument?). We find that emotion categories enhance the prediction of emotionality in arguments, emphasizing the need for discrete emotion annotations in arguments. Across all prompt settings and models, automatic predictions show a high recall but low precision for predicting anger and fear, indicating a strong bias toward negative emotions.

## introduction the role of emotionality received considerable attention in recent research, spanning from a focus on pathos @xcite to the study of emotion intensity and its role on argument persuasiveness @xcite . in natural language processing, most research focused on binary or continuous emotion concepts @xcite , and the role of such variables on argument effectiveness has been confirmed in empirical studies. in argumentation theory and psychology, however, it has also been shown that not only binary emotionality or intensity argument label es gibt impfstoffe, welche unsere dna dauerhaft verändern können. diese impfstoffe nennen sich mrna-impfstoffe. das mrna gelangt in unsere dna und gliedert sich dort mit ein. dadurch wird unsere dna leicht verändert. there are vaccines that can permanently alter our dna. these vaccines are called mrna vaccines. the mrna gets into our dna and integrates into it. this slightly alters our dna. interest, disgust, surprise to fill this gap, we approach argumentative texts as a domain for emotion analysis, more concretely the prominent subtask of emotion categorization. in this research direction, the goal is to assign emotion concepts to predefined textual units, for instance literary texts @xcite , political debates @xcite , or social media texts @xcite . emotions are often expressed implicitly, without explicitly mentioning emotion concepts @xcite @xcite . that renders emotion classification a challenging task, even for humans, who tend to agree more with other readers of an emotional text than with the original emotion experiencer @xcite . statement argument kamele speichern fett in ihren höckern. kamele sind sehr große tiere und benötigen sehr viel energie. um diese energie aus den fettreserven zu erhalten, wird das fett in den höckern gespeichert. da kamele sich meistens in gegenden befinden, in denen sie wenig nahrung finde und dort als lastentiere eingesetzt und lange wege zurücklegen, werden, ist es überaus wichtig, sich vorher einen fettspeicher anzulegen. außerdem schützen die mit fett gefüllten höcker die kamele vor der hitze und der sonne, sie wirken wie eine art polster, dass die übrigen organe vor überhitzung schützt. camels store fat in their hump. camels are very large animals and need a lot of energy. to energy from the fat reserves, the fat is stored in the humps. humps. since camels are usually found in areas where they find little food and are used there as beasts of burden and travel long distances, it is extremely important to build up a fat to build up a fat store beforehand. in addition, the fat-filled humps humps filled with fat protect the camels from the heat and the sun, they act they act like a kind of cushion that protects the other organs from overheating. hence, we crowdsource human emotion and convincingness annotations for the publicly available german defabel corpus @xcite . both convincingness and (discrete) emotion labels are annotated based on the perceived convincingness and evoked emotion in the participants as exemplified in analyzing emo-defabel reveals that joy and pride in arguments are correlated with higher convincingness, while anger is negatively correlated. our experiments demonstrate that the promptbased model categorizations are heavily biased toward negative emotions. the biases on concrete emotions differ between the models. ## annotation we enhance an existing argument dataset with emotion labels as the basis for our study. in total, we request three individual annotations for 300 arguments. data sets. the basis for our annotation is the defabel corpus @xcite . it contains argumentative german texts, annotated via crowdsourcing. the participants were asked to write persuasive arguments supporting a given statement (e.g., "camels store fat in their hump."), selected from the truthfulqa dataset @xcite . the corpus contains 1031 arguments for 35 statements. we select this resource because it contains short, isolated arguments. additionally, this dataset allows us to effortlessly capture the annotators' stances towards each statement. we randomly select 300 arguments for our annotation task, evenly distributed across statements. emotion labelset. our emotion label set starts with the basic emotions (anger, disgust, fear, joy, sadness, surprise) and is expanded by cognitive evaluations (interest) and self-directed states (shame, guilt, pride). we further offer a free text field for mentioning evoked emotions that are not covered by our label set as an exploratory approach. annotation setup. we show one statementargument pair per page. the participants are instructed to read those texts and provide their stance toward the statement on a 5-point scale (strongly agree, . . . , strongly disagree). in addition, we ask how convincing they perceive the argument on a 5-point scale (not convincing at all, . . . , very convincing). for the emotion label, we first ask if the argument evokes an emotion in the participants (yes/no). if they answer yes, they are asked to provide the concrete emotion label from our emotion label set (joy, anger, fear, sadness, disgust, surprise, pride, interest, shame, guilt). participants have the option to input an evoked emotion in case it is not covered by our label set. crowd-sourcing details. we use the platform prolific foot_0 with potato @xcite . participants are prescreened to be in germany, have german as their first and native language, be fluent in german, and have an approval rate of 90-100%. each participant answers the survey for five statement-argument pairs (and one attention check, see figure 6 for an example). we pay each participant 1.20c for one survey, which on average takes 7 minutes. participants can participate up to 12 times, and therefore annotate up to 60 statementargument pairs. in total, the cost of the study amounts to 316.87c. the contributing participants of our studies were on average 35.8 years old (21 minimum, 68 maximum). from this set, 98 identified as male and 96 as female. note that we did not limit the study to participants with these two genders. ## models to investigate the efficiency of llm annotation of emotions in arguments we differentiate between two dimensions, the emotion domain setting (binary, closed-domain, open-domain) and the technique (zero-shot, one-shot, and chain-of-thought) of the prompts. combining these strategies results in eight prompt formulations (cf. ## experiments we now explain the experiments, analyze the human annotations and subsequently answer the research questions stated in the introduction. figure 1 : pair-wise co-occurrences of emotion labels in the human annotation study for 300 arguments. the upper part displays percentages, the lower absolute numbers. ## conclusion with this paper, we expanded on theoretical work on the interplay of emotion categories and argument convincingness and previous work in nlp on binary emotionality of arguments. we presented emo-defabel, the first corpus of discrete emotion classes in arguments and analyzed the interplay of emotions and convincingness in german arguments. we found that positive emotions (joy, pride) are correlated with higher convincingness scores and negative emotions (particularly anger) with low convincingness scores, showcasing the relevance of analyzing discrete emotion categories. when binary emotionality labels are required, we showed that inferring binary labels from discrete emotion classes performs better than directly requesting binary labels from llms, a result that may also affect related work on automatically generating persuasive arguments @xcite . we further find that there are only minor performance differences across prompting approaches and emotion domains. falcon, llama, and gpt show a bias toward predicting negative emotions in arguments. to mitigate these issues, we propose to study fine-tuning or prompt optimization in the context of argument-emotion annotations in future work. specifically, the precision of individual emotion classes has to be improved. additionally, a fine-grained analysis of the argumentative structure and quality could further enhance our understanding of the interplay of emotionality and convincingness in arguments. related to that, we want to point out that our study could be expanded with a perspectivist approach (e.g., focusing on personaaware annotation, human characteristics and demographics) and discrete emotion annotation and analysis of arguments in discourse.