LLM s as annotators of argumentation (2025)

Abstract: Annotated data is essential for most NLP tasks, but creating it can be time-consuming and challenging. Argumentation annotation is especially complex, often resulting in moderate human agreement. While large language models (LLMs) have excelled in increasingly complex tasks, their application to argumentation annotation has been limited. This paper investigates how well GPT-4o and Claude can annotate three types of argumentation in Swedish data compared to human annotators. Using full annotation guidelines, we evaluate the models on argumentation schemes, argumentative spans, and attitude annotation. Both models perform similarly to humans across all tasks, with Claude showing better human agreement than GPT-4o. Agreement between models is higher than human agreement in argumentation scheme and span annotation.

## introduction annotated data is essential in most natural language processing (nlp) tasks, including semantic and pragmatic analysis. while pretrained large language models (llms) have reduced the need for large amounts of annotated training data, labeled data remains crucial for evaluation. however, creating high-quality annotated data can be timeconsuming and expensive, especially when faced with the complex aspects of linguistic meaning involved in annotating a phenomenon like argumentation. annotating argumentation is a challenging task, as it involves not only identifying opinions but how they are argued for. argumentation in itself can often be implicit and context-dependent, and sometimes even subjective, which can lead to differing opinions among annotators. in nlp, the study of argumentation is usually done within the field of argumentation mining, which aims to automatically retrieve and analyze argumentation @xcite . because of the complexity of the task, argumentation annotated datasets used in this field often report lower agreement than annotation of other phenomena in nlp @xcite . these challenges make argumentation especially suitable for investigation. in recent years, llms have excelled at different complex tasks, often outperforming previous baselines @xcite . often, these models are not fine-tuned on data, but instead instructed through prompts to perform various tasks, such as classification. this way of prompting is more similar to annotation of data, rather than training and then classifying. this similarity between prompting and annotation has given rise to several studies comparing how well llms annotate @xcite , with potential advantages in speed and cost @xcite . recently, models are able to handle longer inputs (compare openai's gpt-4's context window of 8102 tokens to gpt-4o's 128k), making it possible to prompt models with actual annotation guidelines rather than shortened instructions. despite the capabilities of llms, not many studies have yet compared argumentation annotation of humans to that of models. because of this, in this study, three swedish datasets annotated with different types of argumentation are used to evaluate how well two llms, gpt-4o and claude sonnet, can annotate argumentation. the datasets are annotated for argumentation schemes, spans of argumentation and attitude, respectively. because these tasks are complex and human annotators often disagree, we are also interested to see how the models annotate in cases of disagreement. for example, is there some annotators the models align with more? more specifically, this study asks: in answering these questions, this paper contributes a novel analysis of the models' capabilities as annotators of argumentation in three different tasks. as far as the author is aware, there are no other studies which analyse argumentation annotation of these kinds. we find that both models can annotate similarly to humans in all argumentation annotation tasks, with claude showing better agreement with humans than gpt-4o. in the argumentation scheme and argumentation span task the agreement between the two models is higher than between human annotators. ## related work in many tasks, it has been shown that llms can perform comparably to human annotators. for example, @xcite use chatgpt for four annotation tasks (stance, topic, relevance, and frames) and find that the model performs similarly to, or better than, human annotators in these tasks, compared to an aggregated gold standard. similarly, in @xcite gpt's performance in several classification tasks is presented. they find that the model overall performs well but struggles with sarcasm and emotion detection. other areas where llms have been used for annotation are stance classification @xcite and grammatical annotation @xcite . however, not all studies find that llms perform well. llms seem to struggle with more complex tasks, for example @xcite find that llms under-perform in the task of event extraction. as mentioned in the previous section, the difference between annotation and classification in these kinds of studies is not always clear. in some studies (for example in liyanage et al. (2024) models are not given instructions similar to that of what a human annotator would receive, but instead a shortened version. as pointed out by @xcite , most studies compare model output to a curated gold standard, without direct comparison to the other annotators. an exception to this is @xcite , who annotate entity-level sentiment in norwegian texts by prompting chatgpt. they compare the model's and five human annotators' annotations to a curated dataset. accuracy and cohen's κ is lower for chatgpt, with the exception of an outlier annotator. they also find that chatgpt's errors deviate from the other annotators' labels more than the human annotators' labels. another example is the study by @xcite , who annotate stance using open source llms. they find that llms show lower agreement with human annotators in cases where human annotators themselves disagree. they also find that in these cases, the stance is less explicit than in other examples. in argumentation mining, studies have shown that llms can perform well on different tasks. for example, abkenar et al. (2024) perform argument discourse unit classification and relation classification using open source models. @xcite explore the argumentation mining tasks of detecting claims, stance and evidence types. they find that gpt-3.5-turbo performs best on complex tasks. @xcite fine-tune open source models and achieve state of the art results for component classification, relation classification and identification. @xcite find that open source llms can outperform the baseline in identifying argumentative relations. there are also examples of llms being used to generate argumentation @xcite . however, llms have not been successful at all tasks. @xcite find that gpt-4 performs below other models in their fallacy detection and classification task. when looking specifically at annotation of argumentation, there are few studies. @xcite examine the annotation of argument quality. in this task, inter-annotator agreement (iaa) among human annotators is between 0.37-0.40 (expert and novice annotators). they calculate iaa across several runs with the same model, treating each run as a new annotator, and it is significantly higher (between 0.73-0.98). they also compare model annotations to human annotations in cases where there is perfect agreement between human annotators, and find that there is moderate agreement between the models and humans for most categories. @xcite investigate how llms can aid in annotation of sematic argument type and find that the models can perform the task but with similar performance to a bert model. they did not compare their results to individual human annotators. ## datasets these datasets represent different annotation approaches in argumentation mining, with varying complexity, document length and genre. the datasets are all in swedish. all datasets were annotated by annotators with a background in linguistics and with swedish as their native language. letters are used to represent the annotators below, for example 'annotator a'. note, however, that the annotators differ between the annotation tasks. the annotations of these datasets are not available online, thus the risk of the model being informed by the annotations beforehand is minimal. dataset statistics are seen in ## method for this study, the models gpt-4o (gpt-4o-2024-08-06) @xcite and claude sonnet (claude-3-7-sonnet-20250219) 1 were chosen. for both models, the temperature was set to 0. this was to make the results more consistent and deterministic. gpt-4o was chosen because it is one of the most prominent and well-known models, and one of the most commonly used ones in annotation studies. it was also is one of the most cost effective models. claude was chosen to provide a comparison with a different model architecture and it also performed well in preliminary experiments. initially, in order to compare the results to an open source model, experiments were run with llama 3.3-8b. 2 however, this model did not perform well enough to be included in this study, possibly due to the model size. a larger llama model was not used due to computational constraints. for each annotation task, the model was given a prompt which consisted of the original annotation guidelines together with short supplemental instructions. the prompt also included an example to be annotated. both the prompt and examples were all in swedish. the length of the guidelines is seen in the guidelines were not changed from the original annotation task, with the exception of removal of tool-specific instructions. the supplementary instructions introduced the task and the guidelines, and told the model in which format it should return the annotations. as previous studies have reported @xcite , care must to be taken when crafting instructions. when writing instructions in this study, variations in wording were found to influence the annotations, and developing instructions that produced the correct output took some trial and error. in this process, both models were included, in order to develop instructions that worked for both of them. for example, mentioning the guidelines early in the instructions would increase the number of annotations returned in the correct format. an example of a prompt can be seen below, see appendix a for the other. the prompt was sent for each document to be annotated. in the case of the argumentation schemes and the argumentative spans, a document would be an editorial or a thread, respectively. for the attitude annotation, ten tweets were sent at a time. the guidelines for the argumentative spans and attitude tasks included annotated examples. in the argumentation scheme task, the guidelines did not include annotated examples. this was done in order to keep the annotation task as similar as possible to the original annotation setup. ## results in general, the models struggled with keeping within the instructions and often added extra knowledge. for example, they would add a motivation for the annotation despite being explicitly asked not to. the models would also often correct the spelling of misspelled words, which made it challenging to automatically compare annotations. some examples were returned without annotation or annotated in the wrong format. due to these issues, around 100 tweets could not be included in the evaluation. in the editorials and the online threads, all examples were included. ## conclusions this paper first examined how well the models could perform annotation tasks when provided with guidelines. in this study we have shown that both models exhibit similar annotation patterns and agreement to that of humans, which leads us to conclude that the models can follow the guidelines and perform the task reasonably well. as these are tasks where humans often disagree, reaching comparable levels of agreement to humans could be a sign that the models, especially claude, 'understands' foot_0 the task. in all tasks, gpt-4o annotated fewer tokens than the human annotators, while claude annotated a similar amount of tokens. this might be because gpt-4o only annotates when highly confident or due to differences in how the models approach the task. while models might respond differently to prompts, gpt-4o consistently annotated fewer tokens even during the prompt design phase. comparing agreement with the annotators, gpt-4o agrees less with the annotators than claude. claude exhibits agreement similar to that of humans, as well as higher agreement in the argumentation scheme task. when manually inspecting annotated examples, claude would often annotate more similarly to humans, while gpt-4o shows similar patterns as another, slightly worse, human annotator. however, both models' annotations were often valid. as there can be cases where multiple interpretations are correct, for example in choosing a component as a premise or conclusion, one can not always conclude that models are wrong even if they choose to annotate differently than humans. however, there were some cases where the models' annotations did not make sense. because of the complexity of these tasks, and the fact that human annotators often disagree in them, the second question asked how models annotate when humans disagree. there were instances where models aligned with specific annotators. in the argumentation scheme task, both models agreed more with one of the annotators. likewise, in the argumentation spans and attitude task, there was higher agreement with some annotators. however, in these tasks, the agreement is also different within the human annotators themselves. most prominently, however, agreement was higher between the models than the humans in the argumentation scheme and spans task, possibly suggesting similarities in their reasoning. in the former task the models also agreed more on argumentation schemes types. for future research, there are several promising directions. first, evaluating how other llms, particularly open-source models, annotate these datasets would help determine whether these findings generalize across different models and versions. second, testing these models in zero-shot settings would reveal the extent of their inherent knowledge about argumentation without explicit guidelines. finally, expanding the analysis to include other argumentation datasets, especially those in english, would provide broader insights into model performance across diverse argumentative contexts. in general, there are many questions to answer regarding how to use llms as annotators. for example, should each separate run be treated as a new annotator? in that case, should a failed run be considered as an annotator failing to perform the task?