A GPT among Annotators: LLM -based Entity-Level Sentiment Annotation (2024)

Abstract: We investigate annotator variation for the novel task of Entity-Level Sentiment Analysis (ELSA) which annotates the aggregated sentiment directed towards volitional entities in a text. More specifically, we analyze the annotations of a newly constructed Norwegian ELSA dataset and release additional data with each annotator’s labels for the 247 entities in the dataset’s test split. We also perform a number of experiments prompting ChatGPT for these sentiment labels regarding each entity in the text and compare the generated annotations with the human labels. Cohen’s Kappa for agreement between the best LLM-generated labels and curated gold was 0.425, which indicates that these labels would not have high quality. Our analyses further investigate the errors that ChatGPT outputs, and compare them with the variations that we find among the 5 trained annotators that all annotated the same test data.

## introduction sentiment analysis constitutes an annotation task that is highly subjective in nature, typically with moderate inter-annotator agreement levels @xcite @xcite . recently, @xcite and @xcite compare the label quality of crowd workers to the labels generated by a large language model (llm). they show how annotations by a llm can surpass the quality obtained from crowd workers, for certain annotation tasks for the english language. @xcite employ multilingual generative language models (mt5 and mbart) for other sentiment analysis tasks through prompts and fine-tuning for czech sentiment analysis. they find that prompting these types of models can be a potentially promising avenue for few-shot or zero-shot scenarios. an important direction in recent work on sentiment analysis focuses on analysis of longer texts @xcite . our task of entity-level sentiment analysis (elsa) follows in this direction. it was introduced and motivated by @xcite and can be defined as providing one sentiment score for each volitional entity in a text, the reader's total impression from reading the entire text. the overall, entity-wise sentiment is central to this work, since the annotated texts are quite long, containing several different entities, whereby each entity may be referenced in several sentences in separate parts of the text. a volitional entity, or just "entity" in our context, is a person or organization mentioned by its proper name in the text. we recently released a manually annotated dataset for the elsa task where the overall sentiment conveyed in the text towards each volitional entity is annotated. foot_0 figure 1 shows an example text with elsa annotations. in this paper we present a number of experiments attempting to generate elsa sentiment annotations by prompting chatgpt, and we perform an in-depth comparison between the llm-generated labels, and the labels provided by five human annotators that all labeled the test-split of our dataset in parallel. with this paper we also make available the annotators' labels for each entity in the test set. we believe that the elsa annotation task offers a challenging testbed for llm-based annotation due to the following characteristics: longer texts the texts are professional published reviews with a mean sentence count of 27.5. ## elsa annotation the newly created elsa dataset contains the texts of the norec fine dataset @xcite ) -a subset of the norwegian review corpus (norec; @xcite ) of professional reviews from a variety of domains, e.g. screen, music, restaurants and literature. the elsa annotations adds sentiment information at the sentence-and documentlevel for each volitional entity in a given text; entities of the types person (per) and organization (org). the texts contain on average mentions of 6 different entities, and each entity is as-signed an individual sentiment label based on the reader's overall impression from the text. sentiment polarity is classified with two intensity levels, providing a set of five possible labels: "positive-standard", "positive-slight", "neutral", "negative-slight", and "negative-standard". the elsa dataset is annotated by 5 annotators, all native norwegian undergraduate nlp-students. after introductory training and fine-tuning of the guidelines, the annotators (single-)annotated separate parts of the main body of the dataset, i.e. the training and development splits. the test set, consisting of 44 documents with a total of 1252 sentences and annotated sentiment toward 247 unique entities, was subsequently annotated by all five annotators. the entire dataset was manually curated by the project leader. curation consisted mainly of error-correction and assigning the majority vote, which was accepted for 90% of the entities. an english translation of the annotator guidelines is found in the supplementary materials. ## llm-based annotation we employ openai's chatgpt (openai, 2023) in order to explore whether a widely used llm can provide sentiment labels for volitional entities in longer norwegian texts, similar to that of native human annotators. we here present the alternatives explored in our experiments. the results from the various experiments can be found in the model prompted was gpt-4, gpt-4-1106-preview. for all our experiments, we instructed the model to output both a justification regarding the sentiment of each entity, and to provide the concluded labels separately in structured format (json). the prompts used in the experiment yielding best results can be found in the appendix. the instructions, prompts and replies for one text during the earlier "gpt06" experiment, can be found in the supplementary material. norwegian or english prompting we experimented with both norwegian and english prompts. we always specified in the instructions that the texts would be in norwegian, and we mostly got norwegian text back. ## findings ## suggestions for future work from the findings reported in this paper, we are personally not encouraged to undertake new annotation projects using llm-based annotation only. but we see a potential for augmenting datasets using a similar approach. when it comes to modelling, both the elsa dataset and other sa datasets may benefit from being extended by llm annotations. including more open source instructiontuned llms would then be essential. modelling with llms is also highly relevant, and the experiments reported here can be used as the starting point for further experiments in that direction. ## conclusions we have presented and analyzed a dataset of annotators' agreement for the task of entity-level sentiment analysis in norwegian, and studied how well gpt-generated labels compare with labels produced by five different human annotators for this particular task. we consider the elsa task to be interesting for such comparisons since the entire text needs to be analyzed for each entity, and the texts are lengthy, varying around a mean of 27.5 sentences per text. we find that the gpt-generated labels have lower accuracy and cohen's kappa than any annotator. but when we try to quantify the magnitude of the deviations, e. g. through mse, we find that one "outlier" annotator has stronger deviations from gold that the gpt-generated labels. this represents new insights for us, since it has been assumed that the errors made by an llm could be more dramatic than those made by humans. similar approaches may therefore be well worth exploring for other sentiment analysis tasks in other languages similarly related to english. particularly in parallel with human annotators. the only method we are aware of which serves to explore how a llm could help with a certain new task, is by annotating data manually, since the task is to identify the sentiment as perceived by humans.