Using C hat GPT for Annotation of Attitude within the Appraisal Theory: Lessons Learned (2024)

Abstract: We investigate the potential of using ChatGPT to annotate complex linguistic phenomena, such as language of evaluation, attitude and emotion. For this, we automatically annotate 11 texts in English, which represent spoken popular science, and evaluate the annotations manually. Our results show that ChatGPT has good precision in itemisation, i.e. detecting linguistic items in the text that carry evaluative meaning. However, we also find that the recall is very low. Besides that, we state that the tool fails in labeling the detected items with the correct categories on a more fine-grained level of granularity. We analyse the errors to find systematic errors related to specific categories in the annotation scheme.

## introduction the present paper investigates the potential of using large language models (llms), specifically chat-gpt for annotating pragmatic categories. recent advances in artificial intelligence (ai), propelled by llms such as chatgpt, have led to substantial improvements in automating complex linguistic tasks such as coherent text generation, text simplification, machine translation, error detection, and question answering. they have produced unprecedented results in a wide range of applications including linguistic annotations. we focus on the feasibility of using chatgpt to annotate linguistic items expressing evaluation, attitude and emotion according to the framework based on appraisal theory @xcite . more specifically, we probe the tool for the annotation of evaluative language in spoken popular science discourse. to this end, we automatically annotate selected english texts extracted from a dataset of ted talks and evaluate the annotation results manually to see how well chatgpt can recognise linguistic items carrying the pragmatic meaning under analysis and if these items were correctly classified according to the pre-defined categories of the appraisal theory. we also analyse erroneous cases to find the systematic failure of the tool in assigning specific categories. so far, especially when investigating complex pragmatic and discourse features, most of the annotation work is still done manually. however, manual corpus annotation is time-consuming and requires specialised skills and training. automating this task would significantly reduce the annotator's workload, save resources and costs, and would allow researchers to build and annotate larger and thus more representative corpora. moreover, manual annotation is error-prone and is subject to inconsistencies which can be avoided by automatic or semi-automatic procedures integrated into the process of annotation. thus, our aim is two-fold. on the one hand, we aim to evaluate chatgpt in a demanding annotation task of complex linguistic phenomena. on the other hand, we want to learn about the challenges present in the data at hand. this knowledge will allow us to detect phenomena that are particularly hard to annotate, which in turn will allow us to create better guidelines for human annotators. the remainder of the paper is structured as follows. in section 2, we outline the basic principles of the appraisal theory and present the categories we consider in our analysis. section 3 describes related works that utilise llms for annotating. in section 4, we provide details on the data used for this study, as well as procedures to annotate the data. we present and discuss our results in section 5. an outlook for future work is given in section 6. we discuss the limitations of our work in section 7. ## theoretical background we use appraisal theory @xcite developed under the general framework of sys-temic functional linguistics (sfl, @xcite . appraisal is related to the interpersonal metafunction of language and it consists of three interrelated domains: attitude (expressions of opinions and feelings), engagement (positioning of the writer/speaker toward other voices within the discourse) and graduation (intensifying or down-toning attitude and engagement). we focus on the three sub-types of attitude: affect (emotions), judgement (moral judgements about human behaviours) and appreciation (evaluations of objects, products and processes). we use the original appraisal annotation scheme and consider these three sub-types of attitude as the main semantic categories (level 1). furthermore, we focus on the sub-categories of attitude on a more fine-grained level (level 2) and their related subvalues that represent the finest level of granularity (level 3). the categories on levels 2 and 3 include (with the 3rd level given in brackets): for affect dis/inclination (fear; desire), un/happiness (misery, antipathy; cheer, affection), in/security (disquiet, apprehension; confidence, trust), and dis/satisfaction (ennui, displeasure; interest, pleasure), for judgement social esteem (normality, capacity, tenacity), and social sanction (veracity, propriety), and for appreciation reaction (impact, quality), composition (balance, complexity), and valuation. our detailed scheme is presented in figure 1 . ## conclusion and future work the present paper evaluates the use of chatgpt as an annotation tool for evaluative language in spoken popular science discourse. our results showed that while the concordance rate between the tool and the human annotator in terms of itemisation precision was rather high (94,49%), it was rather low in terms of the sub-category label classification (35%). although the tool succeeds in recognising evaluative items, it fails in retrieving all of them (as shown by the low recall) and in correctly classifying them. also, the tool produced hallucinations in the output. this is especially problematic as human annotators not only have to manually check the results in terms of recall and the assigned category, but they also have to double-check if the annotated item appears in the text at all. this problem could be eliminated by changing the form of the output (see discussion of limitations below). our observations also show that most of the recognised evaluative items (58,89%) were adjectives. however, a more detailed analysis of partof-speech (pos) categories would be an asset. for this, we plan to automatically pos-tag and parse the corpus, which will also help us to define systematic morpho-syntactic patterns of explicit evaluative expressions. however, it would be also interesting to find out if chatgpt could identify and annotate implicit (indirect) attitudes in texts which is a task demanding even for human annotators, which is also amongst the tasks for our future work. besides that, the tool performed better on some texts than on others. however, as we only analysed one text per discipline we cannot determine if the varying results depend on the text discipline or other factors, such as the speaker's style or the topic of the text. interestingly, regardless of the text length, the number of instances annotated by chatgpt was more or less the same (approximately 30 instances per text). our results show that splitting the texts and performing the annotation task on shorter parts improves the output. this suggests that text length could potentially have an impact on the results. with this assumption, the recall rate of future stud-ies could be improved by adjusting the text length. this, however, does not guarantee that some items will not be missed out completely or not be false positives. we also found that the output varied across several runs of the same prompt. this means that the results of the automatic annotation are not reliable in terms of consistency, which is in line with other studies. while other existing annotation projects found that gpt-4 was more reliable in annotation than other gpt models (e.g. @xcite , we conclude that even the latest model gpt-4 turbo exhibits low consistency compared to human annotator for the annotation of evaluative language. we explain the inconsistency of chatgpt as an annotator of complex linguistic phenomena by the non-deterministic nature of llm: identical input with minor word alterations in prompts leads to different outputs (see reiss, 2023, for more details). thus, improving and designing effective prompts is important for optimal model performance. we will also explore and compare different prompting strategies (few-shot vs. zero-shot). by doing so, we will investigate whether the tool's ability of incontext learning through prompting can increase the concordance rate. moreover, we will test and compare further existing llm tools (e.g. bard 6 ). we also plan to collect annotations by multiple human annotators and compare them with those of chatgpt and other llms. we will use the findings of this study to improve the annotation guidelines for human annotators in terms of itemisation and classification of appraisal categories. crucially, we will follow up on the evaluative items that chat-gpt incorrectly assigned and observe if the human annotators also tend to disagree regarding those linguistic items, explaining what kind of tasks the tool performs best on. we will also analyse the evaluation span as its length seems to have an impact on the chatgpt's output and may also cause inconsistencies in human annotator decisions. besides, we will annotate several texts of the same discipline which will allow us to identify tendencies in terms of stylistic or domain-specific variation. therefore, we plan to analyse texts from different genres and compare the results for different text types. further, we will cross-lingually test chatgpt's annotation of evaluation in german translations of the analysed texts by comparing originals with translations and exploring how eval-6 https://bard.google.com/chat .