C hat GPT vs. Crowdsourcing vs. Experts: Annotating Open-Domain Conversations with Speech Functions (2023)

Abstract: This paper deals with the task of annotating open-domain conversations with speech functions. We propose a semi-automated method for annotating dialogs following the topic-oriented, multi-layered taxonomy of speech functions with the use of hierarchical guidelines using Large Language Models. These guidelines comprise simple questions about the topic and speaker change, sentence types, pragmatic aspects of the utterance, and examples that aid untrained annotators in understanding the taxonomy. We compare the results of dialog annotation performed by experts, crowdsourcing workers, and ChatGPT. To improve the performance of ChatGPT, several experiments utilising different prompt engineering techniques were conducted. We demonstrate that in some cases large language models can achieve human-like performance following a multi-step tree-like annotation pipeline on complex discourse annotation, which is usually challenging and costly in terms of time and money when performed by humans.

## introduction discourse analysis as a method of an abstract dialog representation is used in various nlp tasks: dialog management @xcite , dialog generation @xcite , dialog summarization @xcite , emotion recognition @xcite , etc. mostly, discourse structure is considered to be an interconnected system of linguistic features such as a topic, pragmatics, and semantics. one of the main goals of discourse analysis is to describe pragmatics of actions performed by speakers within a communicative process, i.e., characterise the interlocutors' intentions at a certain moment of their interaction @xcite . despite the fact that there are numerous theoretical approaches to dialog discourse analysis, only a * these authors contributed equally to this work figure 1 : example of dialog annotation with speech functions few of them consider the complexity of conversational nature and allow for annotation on multiple levels @xcite @xcite . in this paper, we propose a multi-dimensional and hierarchical taxonomy of speech functions introduced by @xcite as an alternative for abstract dialog representation. in contrast to other annotation schemes, this taxonomy is topicoriented and includes classes that are very similar in terms of pragmatics (see appendix b). the taxonomy provides a comprehensive, systematic discourse model of dialogues (see figure1). traditionally, discourse annotation is performed manually by trained experts or crowdsourcing workers @xcite . automating it partially or entirely is key to making this complicated process faster and cheaper. we argue that in complex discourse annotation tasks large language models (llms) can be used to establish decent quality silver standards that would be later checked and improved by expert annotators. in this paper, we annotate dailydialog @xcite , a multi-turn casual dialog dataset, using the speech function taxonomy. the annotation is conducted in three ways: 1) by experts with at least b.a. in linguistics; 2) by workers of toloka foot_0 , a crowdsourcing platform; 3) with the use of a large language model, specifically, chat-gpt (gpt-3.5-turbo). we then compare the performance of crowdsourcers and chatgpt using expert annotation results as the gold standard and analyse the findings to prove that llms can achieve humanlike performance on complex discourse annotation tasks. finally, we release the repository with all the code we used to perform the annotation with chatgpt foot_1 . ## taxonomy of speech functions although the original taxonomy of speech functions included 45 classes, we reduced it to 32 labels (see appendix b). created for analysing casual conversations, every speech function describes several functional dimensions performed on different segmentation levels. this approach allows for annotating all the speaker's intentions and communicative actions at each moment of the dialog. ## human annotation using speech function taxonomy the annotation of discourse structures or dialog acts is not a simple task as it requires either linguistic knowledge or trained workers @xcite . additionally, understanding the speaker' intentions in utterances can vary among individuals, further complicating the task. in this section, we compare the results of speech function annotation completed by experts with professional backgrounds in linguistics and crowdsourcing assessors. to evaluate the agreement between the experts and between the assessors, we use fleiss' kappa that is an extension of scott's pi (π) for two coders. fleiss' kappa can deal with any number of annotators, where every item is not necessarily annotated by each annotator. it is the most commonly used method to evaluate taxonomy reliability in tasks related to discourse analysis. however, this method has the limitation of not considering the common mistakes of annotators. therefore, we measured not only inter-annotator agreement but also three most common metrics for multi-class classification tasks with imbalanced data -macro f1, weighted precision and weighted recall, by comparing the workers' annotations to the results of experts. ## methods the annotation task in question required careful instruction preparation even for human annotators as opposed to simpler tasks such as sentiment classification, bot detection, etc. thus, the process of creating the best prompt for an llm is also a challenging and multi-step process. we conduct a number of experiments in order to find the best way to use chatgpt for complex discourse annotation tasks. in all cases, the system_message we used while querying chatgpt api was "you are a professional linguist annotator who has to perform a discourse annotation task". the user_message varied for different experiments. see figure 5 for an example of user_message. to reduce the number of api calls and thus the time and the cost of the annotation, we also used automatic methods other than chatgpt on some steps of the annotation. for example, in all our experiments we used topic shift classifier to detect the beginning of a new topic in a dialog. it is worth noting that chatgpt did not perform well in this particular task. the topic shift classifier was trained using the deeppavlov @xcite library utilizing a double sequence binary classifier model based on roberta-large-mnli, with two sequential utterances as input. the true labels indicate topic change in the utterances. the following hyper-parameters were used to train the model: learning @xmath0, @xmath1, input max @xmath2. to successfully train the model, we used the early-stopping technique. the classifier was able to transfer the knowledge acquired during pre-training on mnli to the related problem of shift identification by using a pre-trained model @xcite . ## conclusion and future work we conducted several experiments on the annotation of casual conversations with speech function taxonomy performed by experts in linguistics, crowdsourcing workers, and chatgpt. in this paper, we took a closer look at the problems of defining multilayer taxonomies in real dialogs and, furthermore, explored whether it is possible to differentiate between those classes when annotating. experiments with chatgpt have demonstrated the potential of using llms for linguistic annotation with accuracy that is close to crowdsourcing workers' performance on some dialogs. even though guiding the model across a tree-like structure of instructions to reach the final label seems to be promising, it still falls short of non-expert performance on such tasks and does not let the researchers explore variations in how non-experts understand discourse structures. it is important to mention that a significant drawback of the method we propose is the neces-