A Fully Automated Pipeline for Conversational Discourse Annotation: Tree Scheme Generation and Labeling with Large Language Models (2025)

Abstract: Recent advances in Large Language Models (LLMs) have shown promise in automating discourse annotation for conversations. While manually designing tree annotation schemes significantly improves annotation quality for humans and models, their creation remains time-consuming and requires expert knowledge. We propose a fully automated pipeline that uses LLMs to construct such schemes and perform annotation. We evaluate our approach on speech functions (SFs) and the Switchboard-DAMSL (SWBD-DAMSL) taxonomies. Our experiments compare various design choices, and we show that frequency-guided decision trees, paired with an advanced LLM for annotation, can outperform previously manually designed trees and even match or surpass human annotators while significantly reducing the time required for annotation. We release all code and resultant schemes and annotations to facilitate future research on discourse annotation.

## introduction discourse analysis is essential in nlp tasks like dialog management, generation, summarization, and emotion recognition @xcite @xcite . traditionally, discourse annotation depends on manual expert labeling, which is costly and time-consuming. llm-based annotation presents a promising alternative, enhancing speed, consistency, and cost-effectiveness @xcite . however, challenges such as biases and domain limitations necessitate careful prompt design and evaluation. in @xcite , the authors explored using chatgpt to automate discourse annotation for labeling chit-chat dialogs using the speech functions (sfs) taxonomy @xcite . sfs categorize communicative acts in dialog, capturing speaker intentions and interactions in a hierarchical structure to analyze conversational flow (see figure 1 for an example and appendix a for the full label set). @xcite al. (2023) (3) complex tree-like scheme with yes/no questions, using a complex tree-like annotation pipeline originally designed for crowdsourced annotation. since breaking a multi-label selection task into smaller sub-tasks using a tree structure has improved human performance in complex discourse annotation @xcite , the authors hypothesized that the same approach could enhance llm-based annotation. prior research also suggests that guiding models with tree-structured prompts significantly improves performance @xcite . @xcite found that the tree-like scheme approach enhances llm accuracy, achieving near-human performance, and suggested that llms could serve as a "silver standard" for annotation. however, newer and more powerful llms have been released since this work was published. these developments not only enhance the potential of llms to be used for annotation but also open the door to automating the creation of treelike schemes, foot_0 enabling a fully automated pipeline. this is especially valuable for large taxonomies, such as intelligent pal @xcite or teach-da @xcite for task-oriented systems, where manually creating tree schemes is complex and time-intensive. this work automates tree schemes creation, making them usable for annotation by crowdsourced workers and llms. a tree scheme is a decision tree that classifies dialog utterances through a series of questions, which can be binary or non-binary, using yes/no or open-ended formats. an example of a tree generated using the pipeline proposed in this work can be found in appendix b. ## related work discourse analysis researchers analyze discourse structures to improve dialog understanding and management, focusing on pragmatics and speaker intent. one of the key frameworks is the dialog act (da) theory @xcite , which assigns pragmatic labels to utterances. the swbd-damsl scheme @xcite , initially created for casual conversations and widely applied to task-oriented systems, classifies dialog acts into 42 classes. taxonomy of speech functions @xcite offers a hierarchical annotation approach, integrating da principles and relational analysis. dialog acts are beneficial in task-oriented dialog agents. for example, @xcite introduce a scheme for embodied agents, improving natural language interactions and task success, and @xcite develop speech act annotated corpus (spaac) scheme, balancing specificity and generalizability in task-oriented dialogs. ## speech functions corpus the experiments described in this work are based on the speech functions corpus, a dataset of dialogs annotated with sfs. this corpus was developed by @xcite , where three experts, each with at least a b.a. in linguistics, annotated the dailydialog dataset @xcite -a multi-turn casual dialog dataset -using the speech functions taxonomy @xcite . the authors of the corpus reduced the original 45 classes proposed by @xcite to a more manageable set of 32 classes. the tag set covers five functional dimensions: turn management, topic organization, feedback, communicative acts, and pragmatic purposes. while all dimensions are embedded within speech functions, they are distributed unevenly across tags, with individual speech functions incorporating between two and five dimensions. figure 1 illustrates an example of a speech function that includes all dimensions: react.rejoinder.support.track.clarify. in this example: (1) react represents turn management, indicating a speaker change or a reaction to a previous utterance; (2) rejoinder corresponds to topic organization, signifying active topic development that influences the dialog flow; (3) support denotes feedback, showing that the speaker is supporting an interlocutor; (4) track falls under communicative acts, identifying questions; (5) clarify serves a pragmatic purpose, indicating a question aimed at obtaining additional information on the current conversation topic. some sfs, however, cover fewer dimensions. for instance, open.attend represents only two: turn management (marking the conversation's beginning) and communicative acts (a greeting). the speech functions corpus includes 64 dialogs, containing 1,030 utterances. appendix c shows an example of an annotated dialog. to evaluate the effect of the taxonomy size on the method proposed in this work, the existing taxonomy was converted into the following subsets: this subset comprises the 20 most frequently occurring classes in the speech functions corpus. it is designed to evaluate how well the model handles frequent classes in the absence of rarer ones. ## framework for tree construction with llms traditionally, discourse annotation has been performed manually by experts or trained annotators, relying on predefined end-class descriptions. however, @xcite investigates whether non-trained, non-expert annotators can reliably annotate coherence relations using a step-wise approach, which functions similarly to a decision tree. their findings indicate that a structured stepwise method can indeed make discourse annotation more accessible to non-experts, facilitating largescale annotation without requiring extensive training. this is achieved by using cognitively plausible primitives rather than relying on complex end labels. a similar observation is made in @xcite , further supporting the viability of this approach. to construct an effective decision tree for annotation, it is essential to design questions that do not require expert-level knowledge of discourse but can instead be answered based on the utterance itself. ideally, related classes should be positioned closely within the tree, forming a hierarchical structure that reflects conceptual similarities between coherence relations. this hierarchical organization not only simplifies decision-making for annotators but also enhances consistency and reliability in annotation. tree construction the pipeline for the tree construction process is illustrated in figure 2 . the core concept of the proposed algorithm is to use an advanced llm to identify distinguishing features that allow it to divide a set of classes into two or more groups. here, a group refers to a subset of the input classes that the model clusters together based on a shared property it identifiesframed through a classification question. for example, given the classes open.demand.fact (requesting factual information at the beginning of a conversation), open.demand.opinion (requesting an opinion at the beginning), open.give.fact (providing factual information at the beginning), and open.give.opinion (providing an opinion at the beginning), the model might divide them as group 1: open.demand.fact, open.demand.opinion, and group 2: open.give.fact, open.give.opinion. to do this, the llm is provided with a table containing class names, definitions, and usage examples as input. to enhance reasoning capabilities, we aim for the model to engage in an inner monologue @xcite before determining how to split the data. to achieve this, the llm is prompted to first generate a set of questions about an utterance that can aid its understanding. specifically, the model is instructed to formulate and answer three such reasoning question-answer pairs at the beginning of its response. recent studies indicate that employing such self-questioning techniques helps models produce more flexible, meaningful outputs, indicating a deeper level of comprehension and reasoning @xcite . an example of the llm's reasoning output can be found in appendix f. after this reasoning step, the llm generates a classification question to determine an utterance's group, providing possible answers mapped to class groups. for example, the model might generate a classification question: "is this the beginning of a conversation?" the possible answers (groups) could be: (1) "yes, this is the beginning of a conversation." and ( 2 what's allenby doing these days? ## tree construction approaches yes/no questions vs. open-ended questions the manually created tree in @xcite is binary and composed exclusively of yes/no questions. as the first approach explored in our work, we implement a binary yes/no question tree using the proposed framework. additionally, we construct a binary tree with open-ended questions. the hypothesis behind this experiment is that allowing the model to generate any questions, rather than limiting it to yes/no questions, can result in a more flexible and nuanced tree for discourse annotation. the prompts used for constructing both the yes/no questions and the open-ended questions tree were developed through iterative refinement. this refinement process involved adjusting and testing prompts until they produced stable and consistent results, ensuring a valid json output with all necessary keys present. the final versions of these prompts are provided in appendix i and appendix j, respectively. binary vs. non-binary trees allowing the model to split nodes into multiple groups, rather than restricting it to two groups, has the potential to create a more granular and detailed annotation tree. however, this added granularity may introduce greater complexity and reduce consistency in the annotation process. this experiment is distinct from the previous one, where we only allowed binary open-ended questions and compared them to yes/no questions. as a result, both the questions and answers in that setting were more high-level, distinguishing data based on a single characteristic. in contrast, non-binary open-ended questions allow for greater granularity and specificity, enabling more nuanced differentiation within the data. experiments described in this section examine the impact of restricting the model to binary splits compared to allowing multiple-group splits on the accuracy of the resulting annotation tree. both the binary and non-binary trees discussed in this section use open-ended questions. a prompt that allows open-ended questions and splitting data into more than two groups is detailed in appendix k. ## results & analysis evaluation results for the top level, the top two levels, and top-20 class subsets on the development set are presented in tables 1, 2 and 3 while the results for the complete sf taxonomy on the development and test sets are shown in tables 4 and 5 , respectively. 3 yes/no questions vs. open-ended questions the findings indicate that open-ended trees outperform yes/no trees across all metrics and data subsets. however, for the complete sf taxonomy, ## open-source models for annotation these results indicate that despite having fewer parameters than the closed-source models, both open-source models notably outperform gpt-3.5. specifically, when using the non-frequency-guided approach, both open-source models achieve performance close to gpt-4o while markedly surpassing gpt-3.5. however, with the frequency-guided approach, llama continues to outperform gpt-3.5 noticeably, but both models' performance metrics fall short of gpt-4o. this trend underscores the ## evaluation on switchboard dialog act corpus to assess the generalizability of our approach, we evaluate two configurations on the swbd-damsl annotation scheme (42 classes). the configurations are: ## conclusions we conducted experiments on generating tree schemes for discourse taxonomies using llms. this paper proposes a framework that supports the entire pipeline, from tree construction to dialog annotation. our configuration with frequency-guided tree creation demonstrates that using llms for both tree scheme generation and annotation can yield results that surpass manual tree construction and crowdsourced annotation while significantly reducing the time required for the entire process.