Training LLM s to Recognize Hedges in Dialogues about Roadrunner Cartoons (2024)

Abstract: Hedges allow speakers to mark utterances as provisional, whether to signal non-prototypicality or “fuzziness”, to indicate a lack of commitment to an utterance, to attribute responsibility for a statement to someone else, to invite input from a partner, or to soften critical feedback in the service of face management needs. Here we focus on hedges in an experimentally parameterized corpus of 63 Roadrunner cartoon narratives spontaneously produced from memory by 21 speakers for co-present addressees, transcribed to text (Galati and Brennan, 2010). We created a gold standard of hedges annotated by human coders (the Roadrunner-Hedge corpus) and compared three LLM-based approaches for hedge detection: fine-tuning BERT, and zero and few-shot prompting with GPT-4o and LLaMA-3. The best-performing approach was a fine-tuned BERT model, followed by few-shot GPT-4o. After an error analysis on the top performing approaches, we used an LLM-in-the-Loop approach to improve the gold standard coding, as well as to highlight cases in which hedges are ambiguous in linguistically interesting ways that will guide future research. This is the first step in our research program to train LLMs to interpret and generate collateral signals appropriately and meaningfully in conversation.

## introduction the virtuosity of llms such as chatgpt has led some to the impression that ai already converses (or will soon be able to converse) as people do. but as language users, llms and humans are quite different. the underlying foundations for learning by these distinct kinds of language users share little in common: humans learn as infants to interact with others well before they learn their first words, and once word learning begins, they can pick up a new word in one or just a few exposures, whereas llms are pre-trained on humanly unfathomable quantities of text without ever learning to inter-act. transformer-based chat programs can generate paragraphs-worth of text remarkably well without modeling the coordination between agents-but is this conversation? whether a sequence of prompts and responses exchanged in a dialogue between an llm agent and a human counts as truly (rather than superficially) "conversational" depends on how conversation is conceptualized. conversation is often presumed to be the passing back and forth of messages (a "message model"); but that does not explain phenomena common to spontaneous conversation such as incremental turns, clarifications, and repair. here we conceptualize conversation as a collaborative process of grounding meanings (seeking and providing evidence) during which two or more partners signal, coordinate, and align their beliefs or cognitive states @xcite . this leads to a broader research agenda that we hope will push generative ai to model phenomena such as a partner's knowledge or theory of mind, mutual beliefs or common ground, as well as when to take initiative in a dialogue. the main contributions of this work include: (i) after grounding the project in psycholinguistic theory (section 2) and related work (section 3), we present the roadrunner-hedge corpus (section 4), a corpus of spontaneous face-to-face narratives annotated for hedging. 1 (ii) we describe a set of experiments on this corpus using zero-shot, few-shot, and fine-tuning methods on modern llms (section 5). (iii) we perform a detailed error analysis pinpointing where llms fail in detecting hedges @xcite . with this analysis, we take an llm-inthe-loop approach to correcting gold annotations, reducing errors in our top performing systems. we conclude with a discussion and implications of our results in section 7, limitations and the future of our work in section 8, and a final summary of our salient contributions in section 9. ## theoretical foundations from psycholinguistics in conversation, people communicate not only about the purpose or topic at hand, but they also communicate meta-information about what they're saying within the context of interaction, or collateral signals @xcite . along with providing evidence for grounding in conversation, about whether a prior turn has been understood as intended @xcite , collateral signals can also provide information about the speaker's relationship with the content of their message-how confident they are in what they are saying, whether it is difficult to recall or express, and whether they would welcome input from their partner. in this project, we focus on a particular kind of collateral signal used for coordination, hedges. ## the roadrunner-hedge corpus for training and testing, we obtained a corpus @xcite of spontaneous narratives produced from memory by 20 speakers who had watched a roadrunner cartoon. each speaker narrated the story face-to-face to an audience, a total of three times: first to a naïve addressee, a second time to the same addressee, and a third time to a new naïve addressee (with the latter two episodes counterbalanced for order). the original experiment was designed to detect differences in collateral signals (intelligibility vs. attenuation of speech and gestures) stemming from the speaker's vs. the addressee's knowledge states-that is, whether the story was new for the speaker (told for the first time) vs. old (retold), compared to the addressee's knowledge state (new vs. heard for the second time). findings included that the attenuation of both referring expressions @xcite and gestures @xcite were driven by both speakers' and addressees' knowledge states-that is, shortened upon retelling the story to the same addressee, but lengthened upon retelling to a new addressee. gold standard coding. the original corpus transcribed the spontaneous narratives in detail, including speaking turns and disfluencies (for details, see @xcite , segmented into lines by installments that corresponded to narrative elementss in the cartoons. we annotated hedges on the original roadrunner corpus to create the gold standard for hedge training and detection (the roadrunner-hedge corpus; see https: //github.com/cogstates/hedging for the annotation codebook). the roadrunner-hedge corpus is distributed as a csv file. it is structured as a total of 5,508 lines, over a quarter of which (@xmath0. the first author annotated hedges in the corpus as in inter-rater reliability. to compute inter-rater reliability, a trained research assistant coded 7 randomly-selected transcripts with no overlapping speakers (10% of the corpus). we calculated cohen's kappa from each word marked as a hedge within each transcript. there was high agreement between coders, with κ = 0.985. ## error analysis while the fine-tuned bert model performed fairly well, a certain number of cases did not align with the gold labels in the data. we performed error analysis to understand whether there were any systematic deviations from the corpus annotation. we conducted an error analysis on the top two performing models, the fine-tuned bert model and the gpt-4o few-shot list (fsl) model (@xmath1.91 and 0.71, respectively). starting with the first fold, we selected the first hundred errors to categorize. these errors are broadly divided into instances where the models failed to detect a hedge (false negatives) and instances where models returned cases that were not annotated hedges (false positives). the remaining errors fell into two other categories: a gold error category, wherein errors in the (human) annotation were discovered, and an "other" category. of the hundred errors sampled from the bert model, approximately the same number of errors were false negatives (26) as false positives (29). of the hundred errors sampled from the gpt-4o fsl model, 66 were false positives and 25 were false negatives (reflecting the low precision and higher recall for this approach; see although the corpus annotation does not include the type of hedge (only the presence or absence of hedge tokens), our error analysis looked at hedge types in order to tease apart model behaviors. we observed systematic differences between models in their types of mismatches with the gold standard. false positives. first, the gpt-4o fsl model inaccurately classified disfluencies (e.g., "uh") as hedges in 37 of the 66 false positives reviewed, whereas bert did not. second, bert showed quite a different pattern of mismatches than gpt-4o when classifying "like", returning false positives that always turned out to be comparatives (e.g., "it's like an open elevator"). these we considered to be true errors in their text form, although some may be ambiguities that could be resolved prosodically. false negatives. tokens denoting approximator hedges (e.g. "that's basically it") were frequently misclassified as false negatives by bert (9 of 26 false negatives reviewed), but never by the gpt-4o fsl model. in addition, other emerged as a category type for situations that could not clearly be described as false positives, false negatives, or gold errors. in the bert model, these cases were typically segmentation errors (i.e., an inner token mislabeled as a beginning token). ## discussion the results show that even enormous, recently released llms cannot reliably recognize hedges. there is no "emergent" ability in llms to understand full human linguistic behavior. on the other hand, when we explicitly train a small, rather old llm (bert) to perform our task by fine-tuning it, it performs quite well. what this shows is that detecting hedges is a capability that can be learned, but it cannot be learned in the manner that llms are taught, namely by simply ingesting large amounts of varied data. we interpret this to mean that if we want to make llms able to converse with humans as humans do, we need to understand what capabilities llms need and how to provide them with the ability to do so. the prevalence of gold errors discovered by the bert model raises two interesting points for discussion. first, some of these discrepancies identified by the bert model were clearly errors made by the human coders; this was true in particular for proxies, which bert coded for hedges more consistently than did human coders. this error analysis allowed us to iteratively improve the human coding before the final analysis, essentially deploying an llm-in-the-loop approach. second, the discrepancies between bert and gold coding on the tokens just and like highlight that these types of hedges have high potential for ambiguity-perhaps the very sort of ambiguity that could be resolved by prosody. ## limitations and future work this work represents the first step in our research program that aims to train llms to use collateral signals in support of human-llm dialogue. once hedges can be recognized by an llm, it remains to be shown that they can be meaningfully interpreted and generated. relevant work by cassell and col-model original gold f1 llm-in-the-loop gold f1 error reduction (%) bert 0.908 ± 0.010 0.925 ± 0.019 18.5% gpt-4o few-shot 0.712 ± 0.021 0.721 ± 0.020 3.1% gpt-4o zero-shot 0.510 ± 0.028 0.551 ± 0.011 8.4% ## conclusion our project is grounded in psycholinguistic theory and aims to capture theory-of-mind aspects of hedging among discourse participants. we present the roadrunner-hedge corpus, with hedges annotated from naturally occurring dialogues by speakers describing roadrunner cartoons. we use the corpus to train and perform experiments on detecting hedges using bert, gpt-4o, and llama-3. we find that fine-tuning bert significantly outperforms state-of-the-art llms in few-shot and zero-shot settings. with our systems outputs, we perform an error analysis and use an llm-in-the-loop approach to correct gold standard annotations. our llm-in-the-loop approach provided further error reductions on all models.