Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversations (2023)

Abstract: When we communicate with other humans, we do not simply generate a sequence of words. Rather, we use our cognitive state (beliefs, desires, intentions) and our model of the audience’s cognitive state to create utterances that affect the audience’s cognitive state in the intended manner. An important part of cognitive state is the common ground, which is the content the speaker believes, and the speaker believes the audience believes, and so on. While much attention has been paid to common ground in cognitive science, there has not been much work in natural language processing. In this paper, we introduce a new annotation and corpus to capture common ground. We then describe some initial experiments extracting propositions from dialog and tracking their status in the common ground from the perspective of each speaker.

## introduction expressions such as "finding common ground" or "establishing common ground" are familiar to everyone. they are often used with reference to two or more people trying to reach some sort of mutual understanding about various topics; for example, politicians during a parliamentary meeting. most often, however, common ground does not involve a conscious effort. establishing and updating common ground (cg) between interlocutors is the key to a successful conversation. it is therefore crucial to develop methods of cg representation and learning. this work focuses on modelling the concept of common ground from the cognitive perspective. specifically, we introduce the first (to the best of our knowledge) corpus annotated for common ground. the procedure developed for the annotation is designed such that it reflects what we think happens when people update their common grounds. we further conduct preliminary machine learning experiments focused on extracting events subject to cg updates, and on predicting the cg updates from the perspective of the two speakers in a dialog. the concept of common ground has been widely studied across various disciplines including linguistics and cognitive science @xcite @xcite , computer science, artificial intelligence, and natural language processing @xcite @xcite , and philosophy @xcite . cg can be described in simple terms as a set of beliefs which interlocutors believe they share, and which they believe other discourse participants also believe they share. however, when people communicate, they do not only exchange their beliefs, but also other components of cognitive state, such as desires, intentions, goals, etc. consequently, the full definition of cg should also contain sets of shared desires, intentions, and so on. because this is the first corpus created for that purpose, we focus solely on the belief component of cg. we do acknowledge, however, that future work will need to extend the notion of cg. this paper is organized as follows. we summarize the nlp literature in section 2. we present our new corpus in section 3. in sections 4, 5, and 6 we discuss our baseline experiments for predicting the events, beliefs about events, and achieving common ground. ## related work the concepts of belief and factivity are closely related, see @xcite , section 2) for a discussion. they have been annotated in multiple language corpora, such as lu @xcite , factbank @xcite , uw @xcite , ldc @xcite , megaveridicality @xcite , @xcite , rp @xcite , and best @xcite , among others. while all of those datasets differ in terms of identification of events and the annotation procedure, they all target one common goal. given some text, they identify the level of commitment to the events expressed in the text, whether that be from the perspective of the author solely (e.g. lu, megaveridicality, rp, uw) or from the author along with other mentioned sources @xcite . what is considered an event is usually tailored to the specific question that a corpus is addressing. for example, cb targets solely final clausal complements as it examines the role of the matrix predicate in determining the speaker's belief, while lu and ldc cb consider all full lexical verbs in a sentence, and factbank and best target verbs and nouns referring to events and states. the genre of the data is also correlated with the task. factback uses wsj newswire, and as a result the author is limited in the expressiveness of their cognitive state, as news-writing requires following certain standards. rp, lu, and cb, on the other hand, use mixed genres, including discussion forums and blogs, which allow for more creative and expressive freedom. all of these corpora also exhibit various scales identifying the level of belief. those are either represented numerically, averaging over annotators' judgements, on a [-3,3] scale (uw, cb, rp), or with categorical labels (lu, factbank, ldc cb, best). one last crucial factor affecting the outcome of annotation are the annotators themselves. rp and uw use crowdsourced annotators, while factbank, lu, lcd cb, and best opt for trained annotators. belief/factivity prediction tasks have attracted a lot of attention in the past two decades. in the early 2000s, rule-based systems were used to detect authors' beliefs @xcite . @xcite , @xcite , and @xcite use svms together with dependency trees and lexical based features. newer, neural network approaches include bidirectional lstms for both single and multi-task setups @xcite use bert representations with graph convolutional neural networks. jiang and de marneffe (2021) use fine-tuned bert for belief detection on various corpora and closely examine shortcomings of the model. @xcite show that combination of specific corpora can improve the model's performance. cg is closely related to theory of mind (tom), a concept first discussed in @xcite , which refers to the capacity in humans @xcite , as well as chimpanzees and orangutans @xcite , to infer information about others' mental states including beliefs, desires, and emotions. cg is an aspect of tom, since beliefs in the cg are beliefs a speaker models as being held by the addressee. this means that humans (as well as other primates) are able to infer some "hidden" information about what other people think, believe, and perceive in particular situations. a natural question for nlp is whether contemporary advanced language models exhibit knowledge of tom. furthermore, if not, what are some possible approaches that will allow machines to infer information about their conversational partner's cognitive states? @xcite tested current large language models (llms) by having them complete narratives inspired by the sally-anne @xcite and smarties @xcite tests, both of which were originally designed to determine if children are aware of the cognitive state of others. he focused specifically on one subcomponent of the tom scale @xcite , namely false belief. the results found that the most advanced gpt models correctly complete both true and false beliefs for characters in various scenarios. for example, if there is a bag of popcorn with a "chocolate" label on it, the models accurately predict that when sam finds the bag she will believe that the bag is full of chocolate. kosinksi then concludes that this is either (1) evidence of the inadequacy of these tests for evaluating tom or (2) evidence of tom emerging as a byproduct of learning on different tasks. @xcite disagrees with this hypothesis and states that it is indeed possible that tom tests are in fact accurate and useful for studying human cognition but not llms. his claim is backed by a series of tom experiments, slightly different from the ones in @xcite , that show that llms fail to make correct false-belief predictions. for example, in the popcorn-chocolate bag scenario they added additional information stating that sam cannot read. gpt 3.5 still guessed with 98% confidence that sam believes the bag is full of chocolate. similarly, @xcite develop a dataset of "mind games" using dynamic epistemic logic and target aspects of tom to show that contemporary llms fail to make correct inferences in other scenarios as well. the experiments conducted by @xcite , @xcite lernould (2023), and ullman (2023) show that while llms can produce accurate completions for false-belief in some settings, the results are highly sensitive to perturbation and far from robust. this on its own should be sufficient evidence that tom cannot possibly be an emergent property of llm learning, at least not yet, and points to a need for higher quality, naturalistic data on how humans reason about aspects of tom, like cg. ## the cg corpus the common ground corpus is annotated on the top of the ldc callhome american speech corpus @xcite , which consists of collections of 120 unscripted dialogs between close friends or family members. the dialogs are available in both written form and audio. since our dataset is the first attempt at annotating cg in a discourse, we chose to start with conversations between two people. another crucial aspect of the callhome corpus is that the interlocutors know each other very well, which allows us to assume that their cg set is non-empty at the start of the conversation. finally, the speakers were allowed to speak freely about any topic, which allows us to capture the speakers' cognitive states in natural settings. in a dialog, each of the interlocutors form their own cognitive states, which includes a model of other speakers' cognitive state. since cg forms a substantial component of a speaker's cognitive state, we model cg for each speaker separately: the cg is not shared. in the majority of cases, the cgs of the two interlocutors converge. however, we also see scenarios in which the cgs for speakers diverge and in which we observe miscommunications. the two main parts of the annotation procedure are event extraction, and belief and cg annotation. ## experiments: events for generating events, we use the base version of flan-t5, which is an instruction-finetuned version of the t5 llm @xcite . it has demonstrated impressive few-shot performance across various tasks, outperforming even larger models. in our experiments, the model receives utterances as input and it generates the corresponding event(s) for each utterance. we have fine-tuned flan-t5 on the training set, which is a small dataset (see alongside the input utterance, the model can also receive contextual information as input. in this paper, the contextual information is provided as the following modes: 1) fixed window: a predetermined number of utterances preceding and/or following the target utterances or events will be included as input for the model. and 2) speakerbased window: the model will receive all preceding and/or following utterances until it encounters an utterance from a speaker different from the speaker in the target utterance or event. the input format of the model is as follows: ## experiments: beliefs for belief classification, we designed a comprehensive set of models utilizing bert-base-cased @xcite and flan-t5. the system is trained to annotate beliefs for each speaker (e.g. bel(a)=ct+) given a gold target event and any additional context. as with event extraction, we experiment with both fixed and speaker-based windowing methods and additionally investigate including forward context. in these experiments, utterances are presented to the model either as raw sentences or as the corresponding events of those utterances. we fine-tune bert and flan-t5 on the training set using the following input format: "preceding context": {preceding events or utterances} ↩→ "target event": {target event} "following context": {following events or utterances} ↩→ ## future work there are many obvious extensions of our work to date. we list a few. audio modality in our modeling, we only use the callhome transcripts. however, intonation in english is related to what the speaker believes the hearers already know. we will incorporate the audio signal into future cg predictions,and will explore multimodal neural architectures. existing belief corpora and systems we intend to incorporate existing belief corpora and systems into the belief prediction task, which could improve the system performance. specifically, we intend to use the state-of-the-art factbank-only belief system from @xcite as a belief tagger, and give those outputs to our system. we also intend to experiment with a multi-task learning (mtl) setup, where we jointly model belief and common ground updates. graph approaches our data can naturally be represented as a graph structure where speaker a and b's utterances are nodes, and the changes in belief and common ground being edge labels. we intend to leverage graph neural network approaches inspired by neural architecture and general insights from @xcite , who model the emotion recognition in conversation (erc) task. higher-order beliefs we explicitly model firstorder beliefs and the cg. many higher-order beliefs follow from the combination of individual belief and cg -for example, if a says that mary is coming to dinner but b rejects it, it does not enter a's model of the cg. as a result, we can infer a higherorder belief, namely that a believes b does not believe the event (since if b believed it, it would be in the cg). however, there may be cases in which higher-order beliefs cannot be inferred from the cg. we intend to annotate and model higher-order beliefs in dialog, i.e. beliefs held about other people's belief which are not derivable from cg. error analysis finally, we intend to conduct a more detailed error analysis that could give us more insight to specific issues that need to be addressed in future work.