The Role of Common Ground for Referential Expressions in Social Dialogues (2022)

Abstract: In this paper, we frame the problem of co-reference resolution in dialogue as a dynamic social process in which mentions to people previously known and newly introduced are mixed when people know each other well. We restructured an existing data set for the Friends sitcom as a coreference task that evolves over time, where close friends make reference to other people either part of their common ground (inner circle) or not (outer circle). We expect that awareness of common ground is key in social dialogue in order to resolve references to the inner social circle, whereas local contextual information plays a more important role for outer circle mentions. Our analysis of these references confirms that there are differences in naming and introducing these people. We also experimented with the SpanBERT coreference system with and without fine-tuning to measure whether preceding discourse contexts matter for resolving inner and outer circle mentions. Our results show that more inner circle mentions lead to a decrease in model performance, and that fine-tuning on preceding contexts reduces false negatives for both inner and outer circle mentions but increases the false positives as well, showing that the models overfit on these contexts.

## introduction people that have a long-term relationship develop an effective way of communication which also targets the relationship as such. we call such conversations "social dialogues" and we expect that common ground plays an important role and has an impact in the way reference is made. as two conversation partners develop a closer bond, they form conventions in how they refer to individuals that are often part of their shared experiences, and these references may become vague and ambiguous to others @xcite . how present and important a particular individual is within the common ground not only influences the ambiguity of the references used, it also influences how readily they can be introduced into the conversation. while less popular or newly introduced individuals (outer circle) need a more elaborate and explicit reference, well-known individuals (inner circle) can be referenced with their name or a short and vague description, which may be difficult to infer for outsiders. for instance, someone's grandmother could be brought up with the reference 'nana'. we would thus expect to see a difference between less important or unknown individuals on the one hand and important individuals on the other hand with respect to their co-reference chains. when agents become more and more part of our lives, we can expect that they also build up long-term relationships with us, just like people do. an agent that needs to engage in social conversation with a human should therefore be sensitive to these changes in the way reference is made as the common ground grows. some parts of this common ground, such as observations within the visual scene, can be established based on the context of the shared environment @xcite . however, references to individuals in their shared experience (i.e. individuals that have been mentioned before) belong to the common ground that is based on the more long-term shared world context that needs to be retained across interactions, and which can also change over time. in this paper, we report on a first analysis of how inner and outer circle people are referred to in social dialogues in which common ground plays a role and we test how sensitive existing co-reference resolution models are to this. to test this, we use a data set consisting of episodes of the friends tv series that has been annotated with mentions of individuals @xcite . this data set contains both social dialogue and long-term connections between mentions that go above the level of a single document, and therefore it could serve as a useful simulation of the buildup of common ground over time. our main contributions are: 1) we frame the problem of resolving co-reference in dialogues as a dynamic process in which common ground plays a role, introducing the concepts of inner and outer circle references, 2) we provide new insight into the way inner and outer circle references are made by "friends" with a lot of common ground, 3) we test the sensitivity of machine learning models to (long-term) common ground in dialogue, 4) we restructure an existing data set of social dialogue in such a way that the existing temporal and topical relations between the conversations are maintained, which can be used for investigating the buildup of common ground and the development of conventions in referencing. this paper is structured as follows: in section 2 we discuss related work, and present our motivation and problem statement. in section 3 we analyze the data on the difference in mention patterns for well-known and lesser-known individuals, and we discuss our approach to testing model performance with respect to references with common ground, and in section 4 we present the results of our tests and perform some error analysis. finally, in section 5 we interpret our results and link them to the broader question of achieving common ground in human-agent communication. ## related work common ground is what we call the established shared information that speakers rely on within a conversation @xcite . it is essential to successful communication. consequently, an agent that communicates with a human also needs to establish common ground to overcome mismatched representations of the world @xcite . in a process called grounding, this common ground also needs to be continually updated @xcite . various research has been done on grounding in human agent-interaction, for instance on grounding in the visual scene in relation to tasks @xcite @xcite . agents which develop this common ground have been shown to perform better on well-known tasks and also adapt better to new tasks @xcite . however, in these task-oriented dialogues, the references tend to be explicit and refer to objects in the shared environment. in social dialogues, which are open, not task-oriented and between people that established a long social relationship, references become more vague quickly, and also refer to objects or individuals which are not present, but part of past (shared) experiences. this makes the references harder to latch on to, and means the agent must relate them to background knowledge rather than to what it sees in front of him. the more interactions the agent has had with a particular human, the more shared experiences and background knowledge it can potentially rely on. however, in natural dialogue, as the number of shared experiences increases, the references become also more conventionalized, and as a result, more ambiguous to outsiders. researchers have shown in simulations and experiments in humanhuman communication how this conventionalization occurs, and how it leads to more efficient but also more vague expressions over time that nonetheless remain understandable for the conversation partners who share the common ground @xcite @xcite . however, these simulations have been limited to task-oriented dialogue. we add to this research an analysis of social dialogues between humans. existing end-to-end coreference resolution models can achieve high scores on well-established datasets such as ontonotes @xcite . however, these datasets do not relate well to our use-case of social dialogue. they consist of snippets of text from news articles, telephone conversations or talkshows, often more formal in nature, which likely makes the references more explicit. most importantly, though, since the data does not contain a temporal aspect in which the various documents relate to each other, it cannot be used to examine how common ground builds up over time and how a model could utilise that common ground. the codi-crac shared task @xcite is aimed at improving coreference resolution performance in dialogue. they also provide a selection of data sets consisting of dialogue, such as the switchboard corpus @xcite and the persuasion for good dataset @xcite . however, these data sets are not ideal for our case either. although they do consist of (social) dialogue, the conversations are between speakers who were previously unacquainted, and who do not share a common background which can be built upon in the conversation. this is a crucial part of the phenomenon that we aim to investigate. therefore, we take an existing dataset containing social dialogue and temporal relations between documents (choi and chen, 2018) and adapt it to analyze the differences in referencing of inner-circle mentions, which are part of the common ground, and outer circle mentions, which are only relevant within the surrounding context. we also test to what extent a state-of-the-art end-to-end coreference resolution model utilizes background knowledge and how it resolves complex third-person references. we hypothesize that the model will perform worse on references that require common ground. a model failing to detect a vague introduction for an otherwise well-known individual could also have problems further on in the conversation, as third-person pronouns referring to this individual are instead linked to a different individual. concretely, this means that the higher the amount of inner circle references in the test set, the lower the overall model performance will be for that set. we further investigate whether this performance can be improved by increasing the background knowledge by training on preceding interactions. we believe that it is valuable to examine common ground buildup over time in the context of coreference resolution. to the best of our knowledge, this is a new approach to the problem of coreference resolution. in the next section, we describe how we created the dataset and how we tested the model. ## method for our experiment, we take the current state-of-theart model in co-reference resolution, spanbert @xcite . we use an implementation of this model by @xcite . this model predicts co-reference chains by calculating scores for pairs of mention and antecedent span representations which have been contextualized using bert. we test this model in two ways. first, we run the pre-trained model on the new data set without fine-tuning. we do this to examine what performance the model can already achieve without knowing anything about the background knowledge except for what may have been learned from public sources such as wikipedia during pretraining. next, we fine-tune the model on data of previous conversations that likely contain common ground information and discourse contexts specific to sitcom characters. we fine-tune three models, one on a small, one on a medium and on a large portion of the previous conversations (see ## discussion our results showed differences between models and across test sets with different ratios. all models perform lower on s2e14 with more inner circle references than on s2e24 with less. for most metrics, the pretrained model performed best on s2e14 (4/1) and the largest fine-tuned model on s2e24 (1/1) ratio, except for entity-based evaluations in which pretrained performed best on both. when breaking down the errors per part-of-speech and across inner and outer references, we found some patterns but it remains difficult to relate these to the different part-of-speech statistics observed. many errors are made for outer circle names and in the case of s2e24 also for inner circle name mentions. remarkable are the false positive (and to some extent false negative) errors in pronominal inner and outer circle mentions in s2e24. since the false positives tend to increase with fine-tuning, we suspect that the fine-tuned models are over-fitting. our experiments do not allow us to draw conclusions towards the potential of more knowledge-rich approaches that incorporate built-up common ground. this is partly because fine-tuned language models are not transparent to what knowledge is picked up from the preceding conversations. clearly, more research on the role of common ground in referencing in social dialogue is necessary. most co-reference resolution models continue to be trained and tested on well-established data sets which are not useful for exploring this phenomenon. although the data set of episodes of friends that we used in this paper has the necessary properties, it too has its drawbacks. most of the dialogues are multi-party dialogues, whereas dyadic dialogue would be a more controlled setting in which to explore the buildup of common ground. the dialogues also partly rely on visual cues, which the model cannot rely on and for which the necessary metadata is not provided in the data set. furthermore, the show is a sitcom, and the many quips might have a detrimental effect on the naturalness of the conversations. therefore, we encourage the further development of more data sets of social dialogue with multiple interactions over time, based on a more natural setting or with fewer speakers involved in the conversation. in this work, we have made a distinction between well-known 'inner circle' and lesser known 'outer circle' referents. we believe it is relevant to be aware of such a distinction in referencing, since people rely on the established references to the inner circle to create a bond and distinguish their shared social circle from the outside world. if we want systems to become a part of this shared social circle and develop their own bond with humans, they too need to learn this way of referencing, and in long-term interaction it could help them reinforce this bond and improve communicative efficiency and enjoyment on the part of the human. in future work, we will further explore how the buildup of common ground influences referential expressions to well-known individuals over time in dyadic social dialogue. this will be done in an interactive setting, where an artificial agent engages in conversation with a human and can use visual cues and human feedback to improve its representation of the common ground. due to the interactive nature of the dialogue, the model will not be a pure co-reference resolution model, but it will build upon properties of both co-reference resolution and entity linking models. in addition, we will use a more explicit modeling of common ground, and include more knowledge-rich features in our model. ## conclusion in this paper, we framed the problem of resolving third-person references in social dialogues as a dynamic process in which common ground plays a role. we made a difference between inner and outer circle references and hypothesized that the former are more difficult to resolve, which was partially confirmed by the model performances on data with more and less inner circle references. training models on preceding data did not show a corresponding increase in performance on inner circle references, indicating that such models do not acquire common ground knowledge, but did improve the performance for outer circle mentions. we propose that co-reference resolution models for social dialogue could benefit from a more knowledge-rich approach in order to better adjust to the common ground, which in turn facilitates the resolution of complex third-person references.