Referring Expressions in Human-Robot Common Ground: A Thesis Proposal (2024)

Abstract: In this PhD, we investigate the processes through which common ground shapes the pragmatic use of referring expressions in Human-Robot Interaction. A central point in our investigation is the interplay between a growing common ground and changes in the surrounding context, which can create ambiguity, variation and the need for pragmatic interpretations. We outline three objectives that define the scope of our work: 1) obtaining data with common ground interactions, 2) examining reference-making, and 3) evaluating the robot interlocutor. We use datasets as well as a novel interactive experimental framework to investigate the linguistic processes involved in shaping referring expressions. We also design an interactive robot model, which models these linguistic processes and can use pragmatic inference to resolve referring expressions. With this work, we contribute to existing work in HRI, reference resolution and the study of common ground.

## introduction while there has been a huge leap in conversational ai in recent years, innovations in multi-modal, situated conversational ai have not seen similar progress. one area which especially deserves attention is situated common ground in human-robot interaction (hri). understanding how common ground and conventions play a role in the use of referring expressions in hri can help create more efficient, enjoyable and successful communication. a robot that does not build up common ground and learn the conventions may have difficulty identifying the referent of a referring expression, leading to confusion and errors. in human conversation, there is an implicit drive to be only as informative as necessary @xcite . this leads to pragmatic behavior in humanhuman conversation, and explains why seemingly t1 t2 t3 guy with blue beard blue beard blue figure 1 : schematic representation of how common ground can shape referring expressions within a changing context. over time, the human and robot form a convention for the entity in the red box. this is associated with a reduction in the utterance length leading to underspecified language. the use and interpretation of this convention can remain consistent even if ambiguous information is introduced, such as the entity in the black box at t3. for instance, consider the following scenario: two close friends, anna (a) and bob (b), frequently meet up at a bar in the centre of town. one of the bartenders there has a distinctive blue beard and a strange personality. anna and bob do not know his name, but often joke about his antics. when talking about him, they call him blue for his beard. the referring expression blue provides enough information to a and b, because they share a common ground due to their situational grounding and a history of previous exchanges at the bar @xcite . the more common ground has been built up, the more information can be left implicit, which enables more efficient communication. this is especially true for referring expressions such as blue, which are the result of a process of conventionformation. a key point in conventions is that they are stable @xcite : a and b could continue using the name blue with each other even when blue changes his beard. pragmatic interpretation of the convention allows a and b to use this referring expression when there is conflicting information, such as another individual with a blue beard. while the effects of common ground on communication have been examined in detail for humanhuman communication, less is known about its role in hri. therefore, this dissertation examines the impact of common ground on the pragmatic use of referring expressions within hri. the pragmatic behaviour is analysed through linguistic, contextual and social factors such as patterns of reference, ambiguity, and convention formation. we model these factors in a multi-modal interactive robot equipped with pragmatic reasoning capabilities, which allows us to assess which factors contribute the most to the use and interpretation of referring expressions in hri. ## background referring expressions are studied within nlp in coreference resolution and entity linking (el) tasks. although there are similarities between the tasks, they have distinct goals, and separate models exist for either task @xcite . the problem we are investigating in this research draws important elements from both tasks, but actually establishes a new research space by combining and expanding on them. on top of linking entities and clustering them within a dialogue, in our work references should be understood in the broader context of the common ground which is built up over multiple interactions. furthermore, we examine the interpretation of references within a situated, multi-modal context rather than the unimodal data that are used in coreference resolution and el. we also examine the production of references as well as their interpretation. in 2024, both downstream tasks could be performed by large language models (llms). however, there could still be issues when applied in a situated multi-modal environment, as llms are still mostly unimodal and not situated. further-more, while llms have been shown to be capable of pragmatic inference to some extent @xcite , fine-tuning is still required to get desirable results @xcite . iterated reference games such as the tangram task @xcite and the photobook task @xcite have studied how spontaneous linguistic conventions form as a result of common ground. these games simulate common ground by invoking repeated references to the same image or figure over a number of rounds. however, common ground is analysed in a static environment rather than within a changing context. in the dynamic onecommon task by @xcite , contexts do change, but convention-formation is not an aspect in this task. all tasks mentioned above are performed in human-human interaction. in hri, the role of conventions and common ground has been studied for gent policies and strategies @xcite , rather than for natural language understanding and generation. ## research goals and questions the main goal of this research project is to better understand the processes through which common ground shapes referring expressions within human-robot interaction. our main research question is: rq to what extent does common ground influence the pragmatic use of referring expressions in human-robot interaction? by simulating the advancement of common ground while changes occur in the surrounding context, we aim to examine how common ground impacts the use of referring expressions and their interpretation within the context. we outline three objectives that need to be tackled in order to answer our research question: ## conclusion this thesis proposal outlines the data that needs to be collected, and the linguistic processes that need to be examined and modeled to understand the role of common ground in shaping referring expressions in human-robot interaction. the findings of this work can be used to design social robots which can sustain meaningful and enjoyable long-term interaction with humans. next to this, the findings obtained for human-robot interaction can also inform us about how common ground influences communication between humans. the thesis will also include an assessment of future steps that need to be taken to further improve social robots.