Processing Referential Ambiguities in Situated Dialogue Systems (2023)

Abstract: Position paper for YRRSDS 2023

## research interests conversations between humans are based on the collection of mutual knowledge, experiences, beliefs, assumptions and even goals of the interlocutors. we estimate these unconsciously, but they attribute meaning outside words that is crucial to understanding the interaction. this common grounding is what connects the meaning of the physical world with our language abstractions @xcite ). however, current intelligent systems do not share this mutual understanding and common grounding, heavily impairing the interaction. users have to simplify queries, be more explicit and repeat information already mentioned so that their language matches the way that dialogue systems communicate. as previous works have demonstrated, dialogue models trained on large scale datasets are not able to capture meaning beyond words (symbols) @xcite @xcite and thus fall short on tasks that require common sense or understanding nuanced meanings. training on text alone or even text and images may not be enough to continue advancing in the field further. spoken dialogue systems (sdss) operate with a different view of the world from us and thus, struggle to draw connections between what can be observed or the result of actions and language, resulting in poor interactions. ## sds research the field of dialogue research should work more closely with other fields related to interaction. dialogue systems trained on text alone cannot fully understand the nuances of language and how these affect the physical world, hence works are increasingly combining natural language and rich image representations to improve text and vi-sion benchmarks @xcite @xcite @xcite . improved multi-modal representations or more complete world views may be crucial for sds to navigate more complex scenarios. sds could also become more robust and flexible in the way that they process natural language. incrementally processing words instead of turns could enable sdss to better understand and coordinate the conversation with a human @xcite , as we often use feedback mechanisms such as backchannels (i.e., 'okay' or 'mhm') or clarifications to signal what has been grounded in a dialogue. fluid human-robot interactions may require keeping track of the conversation context explicitly in real-time @xcite so the sds can self-repair the state when there are issues or misunderstandings @xcite . finally, the field of sds could explore new ways of blending the natural language element of interactions with other modalities beyond vision, such as non-verbal communication. agents that only understand words may not be suitable to interactions outside labs and in unstructured environments. ## suggested topics for discussion here are some of the topics for discussion: