On Referring Language Use in Visually Grounded Dialogue (2023)

Abstract: Position paper for YRRSDS 2023

## research interests my current research lies at the intersection of vision and language; more specifically, i am working on problems related to visually grounded language understanding for conversational agents. by observing conversations between humans, we aim to improve our understanding of referring language use in dialogue, so that we may develop systems capable of engaging in interactions with humans that involve references to a co-observed world. ## spoken dialogue system (sds) research in the coming years, i expect much emphasis to be put on learning to integrate modalities end-to-end. problems that are inherently multimodal, which aside from sdss also includes visually grounded language understanding, ultimately require solutions that respect the interactions between modalities. for example, even though we can use an automatic speech recognition system to transcribe speech and use an llm as the natural language understanding component of the sds to processes the transcription, we would miss out on extralinguistic context, such as the prosodic cues that were present in the speech signal, that may be vital to the interpretation of the message: in trying to understand what message someone is attempting to convey, we do not simply take note of the uttered words; we also pay attention to how those words were uttered. ## suggested topics for discussion hearing (tmh) working towards visually grounded language understanding for conversational agents in the context of the wasp-funded roboground project. before starting his doctoral studies in sweden in 2019, he studied and worked at tilburg university, completing his msc degree (cum laude) in 2017 and working as a junior researcher (full-time) on the horizon 2020-funded l2tor project until 2019. he enjoys thought-provoking discussions that induce existential dread and long walks on the beach.