Conversational Collaborative Robots (2025)

Abstract: Spoken dialogue systems (SDSs) aims to enable natural, interactive and collaborative conversations. My research interest lies in leveraging these situated collaborative conversations to teach new concepts (skills) to collaborative robots (cobots). These cobots, when operating in manufacturing environments such as assembly lines, are envisioned to converse with humans, reach common ground, and learn new skills in one shot without the need for multiple demonstrations. Unlike SDSs in consumer domains, these cobot-based systems must handle conversations in noisy, time-sensitive industrial settings. Motivated by these challenges, my research focuses on building collaborative dialogue systems capable of integrating conversational programming to translate situated dialogue into modular programs, knowing when to ask for clarifications, and adapting the program based on corrections.

## research interests spoken dialogue systems (sdss) aims to enable natural, interactive and collaborative conversations. my research interest lies in leveraging these situated collaborative conversations to teach new concepts (skills) to collaborative robots (cobots). these cobots, when operating in manufacturing environments such as assembly lines, are envisioned to converse with humans, reach common ground, and learn new skills in "one shot" without the need for multiple demonstrations. unlike sdss in consumer domains, these cobot-based systems must handle conversations in noisy, time-sensitive industrial settings. motivated by these challenges, my research focuses on building collaborative dialogue systems capable of integrating conversational programming @xcite to translate situated dialogue into modular programs @xcite , knowing when to ask for clarifications @xcite @xcite , and adapting the program based on corrections @xcite . ## spoken dialogue system (sds) research in the era of llms, the future of sds research is difficult to predict but looks promising. these models lower the barrier to creating culturally adaptive and accessible dialogue systems, while also raising important ethical and regulatory concerns. sds applications are expanding beyond traditional domains into areas like mental health support, creative learning, and industrial automation. this calls for interdisciplinary collaboration. future efforts should focus on making sdss robust to changes in underlying apis and external dependencies. improving automatic evaluation metrics and methodologies is also necessary, as they remain a major bottleneck. despite their impressive fluency, llms often rely on surface-level pattern matching, which may limit their ability to engage in deeper goal-driven, spoken dialogue interactions involving common ground. this makes it essential to investigate and mitigate these limitations to ensure safe and trustworthy dialogue systems. while curating human evaluation data for sds remains resourceintensive, llms could be leveraged to help automate and scale this process. additionally, improving the performance of smaller llms, especially in low-resource settings, will be important for making sds systems widespread and inclusive.