Towards Adaptive Human-Agent Collaboration in Real-Time Environments (2025)

Abstract: My research interests lie in human-agent collaboration and user adaptation, with a particular emphasis on adaptation in real-time collaborative environments.The field of collaborative systems aims to support human teams in completing complex tasks efficiently while ensuring natural and adaptive interaction experiences.I investigate how AI agents can function as effective partners by adapting to their human collaborators.A central focus of my research is the personalization of agent behavior based on user proficiency.This includes methods for adapting the agent’s communication strategies according to the user’s skill level and task experience. To pursue this goal, I collected and analyzed a multimodal dataset of human-human interaction using a real-time collaborative cooking game environment (Wu et al., 2021; Liu et al., 2024).The chosen environment is characterized by its complex task mechanics and strict time constraints, which necessitate seamless coordination and elicit dynamic, natural collaborative behaviors such as role negotiation and error recovery.Through this analysis, I investigated how partners with different levels of task proficiency communicate and coordinate effectively.Based on the findings, I proposed practical design guidelines for future adaptive AI agents, enabling them to adjust their level of guidance and initiative in response to the user’s proficiency.

## spoken dialogue system (sds) research i expect that future spoken dialogue systems (sdss) will evolve from simple assistants into effective collaborative partners, capable of engaging in real-time, handson tasks alongside humans. a key driver of this evolution will be the shift from text-based to voice-based communication. in visually demanding, high-pressure tasks such as the cooking game environment used in my research, text-based chat can increase the user's cognitive and visual load, forcing them to divert attention away from the task itself. voice interaction frees the user's visual channel, allowing them to maintain focus on the shared workspace. this more natural communication modality can also foster a stronger sense of presence and engagement, making the ai feel more like a true partner. however, to realize this potential, the field must address the critical challenge of real-time responsiveness. while speaking is a fast and natural input method for humans, the sds must perform a complex pipeline of speech recognition, intention understanding, planning, and response generation, all within a fraction of a second to keep pace with the collaborative task. the inference latency inherent in today's large-scale models poses a significant obstacle to achieving this fluidity. therefore, i believe a crucial research direction for the next generation of researchers is to develop novel sds architectures that balance advanced reasoning capabilities with the stringent low-latency requirements of simultaneous human-ai collaboration.