Grounding Task Assistance with Multimodal Cues from a Single Demonstration (2025)

Abstract: A person’s demonstration often serves as a key reference for others learning the same task. However, RGB video, the dominant medium for representing these demonstrations, often fails to capture fine-grained contextual cues such as intent, safety-critical environmental factors, and subtle preferences embedded in human behavior. This sensory gap fundamentally limits the ability of Vision Language Models (VLMs) to reason about why actions occur and how they should adapt to individual users. To address this, we introduce MICA (Multimodal Interactive Contextualized Assistance), a framework that improves conversational agents for task assistance by integrating eye gaze and speech cues. MICA segments demonstrations into meaningful sub-tasks and extracts keyframes and captions that capture fine-grained intent and user-specific cues, enabling richer contextual grounding for visual question answering. Evaluations on questions derived from real-time chat-assisted task replication show that multimodal cues significantly improve response quality over frame-based retrieval. Notably, gaze cues alone achieves 93% of speech performance, and their combination yields the highest accuracy. Task type determines the effectiveness of implicit (gaze) vs. explicit (speech) cues, underscoring the need for adaptable multimodal models. These results highlight the limitations of frame-based context and demonstrate the value of multimodal signals for real-world AI task assistance.

## introduction when individuals learn tasks through demonstrations, whether administering medication, assembling machinery, or cooking family recipes, they rely on subtle contextual cues. although pretrained vision language models (vlms) have a user selects a lactose-free product using eye gaze and speech. gaze highlights the product before speech clarifies intent: "i usually get this to avoid lactose." the core challenge lies in reconciling two shortcomings: (1) partial perception, where frame-only processing discards intent-revealing signals like spoken instructions and gaze identification, and (2) static reasoning, where vlms lack mechanisms to adapt to demonstrator-specific patterns. however, most retrieval-augmented generation (rag) approaches limit themselves to captioning frames or clustering segments @xcite , losing these essential signals. meanwhile, existing vlms often struggle to ground fine-grained spatial details in images or video, particularly in the absence of additional guidance @xcite @xcite . consequently, critical moments, such as a chef's glance at oven settings or a mechanic's triple tool check, become lost as actionable signals. to bridge this gap, we introduce mica (multimodal interactive contextualized assistance), an end-to-end system that records and processes a single user demonstration-including eye gaze and speech-and subsequently provides personalized guidance to a different individual. as shown in figure 2 , mica extracts key moments from demonstrations and aligns them with user queries to provide contextualized assistance. mica dynamically leverages available cues, using gaze and speech signals to segment the demonstration, extract key frames, and generate contextualized captions for retrieval-augmented generation. at inference time, it retrieves the most relevant segments and prompts a vision-language model on the contextualized images and captions to produce tailored responses. we evaluate mica on a benchmark derived from real-time chat-assisted interactions. in this setting, participants demonstrate personalized tasks while simultaneously providing multimodal cues. later, different users ask questions about these tasks to an ai chat assistant, aiming to replicate the same procedures. while previous video benchmarks rely solely on rgb and use offline annotators or templates for question generation @xcite , ours integrates video, continuous gaze, and speech with ecologically-valid, on-the-fly user queries. our findings reveal: (1) mica significantly outperforms zero-shot and frame-only rag baselines, leveraging multimodal cues for richer and more contextualized responses. by unifying speech with rgb and nonverbal cues like gaze, mica improves intent-aware task understanding-showing that how demonstrators communicate (through hesitations and attention shifts) is every bit as crucial as what they say. ## problem formulation consider the scenario in which a conversational assistant helps a new user replicate an everyday task after seeing a single demonstration from a different user. formally, given a query (q, i) from a new user which consists of an open-vocabulary question q and an image i (fig 2b ). the goal of the assistant is to generate an answer a (fig 2d ) to the question q informed by the image i and a previous demonstration d (fig 2a ). the answer a must approximate the ground truth human answer the demonstration d includes: 1. egocentric rgb frames i t at each time step t. 2. natural language speech transcriptions @xmath0, s, e), where n represents the text segment spoken by the user, s denotes the start time, and e denotes the end time. 3. eye gaze g @xmath1, d t ), with gaze origin p @xmath2, and hand pose k @xmath3. despite overlaying hand pose keypoints on all frames, they had little effect on interpreting hand cues; we therefore prioritize gaze and speech. ## method the goal of our framework shown in fig. 2c is to incorporate implicit (eye gaze) and explicit (speech) cues to enrich demonstration understanding for vision-language models (vlms). as shown in (fig. 3 ), we first segment demonstrations into meaningful units and then perform key frame extraction and captioning. ## data collection we collected user demonstrations and evaluations with a hololens 2 across three task categories: organizing a room, shopping, and a morning routine. in the demonstration phase (fig. 4 , left), participants performed each task following their own preferences while speaking aloud to provide rich verbal cues. in the live evaluation phase (fig. 4 , right), a new participant asked questions to replicate the same task setup. this produced realistic, user-specific queries answered by referencing the demonstration. our offline evaluation set includes 32 demonstrations and 415 live questions. we annotated each question with a ground-truth answer a * . question accuracy is measured via llm-match (see appendix section s4.2.2). further details on participant instructions, hardware setup, and sample prompts are provided in appendix sectoion s4. ## results we evaluate the effectiveness of various models in extracting user-specific context from demonstra-tions, leveraging implicit (e.g., eye gaze) and explicit (e.g., speech) cues. our goal is to benchmark advanced vlms' performance in understanding nuanced user intent and facilitating accurate responses in real-world applications. baselines. we evaluate the ability of various baselines to extract relevant context from demonstrations using implicit and explicit cues on the annotated evaluation set. for each model, we assess the following methods for in-context learning from the demonstration: ## analysis of implicit cues to ground context our results (section 6.1) show that vlms effectively use eye gaze for contextual understanding, matching or surpassing speech in some cases . three key benefits emerged: (1) gaze provides precise focus cues for disambiguation, (2) it grounds intent by linking attention to objects, and (3) it resolves ambiguous references like 'those' more effectively than speech alone. ## conclusion in this work, we presented a novel framework that leverages vlms with gaze and speech inputs to provide contextualized task assistance from a single demonstration. our approach highlights the potential of using both implicit and explicit cues to enable personalized and efficient task assistance in real-world settings. through experiments across various tasks such as organizing, shopping, and morning routines, we demonstrated that vlms can achieve significant performance gains compared to naive video methods by effectively utilizing implicit gaze cues.