Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake? (2023)

Abstract: Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks. These systems need to have a sophisticated understanding of the user as well as the environment, and make timely accurate decisions on when and what to say. To address this issue, we created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We further proposed two tasks: User and Environment Understanding, and Instructor Decision Making. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance. Our quantitative, qualitative, and human evaluation results show that these models can demonstrate fair performances in some cases with no task-specific training, but a fast and reliable adaptation remains a significant challenge. Our benchmark and baselines will provide a stepping stone for future work on situated task guidance.

## introduction you have probably watched a lot of youtube videos on how to bake a cheesecake, or how to change the car windshield wipers, but something always goes wrong and you just wish there is an expert right there to guide you through. can we design an artificial intelligent system to watch, talk and guide humans step by step to complete a given task? task guidance for human users is a challenging problem, as it requires an interactive system to have a sophisticated understanding of what the user is doing, under the environment setup, and providing appropriate timely guidance. what is more challenging is if we could design a model * work done during a summer internship at the university of michigan. traditional approaches to develop ai agent for interactive task guidance like this require a large amount of task-specific training or rules to recognize object states @xcite , mistakes in actions @xcite , and to interact with humans @xcite , thus limiting their ability to generalize. however, the recent rise of foundation models trained on a large amount of multimodal data from the web @xcite @xcite @xcite creates new opportunities for developing robust open-domain ai agents for this problem. these works have undergone a paradigm shift and begun to explore the zero-and few-shot adaptation of these models to various embodied ai problems @xcite @xcite . in this work, we extend this paradigm shift by examining the application of recent state-of-theart foundation models to situated interactive task guidance. we created watch, talk and guide (wtag), a new multimodal benchmark dataset which includes richly annotated human-human dialog interactions, dialog intentions, steps, and mistakes to support this effort. we define two tasks: (1) user and environment understanding, and (2) instructor decision making to quantitatively and qualitatively evaluate models' task guidance performance. with the inherent complexity of the problem itself, this dataset can help researchers understand the various nuances of the problem in the most realistic human-human interaction setting. we used a large language model (llm) as the backbone for guidance generation, and explored three different mutimodal methods to extract vi-sual and dialog context. our empirical results have shown promising results with foundation models, and highlighted some challenges and exciting areas of improvement for future research. the dataset and code are available at https://github.com/ sled-group/watch-talk-and-guide . ## a dataset for situated task guidance in this work, we introduce watch, talk, and guide (wtag), a new dataset for situated task guidance. wtag includes nearly 10 hours of egocentric videos of human users performing cooking tasks while guided by human instructors through live, natural interaction. synchronized videos and audio transcripts in wtag present a variety of challenging phenomena, including perceptual understanding, communications, natural mistakes, and much more. we hope this dataset can serve as the starting point to dive into this complex problem. egocentric video datasets ( ## task definitions for benchmark evaluation, we extract query points from the wtag dataset whenever one of the following conditions is met: ## methods given the wtag dataset and the two tasks defined above, we explore the application of pre-trained large language and vision foundation models on this problem without task-specific training. for each recording in wtag, we feed the video frame by frame to our system, together with the synchronized dialog transcripts, and only query chatgpt at the three query conditions (section 4). the visual frames go through optional multimodal information extraction process detailed below, to translate the relevant visual context into natural language. the dialog transcripts are extracted at the first frame each utterance appears, and offer conversational context for model predictions. we designed a prompt template (figure 4 ) that includes the ground truth recipe template, user-instructor chat history up to time point t, observations of the user and environment, as well as a list of questions listed in section 4. we then send the prompts to chatgpt foot_5 to answer questions related to the proposed tasks in each query time point. to enrich the prompts and offer more context, we explored the following three methods that translate multimodal precepts from the egocentric video into language: language only (lan): as a baseline for the llm, we extracted the ground truth user and instructor dialog up until time t. all the past utterances are added to the prompt as part of the interaction history, and only the most recent user utterance is added to the observation to avoid model cheating. to enable temporal reasoning, observations include how long the user has been following the recipe so far. this method does not offer any visuallydependent information to the llm backbone, and challenges the model to infer the context purely based on the conversation. scene description (sce): in this method, we generate a free-text scene description by applying blip-2 @xcite to the latest frame image, and insert it into the prompt as part of the observations. depending on the prompts to blip-2, we ask generic questions such as "what is the user do-ing" or "this is a picture of" to get the descriptions. while this approach is flexible and open-domain, there is no control of how much situation-specific the descriptions would be. together with the time lapsed and the dialog history, this method offers more scene level visual context to the llm. object and state detection (obj): in this method, we extract more fine-grained scene information by detecting important objects in the frame image and their corresponding states, and inserting them into the prompt observations. at each query time t, we use egohos @xcite to segment user's hands and the objects that they are interacting with. we then extract the object segments and predict their most likely object names and their corresponding states using clip @xcite . to balance the performance and generalizability of clip predictions, we first extracted a list of most likely objects and their potential states from the recipe through a separate prompt to the llm backbone. the resulting list of objects and states go through a minor manual cleanup before being used by clip. this narrows down the scope of search for clip, and offers better targeted vision to language prediction, but is also easily generalizable to open-world object and state detections. all three methods were queried at the same time points as extracted in section 4 for fair comparison. we reserved 6 recordings (2 of each recipe) for hyperparameter and prompt tuning, and the rest for evaluation. each method was repeated 3 times. chatgpt apis configurations are: max @xmath0, @xmath1, stop @xmath2, -, """, "']. all experiments conducted on a single gpu nvidia rtx a6000 and a intel(r) core(tm) i9-10900x cpu @ 3.70ghz. ## experimental results in this section, we evaluate the three methods above on the following two tasks: user and environment understanding, and instructor decision making. micro f1 scores are reported for each classification task. we further break down the performance, and evaluate how accurately each vision extraction module can translate the scene into language, and how helpful or annoying the model's generated sentences are under a human evaluation. for user and environment understand tasks, the models demonstrated well above random chance performance in user intention prediction and step detection, but struggle with mistake recognition. for instructor decision prediction tasks, the models showed above random chance performance in predicting instructor's intention and instruction types, but issued higher communication frequencies than human instructors. across the three methods, language only (lan) showed comparable performance even without any visual context. ## conclusion and discussions in this work, we explored how to leverage foundation models to guide human users step by step to complete a task in a zero-shot setting given an arbitrary task recipe. we created a new benchmark dataset, watch, talk and guide (wtag), with natural human user and instructor interactions, and mistake guidance. two tasks were proposed to evaluate model's ability on user and environment understanding, as well as instructor decision making. we used a large language model as the backbone for guidance generation, and compared three configurations of inputs incorporating language and visual context through dialog history, scene description, and object and state detection with multimodal foundation models. we conducted quantitative, and human evaluations of the three methods on our dataset, and discovered several challenges for future work. first of all, vision to language translation is challenging. having an accurate and relevant description of what the user is doing and the environment setup is non-trivial. some of the methods we've experimented with often describe untruthful scenes or objects, or offer true but too generic or irrelevant information, e.g., "the user is preparing food." for future work, it would be helpful if these vision to language models could leverage recipe information, and user's attention (e.g., eye gaze), and other sensory inputs (e.g., sound) to achieve richer and more relevant user and environment descriptions. second, the overall performance of each prediction task from the large language model is fairly above random guessing, especially in a zero-shot setting with no task-specific finetuning, but are realistically too low to be useful in practice. there is much room for improvement, but also a great potential to be generalized to more tasks, multi-user or multi-tasking environments. third, there is a big difference between handson task guidance versus watching a "how-to" youtube video. if we look closely, most of the model generated responses are instructions, instead of more situation-aware and personalized responses such as question answering and confirmations. it is easy to repeat recipe instructions or perhaps offer a little more detail leveraging the knowledge base, but harder to communicate with the user in a more situation-relevant way. sometimes, a simple confirmation can boost the user's confidence, and worth a lot more than repeating what the user should do. furthermore, besides user intent and hesitation, there are other types of user states that can be helpful but are not currently modeled by our system. are they familiar with the recipe? do they look confused? are they getting annoyed by all the instructions? are they emotionally stable? all of these can change the way the instructor should talk. it is hard to categorize and collect users' mental states at each time point throughout the task. sometimes experienced human instructors can infer through user actions and utterances, but the large language models have not demonstrated a strong ability in that way in our experiments. lastly, on the model architecture side, we leveraged some of the best performing multimodal and language models currently available, translated the rich audiovisual context collected in wtag to language, and used llm as the only reasoning engine to generate guidance. language is too concise of a medium to offer context, and sometimes a picture is worth a thousand words. the current multimodal foundation models have limited reasoning capabilities to answer complicated questions and offer accurate timely guidance. this opens a wide variety of future opportunities for improving multimodal foundation models for situated task guidance.