R 2 H : Building Multimodal Navigation Helpers that Respond to Help Requests (2023)

Abstract: Intelligent navigation-helper agents are critical as they can navigate users in unknown areas through environmental awareness and conversational ability, serving as potential accessibility tools for individuals with disabilities. In this work, we first introduce a novel benchmark, Respond to Help Requests (R2H), to promote the development of multi-modal navigation helpers capable of responding to requests for help, utilizing existing dialog-based embodied datasets. R2H mainly includes two tasks: (1) Respond to Dialog History (RDH), which assesses the helper agent’s ability to generate informative responses based on a given dialog history, and (2) Respond during Interaction (RdI), which evaluates the effectiveness and efficiency of the response during consistent cooperation with a task performer. Furthermore, we explore two approaches to construct the navigation-helper agent, including fine-tuning a novel task-oriented multi-modal response generation model that can see and respond, named SeeRee, and employing a multi-modal large language model in a zero-shot manner. Analysis of the task and method was conducted based on both automatic benchmarking and human evaluations.

## introduction assisting humans in real-world scenarios is a fundamental capability that ai agents should possess. an ai helper, communicating with humans in natural language based on visual observation in the environment and oracle information could significantly enhance work productivity and serve as an accessibility tool for individuals with disabilities. figure 1 illustrates an example of a delivery person seeking assistance in navigating through an unfamiliar building. with the help of a navigationhelper agent, the delivery person can ask questions ## the r2h benchmark significant differences exist between building a helper agent and developing task performer agents for dialog-based multi-modal navigation tasks. task performers, such as ones in cvdn (thomason et al., 2020), dialfred @xcite , and avdn @xcite , are challenged as command followers, evaluated based on their performance of following dialog histories from human annotations, as shown in figure 2i . however, the helper agent works as a supportive role, responds to questions from the task performer to facilitate task success. therefore, the evaluation of the helper agent's performance could not be solely dependent on the agent itself, but on how the task performer benefited from the response. therefore building the helper agent requires evaluations in a collaborative setting with task performers. involving humans as task performers to evaluate the helper agent is ideal but expensive. alternatively, inspired by @xcite , @xcite and roman et al. (2020) that build helper and performer agents to collaborate as shown in figure 2ii , we introduce the respond to help requests (r2h) benchmark, involving task performer in the evaluation process as shown in figure 2iii ). in this way, the helper agent can be assessed comprehensively and realistically. r2h benchmark tests the agent's ability let me help! i am seeree, a helper agent that can see and respond to your questions. 2 … environment task performers under test bound helper-performer agent pairs under test helper agents under test 1 human annotation … environment environment task performer / 1 2 (i) benchmark for task performers. let me help! i am seeree, a helper agent that can see and respond to your questions. 2 … environment task performers under test helper-performer agent pairs under test helper agents under test 1 human annotation … environment environment pre-trained & fixed performer agent or human / 1 2 (ii) benchmark for helperperformer agent pairs. let me help! i am seeree, a helper agent that can see and respond to your questions. 2 … environment task performers under test helper-performer agent pairs under test helper agents under test 1 human annotation … environment environment human or pre-trained & fixed performer agent / 1 2 (iii) benchmark for helper agents. to respond effectively in a wide range of scenarios, including two novel tasks, the respond to dialog history (rdh) task and the respond during interaction (rdi) task, building upon three existing vision-and-dialog navigation datasets. rdh task evaluates helper agents in a situation where partial human dialog history is provided and the rdi task aims at challenging the helper agents with real collaborative scenarios. ## conclusion in this paper, we introduce the respond to help requests (r2h) benchmark with two tasks, respond to dialog history (rdh) and respond during interaction (rdi), assessing a helper agent's guidance capabilities. with the r2h benchmark, we build and evaluate two navigation-helper agents, seeree and multi-modal llm model. the results show that they both outperformed the baseline model. through further ablation study and human evaluation, we prove the effectiveness of seeree in assisting humans with tasks and argue that the effectiveness of responses cannot be determined by their linguistic resemblance to human dialogue alone.