Successfully Guiding Humans with Imperfect Instructions by Highlighting Potential Errors and Suggesting Corrections (2024)

Abstract: Language models will inevitably err in situations with which they are unfamiliar. However, by effectively communicating uncertainties, they can still guide humans toward making sound decisions in those contexts. We demonstrate this idea by developing HEAR, a system that can successfully guide humans in simulated residential environments despite generating potentially inaccurate instructions. Diverging from systems that provide users with only the instructions they generate, HEAR warns users of potential errors in its instructions and suggests corrections. This rich uncertainty information effectively prevents misguidance and reduces the search space for users. Evaluation with 80 users shows that HEAR achieves a 13% increase in success rate and a 29% reduction in final location error distance compared to only presenting instructions to users. Interestingly, we find that offering users possibilities to explore, HEAR motivates them to make more attempts at the task, ultimately leading to a higher success rate. To our best knowledge, this work is the first to show the practical benefits of uncertainty communication in a long-horizon sequential decision-making problem.

## introduction expecting language models to consistently make accurate predictions in a dynamic world is unrealistic @xcite . evidence shows that these models often falter in unfamiliar situations @xcite . given the inherent fallibility of language models, an important research problem is to enable these models to successfully assist humans even when they make errors. but how is it possible for a model to guide a human toward the right decisions when it cannot precisely specify what those decisions are? this 1 our code and data for model and human evaluation are publicly released at https://lingjunzhao.github.io/hear.html . walk past the couch and turn right ▼ . walk walk straight down the hallway and stop in the bedroom. work demonstrates the feasibility of tackling this problem in a language-guided visual navigation setting. concretely, we develop hear (hallucination detection and remedy), a system that aids human navigation in 3d residential environments using potentially erroneous natural language instructions. the key to the success of hear is its ability to communicate various types of uncertainty information to users. specifically, hear can identify and highlight potential errors in an instruction, and suggest possible corrections. this information prevents misdirection and narrows the search space for users, enabling them to navigate successfully even when given inaccurate instructions. to our best knowledge, our work presents the first study on the effects of uncertainty communication on human decision making in a long-horizon task. although uncertainty communication has been identified as crucial for ai systems, very few studies have investigated how uncertainty information impacts human decisions. previous studies have primarily focused on classification tasks rather than long-horizon tasks, and on numerical uncertainty (i.e., probability) rather than verbal un-certainties @xcite . by demonstrating that presenting uncertainties leads to a substantial performance boost in this navigation task, we provide strong evidence to support the development of these features in sequential decisionmaking ai agents. to build hear, we tackle the problem of detecting and classifying hallucinated phrases in visually grounded instructions. this problem is particularly challenging in the environments we study because of the realisticity and diversity of the visual scenes. our solution involves training two vision-language models: one for hallucination detection and the other for classification (i.e., deciding whether a phrase should be deleted or replaced). we combine these models to identify hallucinations in an instruction, as well as score and rank potential corrections. to train each model, we fine-tune a large vision-language model @xcite with synthetically created data to optimize for a contrastive learning objective. we introduce a practical methodology for generating synthetic data, combining rule-based approaches with large language models. we conduct an evaluation with 80 human users to measure the effectiveness of hear. our results demonstrate that incorporating hear improves user navigation outcomes. specifically, hear increases the likelihood of a user successfully reaching their destination by 13% and reduces the average distance to the true destination by 29%. analyzing human behavior reveals that by providing useful hints, hear motivates humans to put more effort into solving a task, leading to a higher success rate. interestingly, our results suggest that the uncertainty communication capabilities of a system do not need to be flawless to boost user performance. the components of hear are all imperfect: the error detection and correction, and the instruction generation capabilities are all of reasonable quality, but not faultless. however, because these capabilities complement one another, and complement the knowledge of the human user, they ultimately improve user decisions. ## problem setting we consider the problem of generating language instructions to guide a human to follow an intended route in an environment. the concrete goal is to build a speaker model s(w | r), which takes an intended route r as input and generates a corresponding language instruction w as output (figure 1 ). the instruction @xmath0, . . . , w n ) is a sequence of words (e.g., "walk past the couch and turn right. walk down the hallway and stop in the bedroom."). the route @xmath1, a 1 , . . . , o l , a l ) is a sequence of observations and actions, where each observation is a collection of rgb images that capture the view at a location, and each action represents a transition from one location to another. the speaker is evaluated through an instructionfollowing task, in which a human user receives an instruction generated by the speaker and follows it in the corresponding environment. success is achieved if the user reaches the final location along the intended route. to simulate this problem, we employ the mat-terport3d simulator and room-to-room (r2r) dataset @xcite for model training and human experiments. matterport3d is a photorealistic simulator that features images taken from various real residential buildings. the r2r dataset contains pairs of route and language instruction. the instructions contain more than 7,000 object and direction phrases. we follow @xcite to train a t5-based (raffel et al., 2020) speaker model. the instructions generated by this model often contain object or directional phrases that are inconsistent with the scenes along the intended route. we refer to such phrases as hallucinations. we categorize hallucinations into two types: intrinsic hallucination is a phrase that needs to be replaced because it inaccurately describes an observation or action (e.g., an instruction says "walk past the reception desk and out the door on the right", but on the intended route, the door is on the left); extrinsic hallucination is a phrase that needs to be removed because it does not have a correspondence on the input route (e.g., "walk through the office and out of the office. walk into the hallway and turn left", where the second sentence describes a path that does not exist in the environment). upon inspecting 40 sample instructions generated by our speaker, we find that 67.5% of them have hallucinations, and that 20.9% of all the object and direction phrases are hallucinations. ## hear: hallucination detection and remedy in this section, we introduce hear, which augments a speaker model by enabling it to (i) highlight potential hallucinations in an instruction and (ii) produce a list of plausible corrections for each hallucination. we expect that (i) would help a user avoid being misled into incorrect regions, while (ii) would reduce the effort required to locate the correct region. we build two models ( § 4.1, § 4.2, illustrated in figure 2 ) to generate these pieces of information and design an interface to effectively convey them to users ( §4.4). ## experiments the questions that we aim to answer are: (q1) can hear reliably detect hallucinations and provide reasonable suggestions? (q2) does providing hallucination highlights and suggesting corrections improve human navigation performance? (q3) what are the effects of highlights and suggestions on human behavior? to answer q1, we evaluate hear intrinsically with human-annotated data. to answer q2 and q3, we conduct a human evaluation with various systems, including ablated versions of hear and an oracle human-based system. data. to train the hallucination detection model, we synthetically generate a training set with 164,939 pairs of positive and negative examples ( §4.3), which are created from the room-to-room (r2r) @xcite train set (4,675 routes, each route has 3 human-annotated instructions). to train the hallucination type classification model, we generate 117,357 pairs of positive and negative examples, created from the r2r train set. for both evaluations, we first use a speaker model ( § 3) to generate instructions describing routes from the r2r validation seen split. for intrinsic evaluation and model selection, we randomly select and annotate 40 routes from the split as our dev set. for human evaluation, we use the 75 test routes from previous work @xcite as our test set. there is no overlap between the dev set and the test set. ## conclusion we present a novel approach to enhance human task performance by effectively communicating model uncertainties. by encouraging users to refine ai-generated solutions, our approach offers an alternative to the conventional method that focuses on directly improving ai autonomous capabilities while overlooking human capabilities. to fully unlock the potentials of ai technologies, we advocate for viewing ai systems not as independent problem solvers, but as assistants and collaborators of humans. while our research primarily addresses language-guided visual navigation, the insights gained are broadly applicable to other visionlanguage tasks. specifically, we have demonstrated that: (i) it is feasible to generate meaningful error highlights and correction suggestions for vision-language models, and (ii) presenting these highlights and suggestions to human users can improve their decision-making. moreover, our methods for creating synthetic errors and correction suggestions using rules and large language models are generalizable to various contexts.