Do What? Teaching Vision-Language-Action Models to Reject the Impossible (2025)

Abstract: Recently, Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks. These models rely on multimodal inputs, with language instructions playing a crucial role-not only in predicting actions, but also in robustly interpreting user intent, even when the requests are impossible to fulfill. In this work, we investigate how VLAs can recognize, interpret, and respond to false-premise instructions-natural language commands that reference objects or conditions absent from the environment. We propose — Instruct-Verify-and-Act (IVA) — a unified framework that (i) detects when an instruction cannot be executed due to a false premise, (ii) engages in language-based clarification or correction, and (iii) grounds plausible alternatives in perception and action. Towards this end, we construct a large-scale instruction tuning setup with structured language prompts and train a VLA model capable of handling both accurate and erroneous requests. Our approach leverages a contextually augmented, semi-synthetic dataset containing paired positive and false-premise instructions, enabling robust detection and natural language correction. Our experiments show that IVA can improves false premise detection accuracy by 58.89% over baselines, while increasing successful responses in false-premise scenarios by 27.89%.

## introduction vision-language-action (vla) models @xcite @xcite represent a significant advancement in robotics, enabling agents to perform tasks using multimodal inputs by integrating visual perception, natural language understanding, and action generation. however, as these models are increasingly deployed in open-ended, real-world environments, they must handle diverse and often ambiguous instructions issued by users. consider a household when the robot receives a command referencing a non-existent object (a bottle), it detects the false premise, and generates a clarifying response that corrects the instruction, suggesting a valid alternative (a drawer). while large multimodal models have made significant strides in visual grounding and instruction following, prior research in both natural language processing and robotics has typically assumed that user instructions are feasible and contextually grounded. in domains such as visual question answering and dialogue, models have been studied for their robustness to unanswerable or contradictory queries have been developed which can evaluate and correct for false premises in the case of question answering @xcite @xcite @xcite @xcite @xcite @xcite @xcite . however, in the context of embodied agents and robotics, the issue of handling false-premise instructions -recognizing unfulfillable commands and producing helpful language-based corrections -remains largely unexplored. existing robotics benchmarks and vlas focus on execution success given correct instructions, without testing how these systems respond when user intent is unattainable or ambiguous. to address this gap, we develop a unified vla model capable of interpreting and correcting false-premise instructions in robotic settings. our framework combines large-scale instruction tuning with contextually-augmented datasets, enabling the model to detect unfulfillable requests, provide natural language feedback, and suggest alternative courses of action. see figure 1 . we evaluate our approach across eight robotic tasks, measuring false premise detection accuracy and the rate of successful responses in false-premise scenarios. our method achieves an improvement of 97.56% in false premise detection over established baselines, and increases success in false-premise handling by 50.78%. these results demonstrate that language-aware robots can move beyond simple execution -reasoning about user intent, clarifying ambiguity, and interacting naturally even when confronted with the impossible. ## related work vision-language-action models (vlas). vlas integrate visual perception, language understanding, and action generation to produce robot control sequences from visual observations and language instructions. models such as llarva @xcite , π (physical intelligence et al., 2025), gemini robotics @xcite , openvla @xcite , rt-2 @xcite , palm-e embodied multimodal model @xcite , and llara @xcite fine-tune pretrained large multimodal models (lmms) to predict actions, often using special tokens to represent the action space. these models share the goal of adapting lmms for robotic control as they differ in the choice of lmm and action encoding methods. while existing vlas typically leverage language decoders pretrained for high-level tasks (e.g., image captioning @xcite and vqa @xcite ), when deployed in diverse, real-world environments, they exhibit a critical limitation: the inability to handle instructions grounded in false premises. our work addresses this gap by explicitly training vla models to recognize when referenced objects or conditions are absent and to respond appropriately. false premises. the problem of detecting false premises has long been studied in nlp, particularly in qa benchmarks such as squad 2.0 @xcite and false qa @xcite , with research primarily focused on model uncertainty for unanswerable questions @xcite . this concept of false premises has recently spread from nlp to other areas of interest, including visual question answering @xcite , image/text matching @xcite @xcite @xcite , image-grounded conversation @xcite , tool usage @xcite and hallucination detection @xcite . research in human-robot interaction has explored handling imperfect instructions and clarifying ambiguous instructions @xcite @xcite @xcite , however this short paper is the first to explore false premise tasks in the context of vision-language-action models (vlas). ## a vla that instructs, verifies, and acts our method builds upon the llarva model @xcite , a vla architecture designed for robotic instruction following. llarva leverages instruction-tuned large multimodal models (lmms) to jointly interpret visual observations, natural language instructions, and robot proprioception, outputting robot actions along with intermediate visual representations called visual traces. input. llarva takes a visual observation o t , an rgb image at timestep t, and a structured natural-language instruction l t as input. the language instruction encodes the robot type (e.g., franka panda), control mode (e.g., end-effector or joint control), task description (e.g., "close the drawer"), proprioceptive states (internally-sensed joint-angle vectors that indicate the robot's current position before it plans the next move) from the previous h timesteps, and the number of future actions (n) to predict, the instruction tuning template is shown as follows: model. llarva integrates three main components as follows: llarva is initially pretrained on large-scale vision-action instruction data from open x-embodiment (oxe) and subsequently fine-tuned for specific robotic tasks, enabling strong generalization across diverse tasks and environments. false premise instruction dataset. following @xcite , we utilize image-action pairs from the oxe dataset @xcite , providing rich visual, language, and action representations. however, prior work did not explicitly address reasoning about the feasibility of user instructions. to address this gap, we constructed a dedicated dataset explicitly curated to handle false premise scenarios and their corresponding corrections. our false-premise instruction dataset is generated from robotic trajectories available in rlbench @xcite . each task in our dataset includes two categories of false premise instructions: in-domain false premise. these instructions involve geometrically similar and contextually plausible objects derived from related tasks, making the intended correction relatively intuitive. for instance, in the task of closing a jar, if the user prompt is "the task is close the blue safe", the model is expected to respond, "i don't see a safe in the current scene. do you mean jar?". out-of-domain false premise. these instructions contain clearly infeasible or nonsensical requests involving objects or scenarios impossible within the given context. for example, during the open drawer task, if the user prompts "the task is open the top elephant," the model should identify the absurdity and respond appropriately, such as "i couldn't find a elephant in the current scene," subsequently terminating the interaction since the request is fundamentally invalid. for training purposes, our dataset composition strategically includes approximately 20% of episodes containing out-of-domain false premises, and 65% of episodes containing in-domain false premises injected into 10% of their respective steps. this deliberate distribution ensures comprehensive exposure and training of the model in recognizing and handling various types of erroneous instructions. ## results we evaluated iva on two aspects: its ability to detect and correct false-premise (fp) instructions, and its performance on standard, true-premise (tp) tasks. ## conclusion in this work, we introduced the instruct-verify-and-act framework, enabling vla models to robustly detect, clarify, and correct false-premise instructions in robotic settings. our approach achieves strong performance in both false-premise detection and standard task execution, demonstrating the value of explicit false-premise reasoning in vla models. we hope these results demonstrate the potential of language-aware robots to engage more naturally and safely with users, even when faced with ambiguous or impossible commands. future work will focus on extending iva's capabilities to more complex environments and real-world deployments. finally, we do not anticipate a specific negative impact, but, as with any machine learning method, we recommend exercising caution.