HEAL : An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models (2025)

Abstract: Large language models (LLMs) are increasingly being adopted as the cognitive core of embodied agents. However, inherited hallucinations, which stem from failures to ground user instructions in the observed physical environment, can lead to navigation errors, such as searching for a refrigerator that does not exist. In this paper, we present the first systematic study of hallucinations in LLM-based embodied agents performing long-horizon tasks under scene–task inconsistencies. Our goal is to understand to what extent hallucinations occur, what types of inconsistencies trigger them, and how current models respond. To achieve these goals, we construct a hallucination probing set by building on an existing benchmark, capable of inducing hallucination rates up to 40\times higher than base prompts. Evaluating 12 models across two simulation environments, we find that while models exhibit reasoning, they fail to resolve scene-task inconsistencies — highlighting fundamental limitations in handling infeasible tasks. We also provide actionable insights on ideal model behavior for each scenario, offering guidance for developing more robust and reliable planning strategies.

## introduction recent advances in the reasoning and generalization capabilities of large language models (llms) @xcite have led to their increasing adoption as the cognitive core @xcite of embodied agents @xcite @xcite , enabling these systems to interpret instructions in natural language and formulate action plans in complex environments. however, llms have well-known vulnerabilities @xcite . consequently, llmdriven agents @xcite inherit not only the vast world knowledge and reasoning capability of llms, but also their limitations @xcite ; most notably a persistent tendency to hallucinate @xcite . while hallucination is a well-recognized limitation of llms @xcite , its manifestation in embodied agents is qualitatively different. unlike conversational systems, where hallucinations often result in factual errors or incoherent replies @xcite , hallucinations in embodied agents stem from a failure to ground user-provided task instructions in the observed physical environment. this misalignment can lead to consequences far more serious than a simple textual error. for example, if a robot is instructed to "put the knives in the dishwasher" but no dishwasher is present, an llm unable to reconcile the task with the observed scene may hallucinate the existence of the dishwasher, and include it in the generated plan to follow the user instruction. this can cause the robot to place sharp utensils in an empty cabinet or try to press buttons on a bare wall, leading to physical damage, safety hazards, and wasted battery. such behaviors highlight the need for scene-taskconsistent planning in llm-based agents. motivated by these limitations of current llmsmost of which are optimized to complete @xmath0. turn on the washing machine. {"node goals": [ {"name": "washing_machine", "state": "on"}, {"name": "washing_machine", "state": "plugged_in"}], "edge goals": [{"from_name": "clothes_jacket", "relation": "on", "to_name": "washing_machine"}, {"from_name": "soap", "relation": "on", "to_name": "washing_machine"}]} (a) base pipeline {"node goals": [...], "edge goals": [ {"from_name": "clothes_jacket", "relation": "on", "to_name": "washing_machine"}]} {"node goals": [ {"name": "washer", "state": "on"}], "edge goals": [ {"from_name": "detergent", "relation": "on", "to_name": "washing_machine"}]} {"node goals": [ {"name": "washing_machine", "state": "on"}], "edge goals": [ {"from_name": "soap", "relation": "on", "to_name": "washing_machine"}]} these examples demonstrate that when inconsistencies arise between the scene information and the given task description, the llm fails to reconcile the two and generates incorrect plans or object references. given the base prompt, we systematically modify two core input components-the task description and scene information-to elicit hallucinations. our four controlled modifications of the base prompts are: under task description variation, (i) distractor injection-adds non-existent objects to the task description; and under scene variation, (ii) task relevant object removal-omits key objects from the scene; (iii) synonymous object substitution-replaces scene objects with synonyms; and (iv) scene task contradiction-introduces conflicts between the task and scene. under ideal conditions-we shift our focus to understand their failure modes. specifically, we aim to answer the following research questions: rq1: to what extent do llm-based embodied agents hallucinate under scene-task inconsistencies; what types of mismatches are more likely to trigger hallucinations; and what limitations do these failures reveal? rq2: does the absence of hallucination imply correct planning? what are the ideal behaviors in these scenarios for more robust planning? although prior works explore llms in embodied agents @xcite and hallucinations in qa @xcite , studies on hallucinations in embodied agents, especially for long-horizon tasks, are limited. existing efforts @xcite mainly study incidental cases from generic prompts, capturing only surface-level issues and overlooking the failure patterns behind hallucinations. to fill this gap, we present, to the best of our knowledge, the first empirical study that systematically exemplifies and quantifies hallucinations in long-horizon planning through controlled scene-task inconsistencies. because existing datasets either do not explicitly aim to elicit hallucination in long-horizon embodied tasks, or do not target embodied agent tasks, we first construct a new probing set foot_0 that is more likely to cause hallucination. constructing such a probing set faces two main challenges: (i) how to induce hallucinations and (ii) how to detect unwanted behaviors caused by hallucinations. we solve these challenges by systematically modifying base prompts from the existing llm-based embodied agent benchmark @xcite to introduce scene-task inconsistencies. specifically, we modify the two core components of each base prompt-the task description and the scene information-to create mismatches between user instructions and the observed environment (see figure 2 ). with the new hallucination inducing probing set, we conduct an empirical study of popular llms. overall, among our four controlled modifications, synonymous object substitution yields the lowest hallucination rates, suggesting that models can recognize objects conceptually belonging to the same category but often fail to maintain naming consistency, probably due to the inherent tendency of language models to favor varied phrasing over strict lexical alignment. in contrast, the highest hallucination rates occur under scene task contradiction (see figure 1 ), revealing the model's inability to ground planning to perceived environments. we also empirically study the effectiveness of mitigation strategies @xcite . the result shows that, even with feedbackbased self-correction, hallucinations in scene task contradiction remain high, underscoring a fundamental inability of models to recognize the infeasibility of the task. additionally, our small-scale experiments with vision-language models (vlms) suggest that hallucinations are reduced when both image and text inputs are available, emphasizing the importance of cross-modal verification. in summary, our contributions are as follows: ## methodology to conduct our study, we construct a new hallucination probing set by systematically modifying an existing embodied ai benchmark @xcite to introduce scene-task inconsistencies. ## conclusion in this paper, we systematically identify hallucination cases in llm-based embodied agents, offering insights into where and how current models fall short when executing long-horizon tasks under scene-task inconsistencies. the significantly elevated hallucination rates observed with our modified prompts highlight that existing models struggle to reconcile mismatches between user instructions and the environment. we further find that hallucinations persist despite feedback-based mitigation and are only partially reduced with cross-modal inputs. by introducing challenging scenarios along with guidance on ideal model behavior, our work takes an important first step toward enabling more grounded planning in real-world embodied agents.