Training Multi-Modal LLM s through Dialogue Planning for HRI (2025)

Abstract: Grounded natural language understanding in Human-Robot Interaction (HRI) requires integrating linguistic, visual, and world knowledge to ensure effective task execution. We propose an approach that enhances Multi-Modal Large Language Models (MLLMs) with a novel explicit dialogue planning phase, allowing robotic agents to systematically refine their understanding of ambiguous commands through structured clarification steps. This reduces hallucinations and improves task feasibility.To evaluate this approach, we introduce a novel dataset of over 1,100 annotated dialogues in English and Italian, designed for fine-tuning and assessing Multi-Modal models in HRI scenarios. Experimental results show that dialogue planning improves response accuracy and quality, and contributes to cross-lingual generalisation, enabling models trained in one language to transfer effectively to another. To the best of our knowledge, this is the first application of structured, goal-driven, and explicit dialogue planning in Multi-Modal LLMs for grounded interaction.

## introduction grounded natural language understanding in human-robot interaction (hri) presents significant challenges due to the need to integrate linguistic, visual, and world knowledge to ensure effective task execution @xcite . in real-world scenarios, a robotic agent receives multi-modal inputs: a natural language command from a human user and a perceptual representation of the environment, typically in the form of visual input, such as images or derived from more complex sensors, as in @xcite . however, human instructions may be underspecified or ambiguous @xcite , requiring an interactive dialogue where the agent actively requests missing information before taking action. recent advances in multi-modal large language models (mllms) have enabled the joint processing of textual and visual information @xcite . while these models were not originally designed for hri, they provide a promising foundation for robotic applications, allowing agents to interpret visual contexts and language simultaneously. hromei et al., 2024 have explored the application of mllms in multi-modal environments, such as the minecraft-inspired mm-iglu framework, where agents must reason over commands and visual scenes to understand if a command is executable or not. however, standard mllms lack reasoning capabilities for solving specific tasks: they generate responses based on local conversational history but do not construct a structured sequence of information-seeking steps, which is crucial in hri. this limitation risks leading to inefficient interactions, redundant queries, or hallucinated responses @xcite that do not align with the environment. recent research has demonstrated that incorporating planning mechanisms into llms can significantly enhance their robustness and generalisation capabilities. techniques such as chain-of-thought reasoning @xcite have proven beneficial in guiding models through complex decision-making tasks. in this work, we study the role of planning within a multi-modal setting, focusing on grounded interaction with a robotic agent operating in an environment similar to mm-iglu @xcite . figure 1 illustrates the structured dialogue planning mechanism employed in our approach. in a multi-modal human-robot interaction (hri) setting, a robotic agent receives an initial command from the user along with a visual representation of the environment. however, the command may be incomplete, requiring additional clarification before it can be executed correctly. instead of attempting an immediate response, the agent engages in an interactive process to gather missing details. at each turn, the agent processes three inputs: the current world state provided as an image, the accumulated dialogue history containing prior interactions, and the user's latest utterance. given this context, an mllm generates a structured plan identifying the missing information required for task completion. rather than providing an immediate response, the plan serves as a guide for the necessary clarifications before execution. once the plan is generated, it is incorporated into the ongoing interaction to guide the formulation of the next clarification question. the agent iterates through this process, dynamically updating its plan as new information emerges from the dialogue. the clarification questions systematically address ambiguities related to attributes such as the number of blocks, their colour, orientation, or precise positioning within the scene. when all necessary details have been gathered, the agent produces a final recap summarising the inferred command. this final step allows the user to confirm the interpreted instruction before execution, ensuring alignment between the intended and understood task. to this end, benchmarks @xcite have been created for evaluating their capabilities. to evaluate the effectiveness of our approach, we have created a dataset of more than 1, 100 dialogues in english, specifically designed for finetuning and evaluating multi-modal models in hri scenarios 1 . although many state-of-the-art models undergo pre-training on large-scale corpora of text and images, such pre-training does not necessarily equip them with the structured reasoning and dialogue capabilities required for interactive problem-solving. to demonstrate the broad applicability of our approach, independent of the underlying architecture, we apply it to two state-of-the-art multi-modal models, llava @xcite and minicpm @xcite , and demonstrate that: in the remainder of the paper, section 2 discusses related work, section 3 describes the proposed multi-modal architecture and dialogue planning approach, section 4 details experimental evaluation, and section 5 concludes our findings. ## related work effective multi-modal dialogue planning requires integrating structured decision-making into llms, enhancing model outputs with external knowledge, and systematically evaluating the impact of these mechanisms. our work builds upon three key research directions. first, explicit planning mechanisms have been explored to enhance structured reasoning in llms, but primarily outside the context of hri. prior studies have integrated planning into llms for tasks such as code generation @xcite , visual question answering @xcite , response structuring in textual dialogues @xcite , and incremental decision-making @xcite . while these works demonstrate the benefits of intermediate planning steps, they focus on structuring outputs in non-embodied scenarios rather than in interactive, multi-modal dialogue settings. a closely related line of work is dialfred @xcite , which extends the alfred benchmark @xcite to support dialogue-enabled agents capable of asking clarification questions when task instructions are ambiguous. unlike our approach, which introduces explicit plan generation to structure such interactions, dialfred equips agents with a questioning module but lacks systematic integration of planning steps to guide the dialogue evolution. in contrast, planning has also been applied in hri, but predominantly for action execution rather than dialogue structuring. for example, some studies have investigated planning-based decision-making for task execution in embodied agents @xcite @xcite , where models generate step-by-step action sequences to achieve a goal. however, these approaches do not address how planning can be leveraged within dialogue itself to systematically elicit missing information before taking action. second, multi-modal dialogue systems have been enhanced through external knowledge integration, improving response coherence by leveraging perceptual and linguistic information. previous work has focused on enriching dialogue models with implicit visual cues @xcite or persona-driven dialogue structuring @xcite . these contributions highlight the benefits of injecting multi-modal information but do not employ explicit dialogue planning to systematically structure interactions. in par-allel, emma @xcite proposes a unified encoder-decoder framework for embodied task completion in alexa arena @xcite , formulating both action execution and dialogue grounding as a text generation problem. while emma achieves strong results through multitask training and dialogue-guided execution, it does not incorporate an explicit planning module to structure or optimise the clarification process across turns. finally, evaluating dialogue quality remains a challenge, particularly for coherence, informativeness, and engagement. recent studies propose llm-based evaluation techniques @xcite , demonstrating that automated methods can provide reliable assessments of dialogue quality. although they are efficient and scalable, a key issue emerges regarding the reliability of evaluation: the final results may be affected by errors originating from both the system and the evaluator. our work builds upon these foundations by explicitly incorporating dialogue planning into mllms. inspired by prior work on multi-modal task execution (hromei et al., 2024), we extend the scope beyond single-turn decision-making to multiturn interactions, enabling systematic refinement of ambiguous commands through structured dialogue. this approach aligns with prior studies on collaborative dialogue in minecraft @xcite but goes further by introducing an explicit planning mechanism to optimise the interaction process: a unified dialogue plan representation, similar in spirit to the structured interaction histories in @xcite , where a multimodal transformer jointly models visual input and prior dialogue context for instruction-following. however, our method extends beyond reactive modelling by introducing an explicit, generative planning stage that precedes and guides the dialogue: the dynamic form of our approach allows the agent to induce different plans (or re-plan) on the fly, support online search for alternative situations emerging during the dialogue. ## a multi-modal neural architecture for dialogue planning in grounded hri prior approaches to multi-modal interaction in minecraft-like environments, such as mm-iglu @xcite , only considered simple oneturn dialogues. the builder could ask only a single clarification question before executing the command, and questions were drawn from a closed set with limited linguistic variability. this constraint often led to suboptimal task performance, as the system lacked mechanisms to iteratively refine its understanding of the user's intent. to overcome these limitations, we introduce a novel dialogical dataset tailored for multi-modal environments and propose a methodology for training multi-modal language models to handle more dynamic and adaptive conversations. our approach defines structured information categories that the assistant should inquire about, enhancing its ability to formulate clarification questions systematically. inspired by self-reflection @xcite and chain-of-thought reasoning @xcite , our system first generates an information plan to guide the dialogue, identifying missing details, then iteratively refines its understanding by formulating targeted questions. additionally, the agent must continuously reassess dialogue coherence, detecting inconsistencies and dynamically adjusting its strategy as needed. in the following, we present the multi-modal architectures employed in our approach, the underlying dialogue planning mechanism, and the methodology used to construct and train the dialogue models. ## experimental evaluation our evaluation examines the impact of explicit dialogue planning in multi-modal llms by addressing three key aspects: i) whether fine-tuning on a dedicated dataset improves performance over zero-shot inference, ii) how the dialogue planning phase enhances dialogue efficiency at both the turn and full dialogue level, and iii) whether the dialogue planning phase facilitates cross-lingual generalisation. to assess the effects of fine-tuning, we compare zero-shot performance against models fine-tuned on our annotated dataset. for dialogue efficiency, we contrast models trained solely on dialogue history with those incorporating structured dialogue planning, measuring improvements in response quality, reduction of redundant queries, and overall interaction effectiveness. the dataset (see an 80-10-10 ratio. each dialogue is available in both english and italian to enable controlled crosslingual evaluations. the inclusion of a manually translated italian version ensures that crosslingual assessments measure generalisation effects rather than language-specific variations. we assess whether planning enhances cross-lingual robustness by evaluating model transfer between english and italian, testing if structured planning improves performance across languages. turn-driven evaluation. this evaluation focuses on the model's ability to generate responses at the individual turn level. the agent (llm) must determine whether additional clarification is needed before execution. if necessary, it generates a clarification question (q); otherwise, it provides a recap (rc) summarising its understanding of the task. to assess these aspects, we evaluate clarification questions and recap generation separately. to examine the impact of fine-tuning, we test two state-of-the-art multi-modal models with a compact number of parameters: llava @xcite and minicpm @xcite . these models are evaluated in a zero-shot setting, where they rely solely on a system prompt, and in a finetuned setting, where they learn to generate structured plans before responding. for plan generation, the model receives a context consisting of the initial task-defining prompt (see appendix a), the dialogue history up to that turn, and an image. we assess the alignment of generated plans with expected outputs using precision, recall, and f1score for the plan categories defined in section 3. for clarification questions and recap generation, output quality is measured using cosine similarity between the embedding vectors of generated responses and expected outputs, to better capture the semantic and pragmatic quality of responses in our highly contextual, interactive hri scenario: notice that rather than simple text generation (as in translation), we focus on suitable linguistic interactions as capable to preserve all inferences needed by the underlying hri tasks (e.g. posing questions such as potential paraphrases). this capability is very weakly modelled by machine translation metrics such as bleu or rouge. this is computed using the sentence-bert model (reimers and gurevych, 2019) foot_1 . higher cosine similarity indicates greater syntactic and semantic alignment with the expected clarification and recap. fine-tuning was conducted on four a100 gpus using a total batch size of 8 and a learning rate of 5 • 10 -5 . complexity reduction techniques such as lora (rank r and α set to 64) were applied to reduce training costs in terms of time and energy, to the attention and output layers of the language decoding module. moreover, we continued the joint fine-tuning of the vision encoder as well, as detailed in the minicpm paper @xcite . to further assess the impact of explicit planning beyond fine-tuning alone, we conduct a detailed evaluation comparing minicpm in settings with and without structured planning. given its consistently superior performance across previous experiments, we focus exclusively on minicpm for this analysis. figurations: zero-shot, fine-tuning on dialogue history alone, and fine-tuning with explicit planning. in the monolingual @xmath0, fine-tuning without planning significantly improves generation quality compared to the zero-shot model (e.g., question generation rises from 0.548 to 0.768, while recap generation increases from 0.706 to 0.921). however, the addition of planning further boosts performance, with question generation reaching 0.843 and recap generation improving to 0.955. these findings confirm that fine-tuning alone enhances response quality, but planning refines the interaction by making clarification requests more structured and summaries more accurate. we also examine cross-lingual generalisation, where the model is trained in one language and tested in another. the results reveal that models finetuned only on dialogue history experience a notable drop in question generation accuracy when applied across languages (@xmath1: 0.659, @xmath2: 0.638). this degradation suggests that, without structured planning, the model struggles to transfer the reasoning process necessary for effective question formulation. in contrast, incorporating planning significantly mitigates this issue: the @xmath3.735, and the @xmath4.738, both exceeding the performance of the history-only model in monolingual settings. interestingly, recap generation remains more stable across settings, with smaller differences between monolingual and cross-lingual results. this indicates that summarisation is a more transferable skill, whereas effective clarification question generation benefits more directly from structured planning. these findings underscore the role of explicit planning not only in improving dialogue quality but also in strengthening the model's ability to generalise across languages, reinforcing the importance of structured multi-modal training for robust cross-lingual hri applications. ## conclusion this work explored the integration of explicit dialogue planning in multi-modal large language models for human-robot interaction. we proposed a structured planning mechanism to enhance an agent's ability to gather task-relevant information through dialogue before executing user commands. to support this investigation, we introduced a dataset of over 1, 100 annotated dialogues in english and italian, enabling fine-tuning and cross-lingual evaluation. our experimental results demonstrate that finetuning significantly improves performance over zero-shot inference, while explicit planning further enhances response accuracy, reduces redundant exchanges, and improves overall dialogue efficiency. additionally, we show that planning facilitates cross-lingual generalisation, allowing models trained in one language to transfer effectively to another. in this way, we contribute a novel benchmark and methodology to advance future research on grounded, multi-modal dialogue systems for hri. future directions include investigating alternative planning strategies, such as integrating planning into a single-step generation process, exploring more advanced learning paradigms like reinforcement learning, and extending this approach to less structured environments with dynamically evolving configurations. first, the proposed planning mechanism relies on predefined clarification categories, which may not generalise to all hri scenarios. although these categories improve dialogue efficiency, they impose a structured framework that might limit the flexibility needed for open-ended or highly dynamic interactions. second, our dataset consists of dialogues set in a block-based, synthetic environment with static world configurations. while this controlled setting allows for systematic evaluation, real-world hri scenarios often involve environments that change over time and require continuous adaptation. extending this approach to more complex, evolving contexts would be a natural next step. third, the planning step is currently treated as a separate generation process. while this modular approach facilitates structured reasoning, it may introduce additional latency. future work could explore integrating planning directly into the response generation step, potentially reducing inference time while maintaining structured dialogue management. finally, our experiments focus on fine-tuning mllms, but alternative training paradigms, such as reinforcement learning, could further refine planning strategies and enable models to adapt dynamically based on user interactions. investigating such approaches could improve long-term engagement and generalisation in interactive systems.