Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks (2024)

Abstract: Evaluating the generalisation capabilities of multimodal models based solely on their performance on out-of-distribution data fails to capture their true robustness. This work introduces a comprehensive evaluation framework that systematically examines the role of instructions and inputs in the generalisation abilities of such models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity. The proposed framework uncovers the resilience of multimodal models to extreme instruction perturbations and their vulnerability to observational changes, raising concerns about overfitting to spurious correlations. By employing this evaluation framework on current Transformer-based multimodal models for robotic manipulation tasks, we uncover limitations and suggest future advancements should focus on architectural and training innovations that better integrate multimodal inputs, enhancing a model’s generalisation prowess by prioritising sensitivity to input content over incidental correlations.

## introduction designing artificial agents to follow natural language instructions-to understand and act within the context of their environment-is a long-term goal of artificial intelligence @xcite ). an artificial agent should generalise to unseen scenarios by combining concepts and skills underpinning its training data in novel ways @xcite . previous work which proposed several languageguided tasks for tackling this challenge, largely focused on generalising to environments with different scenes from the training ones (e.g., alfred; @xcite . however, relying solely on language for embodied action execution tasks can be inefficient, especially in collaborative settings with high ambiguity, such as visually cluttered scenes @xcite @xcite . multimodal prompts-instructions which interleave vision and language tokens-represent a way to specify commands which can be more flexible and specific than can be explained using text only @xcite @xcite . this capability is crucial for realistic humanrobot collaboration tasks and can be viewed as analogous to pointing at objects within a scene @xcite . for this reason, @xcite presented vima-bench, the first benchmark aimed at studying several axes of generalisation involving novel concepts and tasks, with models receiving instructions combining both language and visual referents. many other benchmarks test for generalisation by solely looking at held-out examples (open x-embodiment collaboration, 2024; @xcite . however, as highlighted by @xcite , generalisation should be evaluated across multiple dimensions when creating truly robust models, capable of performing safely in varied environments. inspired by these ideals, we assess generalisation along key axes such as structural, compositional, and robustness through specific covariate shifts (i.e., input perturbations) as outlined in figure 1 . specifically, we looked at 1) an extensive set of linguistic perturbations on instructions, such as paraphrasing, corrupting the language content, and replacing visual referents with language descriptions; 2) masking entire modalities within instructions; 3) introducing visual perturbations by permuting object order; and 4) increasing the difficulty of the tasks (e.g., placing distractors between source and target). we categorise each perturbation as either plausible (e.g., paraphrases) or unrealistic (e.g., nonsensical instructions). we expect models to be robust to plausible inputs while dropping performance when faced with unrealistic inputs. perturbations instruction observations plausible? hvi tsn rve fmot . gobbledygook words tablet protocols representation panasonic . gobbledygook tokens mask language tokens put the on the. mask visual tokens mask entire instruction put the letter r with blue and red stripes onto t just text put the on the . interleaving modalities drop the on top of the . paraphrases put the on the . distracting difficulty put the on the . extreme difficulty put the on the . extremely distracting difficulty put the on the . order permutation* figure 1 : our evaluation framework. each perturbation affects the instruction or observation inputs, which can be linguistic, visual, or a combination of both. the plausibility of a perturbation relates to a model's expected performance. sensitivity to unreasonable conditions ( ) indicates that a model should not perform the task successfully given the perturbation, while plausible perturbations ( ) suggest that it should still perform successfully. marks such as alfred @xcite , cliport @xcite , arnold @xcite , and ravens @xcite , provides several advantages: 1) it covers the majority of robotic manipulation tasks, 2) it offers more fine-grained levels for assessing the systematic generalisation of models; and 3) it represents a benchmark that allows for careful examinations of specific architecture and training regimes. for this reason, this paper builds on the controllability of vima-bench to extensively study the impact that properties of multimodal prompts and visual representations have on model performance. we applied our novel evaluation setup on multiple state-of-the-art architectures commonly used for different embodied ai tasks and datasets @xcite @xcite @xcite ). 2 we uncover several deficiencies of current "generalist agents" including 1) insensitivity to language perturbations, as they still perform several tasks when provided with gibberish instructions; and 2) inability to handle tasks of increasing difficulty, poten-tially including more distractors in the visual scene. these findings aim to shed light on state-of-the-art model performance and call for more research on systematically assessing model robustness with adequate tasks and settings that are indicative of the generalisation capacities of embodied ai models designed to safely and effectively complete tasks in the real world in collaboration with humans. language in robotic manipulation tasks language plays a crucial role in many embodied ai tasks, providing an interface for task learning @xcite , with many embodied ai tasks that require language instructions which are typically hand-crafted via templates (e.g., vima-bench, cliport, arnold) or crowdsourced (e.g., alfred). however, benchmarks often focus on evaluating generalisation using heldout episodes (open x-embodiment collaboration, 2024) and do not thoroughly evaluate the importance of language (octo model team, 2023; open x-embodiment collaboration, 2024; @xcite . for instance, models trained on alfred have shown to be insensitive to language instructions @xcite , while nonsensical instructions have even improved downstream performance on the vln benchmark @xcite . we focus on tabletop robotic manipulation tasks with natural language instructions to measure performance on a well-scoped action execution task. this allows for assessment of visual grounding capabilities from grounding instructions in the real world, while also removing the extra complexity of sophisticated skills (e.g., slam) required for navigation tasks @xcite or the need to predict fine-grained joint control by relying on inverse-kinematics @xcite @xcite . assessing generalisation and robustness embodied ai systems must generalise to any complex and novel tasks they might face @xcite , making robustness a highly-desired characteristic in models, illustrating how well they can ignore spurious correlations and generalise to new domains and tasks @xcite @xcite . embodied ai benchmarks often assess generalisation through seen/unseen scenes (e.g., @xcite @xcite , assuming that all tasks the agent must complete and the objects the agent must interact with are fully specified at training time. while recent benchmarks evaluate models on unseen objects and scenes @xcite @xcite , there is no notion of systematic or compositional generalisation to new concepts, affordances @xcite , or novel tasks @xcite . although models trained on realistic simulations can transfer learned behaviours to real-world environments (octo model team, 2023; open x-embodiment collaboration, 2024), they remain sensitive to distributional shifts in visual inputs @xcite , an issue that persists even when training data includes perturbations @xcite . furthermore, while models can adapt to relocated targets, they struggle with mid-trajectory linguistic shifts, such as swapping directions from "left" to "right" @xcite . our work extends these findings by examining model behaviour under extreme instruction perturbations, providing insights into how models handle challenging and unconventional scenarios. ## experimental setup evaluation data we use vima-bench to compare model performance across various skills, tasks, and levels of systematicity, as it is best suited for evaluating the role instructions play in generalising from multimodal prompts. 3 specifically, we assess the compositional generalisation capabilities at four distinct levels of systematicity @xcite : object pose sensitivity (l1), combinatorial generalisation (l2), novel objects (l3), and novel tasks (l4). see appendix b for environment and evaluation details. ## the evaluation framework we systematically perturb model inputs at test time to investigate the importance of visual and linguistic information in multimodal prompts. this approach helps us understand how input characteristics contribute to a model's task comprehension. full per-task results are reported in appendix f. ## conclusion we define an evaluation framework for embodied ai grounded in the generalisation framework from @xcite . specifically, we assess generalisation across important axes by means of specific multimodal input perturbations including paraphrases, replacing visual referents with descriptors, and manipulating the instruction syntax as well as entire input modalities. we instantiate this evaluation framework in vima-bench to assess the robustness of state-of-the-art models. overall, our findings indicate that while substitutivity can lead to performance gains, language perturbations do not impact performance as expected. to further explore this effect, we evaluate whether models rely on heuristics to complete tasks by removing individual modalities. we show that models perform tasks even without instructions by relying on spurious correlations within observations, as learned during training. we further prove this effect by showing that performance decreases when the number of objects in an environment increases, and agents can no longer randomly perform the correct sequence of actions. taken together, our findings suggest that it is important to define evaluation frameworks like ours that can assess generalisation across multiple axes in order to have a more reliable characterisation of the overall model performance. in future work, we aim to apply this evaluation framework systematically to other benchmarks as well to discover important architectural insights that will guide the next generation of embodied ai models.