Learning the Effects of Physical Actions in a Multi-modal Environment (2023)

Abstract: Large Language Models (LLMs) handle physical commonsense information inadequately. As a result of being trained in a disembodied setting, LLMs often fail to predict an action’s outcome in a given environment. However, predicting the effects of an action before it is executed is crucial in planning, where coherent sequences of actions are often needed to achieve a goal. Therefore, we introduce the multi-modal task of predicting the outcomes of actions solely from realistic sensory inputs (images and text). Next, we extend an LLM to model latent representations of objects to better predict action outcomes in an environment. We show that multi-modal models can capture physical commonsense when augmented with visual information. Finally, we evaluate our model’s performance on novel actions and objects and find that combining modalities help models to generalize and learn physical commonsense reasoning better.

## introduction large language models (llms) are trained on large corpora of disembodied texts. they are typically pre-trained on a masked language modeling task: the model must predict a masked word in a text given its context. llms have achieved stateof-the-art performance on many nlp tasks @xcite ), but they can also fail on seemingly easy and obvious tasks and in unpredictable ways @xcite . commonsense knowledge is shared knowledge and is often so obvious that it is absent from the llms' training data: people don't mention what is already known to their interlocutors. this includes physical commonsense information, including how executed actions affect the physical attributes of objects; e.g., shape and weight @xcite . humans may learn such knowledge from their embodied environment. but llms, being trained on disembodied text, can make incorrect predictions about physical attributes and how these change when actions occur. for instance, when asked what the weight of a 150 grams potato after it is sliced, gpt-3 @xcite incorrectly answers 75 grams (see appendix a for the exact prompt). gpt-3 is an llm with 175 billion parameters, and nonetheless its disembodied existence limits its physical commonsense estimates. @xcite inject physical commonsense information into llms via their model piglet-a modified llm that is trained on their pigpen simulated 3d environment dataset. piglet estimates how an environment changes as a result of specific actions. in training and testing, the model uses ground-truth symbolic representations of the environment but not the images: it ignores visual sensory observations. these symbolic representations of objects in an environment are chosen to capture the possible effects of actions, and include attributes like weight, size and temperature. however, in an embodied situation, an agent needs to use visual perception to estimate its interpretation of the scene. therefore, the symbolic representations should be treated as latent rather than observed. we propose an alternative to the piglet model, piglet-vis, which uses images directly as input into a multi-modal llm to ground the model to its physical environment. we compare our approach to the original piglet model and evaluate the generalization capabilities gained from using image inputs. at test time, our model foregoes symbolic labels: only the images and the name of the action are observed. thus our model tackles a more challenging task than the original piglet model in that it must not only predict the effect of actions but also (indirectly) estimate the symbolic representations of objects in the images. we also evaluate a model for predicting the effects of actions that trains on pigpen's images and their associated natural language (nl) descriptions, eliminating the need for @xcite . during pre-training the model receives as input the full symbolic representation of two objects (o 0 pre and o 1 pre ) before the action is taken and the symbolic representation of the action itself (a) and is tasked with predicting the attributes of the objects after the action (o 0 post and o 1 post ). during fine-tuning, the action encoder is replaced by an llm to process a natural language description of the action being taken and with what objects. our contributions are three-fold. first, we show that it is possible to predict the physical effects of actions from visual data. second, we show that it is possible to learn the task on training data where formal symbolic representations, which are unobservable in real-world settings, are replaced with nl descriptions (which can be observed through natural interaction). third, we evaluate all our models in a stricter zero-shot setup to promote ways to train agents that generalize. overall our work paves the way for multi-modal models that learn the effects of actions in realistic environments. ## related work commonsense reasoning has been highlighted as a potential weak point of llms in recent years @xcite @xcite . datasets such as pigpen @xcite , commonsenseqa @xcite , vcr @xcite and gd-vcr @xcite help evaluate different aspects of commonsense reasoning in modern llms. in this paper, we focus on physical commonsense reasoning, which involves understanding the (often) unexpressed rules of the physical world. @xcite reported that neural representations found it challenging to infer the link between actions and what they imply about the attributes of objects. accordingly, @xcite introduced the visual commonsense reasoning (vcr) task to test how images can inform question answering models that tackle commonsense information. @xcite designed the piqa benchmark to evaluate physical commonsense reasoning in llms through question answering. @xcite proposed an extension to the clevr dataset, where an agent must reason and answer questions about a scene after a hypothetical action is taken. multiple approaches can improve the capabilities of llms in commonsense reasoning, such as using handcrafted knowledge graphs @xcite or leveraging simulated environments @xcite . piglet, in particular, combines a traditional llm and a "physical dynamics" model to ground an llm @xcite . the physical dynamics model enhances the commonsense knowledge of an llm by fine-tuning it, using trajectories sampled from a realistic environment (see figure 1 ). trajectories are an action and a pair of environment states (before and after the action) expressed in a formal symbolic representation. @xcite found that fine-tuning llms with symbolic data from the simulated environment helped them outperform other models in physical commonsense reasoning tasks: in particular, predicting the effects of an action when executed in a particular state. image inputs offer a way to ground an llm, as they only require general alignment with a text or symbolic input and do not require the comprehensive environment ground-truth labels that piglet uses. @xcite used multi-modal web data to learn actions and their effects from images action apply object decoder action @xmath0, @xmath1." 1 2 observed latent 1 2 ... ... ... name: cup size: small filled: no figure 2 : piglet-vis. we introduce piglet-vis, where we modify the piglet architecture to replace its symbolic object encoder with a vision component that makes use of images of the environment before and after an action is taken to predict the symbolic representation of objects post-action. we use an attention mechanism over the extracted bounding boxes to obtain a visual hidden representation of an object given its name. the only remaining symbolic inputs during pre-training are the action description and object names. and corresponding text descriptions. @xcite used an off-the-shelf resnet50 model @xcite to augment an existing bert language model @xcite with vision capabilities. transformer models such as uniter @xcite , ernie-vil @xcite , visualbert (li et al., 2020), and vilbert @xcite have been applied to visual commonsense reasoning. these models use a joint transformer backbone for images and text and vary their pre-training objectives. however, most of these models are trained on static text-image pairs: they aren't designed to capture the dynamics of an environment, particularly how object attributes change with actions. notably, recent work by @xcite uses clip @xcite and moca @xcite embeddings to predict a post-action image given a set of possible images. in contrast, we focus on adapting an llm with a vision-based component to predict the consequences of actions on the environment. ## method we propose piglet-vis (figure 2 ) for learning the effects of actions on objects from images. we use a pre-trained vision backbone, detr @xcite , as a vision object encoder and combine it with a roberta llm @xcite as an action encoder. we experiment with different configurations of inputs to measure the impact of the various components of our architecture. in particular, we test a variation in which we remove the formal symbolic labels even in training, replacing them with nl text labels. to evaluate our models, we use the pigpen dataset @xcite , which consists of a symbolic and visual representation of an environment before and after an action is taken. however, we filter pigpen to create a viable testing ground for visual grounding of physical actions and more accurately measure generalization capabilities of models. ## results and discussion we evaluate all models on our pigpen-vis split and report the overall (exact match), zero-shot, action-level, and attribute-level accuracy results for all setups in tables 1 and 2 . for completeness, we also evaluate models on the original pigpen to contrast the effects of our filtering operations (see §3.4 and appendix d) and find pigpen-vis is a more challenging subset for all models. the base model provides a low bar estimate of what is achievable using only the action encoder inputs. unsurprisingly, the base model performs worst on overall accuracy, which demands an exact match of all attributes. it does relatively well on (individual) attribute-level accuracy, primarily because it predicts the most common attribute for each object. some actions are also easier than others-for instance, the model reaches 27.38% accuracy on toggleon from only knowing the action and object names. this is likely because toggleon is constrained to a small set of objects and effects. our base+symbolic model obtains similar results to the original implementation by @xcite , with an overall accuracy of 85.03%. however, it performs much worse on the zero-shot split (39.04%) than the original piglet model reported (80.2%) @xcite . this disparity can be explained by the fact that the original zero-shot pigpen dataset was not a true zero-shot dataset, because the physical dynamics model was exposed to the "unseen" objects in its pre-training. the base+symbolic model provides a high bar estimate of what could be achievable if: (i) i pre and i post capture the symbolic environment; and (ii) the vision object encoder can subsequently extract these features. however, as we will argue in section 6, both (i) and (ii) are unrealistic given the constraints of both the dataset and the model. our base+images (piglet-vis) model scores 45.28% in overall accuracy but only 7.53% on the zero-shot set. nevertheless, it outperforms the base model in overall accuracy (@xmath2.0001) and in zero-shot accuracy (@xmath3.08), which demon-strates that the images improve the prediction of the effects of actions. the base+images model also performs significantly better than base on difficult attribute-level accuracies such as distance (@xmath4.0001). however, as before, accuracy on individual attributes benefits from the skewed distributions of their values and does not necessarily translate to high scores on predicting all 38 attributes correctly. utilizing both images and symbolic representations as inputs helps the base+symbolic+images model outperform purely symbolic inputs in overall accuracy, from 85.03% to 86.01% (@xmath5.01). however, image inputs also decrease the model's zero-shot performance from 39.04% to 35.89%, although this isn't statistically significant (@xmath6.05) due to high variance. we suspect that this high variance is caused by an increase in noise in the model resulting from adding images to the symbolic model. however, the overall picture is more complicated, as images can also provide gains on certain actions (e.g., pickup accuracy increases from 80.48% to 86.14%) even though it causes a decrease in many other cases (e.g., toggleon). finally, when we utilize nl descriptions to replace the formal symbolic inputs (action name and object names), base+images+text-labels improves overall accuracy when compared to base+images from 45.47% to 47.55% (@xmath7.02). text inputs appear to improve zero-shot accuracy, but not by a statistically significant margin (@xmath8.31). accuracy also improves in most actions, for instance the slice accuracy improves from 41.64% to 45.57% (@xmath9.03). so the nl descriptions inform the task in a beneficial way, over and above the raw images. but encoding the labels as text rather than formal symbolic representations also adds noise. nevertheless, text labels improve accuracy on actions where the semantic information contained in the label provides a richer context to help generalize to similar objects. for instance, a "cup" and a "mug" are semantically close, and thus learning the effects of actions on a "cup" might help the model predict the same effects on a "mug" even if the word forms are different. in contrast, the formal symbolic representations treat the predicate symbols cup and mug as unrelated, and so don't benefit from the lexical relationships that the llm captures. fully removing the symbolic representations allows us to adapt our model ## conclusion in this paper, we tackle the task of predicting the effects of actions on objects' physical attributes. in contrast to @xcite , our model does not treat the formal symbolic representation of the images as observed. instead, piglet-vis supports inference when the inputs are images alone or images plus nl descriptions and a phrase denoting the action (e.g., "the robot empties the cup"). while pigpen offers challenges for applying a multimodal approach, our model can extract useful information from images, opening the door for generalizing learning physical commonsense to realworld data. importantly, our pigpen-vis split can be used to evaluate the zero-shot capabilities of different model configurations. moreover, while base+symbolic still outperforms base+images, it does so without estimating the attributes of ob-jects and thus solves a much easier but unrealistic task. through base+images+text-labels, we show that, when replacing symbolic inputs, the best solution is to complement image inputs with nl descriptions to leverage information from both modalities. finally, our results show the need to improve the generalization capabilities of multimodal models such that they can learn and adapt to unseen situations.