ACT -Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments (2022)

Abstract: Artificial agents are nowadays challenged to perform embodied AI tasks. To succeed, agents must understand the meaning of verbs and how their corresponding actions transform the surrounding world. In this work, we propose ACT-Thor, a novel controlled benchmark for embodied action understanding. We use the AI2-THOR simulated environment to produce a controlled setup in which an agent, given a before-image and an associated action command, has to determine what the correct after-image is among a set of possible candidates. First, we assess the feasibility of the task via a human evaluation that resulted in 81.4% accuracy, and very high inter-annotator agreement (84.9%). Second, we design both unimodal and multimodal baselines, using state-of-the-art visual feature extractors. Our evaluation and error analysis suggest that only models that have a very structured representation of the actions together with powerful visual features can perform well on the task. However, they still fall behind human performance in a zero-shot scenario where the model is exposed to unseen (action, object) pairs. This paves the way for a systematic way of evaluating embodied AI agents that understand grounded actions.

## introduction recently, embodied agents have been increasingly proposed and evaluated on their capacity to navigate virtual environments, and execute actions within them according to instructions @xcite @xcite . action execution as a means of evaluating language understanding is a long-standing idea (e.g. @xcite ) which has now become feasible thanks to the release * equal contribution † work performed while at the university of trento of various virtual environment platforms @xcite @xcite . thus, the study of embodied agents is an interesting venue to push forward due its both theoretical and practical appeal. however, fully embodied tasks involve a variety of issues, making it hard to understand where we stand with respect to the challenges posed. here, we focus on one core aspect of such embodied tasks: action grounding. @xcite advocate for embodied language understanding as a necessary step to overcome the limitations of the mainstream deeplearning paradigm. among the priorities of their agenda is evaluating models' ability to simulate the effects of actions. evidence from neuroscience favors an embodied meaning of words and specifically of verbs; for instance, the natural theory of language @xcite is based on the assumption that people understand language by imagining (or simulating) the situation being described. in this paper, we focus on grounded verb understanding and predicting the visual transformation their corresponding action causes. more specifically, we consider the idea of verbs as functions @xcite , and see them as transformations of an object within a scene before and after the verb/action is executed. to achieve this, we create a dataset combining ideas and methods from two main papers about learning physical commonsense knowledge @xcite . as in @xcite , we focus on the action-effect relation, namely the physical change that an action causes when executed on an object. and, as in @xcite , we leverage a virtual environment, ai2-thor @xcite , to generate transitions between object states. in sum, we propose act-thor, a dataset for learning the effects of actions, generated using a virtual environment. 1 for each example in our dataset, we collect an image of an object, a label describing an action performed on the object, and an after-image resulting from executing the action (figure 1 ). thus, unlike in @xcite and @xcite , both the before and after world states are images; the linking action acts as a function transforming the former into the latter. then, via the use of images whose generation is carefully controlled, we design a task that intends to capture a challenge embodied ai agents will face in a real setting. out of this collection of image pairs and their corresponding action-label, we design a prediction task based on carefully selected contrast sets containing candidate images representing the effect of other actions performed on the very same object in the same scene (figure 2 ). agents that truly ground verbs, given the before-image and the action label, should be able to select the after-image out of the contrast set. we also evaluate human performance on a sample of the dataset. annotators achieve high accuracy and inter-annotator agreement, evidencing the quality of the design method we propose, and setting an upper bound for multimodal models. we evaluate baseline models based on state-ofthe-art visual features @xcite . we put together ideas and results from different disciplines to shed light on different ways to study action in neural models and pave the way for future research on action grounding. from formal semantics and computational semantics, we take the view of verbs as functions @xcite , and therefore as matrices @xcite and learn, for each action, a matrix able to transform the before-image, into the corresponding after-image. we compare this model with multimodal neural network models of increasing complexity. our experiments show that • virtual environment platforms can be used to generate interesting diagnostic datasets of static images having certain properties and satisfying certain constraints; ## related work actions/verbs have been studied from a variety of perspectives. they have been the subject of thorough investigation in formal semantics @xcite , and have been the focus of widely-used annotated resources @xcite . multimodal approaches to action/verb learning have been especially fruitful. @xcite learn to ground visual representations of actions to formal semantic representations thereof. prior studies have connected visual action representations to concepts such as intention @xcite and causality @xcite @xcite . moreover, textbased models of similarity between actions have been shown to substantially improve when text is combined with visual information @xcite . the breadth of these works suggests the importance of verbs and actions for the development of embodied agents and natural language understanding more broadly. we contribute to this line of research by presenting a new technique for creating datasets that evaluate computational models for action grounding. we build on ideas introduced in @xcite by leveraging a virtual environment platform: rather than filtering and annotating images from the web via crowdsourcing, we generate a controlled dataset using an agent in a virtual environment. within the nlp community, predicting the consequences of actions is often framed as part of learning commonsense knowledge. we share with @xcite a focus on what they call naive physical action-effect relations, namely the causal relation between an action (verb) and the change in physical world state caused by that action. similarly, we propose an "effect prediction" task in which the model predicts an effect (an image) given the action and the object on which the action is performed. in their task, the action and object pair are expressed as verbs and nouns, whereas in our case the noun is presented as a (before) image; hence the model is challenged to learn the visual transformation objects undergo through the action execution. @xcite use ai2-thor to build the pigpen dataset, from which models can learn world dynamics. they do so by letting a virtual agent navigate and interact with its environment to solve certain tasks. however, when building these trajectories, the robot's visual perspective (an image) is ignored in favor of a symbolic representation of the world state (i.e., as a set of visual attributes). hence, models trained on pigpen receive symbolic and natural language inputs representing world information, and produce outputs in the same modality. to achieve our goal, we propose studying the physical action-effect relation highlighted by @xcite , utilizing the image generation approach used in @xcite . finally, as in @xcite , we implement this concept as a classification task. however, inspired by @xcite , we adopt a contrastive setup. our use ## datasets to build our dataset, we generated images using ai2-thor foot_1 , the virtual environment platform introduced in @xcite . in ai2-thor, the user controls a robot as it navigates a room of a virtual house environment, filled with household objects. while navigating the rooms, the robot can interact with objects it encounters by performing actions on them. position and state metadata are available for the robot and all objects in the scene. we use the robot to generate an image representing a scene before and after an action is taken. in the following section, we describe the image generation process, which led to the collection of triples consisting of before and after-images and their linking action. then, we describe act-thor, a dataset built out of this collection to evaluate models' abilities to simulate the consequences of actions on objects situated in virtual scenes. ## models in this section, we describe baseline models designed to assess the role of the embeddings involved: those of the action-label, before-and afterimages. we compare models using frozen visual features extracted from various state-of-the-art feature extractors. 4 training method considering the contrastive nature of our task, inspired by clip's loss function @xcite , we use the infonce loss proposed by @xcite . this loss minimizes the distance with the ground-truth vector while jointly maximizing distances from the "negative" elements in the contrast set (i.e. the ones generated from different actions). let @xmath0, c 2 , ..., c n } , k) be a contrast set where c i are the vector representations for each image, and k is the index of the true after-image. furthermore, let ã be the predicted after-image representation. then we compute the probability that image i is the true after-image as follows: the training loss is l(c, ã) = -log(p(k|c, ã)), cross-entropy loss. at test time, our model predicts the true after -mage index as argmax i p(i|c, ã). the models presented in the following paragraphs all use the approach described above, with the exception of the visual-only model, for which there is no training time. unimodal baselines inspired by @xcite , we verify the impact of visual and language input by considering unimodal baselines. first, we evaluate a model that learns to select as after-image an image of the contrast set whose visual vector is closer to the before-image, based on cosine similarity (hence, visual-only.) in this work, we explore two different visual extractors: 1) moca: we rely on the visual encoder pretrained using al-fred data by @xcite ; 2) clip: we use the visual features extracted by the clip image encoder @xcite . we use the former because it has been extensively used for other embodied ai tasks (e.g., @xcite )) while we use the latter because @xcite showed that clip provides very effective representations for embodied ai tasks. second, we evaluate a model that learns to select the after-image simply based on the action name (action-name). the model architecture is simple: first, the action label is converted into a (learned) embedding. next, each of the afterimages is turned into a feature vector using one of the aforementioned frozen pre-trained feature extractors. then, both the action embedding and each of the after-image feature vectors are projected into a shared space via linear projections. the projected action embedding is taken to be ã, and the appropriate projected image representations to be c 1 , . . . , c n . then training and prediction are performed as described above. multimodal baselines we design three multimodal baseline models. the first of them departs from the mainstream end-to-end approach used in embodied ai, implementing the formal semantics view of "verbs as functions". following baroni and zamparelli (2010), we represent each action as a matrix that is learned from its beforeand after-images. unlike @xcite , we rely on a feed-forward network with a single linear layer, with no bias vector or activation function (hence, action-matrix). given the before-image representation (derived from either moca or clip), we use the action identifier to extract the associated action matrix, which is composed with the before-image vector through matrix multiplication to return the predicted after-image vector. 5 in addition, we design two models trained endto-end to select the after-image given the beforeimage representation and the action name represented as a vector that belongs to a learned embedding matrix. the two variants of this architecture are: concat-linear: a single matrix, acting as a linear layer, which is multiplied by the concatenation of before vector and action vector; and concat-multi: a 2-layer, feed-forward network, including batch-normalization, dropout and activation layers, taking as input the same concatenated vector as above. unseen objects. given the set of object classes o, we split it into a seen subset o s , and an unseen subset o u . we define the training and validation splits so that they contain objects that belong to o s only. on the other hand, we define the test split so that it contains only objects belonging to o u . we manually selected the unseen objects (laptop, cup, and book) so as to guarantee the complete coverage of actions at test time. the test set contains 908 datapoints, the training and validation sets amount to 2762 and 690 datapoints, respectively. 8 this could not evaluate it. 6 the standard practice in ai2-thor is reported in the official documentation. 7 detailed statistics in supplementary material ( ## conclusion in this work, we define act-thor, a controlled benchmark for embodied action understanding in simulated environments. compared to similar benchmarks presented in the literature, we propose a more systematic benchmark for studying the capability of multimodal machine learning models to understand the effects of actions on the objects. additionally, our contrast set formulation allows us to precisely pinpoint the distinctions that models should learn, providing a robust and reliable benchmark for studying action grounding. to assess the quality of our dataset, we first completed a human evaluation demonstrating that humans can complete this task with 83% accuracy. then, we evaluated several unimodal models, and multimodal models using several state-of-the-art visual feature extractors such as clip @xcite . by inspecting models that learn a matrix for each action, we show that the representations learned via our dataset favor the emergence of clusters of actions associated with a change of position, and actions that cause a change in object appearance. these are two salient visual transformations objects undergo, but more could be learned by extending the dataset with other actions or other objects, and by proposing controlled settings inspired by the theoretical view on verb semantics.