On the Limits of Evaluating Embodied Agent Model Generalization Using Validation Sets (2022)

Abstract: Natural language guided embodied task completion is a challenging problem since it requires understanding natural language instructions, aligning them with egocentric visual observations, and choosing appropriate actions to execute in the environment to produce desired changes. We experiment with augmenting a transformer model for this task with modules that effectively utilize a wider field of view and learn to choose whether the next step requires a navigation or manipulation action. We observed that the proposed modules resulted in improved, and in fact state-of-the-art performance on an unseen validation set of a popular benchmark dataset, ALFRED. However, our best model selected using the unseen validation set underperforms on the unseen test split of ALFRED, indicating that performance on the unseen validation set may not in itself be a sufficient indicator of whether model improvements generalize to unseen test sets. We highlight this result as we believe it may be a wider phenomenon in machine learning tasks but primarily noticeable only in benchmarks that limit evaluations on test splits, and highlights the need to modify benchmark design to better account for variance in model performance.

## introduction language guided embodied task completion is an important skill for embodied agents requiring them to follow natural language instructions to navigate in their environment and manipulate objects to complete tasks. natural language is an easy medium for users to interact with embodied agents and effective use of natural language instructions can enable agents to navigate more easily in previously unexplored environments, and complete tasks involving novel combinations of object manipulations. vision and language navigation benchmarks @xcite @xcite provide an agent with natural language route instructions and evaluate their ability to follow these to navigate to a target location. it requires agents to have a deep understanding of natural language instructions, ground these in egocentric image observations and predict a sequence of actions in the environment. other benchmarks study the manipulation and arrangement of objects @xcite @xcite -another crucial skill to complete many tasks that users may desire embodied agents to be able to complete. these tasks additionally require agents to reason about the states of objects and relations between them. language guided embodied task completion benchmarks @xcite @xcite combine these skills -requiring agents to perform both navigation and object manipulation/arrangement following natural language instructions. in this paper, we explore a challenging navigation and manipulation benchmark, al-fred @xcite , where an agent has to learn to follow complex hierarchical natural language instructions to complete tasks by navigating in a virtual environment and manipulating objects to produce desired state changes. the alfred benchmark provides a training dataset of action trajectories taken by an embodied agent in a variety of simulated indoor rooms paired with hierarchical natural language instructions describing the task to be accomplished and the steps to be taken to do so. for validation and testing of models, there are two splits each -seen and unseen splits. the seen validation and testing splits consist of instructions set in the same rooms as those in the training set, while the unseen splits consist of instructions set in rooms the agent has never seen before, with rooms in the unseen test set being different from those in the train and unseen validation set. performance on the unseen validation and test sets are considered to be the best indicators of whether a model can really solve the task as the agent must operate in a completely novel floorplan, and cannot rely for example on memorized locations of large objects such as a fridge or a sink. additionally, the ground truth action sequences are not publicly available for the seen and unseen test sets, and participants must submit prediction acted sequences on the test sets to an evaluation server where they are privately evaluated to obtain test performance. the evaluation server limits the number of submissions that can be made from an account to one per week to discourage directly tuning hyperparameters of a model on the test set. it is expected that following standard procedure in training machine learning models, one may use the validation sets to evaluate models trained with different hyperparameters, or ablating different components on the validation sets and only evaluate the best model on the test sets. since ideally we would want a model to perform well on the unseen test set, it is reasonable to use success rate on the unseen validation set as a metric to choose which model is to be submitted for evaluation on the unseen test set. one technique previously demonstrated to improve performance on alfred is the use of a multi-view setup @xcite where an agent turns or moves its head in place at every time step to obtain additional views before deciding what action to take. in contrast to current models that simply concatenate features from each view, we use view-action matching -explicitly aligning embeddings of actions with embeddings of corresponding views -and using a score from fusing these aligned embeddings to select the next action to be taken. this is inspired by a dominant paradigm for modeling visual navigation tasks called viewpoint selection @xcite where an agent predicts the next action by examining the resultant views each of those would produce and selecting the desired future view. viewpoint selection is possible in some simulators such as r2r where the environment does not get altered by the agent's actions and the agent's movement is confined to a fixed grid. the alfred dataset uses the ai2-thor simulator which supports a wider action space, physics modeling for movement and a more dynamic environment including irreversible actions. hence, it is not possible to obtain the view that would result from an action without taking it, preventing direct application of viewpoint selection. additionally, the agent must decide at each time step whether to perform navigation or manip-ulation actions. in contrast to prior work that uses a single classifier layer over all possible actions treating them equally, we propose a gate module which gives a higher weight to actions of a more relevant action type. we follow standard experimental procedure training our modified models on the train split and using success rate on the unseen validation split to compare to baselines and perform ablation studies. on this set, the proposed model equipped with the aforementioned modules outperforms the stateof-the-art multi-view setup approaches and the ablation study shows each proposed module helps improve the model's performance. however, we observe an unexpected and large performance gap between the unseen validation and test data splits. our model outperforms stateof-the-art baseline models on the unseen validation split, but performs worse than them on the unseen test split. we hypothesize that it may be possible to overfit hyperparameters and design choices to one set of unseen environments (the unseen validation) and hence success on one such set of unseen environments is insufficient to guarantee that a model will generalize to another set of unseen environments (the unseen test). we report this finding as we believe this situation is likely more common during development on machine learning benchmarks, but such intermediate results are unlikely to be published. instead after a poor result on a test set, it is likely that researchers continue further model modifications until a model setting is obtained that performs well on the test set. we believe that such models are likely overfitting to the test set of the benchmark and may not generalize well to a new test set. ## dataset & environment in this paper, we focus on improving models for the alfred @xcite benchmark. alfred is built using the ai2-thor simulator @xcite which consists of 120 indoor scenes across 4 types of rooms. scenes also contain a diverse set of objects that are rearranged in different configurations for each trajectory in the dataset. in alfred, a agent is given a high level natural language goal statement ("put a chilled pan on the counter") as well as step by step natural language instructions corresponding to subgoals to be completed in order for achieving the goal ("turn around and cross the room and then go right and turn to the left to face the stove ... put the pan down on the counter to the right of the toaster"). an agent has access to all these instructions at the start of the task and then has to iteratively predict navigation and manipulation actions in the environment based on egocentric image observations to complete subgoals in order. an agent must predict between a discrete set of possible navigation and manipulation actions, and predict a segmentation mask for the object to be manipulated if a manipulation action is predicted. the performance for an agent is evaluated by comparing the final states of the objects at the end of the action trajectory executed by the agent to the states of the objects at the end of the ground truth trajectory. ## model we employ a vision-language transformer, lxmert @xcite as the base architecture for our model. we encode the language input using a learned word embedding and transformer layer, and action history using a linear layer. following @xcite , we extract image features using a faster r-cnn @xcite pretrained on images from the ai2-thor simulator, and average-pool features of regions into a single vector. the visual and action features are first combined via a liner layer, and then fused with language features through a cross modal transformer layer. view-action matching. we collect the multiple views (front, left, right, up, down) and go through the aforementioned process to obtain a feature v i from the cross modal transformer for each view, and compute its matching score m i with the corresponding action embedding a i using a feedforward network. action-type gate. we additionally learn a gate vector using a linear layer over features of all views at the current time step to better distinguish between navigation and non-navigation actions. this layer is trained to predict high weights for actions of the same type as the ground truth action and low weights otherwise. the predicted weights are multiplied pointwise with match scores m i and the action with the highest resultant score is selected. for example, if the ground truth action at a particular time step is move forward, the gate will ensure that a prediction of toggleoff which is a non-navigation action will receive a higher loss than a prediction of turn right, which is also an incorrect action but of the same type as the ground truth action (navigation). loss. the model is trained via cross-entropy losses for action (teacher-forcing) and object type. ## experiments implementation & training details. we use 2 language and 2 cross-modal lxmert layers for the model, and use 768 as the hidden size. we use adamw @xcite as the optimizer with the learning rate 1 × 10 -5 . all of the experiments are run on aws 'p3.16xlarge' ec2 instances running ubuntu 18.04. we employ pytorch (paszke et al., 2017) to build our models. data splits. following shridhar et al. (2020), we train our models on the train split and use success rate on the unseen validation split to perform model selection, and determine whether our model changes are likely to improve over existing state of the art models. we used the validation splits to evaluate the efficacy of variants of the transformer architecture, number of layers and number of epochs of training to use. we then submitted predictions from the best performing model on the unseen validation split to the evaluation server to obtain scores on the test sets. evaluation metrics. we report two evaluation metrics from shridhar et al. (2020) on validation and test splits. success rate (sr) measures the fraction of episodes whether the predicted model trajectory results in all object state changes produced by the ground truth action trajectory. goal condition success rate (gc) measures the fraction of such desired state changes across all episodes that were accomplished by model-predicted trajectories. ## results we first evaluate the utility of each modeling change on the unseen validation set of alfred. as shown sub-goal performance. considering the proportion of gotolocation to the total number of sub-goal tasks (i.e., 48%) and its role of bridging other sub-goal tasks, navigation is very crucial ability for a agent to successfully perform this challenging alfred task. as shown in validation-test performance gap. when we compare to other baselines in this lack of generalization is more likely in current embodied learning tasks such as vision-andlanguage navigation or embodied task completion in comparison to other machine learning tasks due to the way unseen test sets are defined in embodied learning tasks. while alfred in particular does not introduce new object categories at test time, both validation and test unseen environments are visually different, by design from the training environment and from each other. when we compare models on the validation set, we hope that an increase in performance denotes a model that is more capable of generalizing to any unseen environment. however, it may only be the case that the model only generalizes better to the particular visual differences present in the unseen validation environment. when the benchmark limits access to the test set, as in alfred, when dealing with a model that demonstrates variance when trained with different random seeds, hyperparameters and across training epochs, it is natural to choose the setting that results in the highest performance on the unseen validation set. however, a different setting may in fact be optimal for the unseen test set due to visual differences. while such a design is likely significantly more computationally expensive, it may be necessary to redesign benchmarks to take an average of performance from a few different variants of a model to reliably rank different modelling methods, instead of using scores from individual runs. we may also want to re-evaluate the value of keeping a test set private, as in the case of alfred that avoids prevents allowing models to overfit on the test set, but also makes it difficult to analyze the robustness of model performance between the validation and test sets. we would also like to encourage the reviewing community to enable the publication of modelling techniques whose performance is in the same ballpark as existing state-of-the-art models, but novel in some way, as opposed to solely relying on a model achieving a top score on a leaderboard as a criterion for publication, as this limits the development that could be made using these alternative modeling approaches. ## conclusion we attempted to improve a transformer model for embodied task completion by enabling it to effectively uses multiple views via view-action matching and action-type gating. our view-action matching module computes a matching score between each a view and the embedding of the action used to generate it, and the gate module gives a higher weight to a more appropriate action type. while our model outperformed relevant baselines on the alfred unseen validation split, the trend was reversed on the unseen test split, suggesting that it may not be possible to over-utilize a validation split when making model selection choices so that the resultant model does not perform well on the test split. we choose to publish this result as we believe this phenomenon is likely more common than reported with machine learning benchmarks, but only noticeable to researchers when working on a benchmark with limited access to the test set. we additionally hope that our work encourages the publication of promising modelling approaches that do not work as reliably as expected, so that these can act as a guide to researchers to better inform their future directions.