L EARNING G ENERALIZABLE V ISUAL R EPRESENTATIONS VIA I NTERACTIVE G AMEPLAY
A BSTRACT
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural ﬂexibility for creative problem solving, decision making, and socialization.
Comparatively little is known regarding the impact of embodied gameplay upon artiﬁcial agents.
While recent work has produced agents proﬁcient in abstract games, these environments are far removed from the real world and thus these agents can provide little insight into the advantages of embodied play.
Hiding games, such as hide-and-seek, played universally, provide a rich ground for studying the impact of embodied gameplay on representation learning in the context of perspective taking, secret keeping, and false belief understanding.
Here we are the ﬁrst to show that embodied adversarial reinforcement learning agents playing Cache, a variant of hide-and-seek, in a high ﬁdelity, interactive, environment, learn generalizable representations of their observations encoding information such as object permanence, free space, and containment.
Moving closer to biologically motivated learning strategies, our agents’ representations, enhanced by intentionality and memory, are developed through interaction and play.
These results serve as a model for studying how facets of vision develop through interaction, provide an experimental framework for assessing what is learned by artiﬁcial agents, and demonstrates the value of moving from large, static, datasets towards experiential , interactive, representation learning.
I NTRODUCTION
We are interested in studying what facets of their environment artiﬁcial agents learn to represent through interaction and gameplay.
We study this question within the context of hide-and-seek, for which proﬁciency requires an ability to navigate around in an environment and manipulate objects as well as an understanding of visual relationships, object affordances, and perspective.
Inspired by behavior observed in juvenile ravens (Burghardt, 2005), we focus on a variant of hide-and-seek called cache in which agents hide objects instead of themselves.
Advances in deep reinforcement learning have shown that, in abstract games ( e.g. Go and Chess) and visually simplistic environments ( e.g. Atari and grid-worlds) with limited interaction, artiﬁcial agents exhibit surprising emergent behaviours that enable proﬁcient gameplay (Mnih et al., 2015; Silver et al., 2017); indeed, recent work (Chen et al., 2019; Baker et al., 2020) has shown this in the context of hiding games.
Our interest, however, is in understanding how agents learn to represent their visual environment, through gameplay that requires varied interaction, in a high-ﬁdelity environment grounded in the real world.
This requires a fundamental shift away from existing popular environments and a rethinking of how the capabilities of artiﬁcial agents are evaluated.
Our agents must ﬁrst be embodied within an environment allowing for diverse interaction and providing rich visual output.
For this we leverage AI2-THOR (Kolve et al., 2017), a near photo-realistic, interactive, simulated, 3D environment of indoor living spaces, see Fig.
1a.
Our agents are parameterized using deep neural networks, and trained adversarially using the paradigms of reinforce- ment (Mnih et al., 2016) and self-supervised learning.
After our agents are trained to play cache, we then probe how they have learned to represent their environment.
To this end we distinguish two distinct categories of representations generated by our agents.
The ﬁrst, static image representations (SIRs), correspond to the output of a CNN applied to the agent’s egocentric visual input.
The second, dynamic image representations (DIRs), correspond to the output of the agents’ RNN.
While SIRs are timeless, operating only on single images, DIRs have the capacity to incorporate the agent’s previous actions and observations.
Representation learning within the computer vision community is largely focused on developing
SIRs whose quality is measured by their utility in downstream tasks ( e.g. classiﬁcation, depthprediction, etc) (Zamir et al., 2018).
Our ﬁrst set of experiments show that our agents develop lowlevel visual understanding of individual images measured by their capacity to perform a collection of standard tasks from the computer vision literature, these tasks include pixel-to-pixel depth (Saxena et al., 2006) and surface normal (Fouhey et al., 2013) prediction, from a single image.
While
SIRs are clearly an important facet of representation learning, they are also deﬁnitionally unable to represent an environment as a whole: without the ability to integrate observations through time, a representation can only ever capture a single snapshot of space and time.
To represent an environment holistically, we require DIRs.
Unlike for SIRs, we are unaware of any well-established benchmarks for DIRs.
In order to investigate what has been learned by our agent’s DIRs we develop a suite of experiments loosely inspired by experiments performed on infants and young children.
These experiments then demonstrate our agents’ ability to integrate observations through time and understand spatial relationships between objects (Casasola et al., 2003), occlusion (Hespos et al., 2009), object permanence (Piaget, 1954), and seriation (Piaget, 1954) of free space.
It is important to stress that this work focuses on studying how play and interaction contribute to representation learning in artiﬁcial agents and not on developing a new, state-of-the-art, methodology for representation learning.
Nevertheless, to better situate our results in context of existing work, we provide strong baselines in our experiments, e.g. in our low-level vision experiments we compare against a fully supervised model trained on ImageNet (Deng et al., 2009).
Our results provide compelling evidence that: (a) on a suite of low level computer vision tasks within AI2-THOR, static representations learned by playing cache perform very competitively (and often outperform) strong unsupervised and fully supervised methods, (b) these static representations, trained using only synthetic images, obtain non-trivial transfer to downstream tasks using real-world images, (c) unlike representations learned from datasets of single images, agents trained via embodied gameplay learn to integrate visual information through time, demonstrating an elementary understanding of free space, objects and their relationships, and (d) embodied gameplay provides a natural means by which to generate rich experiences for representation learning beyond random sampling and relatively simpler tasks like visual navigation.
In summary, we highlight the contributions: (1) Cache – we introduce cache within the AI2-THOR environment, an adversarial game which permits the study of representation learning in the context of interactive, visual, gameplay.
(2) Cache agent – training agents to play Cache is non-trivial.
We produce a strong Cache agent which integrates several methodological novelties (see, e.g. , perspective simulation and visual dynamics replay in Sec. 4) and even outperforms humans at hiding on training scenes (see Sec. 5).
(3) Static and dynamic representation study – we provide comprehensive evaluations of how our Cache agent has learned to represent its environment providing insight into the advantages and current limitations of interactive-gameplay-based representation learning.
R ELATED WORK
Deep reinforcement learning for games.
As games provide an interactive environment that enable agents to receive observations, take actions, and receive rewards, they are a popular testbed for RL algorithms.
Reinforcement learning has been studied in the context of numerous single, and multi, agent games such as Atari Breakout (Mnih et al., 2015), VizDoom (Lample & Chaplot, 2017), Go (Silver et al., 2017), StarCraft (Vinyals et al., 2019),
Dota (Berner et al., 2019) and Hide-and-Seek (Baker et al., 2020).
The goal of these works is proﬁciency: to create an agent which can achieve (super-) human performance with respect to the game’s success criteria.
In contrast, our goal is to understand how an agent has learned to represent its environment through gameplay and to show that such an agent’s representations can be employed for downstream tasks beyond gameplay.
Passive representation learning.
There is a large body of recent works that address the problem of representation learning from static images or videos.
Image colorization (Zhang et al., 2016), egomotion estimation (Agrawal et al., 2015), predicting image rotation (Gidaris et al., 2018), context prediction (Doersch et al., 2015), future frame prediction (Vondrick et al., 2016) and more recent contrastive learning based approaches (He et al., 2020; Chen et al., 2020b) are among the successful examples of passive representation learning.
We refer to them as passive since the representations are learned from a ﬁxed set of images or videos.
Our approach in contrast is interactive in that the images observed during learning are decided by the actions of the agent.
Interactive representation learning.
Learning representations in dynamic and interactive environments has been addressed in the literature as well.
In the following we mention a few examples.
Burda et al.
(2019) explore curiosity-driven representation learning from a suite of games.
Anand et al.
(2019) address representation learning of the latent factors used for generating the interactive environment.
Zhan et al.
(2018) learn to improve exploration in video games by predicting the memory state.
Ghosh et al.
(2019) learn a functional representation for decision making as opposed to a representation for the observation space only.
Whitney et al.
(2020) simultaneously learn embeddings of state and action sequences.
Jonschkowski & Brock (2015) learn a representation by measuring inconsistencies with a set of pre-deﬁned priors.
Pinto et al.
(2016) learn visual representations by pushing and grasping objects using a robotics arm.
The representations learned using these approaches are typically tested on tasks akin to tasks they were trained with (e.g., a different level of the same game).
In contrast, we investigate whether cognitive primitives such as depth estimation and object permanence maybe be learned via gameplay in a visual dynamic environment.
P LAYING CACHE IN SIMULATION
We situate our agents within AI2-THOR, a simulated 3D environment of indoor living spaces within which multiple agents can navigate around and interact with objects ( e.g. by picking up, placing, opening, closing, cutting, switching on, etc.).
In past works, AI2-THOR has been leveraged to teach agents to interact with their world (Zhu et al., 2017;
Hu et al., 2018; Huang et al., 2019; Gan et al., 2020b; Wortsman et al., 2019; Gordon et al., 2017), interact with each other (Jain et al., 2019; 2020) as well as learn from these interactions (Nagarajan & Grauman, 2020; Lohmann et al., 2020).
AI2-THOR contains a total of 150 unique scenes equally distributed into ﬁve scene types: kitchens, living rooms, bedrooms, bathrooms, and foyers.
We train our cache agents on a subset of the kitchen [Table] [Table] xB+lnkBgE8liBuAkDOj7QM06nDQ=">AAACAHicbVDLSsNQEL2pr1pfURcu3ASL6KokWnzsit24KhXsA5pSbm4n7aU3D+6diCVk46+4caGIWz/DnX9j2gZR64GBwzkzzMxxQsEVmuanlltYXFpeya8W1tY3Nrf07Z2mCiLJoMECEci2QxUI7kMDOQpohxKo5whoOaPqxG/dgVQ88G9xHELXowOfu5xRTKWevmcj3KNicbVWS3qxjUNAepQUenrRLJlTGPPEykiRZKj39A+7H7DIAx+ZoEp1LDPEbkwlciYgKdiRgpCyER1AJ6U+9UB14+kDiXGYKn3DDWRaPhpT9edETD2lxp6TdnoUh+qvNxH/8zoRuhfdmPthhOCz2SI3EgYGxiQNo88lMBTjlFAmeXqrwYZUUoZpZrMQLic4+355njRPStZpqXxTLlausjjyZJ8ckGNikXNSIdekThqEkYQ8kmfyoj1oT9qr9jZrzWnZzC75Be39C6XwlpE=</latexit> and living room scenes as these scenes are relatively large and often include many opportunities for interaction; but we use all scenes types across our suite of experiments.
Details regarding scene splits across train, validation, and test for the different experiments can be found in Sec.
A.1.
[-"xB+lnkBgE8liBuAkDOj7QM06nDQ=">AAACAHicbVDLSsNQEL2pr1pfURcu3ASL6KokWnzsit24KhXsA5pSbm4n7aU3D+6diCVk46+4caGIWz/DnX9j2gZR64GBwzkzzMxxQsEVmuanlltYXFpeya8W1tY3Nrf07Z2mCiLJoMECEci2QxUI7kMDOQpohxKo5whoOaPqxG/dgVQ88G9xHELXowOfu5xRTKWevmcj3KNicbVWS3qxjUNAepQUenrRLJlTGPPEykiRZKj39A+7H7DIAx+ZoEp1LDPEbkwlciYgKdiRgpCyER1AJ6U+9UB14+kDiXGYKn3DDWRaPhpT9edETD2lxp6TdnoUh+qvNxH/8zoRuhfdmPthhOCz2SI3EgYGxiQNo88lMBTjlFAmeXqrwYZUUoZpZrMQLic
-]
In a game of cache, two agents (a hider and a seeker ) compete, with the hiding agent attempting to place a given object in the environment so that the seeking agent cannot ﬁnd it.
This game is zero-sum with the hiding agent winning if and only if the seeking agent cannot ﬁnd the object.
We partition the game of cache into ﬁve conceptually-distinct stages: exploration and mapping (E&M), perspective simulation (PS), object hiding (OH), object manipulation (OM), and seeking (S); see Figures 1b to 1f.
A game of cache begins with the hiding agent exploring its environment and building an internal map corresponding to the locations it has visited (E&M).
The agent then chooses globally, among the many locations it has visited, a location where it believes it can hide the object so that the seeker will not be able to ﬁnd it (PS).
After moving to this location the agent makes a local decision about where the object should be placed, e.g. behind a TV or inside a drawer (OH).
The agent then performs the low-level manipulation task of moving the object in its hand to the desired location (OM), if this manipulation fails then the agent tries another local hiding location (OH).
Finally the seeking agent searches for the hidden object (S).
Pragmatically, these stages are distinguished by the actions available to the agent (see Table G.1 for a description of all 219 actions) and their reward structures (see Sec.
C.5).
For more details regarding these stages, see Sec.
B.
L EARNING TO P LAY C ACHE
In the following we provide an overview of our agents’ neural network architecture as well as our training and inference pipelines.
For space, comprehensive details are deferred Sec.
D.
Our agents are parameterized using deep convolutional and recurrent neural networks.
In what follows, we will use θ ∈ Θ as a catch-all parameter representing the learnable parameters of our model.
Our model architecture (see Fig 2) has ﬁve primary components: (1) a CNN that encodes input egocentric images from AI2-THOR into SIRs, (2) a RNN model that transforms input observations into
DIRs, (3) multi-layer perceptrons (MLPs) applied to the DIR at a given timestep, to produce the actor and critic heads, (4) a perspective simulation module which evaluates potential object hiding places by simulating the seeker, and (5) a U-Net (Ronneberger et al., 2015) style decoder CNN used during visual dynamics replay (VDR), detailed below.
Static representation architecture .
Given images of size 3 × 224 × 224 , we generate SIRs using a 13-layer U-Net style encoder CNN, CNN θ .
CNN θ downsamples input images by a factor of 32 and produces an our SIR of shape 128 × 7 × 7 .
Dynamic representation architecture .
Unlike an SIR, DIRs combine current observations, historical information, and goal speciﬁcations.
Our model has three recurrent components: a metric map, a 1-layer convolutional-GRU CGRU θ (Cho et al., 2014; Kalchbrenner et al., 2016), and a 1-layer
LSTM LSTM θ
(Hochreiter & Schmidhuber, 1997).
At time t ≥ 0 , the environment provides the agent with observations o t including the agent’s current egocentric image i t and goal g t ( e.g. hide the cup).
CNN θ inputs i t to produce the SIR s t =
CNN θ
( i t ) .
Next the CGRU processes existing information to produce the (grid) DIR (cid:101) h t =
CGRU θ ( s t , o t , h t − 1 ) of shape 257 × 7 × 7 .
(cid:101) h t is then processed by the LSTM LSTM θ to produce the (ﬂat) DIRs ( h ot , h ct ) =
LSTM θ
( (cid:101) h t , o t , h t − 1 ) where h ot , h ct are the so-called output and cell states of the LSTM with length 512 .
Finally the agents’ metric map representation m t − 1 from time t − 1 is updated by inserting a 128 -dim tensor from CNN θ into the position in the map corresponding to the agent’s current position and rotation, to produce m t .
The historical information h t for the next step now includes ( (cid:101) h t , h ot , h ct , m t ) .
Actor-critic heads .
Our agents use an actor-critic style architecture, producing a policy and a value at each time t .
As described in Sec. 3, there are ﬁve distinct stages in a game of cache (E&M, PS,
OH, OM, S).
Each stage has unique constraints on the space of available actions and different reward structures, and thus require distinct polices and value functions – obtained by distinct 2-layer MLPs taking as input the DIR h ot described above.
Perspective Simulation .
Evidence suggests that in humans, perspective taking, the ability to take on the viewpoint of another develops well after infancy (Trafton et al., 2006).
While standard RNNs can perform sophisticated tasks, there is little evidence to suggest they can perform the multi-step ifthen reasoning necessary in perspective taking.
Inspired by AlphaGo’s
(Silver et al., 2017) intuition guided Monte-Carlo tree search and imagination-augmented RL agents (Racani`ere et al., 2017)
we require, in the perspective simulation stage, that our hiding agent evaluate prospective hiding places by explicitly taking on the seeker’s perspective using internal mental rollouts (see Fig. 3c).
To this end, the metric map is processed by a CNN to produce an internal map representation M 1 , which is used to coarsely evaluate potential hiding locations, see Fig.
3b.
For a subset of these locations (cid:96) , M 1 is processed (along with (cid:96) ) to produce a new representation M (cid:96) .
Intuitively, M (cid:96) can be thought of as representing that agent’s beliefs about what the environment would look like after having hidden the object at position (cid:96) .
A “mental” seeker agent deﬁned using an LSTM (not shown in the model ﬁgure for simplicity) then moves about M (cid:96) and obtains observations as slices of M (cid:96) , see Fig.
3c.
The quality of a potential hiding location is then quantiﬁed by the number of steps taken by the “mental” seeking agent to ﬁnd the hidden object.
After scoring hiding locations in this way, the agent randomly samples a position to move to (beginning the OH-stage) with larger probability given to higher scoring positions.
Training and losses.
Our agents are trained using the asynchronous advantage actor-critic (A3C) algorithm (Mnih et al., 2016) with generalized advantage estimation (GAE) (Schulman et al., 2015) and several (self) supervised losses.
As we found it critical to producing high quality SIRs, we highlight our visual dynamics methodology below.
See Sec.
C for details on all losses.
Visual dynamics replay.
During early experiments, we found that producing high quality SIRs using only existing reinforcement learning techniques to be infeasible as the produced gradients suffer from high variance and so signal struggles to propagate through the many convolutional layers.
To correct this deﬁciency we introduce a technique we call a visual dynamics replay (VDR) inspired by experience replay (Mnih et al., 2015) and intrinsic motivation (Pathak et al., 2017).
In VDR, agent state transitions produced during training are saved to a separate process where an encoderdecoder architecture (using CNN θ as the encoder) is trained to predict a number of targets including forward dynamics, predicting an image from the antecedent image and action, and inverse dynamics, predicting which action produced the transition from one image to another.
Given triplet ( i 0 , a, i 1 ) , where i 0 , i 1 are the images before and after an agent took action a , we employ several self-supervised tasks including pixel-to-image predication.
See Sec.
C.2 for more details.
E XPERIMENTS
Our experiments are designed to address three questions: (1) “has our cache agent learned to proﬁciently hide and seek objects?”, (2) “how do the SIRs learned by playing cache compare to those learned using standard supervised approaches and when training using other interactive tasks?”, and (3) “has the cache agent learned to integrate observations over time to produce general DIR representations?”.
For training details, e.g. learning rates, discounting parameters, etc., see Sec.
C.
Playing cache.
Figures 3a to 3f qualitatively illustrate that our agents have learned to explore, navigate, provide intuitive evaluations of hiding spaces, and manipulate objects.
Figures 4a and 4b, show quantitatively that hiders from later in training consistently chose hiding places that our seekers are less likely to ﬁnd.
Also, for hiding spots generated by human experts and by an automated brute force approach, seekers from later in training more frequently ﬁnd the hidden objects.
Interestingly,
[Figure]
the seeker is able to locate objects hidden by humans and the brute force approach with a similar level of accuracy.
Surprisingly, as shown in Table 4c, human seekers ﬁnd objects hidden by humans and by our hiding agent at similarly high rates: on training scenes human seekers ﬁnd our agent’s objects 77% of the time and their fellow humans hidden objects 80% of the time, on test scenes humans are more successful at ﬁnding our agent
’s objects (90% success rate).
Finally note that the (effectively random) h 0 agent hides objects generally very near its starting location and such positions are very frequently found by all agents.
Somewhat surprisingly, the s 1 e 6 agent appears very slightly worse at ﬁnding such positions than the seeker s 0 agent.
This can be explained by the s 1 e 6 agent having learned to not expect such an audacious hiding strategy.
Static tasks.
We evaluate SIRs on a suite of standard computer vision targets such as monocular depth estimation by freezing the SIRs and training decoder models.
These evaluations are performed on synthetic (AI2-THOR) data as well as natural images, which can be considered as out of domain.
[Figure]
E
M
H
Exhaustive
[Figure]
Human hh 1e
h 4.5e6 E
M
H
Exhaustive Cache Agent
Human s 4.5e [Figure] s 1e [Figure]
[Figure]
Predictions with synthetic images - We use SIRs to predict geometry and appearance-based targets namely depth, surface normals, object class, object depth and object normals, as well as affordancebased targets namely traversable surfaces and object openability.
On the test set, our (Cache) SIR consistently outperforms multiple baselines (Fig 5) including: Auto Encoder on AI2-THOR images, a Navigation agent within AI2-THOR, and Classiﬁer – a CNN trained to classify objects in
AI2-THOR images.
Importantly, the Cache SIRs compare favorably (statistically indistinguishable performance for 4 tasks and statistically signiﬁcant improvement for the remaining 2) to SIRs produced by a model trained on the 1.28 million hand-labelled images from ImageNet (Deng et al., 2009), the gold-standard for fully-supervised representation learning with CNNs.
Note that our SIR is trained without any explicitly human labeled examples.
Predictions with natural images - As images from AI2-THOR are highly correlated and synthetic, we would not expect cache SIRs to transfer well to natural images.
Despite this, Fig. 6 shows that the cache SIRs perform remarkably well: for SUN scene classif.
(Xiao et al., 2010) as well as the
NYU V2 depth (Nathan Silberman & Fergus, 2012) and walkability tasks (Mottaghi et al., 2016), the cache SIRs outperform baselines and even begin to approach the ImageNet SIR performance.
This shows that there is strong promise in using gameplay as a method for developing SIRs.
Increasing the diversity and ﬁdelity of synthetic environments may allow us to further narrow this gap.
Dynamic tasks.
While CNN models can perform surprisingly sophisticated reasoning on individual images, these representations are fundamentally timeless and cannot hope to model visual understanding requiring memory and intentionality.
Through experiments loosely inspired by those from developmental psychology we show that, memory enhanced representations emerge within our agents’ DIRs when playing cache.
In the following experiments, we ensure that objects and scenes used to train the probing classiﬁers are distinct from those used during testing.
For a description of all baseline models, see Sec G.
Developmental psychology has an extensive history of studying childrens’ capacity to differentiate between an object being contained in, or behind, another object (Casasola et al., 2003).
Following this line of work, our hider is made to move an object through space until the object is placed within, or behind, a receptacle object (see Fig. 7a).
The DIR is then used to predict the relationship between
[Figure]
[Figure]
the object and the receptacle using a linear classiﬁer.
Cache DIRs dramatically outperform other methods obtaining 71.5% accuracy (next best baseline obtaining 63.4%).
Two and a half month old infants begin to appreciate object permanence (Aguiar & Baillargeon, 2002) and are surprised by impossible occlusion events highlighting the importance of permanence as a building block of higher level cognition, and begin acting on this information at around nine months of age (Hespos et al., 2009).
Inspired by these studies, we ﬁrst ﬁnd that (Fig.
7b) a linear classiﬁer trained on a cache DIR is able to infer the location of an object on a continuous path after it becomes fully occluded with substantially greater accuracy (64%) than the next best competing baseline (59.6%).
Second (Fig. 7c), we test our agents’ ability to remember and act on previously seen objects.
We ﬁnd that our seeker’s DIR can be used to predict whether it has previously seen a hidden object after being forced to rotate away from it, having the object vanish, and taking four additional steps.
Surprisingly we see that the seeker has statistically signiﬁcantly larger odds of rotating back towards the object after seeing it than a control agent (mean difference in log-odds of rotating back towards the object vs. rotating in the other direction between treatment and control agents 0.08, 95% CI ± 0.02) demonstrating its ability to act on its understanding of object permanence.
Numerical markers denote the number of steps since the agent has seen the object.
Piaget argues that the capacity for seriation, ordering objects based on a common property, develops after age seven (Piaget, 1954).
We cannot hope that our agents have learned a seven year old’s understanding of seriation but we hypothesize that rudimentary seriation should be possible among properties central to cache.
One such property is free space, the amount of area in front of the agent that could be occupied by the agent without collision.
We place our agent randomly within the environment, rotate it in 90 ◦ increments (counter-) clockwise, and save SIRs and DIRs after each rotation (see Fig. 7d). We then perform a two-image seriation, namely to predict, from the representation at time t , whether the viewpoint at the previous timestep t − 1 contained more free
[Figure]
Contained or behind?
Where is the object?
Did you see the object?
Which has more free space?
[Figure]
space.
Our cache DIR is able to perform this seriation task with accuracy signiﬁcantly above chance (mean accuracy of 84.92%, 95% CI ± 2.27%) and outperforms all baselines.
D ISCUSSION
Modern paradigms for visual representation learning with static labelled and unlabelled datasets have led to startling performance on a large suite of visual tasks.
While these methods have revolutionized computer vision, it is difﬁcult to imagine that they alone will be sufﬁcient to learn visual representations mimicking humans’ capacity for visual understanding which is augmented by intent, memory, and an intuitive understanding of objects and their physical properties.
Without such representations it seems doubtful that we will ever deliver one of the central promises of artiﬁcial intelligence, an embodied artiﬁcial agent who achieves high competency in an array of diverse, demanding, tasks.
Given the recent advances in self-supervised and unsupervised learning approaches and progress made in developing visually rich, physics-based embodied AI environments, we believe that it is time for a paradigm shift via a move towards experiential, interactive, learning.
Our experiments, in showing that rich interactive gameplay can produce representations which encode an understanding of containment, object permanence, seriation, and free space suggest that this direction is promising.
Moreover, we have shown that experiments designed in the developmental psychology literature can be fruitfully translated to understand the growing competencies of artiﬁcial agents.
Looking forward, real-world simulated environments like AI2-THOR have focused primarily on providing high-ﬁdelity visual outputs in physics-enabled spaces with comparatively little effort being placed on developing other sensor modalities ( e.g. sound, smell, texture, etc.).
Recent work has, however, introduced new sensor modalities (namely audio) into simulated environments (Chen et al., 2020a; Gan et al., 2020a) and we suspect that the variety and richness of these observations will quickly grow within the coming years.
The core focus of this work can easily be considered in the context of these emerging sensor modalities and we hope in future work to examine how, through representation and gameplay, an artiﬁcial agent may learn to combine these separate sensor streams to build a holistic, dynamic, representation of its environment. A CKNOWLEDGMENTS
We thank Chandra Bhagavatula, Jaemin Cho, Daniel King, Kyle Lo, Nicholas Lourie, Sarah Pratt, Vivek Ra- manujan, Jordi Salvador, and Eli VanderBilt for their contributions in human evaluation.
We also thank Oren
Etzioni, David Forsyth, and Aude Oliva for their insightful comments on this project and Mrinal Kembhavi for inspiring this project.
V IRTUAL ENVIRONMENT AND PREPROCESSING
While AI2-THOR can output images of varying dimensions and quality we ﬁnd that simulating images of size 224 × 224 with the “Very Low” quality setting provides a balance of image ﬁdelity and computational efﬁciency.
AI2-THOR provides RGB images with the intensity of each channel encoded by an integer taking values between 0 and 255.
Following standards used for natural images, we preprocess these images by ﬁrst scaling the channels so that they are ﬂoating point numbers between 0 and 1 after which we normalize the R, G, and B channels with means 0.485, 0.456, 0.406, and standard deviations 0.229, 0.224, 0.225.
Agents are constrained to lie on a grid with 0.25 meter increments and may face in one of four cardinal directions with 0 ◦ being north and 90 ◦ being east.
The agent’s camera is set to have a ﬁeld of view of 90 ◦ , is ﬁxed at an angle 30 ◦ below the horizontal, and lies at a height of 1.5765 meters when standing and 0.9015 meters when crouching.
Following conventions in computer graphics, the agent’s position is parameterized a single ( x, y, z ) coordinate triple with x and z corresponding to movement in the horizontal plane and with y reserved for vertical movements.
Other than when standing and crouching, our agent’s vertical position is ﬁxed.
Letting Z denote the integers, an agent’s position is uniquely identiﬁed by a location tuple ( x, z, r, s ) ∈ (0 .
· Z ) × (0 .
25 · Z ) ×{ 0 , 90 , 180 , 270 }×{ 0 , 1 } where x, z are as above, r denotes the agent’s rotation in degrees, and s equalsif and only if the agent is standing.
It will be useful in the following to be able to easily describe subcomponents of a location tuple, to this end we let proj − x : R 4 → R 3 be the projection removing the ﬁrst coordinate of a location tuple so that proj − x ( x, z, r, s )
=
( z, r, s ) , and similarly for proj − z , proj − r , proj − s .
AI2-THOR contains a total of 150 scenes evenly grouped into ﬁve scene types, kitchens (1-30), living rooms (201-230), bedrooms (301-330), bathrooms (401-430), and foyers (501-530).
Excluding foyers, which are reserved for our dynamic image representation experiments and used nowhere else, we consider the ﬁrst 20 scenes of each scene type to be train scenes, the next ﬁve of each type to be validation scenes, and the last ﬁve of each type to be test scenes.
When training our cache agents we only use kitchen and living room scenes as these scenes are relatively large and often include many opportunities for interaction.
B C ACHE STAGES
In cache the hiding agent , or hider , attempts to place a goal object in a given AI2-THOR scene such that the seeking agent , or seeker , cannot ﬁnd it.
This goal object is one of ﬁve standard household items, see Fig.
G.1, being either a loaf of bread, cup, knife, plunger, or tomato.
Recall that we divide the game of cache into the ﬁve stages S stages =
{ E&M, PS, OH, OM, S } .
The hider participates in the ﬁrst four of these stages and the seeker in the last stage.
Each of these stages is associated with a distinct discrete collection of possible actions, see Table G.1.
When a stage s ∈ S stages is instantiated, so that the scene has been chosen and all goals speciﬁed, we call this instantiation an s -episode .
During an s -episode, an agent takes actions, each taking a single unit of time called a step , until the goal state has been reached or a maximum number of steps, N maxsteps s , has been taken.
Episodes begin with time t = 0 and t advances by 1 after every action regardless of whether or not that action is successful.
Note that these episodes must be sequentially instantiated and we call a collection of sequentially instantiated episodes ending with an S -episode a game of cache .
In the following we describe these stages in detail along with the metrics we use to track the quality of the learned policies and assign rewards.
In order to describe these metrics we require the following sets.
First let S visited H,t ( S visited S,t ) be the set of all location tuples visited by the hider (seeker) by time step t of its episode.
Similarly we let S opened H,t ( S opened S,t ) be the set of all objects the hider (seeker) has opened by time step t .
Finally we let S extrap.
H,t be the set of all location tuples that can, assuming the agent can move through objects, be reached by starting at some location in S visited H,t and moving the agent ahead by 0.25m and then either staying put, moving left by 0.25m, or moving right 0.25m.
B. E XPLORATION AND MAPPING (E&M)
A game of cache begins with an E&M episode.
During this episode, the agent may traverse the scene and interact with objects while building a representation of its environment for up to N maxsteps E & M = 200 steps.
In our implementation, this representation is a metric map, see the model architectures section below, which is then available in the subsequent PS- and OH-episodes.
Let S reachable be the set of all feasible location tuples in a given episode and, similarly, let N openable be total number of objects that can be opened.
We evaluate the quality of exploration with three metrics:
[Equation]
and
[Equation]
Q coverage and Q coverage+ evaluate the agent’s ability to traverse the entire scene while Q open% evaluates how frequently the agent interacts with its environment.
B. P ERSPECTIVE S IMULATION (PS)
After exploration the hider is allowed to consult the representation it has developed of its environment and must, from that representation, pick a single location tuple to which the agent will then be moved.
As we do not give the agent a global reference frame it must choose this location tuple by specifying a location relative its ﬁnal position after the E&M episode.
As the hiding agent is not constrained to pick a valid location tuple, it may, for instance, choose a location that is outside of the scene or already occupied by an object, such a choice is considered a failed action.
As the quality of a choice of location tuple in this episode is tied intimately with the outcome of the OH-episode we will measure the progress of training for both episodes using the same metrics described in the section below.
B. O BJECT HIDING (OH)
In the OH-episode the agent has been moved to a location tuple determined in the PS-episode and is now holding the goal object.
In this episode the agent is not allowed to perform any navigational actions beyond standing or crouching, but may interact with objects by opening or closing them.
Most importantly, the agent must choose a location in its current view at which it should hide the item.
To discretize the possible hiding locations recall that the images available to agents are of size 224 × 224 .
Viewing C = { 1 , ..., 224 } 2 as a grid of pixel coordinates, let ( C ij ) 1 ≤ i,j ≤ 7 be a partitioning of C into a 7 × 7 grid so that C ij = { (32 · i + k, 32 · j + l ) | 1 ≤ l, k ≤ 32 } is of size 32 × 32 .
When specifying a location in which to hide the object, the agent picks one of these grid locations along with one of the positional keywords in the set S modality
= { O N T OP , C ONTAINED
I N , B EHIND } .
After choosing a location we begin an OM-episode, described below, during which the agent attempts to move the object through space and to the speciﬁed location.
In order to reduce the complexity of specifying plausible object locations, potentially difﬁcult when, e.g. , specifying a ﬁnal location behind another object, we consider an object to have been successfully placed at the grid location C ij , when using the C ONTAINED I N , B EHIND keywords, even in those cases that it moved to C ( i − 1) j , C
( i +1)
j instead.
If OM-episode is successful given this relaxation, all future actions specifying object locations are considered failures.
If the OM-episode is unsuccessful the hiding agent may continue to specify locations, causing the item to be picked up and a new OM-episode to begin, until manipulation is successful.
If the hiding agent fails to successfully place the object within N maxsteps OH = 15 steps the object is returned to its default position in the agent’s hand and then simply dropped.
Otherwise, the agent may continue to take other actions, closing a cabinet into which it has placed an object for instance, or specify that it is done by performing the ReadyForSeeker action.
Objectively quantifying a hiding location is difﬁcult as the quality of a location is coupled with psychology and artifacts of perception.
Indeed, the phenomenon of an object being lost “in plain sight” suggests that seemingly irrational hiding strategies may be highly successful.
Nevertheless, we require some automated metric with which to measure the quality of a hidden object and, for this purpose, deﬁne the following three metrics.
First we let N visiblefrom be the number of feasible location tuples from which any portion of the goal object can be seen at any distance.
To complement N visiblefrom with a measure that takes the seeking agent’s start location into account, we let N BFSsteps be the number of steps it takes a breadth ﬁrst search (BFS) agent, starting at the seeking agent’s start location, to ﬁnd the goal object.
A single step for the BFS agent corresponds to teleporting to the location tuple closest to the seeker’s start position that has not yet been visited by the BFS agent and checking if the goal object can be seen from any rotation, crouching or standing, from that location.
The BFS agent is allowed to see through openable objects and the goal object is considered found if any portion of the object can be seen at any distance.
If the BFS agent cannot ﬁnd the object then N BFSsteps equals the maximum number of steps, | proj − r (proj − s ( S reachable ))
| , in which case we let δ BFSfoundobject equal 0, otherwise δ BFSfoundobject equals 1.
Finally we also deﬁne the proportional versions of two of the above metrics.
[Equation]
and
[Equation]
B. O BJECT MANIPULATION (OM)
In the object manipulation episode the agent is set at a ﬁxed location tuple holding the goal object and is given a grid location 1 ≤ i, j ≤ 7 along with the positional keyword M ODALITY ∈ S modality .
During this episode the agent can interact with objects by opening them, move the object held in its hand in one of six axis-aligned directions, or drop the goal object.
The object manipulation episode is considered successful if, after dropping the goal object, the goal object is in the correct grid position speciﬁed by i, j and respects the constraints imposed by M ODALITY .
In particular, after the manipulator has dropped the object, we obtain from AI2THOR the sets S O N T OP pix.
= { coordinates of pixels corresponding to the object that are visible from the agent’s current viewpoint } , S B EHIND pix.
= { pixel coordinates of the object that are visible only after making all other objects in the scene invisible } , and the value δ C ONTAINED I N ∈ { 0 , 1 } which equals
if and only if the goal object is contained within another object.
If δ C ONTAINED I N = 1 , then we let S C ONTAINED
I N pix.
= S O N T OP pix.
∪ S B EHIND pix.
, otherwise S C ONTAINED I N pix.
= ∅ .
If |S O N T OP pix.
| + |S B EHIND pix.
| ≤ 10 , the episode is considered a failure.
Otherwise, success depends on which M ODALITY was speciﬁed and on the quantity M = max i,j max( | C i,j ∩S O N T OP pix.
| , | C ij ∩S B EHIND pix.
| , | C i,j ∩ S C ONTAINED I N pix.
| ) , i.e. maximum number of pixels in any single modality and grid location.
If M ODALITY =
O N T OP then the episode was successful if | C ij ∩ S O N T OP pix.
| ≥ 0 .
95 · M , i.e. if there are at least 95% as many pixels in the speciﬁed location and modality as in in the grid location and modality with the most of the object’s pixels.
Similarly if M ODALITY =
B EHIND (or M ODALITY =
C ONTAINED I N ) then the episode was successful if | C ij ∩ S B EHIND pix.
| ≥ 0 .
95 · M (or | C ij ∩ S C ONTAINED I N pix.
| ≥ 0 .
95 · M ).
To track the progress of this episode we simply record successes, namely we let δ hittarget ∈ { 0 , 1 } equal 1 if and only if object manipulation was successful.
B. S EEKING (S)
In the seeking episode the seeking agent is tasked with ﬁnding the object hidden by the hiding agent.
The seeking agent begins from the same location tuple as the hiding agent began in the E&M-episode and given a budget of N maxsteps S = 500 steps before the episode is considered a failure.
During these 500 steps the seeking agent may perform any of the actions allowed in the exploration and mapping episode along with one additional action, ClaimVisible .
If the seeking agent performs the ClaimVisible action successfully the seeking agent has successfully found the goal object and the episode ends.
Just as when attempting to automatically measure the quality of a hiding location, developing objective mea- sures of the quality of a seeking policy is challenging.
As described previously, a seeking strategy that is highly successful against one hiding strategy may be fatally ﬂawed when faced with another.
To remove any dependence on a hiding agent obtained through reinforcement learning we generate, through an exhaustive, brute force approach, three collections of hiding locations for each scene s and object o , S easypos.
s,o , S med.
pos.
s,o , and S hardpos.
s,o .
To deﬁne these collections we use AI2-THOR to, for each goal object o and scene s , gener- ate a collection L o,s = { (cid:96) o,s,i | 1 ≤ i ≤ N o,s } of N o,s (cid:29) 0 possible hiding locations.
For each (cid:96) o,s,i we compute v o,s,i which equals the Q visiblefrom% metric deﬁned above for the given hiding location.
Letting V o,s = { v o,s,i | 1 ≤ i ≤ N o,s } , we consider a hiding position (cid:96) o,s,i hard if v o,s,i is within the lower ﬁfth percentile of V o,s values, medium if v o,s,i ≤ 0 .
15 and v o,s,i is between the ﬁfth and 20th percentiles, and ﬁnally easy if it is neither hard nor medium difﬁculty.
The set S easypos.
s,o , and similarly for S med.
pos.
s,o and S hardpos.
s,o , is then a randomly sampled collection of, approximately, 20 easy locations from L o,s .
We then quantify the quality of a seeking policy by computing the success rate of the seeker when the hiding locations are restricted to those from each of the above three sets.
C T RAINING DETAILS AND LOSSES
We train our cache agent using eight GPUs with one GPU reserved for running AI2-THOR processes, one reserved for VDR, and the other six dedicated to training with reinforcement and self-supervised learning.
Each of the cache stages generates its own losses, detailed below, which are minimized by HOGWILD!
(Recht et al., 2011) style unsynchronized gradient descent to train model parameters.
Reinforcement learning in the E&M, OH, OM, and S stages is accomplished using the asynchronous advantage actor-critic (A3C) algorithm (Mnih et al., 2016) with generalized advantage estimation (GAE) (Schulman et al., 2015).
For the A3C loss we let the discounting parameter γ = 0 .
99 (except for in the OH-stage where γ = 0 . 8 ), the entropy weight β = 0 .
01 , and GAE parameter τ = 1 .
We use the ADAM optimizer (Kingma & Ba, 2015) with AMSGrad (Reddi et al., 2018), moving average parameters β 1 = 0 .
99 , β 2 = 0 .
999 , a learning rate of 10 − 3 for VDR, and varying learning rates for the different cache stages ( 10 − 4 for the E&M, OH, OM, and S stages, 5 · 10 − 4 for the PS-stage).
In the following we give an overview various training details, namely: (1) data augmentation, (2) (self) super- vised losses, (3) and reward structures for the E&M, OH, OM, and S stages necessary for computing the A3C loss.
C. T RAINING DATA AUGMENTATION
To increase diversity of the images seen by our cache agents during training, we apply two forms of data augmentation to AI2-THOR.
Random world rotation.
Recall that our AI2-THOR cache agents are constrained to 90 ◦ rotations and move on a 0.25m × 0.25m grid.
while this is beneﬁcial in that it abstracts many of the technical complexities of movement in true robotics, it has the disadvantage of reducing the variety of images that can be seen by our agent.
In particular, the default orientation of the agent is “axis aligned” such that it is generally moving parallel to walls and objects and thus it will not see objects and rooms from a large variety of orientations.
To remedy this limitation while still maintaining grid-constrained movement, we, with a 50% probability at the start of every new game of cache, randomly rotate the agent by some degree in { 0 , . .
After this initial rotation the agent moves along a 0.25m × 0.25m grid as usual but, because of this rotation, it will now observe objects at a variety of different orientations.
Image augmentation.
To improve generalization, it is common with the computer vision literature to apply random transformations to images before giving them as input to neural network models for prediction.
We follow this approach and, at the beginning of every game of cache, we randomly select:
(a) A resized crop function C l,r,t,b which, given an input image, crops l pixels from the left, r pixels from the right, t pixels from the top, and b pixels from the bottom, and then resizes the cropped image back to the start dimensions with bilinear sampling.
In particular we select ( l, r, t, b ) =
(0 , 0 , 0 , 0) with 0.5 probability and, otherwise, select ( l, r, t, b ) uniformly at random from { 1 , . .
(b) A color jitter function J .
This function J is chosen to vary the brightness, contrast, saturation, and hue of input images by small amounts.
(c) A rotation function R d .
Here R d rotates the input image about its center by d degrees and we choose d a random to equal 0 with probability 0.5 and, otherwise, sample d uniformly at random from {− 3 , . . .
With the above functions in hand, we apply the composed transform R d ◦ J ◦ C l,r,t,b to every image seen during the game of cache.
C. V ISUAL DYNAMICS REPLAY
Here we give an overview of our VDR training procedure.
We direct anyone interested in exact reproduction of our VDR procedure to our code base.
During training our agents save triplets ( i 0 , a, i 1 ) , where i 0 is an antecedent image, a is an action taken (with associated success or failure signal), and i 1 is the subsequent image resulting from taking action a when the agent’s view was i 0 , to a queue that is then periodically emptied and processed by the VDR thread.
The VDR thread processes the triplets by saving them into a buffer, similar to an experience replay buffer (Mnih et al., 2015).
To ensure that certain actions do not dominate the buffer we group and selectively subsample the triplets added to the buffer.
Moreover, the buffer automatically removes triplets to ensure that triplets do not exceed some ﬁxed age and that the buffer does not exceed its max size of approximately 20,000 triplets.
After processing incoming data and storing triplets in the dynamics buffer, the VDR thread then randomly partitions the buffer into batches of size 64 and, for each batch, computes the self-supervised losses detailed below and performs a single gradient step to minimize the sum of these losses.
We call a single pass through all batches an epoch.
VDR then alternates between processing incoming triplets, forming batches, and training for a single epoch.
The losses minimized in VDR include the following.
• Inverse dynamics loss: predicting a from i
and i 1 .
From the VDR model we obtain, from i
and i 1 , a vector v inv.
dyn.
of length N actions = the total number of unique actions across all agents.
By assigning each action a a unique index, index ( a ) ∈ { 1 , ..., N actions } , we then compute the inverse dynamics loss as the negative cross entropy between the point mass distribution δ index ( a ) and softmax ( v inv.
dyn. ) .
• Forward dynamics loss: predicting i 1 from a and i 0 .
From the VDR model we obtain, from i 0 and a , a tensor (cid:98) i n × n∈ R 3 × n × n for n ∈ S = { 14 , 28 , 56 , 112 , 224 } where (cid:98) i n × nrepresents the prediction of i n × nwhich equals i 1 downsampled to have spatial resolution n × n .
Lett ∆ n be the, per pixel, (cid:96) 1 error between (cid:98) i n × nand i n × n, and let ∆ n be its mean.
We let the forward dynamics loss equal (cid:80) n ∈ S ∆ n .
In practice we found it helpful to compute ∆ n as a weighted average with larger weight given to areas of the image that have changed.
• Pixel difference loss: predicting which pixels in i 1 are different from those in i 0 from i 0 and a .
Computed similarly to the forward dynamics loss using a tensor T pix.
diff.
returned from the VDR model except using a binary cross entropy loss to predict which pixels are different between i
and i 1 .
• Pixel quality loss: predicting conﬁdence of predictions of forward dynamics.
From the VDR model we obtain a 1 × 224 × 224 tensor T conﬁdence .
We compute the pixel equality loss as the mean squared error between T conﬁdence and ∆ 224 .
• Action success loss: from a and i 0 predict if the action would be successful.
From the VDR model we obtain, from a and i 0 , T wouldsucceed .
The action success loss is then computed as the binary cross entropy loss between T wouldsucceed and the indicator δ actiondidsucceed .
Note that we do not compute the forward dynamics loss and pixel quality loss on those triplets whose action a (see table Table G.1 for all action types) is RotateLeft , RotateRight , OpenAt , CloseObjects ,
PlaceAt , ReadyForSeeker , or ClaimVisible .
This is because such actions either cause very unpredictable changes ( e.g. RotateLeft results in the agent looking at entirely separate portion of the scene) or no pixel changes ( e.g. a ReadyForSeeker action simply ends an episode and changes nothing).
C. T RAINING PERSPECTIVE SIMULATION
In this section, when we refer to a location tuple, we mean a location tuple without the standing component.
Recall that in the PS-stage the agent must consult its map of the environment and, from this map, choose a location tuple where from which it will begin the OH-stage.
In the following we describe our implementation of the PS-stage.
We begin by describing the two-phase process by which a location tuple is chosen and then describe the losses we use to train this process.
In the below description it will be helpful to deﬁne the set S OHoutcomes = { fail }∪ ( { 0 , 1 }×{ 0 , 1 , 2 }×{ 1 , ..., 7 } 2 ) where fail ∈ S OHoutcomes denotes the event in which the OH-episode did not successfully choose a hiding location (so the object was simply dropped) and ( s, m, i, j ) ∈
S OHoutcomes denotes the event in which, during the OH-stage, the agent successfully hid the object and
• the hiding agent was standing when it hid the object if s = 1 and was otherwise crouching, • the hiding agent hid the object with a PlaceAt|m,i,j action.
For simplicity, we will say an object was hidden at ( (cid:96), e ) to mean that, during the OH-stage, the agent was located at (cid:96) and e ∈ S OHoutcomes occurred.
We now describe the two phase process for choosing a location tuple.
Initial evaluation (recall Fig, 3b).
The agent assigns a score to all location tuples with the goal of highlighting locations that deserve further evaluation.
This score, for a particular location tuple (cid:96) , is computed as the weighted average
[Equation]
where P (cid:96) and V (cid:96) are generated by the agent and are described below.
Mental rollout (recall Fig. 3c).
The agent begins by picking the ﬁve highest scoring location tuples { (cid:96) 1 , ..., (cid:96) 5 } from the previous phase along with ﬁve randomly selected location tuples { (cid:96) 6 , ..., (cid:96) 10 } .
For each 1 ≤ i ≤ 10 , the seeking agent simulates the outcome of running the OH-stage with the agent at at location (cid:96) i three times.
Namely the agent iteratively samples three distinct elements
e i 1 , e i 2 , e i 3 ∈ S OHoutcomes with the probability of sampling e ij ∈ S OHoutcomes at every iteration being proportional to softmax ( P (cid:96) i ) e i j .
Then, for each 1 ≤ i ≤ 10 and 1 ≤ j ≤ 3 , the agent simulates 50 rollout trajectories ( t ijk ) 50 k =1 of the seeking agent in the hypothetical setting in which the object was hidden at ( (cid:96) i , e ij ) .
Let µ ij = 1 50 (cid:80) 50 k =1 #steps ( t ijk ) be the average number of steps taken in the simulated seeker trajectories ( t ijk ) 50 k =
Straightforward analysis, using the probabilities deﬁned in softmax ( P (cid:96) i ) , shows that there is closed-form solution for weights w ij so that µ i = (cid:80) 3 j =1 w ij µ ij is an unbiased estimator of the expected mental rollout path length when hiding at ( (cid:96) i , e ) and e is selected from S OHoutcomes with probability softmax ( P (cid:96) i ) e .
After computing µ = ( µ 1 , ..., µ 10 ) , the location tuple (cid:96) i where the actual OH-episode will begin is chosen by sampling i from a Multinomial ( softmax (0 . 04 · ( µ 1 , ..., µ n ))) distribution.
We now describe the losses minimized so that we obtain high quality initial evaluations of location tuples and mental rollouts of seeker trajectories.
Given a tensor M encoding the agent’s map of the environment (see the Metric mapping section below), the perspective simulation model produces an intermediate representation of this map M 1 .
From M 1 it generates two tensors P (cid:96) and V (cid:96) of size 2 × 3 × 7 × 7 as well as two values p (cid:96) none , v (cid:96) none ∈ R .
For simplicity we will abuse notation and let P (cid:96) none = p (cid:96) none and V (cid:96) none = v (cid:96) none .
Now softmax ( P (cid:96) )
e is a prediction of the probability that, were the agent to choose to hide the object at location (cid:96) , the event e ∈ S OHoutcomes would occur.
This tensor softmax ( P (cid:96) ) is supervised using a cross-entropy loss with the hiding location chosen during the OH-stage.
The entries of the tensor V (cid:96) then represent an estimate of the quality of a hiding location.
The collection of tensors V (cid:96) is supervised as follows.
Let 1 ≤
i (cid:54) = i (cid:48) ≤ 10 and 1 ≤ j (cid:54) =
j (cid:48) ≤ 3 .
Now let L Vij,i (cid:48)
j (cid:48) be the binary cross entropy loss between V (cid:96) i e ij − V (cid:96) i (cid:48)
e i (cid:48) j (cid:48) and sign ( be larger than V (cid:96) i (cid:48)
e i (cid:48) j (cid:48)
if and only if µ ij > µ i (cid:48) j (cid:48) , namely in those settings where mental rollouts suggest that ( (cid:96) i , e ij ) is a better hiding location than ( (cid:96) i (cid:48) , e i (cid:48) j (cid:48) ) .
We thus let our loss supervising the tensors V (cid:96) equal the average L V =
(cid:0) 302 (cid:1) − 1 sum ( {L Vij,i (cid:48)
j (cid:48) | 1 ≤ i ≤ i (cid:48) ≤ 10 , 1 ≤ j ≤ j (cid:48) ≤ 3 , ( i, j ) (cid:54) =
( i (cid:48) , j (cid:48) ) } ) .
Now we discuss losses used to guide learning to produce mental rollouts of seeker trajectories.
Suppose that a full game of cache has taken place.
We begin by collecting several pieces of data from the game.
The location tuple (cid:96) during the PS-episode and the event e ∈ S OHoutcomes that occurred during the OH-stage.
The trajectory ( t i ) ni
=1 (ordered set of location tuples visited) by the seeker.
A { 0 , 1 } -valued vector v wouldsucceed with i th entry equalling 1 if and only if the seeker would have been able to successfully claim the object was visible at the i th step of its trajectory.
The actions a i taken by the seeking agent taken at step i along with their success signal s i .
The policy, probability distribution over actions, π i of the seeker at step i .
The hiding agent then forms, from the intermediary representation M 1 and pair ( (cid:96), e ) , a hiding-placeconditional representation M (cid:96),e .
Using this conditional representation as a “mental environment,” the agent then simulates an internal seeker who followed the same trajectory ( t i ) ni
=
and took the same actions ( a i ) ni
=1 as the true seeking agent.
At every time step i the hiding agent generates a mental policy (cid:101)
π i for the simulated seeker along with a vector of probabilities (cid:98) s i where (cid:98) s
ij denotes
the predicted probability that a seeking agent, at the location t i , taking action with index j would succeed in doing so.
For every i , the mental policy π i is then trained, using a cross entropy loss, to mimic the true seeker policy π i .
Similarly at every step, the predictions (cid:98) s ij of whether actions would, or would no succeed, is supervised using a binary cross entropy loss with s i if j is the index of action a i .
Furthermore each (cid:98) s ij corresponding to the ClaimVisible action is supervised binary cross entropy with v wouldsucceed .
C. A DDITIONAL LOSSES
To improve training sample efﬁciency we augment the usual A3C losses, described below, by also computing several other losses, these losses are averaged across timesteps within each stage and then summed together along with any other losses, e.g. the A3C loss, and gradients are computed for this total loss before being synchronized with the shared model.
As there are a number of additional losses, we will not describe the full mathematical details of these losses and instead describe their intent.
Let s ∈ S stages , t ≥ 0 be the current time step in s -episode, and i t be the image observed at that time step, if applicable.
Other than the A3C loss, there are two additional losses shared among the { E&M, OH, OM, S } stages described below:
Encouraging understanding the effect of actions: we would like for the DIR representation learned by our agents to be self-predictive, namely we would like the DIR to encapsulate how the environment will change when acted on by the agent.
To this end, recall the notation from Section 4 where s t represented the 128 × 7 × 7 SIR output from CNN θ at time t and h ot represents the 512-dimensional hidden output vector created generated by textsclstm θ at time t .
Moreover let a t be the action taken by the agent at time t .
To encourage self-predictivity, we then deﬁne two learnable embedding functions E 1 θ and E 2 θ and deﬁne a loss which is minimized when the dot product E 1 θ ( s t +1 ) (cid:12) E 2 θ ( h o t , a t ) is large while E 1 θ ( s i ) (cid:12) E 2 θ ( h o t , a t ) is small for all i (cid:54)
=
t .
Minimizing this loss encourages h ot to contain enough information to be able to distinguish the true next visual observation s t +1 from all other observations seen during the episode.
Moreover it encourages the SIR s t +1 to contain information that will make it readily distinguishable from other the SIRs of other timesteps.
Localizing changes in input images: similarly to the above, we would like our agent to be able to localize where changes have occurred in input images.
As described in the neural models section below, at every time step the agent generates a 1 × 7 × 7 tensor W s,t where softmax ( W s,t ) 1 ,i,j should is encouraged to be nonzero only if the pixels corresponding to C ij have changed from the previous to the current frame.
To supervise W s,t we ﬁrst generate the 1 × 7 × 7 , { 0 , 1 } -valued, tensor D where D 1 ,i,j equals one if and only if the pointwise difference between the encodings of i
t
and i t − 1 generated by the ﬁrst convolutional block of CNN θ , has 10% of the spatial locations corresponding to the i, j grid location with (cid:96) 2 -norm greater than 10 − 3 .
We then form the loss
[Equation]
Note that minimizing L encourages the sum of the entries in softmax ( W s,t ) corresponding to entries in D equalling one to be large.
Beyond the above losses shared among all of { E&M, OH, OM, S } , we also include several stage-speciﬁc losses described below.
OH-speciﬁc losses
Avoiding early PlaceAt modality collapse: early in training the hiding agent makes many failed PlaceAt actions, as a large number of these actions are impossible for a given viewpoint, incurring a substantial penalty for these repeated failures.
Because of this, we found that, in this stage, the agent often would often quickly learn to only use the O N T OP modality of the PlaceAt as this is the modality is, generally, the easiest for the manipulator to successfully complete.
To prevent this policy collapse, we introduce a loss that encourages the agent to maintain probability mass on PlaceAt actions with C ONTAINED I N and B EHIND modalities.
This is similar to the standard loss in A3C encouraging higher entropy in an agent’s policy.
This loss is removed after 5,000 episodes of training.
Avoiding early PlaceAt spatial collapse: similarly as above, we sometimes found that the agent’s policy would concentrate on PlaceAt actions at a single grid position i, j early in training.
To prevent this, we create a loss encouraging the probability mass over PlaceAt to be similar to a prediction of which PlaceAt actions would be successful coming from the manipulator model (described below).
As above, this loss is removed after 5,000 episodes of training.
Recalling PlaceAt actions: to encourage the agent to remember where it attempted to place the object and where, on those attempts, the object ended we designed two losses penalizing the agent for failing to accurate make such predictions correctly.
Closing objects: after placing an object so that is in the C ONTAINED I N modality we design a loss encour- aging the agent to attempt closing objects.
OM-speciﬁc losses .
Better sample efﬁciency: inspired by hindsight experience replay (Andrychowicz et al., 2017)
, we design a loss that rephrases failed trajectories made during an OH-episode, i.e. a trajectory that placed the object in a location that was not speciﬁed as the goal location, into a successful trajectory where the goal was whichever location the object actually was placed in. 2.
Opening objects before placing things into them: we designed a loss that encouraged the agent, during the OM-episode, to try opening an object whenever the goal location for the object was in a C ONTAINED I N modality.
Predicting if a goal object location was attainable: for better sample efﬁciency and for use during other stages, we encourage the agent to predict whether or not it will be able to manipulate the current object in such a way that it will reach the given goal object location.
In particular, assume that the agent was directed to place the object in location ( m, i, j ) .
The hiding agent then produces, at every step, a 3 × 7 × 7 tensor T hittable,t and, at the end of the episode, computes an average binary cross entropy loss between the entries T hittable,t m,i,j and the indicator δ OM-episodesuccess .
PS-speciﬁc losses .
See the section on training perspective simulation above.
S-speciﬁc losses .
Better sample efﬁciency through self-supervised imitation learning: using the mental representation of the environment created by hiding agent in the PS-stage we compute a shortest path from the seeking agent’s current location to a location from which, in this mental representation, the object is visible and then en- courage the seeking agent to follow this path.
Reducing false claim visible actions: at every step made by the seeker we encourage (or discourage) the agent from taking the ClaimVisible action based on whether or not the goal object is currently visible to the agent.
C. R EWARD STRUCTURE FOR REINFORCEMENT LEARNING
In order to compute the A3C loss we must deﬁne, for each stage s ∈ S stages , a reward structure, that is, a function which assigns a reward r ∈ R to every state-action-state triplet ( s, a, s (cid:48) ) , where s denotes a possible agent state and a an action which, when taken from state s , resulted in the agent moving into state s (cid:48) .
When applicable and unless otherwise stated, agents obtain a penalty of r step = − 0 .
on every step to discourage long trajectories and an additional penalty of r fail = − 0 . 02 whenever an action fails (for instance, when running into a wall).
For clarity of exposition we will deﬁne the reward structure by deﬁning r in the case that agent starts at a generic state s at time t ≥ 0 and performs action a resulting in a transition to state s (cid:48) .
E&M reward structure.
During exploration and mapping, we would like to encourage the agent to comprehensively explore the environment, including interacting with and opening objects if possible, with particular attention being paid to areas that would make good hiding locations.
To this end, but for r E&M,hide ,t described below, we deﬁne the E&M reward structure in Algorithm 1.
Now for 0 ≤ s ≤ t + 1 , at time s let m s be the metric map generated by the agent, (cid:96) s be the location tuple of the agent, and let v s be the value of the position (cid:96) s when evaluating m s using the perspective simulation module.
Now letting unseen s = δ { with action a s the agent opened an object, or entered a location, for the ﬁrst time } , we deﬁne
[Equation]
so that v t captures an average quality of unique states the agent has visited up to and including time t .
We then deﬁne the reward
[Equation]
Note that r E&M,hide ,t will be 0 if the agent has not opened a new object or entered a previously unvisited location tuple and, otherwise, will only be large if the perspective simulation module considers the current state to be substantially more valuable than previously visited states.
Thus r E&M,hide ,t provides a signal through which the exploration agent may learn to visit states that the perspective simulation module considers good hiding locations.
OH reward structure.
Except for the deﬁnition of r OH,percentile , the reward structure for the OH episode is deﬁned in Algorithm 2.
In the below we deﬁne and motivate r OH,percentile .
Suppose that the during the OH episode the hiding agent has successfully completed a PlaceAt action, we would like the reward given to the agent after its ﬁnal action to reﬂect the quality of the hiding location chosen and thus a natural reward for this ﬁnal action is number of steps taken by the seeker to ﬁnd the object N Ssteps normalized by the maximum number of steps allotted to the seeking agent (500), i.e. the reward
[Equation]
Unfortunately r Ssteps fails in two respects.
The reward r Ssteps suffers from very high variance.
As an example, consider the case in which the object is hidden directly behind the seeker.
If the seeker performs two RotateRight actions it will immediately ﬁnd the object but, if it instead chooses to search the area ahead of itself then it may be many steps before it backtracks to its starting position and ﬁnds the object.
In such a case the reward r Ssteps may be very large or very small with high probability.
The number of steps required to ﬁnd the object is not only determined by the quality of the hiding place chosen in the OH episode but also by the topology of the scene and the location tuple chosen in the perspective simulation episode.
Namely, in a large, maze-like, room the OH episode may terminate with the object in the open from the hiding agent’s perspective (a poor hiding place) but still require many steps from the seeker to ﬁnd the object as the location tuple chosen in perspective simulation was far away from the seeker’s starting position
This high variance and confounding with the quality of perspective simulation results in r Ssteps providing a poor signal as to where an object should be hidden in the OH episode.
While the high variance of the reward is intrinsic to the game of cache, especially so in the case of randomly initialized agents, we can partially address the second issue above by considering a transformation of r Ssteps deﬁned as follows.
First we generate 100 sample seeker trajectories ( s i ) 100 i =
when simulating hiding the object at the location chosen during the PS- stage; see the training perspective taking section above for details as to how these trajectories are generated.
Letting (cid:96) i be the length of s i , i.e. the number of steps the seeker took to ﬁnd the goal object in that mental rollout, we then deﬁne
[Equation]
Assuming these mental rollouts are of reasonable quality, r OH,percentile naturally calibrates the reward for the topology and size of the scene.
This helps disentangle the ﬁnal reward in the OH episode from contributions made purely during the PS episode.
OM reward structure.
The OM episode runs for a maximum of 50 steps and has the reward structure for this episode is outlined in Algorithm 3.
Broadly speaking, we reward the agent for dropping the goal object near the target location with proximity measured in the image space and for placing the object in a location respecting the target modality among O N T OP , C ONTAINED I N , and B EHIND .
We found it especially important to give the agent a large negative reward whenever it placed the object outside of the agent’s viewport.
S reward structure.
The reward structure of the seeking episode is simple and encodes our desire for the agent to locate the object in as short as time as is possible without having a large false positive rate, that is without having many instances in which the agent claims to be able to see the object when the object is not actually visible.
This reward structure is given in Algorithm 4.
C. P RETRAINING THE E&M AND OM STAGES
As the ability to navigate about a room and perform object manipulation are basic competencies for the hiding agent in cache, we pretrain our hiding agent on the E&M and OM-stages until it attains reasonable proﬁciency.
The reward structure when training the E&M-stage was essentially identical to that from Algorithm 1 but without the r E&M,nhide, t reward, this trained for approximately 85,000 episodes.
To pretrain the OM-stages we generated, across AI2-THOR scenes, a collection of plausible locations the objects could be placed when at location tuples, and then trained the hiding agent to attempt placing objects in these locations using the reward structure in Algorithm 3; the manipulator agent was pretrained for approximately 1.8 million episodes.
We ran this pretraining procedure for approximately three days.
D N EURAL MODELS
Here we give an overview of the neural network architectures we use to parameterize our agents.
As these architectures can be complex, we do not attempt to make this description sufﬁciently comprehensive to exactly reproduce our results; we instead direct any parties interested in extract reproduction to our code base.
Our neural network architecture contains components that are shared among both the hider and seeker as well as components that are unique to individual stages.
Recall from 4 that shared components include the image processing CNN CNN θ , the recurrent networks CGRU ) θ and LSTM θ and metric mapping components.
Previ- ously unmentioned shared components include embeddings for actions, goal objects, and agent standing state.
Unshared components include, for example, the MLPs deﬁning actor-critic heads.
D. I NFORMATION FLOW TO POLICY AND VALUE
Here we describe how, during the E&M-, OH–, OM-, or S-episode, data ﬂows through our agents’ neural architecture to produce the agent’s policy and value of the current state.
Suppose the agent has received input corresponding to timestep t ≥ 0 in stage s ∈ S stages and must take an action.
This input comprises the following.
• A 3 × 224 × 224 (channels × height × width) tensor corresponding to the image of the agents’ current view.
This tensor is processed CNN θ to produce an SIR s t of shape 128 × 7 × 7 .
• A non-negative integer specifying the agent’s last action (including a ‘none‘ action when t = 0 ) along with a ‘success’ or ‘failure’ signal indicating whether the agent’s last action succeeded.
This pair of inputs is used to index into an embedding matrix resulting in a 16-dimensional embedding vector.
• A non-negative integer specifying whether or not the agent is standing.
This integer is also used to index into an embedding matrix resulting in a 4-dimensional embedding • A non-negative integer specifying the goal object (i.e. object to hide or object to seek).
This integer is also used to index into an embedding matrix resulting in a 16-dimensional embedding vector.
• The 257 × 7 × 7 -dimensional hidden state (cid:101) h t − 1 corresponding to the output of the convolutional GRU CGRU θ from the previous time step.
If t = 0 then (cid:101) h t − 1 are tensors ﬁlled with zeros.
• The 512 -dimensional hidden states
h ot − 1 , h ct − 1 corresponding to the output of the LSTM LSTM θ from the previous time step.
If t = 0 then these tensors are identically zero.
• The agents’ current metric map of it’s environment, this includes a 144-dimensional vector for every unique location tuple the agent has visited.
• (OH only)
If the last action taken in the OH-episode was a PlaceAt , an action embedding specifying the location location the OM-episode placed the object, e.g. if the OM-episode placed the object in the C ij grid location with modality B EHIND , then the agent obtains the embedding for a successful PlaceAt|2,i,j action.
As an object can potentially land in many grid locations and modalities simultaneously, the chosen action corresponds to the location and modality containing the largest number of the object’s visible pixels.
• (OM only)
The modality m and grid location i, j where the object should be placed.
This triple is used to slice into from a parameter tensor of size 3 × 8 × 13 × 13 retrieving a 8 × 7 × 7 tensor which
is concatenated to the 128 × 7 × 7 tensor of visual features.
Excluding the recurrent states and the map, each of the above embeddings are repeated (if necessary) so that they are of shape N × 7 × 7 and concatenated together with the 128 × 7 × 7 SIR.
For instance, the 16-dimensional object embedding becomes a 16 × 7 × 7 tensor T where T : ,i,j is a copy of the original embedding for each i, j .
To this tensor is additionally concatenated the ﬁrst 256-dimension of h ot − 1 after having been tiled to be of shape 256 × 7 × 7 .
This concatenated tensor is then fed into CGRU θ along with the hidden state (cid:101) h t − 1 producing (cid:101) h t .
The tensor (cid:101) h t − 1 is then split into two components.
It’s ﬁnal channel, (cid:101) h t − 1 [256 , : , :] , is removed and becomes the tensor W s,t described in Sec.
C.4.
The remaining 256 × 7 × 7 tensor is then averaged pooled to create the 256 -dimensional vector h t .
Now the network reads from the input metric map using both an attention mechanism and a collection of convolutional layers resulting in an embedding of size 356.
This embedding is concatenated to h t and fed, along with the hidden states ( h ot − 1 , h ct − 1 ) , into LSTM θ producing the hidden states ( h ot , h ct ) .
For each stage X ∈ { E&M, OH, OM, S } there is now a unique MLP MLP Xθ corresponding to a 512 ×linear layer followed by a ReLU non-linearity.
Additionally there are unique networks ACTOR Xθ (correspond- ing to a 512 × (number of actions in stage X ) linear layer followed by a softmax nonlinearity) and VALUE Xθ corresponding to a 512 × 1 linear layer.
Now the agents’ policies and values for each stage are computed as ACTOR Xθ ( MLP Xθ ( h ot + h ot )) and
VALUE Xθ ( MLP Xθ ( h ot + h ot )) respectively.
Finally the agents then write into their map, at a position deﬁned by their current location tuple, a 128- dimensional vector extracted from one of the ﬁnal layers of the CNN θ (when applied to i t ) along along with the 16-dimensional object embedding.
D. M ETRIC MAPPING ARCHITECTURE
Intuitively, our metric mapping architecture is designed so that an agent can record information regarding its current state for later retrieval.
This map is called metric as data is written into the map so that it is spatially coherent, that is, where data is written in memory is uniquely deﬁned by the agent’s location tuple when it is written.
Our metric map model implements is designed to allow for the following.
• Writing: given the current location tuple of an agent, passes a given value through a Gated Recurrent Unit (GRU) (Cho et al., 2014) whose hidden state is taken from the last value saved at that location (if any, otherwise a learned initial state is used) and this output of the GRU then overwrites the previously saved values.
• Reading at location: given a location tuple, returns the value saved at that location.
• Reading with attention: given an input key, uses a soft-attention mechanism to return a weighted sum of saved values.
• Reading as tensor: return a 4 × 2 × 144 × h × w tensor M where M r,s, : ,i,j is the value written into the map when the agent was most recently at location tuple (min( x values written into map ) + j · 0 . 25 , max( z values written into map ) − i · 0 .
25 , s, r ) , h is the number of distinct z values visited by the agent, plus optional padding, as it built the map, and w is the number of distinct x values written into the map, again possibly with optional padding.
Notice that decreasing i [Table]
• Reading egocentric map: the same as reading as map but with spatial coordinates centered on the agent and appropriately transformed so that north is whichever direction the agent is currently facing.
Using the above, the agent can build a map while it explores its environment while querying that map locally and globally so as to make decisions.
D. V ISUAL DYNAMICS REPLAY DECODER ARCHITECTURES
Recall from our training and losses section, that in VDR we are given a triplet ( i 0 , a, i 1 ) , where i 0 , i 1 are two images with i 1 occurring after an agent took action a , and must, accomplish a number of self-supervised tasks.
To accomplish these tasks we require several neural network architectures building upon our SIR generating architecture CNN θ .
These architectures are all convolutional neural networks with the largest of these networks is inspired by spatial transformer networks (Jaderberg et al., 2015), as well as U-Nets, and contains 13 convolutional layers, 10 of which are transposed convolutions which work to upsample the input tensor, the SIR output from CNN θ , to a ﬁnal size of 16 × 224 × 224 after which 1 × 1 convolutions are used to produce (cid:98) i
[Table]
as well as T conﬁdence and T pix.
diff. .
The tensor T wouldsucceed is produced by a convolutional network acting on
CNN θ ( i 0 ) and the embedding of a while the value v inv.
dyn. is produced by a convolutional network applied to a concatenation of CNN θ ’s embeddings of both i
and i 1 .
Implementation details can be found in our code base.
D. P ERSPECTIVE SIMULATION ARCHITECTURE
The perspective simulation architecture operates on the tensor representation of the map, see the metric mapping architecture above, which is reshaped to be of size 1152 ×
h × w where h and w are the height and width of the map respectively.
First this map is concatenated with a grid of distances from the agent start location after which it is processed by several 1 × 1 convolutional blocks to form the intermediary representation M 1 of the map.
An additional 1 × 1 convolution produces, for each location tuple (cid:96) (merging those which only differ in that the agent is, or is not, standing) after and after reshaping, two tensors P (cid:96) and V (cid:96) of size 2 × 3 × 7 × 7 as well as two values p (cid:96) none , v (cid:96) none ∈ R .
For simplicity we will abuse notation and let P (cid:96) none = p (cid:96) none and V (cid:96) none = v (cid:96) none .
Next, given a hypothetical hiding outcome e ∈ S OHoutcomes another convolutional neural network, acting on M 1 along with an embedding of e indicating where in M 1 the object is being hidden, produces the second intermediary representation of the map M e .
A “mental” seeker agent is then deﬁned in this architecture using an LSTM whose observations come from slices of M e .
This mental seeker can move about the map deﬁned by the representation in M e and, at each step i , produces the vectors (cid:101)
π i and (cid:98) s i deﬁned in the training in losses section by using two linear layers acting on the mental agent’s hidden state.
E H UMAN AND SELF EVALUATION FOR CACHE AGENTS
In Figures 4a to 4c we report the results of evaluating our hiding and seeking agents from three points during their training as they compete against one another as well as against hiding places chosen by an exhaustive approach and hiding places chosen by humans.
In the following we discuss how we collected hiding locations for the various agents and ran our seeking trials.
E. G ENERATING HIDING LOCATIONS
Human agents.
We recruited 16 human volunteers to hide objects within AI2-THOR.
Participants were in- structed that they would be randomly positioned in a simulated environment holding an object which they were then expected to hide so that a seeking agent, who started in the same position that they started in, would ﬁnd the object difﬁcult to ﬁnd.
Furthermore participants were instructed to be in the mindset that they might be playing against the same seeker many times and thus should choose diverse hiding locations.
Before beginning, participants were allowed to familiarize themselves with the interface until they judged themselves sufﬁciently competent to begin the task.
Unlike our artiﬁcial agents, humans subjects were not constrained in the number of actions they were allowed to take and, after hiding their object, were allowed to pick it up again and try other hiding locations if desired.
We collected 600 human hiding locations evenly distributed among all object types and training, validation, and test scenes.
To reduce the amount of data to be collected we used only ten (of 40) training scenes (AI2-THOR ﬂoor plans 7, 10, 13, 16, 18, 203, 209, 213, 215, and 218).
Artiﬁcial agents.
We generate hiding locations from our artiﬁcial hiding agent using the agent after it had trained for 0, 10 6 and 4 . 5 · 10 6 episodes.
We call these agents h 0 , h 1 e 6 , and h 4 .
5 e 6 .
For each of these agents we generate 4,000 total hiding locations, 2,000 each for the train and test scenes.
The train (test) hiding locations are generated such that for each (scene, object type) pair we have an equal number of hiding locations.
For fair comparison, we also generated a set of 1,545 additional hiding locations from h 4 .
5 e 6 using the same scenes, object types, and agent starting locations as those from the human trials (with multiple repeats per agent start location).
Exhaustive approach.
Recall the exhaustive automated approach for generating hiding places of easy, medium, and hard difﬁculties detailed in the cache stages section.
Using this approach we generate, for each of these difﬁculties, 3,999 hiding locations in the train scenes and 998 in the test scenes.
Because of our auto- mated, metric-based, approach for generating these hiding spots there were three cases (one in the train scenes and two in the test scenes) where there were not be sufﬁciently many hiding locations 20 desired locations per object, scene, and difﬁculty.
In such cases we reduced the number of locations saved for that object and scene pair for each difﬁculty level so that the total number of hiding places in each difﬁculty type remained equal.
E. R UNNING SEEKING TRIALS
In these trials a seeking agent is required to ﬁnd an object hidden in a hiding location from one of the above collections.
If, through automated checks, we ﬁnd that a hiding place is such that the hidden object cannot be seen from any reachable position in the scene then this hiding location is never shown to any seeking agent and all seeking agents are considered to have failed to ﬁnd the object for that trial.
Artiﬁcial agents.
We run seeking trials using the artiﬁcial seeking agents from the same three points in training and name these seeking agents analogously as s 0 , s 1 e 6 , and s 4 .
5 e 6 .
Human agents.
For seeking we recruited human participants using Amazon’s Mechancical Turk platform.
As in the hiding task participants were allowed to familiarize themselves with the interface until they judged themselves sufﬁciently competent to begin the task and were not constrained in the number actions they were allowed to take.
On starting a trial, the participant was randomly assigned a hiding location generated either by h 4 . 5 e 6 or from the set of 600 human locations.
To ensure fair comparison, we matched each of the 600 human locations to a unique hiding location generated by h 4 .
5 e 6 (from the additional set of 1,545) where the artiﬁcial hiding agent started the same position as the human hider.
After 100 seconds participants were allowed to give up, moving them to the next seeking trial, if they felt the object could not be found.
At the end of these trials we were left with complete seeking trials for all but two instances where invalid data was returned.
F S TATIC IMAGE REPRESENTATION EXPERIMENTS
In these experiments we consider SIRs produced by the model CNN θ when trained: (1) to play cache (taken our ﬁnal cache agent), (2) to do joint object class prediction using precisely the same images seen by our cache agent during training (here an object is considered present in an image if AI2-THOR returns that the object is visible) (3) to complete a navigation task (see Sec.
G.1), (4) using an autoencoding loss (i.e. pixel-wise mean absolute error) with 35,855 AI2-THOR images, or (5) with an image classiﬁcation loss on the 1.28 million hand-labeled images in ImageNet (Krizhevsky et al., 2012).
More details regarding these baselines can be found in Sec. .
We begin by describing our experiments for synthetic images before moving to our natural image experiments.
F. S YNTHETIC IMAGES F.1. E VALUATION DATA AND TASKS
To evaluate the quality of our SIRs we train decoder models on these SIRs to complete a number of distinct tasks described below.
Except for in the classiﬁcation task, where the decoder model is a max-pooling layer followed by a linear layer, the decoder models are designed similarly to those from a U-Net (Ronneberger et al., 2015), for details please see our code.
Depth prediction.
In AI2-THOR we generate a collection of 3 × 224 × 224 RGB images paired with their corresponding 224 × 224 depth maps.
Models are trained to predict the depth map given its corresponding RGB image by minimizing a per-pixel mean absolute error loss.
Surface normals.
In AI2-THOR we generate a collection of 3 × 224 × 224 RGB images where each image is paired with a corresponding 3 × 224 × 224 tensor T where T : ,i,j represents the surface normal at pixel i, j in the RGB image.
Models are trained to predict this tensor of surface normals given its corresponding RGB image by minimizing a per-pixel negative cosine similarity loss.
Classiﬁcation.
In order to generate a dataset for classiﬁcation we ﬁrst select 90 types of objects available in AI2-THOR and assign each of these types a unique index in 1, ..., 90.
Next, in AI2-THOR, we generate a collection of 3 × 224 × 224 RGB images where each image is paired with a { 0 , 1 } -valued vector of length 90 whose j th entry equals 1 if and only if an object whose type has index j is present in the image and whose pixels occupy at least 1% of the image.
Models are trained to predict this vector of 0,1 values from the input image using binary cross entropy.
Openability.
In AI2-THOR we generate a collection of 3 × 224 × 224 RGB images where each image is paired with a a 224 × 224 { 0 , 1 , 2 } -valued tensor T with entry T i,j equaling 1 if pixel i, j in the input image corresponds to an object that can be opened, equalling 2 if the pixel corresponds to an object that can be closed, and otherwise equals 0.
Models are trained to predict the { 0 , 1 , 2 } -valued tensor from the corresponding RGB image by minimizing a per-pixel mean cross entropy loss where the loss associated with predicting a 0 has a weight of 0.2 and the other losses have a weight of 1.0.
Traversable surfaces prediction.
In AI2-THOR we generate a collection of 3 × 224 × 224 RGB images where each image is paired with a corresponding 224 × 224 { 0 , 1 } -valued tensor T with T i,j equalling one if and only if the ij th pixel in the image corresponds to a surface onto which the agent can walk.
Models are trained to predict the { 0 , 1 } -valued tensor given its corresponding RGB image by minimizing a per-pixel binary cross entropy loss.
Object depth.
In AI2-THOR we generate a collection of 3 × 224 × 224 RGB images each where each image is associated with a collection of 224 × 224 { 0 , 1 } -valued masks of, sufﬁciently large, objects in the image paired with depth maps corresponding to the image in which the masked object has been removed.
Models are trained to predict, from an input RGB image and object mask, the depth map of the image where the object has been removed by minimizing a per-pixel weighted mean absolute error loss.
In this loss, pixels corresponding to the masked object are weighted to contribute equally to those pixels where the object is not present.
Object surface normals.
Similar to object depth but predicting surface normals after an object has been removed.
F.1. T RAINING
We train using an ADAM optimizer with ( β 1 , β 2 ) =
(0 . 99 , 0 . 999) , a weight decay of 2e-6, and a plateau-based learning rate scheduler beginning with a learning rate of 1e-2 and decaying by a factor of 10 with a patience of N patience , that is, the scheduler reduces when the loss on the validation set has failed to decrease by some small absolute quantity for N patience subsequent epochs).
When training CNN θ to perform auto-encoding and ImageNet classiﬁcation we used a patience of N patience = 5 and trained the models using all available training data.
When training our other tasks, we used a patience of N patience = 5 and subsampled the training data to include 1,200 examples.
Training was stopped if the learning rate ever decreased below 1e-4.
During training models were evaluated on the validation set after every epoch and, once training completed, the model with the best validation loss was returned as the ﬁnal model.
F.1. E VALUATION
To evaluate our models we compute the loss of each model on each of the test set examples for its respective task.
As any test set results generated within the same scene and at nearby location tuples are correlated, we use a generalized linear mixed effects model (GLMM) to obtain statistically meaningful comparisons between our models.
We run our analysis using the R (R Core Team, 2019) programming language, in particular we use the glmmPQL function in the nlme (Pinheiro et al., 2019) package to ﬁt our GLMM models and then the emmeans (Lenth, 2019) package to obtain p-values of contrasts between ﬁxed effects.
In these analyses we:
• Let the per test set example loss be our endogenous variable.
• Estimate ﬁxed effects corresponding to which SIR was used to produce the loss (auto-encoder, Ima- geNet, or cache).
• Include mixed effects corresponding to scene nested within scene type.
• Model spatial correlation, nested within scene, using spherical correlation where two test set exam- ples corresponding to different SIRs or with different rotations are considered inﬁnitely distant.
• Use a Gamma family model with an identity link.
• Subsample the test set to include at most 50 datapoints per scene and SIR so that the model can be more quickly trained.
To fully reproduce our results please see our code base.
F. N ATURAL IMAGES
We consider three distinct tasks for natural images, scene classiﬁcation on the SUNCG dataset Xiao et al.
(2010), depth estimation on the NYU V2 dataset Nathan Silberman & Fergus (2012), and walkable surface estimation also on the NYU dataset.
As for the synthetic image experiments, we use CNN θ as our backbone model.
F.2. E VALUATION DATA AND TASKS
Scene Classiﬁcation.
In scene classiﬁcation the goal is to predict, from a natural image, the type of scene the image is staged in.
For instance, scene categories include beach, palace, and sauna.
We evaluate our predictions using the (mean-per-class) top-1 accuracy metric.
Depth prediction.
As for synthetic images, our goal is to predict pixel-to-pixel depth but for natural images.
We evaluate our model using the standard absolute relative error of our predictions.
Walkable surface estimation.
This is synonymous with traversable surfaces prediction in AI2-THOR.
We evaluate the quality of our predictions using the standard IOU (intersection over union) metric where we only consider the IOU with walkable segments.
F.2. T RAINING
Scene Classiﬁcation.
Our decoder architecture for scene classiﬁcation corresponds to a 1-by-1 convolutional layer which transforms the 128 × 7 × 7 SIR output from CNN θ to a 64 × 7 × 7 tensor.
Two fully connected layers then transform this tensor into a vector of size 512 followed by vector of size 397.
For training we use the cross-entropy loss.
Depth estimation.
To perform depth prediction our network CNN θ is composed with a feature pyramid network (Lin et al., 2016) and pixel shufﬂe layers Shi et al.
(2016) are used for upscaling.
The Huber loss is minimized for training.
Walkable surface estimation.
This architecture is essentially identical as for depth prediction.
The architecture of this network is the same as the depth estimation network.
Our training object is pixel-wise binary cross entropy.
G D YNAMIC IMAGE REPRESENTATION EXPERIMENTS
In the following we describe our training procedure for our dynamic image experiments along with any task- speciﬁc details including deﬁnitions of our baselines, namely the labels in the legend of Fig.
For tractability of inference, we will always consider our training set D train to be ﬁxed and ignore any randomness, if applicable, in the generation of D train .
Each of our considered tasks is a discrimination task and requires predicting, from a given representation, whether or not some binary condition holds.
We call cases where the condition holds positive examples and others negative examples .
We will accomplish these predictions by ﬁtting a logistic regression models, details are given in the training section below.
G. B ASELINES
We will now describe our baselines referenced in the labels of Fig.
7 and in Sec. 5.
DIR (Cache).
The is the DIR generated by our trained cache agent.
In the tracking experiment (the experiment corresponding to Fig.
7b) this DIR corresponds to the 257 × 7 × 7 tensor (cid:101) h t (recall the notation from Sec. 4) produced by the trained CGRU θ model.
In all other experiments this corresponds to the 512-dimensional vector h ot produced by the LSTM θ architecture.
DIR (Nav.).
For comparison we trained a navigation agent to perform the seeking task with object locations chosen from the sets of hiding locations, of easy and medium difﬁculty, generated by our exhaustive procedure.
This navigation agent was trained, without VDR, for 150,000 episodes at which point performance appeared to saturate.
The navigation agent was trained with a similar architecture to our cache agent with a convolution GRU but without a ﬁnal LSTM θ model.
For the tracking experiment the DIR for this agent was taken as the 257 × 7 × 7 tensor output from its convolutional GRU.
For the other experiments, the DIR was taken as a weighted average pool of the aforementioned 257 × 7 × 7 tensor.
Note that, unlike the cache agent, the navigation does not have any actions corresponding to moving the object in its hand.
Because of this, in the below experiments, whenever we would tell the navigation agent that its last action was a hand movement action we instead tell it that its last action was a Stand action.
DIR (Rand.)
It is well-known that convolutional networks structurally encode powerful inductive biases for image processing and that even randomly initialized networks can provide surprisingly powerful features for downstream tasks.
This raises the question of whether or not the cache DIR performs well simply because our model architecture encodes strong inductive biases and, perhaps, that we do not need to train our model at all to obtain good results for our DIR tasks.
To address this question, DIR (Rand.) is generated identically to DIR (Cache) except that the underlying model weights are kept as their randomly initialized values.
SIR (Cache).
This baseline corresponds to the features produced by CNN θ trained by playing cache.
For the tracking experiment we use the entire 128 × 7 × 7 SIR produced by this CNN, in all other experiments we average pool this tensor to reduce it to a 128-dimensional vector.
SIR (ImageNet).
This model is identical to SIR (Cache) except the underlying CNN was trained with an image classiﬁcation loss on the 1.28 million hand-labeled images in ImageNet (Krizhevsky et al., 2012).
SIR (Class.)
One may wonder if a CNN trained to perform classiﬁcation on AI2-THOR images would obtain similar performance to our cache-trained model.
To answer this question, while training our cache agent we have, in parallel, trained a CNN θ model to classify AI2-THOR objects.
Critically, this model was trained with exactly the same set of images as our cache agent and received exactly the same number of gradient updates.
Objects were considered to be present in an image if they were considered visible by AI2-THOR.
To train this model we minimized a binary cross entropy loss after average pooling the ﬁnal representation from CNN θ to obtain a 128-dimensional vector after which we applied a 128 × 101 linear transformation giving us thebinary logits used for prediction (of the 101 applicable AI2-THOR objects).
As for SIR (Cache), we use the 128 × 7 × 7 tensor for the tracking experiment and, for all other experiments, average pool this vector to be of size 128.
Forward Steps.
This baseline is only included in the “Contained or Behind?” experiment.
In this experiment there is a non-trivial bias in the dataset, namely that an object needs to be moved further forward to be placed behind a receptacle than it does to be placed into it.
To ensure that DIR (Cache) is not simply exploiting this bias, we have included the count of the number of forward steps as a baseline.
DIR (4CNN-Cache/Class/ImgNet).
An straightforward strategy by which to produce a DIR from SIRs is concatenation.
Namely, if we have an SIR s t , then it is we may build a representation that “integrates observations through time” simply by concatenating several such SIRs together.
That is, a DIR representation at time t could simply be the concatenation [ s t − 3 , s t − 2 , s t − 1 , s t ] .
The three baselines DIR (4CNN-Cache), DIR (4CNN-Class), and DIR (4CNN-ImgNet) each take this approach by concatenating the SIR from time t with the SIRs from the three previous timesteps.
For DIR (4CNN-Cache), SIR (Cache) representations are concatenated (and similarly for DIR (4CNN-Class), and DIR (4CNN-ImgNet)).
For the tracking experiments these DIRs are thus of shape 512 × 7 × 7 while for the the other experiments these DIRs are vectors of length 512.
As we wish these DIRs to be strong baselines, in the “Contained or Behind?”
experiment we also concatenate to these DIRs the Forward Steps feature described above (producing 513-dimensional DIRs).
G. T RAINING
We wish to evaluate how our considered DI- and SI- representations perform in our tasks as the amount of data available for training varies when using a training procedure that follows best practices in machine learning by using dimensionality reduction and cross validation.
Our procedure, based on iteratively subsampling D train , follows.
Suppose we are interested in our representations’ performance given a training set of size n > 0 with n < | D train | .
We then repeat the following R > 0 times.
Subsample a training set D n ⊂ D train of size n taking care to retain as close to the same proportion of negative and positive examples as possible.
Perform dimensionality reduction on D n using (a variant of) principal component analysis, reducing the datasets dimensionality so that at least 99.99% of variance is explained by the remaining components, to obtain a dataset (cid:101) D n . 3.
Fit a logistic regression model on (cid:101) D n using cross validation to choose an appropriate weighting of (cid:96) 2 - regularization.
Apply the principal component analysis transformation learned in the second step to the test set D test to obtain a (cid:101) D test and then compute the test-accuracy on this transformed dataset.
This gives us R accuracy values a n 1 , ..., a nR from which we can estimate the accuracy of our model on test examples when training our model on a randomly sampled training set of size n as the mean a n = 1 R (cid:80) R i =1 a ni .
Note that randomness in the estimator a n comes both from our resampling procedure and, assuming our test set is not exhaustive, from the randomness in which examples where chosen to be included in the test set.
The ﬁrst source of randomness can be reduced simply by increasing R while the second can only be reduced by gather more test set examples.
When appropriate, we account for both sources of randomness when plotting conﬁdence intervals.
G. C ONTAINED OR BEHIND RELATIONSHIPS
For these experiments we generate a dataset using the ﬁrst ﬁve scenes of the foyer type in AI2-THOR.
For each of these ﬁve scenes we generate a large collection of paired examples of objects being places into, or behind, one ﬁve receptacle objects while that receptacle sits on one of ﬁve possible tables at varying offsets from the agent.
The placed objects are the same as those hidden in cache and we use ﬁve different receptacle objects of varying size.
In order to make our setting especially difﬁcult, we modify the above training procedure so that, in step 1 above, we restrict our subsampled training set to exclude a randomly selected combination of scenes, objects, receptacles, and tables.
In step 4, our testing set is those examples in which the scene, object, receptacle, and table are all among those excluded from the training set.
This makes the test examples especially challenging as they are, visually, very dissimilar from those see during training.
In this setting, only consider randomness as a result of our resampling procedure in our error estimates and so our inference is restricted to within our generated set of examples.
Here predictions are made using the representation at the timestep after the object has been moved into its position but before it has been let go by the agent ( i.e. before a DropObject action has been taken).
G. O CCLUDED OBJECT TRACKING
As in our contained or behind relationship experiments we generate a dataset using the ﬁrst ﬁve scenes of the foyer type in AI2-THOR and ﬁve table types.
Here, as we are interested in tracking completely occluded objects, we use only three of our ﬁve item items, the cup, knife, and tomato, as these are sufﬁciently small to be frequently occluded.
For each conﬁguration of object, table, and scene, we generate a collection of object trajectories as the object moves behind a collection of books, recall Fig.
7b.
For additional variability we produce many such trajectories with the books positioned in different distances and orientations.
As the object moves behind the occluding books, we record all grid locations C ij that the object passes through into the set P (it is considered to have passed through a grid location C ij if, at any point during its trajectory behind the books, at least 10 pixels of the object were in C ij after making all other objects invisible).
At every step of a trajectory where the object is occluded such that less than 10 pixels of the image correspond to the object we generate | P | data points, one for each grid location C ij ∈ P with the grid location being considered a positive example if, after making all other objects invisible, the moving object has greater than 10 pixels in that grid location and otherwise being consider negative.
In the paragraph below we describe how we obtain the SIR and DIR features corresponding to each such data point.
Note that, as objects tend to occupy only a small portion of an image, there are many more negative examples in our dataset than positive ones.
During training we always subsample negative examples so that there is an, approximately, equal number of positives and negatives.
Furthermore, during testing, we implicitly balance positive negative examples by reweighting; thus the accuracy we report in this setting is a balanced accuracy.
As in the previous experimental design, we modify the above training procedure so that, in step 1 above, we restrict our subsampled training set to exclude a randomly selected combination of scenes, objects, and tables.
In step 4, our testing set is those examples in which the scene, object, and table are all among those excluded from training.
Again as above, only consider randomness as a result of our resampling procedure in our error estimates and so our inference is restricted to within our generated set of examples.
Recall from our Sec.
G.1, all SIRs and DIRs in this experiment have shape N × 7 × 7 for some N ≥ 0 .
Let T be such an SIR or DIR and suppose that, at time t > 0 , the object is fully occluded and X is a 7 × 7 { 0 , 1 } -valued matrix with X ij equalling one only if grid position C ij is considered a positive example.
To obtain, from T , the features used to predict X we simply take a slice from T at position i, j .
Namely the features used to predict the value of X ij are T
[: , i, j ] .
G. O BJECT PERMANENCE WHEN ROTATING
For each of the hiding locations in our automatically generated easy hiding places, recall Sec.
E.1, we let our seeking agent attempt to ﬁnd the object for 500 steps.
If, during one of these episodes, the agent would successfully take a ClaimVisible action at time t , so that it has found the object, we consider the counterfactual setting in which the agent instead rotates right or left once at random and the, previously seen, object vanishes.
The agent then is allowed to take 4 more actions following its policy.
Each of the ﬁve representation generated after the rotation is saved as a positive example.
To create negative examples we replay this same episode, forcing the seeking agent to take all of the same actions, but with the goal object invisible during the entire trajectory.
From this replayed trajectory we save the ﬁnal ﬁve resulting representations as negative examples.
When sampling training sets we consider only training scenes and randomly exclude one object, when sampling test sets only testing scenes and examples including the previously excluded object.
G. F REE SPACE SERIATION
In our ﬁnal experiment we test the ability of our DIRs and SIRs to predict free space.
We construct our dataset for this task by, for every AI2-THOR living room and kitchen scene s , performing the following procedure.
Let N s = min(20 , round (0 . 15 · # unique reachable ( x, z ) positions in the scene on a 0.25m grid ) , and randomly select ( x 1 , z 1 ) , ..., ( x N s , z N s ) locations in s .
For every position ( x i , z i ) perform the following.
a. Teleport the agent to ( x i , z i ) with a random (axis-aligned) rotation, and randomly select a direction in which to rotate (left or right).
The ﬁrst set of rotations is a habituation phase.
b. Suppose we have chosen to rotate right.
Then require the agent to take seven RotateRight actions so that every orientation is seen exactly twice.
c. Save the DIRs, and SIRs, corresponding to the agents evaluation when looking in each orientation for the second time along with whether or not the current orientation has strictly more free space than the previous orientation.
Free space is measured by the count of the grid locations that the agent could
Input: Time step t ≥ 0 , full history of E&M episode H , number of steps taken in episode N
Output: Reward for E&M episode step at time t .
1 begin for i ← 1, ..., N do a i ← i th action taken in episode success i ← was i th action successful extrap t ← S extrap. E & M,t new location t ← true if the location of the agent at time t did not occur previously in the episode. r ← − 0 . if not success t then r ← r − 0 .
else // encourage the agent to explore new areas num new explored ← r ← r + 0 . 2 · ( extrap t − extrap t +1 ) / // encourage the agent to attempt opening objects new opened ← a t opened an object that was not previously opened in the episode if new opened then r ← r + 0 . // encourage the agent to explore areas considered good for hiding if new opened or new location t then r ← r + r E&M,hide ,t if a t is an OpenAt action then // never punish the agent heavily when taking an OpenAt action r ← max( r, − 0 .
001) return r
G.6. W HY DO OTHER DIR S OBTAIN HIGH PERFORMANCE ?
In Fig.
7d we see that several 4CNN-DIRs do nearly as well as DIR (Cache).
Initially this may seem surprising, our other experiments seem to suggest that our cache agent’s DIR is signiﬁcantly more capable than a simple concatenation of SIR features.
We argue, however, that this result is not at all surprising.
If an SIR can be used to predict the amount of free space
Input: Time step t ≥ 0 , full history of OH episode H , number of steps taken in episode N
Output: Reward for OH episode step at time t .
1 begin r ← − 0 .
for i ← 1, ..., N do a i ← i th action taken in episode success i ← was i th action successful if a t
= R EADY F OR S EEKER then if not success
i then r ← r − 0 .
else if success t and a t is an OpenAt action then // encourage the agent to attempt opening objects if the opened object was not previously opened in the episode then r ← r + 0 .
else if a t is a PlaceAt action but a PlaceAt action was previously successful then r ← r − 0 .
else if success t and a t is a PlaceAt action then //
Reward successful manipulation and encourage the agent to try hiding spots that are more difficult to manipulate to
p ← the probability the object manipulation module assigns to successfully accomplishing a t r ← r + 0 . 02 + 1 / 2 · min(max( p, 0 . 0001) − 2 ) , 100) /
else if not success t then
r ← r −
if
there was a successful PlaceAt action taken in the episode and a t is the last successful action that is not a R EADY F OR S EEKER action then
[Figure]
return r Input: Time step t ≥ 0 , full history of OM episode H , number of steps taken in episode N , the target location speciﬁed by the action PlaceAt|m,i,j recall Table G.
Output: Reward for OM episode step at time t .
1 begin hand locations =
empty list for k ← 1, ..., N do a k ← k th action taken in episode success k ← was k th action successful ( x, y, z ) ← world coordinates of agent’s hand at time k rounded to 2 dec.
places
Append ( x, y, z ) to the end of hand locations r ← − 0 .
if not success t then r ← r − 0 .
else if a t is an OpenAt action then // Encourage the agent to attempt opening objects
if the opened object was not previously opened in the episode then
r ← r + 0 .
else if a t =
DropObject then
n obj pixels ← total number of pixels corresponding to the object after making all other objects invisible
if
n obj pixels ≤ 10 then //
Discourage the agent from placing objects off screen r ← r − 1 .
else hit ← 3 × 7 × 7 tensor recording the number of pixels of the object dropped object in each of the grid locations and modalities, recall the discussion in Sec.
B.4 and
Fig.
3d
M ← Compute M as in described in Sec.
B.
hit inds ← list of the multi-indices of hit where the corresponding entry of hit is ≥ 0 .
95 · M // Encourage the agent to place the object close to the goal in pixel space (ignoring modality) r ← r + 0 .
25 min {| i − b | + | j − c | : ( a,b,c ) ∈ hit inds } //
Encourage the agent to place the object in the correct modality (only for B EHIND and C ONTAINED I N modalities)
if m = 0 and a = m for any ( a, b, c ) ∈ hit inds then if
is
hit inds then
←
else if t =
N then
r ← r − 1 .
return r successful location t ← action a t resulted in the entering a previously unvisited location tuple ignoring rotation new opened t ← action a t resulted in the agent opening a previously unopened object r ← −
if not success t then if a t = C LAIM V ISIBLE then // penalize the agent for claiming the object is visible when it cannot be seen if the action failed as the object was not visible in the viewport and not because the agent was too far away from the visible object then r ← r − 0 .
else r ← r − 0 .
else if new location t then // don’t use a step penalty if the seeking agent entered a new location
r ← r + 0 .
else if new opened t then r ← r + 0 .
else if a t = C LAIM V ISIBLE then // the agent has successfully found the object r ← r + 1 .
return r
[Table]
Agent must be holding an object, if so the agent applies a constant force on the object in the desired direction until it reaches a max velocity of 1m/s.
The agent stops the object if it either moves a total of 0.1m from its starting position, the object reaches a max distance (1.5m) from the agent, the object exits the agent’s viewport, or four seconds have elapsed.
Once the agent stops the object, if the object has not moved more 0.001m or rotated at least 0.001 ◦ then the movement is considered failed and the ob- ject is reset to its starting position.
Otherwise hand movement was successful.
The hand object re-
mains in its new position.
OM
RotateHand|D , D ∈ { -X, +X, -Y, +Y, -Z, +Z }
Agent must be holding an object, if so the agent rotates the object by 30 degrees along the axis speciﬁed by D .
So if D = -Y then the object is rotated 30 degrees counter- clockwise along the y-axis.
If this rotation would result in the object colliding with an- other object then the action fails and the object is reset to its starting rotation.
The hand object re-
mains in
its
[Table]
orientation.
OM
[Table]
The agent is
[Table]
an
[Table]
The
[Table]
is
[Table]
OM
[Table]
Recall the partitioning of an image into a 7 × 7 grid as shown in Fig.
1d.
There is an openable object near the center of the i, j th grid location
[Table]
The object near the location speciﬁed [Table] i, j
[Table]
E&M,OH,
OM,S
[Table]
E&M,OH, [Table] respond
[Table]
OH
[Table]
OH
[Table]
S