O BJECT -O RIENTED
M ODEL
L EARNING
THROUGH
M ULTI -L EVEL
A BSTRACTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Object-based approaches for learning action-conditioned dynamics has demonstrated promise for generalization and interpretability.
However, existing approaches suffer from structural limitations and optimization difﬁculties for common environments with multiple dynamic objects.
In this paper, we present a novel self-supervised learning framework, called Multi-level Abstraction Objectoriented Predictor (MAOP), for learning object-based dynamics models from raw visual observations.
MAOP employs a three-level learning architecture that enables efﬁcient dynamics learning for complex environments with a dynamic background.
We also design a spatial-temporal relational reasoning mechanism to support instance-level dynamics learning and handle partial observability.
Empirical results show that MAOP signiﬁcantly outperforms previous methods in terms of sample efﬁciency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different static object layouts.
In addition, MAOP learns semantically and visually interpretable disentangled representations.
I NTRODUCTION
Model-based deep reinforcement learning (DRL) has recently attracted much attention for improving sample efﬁciency of DRL, such as (Heess et al., 2015; Schmidhuber, 2015;
Gu et al., 2016;
Racani ` ere et al., 2017; Finn & Levine, 2017).
One of the core problems for model-based reinforcement learning is to learn action-conditioned dynamics models through interacting with environments.
Pixel-based approaches have been proposed for such dynamics learning from raw visual perception, achieving remarkable performance in training environments (Oh et al., 2015; Watter et al., 2015; Chiappa et al., 2017).
To unlock sample efﬁciency of model-based DRL, learning action-conditioned dynamics models that generalize over unseen environments is critical yet challenging.
Finn et al.
(2016) proposed an action-conditioned video prediction method that explicitly models pixel motion and thus is partially invariant to object appearances.
Zhu et al.
(2018) developed an object-oriented dynamics predictor, taking a further step towards generalization over unseen environments with different object layouts.
However, due to structural limitations and optimization difﬁculties, these methods do not learn and generalize well for common environments with a dynamic background, which contain multiple moving objects in addition to controllable objects.
To address these limitations, this paper presents a novel self-supervised, object-oriented dynamics learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP).
This framework simultaneously learns disentangled object representations and predicts object motions conditioned on their historical states, their interactions to other objects, and an agent’s actions.
To reduce the complexity of such concurrent learning and improve sample efﬁciency, MAOP employs a threelevel learning architecture from the most abstract level of motion detection, to dynamic instance segmentation, and to dynamics learning and prediction.
A more abstract learning level solves an easier problem and has lower learning complexity, and its output provides a coarse-grained guidance for the less abstract learning level, improving its speed and quality of learning convergence.
This multi-level architecture is inspired by humans’ multi-level motion perception from cognitive science studies (Johansson, 1975; Lu & Sperling, 1995; Smith et al., 1998) and multi-level abstraction search in constraint optimization (Zhang & Shah, 2016).
In addition, we design a novel CNN-based spatial-temporal relational reasoning mechanism, which includes a Relation Net to reason about spatial relations between objects and an Inertia Net to learn temporal effects.
This mechanism offers a disentangled way to handle physical reasoning in the setting with partial observability.
Empirical results show that MAOP signiﬁcantly outperforms previous methods in terms of sample efﬁciency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different object layouts.
It can learn from few examples and accurately predict the dynamics of objects as well as raw visual observations in previously unseen environments.
In addition, MAOP learns disentangled representations and gains semantically and visually interpretable knowledge, including meaningful object masks, accurate object motions, disentangled reasoning process, and the discovery of the controllable agent.
R ELATED W ORK
Object-oriented reinforcement learning has received much research attention, which exploits efﬁcient representations based on objects and their interactions.
This learning paradigm is close to that of human cognition in the physical world and the learned object-level knowledge can be robustly generalized across environments.
Early work on object-oriented RL requires explicit encodings of object representations, such as relational MDPs (Guestrin et al., 2003), OO-MDPs (Diuk et al., 2008), object focused q-learning (Cobo et al., 2013), and Schema Networks (Kansky et al., 2017).
In this paper, we present an end-to-end, self-supervised neural network framework that automatically learns object representations and dynamics conditioned on actions and object relations from raw visual observations.
Action-conditioned dynamics learning aims to address one of the core problems for model-based
DRL, that is, constructing an environment dynamics model.
Several pixel-based approaches have been proposed for learning how an environment changes in response to actions through unsupervised video prediction and achieve remarkable performance in training environments (Oh et al., 2015; Watter et al., 2015; Chiappa et al., 2017).
Fragkiadaki et al.
(2016) propose an object-centric prediction method to learn the dynamics model when given the object localization and tracking.
Finn et al.
(2016) develop a dynamics prediction method that explicitly models pixel motions and is partially invariant to object appearances, and its usage for model-based DRL is demonstrated with model predictive controller (Finn & Levine, 2017).
Recently, Zhu et al.
(2018) propose an object-oriented dynamics learning paradigm that enables its learned model to generalize over unseen environments with different object layouts and be robust to changes of object appearances.
However, this paradigm focuses environments with a single dynamic object.
In this paper, we take a further step towards learning object-oriented dynamics model in more general environments with multiple controlled and uncontrollable dynamic objects.
In addition, we design an instance-aware dynamics mechanism to support instance-level dynamics learning and handle partial observations.
Relation-based deep learning approaches make signiﬁcant progress in a wide range of domains such as physical reasoning (Chang et al., 2016;
Battaglia et al., 2016), computer vision (Watters et al., 2017; Wu et al., 2017), natural language processing (Santoro et al., 2017), and reinforcement learning (Zambaldi et al., 2018; Zhu et al., 2018).
Relation-based nets introduce relational inductive biases into neural networks, which facilitate generalization over entities and relations and enables relational reasoning (Battaglia et al., 2018).
OODP (Zhu et al., 2018) is similar to this paper, learning object-level dynamics conditioned on actions and object-to-object relations.
In contrast to OODP, this paper proposes a novel spatial-temporal relational reasoning mechanism, which includes an
Inertia Net for learning temporal effects in addition to a CNN-based Relation Net for reasoning about spatial relations.
This mechanism offers a disentangled way to handle physical reasoning in the setting of partial observability.
Instance Segmentation has been one of the fundamental problems in computer vision.
Instance segmentation can be regarded as the combination of semantic segmentation and object localization.
Many approaches have been proposed for instance segmentation, including DeepMask (Pinheiro et al., 2015), InstanceFCN (Dai et al., 2016), FCIS (Li et al., 2017), and Mask R-CNN (He et al., 2017).
Most models are supervised learning and require a large labeled training dataset.
Liu et al.
(2015) proposes a weakly-supervised approach to infer object instances in foreground by exploiting dynamic consistency in video.
In this paper, we design a self-supervised, three-level approach for learning dynamic rigid object instances.
At the most abstract level, the foreground detection module provides a coarse-grained guidance for producing region proposals at the instance segmentation level.
The instance segmentation level then learns coarse instance segmentation.
This coarse instance segmentation provides a guidance for learning accurate instances at the dynamics learning level, whose instance segmentation considers not only object appearances but also motion prediction conditioned on object-to-object relations and actions.
M ULTI - LEVEL A BSTRACTION O BJECT - ORIENTED P REDICTOR (MAOP)
In this section, we will present a novel self-supervised deep learning framework, aiming to learn object-oriented dynamics models that are able to generalize over unseen environments with different object layouts and multiple dynamic objects.
Such an object-oriented dynamics learning approach requires simultaneously learning object representations and motions conditioned on their historical states, their interactions to other objects, and an agent’s actions.
This concurrent learning is very challenging for an end-to-end approach in complex environments with a dynamic background.
Evidences from cognitive science studies (Johansson, 1975; Lu & Sperling, 1995; Smith et al., 1998) show that human beings are born with prior motion perception ability (in Cortical area MT) of perceiving moving and motionlessness, which enables learning more complex knowledge, such as object-level dynamics prediction.
Inspired by these studies, we design a multi-level learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP), which incorporates motion perception levels to assist in dynamics learning.
[Figure]
down decomposition view.
First, we perform motion detection to produce foreground masks.
Then, we utilize the foreground masks as dynamic region proposals to guide the learning of dynamic instance segmentation.
Finally, we use the learned dynamic instance segmentation networks (including Instance Splitter and Merging Net) as a guiding network to generate region proposals of dynamic instances and guide the learning of Object Detector in the level of dynamics learning.
Figure 1 illustrates three levels of MAOP framework: dynamics learning, dynamic instance segmentation, and motion detection.
The dynamics learning level is an end-to-end, self-supervised neural network, aiming to learn object representations and instance-level dynamics, and predict the next visual observation conditioned on an agent’s action.
To guide the learning of the object representations and instance localization at the level of dynamics learning, the more abstracted level of dynamic instance segmentation learns a guiding network in a self-supervised manner, which can provide coarse dynamic instance mask proposals.
It exploits the spatial-temporal information of locomotion property and appearance pattern to capture the region proposals of dynamic instances.
To facilitate the learning of dynamic instance segmentation, MAOP employs the more coarse-grained level of motion detection, which detects changes in image sequences and provides guidance on proposing regions potentially containing dynamic instance.
As the learning proceeds, the knowledge distilled from the more coarse-grained level are gradually reﬁned at the more ﬁne-grained level by considering additional information.
When the training is ﬁnished, the coarse-grained levels of dynamic instance segmentation and motion detection will be removed at the testing stage.
In the rest of this section, we will describe in detail the design of each level and their connections.
3. O BJECT -O RIENTED D YNAMICS L EARNING L EVEL
The semantics of this level is formulated as learning an object-based dynamics model with the region proposals generated from the more abstracted level of dynamic instance segmentation.
The whole architecture is shown in the top part of Figure 1, which is an end-to-end neural network and can be trained in a self-supervised manner.
It takes a sequence of video frames and an agent’s actions as input, learns the disentangled representations (including objects, relations and effects) and the dynamics of controllable and uncontrollable dynamic object instances conditioned on the actions and object relations, and produce the predictions of raw visual observations.
The whole architecture includes four major components: A) an Object Detector that decomposes the input image into objects;
B) an Instance Localization module responsible for localizing dynamic instances; C) a Dynamics
Net for learning the motion of each dynamic instance conditioned on the effects from actions and object-level spatial-temporal relations; and D) a Background Constructor that computes the background image from the learned static object masks.
Algorithm 1 illustrates the interactions of these components and the learning paradigm of object based dynamics, which is a general framework and agnostic to the concrete form of each component.
In the following paragraphs, we will describe the detailed implementation of every components.
− h :
← InstanceLocalization
I t − h t
D t h
← ∅ 4: do 5: m
RelationNet
Effects temporal m t 2 ← InertiaNet x t h t a t ) 7:
Total effects m t ← m t 1 + m t8:
Predicted instance mask ˆ x t +1 ← Transformation ( x t , m t ) 9: ˆ X t +1 ← ˆ X t +1 (cid:83) ˆ x t + 10: end for 11: Background image B t +1 ← BackgroundConstructor
( I t
S t ) 12: Predicted next frame
ˆ I t +1 ← Merge (ˆ X t +1 , B t +1 )
Object Detector and Instance Localization Module.
Object Detector is a CNN module aiming to learn object masks from input image.
An object mask describes the spatial distribution of a class of objects, which forms the fundamental building block of our object-oriented framework.
Considering that instances of the same class are likely to have different motions, we append an Instance
Localization Module to Object Detector to localize each dynamic instance to support instance-level dynamics learning.
The class-speciﬁc object masks in conjunction with instance localization build the bridge to connect visual perception (Object Detector) with dynamics learning (Dynamics Net), which allows learning objects based on both appearances and dynamics.
Speciﬁcally, Object Detector receives image I t ∈
R H
× W × 3 at timestep t and then outputs object masks O t ∈ [0 , 1] H × W × n O , including dynamic object masks D t ∈ [0 , 1] H × W × n D and static object masks S t ∈ [0 , 1] H × W × n S , where H and W denote the height and width of the input image, n D and n S denotes the maximum class number of dynamic and static objects respectively, and n O = n D + n S .
The entry O u,v,i indicates the probability that the pixel I u,v, : belongs to the ith object class.
Then, Instance Localization Module uses the dynamic object masks to compute each single instance mask X t : , : ,i ∈ [0 ,
] H M × W M (1 ≤ i ≤ n M ) , where H M and W M denote the height and width of the bounding box of this instance and n M denotes the maximum number of localized instances.
As shown in Figure 1, Instance Localization Module ﬁrst samples a number of bounding boxes on the dynamic object masks and then select the regions, each of which contains only one dynamic instance.
As we focus on the motion of rigid objects, the afﬁne transformation is approximatively consistent for all pixels of each dynamic instance mask.
Inspired by this, we deﬁne a discrepancy loss L instance for a sampled region that measures the motion consistence of its pixels and use it as a selection score for selecting instance masks.
To compute this loss, we ﬁrst compute an average rigid transformation of a sampled region between two time steps based on the instance masks and masked image in this region, then apply this transformation to this region at the previous time step, and ﬁnally compared this predicted region with the region at the current time.
Obviously, when a sampled region contains exactly one dynamic instance, this loss will be extremely small, and even zero when the object masks are perfectly learned.
More details of the region proposal sampling and instance mask selection can be found in Appendix A.
Dynamics Net.
Dynamics Net is designed to learn instance-based motion effects of actions, objectto-object spatial relations (Relation Net) and temporal relations of spatial states (Inertia Net), and then reason about the motion of each dynamic instance based on these effects.
Its architecture is illustrated in Figure 2, which has an Effect Net for each class of objects.
An Effect Net consists of one Inertia Net and several Relation Nets.
As shown in the left subﬁgure, instance-level dynamics learning is performed, which means the motion of each dynamic instance is individually computed.
We take as an example the computation of the motion of the i -th instance
X t : , : ,i to show the detailed structure of the Effect Net.
[Figure]
A
RN C,F
IN C ...
[Figure]
As shown in the right subﬁgure of Figure 2, we ﬁrst use a sub-differentiable tailor module introduced by Zhu et al.
(2018) to enable the inference of dynamics focusing on the relations with neighbour objects.
This module crops a w -size “horizon” window from the concatenated masks of all objects
O t centered on the expected location of X t : , : ,i , where w denotes the maximum effective range of relations.
Then, the cropped object masks are respectively concatenated with the constant x-coordinate and y-coordinate meshgrid map (to make networks more sensitive to the spatial information) and fed into the corresponding Relation Nets (RN) according to their classes.
We use C t : , : ,i,j to denote the cropped mask that crops the j -th object class O t : , : ,j centered on the expected location of the i -th dynamic instance (the class it belongs to is denoted as c i , 1 ≤ c i ≤ n D ).
The effect of object class j on class c
i , E t ( c i , j
) ∈ R 2 × n a ( n a denotes the number of actions) is calculated as,
E t ( c i , j ) =
RN c i ,j (cid:16) concat (cid:0) C t : , : ,i,j , Xmap , Ymap (cid:1)(cid:17) .
Note that there are total n D × n O RNs for n D × n O pairs of object classes that share the same architecture but not their weights.
To handle the partial observation problem, we add an Inertia Nets (IN) to learn the self-effects of an object class through historical states, E t self ( c i )
=
IN c i (cid:16) concat (cid:0) X t : , : ,i , X t +1: , : ,i , . . .
, X t + h : , : ,i (cid:1)(cid:17) , where h is the history length and there are total n D INs for n D dynamic object classes that share the same architecture but not their weights.
To predict the motion vector m ti ∈ R 2 for the i -th dynamic instance, all these effects are summed up and then multiplied by the one-hot coding of action a t ∈ { 0 , 1 } n a , that is, m ti = (cid:16)(cid:0) (cid:80) j E t ( c i , j ) (cid:1)
+ E t self ( c i ) (cid:17) · a t .
Background Extractor.
This module extracts the static background of input image based on the static object masks learned by Object Detector and then it is combined with the predicted dynamic instances to predict the next visual observation.
As Object Detector can decompose its observation into objects in an unseen environment with a different object layout, Background Constructor is able to generate a corresponding static background and support the visual observation prediction in novel environments.
Speciﬁcally, Background Constructor maintains an external background memory B ∈ R H × W × 3 which is continuously updated (via moving average) by the static object mask learned by Object Detector.
Denoting α as the decay rate, the updating formula is given by,
B t = α B t − 1 + (1 − α )
I t (cid:80) i
S t : , : ,i , B 0 = 0 .
Prediction and Training Loss.
At the output end of our model, the prediction of the next frame is produced by merging the learned object motions and the background B t .
The pixels of a dynamic instance can be calculated by masking the raw image with the corresponding instance mask and we can use Spatial Transformer Network (STN) (Jaderberg et al., 2015) to apply the learned instance motion vector m ti on these pixels.
First, we transform all the dynamic instances according to the learned instance-level motions.
Then, we merge all the transformed dynamic instances with the background image calculated from Background Extractor to generate the prediction of the next frame.
In this paper, we use the pixel-wise l 2 loss to restrain image prediction error, denoted as
L prediction .
To get earlier feedback before reconstructing images and facilitate the training process, we add a highway loss, L highway =
(cid:80) i (cid:119)(cid:119) (¯ u i , ¯ v i ) t
+ m ti − (¯ u i , ¯ v i ) t +1 (cid:119)(cid:119) 2 2 , where (¯ u i , ¯ v i ) t is the excepted location of i -th instance mask X t : , : ,i .
In addition, we add a proposal loss to utilize the dynamic instance proposals provided by the abstracted problem to guide our optimization, which is given by L proposal = (cid:119)(cid:119) (cid:80) i
( D t : , : ,i − P t : , : ,i ) (cid:119)(cid:119) 2 2 , where P denotes the dynamic instance region proposals computed by the more abstract learning level (i.e., dynamic instance segmentation level).
The total loss of the dynamics learning level is given by combining the previous losses with different weights,
[Equation]
3. D YNAMIC I NSTANCE S EGMENTATION L EVEL
This level aims to generate region proposals of dynamic instances to guide the learning of object masks and facilitate instance localization at the level of dynamics learning.
The architecture is shown in Figure 1.
Instance Splitter aims to identify regions, each of which potentially contains one dynamic instance.
To learn to divide different dynamic object instances onto different masks, we use the discrepancy loss L instance described in Section 3.1 to train Instance Splitter.
Considering that one object instance may be split into smaller patches on different masks, we append a Merging Net (i.e., a two-layer CNN with 1 kernel size and 1 stride) to Instance Splitter to learn to merge redundant masks by a merging loss L merge based on the prior that the patches of the same instance are adjacent to each other and share the same motion.
In addition, we add a foreground proposal loss L forground to encourage attentions on the dynamic regions.
The total loss of this level is given by combining these losses with different weights,
[Equation]
For more complex domains with arbitrary deformation and appearance change, MAOP is also readily to incorporate the vanilla Mask R-CNN (He et al., 2017) or other off-the-shelf supervised object detection methods (Liu et al., 2018) as a plug-and-play module into our framework to generate region proposals of dynamic instances.
In addition, although the network structure of this level is similar to Object Detector in the level of dynamics learning, we do not integrated them together as a whole network because the concurrent learning of both object representations and dynamics model is not stable.
Instead, we ﬁrst learn the coarse object instances only based on the spatial-temporal consistency of locomotion and appearance pattern, and then use them as proposal regions to perform object-oriented dynamics learning at the more ﬁne-grained level, which in turn ﬁne-tunes the object representations.
[Figure]
3. M OTION D ETECTION L EVEL
At this level, we employ foreground detection to detect changing regions from a sequence of image frames and provide coarse dynamic region proposals F p for assisting in dynamic instance segmentation.
In our experiments, we use a simple unsupervised foreground detection approach proposed by
Lo & Velastin (2001).
Our framework is also compatible with many advanced unsupervised foreground detection methods (Lee, 2005; Maddalena et al., 2008;
Zhou et al., 2013; Guo et al., 2014) that are more efﬁcient or more robust to moving camera.
These complex unsupervised foreground detection methods have the potential to improve the performance but are not the focus of this work.
E XPERIMENTS
We evaluate our model on two games, Monster Kong and Flappy Bird , from the Pygame Learning Environment (Tasﬁ, 2016), which allows us to test generalization ability over various scenes with different layouts.
Here, the Monster Kong is the advanced version of that used by Zhu et al.
(2018), which has a more general and complex setting.
The monster wanders around and breathes out ﬁres randomly, and the ﬁres also move with some randomness.
The agent explores with actions up , down , left , right , jump and noop .
All these dynamic objects interact with the environments and objects according to their own physics engine.
Moreover, gravity and jump model has a long-term dynamics effects, leading to a partial observation problem.
To test whether our model can truly learn the underlying physical mechanism behind the visual observations and perform relational reasoning, we set the k -to- m zero-shot generalization experiment, where we use k different environments for training and m different unseen environments for testing.
Flappy Bird is a side-scroller game, where a bird ﬂies between columns of green pipes with action jump and noop .
Since the unseen environments will be similar with the training ones without limitation of samples in this game, we limit the samples for training.
Two experimental settings are shown in Figure 3.
We use random exploration on Monster Kong , and an expert guided random exploration on Flappy Bird because in this domain a totally random exploration will lead to an early death of the agent even at the very beginning.
We compare MAOP with state-of-the-art action-conditioned dynamics learning baselines, AC Model
(Oh et al., 2015), CDNA (Finn et al., 2016), and OODP (Zhu et al., 2018).
AC Model adopts an encoder-LSTM-decoder structure, which performs transformations in hidden space and constructs pixel predictions.
CDNA explicitly models pixel motions to achieve invariance to appearance.
OODP and MAOP both aim at learning object-level dynamics through an object-oriented learning paradigm, which decomposes raw images into objects and perform prediction based on object-level relations.
OODP is only designed for class-level dynamics, while MAOP is able to learn instancelevel dynamics.
Implementation details of MAOP can be found in Appendix C.
4. Z ERO - SHOT G ENERALIZATION AND SAMPLE EFFICIENCY
To make a sufﬁcient comparison with the previous methods on the generalization ability and sample efﬁciency of object dynamics learning and image prediction, we conduct 1-5, 2-5 and 3-5 generalization experiments with a variety of evaluation indices on Monster Kong .
We use n -error accuracy to measure the performance of object dynamics prediction, which is deﬁned as the proportion that the difference between the predicted and ground-true agent locations is less than n pixel.
We also add an extra pixel-based measurement (denoted by object RMSE), which compares the pixel difference near dynamic objects between the predicted and ground-truth images.
To evaluate the image prediction, we adopt a typical image prediction loss (pixel-wise RMSE).
In Figure 4 and
B7 (Appendix B), we plot the learning curve for better visualization of the comparison in unseen environments.
Further, we add a video ( https://github.com/maop2018/maop-video/ blob/master/video.avi ) for better perceptual understanding of the prediction performance in unseen environments.
As shown in Table 1, MAOP signiﬁcantly outperforms other methods in all experiment settings in terms of generalization ability and sample efﬁciency of both object dynamics learning and image prediction.
It can achieve 0 .
84 0-error accuracy, even with a single training environment, which suggests MAOP is good at relational reasoning.
Although AC Model achieves high accuracy in training environments, its performance in unseen scenes is much worse, which is probably because its pure pixel-level inference easily leads to overﬁtting.
CDNA performs better than AC Model in those uncontrolled objects, but still cannot deal with complicated interactions in lack of knowledge on object-to-object relations.
By the structural limitation of OODP, it has innate difﬁculty on frames with multiple dynamic objects.
In addition, we evaluate our performance of long-range prediction.
As shown in Table B4, our model only trained for 1-step prediction can achieve better performance for action-conditioned dynamics prediction than AC Model and CDNA trained for 3-step prediction.
Speciﬁcally, our model has 88 percent probability that the predicted and ground-true agent locations are below 2 pixel when predicting 3 steps of the future, while the probability is 0.62 when predicting 8 steps of the future, which is also a good performance.
We also test our model on Flappy Bird , where we limit the training samples to 100 and 300 to form a sufﬁciently challenging generalization task.
As shown in Table 2, our performance is similar with that on Monster Kong .
Our generalization ability and sample efﬁciency signiﬁcantly outperform other baselines.
Surprisingly, only 100 samples are enough to reach almost perfect 1-error accuracy
[Table]
4. I NTERPRETABLE REPRESENTATIONS AND KNOWLEDGE
MAOP takes a step towards interpretable deep learning and disentangled representation learning.
Through interacting with environments, it learns fruitful visually and semantically interpretable knowledge in an unsupervised manner, which contributes to unlock the “black box” of neural networks and open the avenue for further researches on object-based planning, object-oriented modelbased RL
[Table]
and hierarchical learning.
[Table] [Table]
‡ indicates training with only 100 and 300 samples.
Visual interpretability.
To demonstrate the visual interpretability of MAOP in unseen environments, we visualize the learned masks of dynamic and static objects.
We highlight the attentions of the object masks by multiplying the raw images by the binarized masks.
Note that MAOP does not require the actual number of objects but a maximum number and some learned object masks may be redundant.
Thus, we only show the informative object masks.
As shown in Figure 6, our model captures all the key objects in the environments including the controllable agents (the cowboy and the bird), the uncontrollable dynamic objects (the monster, ﬁres and pipes), and the static objects that have effects on the motions of dynamic objects (ladders, walls and the free space).
We also observe that model can learn disentangled object representations and distinguish the objects by both appearance and dynamic property.
Discovery of the controllable agent.
With the learned knowledge in MAOP, we can easily uncover the action-controlled agent from all the dynamic objects, which is useful semantic information that can be used in heuristic algorithms.
Speciﬁcally, the object that has the maximal variance of total effects over actions is the action-controlled agent.
Denote the total effects as E i =
( (cid:80) j E ( c i , j ))
+
E self ( c i ) , E i ∈
R 2 × n a , the label of the action-controlled agent is calculated as, arg max i V ar ( E i ) .
The histogram in Figure 5 plotting the ground-truth label distribution of our discovered actioncontrolled agents clearly demonstrates that our discovery of the controllable agent achieves perfect 100% accuracy.
Dynamical interpretability.
To show the dynamical interpretability behind image prediction, we test our predicted motions by comparing RMSEs between the predicted and ground-truth motions [Figure] in unseen environments (Table B3 in Appendix B).
Intriguingly, most predicted motions are quite accurate, with the RMSEs less than 1 pixel.
Such a visually indistinguishable error also veriﬁes our dynamics learning.
C ONCLUSIONS AND F UTURE W ORK
This paper presents a self-supervised multi-level learning framework for learning action-conditioned object-based dynamics.
This framework is sample-efﬁcient and generalizes object dynamics and prediction of raw visual observations to complex unseen environments with multiple dynamic objects.
The learned dynamics model potentially enables an agent to directly plan or efﬁciently learn for unseen environments.
Although a random policy or an expert’s policy is used for exploration in our experiments, our framework can support smarter exploration strategies, e.g., curiosity-driven exploration.
Our future work includes extending our model for deformation prediction (e.g., object appearing, disappearing and non-rigid deformation) and incorporating a camera motion prediction network module introduced by Vijayanarasimhan et al.
(2017) for applications such as FPS games and autonomous driving.
A PPENDIX A I NSTANCE L OCALIZATION
Instance localization is a common technique in context of supervised region-based object detection (Girshick et al., 2014; Girshick, 2015; Ren et al., 2015;
He et al., 2017; Liu et al., 2018), which localizes objects on raw images with regression between the predicted bounding box and the ground truth.
Here, we propose an unsupervised approach to perform dynamic instance localization on dynamic object masks learned by Object Detector.
Our objective is to sample a number of region proposals on the dynamic object masks and then select the regions, each of which has exactly one dynamic instance.
In the rest of this section, we will describe these two steps in details.
Region proposal sampling.
We design a learning-free sampling algorithm for sampling region proposals on object masks.
This algorithm generates multi-scale region proposals with a full coverage over the input mask.
Actually, we adopt multi-fold full coverage to ensure that pixels of the potential instances are covered at each scale.
The detailed algorithm is described in Algorithm 2.
Instance mask selection.
Instance mask selection aims at selecting the regions, each of which contains exactly one dynamic instance, based on the discrepancy loss L instance (Section 3.1).
To screen out high-consistency, non-overlapping and non-empty instance masks at the same time, we integrate Non-Maximum Suppression (NMS) and Selective Search (SS) (Uijlings et al., 2013) in the context of region-based object detection (Girshick et al., 2014; Girshick, 2015; Ren et al., 2015;
He et al., 2017; Liu et al., 2018) into our algorithm.
Input:
full coverage
do
do 6:
Initialize candidate set C
= { ( i j ) |
D
i j
7: while C (cid:54) = do 8:
Sample a pixel coordinate ( from
Get a box B
= { ( i i x dx y dy
10: if B is not empty then 11:
Insert B into the proposal set P ← P
{ B } .
12: end if 13:
Update the remain candidate set C
C
14: end while 15: end for 16: end for 17: return P
A PPENDIX B T ABLES AND F IGURES
[Table]
[Table]
MAOP and OODP are trained for 1-step prediction, while AC Model and CNDA are trained for 3-step prediction.
They all are tested fro 3-step and 8-step prediction.
A PPENDIX C I MPLEMENTATION DETAILS FOR EXPERIMENTS
Object Detector in the dynamics learning level and Instance Splitter in the dynamic instance segmentation level have similar architectures with Object Detector in OODP (Zhu et al., 2018).
To augment the interactions of instances when training Instance Splitter, we random sample two region proposals and combine them into a single region proposal with double size.
Denote Conv ( F, K, S ) as the convolutional layer with the number of ﬁlters F , kernel size K and stride S .
Let R () , S () and BN () denote the ReLU layer, sigmoid layer and batch normalization layer (Ioffe & Szegedy, 2015).
The 5 convolutional layers in Object Detector can be indicated as R ( BN ( Conv (16 , 5 , 2))) , R ( BN ( Conv (32 , 3 , 2))) , R ( BN ( Conv (64 , 3 , 1))) ,
R ( BN ( Conv (32 , 1 , 1))) , and BN ( Conv (1 , 3 , 1)) , respectively.
The 5 convolutional layers in Instance Detector can be indicated as R ( BN ( Conv (32 , 5 , 2))) , R ( BN ( Conv (32 , 3 , 2))) ,
R ( BN ( Conv (32 , 3 , 1))) ,
R ( BN ( Conv (32 , 1 , 1))) , and BN ( Conv (1 , 3 , 1)) , respectively.
The architecture of Foreground Detector is similar to binary-class Object Detector and the 5 convolutional layers in Foreground Detector can be indicated as R ( BN ( Conv (32 , 5 , 2))) ,
R ( BN ( Conv (32 , 3 , 2))) ,
R ( BN ( Conv (32 , 3 , 1))) ,
R ( BN ( Conv (32 , 1 , 1))) , and
S ( BN ( Conv (1 , 3 , 1))) , respectively.
The CNNs in Relation Net are connected in the order:
R ( BN ( Conv (16 , 3 , 2))) ,
R ( BN ( Conv (32 , 3 , 2))) , R ( BN ( Conv (32 , 3 , 2))) , and
R ( BN ( Conv (32 , 3 , 2))) .
The last convolutional layer is reshaped and fully connected by the 64-dimensional hidden layer and the 2-dimensional output layer successively.
Inertia Net has the same architecture and hyperparameters as Relation Net.
The hyperparameters for training MAOP in Monster Kong and Flappy Bird are listed as follows:
• The weights of losses, λ 1 , λ 2 , and λ 3 , are 100, 1, and 10, respectively.
In addition, all the l
losses are divided by HW to keep invariance to the image size.
• Batch size is 16 and the maximum number of training steps is set to be 1 × 10 5 .
• The optimizer is Adam (Kingma & Ba, 2014) with learning rate 1 × 10 − 3 .
• The raw images of Monster Kong and Flappy Bird are resized to 160 × 160 × 3 and 160 × 80 × 3 , respectively.
• Overlapping threshold α is 0.5.
• The size of the horizon window w is 33 on Monster Kong , 41 on Flappy Bird .
• The maximum number of static and dynamic masks is 4 and 12 on Monster Kong , on Flappy Bird .
• The maximum instance number of each object class is set to be 15.