Object-Category Aware Reinforcement Learning
Abstract
Object-oriented reinforcement learning (OORL) is a promising way to improve the sample efficiency and generalization ability over standard RL.
Recent works that try to solve OORL tasks without additional feature engineering mainly focus on learning the object representations and then solving tasks via reasoning based on these object representations.
However, none of these works tries to explicitly model the inherent similarity between different object instances of the same category.
Objects of the same category should share similar functionalities; therefore, the category is the most critical property of an object.
Following this insight, we propose a novel framework named Object-Category Aware Reinforcement Learning (OCARL), which utilizes the category information of objects to facilitate both perception and reasoning.
OCARL consists of three parts: (1) Category-Aware
Unsupervised Object Discovery (UOD), which discovers the objects as well as their corresponding categories; (2) Object-Category
Aware Perception, which encodes the category information and is also robust to the incompleteness of (1) at the same time; (3) Object-Centric Modular Reasoning, which adopts multiple independent and object-category-specific networks when reasoning based on objects.
Our experiments show that OCARL can improve both the sample efficiency and generalization in the OORL domain.
Introduction
Reinforcement Learning (RL) has achieved impressive progress in recent years, such as results in
Atari [26] and Go [30] in which RL agents even perform better than human beings.
Despite its successes, RL is also known to be of low sample efficiency [11] and poor generalization ability [4].
Object-oriented RL (OORL)
[13; 9; 25; 18; 41] is a promising way to deal with these limitations.
Inspired by studies in cognitive science [27] that objects are the basic units of recognizing the world,
OORL focuses on learning the invariance of objects’ functionalities in different scenarios to achieve better generalization ability.
In OORL, the agent’s observation is a set of object representations, and the task can be solved via reasoning based on these objects.
Most previous works [13; 18; 9] in OORL employ hand-crafted object features given in advance, which require human expertise and result in low generality, limiting the applications of these methods.
Recent works [41; 40; 39] try to broaden the scope of OORL’s applications by avoiding using additional feature engineering.
Their works can be divided into either top-down approaches [41] which purely rely on reward signals to ground representations to objects, or bottom-up approaches [40; 39] which utilize unsupervised object discovery (UOD) technologies to provide structural object representations.
However, none of these works [41; 40; 39] tries to explicitly model the inherent similarity between different object instances of the same category when reasoning based on objects.
In most cases, objects of the same category should share the same functionalities, such as obeying similar behaviour patterns or having the same effects when interacting with other objects.
Therefore, an intelligent agent should recognize an object’s category to be aware of its functionalities.
Since objects from different categories differ in their functionalities, the objects’ functionalities can also be separately modelled (according to their categories), leading to modularity which is beneficial for generalization [6; 8; 38] in general.
Following the above insight, we propose a framework named O bject- C ategory
A ware R einforcement
L earning (OCARL), which utilizes the category information of objects to facilitate both perception and reasoning.
OCARL consists of three parts: (1) category-aware UOD, which can automatically discover the objects as well as their corresponding categories via UOD and unsupervised clustering.
O bject-
C ategory
A ware P erception (OCAP), a perception module that takes the object category information obtained in (1) as an additional supervision signal.
OCAP can encode the category information into representations and is also robust to the incompleteness of UOD at the same time.
O bject- C entric M odular R easoning (OCMR), a reasoning module that adopts multiple independent and object-category-specific networks, each of which processes the object features of the same corresponding category.
Such a modular mechanism can enhance further the generalization ability of
OCARL.
Our experiments show that OCARL can improve the sample efficiency on several tasks and also enable the agent to generalize to unseen combinations of objects where other baselines fail.
Related Works
Unsupervised Object Discovery Unsupervised object discovery (UOD) methods try to automatically discover objects without additional supervision.
Roughly speaking, there are two main categories in this area: spatial attention models and spatial mixture models.
Spatial attention models such as SCALOR [17] and TBA [16] explicitly factorize the scene into a set of object properties such as position, scale and presence.
Then these properties are utilized by a spatial transformer network to select small patches in the original image, which constitute a set of object proposals.
These methods can deal with a flexible number of objects and thus are prominent in the UOD tasks.
Spacial mixture models such as Slot Attention [23], IODINE [12] and MONet [3] decompose scenes via clustering pixels that belong to the same object (often in an iterative manner).
However, they often assume a fixed maximum number of objects and thus cannot deal with a large number of objects.
SPACE [21], a UOD method used in this paper, combines both the spatial attention model and spatial mixture model.
The spatial attention model extracts objects from the scene, whereas the spacial mixture model is responsible for decomposing the remaining background.
Such a combination enables SPACE to distinguish salient objects from relative complex backgrounds.
Object-Oriented Reinforcement Learning Object-oriented reinforcement learning (OORL) has been widely studied in RL community.
There are many different structural assumptions on the underlying MDP in OORL, such as relational MDP [13], propositional object-oriented MDPs [9], and deictic object-oriented MDP [25].
Although these assumptions differ in detail, they all share a common spirit that the state-space of MDPs can be represented in terms of objects.
Following these assumptions, [5] proposes object-focused Q-learning to improve sample efficiency.
Schema Networks [18] learn an object-based casual model for regression planning.
[33] improve the generalization ability to novel objects by leveraging relation types between objects that are given in advance.
Although these works have been demonstrated to be useful in their corresponding domain, they require explicit encodings of object representations, which limits their applications.
There are other lines of work in OORL such as COBRA [35], OODP [42], and OP3 [32] that try to solve OORL tasks in an end-to-end fashion.
Most of these works fall into a model-based paradigm in which an object-centric model is trained and then utilized to plan.
Although these methods are effective in their corresponding domains, in this paper, we restrict ourselves to the model-free setting, which can generally achieve better asymptotic performance than model-based methods [34; 14].
The most related works should be RRL [41] and SMORL [40].
RRL proposes an attention-based neural network to introduce a relational inductive bias into RL agents.
However, since RRL relies purely on this bias, it may fail to capture the objects from the observations.
On the other hand,
SMORL utilizes an advanced UOD method to extract object representations from raw images, and then these representations are directly used as the whole observations.
However, SMORL adopts a relatively simple reasoning module, which limits its applications in multi-object scenarios.
Method
OCARL consists of three parts: (1) category-aware unsupervised object discovery (category-aware
UOD) module, (2) object-category aware perception (OCAP), and (3) object-centric modular reasoning (OCMR) module.
The overall architecture is shown in Figure 1.
We will introduce (1)(2)(3) in
Section 3.1, Section 3.2, and Section 3.3 respectively.
3.1 Category-Aware Unsupervised Object Discovery
Unsupervised Object Discovery
In this work, SPACE [21] is utilized to discover objects from raw images.
In SPACE, an image x is decomposed into two latent representations: foreground z fg and background z
bg .
For simplicity, we only introduce the inference model of the foreground latent z fg , which is a set of object representations.
We encourage the readers to refer to [21] for more details.
To obtain z fg , x is treated as if it were divided into H × W cells and each cell is tasked with modelling at most one (nearby) object.
Therefore, z fg consists of a set of H × W object representations (i.e. z fg = { z fgij } Hi =
Wj =1 ), each of which is a 3-tuple 1 z fgij =
( z presij , z whereij , z whatij ) .
z presij ∈ { 0 , 1 } is a 1-D variable that indicates the presence of any object in cell ( i, j ) , z whereij encodes z fgij ’s size and location, and z whatij is a latent vector that identifies z fgij itself.
The whole inference model of z fg can be written as:
[Equation]
where x ij is a small patch from the x that is obtained by the proposal bounding box identified by z whereij , which should contain exactly one object if z presij = 1 .
Unsupervised Clustering After applying SPACE on the randomly collected dataset, we can obtain a set of object representations:
D = { z presnij , z wherenij , z whatnij , x nij } Nn =1 Hi =
Wj =1 , where N is the size of the dataset, z presnij , z wherenij , z whatnij , x nij are defined in Eq.(1).
We first select object patches from
D that is of high object-presence probability by a threshold τ : D τ = { x nij ∈ D : p ( z presnij = 1) > τ } .
Roughly speaking, each patch x nij ∈ D τ should contain exactly one object.
Therefore, we first project x nij ∈ D τ into a low-dimensional latent space R d by running IncrementalPCA [28] on D τ , and then adopt KMeans clustering with a given cluster number C upon the projected latent representations.
The object-category predictor q ( z catij | x ij ) =
Ψ KMeans ◦ Ψ IncrementalPCA is the composition of IncrementalPCA and KMeans:
[Equation]
where z catij ∈ R C is an one-hot latent which indicates the category of z ij .
For simplicity, we regard the background (i.e. z presij = 0 ) as special ’object’, thus z catij can be written as ˆz catij =
Ψ concatenate ([1 − z
presij
, z presij · z catij ]) .
Note that ˆ z catij is an one-hot latent ∈ R C +1 .
Although the object-category predictor Eq.(2) is very simple, we find it works well in our experiments.
In practice, the cluster number C of KMeans can be set by leveraging prior knowledge about groundtruth category number, or by analysing some clustering quality metrics such as silhouette coefficients.
In more complex environments, we can also rely on other advanced unsupervised clustering methods such as NVISA [20], CC [19], etc.
3.2 OCAP: Object-Category Aware Perception
The OCAP module is designed to robustly incorporate the object knowledge of the (category-aware)
UOD model defined in Section 3.1.
In a word, OCAP is a plain convolution encoder that accepts the category information from the UOD model as additional supervision signals.
Such a design makes
OCARL robust to the incompleteness of the UOD model: in extreme cases in which the UOD model fails to discover any objects, OCAP degenerates into a plain convolution encoder which is still able to extract useful information from the raw images with the help of reward signals.
Suppose the convolution encoder in OCAP maps the raw image observation X ∈ R 3 × H img × W img to a latent representation Z ∈ R F × H × W , where H, W are the same with those in the UOD model (see
Eq.(1)).
On the other hand, X is also fed into the UOD model to get the (extended) object category information ˆ z cat =
{ ˆ z catij } Hi =1 Wj =1 via Eq.(2).
The OCAP module forces the latent representation Z to encode the object category information, which is implemented by training a additional category predictor f cat : R F → Categorical ( C +1) .
f cat predicts ˆ z cat ij given Z : ,i,j which is a single channel in Z .
Therefore, the additional supervision signal that incorporates the object knowledge of the UOD model is given as:
[Equation]
j =
CrossEntropyLoss ( f cat ( Z : ,i,j ); ˆ z catij ) .
L cat is used for training the convolution encoder in OCAP.
In practice, L cat can be used as an auxiliary loss to any RL algorithm:
[Equation]
where λ cat ∈ R + is a coefficient.
3.3 OCMR: Object-Centric Modular Reasoning
OCMR is a module that takes Z ∈ R F × H × W from the OCAP module as input and outputs a feature vector that summarizes Z .
The key design philosophy of OCMR is to adopt multiple independent and object-category-specific networks, each of which focuses on processing the object features of the same corresponding category.
Compared with using a universal category-agnostic network, the processing logic of each independent network in OCMR is much simpler and therefore easier to master, which in turn allows for improved generalization as we will show in Section 4.3.2.
This design philosophy also agrees with the recent discovery from [24; 6] which says that neural modules of specialization can lead to better generalization ability.
Given Z ∈ R F × H × W from OCAP, we first encode the x-y coordinates information (corresponding to the H × W grid) to each channel, and then map the resulting tensor into (cid:98) Z ∈ R HW × F .
(cid:98) Z is treated as HW objects with x-y coordinate information encoded.
To model the relations between objects, we apply a self-attention [31] module on (cid:98) Z to query information for each object from its related objects:
[Equation]
where W q , W k , W v ∈ R F × F are trainable parameters.
Modeling the relations is important, because one object’s high-level semantic feature can be derived from other objects.
For example, whether a door can be opened is determined by the existence of the key; therefore, the door should query other objects to check whether it is openable.
After the attention module, (cid:98) Q and (cid:98) Z are concatenated together to get
[Equation]
Z = Ψ concatenate ([ (cid:98) Q , (cid:98) Z ]) ∈
R HW × 2 F .
Each (cid:101)
Z i,j, : in (cid:101)
Z is then fed into different independent neural networks according to its corresponding object category ˆ z catij .
In practice, this can be implemented as:
[Equation]
where f c : R 2 F → R F , [ˆ z catij ;1 , ..., ˆ z catij ; C +1 ]
= ˆ z catij .
Eq.(5) can be computed in parallel, resulting a tensor Z out ∈ R HW × F .
Finally, the Z out is added to (cid:98) Z (i.e. a residual connection).
We perform a max operation along the H, W dimension, and using the resulting vector ∈ R F to predict the value function and action probabilities:
[Equation]
Experiment 4.1 Task Description
In this work, we consider two task domains:
Crafter and Hunter .
The observations on both tasks are raw images of shape 64 × 64 .
Crafter [15] is a complex 2-D MineCraftstyle RL task, where complex behaviors are necessary for the agent’ survival.
The environment is procedurally generated by arranging various resources, terrain types, and objects (18 in total).
The agent is rewarded if it can craft new items and accomplish achievements (22 possible achievements in total).
[Table]
Hunter is much simpler than Crafter .
It is also procedurally generated but only contains 4 types of objects: Hunter , Cow , Zombie and Wall 2 .
The agent can control Hunter and can get positive reward (=1) if Hunter catches a Cow or kill a Zombie .
The agent will be given a high reward (=5) if it can accomplish this task by catching/killing all Cow s and Zombie s.
However, once Hunter is caught by a Zombie , the agent will receive a negative reward (=-1), and the episode ends.
In a word, the agent in Hunter should master two different behavior patterns: chase & catch the Cow , and avoid & shoot at Zombie .
Although Hunter is simple, it is a typical OORL task.
We can derive different environment instances from Hunter by setting the number of Zombie s and Cow s. We use
Hunter-ZmCn to denote an environment that spawns ( m Zombie s + n Cow s) at the beginning of each episode, and Hunter-ZmCn/ZnCm an environment that spawns ( m Zombie s + n Cow s) or ( n Zombie s + m Cow s).
4.2 Evaluation
In this section, we evaluate OCARL’s effect both on sample efficiency and generalization ability.
We consider the following baselines: PPO [29], RRL [41], and SMORL [40].
PPO is a general-purpose on-policy RL algorithm with a plain neural network (i.e. Convolution + MLP).
RRL proposes an attention-based neural network to introduce relational inductive biases and iterated relational reasoning into RL agent, which aims to solve object-oriented RL tasks without any other external supervision signals.
SMORL utilizes the UOD model to decompose the observation into a set of object representations which are directly used as the agent’s new observation.
All algorithms (RRL,
SMORL, and OCARL) are (re-)implemented upon PPO to ensure a fair comparison.
For more implementation details, please refer to the Appendix.
4.2.1 Sample Efficiency
The results on Hunter domain are shown in Figure 3.
We can conclude three facts from this figure: (1)
OCARL performs better and is more stable than other baselines on all tasks.
(2) RRL is competitive with OCARL on Hunter-Z4C4 but significantly worse on other simper environments.
This is because
RRL relies solely on the reward signals to ground the visual images into objects; therefore, it requires enough (rewarded) interactions between objects, which simpler environments such as Hunter-Z1C fail to supply.
(3) SMORL performs better than other baselines (PPO, RRL) on simple environments such as Hunter-Z1C0/Z0C1 , Hunter-Z1C1 and Hunter-Z4C0/Z0C4 , but unable to make progress in Hunter-Z4C4 which consists of more objects.
This is because SMORL adopts a simple reasoning module that may not be well-suited to multi-object reasoning.
The results on Crafter domain are shown in Table 1 and Figure 4.
Crafter is a complex environment in that it features sparse (rewarded) interactions between objects and large object category numbers.
Besides, some objects are omitted by the UOD model (see Appendix) because they almost do not appear in the training dataset due to the insufficient exploration of the random policy.
Due to these features, both RRL and SMORL fail to provide noteworthy results (with scores of 4.22 and 3.94, respectively).
On the other hand, OCARL is able to present more meaningful behaviours (with a score of 12.31), such as collecting coal, defeating zombies, making wood pickaxes and so on, as shown in Figure 4.
The learning curves and detailed achievement success rates are provided in the
Appendix.
4.2.2 Generalization
To evaluate the generalization ability, we train agents on Hunter-Z1C0/Z0C1 , Hunter-Z1C and Hunter-Z4C0/Z0C4 separately, and then observe their test performance on Hunter-Z4C4 .
In Hunter-Z1C1 , the agent needs to generalize from a few objects to more objects.
In
Hunter-Z4C0/Z0C4 , the agent never observes the coincidence of Cow and Zombie .
The generalization from Hunter-Z1C0/Z0C1 to Hunter-Z4C4 is the most difficult.
Such a paradigm actually follows the out-of-distribution (OOD) setting, in that the object combination ( 4 Zombie s + 4 Cow s) is never seen in the training environments.
Therefore, although such generalization is possible for humans, it is much harder for RL algorithms.
As shown in Figure 5, OCARL is the only algorithm that make significant progress on the test environment.
Although other baselines can improve their training performance (Figure 3), they are unable to generalize to the test environment.
The generalization performance of OCARL is most impressive on Hunter-Z1C1 , because agent trained on Hunter-Z1C1 even achieves slightly better than that on Hunter-Z4C4 (i e. directly trained on the test environment, see the last figure in Figure 3.).
4.3 Ablation study 4.3.1 Robustness to the Incompleteness of Object Discovery Methods
OCARL relies on UOD methods to discover different objects from given observations, which are not guaranteed to discover all objects, especially in complex environments [36].
Therefore, one may naturally ask to what extent OCARL’s performance is affected by such incompleteness.
To answer this question, we randomly mask out each object (by setting z presij to 0 ) discovered by the
UOD model with a given probability p ∈ [0 , 1] , and evaluate OCARL’s performance after training 5M environment steps on Hunter .
Such random confusion of the UOD model will affect both the OCAP and OCMR modules of OCARL.
Note that OCARL( p = 0 ) is OCARL itself, and OCARL( p = 1 ) is exactly RRL 3 .
The results are shown in Figure 6.
Although increasing p does hinder OCARL’s performance, OCARL can still benefit from the imperfect UOD model in that its performance is still better than RRL in general.
Such robustness should be attributed to the fact that OCARL does not directly use the output of the UOD model as observation but instead treats it as an additional supervision signal that help the encoder to capture object-category information, which gives OCARL a second chance to find other useful information from the raw image.
4.3.2 OCMR Improves Generalization
One key advantage of OCARL is that it can generalize to unseen combinations of objects by leveraging the invariant relations between objects learnt during training.
We argue that such an advantage is established via OCMR, which adopts object-category-specific networks to deal with different objects.
[Table]
To confirm our argument, we consider a variant of OCARL (i.e. OCARL w/o OCMR) in which the
OCMR module is removed and replaced by an RRL-like reasoning module which adopts a universal network (instead of multiple object-category-specific networks as OCARL does).
The comparison between OCARL w/ and w/o OCMR is shown in Figure 7.
OCARL w/o OCMR (almost completely) fails to generalize from Hunter-Z1C0/Z0C1 , Hunter-Z4C0/Z0C4 to Hunter-Z4C4 , while OCARL w/ OCMR does.
In Figure 7, we also consider a variant of OCARL (i.e. OCARL w/o OCAP) in which the OCAP module is disabled by setting λ cat = 0 in Eq.(3).
In this paradigm, only OCMR is in effect.
As shown in Figure 7, OCMR alone can still bring benefits without the help of OCAP (although not as consistently as OCMR+OCAP ).
According to these results, we can conclude that
OCMR is crucial in OCARL’s success.
4.3.3 The Modularity of OCMR
OCMR adopts different modules to deal with object features from different categories.
In this section, we will show that each module in OCMR actually encodes an object-category-specific behaviour pattern.
As stated in Section 4.1, the agent should master two useful behaviour patterns in Hunter : (1) chase & catch the Cow , and (2) avoid & shoot at Zombie .
The existence of these patterns can be observed by evaluating the agent in Hunter-Z0C4 and Hunter-Z4C0 respectively, because agents that master pattern (1) (pattern (2)) should achieve better performance on Hunter-Z0C4 ( Hunter-Z4C0 ).
In
Table 2, we first train agent on Hunter-Z4C4 and then disable the module f Z (corresponding to Zombie ) or f C (corresponding to Cow ) by replacing f C or f E with f bg (corresponding to the ’background’) in Eq.(5).
The results shows that disabling f Z does not affect the performance on
Hunter-Z0C4 ( 8 . 29 → 8 . 13 ), which means that f Z is not relative to behaviour pattern (1).
Instead, f Z is corresponding to behaviour pattern (2) because we can observe a significant performance decline ( 8 . 27 → 1 . 17 ) on Hunter-Z4C0 when we disable f Z .
Similar phenomenon can also be observed on f C .
Conclusion
In this paper, we propose OCARL, which utilize the category information of objects to facilitate both perception.
For the perception, we propose the OCAP, which enables the encoder to distinguish the categories of objects.
For the reasoning, we propose the OCMR, which adopts object-categoryspecific networks to deal with different objects.
Our experiments are carried out on Crafter and
Hunter .
Experiments show that OCARL outperforms other baselines both on sample efficiency and generalization ability.
We also perform several ablation studies to show that (1) OCARL is robust to the incompleteness of UOD methods, (2) OCMR is critical to improving generalization, and (3) each module in OCMR actually encodes an object-category-specific behaviour pattern in Hunter .
Limitation The main limitation is that we only test OCARL’s generalization ability to unseen object combinations, but not novel object instances.
In order to generalize to novel object instances that looks very different from old ones, the agent should interact with the novel objects first and then infer the their underlying categories according to these interactions.
Such a problem is likely to be in the meta-RL domain and needs more efforts to solve, which we would like to leave for future work.
Acknowledgement
This work is partially supported by the National Key Research and Development Program of
China(under Grant 2018AAA0103300), the NSF of China(under Grants 61925208, 62102399,
62002338, U19B2019, 61906179, 61732020), Strategic Priority Research Program of Chinese
Academy of Science (XDB32050200), Beijing Academy of Artificial Intelligence (BAAI) and Beijing Nova Program of Science and Technology (Z191100001119093) , CAS Project for Young Scientists in Basic Research(YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize. A Implementation Details
The implementation of OCARL is available at https://anonymous.4open.science/r/
OCARL-51BF .
Hyper-parameters for SPACE
In this paper, we use SPACE for object discovery.
Most hyperparameters are the same as the default parameters for Atari in SPACE, except that (1) we use a smaller network to deal with 64 × 64 images; (2) For Hunter , we disable the background module in SPACE because the observations in Hunter is quite simple.
(3) We set the dimension of object representations z whatij to be 16, and (in Crafter )
z fg to be 8.
The training data for SPACE is obtained via running a random policy on Hunter-Z4C4 and Crafter for 100000 environment steps.
Hyper-parameters for PPO Our PPO implementation is based on Tianshou [37] which is purely based on PyTorch.
We adopt the default hyper-parameters in Tianshou, which are shown in Table 3.
Hyper-parameters for OCARL OCARL only introduces 2 hyper-parameter λ cat (the coefficient for L cat defined in Eq.(3.2)) and C (the number of object categories).
We set λ cat = 0 .
01 for all experiments, C = 4 , 8 for Hunter and Crafter respectively.
Network Architecture In OCAP, we use the convolution encoder from IMPALA [10], which consists of 3 residual blocks with 16, 32, and 32 channels, respectively.
By the encoder, the 3 × 64 × 64 image observation is encoded into 32 × 8 × 8 .
The category predictor f cat is a simple 2-layer MLP with hidden size 32.
The object-category-specific networks f 1 , ..., f C +1 in the OCMR module consists of C + 1 MLPs, each of which is of 2-layer with hidden size 64.
Implementation for RRL The main implementation differences between RRL and OCARL are that (1) RRL does not use the supervision signals from the UOD model (i.e. λ cat = 0 in Eq.(3)); (2) RRL adopts a different (although very similar) network architecture.
In RRL, we use the same convolution encoder as OCARL.
The reasoning module in RRL is similar to OCARL, except that it utilizes a non-linear universe transformation instead of multiple object-category-specific networks.
We also use only one relational block in RRL as OCARL does, because we found more blocks may hinder the performance in our early experiments.
Implementation for SMORL
We use SPACE [21] instead of SCALOR [17] (as suggested in the
SMORL paper) to obtain object representations for SMORL.
Since SMORL is originally designed for goal-based RL, it uses an attention model to gather information from the set of object representations, in which the goal serves as a query vector.
In our settings, we do not consider goal-based RL; therefore, we use L learnable vectors as queries, which are also used in the original SMORL.
We search over L = [1 , 2 , 4 , 8] and find that L = 4 works best in our experiments.
We also find that it is better to apply an MLP to the object representations before they are fed into the attention module.
To be more specific, we first run SPACE to get the object representations, sharing the same procedure as OCARL.
The information of these objects is first processed by a 2-layer MLP with hidden size 32 and then gathered through an attention module in which the query consists of L = 4 learnable vectors.
The resulting tensor (of shape L × 32 ) is flattened and then fed into a 2-layer MLP with hidden size 64, which finally output the value function and action probabilities.
B Details of SPACE
OCARL utilizes SPACE to discovery objects from raw objects.
In Section 3.1, we have introduced the inference model of z fg .
For completeness, we would like to introduce the remaining parts of
SPACE in this section: (1) inference model of z bg ; (2) generative model of the image x ; and (3)
the
ELBO to train the model.
Inference model of z bg z bg consists of several components
z bgk , and these components are inferred from the image x
in an iterative manner: q ( z bgk | x ) =
(cid:81)
Kk =1 q ( z bgk | z bg<k , x ) .
Each component z bgk is futher divided into two parts: z bgk =
( z mk , z ck ) , where z mk models the mixing weight assigned to the background component (see π k in the generative model of x ), and z ck models the RGB distribution ( p ( x | z bgk ) ) of the background component.
[Table]
Generative model of x
The generative model consists of two parts: p ( x | z fg ) and p ( x | z bg ) .
For p ( x | z fg ) , each z fgij is passed through a decoder to reconstruct the image patch determined by z whereij .
For p ( x | z bg ) , each z bgk is decoded into a background component and all components are mixed together to get the background.
Foreground and background are combined with a pixel-wise mixture model to reconstruct the original image x .
The whole generative model of x is:
[Equation]
In above equations, z presij , z whereij , z whatij have been discussed in Section 3.1, α is foreground mixing probability, and π k is the mixing weight assigned to the background component z bgk .
The ELBO to train the model SPACE is trained using the following ELBO via reparameterization tricks:
[Equation]
[Table]
Stone Pickaxe
[Table]
Stone
[Table]
Make Wood Pickaxe
[Table]
C More Results on Crafter
, while OCARL does.
In Figure 8, we plot the learning curves on the
Crafter domain.
Only OCARL is able to make progress continuously.
Both RRL and SMORL get stuck in local minimal after training for about 1M environment steps.
In Table 4, we give the detailed success rates for 22 achievements on Crafter , which is also reported in Figure 4.
As shown in Table 4,
OCARL presents more meaningful behaviours such as collecting coal, defeating zombies, making wood pickaxe and so on.
[Table] [Table]
D Analysis of Unsupervised Object Discovery
In Figure 9, we plot the discovered object categories in Crafter and Hunter .
Since Hunter is quite simple, our algorithm can assign correct categories for all objects.
In Crafter , there existkinds of objects in total, making it much harder than Hunter .
Since we use only 8 categories, there are some cases that multiple objects with different ground-truth categories are predicted into the same category (such as Category 5 and 7 in Figure 9).
Although there exist assignment mistakes in
Crafter , OCARL is still much better than other baselines.
Figure 9 also shows that some objects in Crafter is omitted by the UOD model, such as , , , , and .
This is probably because the some objects almost does not appear in in training dataset (Table 5) or are treated as background by SPACE.
In Figure 10, we report the object discovered by SPACE.
The combination of SPACE and unsupervised clustering on the discovered objects (i.e. Eq.(2)) can tell us not only where one object locates, but also what category it is.
[Figure] [Figure]