Self-supervised Reinforcement Learning with Independently Controllable Subgoals
Anonymous Author(s)
AfﬁliationAddress email
Abstract: To successfully tackle challenging manipulation tasks, autonomousagents must learn a diverse set of skills and how to combine them.
Recently,self-supervised agents that set their own abstract goals by exploiting the discovered structure in the environment were shown to perform well on many differenttasks.
In particular, some of them were applied to learn basic manipulation skillsin compositional multi-object environments.
However, these methods learn skillswithout taking the dependencies between objects into account.
Thus, the learnedskills are difﬁcult to combine in realistic environments.
We propose a novel selfsupervised agent that estimates relations between environment components anduses them to independently control different parts of the environment state.
In addition, the estimated relations between objects can be used to decompose a complex goal into a compatible sequence of subgoals.
We show that, by using thisframework, an agent can efﬁciently and automatically learn manipulation tasks inmulti-object environments with different relations between objects.
Introduction
Autonomous agents that need to solve manipulation tasks in environments with many objects haveto master a variety of skills.
In addition, such agents should be able to properly combine these skillsto solve complex tasks.
In modular environments, the agent must explore many different ways howit can control the environment [1].
Self-supervised agents that imagine their own goals can automatethis process and learn many skills without external reward signals [1, 2, 3, 4, 5, 6, 7].
One of themain challenges for goal-based autonomous agents is the choice of a suitable goal space and thecorresponding reward function [8].
As this choice determines the difﬁculty of the learning, it iscrucial to exploit all available structure in the environment state for construction of the goal space.
One natural way to represent the state in modular environments is to use an object-centric representation: the environment state is represented as a set of components, with each component corresponding to the state of an individual object [9, 10, 11].
Such representations can be learned in anunsupervised fashion from high-dimensional observations such as images [12, 13, 14, 9, 15, 16, 17].
Therefore, methods that use object-centric representations can be readily extended to take highdimensional data as input.
A simple approach to use object-centric representations in autonomous learning is to ﬁrst learn how to control each object individually (using the objects’ representationsas subgoals), and then combine learned skills to control multiple objects [10].
However, in an environment where different objects interact with each other, this method might learn an incompatiblesequence of skills, i.e. achieving one of the subgoals can destroy another previously achieved sub- goal.
For example, moving one object from a stack of objects may change the position of the others.
One line of work that aims at learning sequences of skills that are compatible is Hierarchical Rein- forcement Learning (HRL) [18, 19, 20].
In principle, hierarchical agents should be able to transforma task into a sequence of subtasks that they solve sequentially.
However, to date, existing hierarchi- cal agents have mostly been applied to learn navigation or reaching tasks where learned skills do
Random agent,
Observed environment dynamics
[Figure]
Interaction graph 3 rd obj
Arm 4 th obj 1 st obj
Action 2 nd obj
Goal-conditioned policy,
Soft Actor-Critic
Independently controllable subgoals
Arm 4 th obj 3 rd obj
Arm 2 nd obj
Arm
Arm 2 nd obj 3 rd obj
Agent with compositionalskills,
[Figure]
SRICS learns to solve an external goal s goal by decomposing it into an ordered list of subgoals g i and solving each using SAC [24] with a goal-conditioned policy π θ .
As a result, the agent attempts to solve all the discovered subgoals one-by-one, without destroying previously solved subgoals.
not interact with each other.
It is unclear how sensitive hierarchical agents are to possible interactions between learned skills.
In this paper, we investigate another approach by reformulating theagent’s subtasks and the corresponding reward signals.
Similar to Thomas et al.
[21], we train anagent such that it is motivated to control a particular component of the environment state representation while minimally affecting other components.
Such an agent can learn to control componentsindependently from other components, thus making the learned skills compatible with each other.
As the environment state representation is not necessarily disentangled as in Thomas et al.
[21], ourmethod should additionally account for possible relations between components.
We propose a novelselectivity reward signal that uses an interaction graph to determine a set of components that canbe selectively controlled without interacting with the remaining scene.
The interaction graph can beinferred from observed objects dynamics collected by a random policy without supervision [22, 23].
Thus, we combine learning of such interaction graphs with a goal-conditional reinforcement learning (RL) method that operates on object-centric representations [10] and uses the selectivity rewardsignal.
During training (schematically depicted in Fig. 1), our SRICS agent (for S elf-supervised
R elational RL with I ndependently C ontrollable S ubgoals) learns how to efﬁciently achieve different subgoals (and control the corresponding subspaces) while being incentivized to minimize itseffects on other parts of the environment.
Our main contributions are as follows:
• We show that the global interaction graph can be estimated from data using a recurrent graphneural network (GNN) dynamical model combined with a sparsity prior.
• We propose a goal-directed selectivity reward function that allows an agent to learn how tocontrol environment components independently from one another.•
We develop SRICS, an algorithm that uses the inferred interaction graph to learn simple andindependently controllable subtasks and decompose a complex goal into a compatible sequenceof subgoals.
Modular Goal-conditional Reinforcement Learning
We are interested in an agent that can solve multiple tasks in an environment.
In particular, weconsider goal-based task encodings where each task corresponds to an environment state the agenthas to reach, denoted as the goal state g .
The task is then given to the agent by conditioning its policyπ ( a t | s t , g ) on the goal g , and the agent’s objective is to maximize the expected goal-conditionalreturn:
E g ∼ P (cid:34) T (cid:88) t =
E s t ∼ ρ π , a t ∼ π, s t +1 ∼ d
[ r ( s t +1 , g )] (cid:35) where d is an unknown dynamics distribution, ρ π is the state marginal distribution induced by theagent’s policy π
and P is some distribution over the space of goals G that the agent receives fortraining.
A common approach to deﬁne the reward function in this setting is the negative distanceof the current state to the goal: r ( s , g )
= −(cid:107) s − g
(cid:107) .
In general, however, the goal space G doesnot need to be equal to the state space S , but can be any task embedding space with potentiallydifferent dimensionality [1, 25].
As some tasks cannot be expressed as desired regions of the statespace, the goal g can parameterize a more general objective r ( s , g ) that the agent should maximize.
Many environments are modular, in the sense that an agent’s overall goal (e.g. manipulating manyobjects) can be decomposed into different subgoals (e.g. manipulating individual objects) that canbe sequentially achieved.
2.1 Object-Centric Representations
We use object-centric representations for the state space S .
That is, the state space S is a directproduct of all object subspaces, S = S 1 × . . .
×S K , where each S j corresponds to the state of an entity.
The state of an entity is encoded by its position s j, where and an identiﬁer s j, what .
The semantics behindan entity are unknown to the agent, i.e. the state of both the agent and the objects to manipulate areencoded identically.
The agent has no information about which objects are controllable and how theyare related.
Object-centric representations could be learned in an unsupervised way from sequentialimage data [9, 12, 13] and learning them is an orthogonal line of research.
In this work, we focus onusing object-centric representations for decomposing the goal to subgoals that are compatible witheach other and can be achieved sequentially to solve the original goal.
The choice of the goal space plays a crucial role in determining the difﬁculty of the learning task.
Ifthe environment state consists of independent parts, it is easiest to learn to control these componentsindependently [10].
However, in the case of interactions between these components, learning toachieve subgoals in such environments and combining learned skills could be harmful to achievingthe original compositional goal.
For example, assume that the subgoals consist of moving objects todifferent positions.
By solving a single subgoal without taking the other subgoals into consideration,the agent might unintentionally rearrange objects from previous subtasks, resulting in an overalldeterioration instead of an improvement.
In the next section, we present a method that accountsfor such dependencies by learning an interaction graph to decompose a goal into independentlycontrollable subgoals and introduce a corresponding reward function.
Self-Supervised Relational RL with Independently Controllable Subgoals
In the setting we consider, at the training stage, the agent only receives a single compositional goalfrom the environment.
The agent could try to solve the goal using the usual negative distance tothe goal as a reward signal.
However, achieving the compositional goal is quite a complex task byitself.
This challenge can be addressed by discovering simple skills and combining them to solvethe compositional goal.
To achieve this, the agent needs to rely on self-supervision in the formof splitting the goal into subgoals and internally constructing the reward signal connected to eachsubgoal.
The agent uses data collected from the environment to discover how different parts of the environment are related, including the agent itself, and then uses the discovered relations for the constructionof subtasks that are solvable and can be easily combined.
First, we describe how to use object-centricrepresentations to estimate a graph of relations between objects, and then show how to utilize thelearned graph during agent training, and for goal decomposition during evaluation.
[Figure]
Next, the function d act computes the action’s j are combined in the GRU.
Finally, object’s state estimation ˆ s is estimated from the hidden state h
j t +1 using the prediction function d pred
3.1 Estimation of the Latent Interaction Graph with a GNN Dynamical Model
Relational information in the environment can help the autonomous agent to gain control over different parts of the environment.
For example, if some parts of the environment cannot be affected,the agent will be more efﬁcient by not trying to control them.
Recently, several methods to estimatethis relational information in an unsupervised way were proposed [23, 26, 27, 28, 29].
Most of themassume that the relations are static [23, 28, 30].
As this is not the case in many robotic manipulationapplications, we propose to use a similar approach suitable for constantly changing relations.
Forthis, we use a graph neural network (GNN) [22, 31] to model the forward dynamics of the objects.
Because states could be non-Markovian, we use a recurrent dynamical model.
Speciﬁcally, weincorporate recurrence in the GNN model by adding a Gated recurrent unit (GRU) [32] to the GNNmessage passing operation (see Fig. 2a).
We use the functions d int and d act to model the object-objectinteraction effect and the action’s effect, respectively.
Next, both effects are combined in the GRU.
More formally:
[Equation]
where h j, int t and h j, act t are vectors representing interaction and action effects, whereas h jt is thehidden state for object
j at time step t .
To model dynamics with sparse interactions between objects, we model d int as the product of aninteraction weight w ijt ∈ { 0 , 1 } and an interaction effect function d int-eff :
[Equation]
The interaction weight w ijt represents the belief in the absence or presence of the interaction betweenobject i and object j at time step t .
We model the weight’s distribution as
[Equation]
where d int-pres is the interaction presence function.
As we are interested in the estimation of theconnections that are necessary for predictions, we additionally encourage the interaction weightsdistribution q (cid:16) w ijt | s t (cid:17) to be close to the sparsity prior p prior .
In our case, the sparsity prior p prior isthe Bernoulli distribution with a large probability for zero (see App. G).
Finally, we use a function d pred to predict the change in coordinates (see Fig. 2b):
[Equation]
All functions in Eqs.
1–4 are modeled by small MLPs with parameters φ .
Now, as we deﬁned all the parts of the GNN dynamical model, we describe how to estimate the interaction graph using a variational approach.
First, similar to [23], we train our model by minimizingthe negative ELBO loss:
[Equation]
where ˆ s j, where is the prediction of the position of object j , σ 2 is a ﬁxed variance parameter and D KLdenotes the Kullback-Leibler divergence.
After training, we predict interaction weights w t for eachtimestep independently, then we average them across the whole dataset.
Next, we estimate the globalinteraction graph by thresholding the average interaction weights to ﬁnd the most active relations.
Finally, we identify which object is directly controlled by the actions by ﬁnding the node that is mostcorrelated with the action variable a t .
In the graph of our running example as in Figure 1, we denotethis node as ”arm” since in all experiments the identiﬁed node corresponds to a simulated robot arm.
We add the action node with index 0 and the corresponding edge to the most correlated object to thegraph (see App.
G for the details and the graph learning results).
3.2 Learning to Independently Control Objects using the Interaction Graph
In this section, we show how the agent can use the learned interaction graph to solve compositional goal s goal that consists of goals for individual objects s goal ,i .
The SRICS agent sequentiallygains control over the objects without affecting the previously moved objects.
To achieve this,the SRICS method ﬁrst identiﬁes a set of objects P i that could be used to actively control object i by analyzing the discovered relations in the interaction graph.
For each node i , we ﬁndthe set P i of all nodes that lie in a path from the action node 0 to object node i .
These ancestral nodes P i are the objects that could be used by the agent to control object i .
All the othernodes are not required and thus should not be affected during the manipulation of object i .
Arm 2 nd obj 3 rd obj
Arm 4 th obj 1 st obj
Action 2 nd obj 3 rd obj
Next, we introduce the reward signal that uses P i to incentivize the agent to learn to control an object withoutmoving others (line 8 of Alg. 1).
In order to achievethis, we propose to replace the original subgoal s goal ,i by anovel independently controllable subgoal g i that consistsof the subgoal s goal ,i and the ancestral nodes P i :
[Equation]
In contrast to the original notion of a subgoal which onlyspeciﬁes a state component s goal ,i that the agent shouldreach, an independently controllable subgoal g i also includes information about which objects should not be interacted with to reach the target state component.
We now formulate the goal-directed selectivity rewardsignal that explicitly incentivizes the agent to leave allobjects except i and P i untouched.
As opposed to the usual reward signal, it depends on the independently controllable subgoal g i and reads:
[Equation]
The ﬁrst term is the usual goal-based negative distance to the goal, which is needed to learn directedcontrol over object i .
The second term includes the selectivity that we deﬁne as
[Equation]
if subgoal is not solved
[Equation]
The selectivity sel i incentivizes the agent to maximize its inﬂuence [33, 34, 35] on object i whilehaving a minimal effect on objects j ∈ / P i (corresponding to non-ancestral nodes in graph G ) untilthe subgoal corresponding to the object i is solved.
Selectivity reaches its maximum value of 1 whenthe agent changes only the state of the object i without affecting any objects j ∈ / P i .
In App.
F, weshow that selectivity naturally increases during learning to control the environment and that using itas a reward signal increases efﬁciency and stability.
3.3 SRICS Policy Architecture and Training
Similar to the SMORL agent [10], we use a goal-conditioned attention policy for achieving subgoals.
This kind of policy receives a set of object-centric representations as input together with the currentsubgoal representation.
The aforementioned approach allows us to learn several different skillsusing only one policy.
In addition, it is compatible with a different number of objects as inputs, thusallowing to use the agent in novel situations with a different number of objects.
For more details onthe goal-conditioned attention policy, we refer to App.
E.
SRICS can be trained with any off-policy goal-conditioned RL algorithm.
In particular, we use
Soft-Actor Critic (SAC) [24] with Hindsight Experience Replay (HER) [25] as a method to improvesample efﬁciency.
The training of SRICS is presented in Alg. 1.
3.4 Subgoal Ordering during Evaluation
After training, the agent can be applied to more complex tasks than the simple subtasks it was trainedon.
During the evaluation stage (Fig. 13 in App. I), SRICS encodes the compositional goal given bythe environment into a set of independently controllable subgoals.
Subsequently, it orders them bythe depth of the corresponding nodes in the interaction graph G .
Due to this order, subgoals that havea large number of dependencies are attempted ﬁrst and subgoals that have only a few dependencies,like the robotic arm itself, are attempted as the later subgoals.
The order of the independentlycontrollable subgoals makes them compatible with each other.
For example, the agent has to ﬁrstrearrange all objects that need to be manipulated and then try to “solve” the arm subgoal, withoutdestroying the already rearranged objects.
More details can be found in App. I.
Related Work
In self-supervised reinforcement learning , self-supervision refers to the agent constructing its owngoals together with the corresponding reward signal and using them to learn to solve self-proposedgoals [1, 29, 36, 37, 2, 3, 6, 38, 39, 40, 41, 42, 4, 10].
Self-supervised agents can acquire a diverseset of general-purpose robotic skills.
In the case of complex tasks, it is often beneﬁcial to discoversimpler subgoals and learn to solve them [20].
From this point of view, recent hierarchical RL (HRL)agents [20, 19, 43, 44, 18, 45] that try to solve external tasks by proposing several levels of internalsubgoals are also self-supervised agents.
Levy et al.
[20], Nachum et al.
[19] and Wang et al.
[43] propose to learn several goal-conditionedpolicies.
In the HIRO agent [19], lower-level controllers are supervised with goals that are learnedand proposed automatically by the higher-level controllers.
In contrast, the HAC agent [20] trainseach level of the hierarchy independently of the lower levels.
The I 2 HRL agent [43] additionallyallows bi-directional communication among HRL levels and inﬂuence-based exploration to maketraining more stable and efﬁcient.
As such agents need to discover all the structure in the environment while learning on several levels, such approaches struggle to solve complex tasks in modularenvironments [46].
Next, we review agents operating in environments where some structure is given.
The SMORL agent [10] exploits learned object-centric representations for gaining control over different objects in a self-supervised way and combines the learned skills for solving more complexcompositional tasks.
However, Zadaianchuk et al.
[10] assume independence of different objects,restricting the use of the SMORL agent to settings where objects almost do not interact with eachother.
CURIOUS [8] and CWYC [29] exploit the modular structure of the goal space for efﬁcientexploration in a given goal space.
Colas et al.
[8] use a policy that obtains the goal module identiﬁertogether with the goal value.
Blaes et al.
[29] also learn a relational graph between tasks.
Bothagents use a given modular structure for a learning curriculum [36], however, discovered subtasksare evaluated independently.
In realistic applications, autonomous agents usually do not have any well-structured representation.
Nevertheless, agents can potentially infer it from data.
We cover several directions that could beuseful for such structure discovery.
The ﬁrst line of works [13, 12, 9, 14, 16] learns object-centricrepresentations from images or videos.
Such representations could be potentially used in combination with the SRICS agent.
The second line [23, 27, 26, 28, 30] studies how object relationscan be discovered from data.
The improvements in both of these lines could lead to more generalself-supervised agents that use a discovered structure for the generation of goals.
5 Training examples × 100 .0 .0 .0 .0 .0 .0 .
A vg .
d i s t .
Rearrange with 3 objects     5 Training examples × 100 .
Rearrange with 4 objects
[Figure]
Training examples × 100 .
[Figure]
[Figure] [Figure]
Relational
[Figure]
65,& 6025/ 6$&+( +$&
Experimental Results
In this section, we present our experiments that address the following questions:
• How does SRICS perform compared to prior goal-conditioned RL methods on multi-object
continuous control manipulation tasks?
• What is the performance gain obtained from the goal-directed selectivity reward and subgoal
ordering during evaluation?
• How does our agent perform in an environment with an unseen combination of objects?
We run SRICS and the baseline algorithms in the Multi-Object Rearrange from Zadaianchuk et al.[10] and the novel Multi-Object Relational Rearrange environments.
The latter environment incorporates additional physical connections between objects such as spring connections.
Both environments are based on the multiworld package for continuous control tasks introduced by Nair et al.[2] and use MuJoCo [47] as a realistic simulator.
They contain a 7-DoF Sawyer arm where the agentneeds to manipulate a variable number of pucks on a table.
In the ﬁrst environment, the task is torearrange the objects from random starting positions to random target positions.
In the second environment, we add a spring connection between some of the objects and constrain other objects to bestatic (see App. C).
This makes the resulting interaction graph more challenging and thus providesadditional insights on the sensitivity of the agent to different interactions between objects.
For bothenvironments, we measure the performance of the algorithms as the average distance of all objects(including the robotic arm) to their goal positions (computed on the last step of the episode).
5.1 Comparative Analysis
Arm  4 Object 0 .0 .0 .0 .0 .1 .
S ub t a s k s u cce ss r a t e
Relational Rearrange with 4 objects
[Figure]
during evaluation in the Relational Rearrange environment.
Both methods can solve
Arm reaching subgoal, whereas on other subtasks SRICS performs better than SMORL.
As manipulation tasks in compositional environments can be approached from different perspectives, we provide a comparison with a state-of-theart method from each perspective.
In terms of problem assumptions, our work is closest to that of
SMORL [10] which uses object-centric representations for subgoals and reward construction.
In contrast to SRICS, SMORL executes subgoals in a random order and thus can potentially destroy previously solved subgoals.
In addition, the SMORLagent does not have the incentive to inﬂuence thesubgoal object during training.
Another approachto learn goal-conditioned policy with coherent behavior is using Soft Actor-Critic (SAC) [24] with
Hindsight Experience Replay (HER) [25] relabeling.
This method tries to achieve the overall goal withoutsplitting it into subgoals.
Finally, we consider the Hierarchical Actor-Critic (HAC) [20] methodthat tries to solve compositional tasks on several levels and is state of the art on several continuouscontrol tasks.
We show the results in Fig.
3 and Fig.
The performance of SRICS is signiﬁcantly better than allother algorithms in both environments.
SMORL is able to partially rearrange pucks on a table in thesimpler Multi-Object Rearrange environment.
However, its random subgoals ordering is inefﬁcientfor arranging all the objects including the arm.
In addition, even when evaluating only based on thepuck subtasks (see App. H) , SRICS outperforms SMORL, which further demonstrates the beneﬁtsof using a goal-directed selectivity reward signal.
Moreover, in the more complex Multi-Object
Relational Rearrange environment, the gap between SRICS’s and SMORL’s performance is evenlarger.
Furthermore, in all environments SAC is only able to solve the Arm subtasks, whereas HACperformance is close to that of a random agent.
We present further comparison in more challengingenvironments with 6 different objects and velocity-based state representations in App.
D.
5.2 Ablative Analysis
Here, we study the importance of different ingredients of our method for the overall performance ofthe agent.
First, we ablate the selectivity term in our reward signal, using only the negative distancebetween the object and the desired position as a reward signal.
We then additionally ablate the ordering of subgoals described in Sec. 3.4, using instead a random ordering of all subgoals.
The resultsof the ablations are presented in Fig.
Both ablations signiﬁcantly deteriorate the performance of
SRICS, showing the importance of both the goal-directed selectivity reward signal and the correctordering in the goal decomposition for object manipulation in multi-object environments.
[Figure] [Figure]
5.3 Generalization to Unseen Object Combinations
[Figure]
As SRICS can be used with different sets of objects as inputs, we investigate its performance on unseen combinations of objects.
We train SRICS on the Multi-Object Rearrange environment, where4 different combinations of 3 objects are presented.
We leave out one of the combinations for evaluation and use the other 3 combinations for training.
The performance on this modiﬁed environmentis indistinguishable from SRICS’s performance on Multi-Object Rearrange with 3 objects, showingthat SRICS can operate on novel combinations of objects (details in App. B).
Conclusion and Future Work
In this work, we introduce SRICS, a self-supervised RL method that learns the relational structureof the environment and exploits this structure to learn a compatible sequence of skills to solve a difﬁcult compositional goal.
In a range of experiments in multi-object environments with robotic armmanipulation tasks, we demonstrate that SRICS is effective at discovering the most active dynamicrelations between objects and can successfully rearrange multiple objects even in the presence ofobject interactions.
There are several interesting directions for future work.
First, one can extend SRICS to image-basedobject-centric representations, making it more applicable to realistic robotic settings where onlyhigh-dimensional sensory information is provided as input to the agent.
Moreover, we expect that
SRICS can be combined with different modular curriculum learning and exploration strategies [8,29].
Finally, we expect that active training of the dynamic interaction graph (i.e. when the data fortraining is collected by the agent that actively explores the environment) could further improve thediscovery of important structures in the environment.