C AUSAL W ORLD : A R OBOTIC M ANIPULATION B ENCHMARK FOR C AUSAL S TRUCTURE AND T RANSFER L EARNING
A BSTRACT
Despite recent successes of reinforcement learning (RL), it remains a challenge for agents to transfer learned skills to related environments.
To facilitate research addressing this problem, we propose CausalWorld , a benchmark for causal structure and transfer learning in a robotic manipulation environment.
The environment is a simulation of an open-source robotic platform, hence offering the possibility of sim-to-real transfer.
Tasks consist of constructing 3D shapes from a set of blocks - inspired by how children learn to build complex structures.
The key strength of
CausalWorld is that it provides a combinatorial family of such tasks with common causal structure and underlying factors (including, e.g., robot and object masses, colors, sizes).
The user (or the agent) may intervene on all causal variables, which allows for ﬁne-grained control over how similar different tasks (or task distributions) are.
One can thus easily deﬁne training and evaluation distributions of a desired difﬁculty level, targeting a speciﬁc form of generalization (e.g., only changes in appearance or object mass).
Further, this common parametrization facilitates deﬁning curricula by interpolating between an initial and a target task.
While users may deﬁne their own task distributions, we present eight meaningful distributions as concrete benchmarks, ranging from simple to very challenging, all of which require long-horizon planning as well as precise low-level motor control.
Finally, we provide baseline results for a subset of these tasks on distinct training curricula and corresponding evaluation protocols, verifying the feasibility of the tasks in this benchmark.
I NTRODUCTION
do(floor_color=’white’, block_size=0.065, …etc)
Benchmarks have played a crucial role in advancing entire research ﬁelds, for instance computer vision with the introduction of CIFAR-10 and ImageNet (Krizhevsky et al., 2009; 2012).
When it comes to the ﬁeld of reinforcement learning (RL), similar breakthroughs have been achieved in domains such as game playing (Mnih et al., 2013; Silver et al., 2017), learning motor control for high-dimensional simulated robots (Akkaya et al., 2019), multi-agent settings (Baker et al., 2019;
Berner et al., 2019) and for studying transfer in the context of meta-learning (Yu et al., 2019).
Nevertheless, trained agents often fail to transfer the knowledge about the learned skills from a training environment to a different but related environment sharing part of the underlying structure.
This can be attributed to the fact that it is quite common to evaluate an agent on the training environments themselves, which leads to overﬁtting on these narrowly deﬁned environments (Whiteson et al., 2011), or that algorithms are com- ∗ Equal contribution. to
[Figure]
pared using highly engineered and biased reward functions which may result in learning suboptimal policies with respect to the desired behaviour; this is particularly evident in robotics.
In existing benchmarks (Yu et al., 2019; Goyal et al., 2019a; Cobbe et al., 2018; Bellemare et al., 2013; James et al., 2020) the amount of shared causal structure between the different environments is mostly unknown.
For instance, in the Atari Arcade Learning environments, it is unclear how to quantify the underlying similarities between different Atari games and we generally do not know to which degree an agent can be expected to generalize.
To overcome these limitations, we introduce a novel benchmark in a robotic manipulation environment that we call CausalWorld .
It features a diverse set of environments that, in contrast to previous designs, share a large set of parameters and parts of the causal structure.
Being able to intervene on these parameters (individually or collectively) permits the experimenter to evaluate agents’ generalization abilities with respect to different types and magnitudes of changes in the environment.
These parameters can be varied gradually, which yields a continuum of similar environments.
This allows for ﬁne-grained control of training and test distributions and the design of learning curricula.
A remarkable skill that humans learn to master early on in their life is building complex structures using spatial-reasoning and dexterous manipulation abilities (Casey et al., 2008; Caldera et al., 1999; Kamii et al., 2004).
Playing with toy blocks constitutes a natural environment for children to develop important visual-spatial skills, helping them ‘generalize’ in building complex composition designs from presented or imagined goal structures (Verdine et al., 2017; Nath & Sz¨ucs, 2014; Dewar, 2018; Richardson et al., 2014).
Inspired by this, CausalWorld is designed to aid in learning and investigating these skills in a simulated robotic manipulation environment corresponding to the open-source TriFinger robot platform from W¨uthrich et al.
(2020), which can be built in the real world.
Tasks are formulated as building 3D goal shapes using a set of available blocks by manipulating them - as seen in Fig.
This yields a diverse familiy of tasks, ranging from relatively simple (e.g. pushing a single object) to extremely hard (e.g. building a complex structure from a large number of objects).
CausalWorld improves upon previous benchmarks by exposing a large set of parameters in the causal generative model of the environments, such as weight, shape and appearance of the building blocks and the robot itself.
The possibility of intervening on any of these properties at any point in time allows one to set up training curricula or to evaluate an agent’s generalization capability with respect to different parameters.
Furthermore, in contrast to previous benchmarks (ChevalierBoisvert et al., 2018; Cobbe et al., 2018), researchers may build their own real-world platform of this simulator at low cost, as detailed in W¨uthrich et al.
(2020), and transfer their trained policies to the real world.
[Table]
Finally, by releasing this benchmark we hope to facilitate research in causal structure learning, i.e. learning the causal graph (or certain aspects of it) as we operate in a complex real-world environment whose dynamics follow the laws of physics, which induce causal relations between the variables.
Changes to the variables we expose can be considered do-interventions on the underlying structural causal model (SCM).
Consequently, we believe that this benchmark offers an exciting opportunity to investigate causality and its connection to RL and robotics.
Our main contributions can be summarized as follows:
• We propose CausalWorld , a new benchmark comprising a parametrized family of robotic manipulation environments for advancing out-of-distribution generalization and causal structure learning in RL.
• We provide a systematic way of deﬁning curricula and disentangling generalization abilities of RL agents with respect to different changes in the environment, since we allow for dointerventions to be performed on different environment variables (parameters and states) individually.
• We establish baseline results for some of the available tasks under different learning algorithms, thus verifying the feasibility of the tasks.
• We show how different learning curricula affect generalization across different axes by reporting some of the in-distribution and out-of-distribution
generalization capabilities of the trained agents.
C AUSAL W ORLD B ENCHMARK
Here we make the desiderata outlined in the introduction more precise:
The set of environments should be sufﬁciently diverse to allow for the design of challenging transfer tasks.
We need to be able to intervene on different properties (e.g. masses, colors) individually, such that we can investigate different types of generalization.
It should be possible to convert any environment to any other environment by gradually changing its properties through interventions; this requirement is important for evaluating different levels of transfer and for deﬁning curricula.
The environments should share some causal structure to allow algorithms to transfer the learned causal knowledge from one environment to another.
There should be a uniﬁed measure of success, such that an objective comparison can be made between different learning algorithms.
The benchmark should make it easy for users to deﬁne meaningful distributions of environments for training and evaluation.
In particular, it should facilitate evaluation of indistribution and out-of-distribution performance.
The simulated benchmark should have a real-world counterpart to allow for sim2real.
In light of these desiderata, we propose a setup in which a robot must build goal shapes using a set of available objects.
It is worth noting that similar setups were proposed previously in a less realistic setting, e.g. in (Janner et al., 2018; Bapst et al., 2019; McCarthy et al.
; Akkaya et al., 2019; Fahlman, 1974; Winston, 1970; Winograd, 1972).
Speciﬁcally, a task is formulated as follows: given a set of available objects, the agent needs to build a speciﬁc goal structure , see Fig.
1 for an example.
The vast amount of possible target shapes and environment properties (e.g. mass, shape and appearance of objects and the robot itself) makes this a diverse and challenging setting to evaluate different generalization aspects.
CausalWorld is a simulated version (using the Bullet physics engine (Coumans et al., 2013)) of the open-source TriFinger robot platform from W¨uthrich et al.
Each environment is deﬁned by a set of variables such, as gravity, ﬂoor friction, stage color, ﬂoor color, joint positions, various block parameters (e.g. size, color, mass, position, orientation), link colors, link masses and the goal shape.
See Table 3 in the Appendix for a more extensive list of these variables.
Desideratum 1 is satisﬁed since different environment properties and goal shapes give rise to very different tasks, ranging from relatively easy (e.g. re-positioning a single cube) to extremely hard (e.g. building a complex structure).
Desideratum 2 is satisﬁed because we allow for arbitrary interventions on these properties, hence users or agents may change parameters individually or jointly.
Desideratum 3 is satisﬁed because the parameters can be changed gradually.
Desideratum 4 is satisﬁed because all the environments share the causal structure of the robot, and one may also use subsets of environments which share even more causal structure.
We satisfy desideratum 5 by deﬁning the measure of success for all environments as the volumetric overlap of the goal shape with available objects.
Further, by splitting the set of parameters into a set A, intended for training and in-distribution evaluation, and a set B, intended for out-of-distribution evaluation, we satisfy desideratum 6.
Finally, since the TriFinger robot (W ¨ uthrich et al., 2020) can be built in the real-world, we satisfy desideratum 7.
Desideratum 7 and 2 are in partial conﬂict since sim2real is only possible for the tasks which are constrained to the variables on which the robot can physically act upon.
Task generators: To generate meaningful families of similar goal shapes, CausalWorld allows for deﬁning task generators which can generate a variety of different goal shapes in an environment.
For instance, one task generator may generate pushing tasks, while another one may generate towerbuilding tasks (see Fig. 2).
Each task generator is initialized with a default goal shape from its corresponding family and comes with a sampler to sample new goal shapes from the same family.
Additionally, upon construction, one can specify the environments’ initial state and initial goal shape structure when deviating from the default.
The maximum episode time to build a given shape is number of blocks × 10 seconds.
CausalWorld comes with eight pre-deﬁned task generators (see Fig. 2).
• Three generators create goal shapes with a single block: Pushing with the goal shape on the ﬂoor, Picking having the goal shape deﬁned above the ﬂoor and Pick and Place where a ﬁxed obstacle is placed between the initial block and goal pose.
Stacking2 involves a goal shape of two stacked blocks, which can also be considered one instance of the Towers generator.
• The remaining generators use a variable number of blocks to generate much more complex and challenging target shapes, as detailed in the appendix: Towers , Stacked Blocks , Creative Stacked Blocks and General .
Given that building new environments using current physics simulators is often tedious, we provide a simple API for users who wish to create task generators for new challenging shape families, which may be added to CausalWorld’s task generators repository.
Action and Observation Spaces: The robot can be chosen to operate in either joint position control mode, joint torque control mode, end-effector position control mode, or the delta of each.
In any of these cases, the action is 9-dimensional (one per joint).
We provide two observation modes: structured and pixel .
In the structured mode, the observation vector is constructed using a rule for the ordering of the relevant variables, such as joint positions, joint velocities, block positions, etc.
Thus, the size of the observation space depends on the number of blocks, which could potentially change with every new goal sampled, e.g. in Towers, (Creative)
Stacked Blocks and General.
In contrast, in the pixel mode, the agent receives six RGB images (hence the dimension of the observation is 6 × 3 × 128 × 128 ), the ﬁrst three images are rendered from the three cameras mounted around the TriFinger robot, and the last three images specify the goal image of the target shape rendered from the same cameras.
Additionally, CausalWorld allows users to set up a fully customized observation space.
Rewards: The reward function r is deﬁned as the fractional volumetric overlap of the blocks with the goal shape, which ranges between 0 (no overlap) and 1 (complete overlap).
Since this reward function is shared across all tasks, an agent that learned r from some training tasks could in principle use it to solve unseen goal structures.
There is also the possibility of modifying the reward function to 1) sparsify the reward further by returning a binary reward signal instead, or 2) add a dense reward function in order to introduce inductive biases via domain knowledge and solution guidance.
We hope that the considerable complexity and diversity of goal shapes motivate and accelerate the development of algorithms that are not dependent on hand-tuned reward functions.
[Figure]
Training and evaluation spaces:
In this benchmark, a learning setting consists of an allowed training space (ATS) and an evaluation space (ES), both of which are subspaces of the full parameter space.
During training, in the simplest setting, parameters are sampled iid from the ATS.
However, unlike existing benchmarks, CausalWorld allows in addition for curricula within the ATS as well as settings where the agent itself intervenes on the parameters within an episode (see Fig. 3).
Similarly, during evaluation, parameters may be sampled iid from the evaluation space at each episode reset, or there can be interventions within an episode.
Moreover, in order to retrieve the setting considered in most RL benchmarks, we could set the ATS and the ES to be identical and intervene only on object and robot states (and keep other environment properties constant) at each episode reset.
However, to evaluate out-of-distribution generalization, one should set the two spaces (ATS and ES) to be different; possibly even disjoint.
Additionally, to evaluate robustness with respect to a speciﬁc parameter (e.g. object mass), one may deﬁne the training and evaluation spaces which only differ in that particular parameter.
In order to facilitate the deﬁnition of appropriate training and evaluation settings, we pre-deﬁne two disjoint sets, A i and B i , for each parameter i .
Through this, one can for instance deﬁne the training space to be A 1 × A 2 × ... and the evaluation space to be B 1 × B 2 × ... to assess generalization with respect to all parameters simultaneously.
Alternatively, the evaluation space could be deﬁned as A 1 × A 2 × ...
× B i × A
i +1 × ... to assess generalization with respect to parameter i only.
Lastly, users may also deﬁne their own spaces which could then be integrated into the benchmark to give rise to new learning settings.
Intervention actors: To provide a convenient way of specifying learning curricula, we introduce intervention actors.
At each time step, such an actor takes all the exposed variables of the environment as inputs and may intervene on them.
To encourage modularity, one may combine multiple actors in a learning curriculum.
This actor is deﬁned by the episode number to start intervening, the episode number to stop intervening, the timestep within the episode it should intervene and the episode periodicity of interventions.
We provide a set of predeﬁned intervention actors, including an actor which samples parameters randomly at each episode reset, which corresponds to domainrandomization.
It is also easy to deﬁne custom intervention actors, we hope that this facilitates investigation into optimal learning curricula (see Fig. 3).
Probing the Causal Structure in RL: The problem setting in RL is usually formulated using the language of Markov Decision Processes (MDPs) or Partially Observable Markov Decision Processes (POMDPs) (Sutton & Barto, 1999), but can be also represented by Structural Causal Models (SCMs), as shown in (Buesing et al., 2018), refer to section E in the Appendix for a detailed explanation.
This is achieved by formulating all conditional probability distributions as deterministic functions that take independent noise variables as inputs.
These independent noise variables can specify different scenarios while the deterministic functions reﬂect the causal mechanisms of the system (Sch¨olkopf et al., 2021).
Changes in the environment can stem from two different sources:
The agent may alter the state of the environment (e g. the position of a block) indirectly, through its actions (e g. pushing the block by applying appropriate torques at the motors).
During the execution of a learning curriculum or an evaluation protocol, we may directly intervene on any variable of the SCM, including all the latent variables of the causal model that are not accessible to the RL agent (such as gravity, object mass or color).
(1) is the default type of admissible interventions in RL benchmarks, whereas CausalWorld allows for interventions of type (2) in addition.
The idea is that interventions on these latent variables, e.g. during a learning curriculum, will allow the agent to distinguish between spurious correlations that are only present in a particular setting and true causal relations that will hold across all settings (i.e. interventions).
If the agent is able to learn such a representation of the underlying SCM structure, we would expect it to perform well even in out-of-distribution scenarios (Sch ¨ olkopf et al., 2021;
Dittadi et al., 2021) because the causal structure remains the same, even when the functional form of certain relations may vary (e.g. when transferring to the real robot).
Moreover, we hope that by having access to a broad range of interventions in CausalWorld it will aid the inference of the underlying SCM structure through the different causal discovery methods (see Figure 4 for a subset of the expected SCM to be learned), which in turn addresses the lack of causal discovery benchmarks for real world challenges. [Figure]
[Figure]
R ELATED W ORK
Previous benchmarks for RL mostly focused on the single task learning setting such as OpenAI Gym and DM control suite (Tassa et al., 2018; Brockman et al., 2016).
In contrast, a recent line of work, e.g. Meta-World and RLBench (Yu et al., 2019; James et al., 2020), aims at studying multi-task learning as well as meta-learning.
Such benchmarks mostly provide non-parametric, hand-designed task variations, it is hence unclear how much structure is shared between them.
For instance, it is not clear how different it is to “open a door” compared to “opening a drawer”.
To address the ambiguity in the shared structure between the tasks, CausalWorld was designed to allow interventions to be performed on many environment variables giving rise to a large space of tasks with well-deﬁned relations between them, which we believe is a missing key component to address generalization in
RL.
A detailed comparison between CausalWorld and similar benchmarks is shown in Table 1.
Similar parametric formulations of different environments were used in the generalization-for-RL literature, which has played an important role in advancing the ﬁeld (Packer et al., 2018; Rajeswaran et al., 2017; Pinto et al., 2017; Yu et al., 2017; Henderson et al., 2017a; Dulac-Arnold et al., 2020;
Chevalier-Boisvert et al., 2018).
In these previous works, variables were mostly changed randomly, as opposed to the full control over the variables provided by CausalWorld .
Another important open problem for the RL community is the standardization of the reported learning curves and results.
RL methods have been shown to be sensitive to a range of different factors (Henderson et al., 2017b).
Thus it is crucial to devise a set of metrics that measure reliability of
RL algorithms and ensure their reproducibility.
Chan et al.
distinguishes between several evaluation modes like ”evaluation during training” and ”evaluation after learning”.
Osband et al.
(2019) recently proposed a benchmarking suite that disentangles the ability of an algorithm to deal with different types of challenges.
Its main components are: enforcing a speciﬁc methodology for an agent’s evaluation beyond the environment deﬁnition and isolating core capabilities with targeted ’unit tests’ rather than integrating the general learning ability.
Moreover, causality has been historically studied from the perspective of probabilistic and causal reasoning (Pearl, 2009), cognitive psychology (Grifﬁths & Tenenbaum, 2005), and more recently in the context of machine learning (Goyal et al., 2019b; Sch¨olkopf et al., 2021; Baradel et al., 2019;
Bakhtin et al., 2019).
On the contrary, we believe its link to robotics is not yet drawn systematically.
To bridge this gap, one of the main motivations of CausalWorld was to facilitate research in causal learning for robotics, such as observational discovery of causal effects in physical reality, counterfactual reasoning, and causal structure learning.
E XPERIMENTS
To illustrate the usage of this benchmark and to verify the feasibility of some basic tasks, we evaluate current state-of-the-art model-free (MF-RL) algorithms on a subset of the goal shape families described in Section 2 and depicted in Fig.
2: (a) Pushing, (b) Picking, (c) Pick and Place, and (d)
Stacking2.
These goal shapes reﬂect basic skills that are required to solve more complex construction tasks.
Setup: The idea here is to investigate how well an agent will perform on different evaluation distributions, depending on the curriculum it has been trained with.
We train each method under the following curricula:
• Curriculum 0: no environment changes; each episode is initialized from the default task lying in space A - note that here the initial state never changes (i.e. no interventions).
• Curriculum 1: goal shape randomization; at the beginning of each episode a new goal shape is sampled from space A (i.e. interventions on goal position and orientation).
• Curriculum 2: full randomization w.r
t. the task variables 5 ; every episode a simultaneous intervention on all variables is sampled from space A (i
e. can be seen as equivalent to extreme domain randomization in one space).
The curriculum will, as expected, affect the generalization capabilities of the trained agents.
With
CausalWorld ’s formulation, these generalization capabilities can easily be disentangled and benchmarked quantitatively, as explained in Section 2.
For each of the goal shape families (a, b, c, d from Fig. 2), we train agents under the three described curricula using the following MF-RL algorithms: The original Proximal Policy Optimization (PPO) from Schulman et al.
, Soft
Actor-Critic (SAC) from Haarnoja et al.
and the Twin Delayed DDPG (TD3) from Fujimoto et al.
We provided these methods with a hand-designed dense reward function as we did not observe any success with the sparse reward only.
Each of the mentioned setups is trained for ﬁve different random seeds, resulting in 180 trained agents.
Pushing
Picking
Pick and Place
Stacking 0 .0 .1 .f
r a c t
i
o
n a l s u cce ss 1010time steps 0 .0 .
.1010time
steps 1010time steps 1010time steps c u
rr i c
u l u mc u
rr i c
u l u mc u
rr i c
u l u m
PPO
SAC
TD
Training model-free RL methods: We report the training curves averaged over the random seeds in Fig.
As can be seen from these fractional success training curves, MF-RL methods are capable of solving the single block goal shapes (pushing, picking, pick and place) seen during training time given enough experience.
However, we observe that none of the methods studied here managed to solve stacking two blocks.
The score below 0.5 indicates that it only learns to push the lower cube into the goal shape.
This shows that multi-object target shapes can become nontrivial quickly and that there is a need for methods making use of the modular structure of object-based environments.
To no surprise, the training curriculum has a major effect on learning.
For example, methods rarely manage to pick up any signiﬁcant success signal under extreme domain randomization as in curriculum 2, even after 100 million timesteps.
Note that these curves represent the scores under the conditions of the training environments.
Next, we will discuss shared evaluation protocols that allow to benchmark and compare agents trained under different conditions.
Benchmarking generalization capabilities along various axes: For each of the four goal shape families, we deﬁne a set of 12 evaluation protocols that we consider meaningful and representative for benchmarking the different algorithms.
In the protocols presented here, we sample the values from a protocol-speciﬁc set of variables at the start of each episode while keeping all other variables ﬁxed to their default values.
We evaluate each agent on 200 episodes by computing the fractional success score at the last time step of each episode and reporting the mean.
These evaluation protocols allow to disentangle generalization abilities, as they show robustness with respect to different types of interventions, see Fig.
The following are some of the observations we made for pushing
: • Agents that were trained on the default pushing task environment (curriculum 0) do well (as expected) on the default task (P0).
Interestingly, we likewise see a generalization capability to initial poses from variable space
A (P4).
This can be explained by a substantial exploration of the block positions via manipulation during training.
In contrast, we see that the agents exhibit weaknesses regarding goal poses (P5), they seem to overﬁt to their training settings in this case.
P P
P P
P P
P P
P P
P
P11 spaceA
B
A
A A
B
A
B B
A
B variablesbm bm bs bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step            0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO(0) PPO(1)
PPO(2)
SAC(0) SAC(1)
SAC(2)
TD3(0) TD3(1)
TD3(2)
• For agents trained with goal pose randomization (curriculum 1) we see similar results as with curriculum 0, with the difference that agents under this curriculum generalize robustly to different
as
• Finally, agents that experience extreme domain randomization (curriculum 2) at training time, fail to learn any relevant skill as shown by the ﬂat training curve in Fig.
An explanation for this behavior could be that the agent might need more data and optimization steps to handle this much more challenging setting.
Another possibility is that it may simply not be possible to ﬁnd a strategy which simultaneously works for all parameters (note that the agent does not have access to the randomized parameters and hence must be robust to them).
This poses an interesting question for future work.
As expected, we observe that an agent’s generalization capabilities are related to the experience gathered under its training curriculum.
CausalWorld allows us to explore this relationship in a differentiated manner, assessing which curricula lead to which generalization abilities.
This will not only help uncover an agent’s shortcomings but may also aid in investigating novel learning curricula and approaches for robustness in RL.
Lastly, we note that this benchmark comprises extremely challenging tasks that appear to be out of reach of current model free methods without any additional inductive bias.
C ONCLUSION
We have introduced a new benchmark - CausalWorld - to facilitate research in causal structure and transfer learning using a simulated environment of an open-source robot, where learned skills could potentially be transferred to the real world.
We showed how allowing for interventions on the environment’s properties yields a diverse familiy of tasks with a natural way of deﬁning learning curricula and evaluation protocols that can disentangle different generalization capabilities.
A natural extension of our work is to develop new RL algorithms that focuses on out of distribution generalization (whether its a different subspace of the state space or a completely different task).We hope that the ﬂexibility and modularity of CausalWorld will allow researchers to easily deﬁne appropriate benchmarks of increasing difﬁculty as the ﬁeld progresses, thereby coordinating research efforts towards ever new goals.
A CKNOWLEDGMENTS
The authors would like to thank Felix Widmaier, Vaibhav Agrawal and Shruti Joshi for the useful discussions and for the development of the TriFinger robot’s simulator (Joshi et al., 2020), which served as a starting point for the work presented in this paper.
AG is also grateful to Alex Lamb and
Rosemary Nan Ke for useful discussions.
We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting FT.
A PPENDIX A O BSERVATIONS
Observations in CausalWorld has two modes, ”structured” and ”pixel”.
When using ”pixel” mode, 6 images are returned consisting of the current images rendered from 3 different views on top of the TriFinger platform, showing the current state of the environment, as well as the 3 equivalent goal images rendered from the same points of view, showing the goal shape that the robot have to build by the end of the episode.
[Figure] [Figure]
Scene
[Figure]
OR
O b j ec t F ea t u r e s
P a r t i a l G o a l
F e a t u r e s
B T RI F INGER P LATFORM
The robot from (W ¨ uthrich et al., 2020) shown in ﬁgure 9 is open-sourced and can be reproduced and built in any research lab; since its inexpensive (about $5000), speeding up sim2real research.
C T ASK G ENERATORS
Pushing : task where the goal is to push one block towards a goal position with a speciﬁc orientation; restricted to goals on the ﬂoor level.
Picking : task where the goal is to pick one block towards a goal height above the center of the arena; restricted to goals above the ﬂoor level.
Pick And Place : task where the arena is divided by a ﬁxed long block and the goal is to pick one block from one side of the arena to a goal position with a variable orientation on the other side of the ﬁxed block.
Stacking2 : task where the goal is to stack two blocks above each other in a speciﬁc goal position and orientation.
Towers : task where the goal is to stack multiple n blocks above each other in a speciﬁc goal position and orientation - exactly above each other creating a tower of blocks.
Stacked Blocks : task where the goal is to stack multiple n blocks above each other in an arbitrary way to create a stable structure.
The blocks don’t have to be exactly above each other; making it more challenging than the ordinary towers task since the its harder to come up with a stable structure that covers the goal shape volume.
Creative Stacked Blocks : exactly the same as the Stacked Blocks task except that the ﬁrst and last levels of the goal are the only levels shown or ”imposed” and the rest of the structure is not explicitly speciﬁed, leaving the rest of the goal shape to the imagination of the agent itself; this is considered the most challenging since its it needs the agent to understand how to build stable structures and imagine what can be ﬁlled in the middle to connect the two levels in a stable way. 8.
General : the goal shape is an arbitrary shape created by initially dropping an arbitrary number of blocks from above the ground and waiting till all blocks come to a rest position where this becomes the goal shape that the agent needs to ﬁll up afterwards.
[Table] [Table]
[Table]
D T RAINING D ETAILS
The experiments were carried out using the stable baselines implementation of PPO, SAC and TD3.
We used a 2 layer MLP Policy [256,256] for all the policies.
PPO was trained onworkers up to 100 million timesteps in parallel and SAC as well as TD3 were trained serially formillion timesteps.
[Table]
entropy coef
[Table]
value function
[Table] [Table]
gradient clipping
[Table]
max)
[Table]
n minibatches per update
[Table]
n
[Table]
P P
P P
P P
P P
P P
P
P11 spaceA
B
A
A A
B
A
B B
A
B variablesbm bm bs bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step            0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO
[Table] [Table]
PPO(
[Table]
SAC
[Table]
[Table] [Table]
SAC(
[Table] [Table]
[Table] [Table]
P P
P P
P P
P P
P P
P
P11 spaceA
B
A
A A
B
A
B B
A
B variablesbm bm bs bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step            0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO(
[Table] [Table]
[Table] [Table]
SAC
[Table]
P
P
P P P
P P
P P
P P
P11 spaceA
B
A
A A
B
A
B B
A
B variablesbm bm bs bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step            0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO(0) PPO(1)
PPO(2)
SAC(0) SAC(1)
SAC(2)
TD3(0) TD3(1)
TD3(2)
P
P P
P P
P P
P P
P
P10 spaceA
B
A A
B
A
B B
A
B variablesbm bm bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step           0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO(0) PPO(1)
PPO(2)
SAC(0) SAC(1)
SAC(2)
TD3(0) TD3(1)
TD3(2)
E C AUSALITY IN R EINFORCEMENT L EARNING
Deﬁnition [-
-]1: A partially observable Markov decision processes (POMDP) is deﬁned by the tuple ( S, A, T, R, Ω , O, γ, ρ 0 , H ) with states s ∈ S , actions a ∈ A and observations o ∈ Ω determined by the state and action of the environment O
( o | s, a ) .
T ( s t +1 | s t , a t ) is the transition probability distribution function, R ( s t , a t ) is the reward function, γ is the discount factor, ρ 0 ( s ) is the initial state distribution at the beginning of each episode, and H is the time horizon per episode.
The objective of RL algorithms is to learn a policy π ( a t | h t ) with history H t
= ( O 1 , A 1 , ..., A t − 1 , O t ) that maximizes the discounted expected reward J ( π ) =
E π (cid:104)(cid:80)
Ht
=
γ t r t (cid:105) .
Deﬁnition [-
-]2: A structural causal model (SCM) M over X =
( X 1 , ..., X N ) is given by a DAG
G over nodes X , independent noise RVs U =
( U 1 , ..., U N ) with distributions P U i and functions f 1 , ..., f N such that X i =
f i ( pa i , U i ) , where pa i ⊂
X are the parents of X i in G .
An SCM entails a distribution P with density p over ( X, U ) .
Deﬁnition [-
-]3: An intervention I in an SCM M consists of replacing the RHS f i ( pa i , U i ) by f
I i ( pa I i , U i ) with i ⊆ { 1 , ..., N } where pa
I
i are the parents in a new DAG G I .
The resulting SCM is denoted with M do ( I ) with distribution P do ( I ) and density p do ( I ) .
Representing POMDPs as SCMs: A POMDP can be presented as an SCM M by unrolling the POMDP through time by expressing all conditional distributions, e.g the transition kernel P S t +1 | S t ,A t , as deterministic functions (which reﬂects the causal mechanisms of the system) with independent noise variables U , such as S t +1 = f st ( S t , A t , U st ) . This is always possible using autoregressive formalization as shown in (Buesing et al., 2018).
The distribution P π over trajectories T determined by the SCM, which means running a different policy µ (instead of π ) in the environment is an intervention by itself I ( π → µ ) resulting in model distribution P do ( I ( π → µ )) . Given the SCM, one can reason about the alternative outcomes of different actions or even different environment properties (such as friction) (known as counterfactual inference) which is not possible if we only model the system as a conditional distribution P O | A,U c .
Generalizing agents: An agent should generalize well to changes in the environment if it achieves an equal performance after an intervention I is performed on the SCM M resulting in M do ( I ) .
This can be accomplished for instance as explained in (Sch ¨ olkopf et al., 2021), by reusing the learned causal mechanisms.
For example, if an intervention was performed on the colour of one of the available blocks in the environment, it won’t change the underlying causal mechanisms.
Thus, this sort of interventions on a non causal variable shouldn’t affect the resulting actions of the agent.
Nevertheless, this is often not the case with RL agents trained using visual observations.
(Alver & Precup, 2020) showed recently that algorithms which were designed speciﬁcally for Meta-RL can display strong overﬁtting when they are evaluated on challenging visual tasks.
On the other hand, if an intervention was performed on a causal variable with respect to a policy π , such as the mass of an object, the resulting action sequence will be expected to change for robust control.
Additionally, if an intervention is performed on some of the goal variables, the reward function might also change and a robust controller should also be able to react to it accordingly.