C AUSAL W ORLD : A R OBOTIC M ANIPULATION B ENCHMARK FOR C AUSAL S TRUCTURE AND T RANSFER L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite recent successes of reinforcement learning (RL), it remains a challenge for agents to transfer learned skills to related environments.
To facilitate research addressing this, we propose CausalWorld , a benchmark for causal structure and transfer learning in a robotic manipulation environment.
The environment is a simulation of an open-source robotic platform, hence offering the possibility of sim-to-real transfer.
Tasks consist of constructing 3D shapes from a given set of blocks - inspired by how children learn to build complex structures.
The key strength of CausalWorld is that it provides a combinatorial family of such tasks with common causal structure and underlying factors (including, e.g., robot and object masses, colors, sizes).
The user (or the agent) may intervene on all causal variables, which allows for ﬁne-grained control over how similar different tasks (or task distributions) are.
One can thus easily deﬁne training and evaluation distributions of a desired difﬁculty level, targeting a speciﬁc form of generalization (e.g., only changes in appearance or object mass).
Further, this common parametrization facilitates deﬁning curricula by interpolating between an initial and a target task.
While users may deﬁne their own task distributions, we present eight meaningful distributions as concrete benchmarks, ranging from simple to very challenging, all of which require long-horizon planning and precise low-level motor control at the same time.
Finally, we provide baseline results for a subset of these tasks on distinct training curricula and corresponding evaluation protocols, verifying the feasibility of the tasks in this benchmark.
I NTRODUCTION
[Figure]
Benchmarks have played a crucial role in advancing entire research ﬁelds, for instance computer vision with the introduction of CIFAR10 and ImageNet (Krizhevsky et al., 2009; 2012).
When it comes to the ﬁeld of reinforcement learning (RL), similar breakthroughs have been achieved in domains such as game playing (Mnih et al., 2013;
Silver et al., 2017), learning motor control for high-dimensional simulated robots (Akkaya et al., 2019), multi-agent settings (Baker et al., 2019; Berner et al., 2019) and for studying transfer in the context of meta-learning (Yu et al., 2019).
Nevertheless, trained agents often fail to transfer the knowledge about the learned skills from a training environment to a different but related environment sharing part of the underlying task structure.
This can be attributed to the fact that it is quite common to evaluate an agent on the training environments themselves, which leads to overﬁtting on these narrowly deﬁned environments (Whiteson et al., 2011), or that algorithms are compared using highly engineered and biased reward functions which may result into learning suboptimal policies with respect to the desired behaviour; this is particularly evident in robotics.
[Figure]
In existing benchmarks (Yu et al., 2019; Goyal et al., 2019; Cobbe et al., 2018; Bellemare et al., 2013; James et al., 2020)
the amount of shared causal structure between the different environments is mostly unknown, e.g. in the Atari Arcade Learning environments, it is unclear how to quantify the underlying similarities between different Atari games and we generally do not know to which degree an agent can be expected to generalize.
To overcome these limitations, there is a need for a benchmark with a diverse set of environments which, in contrast to previous designs, should share a large set of parameters and parts of the causal structure.
Being able to intervene on these parameters (individually or collectively) allows the experimenter to evaluate agents’ generalization ability with respect to different types and extents of changes in the environment.
These parameters can be varied gradually, which yields a continuum of similar environments that allows for ﬁne-grained control of training and test distributions.
The objective of this work is to address these points by introducing a novel benchmark in a robotic manipulation environment which we call CausalWorld .
A remarkable skill that humans learn to master relatively early on in their life is building complex structures using their spatial-reasoning and dexterous manipulation abilities (Casey et al., 2008;
Caldera et al., 1999; Kamii et al., 2004).
Playing with toy blocks constitutes a natural environment for children to develop important critical and visual-spatial skills, helping them ‘generalize’ in building complex composition designs from presented or imagined goal structures (Verdine et al., 2017;
Nath & Sz ¨ ucs, 2014; Dewar, 2018; Richardson et al., 2014).
Inspired by this, CausalWorld is designed to aid in learning and investigating these skills in a corresponding simulated robotics manipulation environment of the open-source TriFinger robot platform from W¨uthrich et al.
(2020), which can be built in the real world.
Tasks are formulated as building 3D goal shapes using a set of available blocks by manipulating them - as seen in Fig.
This task deﬁnition yields a combinatorially large and diverse space of target structures.
We subsequently provide eight distinct task generators which generate different environments covering distinct target shape families of interest.
The resulting environment instances range from simple to extremely challenging.
Additionally, by using a common goal formulation, we can deﬁne a success metric which is shared across all environments: We measure the volumetric overlap of the objects with the goal shape.
CausalWorld improves upon previous benchmarks by exposing a large set of parameters in the causal generative model of the environments, such as properties of blocks, goals, robot links and others like gravity.
The possibility to intervene on any of these environment properties at any point in time allows one to set up training environments in a curriculum manner and evaluate agents across different generalization axes using a broad set of evaluation protocols.
Furthermore, emphasizing the real-world relevance of this benchmark as opposed to earlier ones (Chevalier-Boisvert et al., 2018; Cobbe et al., 2018), researchers may build their own real-world platform of this simulator at low cost, as detailed in W¨uthrich et al.
(2020), and transfer their trained policies to the real world.
[Table]
Finally, by releasing this benchmark we hope to facilitate research in causal structure learning, i.e. learning the causal graph or certain aspects of it, as we operate in a complex real-world environment whose dynamics follow the laws of physics which induce causal relations between the variables.
Changes to the variables we expose can be considered do-interventions on the underlying structural causal model (SCM).
Consequently, we believe that this benchmark offers an exciting opportunity to investigate causality and its connection to RL and robotics.
Our main contributions can be summarized as follows:
• We propose CausalWorld , a new benchmark comprising a parametrized family of robotic manipulation environments for advancing out-of-distribution generalization and causal structure learning in RL.
• We provide a systematic way of deﬁning curricula and disentangling generalization abilities of trained agents by allowing do-interventions to be performed on all environment variables (parameters and states).
• We establish baseline results for some of the available tasks under different learning algorithms, thus verifying the feasibility of the tasks.
• We show how different learning curricula affect generalization across different axes by reporting some of the in-distribution and out-of-distribution
generalization capabilities of the trained agents.
C AUSAL W ORLD B ENCHMARK
Here, we make the desiderata outlined in the introduction more precise:
The set of environments should be sufﬁciently diverse to design challenging transfer tasks.
We need to be able to intervene on different properties (e.g. masses, colors) individually, such that we can investigate different types of generalization.
It should be possible to convert any environment to any other environment by gradually changing its properties through interventions; this requirement is important for evaluating different levels of transfer and for deﬁning curricula.
The environments should share some causal structure to allow algorithms to transfer the learned causal knowledge from one environment to another.
There should be a uniﬁed measure of success, such that an objective comparison can be made between different learning algorithms.
The benchmark should make it easy for users to deﬁne meaningful distributions of environments for training and evaluation.
In particular, it should facilitate evaluation of indistribution and out-of-distribution performance.
The simulated benchmark should have real-world counterpart to allow for sim2real.
In light of these desiderata, we propose a setup in which a robot must build goal shapes using a set of available objects.
It is worth noting that similar setups were proposed previously in a less realistic setting as in (Janner et al., 2018; Bapst et al., 2019; McCarthy et al.
; Akkaya et al., 2019;
Fahlman, 1974; Winston, 1970; Winograd, 1972).
Speciﬁcally, a task is formulated as follows: given a set of available objects with different properties (such as size, mass, friction, etc.)
the agent needs to build a speciﬁc goal structure , see Fig.
1 for an example.
The vast amount of possible target shapes and environment properties makes this a diverse and challenging setting to evaluate different generalization aspects.
CausalWorld is a simulated version (using the Bullet physics engine (Coumans et al., 2013)) of the open-source TriFinger robot platform from W¨uthrich et al.
Each environment is completely deﬁned by a set of variables such as gravity, ﬂoor friction, stage color, ﬂoor color, joint positions, various block parameters (e.g. size, color, mass, position, orientation), link colors, link masses and the goal shape.
See Table 3 in the Appendix for a subset of these variables.
Desideratum 1 is satisﬁed since different environment properties and goal shapes give rise to very different tasks, ranging from relatively easy (e.g. re-positioning a single cube) to extremely hard (e.g. building a complex multi-level goal shape).
Desideratum 2 is satisﬁed because we allow for arbitrary interventions on these properties, hence users or agents may change parameters individually or jointly.
Desideratum 3 is satisﬁed because the parameters can be changed gradually.
Desideratum 4 is satisﬁed because all the environments share the causal structure of the robot, and one may also use subsets of environments which share even more causal structure.
We satisfy desideratum 5 by deﬁning the measure of success for all environments as the volumetric overlap of the goal shape with available objects.
Further, by splitting the set of parameters into a set A, intended for training and in-distribution evaluation, and a set B, intended for out-of-distribution evaluation, we satisfy desideratum 6.
Finally, since the TriFinger robot (W ¨ uthrich et al., 2020) can be built in the realworld, we satisfy desideratum 7.
Desideratum 7 and 2 are in partial conﬂict since sim2real is only possible for the tasks which are constrained to the variables on which the robot can physically act upon.
Task generators: The space of feasible goal shapes one can match with a given set of objects is combinatorially exponential.
However, only a small subset of these goal shapes comprise topologically meaningful shapes.
To cover collections spanning a family of similar goal shapes,
CausalWorld allows for deﬁning task generators which can procedurally generate a variety of different goal shapes in an environment.
Each task generator is initialized with a default goal shape from its corresponding family and comes with a sampler to sample new goal shapes from the same family.
Additionally, upon construction, one can specify the environments’ initial state and initial goal shape structure when deviating from the default.
The maximum episode time to build a given shape is number of blocks × 10 seconds.
Initially, CausalWorld contains eight task generators that could be used to sample goal shapes from different goal shape families (see Fig. 2).
• Three generators are constrained to a goal shape with a single block:
Pushing with the goal shape on the ﬂoor, Picking having the goal shape deﬁned above the ﬂoor and Pick and Place where a ﬁxed obstacle is placed between the initial block and goal pose.
Stacking2 involves a goal shape of two stacked blocks, which can also be considered one instance of the towers generator.
• The remaining generators use a variable number of blocks to generate much more complex and challenging target shapes with details in the appendix: Towers , Stacked Blocks , Creative Stacked Blocks and General .
Given that building new environments using current physics simulators is usually tedious, we provide a simple API for users who wish to create new task generators, for new challenging shape families which may be added to CausalWorld’s task generators repository.
Action and Observation Spaces: The robot’s action space A ∈ R 9 can be chosen to operate in either joint position control mode, joint torque control mode, end-effector position control mode, or the delta of each.
To address the different challenges in using high-dimensional visual observations as well as using a structured representation, we provide two observation modes: structured as well as pixel .
In the structured mode, the low-dimensional observation vector o follows a common rule for the ordering of the relevant variables, such as joints position, joints velocity, blocks linear velocity, time left for the task..etc.
Thus, the observation space size depends on the number of blocks, which could potentially change with every new goal sampled, e.g. in Towers, (Creative) Stacked
Blocks and General; therefore O ∈ R d o where d o varies across different environments.
On the contrary, in the pixel mode, the agent receives six different RGB images, where O ∈ R 6 × 3 × 128 × 128 , the ﬁrst three images are rendered from different cameras mounted around the TriFinger robot, and the last three images specify the goal image of the target shape rendered from the same cameras.
This mode can be mirrored on the real robotic platform and aids in investigating object-based learning approaches from pixel data as well as learning visual goal conditioned policies.
Additionally,
CausalWorld allows for setting up a fully customized observation space, if needed.
Rewards: The reward function r is deﬁned uniformly across all possible goal shapes as the fractional volumetric overlap of the blocks with the goal shape, which ranges between 0 (no overlap) and 1 (complete overlap).
This shared success metric can be returned at each time step where its scale is independent of the goal shape itself.
Thus, an agent that learned this shared reward function r from several different tasks could in principle use it to solve unseen goal structures.
There is also the possibility of modifying the reward function to 1) sparsify the reward further by returning a binary reward signal instead, or 2) add a dense reward function in order to introduce inductive biases via domain knowledge and solution guidance.
We hope that the considerable complexity and space of goal shapes motivate and accelerate the development of algorithms that are not dependent on highly tuned reward functions anymore by putting more focus on optimal learning curricula instead, which might help to overcome these limitations, and CausalWorld provides a convenient way for specifying them via intervention actors.
[Figure]
Training and evaluation spaces:
In this benchmark, a learning setting consists of an allowed training space (ATS) and an evaluation space (ES), both of which are subspaces of the full parameter space.
During training, in the simplest setting, parameters are sampled iid from the ATS.
However, unlike existing benchmarks, CausalWorld allows for curricula within the ATS as well as settings where the agent itself intervenes on the parameters within an episode (see Fig. 3).
Similarly, during evaluation, parameters may be sampled iid from the evaluation space at each episode reset, or there can be interventions within an episode.
Moreover, in order to retrieve the setting considered in most RL benchmarks, we could set the ATS and the ES to be identical and intervene only on object and robot states (and keep other environment properties constant) at each episode reset.
However, to evaluate out-of-distribution generalization, one should set the two spaces (ATS and ES) to be different; possibly even disjoint.
Additionally, to evaluate robustness with respect to a speciﬁc parameter (e.g. object mass), one may deﬁne the training and evaluation spaces to only differ in that particular parameter.
In order to facilitate the deﬁnition of appropriate training and evaluation settings, we pre-deﬁne two disjoint sets, A i and B i , for each parameter i .
Through this, one can for instance deﬁne the training space to be A 1 × A 2 × ... and the evaluation space to be B 1 × B 2 × ... to assess generalization with respect to all parameters simultaneously.
Alternatively, the evaluation space could be deﬁned as A 1 × A 2 × ...
× B i × A
i +1 × ... to assess generalization with respect to parameter i only.
Lastly, users may also deﬁne their own spaces which could then be integrated into the benchmark to give rise to new learning settings.
Intervention actors: To provide a convenient way of specifying learning curricula, we introduce intervention actors.
At each time step, such an actor takes all the exposed variables of the environment as inputs and may intervene on them.
To encourage modularity, one may combine multiple actors in a learning curriculum.
This actor is deﬁned by the episode number to start intervening, the episode number to stop intervening, the timestep within the episode it should intervene and the episode periodicity of interventions.
We provide a set of predeﬁned intervention actors, including an actor which samples parameters randomly at each episode reset, which corresponds to domainrandomization.
It is also easy to deﬁne custom intervention actors, we hope that this facilitates investigation into optimal learning curricula (see Fig. 3).
R ELATED W ORK
Previous benchmarks proposed for RL mostly focused on the single task learning setting such as
OpenAI Gym and DM control suite (Tassa et al., 2018; Brockman et al., 2016).
Although, a recent line of work, e.g. Meta-World and RLBench (Yu et al., 2019; James et al., 2020) aim at studying multi-task learning as well as meta-learning, respective benchmarks mostly exhibit nonparametric hand-designed task variations which makes it ambiguous and not explicit how much structure is shared between them.
For instance, it is not clear how different it is to “open a door” compared to “opening a drawer”.
To address the ambiguity in the shared structure between the tasks, CausalWorld was designed to allow interventions to be performed on many environment variables giving rise to a large space of tasks with well-deﬁned relations between them, which we believe is a missing key component to address generalization in RL.
Similar parametric formulations of different environments were used in experiments in the generalization for RL literature, which have played an important role in advancing the ﬁeld (Packer et al., 2018; Rajeswaran et al., 2017; Pinto et al., 2017; Yu et al., 2017; Henderson et al., 2017a;
DulacArnold et al., 2020; Chevalier-Boisvert et al., 2018).
In these previous works, variables were mostly assigned randomly as opposed to the full control over the variables in CausalWorld by allowing do-interventions.
Another important remaining challenge for the RL community is the standardization of the reported learning curves and results.
RL methods have been shown to be sensitive to a range of different factors (Henderson et al., 2017b).
Thus it is crucial to devise a set of metrics that measure reliability of RL algorithms and ensure their reproducibility.
Chan et al.
distinguishes between several evaluation modes like ”evaluation during training” and ”evaluation after learning”.
Osband et al.
(2019) recently proposed a benchmarking suite that disentangles the ability of an algorithm to deal with different types of challenges.
Its main components are: enforcing a speciﬁc methodology for an agent’s evaluation beyond the environment deﬁnition and isolating core capabilities with targeted ’unit tests’ rather than integrating the general learning ability.
E XPERIMENTS
To verify the feasibility of this benchmark, we evaluate current state-of-the-art model-free (MF-RL) algorithms on a subset of the goal shape families described in Section 2: (a) Pushing, (b) Picking, (c) Pick and Place, and (d) Stacking2.
We argue that these goal shapes reﬂect primitive skills that are required to solve far more complex target shape families.
Setup: One objective of this benchmark is to deviate from the widely adopted approach of training and evaluating agents in narrowly deﬁned environments.
To emphasize the effect of different training distributions, we set up three handcrafted training curricula:
• Curriculum 0: no environment changes; each episode is initialized from the default task lying in space A - note that here the initial state never changes (i
e. no interventions).
• Curriculum 1: goal shape randomization; at the beginning of each episode a new goal shape is sampled from space A (i.e. interventions on goal position and orientation). Pushing
Picking Pick and Place Stacking 0 .0 .1 .f r a c t
i
o
n a l s u cce {+ss 1010time steps 0 .0 .
+}
.1010time
steps 1010time steps 1010time steps c u
rr i c
u l u mc u
rr i c
u l u mc u
rr i c
u l u m
PPO
SAC
TD
• Curriculum 2: full randomization w.r.t.
the task variables 2 ; every episode a simultaneous intervention on all variables is sampled from space A (i
e. can be seen as equivalent to extreme domain randomization in one space).
The curriculum will affect the generalization capabilities of the trained agents.
With
CausalWorld ’s formulation, these generalization capabilities can easily be disentangled and benchmarked quantitatively, as explained in Section 2.
For each of the goal shape families (a, b, c, d from Fig. 2), we train agents under the three described curricula using the following MF-RL algorithms: The original Proximal Policy Optimization (PPO) from Schulman et al.
, Soft
Actor-Critic (SAC) from Haarnoja et al.
and the Twin Delayed DDPG (TD3) from Fujimoto et al.
Each task-generator was equipped further with a hand-designed dense reward function as we did not observe any success with the sparse reward only.
Each of the mentioned setups is trained for ﬁve different random seeds, resulting in 180 trained agents.
Challenging model-free RL methods: We report the training curves averaged over the random seeds in Fig.
As can be seen from these fractional success training curves, MF-RL methods are capable of solving the single block goal shapes seen during training time given enough experience.
However, we observe that none of the methods studied here managed to solve stacking two blocks.
The score below 0.5 indicates that it only learns to push the lower cube into the goal shape.
This emphasizes that multi-object target shapes can become nontrivial quickly and that there is a need for methods making use of the modular structure of object-based environments.
To no surprise, the training curriculum has a major effect on learning, but the interpretation of generalization capabilities becomes much more explicit in the following subsection.
For example, methods rarely manage to pick up any signiﬁcant success signal under full extreme domain randomization as in curriculum 2, even after 100 million timesteps.
Note that these curves represent the scores under the shapes and conditions of the actual training environments.
Therefore, we need the capability to setup different protocols, in other words standardized sets of evaluation environment, that allow to benchmark learned skills of different agents. P P P
P
P
P P P P
P
P
P11 spaceA
B
A
A
A
B
A
B
B
A
B variablesbm bm bs bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step            0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO(0) PPO(1)
PPO(2)
SAC(0) SAC(1)
SAC(2)
TD3(0) TD3(1)
TD3(2)
[Figure]
Curriculum
InterventionVariable(s)
AllowedTrainingSpace nonegoal pose
A
[Figure]
all
A
[Figure]
InterventionVariable(s)
EvaluationSpace
[Figure]
none -
[Figure]
A
[Figure]
A
[Figure]
A
[Figure]
goal pose A [Figure] A
[Figure] A
P all
B Benchmarking generalization capabilities along various axes: For each of the four goal shape families, we deﬁne a set of 12 evaluation protocols that we consider meaningful and representative for benchmarking the different algorithms. In the protocols presented here, we sample the values from a protocol-speciﬁc set of variables at the start of each episode while keeping all other variables ﬁxed to their default values.
After evaluating an agent on 200 episodes, we compute the fractional success score at the last time step of each episode and report the mean.
These evaluation protocols allow to disentangle generalization abilities, as they show robustness with respect to different types of interventions Fig.
5, The following are some of the observations we made for pushing:
• Agents that were trained on the default pushing task environment (curriculum 0) do well (as expected) on the default task (P0).
Interestingly, we likewise see a generalization capability to initial poses from variable space
A (P4).
This can be explained by a substantial exploration of the block positions via manipulation during training.
Similarly, we see that the agents exhibit weaknesses regarding goal poses (P5) but overﬁt on their training settings instead.
• For agents trained with goal pose randomization (curriculum 1) we see similar results as with curriculum 0, with the difference that agents under this curriculum generalize robustly to different goal poses (P5).
This is because it saw many goal poses in space A. • Finally, agents that experience extreme domain randomization (curriculum 2) at training time, fail to learn any relevant skill as shown by the ﬂat training curve.
A possible explanation for this behavior is that the agent might need more data and optimization steps to handle this much more challenging setting.
An- other possible explanation is that it may simply not be possible to ﬁnd a strategy which simultaneously works for all parameters (note that the agent does not have access to the randomized parameters and hence must be robust to them).
This poses an interesting question for future work.
Given the above experiments, we can see that an agent’s generalization capabilities are related to the experience gathered under its training curriculum.
Our evaluation protocols allow us to explore this relationship in a differentiated manner, assessing which curricula lead to which generalization abilities.
This will not only help uncover an agent’s shortcomings but will likewise aid in investigating novel approaches for robustness in RL.
Lastly, we want to note that this benchmark comprises extremely challenging tasks that appear to be out of reach for model free methods without any additional inductive bias.
C ONCLUSION
We have introduced a new benchmark - CausalWorld - to accelerate research in causal structure and transfer learning using a simulated environment of an open source robot, where skills learned could potentially be transferred to the real world.
We showed how enabling do-interventions on the environment’s properties yields a natural way of deﬁning learning curricula and evaluation protocols that can disentangle different generalization capabilities.
We showed how different training curricula can affect generalization capabilities with respect to different environment properties.
Finally, we hope that the modular and ﬂexible formulation of the environments in CausalWorld will beneﬁt research into a wide range of topics related to transfer in reinforcement learning and causal structure learning.
A PPENDIX A O BSERVATIONS
Observations in CausalWorld has two modes, ”structured” and ”pixel”.
When using ”pixel” mode, 6 images are returned consisting of the current images rendered from 3 different views on top of the TriFinger platform, showing the current state of the environment, as well as the 3 equivalent goal images rendered from the same points of view, showing the goal shape that the robot have to build by the end of the episode.
[Figure] [Figure]
Scene features
OR
O b j ec t F ea t u r e s
P a r t i a l G o a l
F e a t u r e s
B T RI F INGER P LATFORM
The robot from (W ¨ uthrich et al., 2020) shown in ﬁgure 9 is open-sourced and can be reproduced and built in any research lab; since its inexpensive (about $5000), speeding up sim2real research.
C T ASK G ENERATORS
Pushing : task where the goal is to push one block towards a goal position with a speciﬁc orientation; restricted to goals on the ﬂoor level.
Picking : task where the goal is to pick one block towards a goal height above the center of the arena; restricted to goals above the ﬂoor level.
Pick And Place : task where the arena is divided by a ﬁxed long block and the goal is to pick one block from one side of the arena to a goal position with a variable orientation on the other side of the ﬁxed block.
Stacking2 : task where the goal is to stack two blocks above each other in a speciﬁc goal position and orientation.
Towers : task where the goal is to stack multiple n blocks above each other in a speciﬁc goal position and orientation - exactly above each other creating a tower of blocks.
Stacked Blocks : task where the goal is to stack multiple n blocks above each other in an arbitrary way to create a stable structure.
The blocks don’t have to be exactly above each other; making it more challenging than the ordinary towers task since the its harder to come up with a stable structure that covers the goal shape volume.
Creative Stacked Blocks : exactly the same as the Stacked Blocks task except that the ﬁrst and last levels of the goal are the only levels shown or ”imposed” and the rest of the structure is not explicitly speciﬁed, leaving the rest of the goal shape to the imagination of the agent itself; this is considered the most challenging since its it needs the agent to understand how to build stable structures and imagine what can be ﬁlled in the middle to connect the two levels in a stable way. 8.
General : the goal shape is an arbitrary shape created by initially dropping an arbitrary number of blocks from above the ground and waiting till all blocks come to a rest position where this becomes the goal shape that the agent needs to ﬁll up afterwards. [Table] [Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
D T RAINING D ETAILS
The experiments were carried out using the stable baselines implementation of PPO, SAC and TD3.
We used a 2 layer MLP Policy [256,256] for all the policies.
PPO was trained onworkers up to 100 million timesteps in parallel and SAC as well as TD3 were trained serially formillion timesteps.
[Table]
entropy coef
[Table]
value function coef.
[Table]
gradient clipping
[Table]
max)
[Table]
n minibatches per update
[Table]
n
[Table]
epochs
[Table]
P P
P P
P P
P P
P P
P
P11 spaceA
B
A A
A
B
A
B B
A
B variablesbm bm bs bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step            0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO
[Table] [Table]
P P
P P
P P
P P
P P
P
P11 spaceA
B
A A
A
B
A
B B
A
B variablesbm bm bs bp gp bp, gp bp,
[Table]
[Table]
bp,
[Table] [Table]
[Table] [Table]
, ﬀ all all time step            0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO
[Table]
P P
P P
P P
P P
P P
P
P11 spaceA
B
A A
A
B
A
B B
A
B variablesbm bm bs bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step            0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO(0) PPO(1)
PPO(2)
SAC(0) SAC(1)
SAC(2)
TD3(0) TD3(1)
TD3(2)
P
P P
P P
P P
P P
P
P10 spaceA
B
A A
B
A
B B
A
B variablesbm bm bp gp bp, gp bp, gp,bm bp, gp,bm bp, gp, bm, ﬀ all all time step           0 .0 .0 .0 .1 .f
r a c t
i
o
n a l s u cce ss
PPO(0) PPO(1)
PPO(2)
SAC(0) SAC(1)
SAC(2)
TD3(0) TD3(1)
TD3(2)