CLUTR: C URRICULUM L EARNING VIA U NSUPERVISED T ASK R EPRESENTATION L EARNING
A BSTRACT
Reinforcement Learning (RL) algorithms are often known for sample inefﬁciency and difﬁcult generalization.
Recently, Unsupervised Environment Design (UED) emerged as a new paradigm for zero-shot generalization by simultaneously learning a task distribution and agent policies on the sampled tasks.
This is a nonstationary process where the task distribution evolves along with agent policies; creating an instability over time.
While past works demonstrated the potential of such approaches, sampling effectively from the task space remains an open challenge, bottlenecking these approaches.
To this end, we introduce CLUTR: a novel curriculum learning algorithm that decouples task representation and curriculum learning into a two-stage optimization.
It ﬁrst trains a recurrent variational autoencoder on randomly generated tasks to learn a latent task manifold.
Next, a teacher agent creates a curriculum by maximizing a minimax REGRET-based objective on a set of latent tasks sampled from this manifold.
By keeping the task manifold ﬁxed, we show that CLUTR successfully overcomes the non-stationarity problem and improves stability.
Our experimental results show CLUTR outperforms PAIRED, a principled and popular UED method, in terms of generalization and sample efﬁciency in the challenging CarRacing and navigation environments: showing an 18x improvement on the F1 CarRacing benchmark.
CLUTR also performs comparably to the non-UED state-of-the-art for CarRacing, outperforming it in nine of the 20 tracks.
CLUTR also achieves a 33% higher solved rate than
PAIRED on a set of 18 out-of-distribution navigation tasks.
I NTRODUCTION
Deep Reinforcement Learning (RL) has shown exciting progress in the past decade solving many challenging domains including Atari (Mnih et al.
, Dota (Berner et al.
, Go (Silver et al. (2016)).
However, deep RL is sample-inefﬁcient.
Moreover, out-of-box deep RL agents are often brittle: performing poorly on tasks that they have not encountered during training, or often failing to solve them altogether even with the slightest change ( Cobbe et al.
(2019), Azad et al.
(2022), Zhang et al.
Curriculum Learning (CL) algorithms showed promise to improve (Portelas et al.
(2020), Narvekar et al.
RL sample efﬁciency by employing a teacher algorithm that attempts to train the agents on tasks falling at the boundary of their capabilities, i.e., tasks that are slightly harder than the agents can currently solve.
Recently, a class of unsupervised
CL algorithms, called Unsupervised Environment Design (UED) [Dennis et al.
(2020),Jiang et al.
(2021a)], has shown impressive generalization capabilities which require no training tasks as input.
UEDs automatically generate tasks by sampling from the free parameters of the environment (e.g., the start, goal, and obstacle locations for a navigation task) and attempt to improve sample efﬁciency and generalization by adapting a diverse task distribution at the agent’s frontier of capabilities.
Protagonist Antagonist Induced Regret Environment Design (PAIRED) (Dennis et al.
(2020)) is one of the most principled UED algorithms.
The PAIRED teacher is itself an RL agent with actions denoting different task parameters.
PAIRED aims at generating tasks that maximize the agent’s regret, deﬁned as the performance gap between an optimal policy and the student agent.
Theoretically, upon convergence, the agent learns to minimize the regret, i.e., will solve every solvable task.
Such a robustness guarantee makes regret-based teachers well suited for training robust agents.
Despite the strong robustness guarantee, PAIRED is still sample inefﬁcient in practice.
Primarily because training a regret-based teacher is hard (Parker-Holder et al.
First, the teacher receives a sparse reward only after specifying the full parameterization of a task; leading to a long-horizon credit assignment problem.
Additionally, the teacher agent faces a combinatorial explosion problem if the parameter space is permutation invariant—e.g., for a navigation task, a set of obstacles corresponds to factorially different permutations of the parameters 1 .
More importantly, to generate tasks at the frontier of agents’ capabilities, the teacher needs to simultaneously learn a task manifold and navigate it to induce a curriculum.
The teacher learns this task manifold implicitly based on regret.
However, as the student is continuously co-learning with the teacher, the task manifold is also evolving over time.
Hence, the teacher needs to simultaneously learn the evolving task manifold, as well as how to navigate it effectively—which is a difﬁcult learning problem.
To address the above-mentioned challenges, we present Curriculum Learning via Unsupervised Task
Representation Learning (CLUTR).
At the core of CLUTR, lies a hierarchical graphical model that decouples task representation learning from curriculum learning.
We develop a variational approximation to this problem and train a Recurrent Variational AutoEncoder (VAE) to learn a latent task manifold.
Unlike PAIRED, which builds the tasks from scratch one parameter at a time, the CLUTR teacher generates tasks in a single timestep by sampling points from the latent task manifold and uses the generative model to translate them into complete tasks.
The CLUTR teacher learns the curriculum by navigating the pretrained and ﬁxed task manifold via maximizing regret.
By utilizing a pretrained latent task-manifold, the CLUTR teacher can train as contextual bandit – overcoming the long-horizon credit assignment problem – and create a curriculum much more efﬁciently – improving stability at no cost to its effectiveness.
Finally, by carefully introducing bias to the training corpus (such as sorting each parameter vector), CLUTR solves the combinatorial explosion problem of parameter space without using any costly environment interaction.
Our experimental results show that CLUTR outperforms PAIRED, both in terms of generalization and sample efﬁciency, in the challenging pixel-based continuous CarRacing and partially observable discrete navigation tasks.
In CarRacing, CLUTR achieves 18x higher zero-shot generalization returns than PAIRED, while being trained on 60% fewer environment interactions on the F1 benchmark, modeled on real-life F1 racing tracks.
Furthermore, CLUTR performs comparably to the non-UED attention-based SOTA(Tang et al.
, outperforming it in nine of the 20 test tracks while requiring fewer than 1% of its environment interactions.
In navigation tasks, CLUTR achieves higher zero-shot generalization in 14 out of the 18 test tasks, achieving a 33% higher solved rate overall.
Furthermore, we empirically validate our hypotheses to justify the algorithmic decisions choices behind CLUTR.
In summary, we make the following contributions: i) we introduce CLUTR, a novel UED algorithm by augmenting the PAIRED teacher with unsupervised task-representation learning that is derived from a hierarchical graphical model for curriculum learning, ii) CLUTR, by decoupling task representation learning from curriculum learning, solves the long-horizon credit assignment and the combinatorial explosion problems faced by PAIRED.
iii)
Our experimental results show CLUTR outperforms PAIRED, both in terms of generalization and sample efﬁciency, in two challenging sets of tasks:
CarRacing and navigation.
R ELATED W ORK
Unsupervised Curriculum Design: Dennis et al.
(2020) was the ﬁrst to formalize UED and introduced the minimax regret-based UED teacher algorithm, PAIRED with a strong theoretical robustness guarantee.
However, gradient-based multi-agent RL has no convergence guarantees and often fails to converge in practice(Mazumdar et al.
Pre-existing techniques like Domain Randomization (DR) (Jakobi (1997), Sadeghi & Levine (2016), Tobin et al.
and
minimax adversarial curriculum learning( Morimoto & Doya (2005), Pinto et al.
(2017)) also fall under the category of UEDs.
DR teacher follows a uniform random strategy, while the minimax adversarial teachers follow the maximin criteria, i.e., generate tasks that minimize the returns of the agent. POET(Wang et al. and
Enhanced POET(Wang et al.
(2020)) also approached UED, before
PAIRED, using an evolutionary approach using a co-evolving population of tasks and agents.
Recently, Jiang et al.
(2021a)
proposed Dual Curriculum Design (DCD): a novel class of UEDs that augments UED generation methods (e.g., DR and PAIRED) with replay capabilities.
DCD involves two teachers: one that actively generates tasks with PAIRED or DR, while the other curates the curriculum to replay previously generated tasks with Prioritized Level Replay(PLR)(Jiang et al.
(2021b)).
Jiang et al.
(2021a) shows that, even with random generation (i.e., DR), updating the students only on the replayed level (but not while they are ﬁrst generated, i.e., no exploratory student gradient updates as PLR) and with a regret-based scoring function, PLR can also learn minimax-regret agents at Nash Equilibrium and call this variation Robust PLR.
It also introduces
REPAIRED, combining PAIRED with Robust PLR.
Parker-Holder et al.
(2022) introduces ACCEL, which improves on Robust PLR by allowing edit/mutation of the tasks with an evolutionary algorithm.
While CLUTR and PAIRED-variants actively adapt task generation to the performance of agents, other algorithms such as PLR generates task from a ﬁxed task distribution.
Different from
PAIRED-variants, which are susceptible to instability due to evolving task-manifold, CLUTR introduces a novel variational formulation with a VAE-style pretraining for task-manifold learning.
Representation Learning: Variational Auto Encoders (Kingma & Welling (2013), Rezende et al.
(2014), Higgins et al.
(2016)) have widely been used for their ability to capture high-level semantic information from low-level data and generative properties in a wide variety of complex domains such as computer vision (Razavi et al.
(2019), Gulrajani et al.
(2016), Zhang et al.
(2021), Zhang et al.
natural language (Bowman et al.
(2015), Jain et al.
, speech (Chorowski et al.
(2019)), and music (Jiang et al.
VAE has been used in RL as well for representing image observations (Kendall et al.
, Yarats et al.
and generating goals (Nair et al.
While CLUTR also utilizes similar VAEs, different from prior work, it combines them in a new curriculum learning algorithm to learn a latent task manifold.
Florensa et al.
(2018) proposed a curriculum learning algorithm with latent-space goal generation using a Generative Adversarial
Network.
B ACKGROUND 3. U NSUPERVISED E NVIRONMENT D ESIGN (UED)
As introduced by Dennis et al.
(2020) UED is the problem of inducing a curriculum by designing a distribution of concrete, fully-speciﬁed environments, from an underspeciﬁed environment with free parameters.
The fully speciﬁed environments are represented using a Partially Observable Markov
Decision Process (POMDP) represented by ( A, O, S, T , I , R ,  ) , where A , O , and S denote the action, observation, and state spaces, respectively.
I !
O is the observation function, R : S !
R is the reward function,
T : S ⇥ A !
( S ) is the transition function and  is the discount factor.
The underspeciﬁed environments are deﬁned in terms of an Underspeciﬁed Partially Observable Markov
Decision Process (UPOMDP) represented by the tuple M =
( A, O, ⇥ , S M , T M , I M , R M ,  ) .
⇥ is a set representing the free parameters of the environment and is incorporated in the transition function as T M : S ⇥ A ⇥ ⇥ !
( S ) .
Assigning a value to ~✓ results in a regular POMDP, i.e.,
UPOMDP + ~✓ = POMDP.
Traditionally (e.g., in Dennis et al.
(2020) and Jiang et al.
(2021a)) ⇥ is considered as a trajectory of environment parameters ~✓ or just ✓ —which we call task in this paper.
For example, ✓ can be a concrete navigation task represented by a sequence of obstacle locations.
We denote a concrete environment generated with the parameter ~✓ 2 ⇥ as M ~✓ or simply M ✓ .
The value of a policy ⇡ in M ✓ is deﬁned as V ✓ ( ⇡ ) =
E [
P
Tt =
r t  t ] , where r t is the discounted reward obtained by ⇡ in M ✓ .
3. PAIRED
PAIRED Dennis et al.
(2020) solves UED with an adversarial game involving three players 2 : the agent ⇡ P and an antagonist ⇡
A , are trained on tasks generated by the teacher ˜ ✓ .
PAIRED objective is: max ˜ ✓ , ⇡ P min ⇡ A U ( ⇡ P , ⇡ A , ˜ ✓ ) =
E ✓ ⇠ ˜ ✓ [ R EGRET ✓ ( ⇡ P , ⇡ A )] .
Regret is deﬁned by the difference of the discounted rewards obtained by the antagonist and the agent in the generated tasks, i.e.,
R EGRET ✓ ( ⇡ P , ⇡ A ) =
V ✓ ( ⇡ A )  V ✓ ( ⇡ P ) .
The PAIRED teacher agent is deﬁned as ⇤ : ⇧ !
( ⇥ T ) , where ⇧ is a set of possible agent policies and ⇥ T is the set of possible tasks.
The teacher is trained with an RL algorithm with U as the reward while, the protagonist and antagonist agents are trained using the usual discounted rewards from the environments.
Dennis et al.
(2020) also introduced the ﬂexible regret objective, an alternate regret approximation that is less susceptible to local optima.
It is deﬁned by the difference between the average score of the agent and antagonist returns and the score of the policy that achieved the highest average return.
C URRICULUM L EARNING VIA U NSUPERVISED T ASK R EPRESENTATION L EARNING
In this section, we formally present CLUTR as a latent UED and discuss it in details.
4. F ORMULATION OF CLUTR
At the core of CLUTR is the latent generative model representing the latent task manifold.
Let’s assume that R is a random variable that denotes a measure of success over the agent and antagonist agent and z be a latent random variable that generates environments/tasks, denoted by the random variable E .
We use the graphical model shown in Figure-1 to formulate CLUTR.
Both E and R are observed variables while z is an unobserved latent variable.
R can cover a broad range of measures used in different UED methods including PAIRED and DR (Domain Randomization).
In PAIRED, R represents the R EGRET .
We use a variational formulation of UED by using the above graphical model to derive the following ELBO for CLUTR, where V AE ( z, E ) denotes the VAE objective:
[Equation]
We share the details of this derivation in Section A.1 of the Appendix.
The above ELBO (Eq.1) deﬁnes the optimization objective for CLUTR, which can be seen as optimizing the VAE objective with a regret-based regularization term and vice versa.
As previously discussed, it is difﬁcult to train a UED teacher while jointly optimizing for both the curriculum and task representations.
Hence we propose a two-level optimization for CLUTR.
First, we pretrain a VAE to learn unsupervised task representations, and then in the curriculum learning phase, we optimize for regret to generate the curriculum while keeping the VAE ﬁxed.
In Section 5.4, we empirically show that this two-level optimization performs better than the joint optimization of Eq.1, i.e., ﬁnetuning the VAE decoder with the regret loss during the curriculum learning phase.
4. U NSUPERVISED L ATENT T ASK R EPRESENTATION L EARNING
As discussed above, we use a Variational AutoEncoder(VAE) to model our generative latent taskmanifold.
Aligning with Dennis et al.
(2020) and Jiang et al.
(2021a), we represent task ✓ , as a sequence of integers.
For example, in a navigation task, these integers denote obstacle, agent, and goal locations.
We use an LSTM-based Recurrent VAE (Bowman et al.
to learn task representations from integer sequences.
We learn an embedding for each integer and use cross-entropy over the sequences to measure the reconstruction error.
This design choice makes CLUTR applicable to task parameterization beyond integer sequences, e.g., to sentences or images.
To train our VAEs, we generate random tasks by uniformly sampling from ⇥ T , the set of possible tasks.
Thus, we do not require any interaction with the environment to learn the task manifold.
Such unsupervised training of the task manifold is practically very useful as interactions with the environment/simulator are much more costly than sampling.
Furthermore, we sort the input sequences, fully or partially, when they are permutation invariant, i.e., essentially represent a set.
By sorting the training sequences, we thus avoid the combinatorial explosion faced by the PAIRED teacher. [Table]
4. CLUTR
We deﬁne CLUTR following the objective given in Eq. 1.
CLUTR uses the same curriculum objective as PAIRED, R EGRET ( R, E ) =
R EGRET ✓ ( ⇡ P , ⇡ A ) where, ✓ denotes a task, i.e., a concrete assignment to the free parameters of the environment E .
Unlike PAIRED teacher, which generates ✓ directly, the CLUTR teacher policy is deﬁned as ⇤ : ⇧ !
( Z ) , where ⇧ is a set of possible agent policies and Z is as the latent space.
Thus, the CLUTR teacher is a latent environment designer, which samples random z and ✓ is generated by the VAE decoder function G : Z ! ⇥ .
We present the outline of the CLUTR in Algorithm 1.
CLUTR outline is very similar to PAIRED, differing only in the ﬁrst two lines of the main loop to incorporate the latent space.
Algorithm
Pretrain VAE with randomly sampled tasks from ⇥
Randomly initialize Agent ⇡ P , Antagonist ⇡ A , and Teacher ˜ ⇤ ; while not converged do
Generate latent task vector:
z ⇠ Z from the teacher Create POMDP M ✓ where ✓ =
G ( z ) and G is the VAE decoder function
Collect Agent trajectory ⌧ P in M ✓ .
Compute: U ✓ ( ⇡ P ) =
P
Ti =
r t  t
Collect Antagonist trajectory ⌧ A in M ✓ .
Compute: U ✓ ( ⇡ A ) =
P
Ti =
r t  t
Compute: R EGRET ✓ ( ⇡ P , ⇡ A ) =
U ✓ ( ⇡ A )  U ✓ ( ⇡ P )
Train Protagonist policy ⇡ P with RL update and reward R end while
Now we discuss a couple of additional properties of CLUTR compared to PAIRED-variants.
CLUTR teacher samples from the latent space Z and thus generates a task in a single timestep.
Note that this is not possible for the PAIRED/REPAIRED teacher, as they generate one task parameter at a time, conditioned on the state of the partially-generated task so far.
PAIRED-variants typically observe the state of the partially generated task to generate the next parameter. Hence depending on the state space, they require designing different teacher architectures for different environments.
CLUTR teacher architecture, however, is agnostic of the problem domain.
Hence the same architecture can be used across different problems.
4. C OMPARISON OF CLUTR WITH C ONTEMPORARY UED METHODS
As discussed in 2, contemporary UED methods can be characterized by their i) teacher type (random/ﬁxed or, learned/adaptive) and ii) the use of replay.
To clearly place CLUTR in the context of contemporary UEDs, we discuss another important aspect of curriculum learning algorithms: how the task manifold is learned.
The random UED teachers do not learn a task manifold.
Regret-based teachers such as PAIRED and REPAIRED learn an implicit (e.g., the hidden state of the teacher
LSTM) task-manifold
but it is not used explicitly.
It is trained via RL based on regret estimates of the tasks they generate.
Hence, these task-manifolds depend on the quality of the estimates, which in turn depends on the overall health of the multi-agent RL training.
Furthermore, they do not take into account the actual task structures.
In contrast, CLUTR introduces an explicit task-manifold modeled with VAE that presents a local neighborhood structure capturing the similarity of the tasks.
Hence, similar tasks (in terms of parameterization) would be placed nearby in the latent space.
Intuitively this local neighborhood structure should facilitate the teacher to navigate the manifold.
The above discussion illustrates that CLUTR along with PAIRED and REPAIRED form a category of UEDs that generates tasks based on a learned task-manifold, orthogonal to the random generation-based methods, while CLUTR being the only one utilizing an unsupervised generative task manifold.
Table 1 summarizes the differences.
E XPERIMENTS
In this section, we ﬁrst evaluate CLUTR’s performance compared to the existing UEDs in PixelBased Car Racing with continuous control and dense rewards.
As discussed in Section 4.4, our primary comparison is with PAIRED and REPAIRED—the only two existing UED methods that learn task-manifolds to generate tasks.
For completeness, we also compare CLUTR with UEDs with random teachers.
Furthermore, we compare with PAIRED on partially observable navigation tasks with discrete control and sparse rewards.
We then empirically evaluate the following hypotheses:
H1 : CLUTR generates a more efﬁcient curriculum.
(Section 5.3) H2 : Learning task representations and curriculum simultaneously degrades the performance (5.4) H3 :
Training VAE on sorted data solves the combinatorial explosion problem.
(Section 5.5)
At last, we analytically compare the CLUTR and PAIRED curricula.
Full details of the environments, network architectures of the teacher and student agents, the VAE, the training hyperparameters, and further analysis and evaluation are discussed in the Appendix.
5. CLUTR P ERFORMANCE ON P IXEL -B ASED C ONTINUOUS C ONTROL C AR R ACING E NVIRONMENT
The CarRacing environment (Jiang et al.
(2021a), Brockman et al.
(2016)) requires the agent to drive a full lap around a closed-loop racing track modeled with B´ezier Curves (Mortenson (1999)) of up to 12 control points.
CLUTR was trained with the Flexible Regret Objective for 2M timesteps— around which the agent’s training return converges to its maximum.
We also experimented with the standard regret objective and obtained better performance than PAIRED, which we discussed in Section C.2 of the appendix.
We train the VAE on 1 Million randomly generated tracks for Million gradient updates.
Note that only one VAE was trained and used for all the experiments ( independent runs).
We evaluate the agents on the F1 benchmark (Jiang et al.
(2021a))
that contains 20 test tracks modeled on real-life F1 racing tracks.
These tracks are signiﬁcantly out of distribution than any tracks that the UED teachers can generate with just 12 control points.
Further details on the environment, network architectures, and VAE training can be found in Section B.1,B.2, and B. of the appendix, respectively.
Figure 2a shows the mean return obtained by the CLUTR, PAIRED, and REPAIRED on the
F1 test tracks.
CLUTR outperforms PAIRED and REPAIRED by a huge margin: showing an 18x higher mean return than PAIRED and 1.6x than REPAIRED, outperforming both of them in all of the 20 test tracks.
Note that CLUTR was trained only for 2M timesteps, while both PAIRED and
REPAIRED were trained for 5M timesteps.
Figure 2b tracks the agents’ generalization capabilities by periodically evaluating them on four unseen tracks Vanilla, Singapore, Germany, and Italy.
These tracks were selected aligning with Jiang et al.
(2021a).
Based on these selected tracks, CLUTR shows much better generalization and sample efﬁciency—achieving better performance and faster improvement.
Further experiment results are shared in Section C of the appendix.
We also compare CLUTR to other existing UEDs with random task generation on the F1 benchmark for completeness.
CLUTR outperforms Domain Randomization and PLR, falling short only to Robust PLR, which achieves an overall 1.13X higher returns.
Nonetheless, CLUTR performs
[Figure]
The results show mean and standard error of 10 independent runs.
Further details can be found in
Section C in Appendix comparably to Robust PLR on seven of the 20 tracks and outperforms in one.
Furthermore, CLUTR shows comparable performance to the non-UED attention-based state-of-the-art method for CarRacing (Tang et al.
(2020)), despite not using a self-attention policy and training on signiﬁcantly fewer environment timesteps ( < 1% ).
Moreover, CLUTR outperforms it on nine of the 20 tracks.
Detailed results on individual tracks are presented in Table 4 of the Appendix.
5. CLUTR P ERFORMANCE ON P ARTIALLY O BSERVABLE N AVIGATION T ASKS ON M INI G RID
We also compare CLUTR with PAIRED on the popular MiniGrid environment, originally introduced by Chevalier-Boisvert et al.
and adopted by Dennis et al.
(2020) for UEDs.
In these navigation tasks, an agent explores a grid world to ﬁnd the goal while avoiding obstacles and receives a sparse reward upon reaching the goal.
To train CLUTR VAE, we generate 1 Million random grids, with the obstacle locations sorted, and the number of obstacles uniformly varying from zero to 50, aligning with Dennis et al.
We used the standard regret objectives.
Note that the results reported in the original PAIRED paper are obtained after 3 Billion timesteps of training, while we run both PAIRED and CLUTR for 500M timesteps (5 independent runs) due to the huge computational resource and time needed to run a training with 3 Billion timesteps.
For the same computational constraints, we compare only with PAIRED in this environment.
Figure 4 shows zero-shot generalization performance of CLUTR and PAIRED 18 unseen navigation tasks from Dennis et al.
based on the percent of environments the agent solved, i.e., solved rate.
CLUTR achieves superior generalization solving 64% of the unseen grids, while PAIRED achieves 43%, which is 33% lower compared to CLUTR.
From ﬁgure 4 it can be seen CLUTR outperforms PAIRED achieving a higher mean solve rate on 14 out of the 18 test navigation tasks.
Figure 3 shows solved rates on four selected grids (Sixteen Rooms, Sixteen Rooms with Fewer Doors, Labyrinth, and Large Corridor) during training.
CLUTR shows better sample efﬁciency, as well as generalization than PAIRED.
5. E FFICIENCY OF THE C URRICULUM : CLUTR VS PAIRED
Figure 5 shows the mean regret on the teacher-generated tasks for both CarRacing and navigation tasks.
CLUTR shows a lower regret than PAIRED, meaning the performance gap between the agent and the antagonist is lower in CLUTR.
From a curriculum learning perspective, we want to train
[Figure] Figure CLUTR shows a smaller regret value indicating a less performance gap between the agent and the antagonist. CLUTR also converges faster.
the agent on tasks that are slightly harder than it can already solve or, those tasks that it can solve already but can obtain better returns.
In practice, both the agent and the antagonist are trained in the same training context e.g., the same hyper-parameters, model architecture, and tasks, differing only by their random initial weights.
Hence, a lower regret means that the teacher is generating tasks that are either slightly harder than the tasks the agent can solve now (because the other agent is solving them) or, tasks in which the antagonist is performing slightly better.
Hence, the tasks are more likely at the agent’s frontier of capability.
The curves also show that CLUTR and PAIRED show similar convergence patterns, while CLUTR converges sooner to a better local optimum.
These observations, in addition to the empirical performance, indicate that CLUTR is generating a more efﬁcient curriculum than PAIRED.
5. L EARNING TASK MANIFOLD AND CURRICULUM : J OINT VS T WO - STAGED O PTIMIZATION
etuned We hypothesize that learning the task representations and the curriculum simultaneously results in a difﬁcult training problem due to the non-stationarity of the task manifold.
To test this, instead of keeping the task representations ﬁxed, we continue ﬁnetuning our decoder on the regret loss during the teacher-student curriculum learning phase.
This experiment shows a 58% performance drop in the F1 benchmark, labeled ‘Finetuned VAE’ in Figure 6.
This empirically validates our hypothesis that pretraining a latent task space and then learning to navigate it to induce curriculum indeed is easier and can lend to better UED.
5. I MPACT OF SORTING VAE DATA ON SOLVING C OMBINATORIAL E XPLOSION
We hypothesized that training a VAE on sorted sequences can solve the combinatorial explosion problem.
To test this, we run CLUTR with an alternate VAE trained 5X longer
[Figure]
This experiment shows a 59% performance drop on the F1 benchmark, labeled ‘Shufﬂed VAE’ in Figure 6, empirically validating our hypothesis.
Further details are discussed in Section C.3 of the Appendix.
5. C URRICULUM C OMPLEXITY
In this section, we compare the curriculum generated by CLUTR and PAIRED, with snapshots of tasks generated by these methods during different stages of the training (Figure 7).
We illustrate one common mode of failure/ineffectiveness shown by PAIRED:
The curriculum starts with arbitrarily complex tasks, which none of the agents can solve at the initial stage of training.
After a while, PAIRED starts generating rudimentary degenerate tasks.
If enough training budget is given, PAIRED eventually gets out of the degenerative local minima, and the curriculum complexity starts to emerge.
On the other hand, CLUTR does not show such degeneration and generates seemingly interesting tasks throughout.
The examples shown Figure 7 illustrates this.
C ONCLUSION AND F UTURE W ORK
In this work, we propose CLUTR, an unsupervised environment design method via unsupervised task representation learning.
CLUTR augments PAIRED with a latent task space, decoupling task representation learning from curriculum learning.
CLUTR poses several advantages over PAIREDvariants, including solving the long-horizon credit assignment and the combinatorial explosion of the parameter space.
Our experiments show CLUTR outperforms PAIRED-variants in terms of sample efﬁciency and generalization.
Even though CLUTR and other regret-based UEDs empirically show good generalization on humancurated complex transfer tasks, they rarely can generate human-level task structures during training.
An interesting direction would be to enable UED algorithms to generate realistic tasks.
Another important direction would be to reduce the gap between the theoretical and practical aspects of regret-based multi-agent UED algorithms, which are subject to the quality of regret estimates and multi-agent RL training. At last, random generator algorithms like Robust PLR or even, DR have been shown to perform better than learned generator approaches like CLUTR or PAIRED.
An interesting direction would be to investigate the conditions/environments under which a random generator performs better than an adaptive generator and vice versa. At last, we are excited about latent-space curriculum design and hope our work will encourage further research in this domain. A CKNOWLEDGMENTS This project is a collaboration under Google-BAIR(Berkeley Artiﬁcial Intelligence Research Lab)
Commons program. We gratefully acknowledge Natasha Jaques for their valuable feedback. We also acknowledge Raymond Chong, Adrian Liu and Sarah Bhaskaran for the valuable discussions
E THIC S TATEMENT
Unsupervised Environment Design can be applied to many real-world applications and shares many similar ethical concerns and considerations with other Artiﬁcially Intelligent(AI) systems.
For example, AI systems can cause more unemployment or be used for reasons/applications that have a negative societal impact, for which responsible usage of such AI systems must be promoted and established.
During our research, all the experiments were done in simulation and no human or living subjects were used.
R EPRODUCIBILITY
Our code, saved checkpoints, and training data are available at https://github.com/clutr/ clutr