CLUTR: C URRICULUM L EARNING VIA U NSUPERVISED T ASK R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Reinforcement Learning (RL) algorithms are often known for sample inefficiency and difficult generalization.
Recently, Unsupervised Environment Design (UED) emerged as a new paradigm for zero-shot generalization by simultaneously learning a task distribution and agent policies on the sampled tasks. This is a nonstationary process where the task distribution evolves along with agent policies; creating an instability over time.
While past works demonstrated the potential of such approaches, sampling effectively from the task space remains an open challenge, bottlenecking these approaches. To this end, we introduce CLUTR: a novel curriculum learning algorithm that decouples task representation and curriculum learning into a two-stage optimization.
It first trains a recurrent variational autoencoder on randomly generated tasks to learn a latent task manifold. Next, a teacher agent creates a curriculum by maximizing a minimax REGRET-based objective on a set of latent tasks sampled from this manifold. By keeping the task manifold fixed, we show that CLUTR successfully overcomes the non-stationarity problem and improves stability.
Our experimental results show CLUTR outperforms PAIRED, a principled and popular UED method, in terms of generalization and sample efficiency in the challenging CarRacing and navigation environments: showing an 18x improvement on the F1 CarRacing benchmark.
CLUTR also performs comparably to the non-UED state-of-the-art for CarRacing, outperforming it in nine of the 20 tracks.
CLUTR also achieves a 33% higher solved rate than
PAIRED on a set of 18 out-of-distribution navigation tasks.
I NTRODUCTION
Deep Reinforcement Learning (RL) has shown exciting progress in the past decade solving many challenging domains including Atari (Mnih et al.
, Dota (Berner et al.
, Go (Silver et al. (2016)).
However, deep RL is sample-inefficient. Moreover, out-of-box deep RL agents are often brittle: performing poorly on tasks that they have not encountered during training, or often failing to solve them altogether even with the slightest change ( Cobbe et al.
(2019), Azad et al. (2022), Zhang et al. Curriculum Learning (CL) algorithms showed promise to improve (Portelas et al. (2020), Narvekar et al.
RL sample efficiency by employing a teacher algorithm that attempts to train the agents on tasks falling at the boundary of their capabilities, i.e., tasks that are slightly harder than the agents can currently solve.
Recently, a class of unsupervised CL algorithms, called Unsupervised Environment Design (UED) [Dennis et al.
(2020),Jiang et al. (2021a)], has shown impressive generalization capabilities which require no training tasks as input. UEDs automatically generate tasks by sampling from the free parameters of the environment (e.g., the start, goal, and obstacle locations for a navigation task) and attempt to improve sample efficiency and generalization by adapting a diverse task distribution at the agent’s frontier of capabilities.
Protagonist Antagonist Induced Regret Environment Design (PAIRED) (Dennis et al.
(2020)) is one of the most principled UED algorithms.
The PAIRED teacher is itself an RL agent with actions denoting different task parameters. PAIRED aims at generating tasks that maximize the agent’s regret, defined as the performance gap between an optimal policy and the student agent. Theoretically, upon convergence, the agent learns to minimize the regret, i.e., will solve every solvable task.
Such a robustness guarantee makes regret-based teachers well suited for training robust agents.
Despite the strong robustness guarantee, PAIRED is still sample inefficient in practice.
Primarily because training a regret-based teacher is hard (Parker-Holder et al.
First, the teacher receives a sparse reward only after specifying the full parameterization of a task; leading to a long-horizon credit assignment problem.
Additionally, the teacher agent faces a combinatorial explosion problem if the parameter space is permutation invariant—e.g., for a navigation task, a set of obstacles corresponds to factorially different permutations of the parameters. More importantly, to generate tasks at the frontier of agents’ capabilities, the teacher needs to simultaneously learn a task manifold and navigate it to induce a curriculum. The teacher learns this task manifold implicitly based on regret.
However, as the student is continuously co-learning with the teacher, the task manifold is also evolving over time. Hence, the teacher needs to simultaneously learn the evolving task manifold, as well as how to navigate it effectively—which is a difficult learning problem.
To address the above-mentioned challenges, we present Curriculum Learning via Unsupervised Task Representation Learning (CLUTR).
At the core of CLUTR, lies a hierarchical graphical model that decouples task representation learning from curriculum learning. We develop a variational approximation to this problem and train a Recurrent Variational AutoEncoder (VAE) to learn a latent task manifold.
Unlike PAIRED which builds the tasks from scratch one parameter at a time, the CLUTR teacher generates tasks by sampling points from the latent task manifold and uses the generative model to translate them into actual complete tasks.
The CLUTR teacher learns the curriculum by navigating the pretrained and fixed task manifold via maximizing regret. By utilizing a pretrained latent task-manifold, the CLUTR teacher can train using single-step RL – overcoming long-horizon credit assignment problem – and can create a curriculum much more efficiently – improving stability at no cost to its effectiveness. Additionally, by carefully introducing bias to the training corpus (such as sorting each parameter vector), CLUTR solves combinatorial explosion problem of parameter space without using any costly environment interaction.
Lastly, the theoretical robustness guarantee provided by PAIRED is agnostic of the teacher model class and still holds for CLUTR.
Our experimental results show that CLUTR outperforms PAIRED, both in terms of generalization and sample efficiency, in the challenging pixel-based continuous CarRacing and partially observable discrete navigation tasks. In CarRacing, CLUTR achieves 18x higher zero-shot generalization returns than PAIRED, while being trained on 60% fewer environment interactions on the F1 benchmark, modeled on real-life F1 racing tracks. Furthermore, CLUTR performs comparably to the non-UED attention-based SOTA(Tang et al.
, outperforming it in nine of the 20 test tracks while requiring fewer than 1% of its environment interactions.
In navigation tasks, CLUTR achieves higher zero-shot generalization in 14 out of the 18 test tasks, achieving a 33% higher solved rate overall. Furthermore, we empirically validate our hypotheses to justify the algorithmic decisions choices behind CLUTR.
In summary, we make the following contributions: i) we introduce CLUTR, a novel UED algorithm by augmenting the PAIRED teacher with unsupervised task-representation learning that is derived from a hierarchical graphical model for curriculum learning, ii) CLUTR, by decoupling task representation learning from curriculum learning, solves the long-horizon credit assignment and the combinatorial explosion problems faced by PAIRED.
iii)
Our experimental results show CLUTR outperforms PAIRED, both in terms of generalization and sample efficiency, in two challenging sets of tasks: CarRacing and navigation.  R ELATED W ORK Unsupervised Curriculum Design: Dennis et al.
(2020) was the first to formalize UED and introduced the minimax regret-based UED teacher algorithm, PAIRED with a strong theoretical robustness guarantee.
However, gradient-based multi-agent RL has no convergence guarantees and often fails to converge in practice(Mazumdar et al.
Pre-existing techniques like Domain Randomization (DR) (Jakobi (1997), Sadeghi & Levine (2016), Tobin et al.
and
minimax adversarial curriculum learning( Morimoto & Doya (2005), Pinto et al.
(2017)) also fall under the category of UEDs.
DR teacher follows a uniform random strategy, while the minimax adverserial teachers follow the maximin criteria, i.e., generate tasks that minimize the returns of the agent.
Recently, Jiang et al.
(2021a)
proposed Dual Curriculum Design (DCD): a novel class of UEDs that augments UED generation methods (e.g., DR and PAIRED) with replay capabilities.
DCD involves two teachers: one that actively generates tasks with PAIRED or DR, while the other cu- rates the curriculum to replay previously generated tasks with Prioritized Level Replay(PLR)(Jiang et al.
(2021b)).
Jiang et al.
(2021a) shows that, even with random generation (i.e., DR), updating the students only on the replayed level (but not while they are first generated, i.e., no exploratory student gradient updates as PLR) and with a regret-based scoring function, PLR can also learn minimax-regret agents at Nash Equilibrium and call this variation Robust PLR.
It also introduces
REPAIRED, combining PAIRED with Robust PLR.
Parker-Holder et al.
(2022) introduces ACCEL, which improves on Robust PLR by allowing edit/mutation of the tasks with an evolutionary algorithm.
While CLUTR and PAIRED-variants actively adapt task generation to the performance of agents, other algorithms such as PLR generates task from a fixed task distribution.
Different from
PAIRED-variants, which are susceptible to instability due to evolving task-manifold, CLUTR introduces a novel variational formulation with a VAE-style pretraining for task-manifold learning.
Representation Learning: Variational Auto Encoders (Kingma & Welling (2013), Rezende et al.
(2014), Higgins et al.
(2016)) have widely been used for their ability to capture high-level semantic information from low-level data and generative properties in a wide variety of complex domains such as computer vision (Razavi et al.
(2019), Gulrajani et al.
(2016), Zhang et al.
(2021), Zhang et al.
natural language (Bowman et al.
(2015), Jain et al.
, speech (Chorowski et al.
(2019)), and music (Jiang et al.
VAE has been used in RL as well for representing image observations (Kendall et al.
, Yarats et al.
and generating goals (Nair et al.
While CLUTR also utilizes similar VAEs, different from prior work, it combines them in a new curriculum learning algorithm to learn a latent task manifold.
B ACKGROUND 3. U NSUPERVISED E NVIRONMENT D ESIGN (UED)
As introduced by Dennis et al.
(2020) UED is the problem of inducing a curriculum by designing a distribution of concrete, fully-specified environments, from an underspecified environment with free parameters.
The fully specified environments are represented using a Partially Observable Markov
Decision Process (POMDP) represented by ( A, O, S, T , I , R , γ ) , where A , O , and S denote the action, observation, and state spaces, respectively.
I → O is the observation function, R : S
→ R is the reward function, T : S ×
A → ∆( S ) is the transition function and γ is the discount factor.
The underspecified environments are defined in terms of an Underspecified Partially Observable Markov
Decision Process (UPOMDP) represented by the tuple M =
( A, O, Θ , S M , T M , I M , R M , γ ) .
Θ is a set representing the free parameters of the environment and is incorporated in the transition function as T M : S
×
A × Θ → ∆( S ) .
Assigning a value to ⃗θ results in a regular POMDP, i.e.,
UPOMDP + ⃗θ = POMDP.
Traditionally (e.g., in Dennis et al.
(2020) and Jiang et al.
(2021a))
Θ is considered as a trajectory of environment parameters ⃗θ or just θ —which we call task in this paper.
For example, θ can be a concrete navigation task represented by a sequence of obstacle locations.
We denote a concrete environment generated with the parameter ⃗θ ∈ Θ as M ⃗ θ or simply M θ .
The value of a policy π in M θ is defined as V θ ( π ) =
E [ (cid:80) Tt =
r t γ t ] , where r t is the discounted reward obtained by π in M θ .
3. PAIRED
PAIRED Dennis et al.
(2020) solves UED with an adversarial game involving three players 1 : the agent π P and an antagonist π A , are trained on tasks generated by the teacher ˜ θ .
PAIRED objective is: max ˜ θ,π P min π A U ( π P , π A , ˜ θ ) =
E θ ∼ ˜ θ
[ R EGRET θ ( π P , π A )] .
Regret is defined by the difference of the discounted rewards obtained by the antagonist and the agent in the generated tasks, i.e., R EGRET θ ( π P , π A ) =
V θ ( π A ) − V θ ( π P ) .
The PAIRED teacher agent is defined as Λ : Π → ∆(Θ T ) , where Π is a set of possible agent policies and Θ T is the set of possible tasks.
The teacher is trained with an RL algorithm with U as the reward while, the protagonist and antagonist agents are trained using the usual discounted rewards from the environments. Gur et al. (2021) introduced the flexible regret objective, an alternate regret approximation that is less susceptible to local optima.
It is defined by the difference between the average score of the agent and antagonist returns and the score of the policy that achieved the highest average return.
C URRICULUM L EARNING VIA U NSUPERVISED T ASK R EPRESENTATION L EARNING
In this section, we formally present CLUTR as a latent UED and discuss it in details.
4. F ORMULATION OF CLUTR
At the core of CLUTR is the latent generative model representing the latent task manifold.
Let’s assume that R is a random variable that denotes a measure of success over the agent and antagonist agent and z be a latent random variable that generates environments.
We use the graphical model shown in
Figure-1 to formulate CLUTR.
Both E and R are observed variables while z is an unobserved latent variable.
R can cover a broad range of measures used in different UED methods including PAIRED and DR (Domain Randomization).
In PAIRED, R represents the R EGRET .
We use a variational formulation of UED by using the above graphical model.
We first define the variational objective as the KL-divergence between an approximate posterior distribution and true posterior distribution over latent variable z , where both R and E are given.
[Equation]
Next, we write the ELBO as E z ∼ q ( z ) [ logq ( z )] − E z ∼ q ( z ) [ logp ( z, R, E )] , which can be reduced to,
[Equation]
We share the details of this derivation in Section B of the Appendix.
The above ELBO defines the optimization objective for CLUTR. As we mentioned earlier, it is difficult to train a UED teacher while jointly optimizing both the curriculum and task representations. Hence we propose a two-level optimization for CLUTR. First, we pretrain a VAE to learn unsupervised task representations and then we optimize for regret to generate curriculum, keeping the representations fixed.
We empirically compare this two-level optimization to jointly optimizing the objective in Eq.1 (Section 5.4). 4. U NSUPERVISED L ATENT T ASK R EPRESENTATION L EARNING As discussed above, we use a Variational AutoEncoder(VAE) to model our generative latent taskmanifold. Aligning with Dennis et al.
(2020) and Jiang et al. (2021a), we represent task θ , as a sequence of integers. For example, in a navigation task, these integers denote obstacle, agent, and goal locations. We use an LSTM-based Recurrent VAE (Bowman et al.
to learn task representations from integer sequences.
We learn an embedding for each integer and use cross-entropy over the sequences to measure the reconstruction error. This design choice makes CLUTR applicable to task parameterization beyond integer sequences, e.g., to sentences or images. To train our VAEs, we generate random tasks by uniformly sampling from Θ T , the set of possible tasks. Thus, we do not require any interaction with the environment to learn the task manifold.
Such unsupervised training of the task manifold is practically very useful as interactions with the environment/simulator are much more costly than sampling.
Furthermore, we sort the input sequences, fully or partially, when they are permutation invariant, i.e., essentially represent a set.
By sorting the training sequences, we thus avoid the combinatorial explosion faced by the PAIRED teacher.
4. CLUTR
We define CLUTR following the objective given in Eq. 1.
CLUTR uses the same curriculum objective as PAIRED, R EGRET ( R, E ) =
R EGRET θ ( π P , π A ) where, θ denotes a task, i.e., a concrete assignment to the free parameters of the environment E .
Unlike PAIRED teacher, which generates θ directly, the CLUTR teacher policy is defined as Λ : Π → ∆( Z ) , where Π is a set of possible agent policies and Z is as the latent space. Thus, the CLUTR teacher is a latent environment designer, which samples random z and θ is generated by the VAE decoder function G : Z → Θ .
We present the outline of the CLUTR algorithm
(Algorithm 1). Algorithm
Pretrain VAE with randomly sampled tasks from Θ
Randomly initialize Agent π P , Antagonist π A , and Adversary ˜Λ ; while not converged do
Use adversary to sample latent task vector:
z ∼ Z
Create POMDP M θ where θ =
G ( z ) and G is the VAE decoder function
Collect Agent trajectory τ P in M θ .
Compute: U θ ( π P ) =
(cid:80) T i =
r t γ t
Collect Antagonist trajectory τ A in M θ .
Compute: U θ ( π A ) =
(cid:80)
Ti =
r t γ t
Compute: R EGRET θ ( π P , π A ) =
U θ ( π A ) − U θ ( π P )
Train Protagonist policy π P with RL update and reward R ( τ P ) =
U θ ( π P )
Train Antagonist policy π A with RL update and reward R ( τ A ) =
U θ ( π A )
Train Adversary policy ˜Λ with RL update and reward R ( τ ˜Λ )
= R EGRET end while
Now we discuss a few additional properties of CLUTR compared to PAIRED-variants: 1. CLUTR teacher samples from the latent space Z , hence it can be trained using single-step RL.
Note that, it is not possible to train the PAIRED/REPAIRED teacher using single-step RL, as they generate one task parameter at a time conditioned on the state of the partiallygenerated task so far, which might not be even possible for some simulators. PAIRED-variants require designing different teacher architectures for different environments: depending on the state-space of the underlying POMDP.
CLUTR teacher architecture, however, is agnostic of the POMDP/UPOMDP it is solving. Hence the same architecture can be used across different problems. CLUTR does not impose any new assumptions on the set of possible agent policies Π , hence CLUTR holds the same robustness guarantee PAIRED provides. 4. C OMPARISON OF CLUTR WITH C ONTEMPORARY UED METHODS
As discussed in 2, contemporary UED methods can be characterized by their i) teacher type (random/fixed or, learned/adaptive) and ii) the use of replay. To clearly place CLUTR in the context of contemporary UEDs, we discuss another important aspect of curriculum learning algorithms: how the task manifold is learned. The random UED teachers do not learn a task manifold. Regret-based teachers such as PAIRED and REPAIRED learn an implicit (e.g., the hidden state of the teacher
LSTM) task-manifold but it is not used explicitly. It is trained via RL based on regret estimates of the tasks they generate.
Hence, these task-manifolds depend on the quality of the estimates, which in turn depends on the overall health of the multi-agent RL training. Furthermore, they do not take into account the actual task structures. In contrast, CLUTR introduces an explicit task-manifold modeled with VAE that presents a local neighborhood structure capturing the similarity of the tasks.
Hence, similar tasks (in terms of parameterization) would be placed nearby in the latent space.
Intuitively this local neighborhood structure should facilitate the teacher to navigate the manifold. The above discussion illustrates that CLUTR along with PAIRED and REPAIRED form a category of UEDs that generates tasks based on a learned task-manifold, orthogonal to the random generation-based methods, while CLUTR being the only one utilizing an unsupervised generative task manifold. Table 1 summarizes the differences. [Table]
E XPERIMENTS
In this section, we first evaluate CLUTR’s performance compared to the existing UEDs in PixelBased Car Racing with continuous control and dense rewards. As discussed in Section4.4, our primary comparison is with PAIRED and REPAIRED—the only two existing UED methods that learn task-manifolds to generate tasks. For completeness, we also compare CLUTR with UEDs with random teachers. Furthermore, we compare with PAIRED on partially observable navigation tasks with discrete control and sparse rewards.
We then empirically evaluate the following hypotheses:
H1 : CLUTR makes training a regret-based teacher easier. (Section 5.3) H2 : Learning task representations and curriculum simultaneously degrades the performance (5.4) H3 :
Training VAE on sorted data solves the combinatorial explosion problem.
(Section 5.5)
At last, we analytically compare the CLUTR and PAIRED curricula. Full details of the environments, network architectures of the teacher and student agents, the VAE, the training hyperparameters, and further analysis and evaluation are discussed in the Appendix.
5. CLUTR P ERFORMANCE ON P IXEL -B ASED C ONTINUOUS C ONTROL C AR R ACING E NVIRONMENT
The CarRacing environment (Jiang et al.
(2021a), Brockman et al.
(2016)) requires the agent to drive a full lap around a closed-loop racing track modeled with B ´ ezier Curves (Mortenson (1999)) of up to 12 control points. We train the VAE on 1 Million randomly generated tracks for 1 Million gradient updates.
CLUTR was trained with 2M timesteps around which the agent’s training returns converge to its maximum. Note that only one VAE was trained and used for all the experiments ( independent runs). We evaluate the agents on the F1 benchmark (Jiang et al.
(2021a))
that contains 20 test tracks modeled on real-life F1 racing tracks. These tracks are significantly out of distribution than any tracks that the UED teachers can generate with just 12 control points. Further details on the environment, network architectures, and VAE training can be found in Section C.1,C.2, and C. of the appendix, respectively.
Figure 2a shows the mean return obtained by the CLUTR, PAIRED, and REPAIRED on the F1 test tracks. CLUTR outperforms PAIRED and REPAIRED by a huge margin: showing an 18x higher mean return than PAIRED and 1.6x than REPAIRED, outperforming both of them in all of the 20 test tracks.
Note that CLUTR was trained only for 2M timesteps, while both PAIRED and REPAIRED were trained for 5M timesteps. Figure 2b tracks the agents’ generalization capabilities by periodically evaluating them on four unseen tracks Vanilla, Singapore, Germany, and Italy. These tracks were selected aligning with Jiang et al.
(2021a).
Based on these selected tracks, CLUTR shows much better generalization and sample efficiency—achieving better performance and faster improvement.
Further experiment results are shared in Section D of the appendix. We also compare CLUTR to other existing UEDs with random task generation on the F1 benchmark for completeness.
CLUTR outperforms Domain Randomization and PLR, falling short only to Robust PLR, which achieves an overall 1.13X higher returns. Nonetheless, CLUTR performs comparably to Robust PLR on seven of the 20 tracks and outperforms in one. Furthermore, CLUTR shows comparable performance to the non-UED attention-based state-of-the-art method for CarRac-
[Figure] ing (Tang et al.
(2020)), despite not using a self-attention policy and training on significantly fewer environment timesteps ( < 1% ).
Moreover, CLUTR outperforms it on nine of the 20 tracks.
Detailed results on individual tracks are presented in Table 2 of the Appendix.
5. CLUTR P ERFORMANCE ON P ARTIALLY O BSERVABLE N AVIGATION T ASKS ON M INI G RID
We also compare CLUTR with PAIRED on the popular MiniGrid environment, originally introduced by Chevalier-Boisvert et al.
and adopted by Dennis et al.
(2020) for UEDs.
In these navigation tasks, an agent explores a grid world to find the goal while avoiding obstacles and receives a sparse reward upon reaching the goal.
To train CLUTR VAE, we generate 1 Million random grids, with the obstacle locations sorted, and the number of obstacles uniformly varying from zero to 50, aligning with Dennis et al.
Note that the results reported in the original PAIRED paper is obtained after 3 Billion timesteps of training, while we run both PAIRED and CLUTR for 500M timesteps due to the huge computational resource and time needed to run a training with 3 Billion timesteps. For the same computational constraints, we compare only with PAIRED in this environment.
[Figure]
Figure 3 shows zero-shot generalization performance of CLUTR and PAIRED 18 unseen navigation tasks from Dennis et al.
based on the percent of environments the agent solved, i.e., solved rate.
CLUTR achieves superior generalization solving 64% of the unseen grids, while PAIRED achieves 43%, which is 33% lower compared to CLUTR.
From figure 3 it can be seen CLUTR outperforms PAIRED achieving a higher mean solve rate on 14 out of the 18 test navigation tasks. Figure 4 shows solved rates on four selected grids (Sixteen Rooms, Sixteen Rooms with Fewer
Doors, Labyrinth, and Large Corridor) during training.
CLUTR shows better sample efficiency, as well as generalization than PAIRED.
5. T EACHER T RAINING P ERFORMANCE : CLUTR VS PAIRED
Figure 5 shows the mean regret on the teacher-generated tasks for both CarRacing and navigation tasks.
CLUTR shows a lower regret than PAIRED, meaning the performance gap between the agent and the antagonist is lower in CLUTR.
From a curriculum learning perspective, we want to train the agent on tasks that are slightly harder than it can already solve or, those tasks that it can solve already but can obtain better returns.
In practice, both the agent and the antagonist are trained in the same training context e.g., the same hyper-parameters, model architecture, and tasks, differing only by their random initial weights.
Hence, a lower regret means that the teacher is generating tasks that are either slightly harder than the tasks the agent can solve now (because the other agent is solving them) or, tasks in which the antagonist is performing slightly better.
Hence, the tasks are at the agent’s frontier of capability.
The curves also show that CLUTR and PAIRED show similar convergence patterns, while CLUTR is converging sooner to a better local optimum. These observations indicate that training CLUTR teacher is easier than PAIRED. 5. L EARNING TASK MANIFOLD AND CURRICULUM : J OINT VS T WO - STAGED O PTIMIZATION
We hypothesize that learning the task representations and the curriculum simultaneously results in a difficult training problem due to the non-stationarity of the task manifold. To test this, instead of keeping the task representations fixed, we continue finetuning our decoder on the regret loss during the teacher-student curriculum learning phase. This experiment shows a 58% performance drop in the F1 benchmark, labeled ‘Finetuned VAE’ in Figure 6.
This empirically validates our hypothesis that pretraining a latent task space and then learning to navigate it to induce curriculum indeed is easier and can lend to better UED. 5. I MPACT OF SORTING VAE DATA ON SOLVING C OMBINATORIAL E XPLOSION We hypothesized that training a VAE on sorted sequences can solve the combinatorial explosion problem.
To test this, we run CLUTR with an alternate VAE trained 5X longer on a non-sorted and 10X bigger version of the original dataset.
This experiment shows a 59% performance drop on the
[Figure]
F1 benchmark, labeled ‘Shuffled VAE’ in Figure 6, empirically validating our hypothesis.
Further details are discussed in Section D.3 of Appendix.
5. C URRICULUM C OMPLEXITY
In this section, we compare the curriculum generated by CLUTR and PAIRED, with snapshots of tasks generated by these methods during different stages of the training (Figure 7).
We illustrate one common mode of failure/ineffectiveness shown by PAIRED:
The curriculum starts with arbitrarily complex tasks, which none of the agents can solve at the initial stage of training.
After a while, PAIRED starts generating rudimentary degenerate tasks.
If enough training budget is given, PAIRED eventually gets out of the degenerative local minima, and the curriculum complexity starts to emerge.
On the other hand, CLUTR does not show such degeneration and generates seemingly interesting tasks throughout.
The examples shown Figure 7 illustrates this.  C ONCLUSION AND F UTURE W ORK In this work, we propose CLUTR, an unsupervised environment design method via unsupervised task representation learning. CLUTR augments PAIRED with a latent task space, decoupling task representation learning from curriculum learning.
CLUTR poses several advantages over PAIRED-variants, including: solving the long-horizon credit assignment and the combinatorial explosion of the parameter space.
Our experiments show CLUTR outperforms PAIRED-variants in terms of sample efficiency and generalization.
Even though CLUTR and other regret-based UEDs empirically show good generalization on humancurated complex transfer tasks, they rarely can generate human-level task structures during training.
An interesting direction would be to incorporate some prior with minimal human input to generate realistic tasks during training.
Also, the theoretical guarantee holds only when the multi-agent RL training backing these methods reaches Nash equilibrium. However, in practice, Deep multi-agent RL has no convergence guarantee (Mazumdar et al. Furthermore, the regret estimates can be poor leading to a poor curriculum.
E THIC S TATEMENT
Unsupervised Environment Design can be applied to many real-world applications and has similar ethical concerns and considerations of other Artificially Intelligent(AI) systems.
For example, AI systems can cause more unemployment or be used for reasons/applications that have a negative societal impact—for which responsible usage of such AI systems must be promoted and established.
During our research all the experiments were done in simulation and no human or living subjects were used.
R EPRODUCIBILITY
Upon publication we will share our code and dataset. [Equation] B D ETAILED O BJECTIVE D ERIVATION We use a hierarchical graphical model to formulate the latent environment design problem.
Let’s assume that R is a random variable that denotes a measure of success defined using the protagonist and antagonist agents and z be a latent random variable. We use the graphical model in Figure-8 where z generates an environment E and R is the success defined over E . Both E and R are observed variables while z is an unobserved variable. R covers a broad range of measures used in different UED methods including PAIRED and DR (Domain Randomization).
In PAIRED, R represents the R EGRET as the difference of returns between the antagonist and protagonist agents and it depends on the environments that the agents are evaluated on. We use a variational formulation of UED by using the above graphical model. We first define the variational objective as the KL-divergence between an approximate posterior distribution and true posterior distribution over latent variable z , [Equation]
where both R and E are given. Next, we write the ELBO, [Equation] R
[Equation]
We can also induce an objective that includes minimax R EGRET .
Let R be distributed according to an exponential distribution, p ( R | E ) ∝ exp ( R EGRET ( π P , π A | E )) , we derive,
[Equation]
where the normalizing factor is ignored.
C T RAINING D ETAILS C. E NVIRONMENT D ETAILS
Car Racing : The CarRacing environment was originally proposed by OpenAI Gym Brockman et al.
(2016), and later has been reparameterized by Jiang et al.
(2021a) with B´ezier Curves( Mortenson (1999)) for UED algorithms.
This environment requires the agents to drive a full lap around a closed-loop track.
The track is defined by a B´ezier Curve modeled with a sequence of uptoarbitrary control points, each spaced within a fixed radius B/ 2 of the center of the B × B field.
This sequence of control points can uniquely identify a track, subject to a set of predefined curvature constraints Jiang et al.
(2021a).
The control points are encoded in a 10 × 10 grid—a discrete downsampled version of the racing track field.
Each control point hence is a integer denoting a cell of the grid and the cell coordinates are upscaled to match the original scale of the field afterwards.
This ensures no two control points are too close together, preventing areas of excessive track overlapping.
The track consists of a sequence of L polygons and the agent receives a reward of 1000 /L
upon visiting each unvisited polygon and a penalty of − 0 .
1 at each time step to incentivize completing the tracks faster.
Episodes terminate if the agent drives too far off-track but is not given any additional penalty.
The agent controls a 3 dimensional continuous action space corresponding to the car’s steer: torque ∈ [ − 1 . 0 , 1 . 0] , gas: acceleration ∈ [0 , 0 , 1 . 0] , and brake: deceleration ∈ [0 . 0 , 1 . 0] .
Each action is repeated 8 times.
The agent receive a 96 × 96 × 3 RGB pixel observation.
The top 84 ×portion of the frame contains a clipped, egocentric, bird’s eye view of the horizontally centered car.
The bottom 12 × 96 segment simulates a dashboard visualizing the agent’s latest action and return. Minigrid : The environment is partially observable.
Each navigation task is represented with a sequence of integers denoting the locations of the obstacles, the goal, and the starting position of the agent: on a 15 by 15 grid similar to Dennis et al.
The PAIRED teacher outputs a sequence of 52 integers, while the first 50 numbers denote the location of obstacles and may contain duplicates to allow the generation of tasks with fewer than 50 obstacles.
C. N ETWORK A RCHITECTURES
All the student and teacher agents are trained with PPO Schulman et al.
Student Architecture
For CarRacing, we use the same student architecture as Jiang et al.
(2021a).
The architecture consists an image embedding module composed of 2D Convolutions with square kernels of sizes 2,2,2,2,3,3, stride lengths 2,2,2,2,1,1 and channel outputs of 8, 16, 64, 128, 256 stacked together.
The image embedding is of size 256 and is passed through a Fully Connected (FC) layer of 100 hidden units and then passed through ReLU activations.
This embedding is then passed through two FC withhidden neurons, and then a softplus layer, and finally added to 1 for the beta distribution used for the continuous action space.
Further details can be found in Jiang et al.
(2021a).
For navigation tasks, we use the same student architecture as Dennis et al.
The observation is a tuple with a 5 × 5 × 3 grid observation and a direction integer in [0 − 3] .
The grid view is fed to a convolutional layer with kernels of size 3 with 16 filters and the direction integer is passed through
a
FC with 5 units.
This is followed by an LSTM of size 256, and then to two FC layers with 32 units, which connect to the policy outputs.
The value network uses the same architecture.
Teacher Architecture : For CarRacing, CLUTR teacher takes a random noise and generates a continuous vector, i.e., the latent task vector.
We pass the random noise through a simple MultiLayer
Perceptron (MLP) with one hidden layer of 8 neurons as the teacher.
The output of this layer is fed through two separate fully-connected layers, each with a hidden size of 100 and an output dimension equal to the latent space dimension, followed by soft plus activations.
We then add 1 to each component of these two output vectors, which serve as the α and β parameters respectively for the Beta distributions used to sample each latent dimension.
In our experiments, we used a 64-dimensional latent task space.
For Minigrid experiments, we use a network architecture similar to ?
but take only the random noise as input.
VAE architecture We use the architecture proposed in Bowman et al.
C. H YPERPARAMETERS
All our agents are trained with PPO( ? ).
We did not perform any hyperparameter search for our experiments.
The CarRacing experiments used the same parameters used in Jiang et al.
(2021a)
and the Minigrid experiments used the parameters from Dennis et al.
VAE was trained on the parameters from Bowman et al.
[Figure]
. VAE T RAINING
[Figure]
ATA
For CarRacing, we follow the same parameterization as Jiang et al.
(2021a): each track is defined with a sequence of up to 12 integers denoting control points of a B´ezier Curve. .
Each control point is represented with an integer.
We generate 1M random sorted integer sequences of fixed length 12 with duplicates—which enables generating tracks defined with less than 12 control points.
For navigation tasks we use the parameterization of Dennis et al.
(2020), generating upto 50 obstacles for each task for a 15 × 15 grid, surrounded by walls, effectively an active area of 13 × 13 .
Hence, each location is numbered in 1 to 169.
Every number except the last two of the sequence represent obstacle locations, and the last two for the goal and agent location, respectively.
The parameter vector is thus partially permutation invariant.
We uniformly generate 1M sequences of variable length between 2 and 52 (inclusive).
The obstacle locations are sorted.
D D ETAILED R ESULTS ON C AR R ACING
For CarRacing, we use the flexible regret approximation introduced in Gur et al.
The flexible regret objective is a more robust variant of regret estimation used in Dennis et al.
It is defined by the difference between the average score of the agent and antagonist returns and the score of the policy that achieved the highest average return.
Thus, the flexible objective blurs the distinction between the agent and the antagonist.
Hence we designate the agent achieving the higher average training return during the last 10 steps as the agent.
D. D ETAILED C OMPARISON ON F ULL F1 DATASET
Figure 9 compares how different UEDs perform during training by periodically evaluating them on Four Selected Tracks: Vanilla, Singapore, Germany, and Italy.
These tracks were selected aligning with Jiang et al.
(2021a).
Based on these selected tracks, CLUTR improves a bit after 2M timesteps later the performance plateaus.
Robust PLR starts slowly but surpasses all the other methods after 5M timesteps.
Table 2 shows the comparison between the final agents trained with CLUTR and other UED algorithms.
It is to be noted
that, all the
UED methods except CLUTR was trained for 5M timesteps where CLUTR was run for 2M timesteps.
CLUTR outperforms PAIRED by a big margin with 18x bigger mean return on the entire F1 Dataset.
CLUTR also outperforms
Domain Randomization, PLR, and REPAIRED and only falls short to Robust PLR.
Nonetheless, CLUTR shows competitive results compared to Robust PLR, showing comparable results in seven out of the 20 test tracks and outperforming in the Netherlands track.
CLUTR also outperforms the non-UED SOTA on the full F1 dataset.
CLUTR outperforms the Attention Agent on 9 out of the 20 tracks and shows comparable performance in one.
D. CLUTR WITH STANDARD REGRET LOSS
In Figure 10 we show how CLUTR performs with the standard regret approximation as introduced in
Dennis et al.
(2020), after 5M timesteps of training. To test the Zero-shot generalization on the Car Racing tasks, we evaluate both CLUTR and PAIRED agents on 20 levels replicating official humandesigned real Formula One (F1) tracks, also obtained from Jiang et al. (2021a).
CLUTR outperforms
PAIRED in all 20 test-tracks as seen in Figure 10.
Note that these tracks cannot be generated with just 12 control points and hence are significantly OOD than any tracks that the CLUTR and PAIRED
[Figure]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] 404 ±
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] teachers could generate during training. PAIRED struggles to train a robust agent within the allotted training budget.
As mentioned in Jiang et al.
(2021a) PAIRED overexploits the relative strengths of the antagonist over the protagonist and generates a curriculum that gradually reduces the level complexity.
However, CLUTR overcomes this and generates a curriculum where antagonist and protagonist closely compete with each other as seen in Figure 10 and shows a robust generalization on the unseen F1 benchmark.
Figure 12 compares sample efficiency of CLUTR with PAIRED by evaluating the agents on four selected tracks: Vanilla, Singapore, Germany, Italy during training.
It can be seen that CLUTR outperforms PAIRED significantly.
These test environments were used in anyway neither during training CLUTR( or, PAIRED) now while designing it. Figure 13 shows how CLUTR performs without the flexible loss in CarRacing domain. The regret is lower without the regret loss, but shows similar pattern.
The CLUTR agent achieves better returns with flexible loss throughout the training. [Figure] D. E XTENDED A NALYSIS ON I MPACT OF SORTING TRAINING DATA FOR VAE TRAINING
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
The non-sorted dataset was generated by shuffling each track of the original VAE training dataset 10 different times, resulting in a 10X bigger dataset (10M tracks).
It was trained for 5X longer for 5M training steps. We planned on training for 10M gradient steps (10X than the original VAE) but stopped at 5M as it converged much sooner.
We ran both CLUTR and CLUTR-shuffled, i e., CLUTR with a VAE trained on non-sorted data up to 5M timesteps. CLUTR-shuffled shows inferior performance and also signs of unlearning compared to CLUTR.
Figure 14 shows detailed experiment results.