C AN A GENTS R UN R ELAY R ACE WITH S TRANGERS ? G ENERALIZATION OF RL TO O UT - OF -D ISTRIBUTION T RAJECTORIES
A BSTRACT
In this paper, we evaluate and improve the generalization performance for reinforcement learning (RL) agents on the set of “controllable” states, where good policies exist on these states to achieve the goal.
An RL agent that generally masters a task should reach its goal starting from any controllable state of the environment instead of memorizing a small set of trajectories.
To practically evaluate this type of generalization, we propose relay evaluation which starts the test agent from the middle of other independently well-trained stranger agents’ trajectories.
With extensive experimental evaluation, we show the prevalence of generalization failure on controllable states from stranger agents.
For example, in the Humanoid environment, we observed that a well-trained Proximal Policy Optimization (PPO) agent, with only 3.9% failure rate during regular testing, failed on 81.6% of the states generated by well-trained stranger PPO agents.
To improve "relay generalization", we propose a novel method called Self-Trajectory Augmentation (STA), which will reset the environment to the agent’s old states according to the Q function during training.
After applying STA to the Soft Actor Critic’s (SAC) training procedure, we reduced the failure rate of SAC under relay-evaluation by more than three times in most settings without impacting agent performance and increasing the needed number of environment interactions. Our code is available at https ://github com/lan lc
/STA.
I NTRODUCTION
Generalization is critical for deploying reinforcement learning (RL) agents into real-world applications.
A well-trained RL agent that can achieve high rewards under restricted settings may not be able to handle the enormous state space and complex environment variations in the real world.
There are many different aspects regarding the generalization of RL agents.
While many existing works study RL generalization under environment variations between training and testing Kirk et al.
(2021), in this paper, we study the generalization problem in a simple and fixed environment under an often overlooked notion of generalization — a well-generalized agent that masters a task should be able to start from any “controllable” state in this environment and still reach the goal.
For example, a self-driving system may need to take over the control from humans (or other AIs trained for different goals such as speed, gas-efficient, or comfortable) in the middle of driving and must continue to drive the car safely.
We can make little assumptions about what states the cars are at when the take-over happens, and the self-driving agent must learn to drive generally.
Although this problem may look ostensibly easy for simple MDPs (e.g., a 2-D maze), most real RL agents are trained by trajectories generated by its policy, and it is hard to guarantee the behavior of an agent on all “controllable” states.
Roughly speaking, in the setting of robotics (e.g., Mujoco environments), we can define a state as controllable if there exists at least one policy that can lead to a high reward trajectory or reach the goal from this state.
Unfortunately, most ordinary evaluation procedures of RL nowadays do not take this into consideration, and the agents are often evaluated from a fixed starting point with a very small amount of perturbation (Todorov et al., 2012; Brockman et al., 2016).
In fact, finding these controllable states themselves for evaluation is difficult.
[Figure]
The first contribution of this work is to propose relay-evaluation , a proxy to evaluate agent generalization performance on controllable states in a fixed environment.
Relay-evaluation involves running an agent from the middle states of other independently trained agents’ high-reward trajectories.
This is similar to running a relay race with another stranger agent , where the stranger agent controls the robot first, and the test agent takes the reins of the robot later.
It naturally finds a diverse set of controllable states for the test agents because the sampled states come from high reward trajectories of well-trained agents.
The stranger agents can be trained using a variety of RL algorithms, not limited to the one used for the test agent.
Our extensive experiments on 160 agents trained in four environments using four algorithms show that many representative RL algorithms have unexpectedly high failure rates under relay-evaluation.
For example, in the Humanoid environment, Proximal Policy Optimization (PPO) agents and Soft
Actor Critic (SAC) agents have average failure rates of 81 . 6% and 38 . 0% under relay-evaluation even when the stranger agents have trained with the same algorithms (different random seeds), which is surprisingly high compared to the original failure rates (PPO: 3 . 9% , SAC: 0 .
The failure of the agents under this setting shows that they may not genuinely understand the dynamics of the environments and learn general concepts like balancing the robot to avoid failing, but rather memorize a small set of actions specifically for a limited number of states it encountered.
In Figure 1a, we illustrate the t-SNE for states in trajectories of 6 agents trained with 3 algorithms and observe that even trained with the same algorithm, the states generated by different agents are quite distinctive.
Figure 1b shows that the SAC 1 agent only has a low failure rate on its own states, which is the dots colored blue in Figure 1a.
Our second contribution is to propose a novel training method called Self-Trajectory Augmentation (STA) that can significantly improve agents’ generalization without significantly increasing training costs.
We first conduct a motivative experiment to augment the initial state set of an agent during training by the states generated by a set of pretrained stranger agents, and we find that as we increase the number of stranger agents, the relay-evaluation of the agent improves significantly.
However, pretraining additional models is time-consuming and may be impractical in complex environments.
Therefore, we propose a novel method called Self-Trajectory Augmentation (STA), where we randomly set the agent to start from its old trajectories.
Since the distribution of visited states often varies during training, reviewing an agent’s old trajectories can be beneficial for generalization.
After applying STA to the standard SAC training procedure, the failure rates of SAC agents are reduced by more than three times in most settings without sacrificing agent performance under ordinary evaluation, and it has minimal impact on convergence speed.
In Figure 1c, our STA agent is uniformly successful on most states from 6 stranger agents.
R ELAY E VALUATION
In this section, we first introduce the definition of relay-evaluation in Sec.2.1.
Then we conduct extensive experiments to evaluate the relay-generalization of representative RL algorithms in Sec.2.3.
2. N OTATIONS
Single-player environments of RL are normally formalized as a Markov decision process (MDP).
It can be defined by a tuple M = ⟨S , A , T , R, d 0 ⟩ , where S is the state space,
A is the action space, T ( s t +1 | s t , a t ) :
S ×
A × S (cid:55)→ R is the transition function that indicates the probability of reaching state s t +1 after playing action a t on state s t ; R ( s t , a t ) : S × A (cid:55)→ R is the reward function, and d 0 ( s 0 ) is the distribution of the initial state s 0 .
With d 0 , we can define the initial state set S
as ∀ s ∈ S 0 ⇔ d 0 ( s ) > 0 .
We define an agent’s policy as π ( s ) which simply outputs the best action.
For each π ( s ) , we define its the trajectories distribution as T π .
Note that, a trajectory τ ∼ T π is a list of tuple [( s 0 , a 0 , r 0 , s 1 ) , ( s 1 , a 1 , r 1 , s 2 ) , . . .
, ( s T − 1 , a T − 1 , r T − 1 , s T )] , where s 0 ∈ S 0 and for each tuple T ( s t +1 | s t , a t ) > 0 and R ( s t , a t )
=
r t .
We named the return of a trajectory τ as the sum of all the rewards of a trajectory (cid:80) T − 1 i =
r i , where T is the length of the trajectory and r i is the reward of playing a i at s i .
2. P ROBLEM D EFINITION
The goal of relay-evaluation is to evaluate test agent π test performance on any “controllable” state in the state space S .
Here, we define “controllable” state as:
Definition 2.1 (Controllable states)
A state s ∈ S is controllable if there exists a trajectory τ = { . . .
, ( s t = s, a t , r t , s t +1 ) , . . .
} where s is one of its states s t , and the trajectory has a high return (cid:80) T −i =
r i and t < = T − L .
Here T is the length of the trajectory, L is the number of steps the agent runs after state s .
It is used to ensure it is possible to play L time steps starting from s without being terminated by the environment.
Particularly, we focus on the case of catastrophic failures when starting with these controllable states.
In our setting of continuous control environments, we define catastrophic failure as the environment being terminated by the simulator (e.g., the agent completely falls down), but one can also define catastrophic failure as getting an extremely low return.
Conceptually, controllable states are those “decent” states where catastrophic failures should not happen.
We only evaluate the test agent on controllable states since there may not exist a high return policy for a random state since some states are unrecoverable.
To find controllable states, we propose to use independently well-trained agents (which we refer to as “stranger” agents) that can generate high return trajectories.
Therefore, in the relay-evaluation, we evaluate a test agent π test with the controllable states on the trajectories of another agent π gen .
We first use π gen to generate M trajectories and select η × M of them with top returns, where η ∈ [0 , 1] .
We then sample states s t , t ∈ { 1 , 2 , . .
, T − L } from those high return trajectories and regard those states to be controllable.
If the test agent can continue playing for L time steps without failing from a state s t , then we say that the test agent passes the relay-evaluation on this state s t .
After testing π test on all the controllable states generated by π gen , we can then compute the failure rate and average return.
Note that we focus on the failure rates in most of the tables since it’s surprising that most agents encounter unexpected high failure rates under relay-evaluation, while the average return is also reported in some cases to reflect the average performance.
For a test agent, we conduct relay-evaluation on multiple generating agents, including agents independently trained by the same or different algorithms, as detailed below.
2. R ESULTS OF R ELAY -E VALUATION ON EXISTING ALGORITHMS
Experiment setup.
We conduct our experiments in four Mujoco environments in OpenAI Gym:
Humanoid-v3, Walker2d-v3, Hopper-v3, and Ant-v3 with the standard setting of 1,000 steps.
We do not include HalfCheetah-v3 since the simulator does not terminate even when the robot has already fallen over, so there is no standard way to determine catastrophic failures.
We first select three popular algorithms, including Soft Actor-Critic (SAC) (Haarnoja et al., 2018), Twin Delayed DDPG (TD3) (Fujimoto et al., 2018), Proximal Policy Optimization (PPO) (Schulman et al., 2017), as well as two robust training methods aiming to improve agent performance under perturbed observations,
SA-PPO and ATLA-PPO (Zhang et al., 2020; 2021).
Note that the robustness of SA-PPO and
ATLA-PPO agents are not aligned with the generalization setting in our paper, but they are included to see if there exists a connection between robustness and generalization.
All the agents have a
[Table]
decent average return under the ordinary evaluation which samples the initial state s 0 according to distribution d 0 .
We independently train N = 10 agents for each algorithm in each environment.
Every agent takes turns being the test agent and the stranger agent of other agents.
For each agent π gen , we first generate M = 200 trajectories and keep the top 100 trajectories that have a higher return.
For each high-return trajectory, we sample K = 5 states for other agents to test.
Generalization Results of Existing Algorithms.
For each agent, we conduct relay-evaluation using the trajectories generated by all the other agents as stranger agents, trained by either the same or different algorithms.
To investigate the failure rate of test agents trained by Algorithm A when testing the trajectories of stranger agents trained by Algorithm B, we report the average failure rate for each
A-B pair in Table 1.
Each row in Table 1 shows the results of stranger agents trained by a certain algorithm
B. The “Reference” column shows the failure rates of the stranger agents starting from their own controllable state, which is very low.
This indicates that these states are indeed controllable.
For the remaining columns, we show the failure rates of relay-evaluation, and clearly, we observe that all the agents have significantly higher failure rates than the reference failure rates.
For example, in Humanoid, SAC agents have a 38% of failure rate on other SAC agents’ trajectories and even higher failure rates when the stranger agent is trained by PPO or SA-PPO.
In general, SAC agents have the lowest failure rate across all environments.
This is probably due to the entropy term added to their goal function, which promotes exploration.
However, according to our experiments, their state distribution still collapses into a small set of successful trajectories (Figure 1a) and can not handle other agents’ successful trajectories properly.
SA-PPO and ATLA-PPO are robustly-trained agents that perform well under perturbations on state observations.
This is reflected in its smaller failure rate on Humanoid and Hopper.
However, they perform badly when taking the reins from other agents, showing relay generation is quite different from adversarial robustness since we care about all controllable states instead of slightly perturbed states.
Additionally, we observe that Ant-v3 is relatively easy because the robot has four legs, which is easier to keep the robot from falling.
The hardest environment is Humanoid-v3 since it has the most complicated robot to balance.
The failure rate of TD3 even reaches 96.2% on SA-PPO’s trajectories.
Why do agents fail?
To investigate why the agents perform poorly on relay evaluation, we visualize the state distribution generated by different agents.
We select the best two Humanoid agents of SAC,
TD3, and PPO with the highest average return from the same agents used in Table 1.
We generatetrajectories for each agent and sample 5 , 000 states from them.
We then use t-SNE (Van der Maaten & Hinton, 2008) to reduce the dimension of all the states into 2D and visualize them in Fig.
1a.
We color each dot according to which agent generate the states.
For example, the best and the second best SAC agent’ states are colored red and blue.
We can see that the states of the six agents are well separated, which means that their distributions are very different, even when trained with the same algorithms.
The results hold the same in all the environments.
This observation suggests that the state
[Table]
distribution generated by different agents is quite different, with almost no overlap.
Therefore all the other agents’ trajectories are out-of-distribution for a particular agent.
How do the agents fail?
We directly observe how the agents control the robot in Mujoco to understand how the agents fail and the implications of passing the relay-evaluation.
Fig.
2 shows the snapshots of some different SAC agents.
We observe that the agents normally maintain the balance of the upper part of the robot by fixing it to a special posture.
However, different agents’ postures are different, and they use different ways to keep the balance.
Therefore, when an agent takes over the control from another agent, it needs to correct the posture to its own.
This indicates that if an agent can pass the relay-evaluation, it generally knows how to correct the robot without losing balance, even if the starting posture is unseen.
Can the agents detect that it is going to fail?
Out-ofdistribution (OOD) detection has long been an important question in both RL and supervised learning, and we hope to identify the states where the agents may fail using value functions.
We conduct this experiment on SAC since it has the best performance in most of the settings.
Since
SAC itself does not include a state value function, we use
Q ( s, π ( s )) to serve as its evaluation of a given state, where
Q and π are its action-value function and its policy.
The results shown in Table 2 are separated into two parts.
The “Failure States” part shows the results of states that the test SAC agents failed on, and the states that the test SAC agent passed is shown in “Success States”.
The “Reference
Return” and “SAC Ret” columns show the average return of the stranger agent and the test SAC agent on the states for extra L = 500 steps.
The “SAC Q” shows the output
Q ( s, π ( s )) .
We can see that the Q ( s, π ( s )) of the failure states is lower than the success states.
We also notice that when the SAC agents succeed, they can perform almost as well as the stranger agents.
However, the prediction Q ( s, π ( s )) of the success state is way lower than the prediction Q ( s, π ( s )) on its own states: Humanoid: 590 ± 21 , Walker2d: 522 ± 73 , Hopper: 363 ± 20 , and Ant: 480 ± 24 .
This indicates that the Q function Q ( s, π ( s )) gives some indications about OOD and failing states.
We will use this information to design our algorithms.
I MPROVING THE G ENERALIZATION TO O THER A GENTS T RAJECTORIES
In this section, we aim to improve the relay-generalization of the existing algorithms.
We first investigate a naive method in Section 3.1.
Next, based on the investigation, we proposed our Self-
Trajectory Augmentation (STA) in Section 3.2.
Finally, we conduct the experiments to evaluate both methods in Section 3.3.
3. A N AIVE M ETHOD : L EVERAGING P RETRAINED A GENTS ’ T RAJECTORIES
To improve relay-generalization, a naive way is to let the agent “see” at least some of the trajectories generated by other agents during training.
One straightforward solution is to add the other agents’ tuples ( s t , a t , r t , s t +1 ) into the replay buffer as training data.
However, most RL algorithms are not able to handle out-of-distribution actions except offline RL (Levine et al., 2020).
Therefore, our naive method lets the agent “see” other agents’ trajectories by letting the agent have chances to start playing from the controllable states of some pretrained agents during training.
Before training, we first pretrain additional N a agents.
Next, we use states of those agents’ top η naive = 0 .
75 trajectories as controllable states S η naive pretrain .
During training, whenever the agent starts to generate a new trajectory, besides generating a new trajectory normally, it has p 0 probability to uniformly sample a state s pretrain from S η naive pretrain as the initial state of this new trajectory.
Starting from s pretrain , once the agent successfully plays L r steps, we will consider it knows how to play from s pretrain and let the agent start a new trajectory.
Our naive method is similar to training in an augmented environment that has a more diverse initial set state.
Therefore, we do not need to modify the training algorithm.
3. S ELF -T RAJECTORY A UGMENTATION (STA)
1: Variable: Q , π , S STA , D 2:
Parameter: p 0 , L r , d3: for each iteration do 4: x ∼ U (0 , 1) 5: if x > p 0 then 6: s 0 ∼ d7:
τ = gen_trajectory(s 0 , L max ) 8: else 9: s
= select_state( S STA ) 10: τ = gen_trajectory(s 0 , L r ) 11: for s in τ do 12:
D
= D ∪ { s } 13: if is_qualified(s) then 14:
S STA =
S STA ∪ { s } 15: train Q , π with D
Although the naive method can improve the generalization of the agents (Section 3.3), training additional agents is too time-consuming.
Hence, we propose Self-Trajectory
Augmentation (STA), which only uses the states of the agent’s own historical trajectories to augment the initial set.
The framework STA algorithm is shown in Alg.
For the variables, Q, π are the agent’s action value and policy that need to be trained; D is the replay buffer to store the training data; S STA is the set that stores the historical states during training; d 0 is the original initial state distribution.
At each iteration, with probability 1 − p 0 (line 5-7) we generate the trajectory as usual, which means sampling sfrom d 0 and letting the agent plays for L max steps (e.g., in Mujoco, L max = 1 , 000 ).
Otherwise, with probability p 0 (line 9-10), we select a state form S STA and let the agent plays for L r steps.
Since the failures usually happen early after taking over from another trajectory, we found a small L r ≪
L max is sufficient for improving relay generalization.
After generating the trajectory, we will store the trajectory into the replay buffer (line 12) and store the qualified state (line 13) into S STA (line 14).
Definition of qualified state (Alg. 1 line 13)
Ideally, a state s is qualified if it is controllable.
However, during training, we cannot simply use the return of the trajectory to define if its middle states are controllable.
This is mainly due to the exploration during training – the agent may play pretty well at the beginning but ends up having a low return because the algorithm tries to explore a bad action.
In this case, we still want to include the beginning states into S STA .
Hence, we use the sum of the next λ = 50 rewards as the scoring function score ( s t ) = (cid:80) t + λ − 1 i =
t r i to measure whether a state is controllable.
If a state’s score is among the top η STA ratio of the states generated in the latest epoch S latest , the state will be added into S STA .
For convenience, we maintain a threshold ω so that η STA ratio of the states in S latest has a higher score than ω .
Finally, we define the function is_qualified( s ) in line 13 as if score ( s ) > ω .
Note that if the rewards of the environment are sparse, one might consider other scoring functions.
Select a state from S STA (Alg. 1 line 9).
We select a state from S STA with two steps.
First, we random sample N c states that score ( s ) × γ ≥ ω max form S STA as candidates, where ω max is the largest ω encountered during training.
Although we only add states with score ( s ) > ω into S STA ,
[Table]
as the agent becomes stronger, ω will also increase.
Some of the states in S STA will be considered uncontrollable since their score are too low.
Hence, we use a ratio γ > = 1 to avoid low-score states ( < ω max /γ ) being our candidates.
For the second step, among the candidates, we want to select a state with which the current agent is unfamiliar.
Based on the results in Table 2, we use Q ( s, π ( s )) to predict which state is harder for the current agent.
That is, we select the state with the lowest
Q ( s, π ( s )) from the candidates as the new initial state (line 9).
3. E XPERIMENTAL R ESULTS
In this section, we evaluate the agents trained by our methods under the same settings and strange agents we used in Table 1.
Since SAC performs best in Table 1, we only apply our methods based on it.
In the following paragraphs, we first show that the naive method can improve the relay-generalization of SAC.
Next, we show that our STA can also improve the SAC agent without slowing the training process.
Moreover, STA is two times better than the naive method on the hardest controllable states.
The naive method.
In this experiment, we evaluate whether using pretrained models can improve the relay-generalization of an agent.
For the experimental settings: we add the controllable states in the top η naive = 0 .
75 ratio of each pretrain agent’s trajectories into the starting set; we use L r =and p
≈ 0 .
9 to increase the sampling frequency of the starting set.
The results of using different numbers N a of pretrained agents are shown in Table 3.
Although the pretrained agent is trained by
SAC, we observe that the failure rates on all kinds of algorithms decrease as more pretrained agents are included in S η naive pretrain .
This indicates that the method can successfully improve relay generalization.
However, in Humanoid, the performance has converged when N a = 8 .
We think it is because
Humanoid is so complex that adding more agents still cannot cover all the playing styles.
Hence, the failure rates do not keep dropping after N a ≥ 8 .
We also notice that when only using 1 pretrained agent ( N a = 1 ), the failure rate surprisingly increased in the Humanoid environment; this might be because the agent will overfit the only pretrain agent’s trajectories.
Additional results like using different L r = { 50 , 100 , 200 , 500 } are shown in the Appendix.
Self-Trajectory Augmentation (STA).
In this experiment, we compare the base SAC, our naive baseline with 16 pretrained agents, and our STA method with tuned parameters (see Appendix).
Note that the environment step number of training each agent is all three million.
Therefore, the training time of STA will be almost 17 times faster than the naive method.
The comparison of both the average return and failure rate are shown in Table 4.
We first show the performance under ordinary evaluation ( s 0 ∼ d 0 ) of each environment (“ordinary” rows).
The results show that both the naive method and STA have almost equal or even higher ordinary performances, although agents are trained under augmented MDP, which is different from the original
MDP.
Moreover, the failure rates of STA in the ordinary setting are also lower than SAC in all the environments, which means that the STA agents are more robust.
Next, we show the relay-evaluation results on different stranger algorithms.
According to the rest of the rows (except the “ordinary” row) in Table 4, we notice that STA has three times lower failure rates compared to SAC in most of the settings.
In addition, since we only sample N c = 5 candidates, the training time is almost the same as SAC.
Hence, we claim that we achieve a better relay-generalization than SAC without losing the ordinary performance and the converging speed.
We also compare the results between STA
Since all the agents are trained with three million environment interactions, training an STA agent is 15.9 times faster than a Naive N a =16 agent.
The “ordinary” rows show that STA will not decrease the regular performance.
The rows except “ordinary” show that STA can achieve significantly lower failure rates under relay-evaluation compared to SAC and achieve performance similar to
Naive N
[Table] [Table]
and the naive method.
We find out that our method has a better performance in harder environments.
For example
[Table]
STA has only 25 .
2% failure rate on the Humanoid SA-PPO.
However, in the easiest environment, Hopper, the naive method with N a = 16 pretrained agents has a twice lower rate than
STA.
We suggest it is because Hopper has the simplest robot that only has one lag.
Hence, we suggest that most of the hoping styles have a similar version in the 16 pretrained agents.
Therefore, the naive method can easily play on other agents’ trajectories.
Besides showing the best performance of STA, we also conduct some ablation studies.
First, we directly observe how STA takes over the control of other agents.
Same as SAC agents, we notice that STA agents also run with one kind of posture.
Even when the STA starts from other agents’ posture, it will change it into its own and keep running.
We consider this a good thing.
For example, if a test agent is faster than the stranger agent, we will want the test agent to run in its own way, which is faster.
We also conduct an experiment on different numbers of candidates N c to show the importance of starting the trajectories from the states that has a lower Q ( s, π ( s )) .
The results are shown in Table 5.
We first look at the column where N c = 1 .
It shows the results of simply randomly selecting a state from the initial set as we did in the naive method.
Compared to SAC, N c = 1 is only slightly better.
However, as we use a larger N c , the result becomes comparable to the naive method, especially on Humanoid.
We also notice that different N c have little effect on Hopper.
This is because there is only a small Q ( s, π ( s )) difference between the familiar states and the unfamiliar state according to Table 2.
Hence, we should consider other methods in Hopper.
R ELATED W ORK
Many methods have been designed to evaluate the generalization of an RL agent (Kirk et al., 2021).
They can be categorized by which part of the RL environment are different between training and testing, including (1) the environment, (2) the observation emission function, (3) the transition function, (4) the reward function, and (5) the initial state set.
For (1) , researchers studied training and testing in totally different environments.
For example, Cobbe et al.
(2019) and Dennis et al.
(2020) randomly create different environments and separate them into training and testing environments.
For (2) , researchers change the surface of the objects in the environment (Tobin et al., 2017; Zhang et al., 2018b;
Zhao et al., 2019; James et al., 2019).
For (3) , researchers change the parameter of the transition function like the gravity strength or the weight of the ball, or add an external force to the agent.
(Packer et al., 2018;
Pinto et al., 2017; Zambaldi et al., 2018)
For the (4) , researchers (Mishra et al., 2018; Dosovitskiy et al., 2017; Chevalier-Boisvert et al., 2019; Lynch & Sermanet, 2021) normally want to see if the agent can adapt quickly from one task to another with the general knowledge learned in the first task, which is similar to Meta-RL (Finn et al., 2017).
Finally, for (5) , the last category that only selects different initial states (Portelas et al., 2019; Peng et al., 2018;
Kaplanis et al., 2018) is where our relay-generalization belongs.
This class of generalization does not require research to modify the environment.
In one of the most related previous work, Zhang et al.
(2018a) only train on limited states in the initial set d 0 and test on the other states in the initial set d 0 .
Their results show that if a PPO agent only trains on less than 5 states, it will be hard to avoid failing on other states in the initial set d 0 .
Another common way to categorize the research of RL generalization is by the relationship between the training and testing distribution.
The first category is Independent-and-Identical Distribution (IID), where both training and testing environment are sampled from the same distribution (Zhang et al., 2018a; van der Pol et al., 2020; Bica, 2021;
Arndt et al., 2020).
The second category is Out-of-Distribution (OOD), where researchers separate the environments into training and testing.
For example, in Sim2Real, the training is only in the simulator and the testing is in the real world.
Another example is curriculum learning (OpenAI et al., 2019; Huang et al., 2022; Bellemare et al., 2013; Muratore et al., 2020) where the agent trains in both easy and hard environments and only tests in a hard environment.
Our relay-generalization belongs to OOD, since even when the test agent and trajectory generation agent are trained by the same algorithm, Figure 1a demonstrates that they have very different state distributions in their trajectories.
To the best of our knowledge, the notion of controllable states and relay evaluation has not been discussed in the literature.
Beyond the generalization of RL, many other RL domains are related to our works.
First, in hierarchical reinforcement learning, Pateria et al. (2021); Brochu et al. (2010); Nachum et al.
(2018) train multiple agents with different small missions and let them complete a task together by taking turns.
However, in such a scenario, the agents are trained together.
Therefore, they are familiar with previous agents’ trajectories.
Second, both our naive method and the offline RL (Levine et al., 2020; Kumar et al., 2020; Ostrovski et al., 2021) leverage the trajectories of other agents.
Third, population-based training (Jaderberg et al., 2017; Derek & Isola, 2021) can be a potential way to generate the controllable states and increase relay-generalization.
Finally, the adversarial robustness of agents have been studied in the context of state, observation and reward perturbations (Huang et al., 2017; Ilahi et al., 2021;
Zhang et al., 2020; 2021;
Ding et al., 2022; Lan et al., 2022).
However, these studies focus on the performance of agents under imperceptible adversarial perturbations, which is totally different from our definition of relay generalization.
C ONCLUSION
We propose relay-evaluation, a proxy to evaluate RL agents’ generalization performance on controllable states in a fixed environment.
Relay-evaluation involves running an agent from the middle states of other independently-trained agents’ high-reward trajectories.
Extensive studies under the MuJoCo environment demonstrate that many representative RL algorithms have unexpectedly high failure rates under relay-evaluation, indicating a significant limitation of existing RL methods.
To overcome this challenge, we propose a novel method called Self-Trajectory Augmentation (STA), which improves the generalization of existing RL agents without impacting their ordinary performance.
This paper opens up a new research direction in RL, and there are several limitations to be addressed in the future.
For instance, how to find unexplored controllable states more effectively and how to “certify” that an agent can work (or at least won’t fail) on all controllable states are interesting future challenges.
A CKNOWLEDGEMENT
This work is supported in part by NSF under IIS-2008173, IIS-2048280 and an Okawa research grant.
A D ETAILS FOR E XPERIMENTAL S ETUP
For SAC and TD3, we follow the implementation of OpenAI Spinning Up (Achiam, 2018), which uses networks of size (256, 256) with relu units and 3 million environment interactions.
For PPO, we follow the implementation of Engstrom et al., 2019 since its PPO performance is higher than
OpenAI Spinning Up.
The total number of environmental interactions is one million.
For SA-PPO and ATLA-PPO, we follow the implementation in (Zhang et al., 2021).
SA-PPO and ATLA-PPO are representative algorithms in terms of observation robustness.
That is, the agent can function normally even if a small adversarial perturbation is added to the observation.
We include this kind of robust agent to show that our problem is different from the normal adversarial robustness problem in RL.
For the humanoid environment, the SA-PPO algorithm we used is the SGLD algorithm.
For other environments, we use ATLA-PPO.
For the naive method , both the pretrained agents and the agent follow the implementation of SAC of OpenAI Spinning Up.
Hence, training a naive method with N a agents requires N a + 1 times of training time.
The default parameter settings are: η navie = 0 .
75 since we want to include harder states; p 0 ≈ 0 .
9 , L r = 100 since we want to sample more initial states form S η naive pretrain .
For the STA method , we also follow the implementation of SAC of OpenAI Spinning Up.
Since the candidate number N c we use is less than 10, the training time is almost the same as training a
SAC agent.
The default parameter settings are: η STA = 0 .
75 since we want to include harder states as we did in the naive method; p 0 ≈ 0 .
9 , L r = 100 since we want to sample more initial states form
S STA ; λ
= 50 <
L r since we want to include the states in the trajectories that starts from states in
S STA .
For the STA parameters used in Table 4, we set the number of candidates N c = 5 ; we set the qualifying ratio γ = 1 . 0 in Humanoid and γ = 1 . 6 in Walker2d and Hopper
[Table]
B E XPERIMENTS OF OUR NAIVE METHOD WITH DIFFERENT PLAYING LENGTHS
In this section
[Table]
we show the results of our naive method with different limitations of the extra steps of the trajectories starting from starting set.
The results are shown in Table 6.
According to the table, the failure rate will increase when L r is too small or too large.
Since L r = 100 has the lowest failure rate in almost all settings, we will use
L r = 100 as the default in other experiments.
[Table]
C E XPERIMENTS OF STA WITH D IFFERENT C ANDIDATE C RITERIA
In this section, we evaluate the performance of using different γ .
In STA, we use ω max /γ to serve as the criteria to be the threshold of being one of the N c candidates.
Only states with score ω max /γ in
S STA can be the candidate of the initial state s 0 .
If γ is larger, then states with lower scores can also be the candidates
[Table]
The results different γ with N c = 5 are shown in Table 7.
According to the results, the best γ of each environment is different.
For example, in Humanoid, it is important to only samples the states that have higher scores as candidates (small γ ).
[Table]
D I NFINITY REPLAY BUFFER
In this section
[Table]
we show that our STA method is better than using an unlimited-size of replay buffer.
The original replay buffer size is one million.
The maximum size of an unlimited-size of replay buffer is three million since we only have three million environment interactions.
The results are shown in
Table 8.
According to the results
[Table]
SAC with an infinite replay buffer still has poor performance on relay evaluation compared to our method.
Moreover, the average return under ordinary evaluation is lower than the normal SAC, while our method is greater or equal to the normal SAC.
[Table]
E U SING A VERAGE R EWARD AS S CORING F UNCTION
In this section
[Table]
we show the STA result of using the average score score ( s t )
=
(cid:80) T − 1 i =
t r i / ( T − t ) instead of using the reward sum of next λ steps score ( s t ) =
(cid:80) t + λ − 1 i =
t r i .
In STA, the scoring function is a quick way to estimate if a state is controllable.
We only add the states that have high enough scores into the S STA .
According to the results, using the average reward can also reduce the failure rate compared to the original SAC.
However, its failure rate is still higher than using our score function (column STA)
[Table]
which uses the sum of the reward of the next λ step.
[Table]
F T HE FAILURE DEFINITION OF M UJOCO
In Mujoco, by default, most of the environments will terminate the simulation when the agent is unhealthy.
Normally, unhealthy means the robot has fallen over.
By doing so, the agent can start a new trajectory without wasting more time on the "unhealthy" trajectory.
Each environment has its own way of detecting unhealthy.
The Humanoid environment requires the z-position (height) of the robot need to be in a predefined range.
The Ant and Walker2d environment define the robot as unhealthy if any of the following happens:
• Any of the state space values is no longer finite.
• The height of the walker is not in a predefined range.
• The absolute value of the angle is not in a predefined range.
The Hopper environment defines the robot as unhealthy if any of the following happens:
• An element of observation[1:] is not in a predefined range.
• The height of the walker is not in a predefined range.
• The absolute value of the angle is not in a predefined range.
Please see more details at https://www.gymlibrary.dev/environments/mujoco/.
G D EFINE F AILURE WITH R ETURNS
In this section, we define the failure of relay-evaluation according to the reward sum of the next
L = 500 steps instead of using the failure defined by Mujoco.
The first column shows the failure rate using the failure definition of the Mujoco.
For the remaining columns, we allow the agent to keep playing even if the current state is "failed," according to Mujoco.
In the end, we define a trajectory as a failure trajectory according to its return.
For example, for the second column, we define a trajectory as failed when its return is lower than 500.
Note that the average return of SAC should be 2828.
According to the results, no matter using which kind of definition of failure, SAC still has an unacceptable failure rate, especially on the SA-PPO.
This shows that the termination criteria in the
MuJoCo environments are not too strict and most of the terminated trajectories have no chance to get a high score.
[Table]
H S TATE D ISTRIBUTION OF AN STA A GENT ’ S O LD T RAJECTORIES
In this section, we plot the historical states of an STA agent with the states we showed in Fig 1a.
As the figure shows, although the distribution of historical states of the STA agent is more diverse than the distribution of other agents’ states, there are still many areas that it does not cover.
Hence, this suggests that the STA agents have a better performance on relay evaluation due to being more general.
I E VALUATING SAC WITH D IFFERENT L IN R ELAY -E VALUATION
In this section
[Table]
we conduct relay-evaluations with different L .
We select SAC as our test agent.
Our result is shown in Table 11.
The results show that most of the failures happened in the first 100 steps after the test agent took over the control.
The results also show that for some of the cases, even if the agent does not fail for 100 steps, it may still fail in the next 400 steps.
[Table]
J G ENERATING C ONTROLLABLE S TATES WITH D IFFERENT H YPERPARAMETERS
In this section
[Table]
our goal is to generate more controllable states to test our agents by training with different hyperparameters.
We first train 40 SAC agents with random hyperparameters and use the top 10 agents to serve as stranger agents.
The hyperparameters that we will randomly choices are: the learning rate of the policy π , the learning rate of the Q function, the batch size, the network structure, the training frequency
[Table]
and the α in the
SAC goal function.
The results are shown in Table 12.
According to Table 12, we observe that the states generated by random hyperparameters are easier for other agents to pass the relay-evaluation, especially when evaluating SAC and TD3 agents.
A potential reason is that some of these hyperparameters lead to worse policies and generate slower trajectories, and it’s easier for the target agent to take over from those slower states.
[Table]
K T HE ORDINARY RETURN OF THE TEST AGENTS IN T ABLE
This section shows the average returns of the agents we used in Table
[Table] [Table]