Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation
Abstract
The generalization gap in reinforcement learning (RL) has been a signiﬁcant obstacle that prevents the RL agent from learning general skills and adapting to varying environments.
Increasing the generalization capacity of the RL systems can signiﬁcantly improve their performance on real-world working environments.
In this work, we propose a novel policy-aware adversarial data augmentation method to augment the standard policy learning method with automatically generated trajectory data.
Different from the observation transformation based data augmentations, our proposed method adversarially generates new trajectory data based on the policy gradient objective and aims to more effectively increase the
RL agent’s generalization ability with the policyaware data augmentation.
Moreover, we further deploy a mixup step to integrate the original and generated data to enhance the generalization capacity while mitigating the over-deviation of the adversarial data.
We conduct experiments on a number of RL tasks to investigate the generalization performance of the proposed method by comparing it with the standard baselines and the stateof-the-art mixreg approach.
The results show our method can generalize well with limited training diversity, and achieve the state-of-the-art generalization test performance.
1. Introduction
Beneﬁting from the power of deep neural networks, deep reinforcement learning (RL) has recently demonstrated incredible performance on many human-level challenging tasks.
In addition to traditional board games like Go (Silver et al., 2016; 2017), deep reinforcement learning agents have defeated professional human players in large scale video games like StarCraft (Vinyals et al., 2017) and Dota 2 (Berner et al., 2019).
Meanwhile, deep RL systems also suffer from the vulnerabilities of deep neural networks such as overﬁtting and data memorization, which often induce generalization gaps between the training and testing performances of the RL agents.
The generalization gap prevents the RL agents from learning general skills in a simulation environment to handle the real-world working environments (Cobbe et al., 2020; Tobin et al., 2017).
As a result, deep
RL agents are reportedly performing poorly on unseen environments, especially when the training environments lack diversities (Zhang et al., 2018b;a; Cobbe et al., 2020;
Song et al., 2019; Cobbe et al., 2019).
Towards the goal of reducing the generalization gap, previous works have exploited conventional data augmentation techniques such as cropping, translation, and rotation to augment the input observations and increase the diversity of training data in RL (Shorten & Khoshgoftaar, 2019; Cobbe et al., 2019; Lee et al., 2019;
Laskin et al., 2020).
Some have also explored the selection of data augmentation techniques (Raileanu et al., 2020; Jiang et al., 2021) and improved the RL architectures (Raileanu & Fergus, 2021;
Ball et al., 2021).
Recently, a mixture regularization method has been introduced to learn generalizable RL systems (Wang et al., 2020), which deploys Mixup to increase the data diversity and yields the state-of-the-art generalization performance.
However, these methods work speciﬁcally at the level of the input observation data without taking the RL system into consideration, which prevents them from generating data that are most particularly suitable for the target RL system and hence could limit their performance’s improvement.
In this paper, we propose a novel policy-aware adversarial data augmentation method with Mixup enhancement (PAADA+Mixup) to improve the generalization ability of policy learning based RL systems.
This method ﬁrst generates adversarial augmenting trajectory data by minimizing the expected rewards of the given RL policy based on the initial observed source trajectory data.
Next it combines each adversarial augmenting trajectory and the corresponding ob- servation source trajectory together by randomly selecting each observation step from one of them.
The combined trajectory data can then be used to update the policy of the RL system.
As deep RL systems typically inherit the vulnerability of deep neural networks to adversarial examples (Lin et al., 2017), some previous works have investigated the topic of adversarial attacks in deep RL (Zhao et al., 2020; Gleave et al., 2019).
Our work however is the ﬁrst that deploys adversarial data augmentation in online
RL systems to improve their generalization capacity.
The proposed adversarial augmentation is conducted in a policyaware manner to induce direct impact on the to-be-learned policy.
Moreover, we further deploy a mixup operation on the combined trajectory to enhance the robustness and generalization of the RL system.
We conduct extensive experiments on several RL tasks to investigate the proposed approach under different generalization test settings.
The experimental results show the proposed PAADA greatly outperforms the strong RL baseline method, proximal policy optimization (PPO)
(Schulman et al., 2017).
With the mixup enhancement, PAADA+Mixup can achieve the state-of-theart performance, surpassing mixreg (Wang et al., 2020) with notable performance gains.
The main contributions of the proposed work can be summarized as follows:
• We introduce adversarial data augmentation to deep RL and develop the very ﬁrst policy-aware adversarial data generation method to improve the generalization capacity of deep RL agents.
• We integrate the generalization strengths of both adversarial data generation and mixup and demonstrate superior empirical performance than using either one of them alone.
• We conduct experiments on the Procgen benchmark with different generalization settings.
Our proposed method demonstrates good generalization performance with limited training environments and outperforms the state-of-the-art mixreg approach.
2. Related Works
Generalization gap has been an increasing concern in deep reinforcement learning.
Recent studies (Zhang et al.
, 2018b;a) show that the main cause of generalization gap is the over-ﬁtting and memorization inherited from deep neural networks (Arpit et al., 2017).
Data augmentation is a conventional technique for solving over-ﬁtting in deep learning (Shorten & Khoshgoftaar, 2019).
In recent studies, some data augmentation approaches broadly applied in deep learning have been brought into reinforcement learning.
Cobbe et al.
(2019) introduce the cutout technique into deep
RL, where data are generated by partially blocking the input observations with randomly generated black occlusions.
Lee et al.
(2019) propose a randomized convolutional network to perturb the input observations.
Laskin et al.
(2020) propose to integrate the commonly used data augmentation skills such as cropping, translation, and rotation to improve the generalization of RL.
Ball et al.
(2021) introduce augmented world models to speciﬁcally address the generalization of model-based ofﬂine RL problems.
Raileanu et al.
(2020) propose a data-regularized actor-critic approach to regularize policy and value functions when applying data augmentation, whereas the upper conﬁdence bound (Auer, 2002) is borrowed to select the proper data augmentation method.
Wang et al.
(2020) propose a simple but efﬁcient mixture regularization approach, mixreg , to improve the generalization capacity in RL systems.
Following the mixup method (Zhang et al., 2017) in supervised learning, mixreg generates new observations from two randomly selected observations through linear combinations.
The deep RL agents trained on mixreg augmented observations demonstrate signiﬁcant improvements in generalization.
Inspired by the success of data augmentation on generalization of deep reinforcement learning, we introduce a novel adversarial data augmentation technique to deep RL.
Deep neural networks are known to be vulnerable to adversarial examples (Szegedy et al., 2013).
Many previous works have studied the attack and defence strategies of deep neural networks regarding adversarial examples (Yuan et al., 2019;
Madry et al., 2017; Akhtar & Mian, 2018).
Lin et al.
(2017) show deep reinforcement learning agents inherited the vulnerability of deep neural networks to adversarial examples.
They propose two tactics to perform adversarial strategies to attack RL agents: the strategically-timed attack minimizes the agent’s reward for small subsets of time steps, while the enchanting attack lures the agent to a speciﬁc target state.
Following this, a few works have further contributed to this rising topic of adversarial attacks in deep RL.
Zhao et al.
(2020) propose an approach to generate adversarial observations without previous knowledge on the network architecture and RL algorithm of the deep reinforcement learning agent.
Instead of adding perturbations to the observations, Gleave et al.
(2019) propose to train the agent with an adversarial policy.
Distinct from adversarial attacks, our study focuses on adversarial data augmentation.
Adversarial data augmentation has been investigated in a number of works on supervised learning, but has not been explored to enhance generalization in deep RL systems.
Goodfellow et al.
ﬁnd the linear property of deep neural networks is vulnerable to adversarial perturbations and propose to train the supervised model with adversarial examples to improve generalization.
Sinha et al.
(2017) propose a Lagrangian formulation of adversarial perturbations in a Wasserstein ball (Lee &
Raginsky, 2017) to enhance the robustness of deep learning models.
Volpi et al.
(2018) further propose to exploit adversarial data augmentation for domain adaptation with unknown target domains.
3. Method
This study focuses on increasing the generalization ability of a deep RL agent.
Following the previous generalization study in RL (Wang et al., 2020), we consider the following
RL setting.
The agent is trained on a set of n environments {K 1 , ..., K n } sampled from a distribution p ( K ) to learn an optimal policy π ∗ , and then tested on another set of environments { ˆ K 1 , ..., ˆ K m } sampled from p ( K ) .
Its generalization performance is measured as the zero-shot expected cumulative reward in the test environments:
[Equation]
where τ denotes a trajectory ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , · · · , r T ) ,
D testπ ∗ denotes the distribution of τ in the test environments under policy π ∗ , and γ ∗ ∈ (0 , 1] is the discount factor.
Moreover, we assume the training environments can be much fewer than the test environments such as n = (cid:100) ξm (cid:101) with ξ ∈ (0 , 1] .
The goal is to train an optimal policy π ∗ that can generalize well in terms of the expected cumulative test reward above in Eq.(1).
Towards this goal, in this section we present a policy-aware adversarial data augmentation with mixup enhancement (PAADA+Mixup) method for the
RL training process.
3.1. Policy-Aware Adversarial Data Augmentation
When the training and test environments are different, a standard RL learning algorithm, e.g, Proximal Policy Optimization (PPO)
(Schulman et al., 2017), will inevitably suffer from the domain gap between the training and test environments, and demonstrate generalization gaps between the training and test performances.
Inspired by the effectiveness of adversarial data augmentation in supervised learning (Volpi et al., 2018), we propose to augment the deep
RL process by generating adversarial trajectories from the current policy, aiming to adaptively broaden the experience of the RL agent and increase its generalization capacity to unseen test environments.
Speciﬁcally, given the current parametric policy π θ with parameter θ , in each epoch of the policy optimization based
RL training, the standard procedure is to collect a set of trajectories { τ 1 , · · · , τ n } from the training environments, and then update the policy parameter θ by performing gradient ascent with respect to the policy optimization objective over the observed trajectories.
For policy gradient, the following surrogate objective is often used:
[Equation]
where ˆ E t [ · ] denotes the empirical average over the set of transitions in the collected trajectories; ˆ A t is the estimated advantage function at timestep t , and can be approximated as ˆ A t =
r t − V ( s t ) , where V ( s t ) is the value function at state s t (Degris et al., 2012).
For the more advanced high-performance policy gradient algorithm PPO (Schulman et al., 2017), a clipping modulated objective is typically used:
[Equation] [Equation]
where ρ θ = π θ ( a t | s t )
π θ (cid:48)
( a t | s t ) and (cid:15) is a small constant.
This PPO objective regularizes the new policy π θ from being severely deviated from the previous policy π θ (cid:48) and aims to avoid large destructive policy updates associated with the vanilla policy gradient.
Although PPO has demonstrated great performance in standard RL, it may be overly bounded to the available training observations and hence yields poor test performance in the generalization settings.
We therefore propose to expand the observation space by generating adversarial examples based on the collected trajectories.
3.1.1. A DVERSARIAL T RAJECTORY G ENERATION
For each observed trajectory τ , we generate an adversarial example for each of its observation points (i.e., transitions),
P t = ( x t , y t ) with x t = s t and y t
= ( a t , r t ) .
That is, we ﬁnd the worst example P =
( x, y ) in the close neighborhood of the current point P t = ( x t , y t ) by minimizing the cumulative award objective L the RL agent needs to maximize, such as
[Equation]
where D ( · , · ) denotes a distance metric such as the Wasserstein distance and the constraint bounds the point P to be within the neighborhood of P t .
For two points P
=
( x, y ) and P t = ( x t , y t ) , the Wasserstein distance D ( P, P t ) can be speciﬁed as follows through a transportation cost c (Volpi et al., 2018):
[Equation]
Moreover, the constrained problem in Eq.(4) can be equivalently reformulated as the following regularized optimization problem with a proper Lagrangian parameter γ :
[Equation]
Input: π θ , V , ( s t , a t , r t ) ; stepsize η , maximum step number K max , tolerance (cid:15) a
Output: adversarial observation s 1: s = s t 2: for k = 1 , ..., K max do 3: if (cid:107)∇ s [log π θ ( a t | s )( r t − V ( s ))
+ γ ( s − s t ) 2 ] (cid:107) 2 < (cid:15) a then break 4: s = s − η ∇ s [log π θ
( a t | s )( r t − V ( s ))
+ γ ( s − s t ) 2 ]
As this point generation is conducted under the current policy π θ and does not involve policy update, we use the policy gradient objective L PG as the objective L for adversarial example generation due to its simplicity and easy computation.
Therefore for each observation point ( s t , a t , r t ) in the collected trajectory data, its corresponding adversarial point (ˆ s t , a t , r t ) will be generated as follows:
[Equation]
We can solve this generation problem by performing gradient descent with a stepsize η and a maximum step number
K max .
The algorithm is shown in Algorithm 1.
With this simple procedure, for each observation trajectory τ , we can produce a corresponding adversarial trajectory ˆ τ by generating an adversarial point (ˆ s t , ˆ a t = a t , ˆ r t = r t ) for each observation point ( s t , a t , r t ) in τ .
3.1.2. T RAJECTORY A UGMENTATION
Given the observed source trajectory τ and the generated adversarial trajectory ˆ τ , simply appending one after the other to fed to the deep RL agent for training turns out not to be a suitable solution, as it may cause the policy parameter update to dramatically switch between very different directions.
Moreover, the suitable degree of augmentation could also vary for different RL tasks.
Here we propose to augment the source trajectory τ with the adversarial trajectory ˆ τ by combining them into a new trajectory τ with an augmentation degree ν ∈ [0 , 1] .
Speciﬁcally, we construct the new trajectory τ by randomly selecting (cid:98) ν · | τ |(cid:99) points (transitions) from the adversarial trajectory ˆ τ and taking the other (cid:100) (1 − ν ) · | τ |(cid:101) points from the original trajectory τ .
In this way, we not only can better blend the adversarial points with the original observations, but also have control over the contribution degree of the augmentation data through the hyperparameter ν .
We deploy this policy-aware adversarial data augmentation scheme on the PPO method.
The overall training algorithm
Input: initial policy parameter θ , initial value function parameter φ , the pre-training epoch number
K pre , the augmentation degree ν
Output: trained policy π θ 1: for k = 1 , 2 , ... do 2:
Collect a set of trajectories { τ 1 , τ 2 , ..., τ n } by running policy π θ on the training environments 3: for τ i in { τ 1 , τ 2 , ..., τ n } do 4:
Compute the advantage estimates { A t } for all the t transition points in τ i 5: if k < K pre then 6: ¯ τ i =
τ i ; continue 7: for t = 0 , 1 , 2 , ..., | τ i | − 1 do 8:
Generate the adversarial state
ˆ s t with Eq.(7) 9: ˆ A t =
r t − V φ (ˆ s t ) 10:
Add ( ˆ s t , a t , r t , ˆ A t ) into the augmentation trajectory ˆ τ i 11:
Combine τ i and ˆ τ
i into an augmented trajectory τ i with the augmentation degree ν 12: %[place holder for additional step] 13:
Update the policy function parameter θ by maximizing the PPO-Clip objective in Eq.(3) on the augmented trajectories { ¯ τ 1 , ¯ τ 2 , ..., ¯ τ n } 14:
Update the value function parameter φ on the augmented trajectories is depicted in Algorithm 2, which ﬁrst pre-trains the RL agent for K pre epochs and then performs adversarial data augmentation in the ensuing epochs.
3.2. Enhancement with Mixup
In addition to the adversarial data augmentation technique above, we consider further enhancing the diversity of the training data with a Mixup procedure.
Mixup generates data points through linear interpolation and has demonstrated effective performance in both supervised learning (Zhang et al., 2017) and reinforcement learning (Wang et al., 2020).
Here we propose to deploy the Mixup procedure on each augmented trajectory τ i generated above.
Speciﬁcally, for a trajectory τ i with | τ i | transition points, we ﬁrst make a copy of τ i as τ (cid:48)
i , and randomly shufﬂe the indices { 0 , 1 , · · · , | τ i |− 1 } into { I 0 , I 1 , · · · , I | τ i |− 1 } .
Then we linearly combine τ i and τ (cid:48)
i with the following mixup steps:
[Equation]
while ¯ a t is set as ¯ a t with probability λ and set as ¯ a (cid:48)
I t with probability 1 − λ .
The hyperparameter λ is sampled from a beta distribution λ ∼ B ( α, β ) .
Normally, the parameters of the beta distribution are set to α = β as suggested in (Zhang et al., 2017).
However, when the training environments are limited, it is beneﬁcial to have different α and β values to shift the mean value of the λ samples.
This Mixup step can be deployed on the augmented trajectory and performed in line 12 within the trajectory loop in Algorithm 2.
4. Experiments
We conducted experiments to validate the empirical performance of the proposed method under different generalization settings.
In this section, we report our experimental settings and results.
4.1. Experiment Setting
We conduct experiments on the Procgen benchmark (Cobbe et al., 2020), which contains procedurally generated environments designed to test the generalization ability of deep
RL agents.
Each environment takes visual input and has signiﬁcant change among different levels of the environments.
A Procgen environment can generate a maximum of 500 different levels for the RL generalization task.
We choose 4 game environments (starpilot, dodgeball, climber, fruitbot) from this benchmark as different RL tasks and treat different levels of each RL task as different training and test environments from the generalization perspective.
Follow the settings in (Cobbe et al., 2020; Wang et al., 2020), we do not limit the levels in the testing environments and use the total m = 500 levels for testing, while a relatively smaller number, n = (cid:100) ξm (cid:101) , of level environments, are sampled for training.
In particular, we consider different ξ values such as ξ ∈ { 0 . 25 , 0 . 5 , 1 } .
A smaller ξ value indicates a more difﬁcult generalization setting as the diversity of training environments is reduced.
We adopt PPO (Schulman et al., 2017) as our RL baseline, although the proposed methodology in principle can be generalized into other RL methods as well.
In addition, we compare our approach, PAADA+Mixup, to the stateof-the-art generalization method, mixreg, which has been shown to outperform the conventional data augmentation techniques in (Wang et al., 2020).
Following the Procgen benchmark, we adopt the same convolutional neural network architecture as IMPALA (Espeholt et al., 2018).
We use the mean episode return in each epoch of the zero-shot testing as the generalization evaluation metric on each of the four
Procgen benchmark games (starpilot, dodgeball, climber, fruitbot).
Moreover, following (Cobbe et al., 2020) we compute the mean normalized return over the four games to summarize the overall generalization performance.
Hyperparameters For the adversarial observation generation, the stepsize η is set to 10, the maximum number of steps K max is set to 50, and the tolerance (cid:15) a is set to 5 e − 6 .
For the training algorithm of the proposed approach,
PAADA, we set the pre-training epoch number, K pre , as 50.
That is, in the ﬁrst 50 training epochs, the deep RL agent is trained with standard PPO (Schulman et al., 2017).
After 50 epochs, our augmentation method is applied to enhance the generalizability.
In the evaluation plots, the epoch number does take these pre-training epochs into account.
The
Lagrangian hyperparameter γ in Eq.(6) and Eq.(7) is set to 0 .
In each training epoch, we collect n trajectories and each trajectory has 256 transitions.
For testing, m trajectories, one from each testing environment, are used.
The discount factor γ ∗ is set to 0.999.
Each experiment is tested on 3054 training epochs in total.
For the Mixup procedure, we use α = 0 .
2 and β = 0 .
2 for ξ = 1 , and increase β to β = 0 .
5 for ξ = 0 .
5 and to β = 1 for ξ = 0 .
4.2. Experiments with Full Set of Training Environments
In the ﬁrst set of less challenging experiments, for each game, we have ξ = 1 and use the full set of 500 training environments, while testing on unlimited environments.
For the proposed approach PAADA, we tested two different variants of it:
(1) PAADA ( ν = 0 . 5 ), which uses an augmentation degree ν = 0 . 5 and indicates that 50% of the observation transitions of the adversarial trajectory are randomly merged into the original observation trajectory.
(2) PAADA+Mixup ( ν = 0 . 5 ), which indicates the Mixup enhancement is added to PAADA ( ν = 0 .
We compared these variants with the PPO baseline and one state-of-the-art method, mixreg.
The training and test evaluation results on the four games are reported in Figure 1, where dotted lines are used to present the training evaluation results and solid lines are used to present the test evaluation results.
The test results demonstrate the working performance of the system.
The left four plots in Figure 1 report the comparison results on the four games separately.
We can see that the policy-aware adversarial data augmentation alone outperforms the baseline PPO:
PAADA ( ν = 0 . 5) greatly improves the generalization performance of PPO on the four games and the performance gain in terms of the mean normalized return is very notable.
With the Mixup enhancement, PAADA+Mixup ( ν = 0 . 5) further improves its generalization performance, and yields similar performance to the state-of-the-art method, mixreg, on three games, dodgeball, fruitbot and starpilot , and outperforms mixreg on climber with large margins.
The right plot in Figure 1 summarizes the overall performance over the four games in terms of a mean normalized return metric.
The mean normalized return is a useful metric that allows us to compare the overall performance of different reinforcement learning systems across multiple environments (Cobbe et al., 2020).
We can see from the plot that PAADA+Mixup ( ν = 0 . 5) outperforms all the other methods, including mixreg, with distinct performance gains.
These results clearly demonstrate the efﬁcacy of the proposed adversarial based data augmentation and mixup.
4.3. Experiments on Partial Set of Training Environments
In this set of experiments, we consider more challenging generalization settings with ξ < 1 .
That is, the number of training environments are much smaller and hence the training diversity is greatly reduced.
In particular, we consider ξ = 0 .
5 and ξ = 0 .
In such settings, with limited training diversity, it is more difﬁcult to achieve better generalization.
Again we compared the two variants of the proposed method,
PAADA ( ν = 0 . 5) and PAADA+Mixup ( ν = 0 . 5) , with
PPO and mixreg on the four game tasks.
Figure 2 reports the testing results of the comparison methods with 50% training environments, i.e., ξ = 0 .
We can see that in this setting, both PAADA ( ν = 0 . 5) and PAADA+Mixup ( ν = 0 .
5) outperform PPO on all the four games with clear performance gains.
Between the two variants, mixup still produces certain improvements on three games, starpilot, dodgeball and fruitbot , whereon PAADA+Mixup ( ν = 0 .
5) outperforms
PAADA ( ν = 0 . 5) .
Moreover, PAADA+Mixup ( ν = 0 . 5) outperforms mixreg on three games, climber, fruitbot and starpilot , while mixreg produces the best performance on dodgeball .
Nevertheless, as shown in the right plot of Figure 2, PAADA+Mixup ( ν = 0 . 5) demonstrates a consistent and clear advantage over mixreg in terms of the overall mean normalized return.
Figure 3 reports the results of the comparison methods with 25% training environments, i.e., ξ = 0 . 25 .
This setting is more challenging than the one above with ξ = 0 .
In this setting, we can see that PAADA+Mixup ( ν = 0 . 5) still outperforms mixreg on dodgeball, fruitbot and starpilot , but produces inferior result on climber ,
whereon PAADA ( ν = 0 . 5) produces the best results.
In terms of mean normalized return, as shown in the right plot of Figure 3, all three data augmentation methods demonstrate similar superior generalization capacity over the baseline PPO, All these experiments show our propose adversarial data augmentation with mixup enhancement can effectively increase the generalization capacity of RL systems.
4.4. Overall Generalization Capacity Comparison
To demonstrate a more concrete comparison between all the methods and across different settings and games, we collected the test results for the last 100 of the 3054 training epochs and reported the mean and standard deviation of these test returns in Table 1.
From these concrete result numbers we can see that all the three methods with generalization strategies, PAADA ( ν = 0 . 5) , PAADA+Mixup ( ν =
[Table]
58 ±
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
9 . 66 ± 1 .5 .
[Table] [Table]
[Table] [Table]
2 . 85 ± 0 .3 .
98 ± 0 .4 .
27 ±
[Table]
0 . 5) , and mixreg, consistently outperform PPO across different game tasks and different generalization settings.
This suggests both adversarial data augmentation and mixup are individually effective in improving the generalization performance of the baseline RL.
By effectively integrating both adversarial data augmentation and mixup enhancement, our proposed PAADA+Mixup ( ν = 0 . 5) outperforms the stateof-the-art mixreg across almost all cases (12 out of 15 cases) except on dodgeball with ξ = 1 , 0 .
5 and on climber with ξ = 0 .
These results again validated the efﬁcacy of the proposed method.
5. Conclusion and Future Work
In this paper, we proposed a novel policy-aware adversarial data augmentation method with Mixup enhancement (PAADA+Mixup) to improve the generalization capacity of RL systems.
It generates augmenting trajectories by adversarially minimizing the expected reward that a RL agent would desire to maximize , and then uses them to augment the original trajectories under controlled augmentation degrees.
Moreover, a mixup operation is further deployed to enhance the diversity of the augmented trajectory.
This is the ﬁrst work that deploys adversarial data augmentation to learn generalizable RL systems in an online manner.
It also presents the ﬁrst experience of integrating adversarial augmentation and mixup generalization.
We conducted experiments on the Procgen benchmark by comparing the proposed method with both the baseline PPO and the state-of-the-art method under different generalization settings.
The results show PAADA surpasses PPO in general, while PAADA+Mixup outperforms the state-of-the-art mixreg with notable performance gains, especially in the challenging generalization settings.
Our empirical study indicates that when the generalization setting is very challenging—that is, there are very limited training environments, the performance gains of the existing generalization methods can decrease substantially.
This is a fundamental challenge for methods that focus on generalizing the training data.
Our future work aims to tackle this challenge by extending the generalization effort into the test time.
In addition to the training time generalization, we will consider enhancing the performance of RL systems by performing test time adaptation