Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning
Abstract
In deep reinforcement learning (RL), data augmentation is widely considered as a tool to induce a set of useful priors about semantic consistency and to improve sample efficiency and generalization performance.
However, even when the prior is useful for generalization, distilling it to RL agent often interferes with RL training and degenerates sample efficiency.
Meanwhile, the agent is forgetful of the prior due to the non-stationary nature of RL.
These observations suggest two extreme schedules of distillation: (i) over the entire training; or (ii) only at the end.
Hence, we devise a stand-alone network distillation method to inject the consistency prior at any time (even after RL), and a simple yet efficient framework to automatically schedule the distillation.
Specifically, the proposed framework first focuses on mastering train environments regardless of generalization by adaptively deciding which or no augmentation to be used for the training.
After this, we add the distillation to extract the remaining benefits for generalization from all the augmentations, which requires no additional new samples.
In our experiments, we demonstrate the utility of the proposed framework, in particular, that considers postponing the augmentation to the end of RL training.
https://github.com/kbc6723/es-da
Introduction
Deep reinforcement learning (RL) aims at finding a neural network to represent policy or value functions taking raw observation as input, of which the most common form in practice is visual data or images of high-dimensionality, e g., video games [24], board games [31, 30] and robot controls [34, 19].
RL handling high-dimensional input often suffers from poor sample efficiency and generalization capability, mainly due to the curse of dimensionality [4, 16].
To overcome these issues, it has been widely considered to augment data based on prior knowledge that a set of transformations preserve the meaning or context of input observations, e g., cropping out unimportant parts of images, and changing colors [21, 20, 22, 15].
On one hand, RL agent can be directly fed with the original and augmented data so that it implicitly learns a representation with the prior and improves the sample efficiency and generalization [21].
On the other hand, the prior knowledge in data augmentation can be explicitly distilled via a self-supervised learning, which introduces additional regularization to ensure consistency between responses to original and augmented inputs [26, 15].
However, data augmentation shows highly task-dependent effect in RL, and it is prone to generate severe interference with the training even when it truly conveys a useful prior to train and test environments [21, 26, 15].
We address the problem of alleviating the interference between data augmentation and RL training to improve sample efficiency for acquainting train tasks, and generalization capability for unseen test environments.
This problem in (online) RL is more critical and challenging than that in supervised learning (SL) since the objective function and data distribution are time-varying in RL, while they are not in SL.
Indeed, according to [1, 11], it is sufficient to partly apply data augmentation just for a short period of SL at any time.
Meanwhile, we empirically observe that the prior from data augmentation can be easily forgotten in RL of the non-stationary nature (see Section 5.3), i.e., the effect of augmentation is time-sensitive .
Based on our observation about the interference and the time-sensitivity, we propose two simple yet effective methods according to timings of data augmentation :
In tra D istillation with A ugmented observations (InDA) and Ex tra D istillation with A ugmented observations (ExDA).
Data augmentation beneficial for the sample efficiency needs to be applied over the entire RL training, i.e., InDA.
Conversely, data augmentation useful only for the generalization should be postponed to the end of
RL training, i.e., ExDA, so that we can minimize the interference in the training, while enjoying the benefit in the testing.
InDA and ExDA are equipped with D istillation with A ugmented observation (DA).
DA is a stand-alone self-supervised learning which enables us to induce the prior after RL training, and shows a relatively small interference with RL training by explicitly preserving the response of RL agent to the original input.
The best timing (InDA or ExDA) depends on traits of train task and augmentation.
We hence suggest a framework of adaptive scheduling, named UCB-ExDA, that (i) first aims at maximizing the sample efficiency by adaptively selecting which or no augmentation to be used during RL training ; and then, (ii) distill the priors from all the augmentations after RL training .
Specifically, inspired by [26], we devise UCB-InDA for the adaptive selection in the first part by modifying the upper confidence bound (UCB) algorithm [3] for multi-armed bandit, where differently from [26], the set of arms includes all the augmentations and the option to not augment .
In summary, UCB-ExDA is nothing but executing
UCB-InDA followed by ExDA.
Our experiment demonstrates the utility of the proposed framework.
Our contributions are summarized as follows:
(i)
We devise DA (Section 4.1) which explicitly preserves the knowledge of RL agent so that enables distilling the consistency prior of augmentation into RL agent not only during but also after RL training, while other methods [21, 26, 15] need to be applied concurrently with RL training and thus show relatively strong interference in our experiments (Section 5.2).
(ii)
We identify the simple yet effective timings of data augmentation: either InDA or ExDA (Section 4.2, Section 4.3), based upon the discovery of the time-sensitivity (Section 5.3) that has not been observed in SL [11], i.e., the proposed timings are effective particularly for RL.
(iii)
We finally establish UCB-ExDA which automatically decides the best timing of augmentation, and demonstrate its superiority compared to others (Section 5.4).
The advantage of UCB-ExDA is particularly substantial when the best strategy is ExDA postponing augmentation to the end of RL training.
Related Works
Augmented experience in RL.
To solve the problem of poor generalization and sparse data, a popular approach is to fabricate virtual experiences and train the RL agent to learn with them.
Domain randomization is a technique to produce such experiences from a simulator of a targeted system, [34, 25, 27].
Accurate simulators of practical systems are difficult to obtain, and it has limited applicability.
However, visual augmentation has no such limit because the method uses simple image transformations such as cropping, tilting, and color jitter, although applications require a careful understanding of the targeted system to guide the design of an appropriate image transformer.
A method of curriculum learning for domain randomization, in which the difficulty is gradually increasing [27] provided insights that coincide with some of our findings.
However, we provide a further understanding of the types of visual augmentation that should be early or late during training.
Regularization from augmented data in vision-based RL has been implemented in various learning frameworks, including but not limited to representation [15, 33], self-supervised [26], and contrast [32].
One proposed algorithm [26] applies the UCB algorithm [3] to automatically select the most effective augmentations over RL training, where each augmentation is considered as an arm and then evaluate the effectiveness of augmentation by using a sliding window average.
The idea of adapting augmentation concurs with our main message regarding the timing of augmentation.
In [26], ’not augmenting’ is not an option, whereas our findings indicate that it should be.
In addition, [26] does not consider post augmentation followed by RL training, as in ExDA.
Other time-sensitivity in deep learning.
During deep learning, the early stage of training often has a significant effect [7, 1].
Therefore, we devised time-sensitive methods that adapt to the progress of training, such as learning rate decay [37] and curriculum learning [35].
Golatkar et al .
[11] studied such a time-sensitivity of regularization techniques for SL, where the effect of data augmentation in different time does not change much.
We find that the time-sensitivity of augmentation can be significant in RL.
This contrast may occur because of the non-stationary nature of RL, which SL does not have.
Although a set of techniques originally developed for SL such as convolutional neural network, weight decay, batch normalization, dropout and self-supervised learning improve deep RL [17, 6, 23, 9, 32, 36, 14], a thorough study should be conducted before introducing a method from different learning frameworks, because we find the contrasting time-sensitivities of data augmentation.
This spirit is also shared with an application [18] of implicit bias in SL [13, 2, 10] to RL.
Background
Notation.
We consider a standard agent-environment interface of vision-based reinforcement learning in a discrete Markov decision process of state space S , action space A , and kernel P =
P ( s t +1 , r t | s t , a t ) which determines the state transition and reward distribution.
The goal of the RL agent is to find a policy that maximizes the expectation of cumulative reward (cid:80) t ′ − 1 t =
γ t r t , where t ′ is terminating time and γ ∈ [0 , 1] is discount factor.
At each timestep t , the agent selects an action a t ∈ A and receives a reward r t and an image o t +
= O
( s t +1 )
∈ R k × k ′ as an observation of the next state s t +1 .
Data augmentation can be described by an image transformation function ϕ : R k × k ′ (cid:55)→ R k × k ′ of which output is presumed to have the same or similar semantics of the input.
Baseline RL algorithm.
Throughout this paper, we use Proximal Policy Optimization (PPO)
[28] as a baseline, although we believe our findings and methods can be easily adjusted to others.
PPO is a representative on-policy RL algorithm to learn policy π θ ( a | o ) and value function V θ of neural agent parameterized by θ .
Storing a set of recent transitions τ t :=
( o t , a t , r t , o t +1 ) in experience buffer D , the agent is updated to minimize the following loss function:
[Equation]
where α is a hyperparameter and canonical regularization terms are omitted.
The clipped policy objective function L π and value loss function L V are defined as:
[Equation]
where the expectation ˆ E is taken with respect to τ t ∼ D , we denote by θ old the parameter before the update, ρ t ( θ ) is the importance ratio π θ ( a t | o t )
π θ old ( a t | o t ) , and ˆ A t is the generalized advantage estimation [28].
Method
In what follows, we present our methods: D istillation with A ugmented observations (DA; Section 4.1),
In tra DA (InDA; Section 4.2), Ex tra DA (ExDA; Section 4.3), and then the adaptive scheduling frameworks based on UCB (UCB-InDA and UCB-ExDA; Section 4.4).
DA is a stand-alone knowledge distillation method, which can be used at any time to instill the underlying prior of augmentation into a given RL agent.
InDA and ExDA conduct either DA or PPO in each epoch but have different schedules (Figure 1), where InDA interleaves PPO and DA, whereas ExDA performs PPO first then
DA.
UCB-InDA adaptively decides which or no augmentation to be used in each DA epoch of InDA based on the UCB of estimated gain from each option.
UCB-ExDA performs ExDA preceded by
UCB-InDA.
4.1 Distillation with augmented observations (DA)
The underlying prior of augmentation can be infused by minimizing a measure of inconsistency between the agent’s responses to original and augmented inputs (resp.
o t and ϕ ( o t ) ).
For instance,
[Figure]
with PPO agent learning policy π θ and value V θ , Raileanu et al .
[26] proposes the following measure:
[Equation]
which uses Kullback-Leibler divergence (first term) and mean squared deviation (second term) for policy and value inconsistencies, respectively.
Noting that L dis ( θ ; ϕ ) can be minimized to be zero by a constant response to all inputs, the distillation with Eq (4) can distort the RL agent, in particular, when applying it outside of RL training.
We hence devise a network distillation technique (DA) which explicitly preserves the RL agent’s response to original input and thus is applicable even after RL training.
DA distills the knowledge of
RL agent θ old into θ using not only original but also augmented observations.
More formally, the loss of DA is written as:
[Equation]
We here denote the identity transformation by I such that I ( o ) =
o for all o , and extend the definition of L dis in Eq (4) as follows:
[Equation]
In Eq (5), the first term ensures that θ and θ old behave identically for the original inputs, and the second one infuses the consistency prior.
In Figure 2, L dis ( θ, I ; θ old ) , L dis ( θ, ϕ ; θ old ) , and L dis ( θ, ϕ ; θ ) graphically correspond to (A), (B), and (C), respectively.
From this, it follows that minimizing L DA in Eq (5) eventually reduces L dis ( θ, ϕ ; θ ) in Eq (4).
In addition, the minimization of L DA secures the responses of θ to the original inputs (which can be pre-computed to reduce computation cost) to the those of θ old , while the alternatives (e.g., (A)+(C): L dis ( θ, I ; θ old ) + L dis ( θ, ϕ ; θ ) ) does not and thus may generate interference with RL training.
Our experiments (Section 5.2; Figure 4) justifies the design of DA by showing substantial advantage compared to the other alternatives such as DrAC [26] using L dis ( θ, ϕ ; θ ) in Eq (4).
We note that this advantage becomes much clearer when a wrong augmentation is given, c.f., the supplementary material.
4.2 Intra distillation with augmented observations
InDA (Algorithm 1) alternates between minimizing L PPO and L DA , i.e., PPO and DA are explicitly separated, whereas they are often executed simultaneously in other methods [26].
Such a clear separation reduces the interference [15].
We can control the frequency and timing of applying distillation with hyperparameters I , S and T , where we perform DA after each I rounds of RL training only if the current epoch n is in the interval of [ S, T ] , i.e., S and T are the epochs to begin and terminate DA, respectively.
We vary S and T to study the time-dependency of data augmentation.
We denote InDA [ S, T ] to indicate the period to apply DA, while we omit the indication when DA is applied over the entire period.
We provide further details on InDA in the supplementary material.
Require: N, I, ϕ , S , T 1: Initialize θ close to origin.
2: for n = 1 , 2 , . . .
, N do 3: //
RL training 4:
Store sampled transitions to D ; 5:
Optimize RL objective L PPO ( θ ) with D ; 6: //
Distillation 7: if n ∈ [ S, T ] and mod ( n − 1 , I ) =
then 8:
Store θ old ← θ ; 9:
Minimize L DA ( θ ) for D , θ old and ϕ ; 10: end if 11: end for
Require: N , M , ϕ 1: Initialize θ close to origin.
2: //Pre-training phase with RL algorithm 3: for n = 1 , 2 , . .
. , N do 4:
Store sampled transitions to D ; 5:
Optimize RL objective L PPO ( θ ) with D ; 6: end for 7: 8: Store θ old ← θ ; 9: //
Distillation at the end of RL training 10: for m = 1 , 2 , . .
, M do 11:
Minimize L DA ( θ ) for D , θ old and ϕ ; 12: end for
4.3 Extra distillation with augmented observations
ExDA (Algorithm 2) performs the distillation after the end of RL training, where the lengths of DA and RL training are parameterized by M and N , respectively.
We note that computational cost can be reduced by removing the value inconsistency measure ˆ E o t ∼D (cid:2) ( V θ old ( o t ) − V θ ( ϕ ( o t ))) 2 (cid:3) from
L dis in Eq (6) because the value consistency is not necessary for RL training nor distillation in the actor-critic framework and including it has a potential risk of generating additional interference.
In the supplementary material, it is empirically verified that this reduction does not degrade RL performance.
We leave more interesting details in the supplementary.
For instance, one can consider re-initializing θ before starting DA as a part of exploiting the implicit bias [13, 18] to improve generalization.
However, test performance is often dropped.
This is mainly because the dataset to distill π θ old has much less diversity than that observed along the trajectory.
Thus, we use no re-initialization for the experiments in the main paper.
4. UCB-based adaptive scheduling frameworks
The training benefit by augmentation differs depending on the task.
This dependency complicates the choice of whether to use InDA or ExDA for augmentation.
Hence, we devise an auto-augmentation method, called UCB-InDA, inspired by UCB-DrAC [26], where each augmentation is corresponded to an arm in multi-armed bandit (MAB) problem and assessed its potential gain in training with upper confidence bound (UCB) [3].
More formally, in UCB-InDA, the set of arms is the set of augmentations, Φ = { ϕ 1 , . . .
, ϕ K } , which must include the identity function I , i.e., the option not to augment.
The inclusion of identity function is small but makes crucial difference than UCB-DrAC [26] since we observe that using augmentation sometimes needs to be postponed after RL training for the sake of better sampling complexity and test performance.
A round of MAB corresponds to every I epoch of InDA, where we let ϕ k ( s ) ∈ Φ be the augmentation selected at the s -th round of DA.
Let G ( s ) be the average return, the sum of estimated advantage ˆ A and predicted value V θ , over ( I − 1) epochs of PPO followed by the s -th DA.
UCB algorithm assumes that each arm generates reward independently drawn from a fixed distribution, and estimates the empirical mean over the entire sampling process.
However, in RL, the return G ( s ) is non-stationary, so UCB-InDA computes moving average gain ¯ G k ( s ) , instead, taken over a certain number (chosen to be 3 in our experiment) of most recent rounds selecting ϕ k as Raileanu et al .
[26] proposed.
Then, inspired by UCB1 algorithm [3], UCB-InDA selects action k ( s ) at round s as follows:
[Equation]
where c is the UCB exploration coefficient, and N k ( s ) is the number of times selecting ϕ k up to round s .
We refer to the supplementary material for the hyperparameter choice.
We remark that compared to
UCB-DrAC [26], the proposed UCB-InDA has subtle but important differences, summarized in two folds: (i) the inclusion of identity transformation (i.e., no augmentation) and (ii) the distillation with
[Figure]
augmentation via InDA.
The gain of each component is numerically studied in Section 5.
Finally, we note that UCB-ExDA is nothing but UCB-InDA followed by ExDA.
4.5 PAGrad
When we use an additional regularization term in training as below,
[Equation]
a direction of gradient from the additional term can differ with a direction of standard training.
Similarly, in multitask learning, each gradient of tasks affects each other during solving several tasks simultaneously.
To relax the negative effect by a gradient conflicting, they project each task’s gradient onto the normal vector of the other task’s gradient [38].
By borrowing this method, we project a gradient of regularization ( g reg ) on the normal vector of a gradient of RL training ( g train ) when only they conflict (Figure 3).
We can reduce a negative effect from regularization by subtracting a part of g reg according to an opposite direction of g train .
We call the modified gradient as PAGrad (Projecting
Auxiliary Gradient).
Computation of modified gradient is described in Algorithm 3.
To verify the effect of projection, we use DrAC [26], which use self-distillation to use data augmentation in RL.
Noting the distillation is an auxiliary task for RL, we devise DrAC+PAGrad (projecting auxiliary gradient) which is a gradient-based method towards the negative of the following direction:
[Equation]
(9) where we remove the components of ∇ L dis opposite to ∇ L PPO which may disturb optimizing the main objective L PPO .
Experiment 5.1 Setups
Train and test tasks.
We use the OpenAI Procgen benchmark of 16 video games [5], where a main character tries to achieve a specific goal, e.g., finding exit (Maze) or collecting coins (Coinrun), while avoiding enemies given a 2D map.
At each time t , visual observation o t is given as an image of size 64 × 64 .
A train or test task is to achieve a high score on a set of environments configured by game and mode, where a mode describes predefined sets of levels (e.g., complexity of map) and backgrounds.
Cobbe et al .
[5] provide easy mode for each game, consisting of 200 levels and a certain set of backgrounds.
We simplify easy mode and train agents in easybg mode, of which the only difference from easy mode [5] is showing only a single background.
This is useful to investigate the case that using visual augmentation is helpful for testing but not for training.
We denote the task configuration by game_name( mode ), e.g., Coinrun( easybg ).
Then, we evaluate generalization capabilities using two modes: test-bg and test-lv , which contain unseen backgrounds and levels, respectively, in addition to easybg mode that we use for training.
Types of augmentation.
For clarity, we mainly focus on two types of visual augmentation, each of which conveys distinguishing inductive bias:
(a) Random color transforms an image by passing through either color jitter layer or random convolutional layer proposed in [22].
From this, we can impose the consistency to color changes, which may provide a strong generalization to diverse backgrounds of test-bg mode.
(b) Random crop leaves a randomly-selected rectangle and pads zeros to the rest of the image [26].
This augmentation is particularly useful in fully-observable games (e
g., Chaser and Heist), because it imposes an efficient attention mechanism.
We also report the result with other augmentations including color jitter, random convolution, gray, and cutout color in the supplementary material, from which the same messages can be interpreted.
Baseline methods for RL with data augmentation.
We mainly compare the proposed methods (InDA and ExDA) to the following baselines:
(a) RAD [21] simply feeds PPO with experiences of original and augmented observations.
(b) DrAC [26] is a method to simultaneously minimizing L PPO in Eq (1) and L dis in Eq (4).
(c) DrAC+PAGrad is a variant of DrAC, which we devise to investigate another mechanism to relieve the interference between RL training and augmentation in Section 4.5.
The supplementary material presents further details and experiments, which we omitted for simplicity.
All results in the main paper are averaged over five runs.
of PPO.
ExDrAC is a variant of DrAC without RL training, i.e., minimizing only
L dis in Eq (4).
5.2 Benefit of separating distillation from RL training.
In Figure 4, after training PPO agent up to 20M timesteps, we start to suddenly apply one of the different distillation methods with random crop.
We report averaged scores over 6 environments (Bigfish, Dodgeball, Plunder, Chaser, Heist, Maze) after normalized by the highest train score of PPO on each environment.
After 20M timesteps, ExDA and ExDrAC have no RL training but minimize
L DA in Eq (5) and L dis in Eq (4), respectively.
The substantial gap between ExDA and ExDrAC justifies the design of DA explicitly preserving the knowledge from RL when distilling the prior.
More importantly, it is remarkable that ExDA promptly learns the generalization ability once it starts to distill the prior.
This validates the idea of postponing the distillation after RL training.
We now compare the distillation methods (InDA, DrAC, and DrAC+PAGrad) concurrently optimizing the RL objective and distilling the prior in Figure 4.
Each method has performance degeneration due to the interference generated by distillation.
However, InDA and DrAC+PAGrad have clearly smaller
[Figure]
degeneration than DrAC which is the only one without any separation of optimizing the RL objective and distillation loss.
We note that DrAC+PAGrad has more interference than InDA, and it seems to fail to impose the prior since there is not much difference to PPO in testing.
Hence, this verifies the superiority of InDA which enables distilling the prior while alleviating the interference.
5.3 Effective timings of distillation
In what follows, we aim to identifying effective timings of distillation.
To this end, in Figure 5, we test several schedules of applying DA on two representative environments of distinguishing traits.
The supplementary material presents the result on more environments, which is similar one of the representatives.
An effective timing: InDA.
Chaser with random crop (Figure 5(a) and 5(b)) represents the case when augmentation improves the sample efficiency of training and thus the generalization ability in training.
To compare the generalization ability, we measure policy distance between original and augmented observations using Jenson-Shannon divergence (Figure 5(c)).
InDA[0, 5]’s policy distance is increased after it stops using DA.
Thus, the generalization ability is degraded, if we do not continue to use DA.
In this case, it is clear that InDA should be applied during the entire RL training, as InDA[0, 25] shows the best performance in both training and testing.
In addition, it is also important to apply DA as soon as possible since the effect of InDA[0,5] applying DA in the beginning is relatively prompt and significant comparing to that of InDA[20,25].
This suggests that the automatic framework needs to explore more in the early stage.
An effective timing: ExDA.
Heist with random crop (Figure 5(d) and 5(e)) shows the opposite use of data augmentation to what Chaser case suggests, i.e., postponing data augmentation as much as possible.
Random crop generates a slight interference, although it immediately improves the generalization ability.
We remark that the inductive bias from the random crop is easily forgotten, as the test performance of InDA[0, 5] is saturated right after stopping the distillation.
This can be explained by the time-varying nature of sample distribution and objective in RL.
Interestingly, it is in contrast to the data augmentation in SL, where an early distillation is sufficient to impose the prior [11].
On the other hand, the test performance curve of InDA[20, 25] soars right after DA.
Furthermore, Figure 5(f) shows that InDA[20, 25] narrows the gap between two policies on the original and augmented observation.
Recalling the interference between RL training and distillation, this suggests postponing the distillation at the end of RL training, and motivates our ExDA.
Performance benchmark on InDA and ExDA.
In Table 1, we summarize the train and test performances of InDA and ExDA on a set of games with different augmentations and modes.
ExDA outperforms other baselines with random color on test-bg .
Moreover, we note that ExDA consumes only 0.5M timesteps to inject the prior at the end of RL training, whereas the others use all the training data.
ExDA in both sample efficiency and generalization ability with random crop on test-lv .
It is worth noting that DrAC+PAGrad is slightly better than DrAC, while there is a substantial gap between
InDA and each DrAC-based algorithm.
This again confirms the benefit from the separated distillation of InDA observed in Figure 4.
These results demonstrate that each combination of environment and augmentation has a suitable time at which to apply augmentation, and the gain from using the right distillation timing, i.e., online (InDA, DrAC, or DrAC+PAGrad) or offline (ExDA), is rigid regardless of the choice of algorithms.
[Table]
Randcolor
[Table]
5.4 Adaptive scheduling methods
Adaptive selecting of timings: UCB-InDA and UCB-ExDA.
It is hard to know in advance whether a certain augmentation helps the training or not [26].
We hence employ UCB-InDA which estimates the gain or damage from each augmentation from trial and error and identifies the one
[Figure]
[Table]
and UCB-DrAC are trained as automatically selecting the augmentation during training.
UCB-ExDA trains ExDA after UCB-InDA with both augmentation.
with most help.
Recall Table 2 where PPO (without augmentation) shows much better training performance than InDA in Heist.
As shown in Figure 6(b), UCB-InDA is able to identify that no augmentation is best for training in Heist.
This implies that ExDA is more appropriate than InDA.
Conversely, random crop is selected on Chaser (Figure 6(a)).
As the result, we can automatically select InDA or ExDA appropriately for each task.
Fully exploitation of augmentation In Table 2, when both random color and random crop are used to improve generalization on both test-bg and test-lv , we report numerical evaluation of UCB-InDA and UCB-ExDA with other baselines on train and test tasks.
Decreased train performance of DrAC and InDA compare to PPO show the difficulty of simultaneous training with several augmentations.
Train performance of UCB-InDA and UCB-DrAC are improved by adaptive selecting, especially,
UCB-InDA is better than UCB-DrAC.
The gap is made due to the robustness about the change of augmentation during training.
In terms of generalization, UCB-ExDA clearly surpasses UCB-InDA thanks to ExDA to extract all the priors from the complete set of data augmentations at the end of RL training.
Discussion
We have identified two most effective yet simple timings (InDA and ExDA) of data augmentation for
RL, and proposed UCB-ExDA framework to adaptively select the best scheduling augmentations.
We note that the effectiveness of this framework is restricted but specialized for RL with the unique nonstationary nature.
Indeed, in SL without shift of data distribution and objective, it is sufficient to apply data augmentation at the beginning [11].
Our framework employs the most basic multi-armed bandit algorithm with a finite set of data augmentation.
It is interesting to investigate a room to improve by further optimizing continuous parameters of data augmentation for RL, c.f., an auto-augmentation technique to optimize continuous augmentation parameter per sample for SL [12].
Another promising direction is to accelerate the distillation process of DA by data condensation with augmentation [39].
This is possible with our framework clearly separating between RL training and distillation, and may be particularly useful to train distributed RL agents since a condensed data for an agent’s distillation is usable for the other.
Acknowledgments and Disclosure of Funding
We thank Kimin Lee for helpful discussions.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01906, Artificial Intelligence Graduate School Program (POSTECH)) and
Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-02068, Artificial Intelligence Innovation Hub) and the
National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
(No.
2021M3E5D2A01023887).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We claim our three main contributions in 1 and target on DeepRL.
(b) Did you describe the limitations of your work?
[Yes] We describe the limitations of our work about a limited set of data augmentation for UCB methods in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We believe that our research will not have a negative impact on society.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We read the ethics review guidelines.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Our works are based on experimental results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
Our works are based on experimental results.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We describe our training environments and include code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We describe training details in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We indicate standard deviation with score in Figureand Table
except normalized score with several environments such as Figure 4 and Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We explain about training time in the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Our experiment environments are baed on OpenAI Procgen [5].
(b) Did you mention the license of the assets?
[Yes] We include the license of the asssets in the code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[Figure]
This section explains Modified Procgen Environments, which is designed to verify different types of generalization, backgrounds, and levels.
Open AI Procgen environments [5] share background themes such as space_backgrounds , platform_backgrounds , topdown_backgrounds , water_backgrounds , water_surface_backgrounds .
We create new difficulties as Easybg , Easybg-test , Easy-test .
Easybg generates environments which contain only one for each background, wall and agent theme.
Easybgtest and Easy-test are for test about background change after trained on Easybg and Easy .
Wall theme in (Climber, Coinrun, Jumper, Ninja) and Agent theme in (Climber, Coinrun) also compose with only one image resource in Easybg .
Figure 7 presents an example set of modes that we use in evaluation.
Furthermore, We fix the exit_wall_choice and enemy theme in Dodgeball.
We describe the usage themes in each environment, which are grouped by backgrounds theme as below:
• space_backgrounds (Bossfight, Starpilot) Background: "space_backgrounds/deep_space_01.png" • platform_backgrounds (Caveflyer, Climber, Coinrun, Jumper, Miner, Ninja) Background:"platform_backgrounds/alien_bg.png", Coinrun (Agent color: Beige, Wall themes: Dirt), Climber (Agent color: Blue, Wall themes: tileBlue), Jumper (Wall theme: tileBlue), Ninja (Wall theme: bricksGrey) •
topdown_backgrounds (Chaser, Dodgeball, Fruitbot, Heist, Leaper, Maze) Background:"topdown_backgrounds/floortiles.png",
Dodgeball (
Enemy
theme:
"misc_assets/character1.png", Exit_wall_choice: 0) • water_backgrounds (Bigfish) Background:"water_backgrounds/water1.png" • water_surface_backgrounds (Plunder)
Background:"water_backgrounds/water1.png"
Easybg-test uses backgrounds in each background group, except the one used in Easybg .
Easy-test is only defined for Climber, Jumper, Ninja, and they compose with topdown_backgrounds .
B Implementation details
In this section, we explain about InDA, ExDA, UCB-InDA and other baselines.
We train the agent with IMPALA-CNN [8] in every experiment.
B.1 InDA
We use PPO [29] as a base RL algorithm, For data efficiency, we store the observations during RL training in buffer D O .
Before DA phase, we also make policy buffer D Π , value function buffer D V and augmented observation buffer D ϕ for distillation, because we only use one network model.
We randomly sample pairs of ( o, π, V ) from buffer, and minimize loss function L DA ( θ ) .
We reuse the sample three times like PPO, it can be controlled by # Epochs of DA.
We did a greed searches for learning rate of DA l DA ∈ [1 × 10 − 3 , 5 × 10 − 4 , 2 × 10 − 4 , 1 × 10 − 4 , 5 × 10 − 5 ] and interval
I ∈ [1 , 5 , 10] and found the best combination l DA = 10 − 4 and interval I =
We fix the buffer size D O = 40960 , because we collect the observations during five RL phases (5 × 256 × 32) .
We describe the every hyperparameter as below:
Hyperparamter
Value γ 0.
λ 0.
# Timesteps per rollout # Epochs per rollout # Minibatches per epoch
Reward Normalization
Yes # Workers # Environments per worker
Total timesteps 25M
LSTM
No
Frame Stack
No
Optimizer
Adam optimizer
Entropy bonus 0.
PPO clip range 0.
Learning rate 5 × 10 −
Interval
I
Size of D O # Epochs of DA
Learning rate of DA l DA 1 × 10 −
Image transformation ϕ
Any augmentation
B.2 ExDA
In ExDA, we generate and store ( o, π, V ) using f θ old in buffer D .
The optimal buffer size depends on the episode length of each environment.
However, we standardize the buffer size as 0.5M in every environment.
We augment the observation with three epochs intervals when using randomized augmentation methods.
We did greed searches for # minibatches [1024 , 2048 , 4096] and learning rate [5 × 10 − 4 , 1 × 10 − 3 , 2 × 10 − 3 ] .
As a result, we select # of minibatches 4096 and a learning rate 1 e − 3 .We describe every hyperparameter as below:
[Table]
Value Size
[Table] [Table]
5M # Epoch # Minibatches per epoch
[Table]
Learning rate 1 ×
[Table]
−# Workers Optimizer Adam optimizer Image transformation ϕ
Any augmentation
B.3 UCB-InDA
We use UCB-InDA as a discriminator to determine the necessity of augmentation during the training.
The gain of an augmentation is a mean of return during interval I, G ( s )
=
1 I (cid:80) j − 1 i =
R ( s + j ) .
The return is computed by the sum of estimated advantage and predicted value, which are expected value of the agent trajectory, R ( s )
=
ˆ E ( o t ,a t ) ∼ π θ [ ˆ A t + V θ ( o t )] .
The ˆ A t is advantage from Generalized
Advantage Estimator [28].
Thus, we can evaluate how augmentation affects the return on the agent trajectory.
However, the distribution of return is non-stationary, as the agent policy is changed.
Therefore, we use the window average gain ¯ G ϕ rather than the whole gain from the past evaluation.
Furthermore, the drastic change of return causes the gap of gain between the augmentation according to sampling time at the transient time of training and leads to poor exploration about some augmentation methods.
For stable exploration, we fix the minimum exploration frequency and use forced exploration method after the minimum exploration as below:
[Equation]
11) where ϕ max = arg max ϕ ∈ Φ ¯ G ϕ , ϕ min =
arg min ϕ ∈
Φ ¯ G ϕ .
We set the hyperparameter as below
[Table]
We compare ExDA and InDA with PPO [5], DrAC [26], Rand-FM [22], RAD [21].
Every baseline is based on PPO [5] and we adopt the implementation of PPO in
• DrAC [26] regularizes both policy and value function as self-supervised learning.
Regularization term have hyperparameter α r for ratio with PPO objective.
We use the hyperparameter recommended by the author.
• Rand-FM
[22] is composed with random convolution networks and feature matching.
They also need hyperparameter β for ratio between feature matching and PPO objective.
We use same β with author.
• RAD [21] naively use augmented observations in state distribution.
Thus, there are no additional hyperparameters.
We describe the hyperparameter of baselines in below table:
Hyperparameter
Value γ 0.
λ 0.
# of timesteps per rollout # of epochs per rollout # of Minibatches per epoch
Reward Normalization
Yes # of Workers # of environments per worker
Total timesteps 25M
LSTM
No
Frame Stack
No
Optimizer
Adam optimizer
Entropy bonus 0.
PPO clip range 0.
Learning rate 5 × 10 −α r (DrAC) 0.
β (Rand-FM) 0.
C Data augmentation
In our experiments, we use five augmentation methods: crop, grayscale, cutout color, random convolution and color jitter .
We refer the implementation of augmentations from Lee et al.
[22] ( random convolution ), Laskin et al.
[21] ( cutout color, color jitter ) and Raileanu et al.
[26] ( grayscale, crop ).
We expect the generalization about background change from random convolution, color jitter, gray, cutout color .
About the change of levels, we use crop and cutout color for generalization.
Examples of data augmentation are represented below:
[Figure]
D Interference from increased complexity of learning
Even if the policy for the original observation is fixed and augmentation is used, learning may still be hindered.
In Figure 9, we report training curves of the agents on train tasks of different degree of background diversity.
InDA with random convolution is anticipated to lead the prior on color consistency, which seems helpful to handle background diversity.
However, PPO outperforms InDA in terms of the sample complexity to master train tasks, while the gap is decreasing as the background diversity increases.
This implies that diverse backgrounds make hard to train by increased complexity of learning, and also data augmentation can cause similar difficulty even with right prior, when using it during RL training.
We further remark that ExDA using random convolution after PPO trained on
Heist( easybg-1 ) achieves 8.15 on Heist( easybg-15 ), which is much higher than PPO’s 6.53 trained on
Heist( easybg-15 ).
This suggests the importance of simplifying the train task and the utility of ExDA which completely separates RL training and distillation with augmentation.
E Robustness in loss function change
In ExDA, we transfer the policy after training 20M time steps with PPO.
Thus, we explain why other augmentations are not used after pre-training.
We compare the results of training and test performance with Drac [26], Rand-FM [22], Rad [21] when we train each method for 5M after training PPO for 20M time steps.
We use random convolution and crop as data augmentation methods, and we do not compare with RAD when we use crop in Figure 12 and Figure 13.
The crop method used in our paper do not work well in RAD, because they use a different crop method with [26] in their paper [21].
InDA is more stable than others in training, and it affects generalization performance.
Every training curves decline immediately after starting training with augmented observations at 20M time steps.
The objective function is changed to each baseline, and augmented data is newly added to data distribution.
Thus, the optimizer should find a new optimal point for new objective function and data.
During find the new optimal points, the agent learns along with the different directions from the optimization direction in pure PPO.
Thus, performance can be degraded because the learning direction on loss landscape is different from maximizing rewards on non-augmented data in PPO.
In spite of using self-supervised learning or representation learning, the policy is changed because they update the same network’s parameter for matching policy or latent features, such as DrAC [26] and Rand-FM [22].
However, InDA is more stable than the others because we distill the fixed policy and value using DA.
It does the stable training through conserving the policy on non-augmented observations during optimizing for augmented data.
Coinrun     25 Timesteps (M)
S c
o r e
Maze     25 Timesteps (M)
S c
o r e
Fruitbot     25 Timesteps (M) - -
S c
o r e
Climber     25 Timesteps (M)
S c
o r e
Ninja     25 Timesteps (M) -
S c
o r e
Jumper     25 Timesteps (M)
S c
o r e
Heist     25 Timesteps (M)
S c
o r e
[Figure] [Figure]
e
[Figure] [Figure]
[Figure] [Figure]
S c
o r e
[Figure] [Figure]
S c
o r e
Jumper     25 Timesteps (M)
S c
o r e
Heist     25 Timesteps (M)
S c
o r e
25 Timesteps (M) -
S c
o r e
Heist
[Figure]
Timesteps (M)
S c
o r e
Maze     25 Timesteps (M)
S c
o r e
Dodgeball
[Figure]
(M) -
S c
o r e
Chaser
[Figure] [Figure]
[Figure] [Figure]
r
[Figure]
Plunder
[Figure] [Figure]
[Figure]
S c
o r e
[Figure]
[Figure] [Figure]
S c
o r e
Bigfish     25 Timesteps (M) -
S c
o r e
Dodgeball
[Figure] [Figure]
S c
o r e
Chaser
[Figure] [Figure]
S c
o r e
Plunder     25 Timesteps (M)
S c
o r e
F Robustness against wrong augmentation
In this section, we verify the robustness of ExDA from an obvious wrong augmentation: just making an black image, denoted by black .
Using this clearly interferes with RL training.
We compare PPO,
InDA, ExDA, UCB-InDA, UCB-ExDA and DrAC.
The hyperparameter is almost the same with
Section 5.4 except we use black instead of random color and random crop .
For UCB-based methods, we use 4 arms: black , random color , random crop , and no aug .
As expected, ExDA preserves the PPO score, but InDA and DrAC degrade the score.
Furthermore, UCB-InDA improves the performance in
Chaser, and also it almost preserves the score in Heist.
Lastly, UCB-ExDA also maintains the score from UCB-InDA.
[Table]
G Ablation study of ExDA G.1 Initialization and regularization term
In this section, we do an ablation study about the factor of ExDA.
We mention the loss function and re-initialization issue in subsection 4.3.
ExDA does not have to minimize L V D because the value function is useless after RL training.
The below results show that L V D cannot give any benefit in
ExDA.
Thus, we only use L PD for computational complexity.
Furthermore, we also compare to verify the effect of non-stationarity with a re-initialized agent before distillation.
Igl et al.
[18] argued that the non-stationarity causes the reduction of generalization.
However, the re-initialization is not critical in test performance, as shown in Figure 15.
Moreover, sometimes re-initialization makes it difficult to distill training performance such as Fruitbot and Ninja in Figure 14.
We use random convolution as an augmentation method in here.
Ninja      30 Epochs8.2 8.
S c
o r e
Jumper      30 Epochs 8.
S c
o r e
Climber      30 Epochs 10.
S c
o r e
Coinrun      30 Epochs 8.
S c
o r e
Maze      30 Epochs 9.45 9.
S c
o r e
Heist      30 Epochs 6.
S c
o r e
[Figure]
30 Epochs
S c
o r e
[Figure]
Ninja      30 Epochs5.
S c
o r e
Jumper      30 Epochs6.
S c
o r e
Climber      30 Epochs 6.
S c
o r e
Coinrun      30 Epochs5.56.57.58.
S c
o r e
Maze      30 Epochs5.56.57.58.59.
S c
o r e
Heist      30 Epochs5.56.57.58.
S c
o r e
Fruitbot      30 Epochs
S c
o r e
[Figure]
G.2 ExDA after InDA with various backgrounds
When augmentation helps the training, ExDA struggle to follow the training performance of InDA because ExDA’s training performance is limited by pre-trained agent’s policy.
Thus, we use InDA for
ExDA’s pre-training , and call it as ExDA (InDA).
As shown in Table 5, ExDA (InDA) is comparable to InDA, but not beyond.
Thus, unless there is a meaningful difference in training performance,
ExDA has no better generalization than InDA.
However, in computational complexity, ExDA is more efficient than others such as InDA and DrAC when they have a similar performance.
In the following section, we discuss computational complexity.
[Figure]
[Table] [Table]
G.3 Computational complexity
We compare the computational complexity with ExDA and InDA.
InDA do DA for every 25M observations during training and reuse the sample in three times.
However, ExDA only use 0.5M for
DA during 30 epochs.
Thus, ExDA is almost 5 times more efficient than InDA by rough calculation.
Furthermore, the ExDA saves the time for augmentation compared to InDA.
When we train with same computational setting (GPU: GeForce RTX 2080 TI), ExDA only consumes 5 hours + 2 hours
(PPO) when using random convolution, but, InDA consumes 18 hours.
Thus, we recommend ExDA when InDA cannot give a meaningful gain in training performance.
H Comparison between InDA and ExDA using the same steps of RL training
In every experiment in the paper, we set the evaluation setup to be somewhat unfavorable to ExDA (using 20M time steps of RL training followed by additional 0.5M steps of distillation; denoted by ExDA(20M)) compared to InDA or other baselines (25M time steps of RL training) to clearly avoid potential complaint about the extra 0.5M steps for ExDA, It is obvious that the performance of
ExDA is improved if we put more time steps for RL training, and thus the benefit of ExDA compared to InDA becomes more conspicuous if ExDA is the effective timing.
Thus, we do an additional experiment evaluating ExDA (25M) using 25M RL time steps and 0.5M distillation time steps as below table:
[Table]
The result of ExDA(25M) reinforces our main message: postponing data augmentation when it generates a severe interference with RL training in Table 6.
ExDA(25M) has a slight drop in train score compared to PPO, but it could be eventually removed if we put (slightly) more steps for distillation.
I UCB with a large set of augmentation
In Figure 6, we only use three arms such as random color , random crop and no augmentation .
Thus, we try to select the usuful augmentations at each time among a large set of augmentation, which contains gray , cutout color , random convolution , color jitter , random crop and no augmentation .
In
Figure 16(a), we show the result of arm selection with UCB-InDA.
The identity function is most frequently used in the same as Figure 6.
On the other hand, we ablate the neccesity of identity function in UCB-InDA, it shows that the identity function is needed.
[Figure]
J Time matter in training
This section shows every result of Figure 5 about time dependency with InDA.
We experiment with random convolution, crop, color jitter and evaluate the test on unseen backgrounds ( random convolution, color jitter ) and levels ( random crop ).
However, the effect of generalization is hard to recognize in most cases, as shown in Appendix K.
Thus, we mainly discuss the most effective augmentation, such as random convolution and crop in the main paper, and only represent some environments that have helped the generalization by color jitter.
easybg mode is used as default mode with three easy mode (Climber, Jumper, Ninja) in our experiments.
The shaded regions and solid line represent the standard deviation and mean, across five runs.
Train Random conv
Jumper     25 Timesteps (M)
S c
o r e
Ninja     25 Timesteps (M)
S c
o r e
Climber     25 Timesteps (M)
S c
o r e
Fruitbot     25 Timesteps (M) -
S c
o r e
Maze     25 Timesteps (M)
S c
o r e
Heist     25 Timesteps (M)
S c
o r e
Coinrun     25 Timesteps (M)
S c
o r e
Jumper (easy)     25 Timesteps (M)
S c
o r e
Climber (easy)     25 Timesteps (M)
S c
o r e
Ninja (easy)     25 Timesteps (M)
S c
o r e (S,T)
Test Random conv
[Figure]
S c
o r e
[Figure] [Figure]
(M)
[Figure]
S c
o r e
[Figure] [Figure]
(M)
S c
o r e
[Figure] [Figure]
(M) -
[Figure]
S c
o r e
[Figure] [Figure]
(M)
[Figure] [Figure]
S c
[Figure]
r e
[Figure] [Figure]
[Figure] [Figure]
[Figure]
Train Crop
[Figure] [Figure]
c o r e
[Figure] [Figure]
[Figure] [Figure]
r e
Plunder
[Figure] [Figure]
S c
o r e
Dodgeball
[Figure] [Figure]
Test Crop
[Figure]
Ninja     25 Timesteps (M)
S c
o r e
Maze     25 Timesteps (M)
S c
o r e
Heist     25 Timesteps (M)
S c
o r e
Fruitbot     25 Timesteps (M) -
S c
o r e
Coinrun     25 Timesteps (M)
S c
o r e
Jumper     25 Timesteps (M)
S c
o r e
Climber     25 Timesteps (M)
S c
o r e
Ninja (easy)     25 Timesteps (M)
S c
o r e
Jumper (easy)     25 Timesteps (M)
S c
o r e
Climber (easy)     25 Timesteps (M)
S c
o r e (S,T)
Test Color jitter
Ninja     25 Timesteps (M) 1.
S c
o r e
[Figure]
25 Timesteps (M)
[Figure]
S c
o r e
[Figure] [Figure]
Timesteps (M) 2.53.54.55.56.
S c
o r e
[Figure]
25 Timesteps (M) -
S c
o r e
[Figure]
25 Timesteps (M)
[Figure]
S c
o r e
[Figure]
[Figure] [Figure]
[Figure] [Figure]
S
[Figure] [Figure]
[Figure]
S c
[Figure]
K Benchmark on Modified Open AI Procgen
We compare the training and test performance on various environments with each augmentation.
We also use DrAC [26], RAD [21], DrAC+PAGrad as baselines.
In every result, we train the agent and standard deviation is calculated after five runs.
We show the result about random conv, color jitter, random crop in Table 1.
Additionally, we evaluate benchmark with gray and cutout color in
Table 7.
For benchmark, we classify Procgen environments with each characteristic as Figure 23.
Furthermore, we attach detail results on each environments with Oracle and Rand-FM [22].
Red one is the Oracle score, which is trained on test environments such as easybg-test, easy-test .
For your information, RAD does not work well when using crop, because we use [26]’s crop method which is different with [21].
Augmentation
[Table] [Table]
[Table] [Table]
the best method.
K.1 Random convolution
[Table] [Table]
[Table] [Table]
[Table] [Table]
K.3 Color jitter Easy
[Table]
[Table] [Table]
[Table] [Table]
K.4 Gray
[Table] [Table]
[Table] [Table]
[Table]
Coinrun 5.
[Table]
[Table] [Table]
[Table] [Table]
Fruitbot 11.
[Table]
[Table] [Table]
[Table] [Table]
Heist
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Jumper 6.
[Table] [Table]
[Table] [Table]
[Table] [Table]
Maze
[Table] [Table]
[Table] [Table]
[Table] [Table]
Ninja
[Table] [Table]
[Table] [Table]
[Table]
InDA
ExDA
Climber
[Table] [Table]
K.5 Cutout color
[Table]
Maze
[Table]
Ninja
[Table]
Bigfish
[Table]
Dodgeball
[Table]
Plunder
[Table] [Table]
[Table] [Table]
[Table] [Table]
L Primitive evaluation on DeepMind Control Suite with SAC
We experiment on DeepMind Control Suite (DMC) with a preliminary experiment which provides justification for our proposed method ExDA due to the limited time and computation resources.
We first note that [26] has demonstrated that UCB-DrAC, which is similar to UCB-InDA, can accelerate
RL training in DMC.
Hence, in order to justify our proposed method UCB-ExDA, which combines
UCB-InDA and ExDA, it would be sufficient to show the existence of the cases showing a benefit of ExDA in DMC.
Furthermore, we use SAC as the base RL algorithm instead of PPO to show the versatility of our method ExDA.
SAC is off-policy actor-critic RL algorithm which trains actor and critic networks separately with a replay buffer.
Thus, we can use stored data in the replay buffer after
RL training
[Table]
and also we can only distill the output of the actor-network, because the actor-network determines the policy π from observations, while the critic-network computes the value function V from observations.
We denote the actor-network parameters using θ same as the notation of PPO in Section 4.
We ablate the effect of ExDA to compare w/ and w/o inconsistency loss of policy on original observations.
Thus ExDA is trained with Eq 5 after SAC, but ablated ExDA is trained with
Eq 12.
[Figure]
[Equation] [Equation]
Both methods are trained 10 epochs after 10K time-steps SAC training and detailed hyper-parameters are described below the table.
We compare the mean reward of each method SAC, ExDA, abl-ExDA on three environments Walker-Walk, Finger-Spin, Cartpole-Swingup and 5 seeds.
We evaluate each reward from 50 evaluation episodes in train environments.
As below Figure 24, ExDA almost maintains the mean reward of SAC, abl-ExDA degrades the performance of SAC by distilling policy only to augmented observations except the original.
Hyperparameter
Value # of action repeat # of frame stack
Data augmentation
Random Convolution
Learning rate of DA 10 −
Batch size # of train steps # of distill epochs
Replay buffer capacity
Init steps
Learning rate of critic 10 −β of critic 0.
τ of critic 0.
Target update frequency of critic
Learning rate of actor 10 −β of actor 0.
Log std min of actorLog std max of actor
Update frequency of actor
Encoder type pixel
Feature dimension of encoder
Learning rate of encoder 10 −τ of encoder 0.
# of layers # of filters
Latent dimension
Discount factor 0.
Learning rate of α 10 −β of α 0.