Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning
(s
Abstract
In deep reinforcement learning (RL), data augmentation is widely considered as atool to induce a set of useful priors about semantic consistency and improve sampleefficiency and generalization performance.
However, even when the prior is usefulfor generalization, distilling it to RL agent often interferes with RL training anddegenerates sample efficiency.
Meanwhile, the agent is forgetful of the prior due tothe non-stationary nature of RL.
These observations suggest two extreme schedulesof distillation: (i) over the entire training; or (ii) only at the end.
Hence, we devisea stand-alone network distillation method to inject the consistency prior at any time(even after RL), and a simple yet efficient framework to automatically schedulethe distillation.
Specifically, the proposed framework first focuses on masteringtrain environments regardless of generalization by adaptively deciding which orno augmentation to be used for the training.
After this, we add the distillationto extract the remaining benefits for generalization from all the augmentations,which requires no additional new samples.
In our experiments, we demonstratethe utility of the proposed framework, in particular, that considers postponing theaugmentation to the end of RL training.
Introduction
Deep reinforcement learning (RL) aims at finding a neural network to represent policy or valuefunctions taking raw observation as input, of which the most common form in practice is visualdata or images of high-dimensionality, e g., video games [23], board games [29, 28] and robotcontrols [32, 18].
RL handling high-dimensional input often suffers from poor sample efficiency andgeneralization capability, mainly due to the curse of dimensionality [4, 15].
To overcome these issues,it has been widely considered to augment data based on prior knowledge that a set of transformationspreserve the meaning or context of input observations, e g., cropping out unimportant parts of images,and changing colors [20, 19, 21, 14].
On one hand, RL agent can be directly fed with the original andaugmented data so that it implicitly learns a representation with the prior and improves the sampleefficiency and generalization [20].
On the other hand, the prior knowledge in data augmentation canbe explicitly distilled via a self-supervised learning, which introduces additional regularization toensure consistency between responses to original and augmented inputs [25, 14].
However, data augmentation shows highly task-dependent effect in RL, and it is prone to generatesevere interference with the training even when it truly conveys a useful prior to train and test environments [20, 25, 14].
We address the problem of alleviating the interference between data augmentationand RL training to improve sample efficiency for acquainting train tasks, and generalization capabilityfor unseen test environments.
This problem in (online) RL is more critical and challenging than thatin supervised learning (SL) since the objective function and data distribution are time-varying in RL,while they are not in SL.
Indeed, according to [1, 10], it is sufficient to partly apply data augmentation just for a short period of SL at any time.
Meanwhile, we empirically observe that the prior from dataaugmentation can be easily forgotten in RL of the non-stationary nature (see Section 5.3), i.e., theeffect of augmentation is time-sensitive .
Based on our observation about the interference and the time-sensitivity, we propose two simple yeteffective methods according to timings of data augmentation :
In tra D istillation with A ugmentedobservations (InDA) and Ex tra D istillation with A ugmented observations (ExDA).
Data augmentationbeneficial for the sample efficiency needs to be applied over the entire RL training, i.e., InDA.
Conversely, data augmentation useful only for the generalization should be postponed to the end of
RL training, i.e., ExDA, so that we can minimize the interference in the training, while enjoying thebenefit in the testing.
InDA and ExDA are equipped with D istillation with A ugmented observation(DA).
DA is a stand-alone self-supervised learning which enables us to induce the prior after RLtraining, and shows a relatively small interference with RL training by explicitly preserving theresponse of RL agent to the original input.
The best timing (InDA or ExDA) depends on traits of train task and augmentation.
We hence suggesta framework of adaptive scheduling, named UCB-ExDA, that (i) first aims at maximizing the sampleefficiency by adaptively selecting which or no augmentation to be used during RL training ; and then,(ii) distill the priors from all the augmentations after RL training .
Specifically, inspired by [25], wedevise UCB-InDA for the adaptive selection in the first part by modifying the upper confidence bound(UCB) algorithm [3] for multi-armed bandit, where differently from [25], the set of arms includes allthe augmentations and the option to not augment .
In summary, UCB-ExDA is nothing but executing
UCB-InDA followed by ExDA.
Our experiment demonstrates the utility of the proposed framework.
Our contributions are summarized as follows:
(i)
We devise DA (Section 4.1) which explicitly preserves the knowledge of RL agent so thatenables distilling the consistency prior of augmentation into RL agent not only during but
also after RL training, while other methods [20, 25, 14] need to be applied concurrently withRL training and thus show relatively strong interference in our experiments (Section 5.2).
(ii)
We identify the simple yet effective timings of data augmentation: either InDA or ExDA(Section 4.2, Section 4.3), based upon the discovery of the time-sensitivity (Section 5.3) thathas not been observed in SL [10],
i
e., the proposed timings are effective particularly for RL.(iii)
We finally establish UCB-ExDA which automatically decides the best timing of augmenta-
tion, and demonstrate its superiority compared to others (Section 5.4).
The advantage of
UCB-ExDA is particularly substantial when the best strategy is ExDA postponing augmen-
tation to the end of RL training
Related Works
Augmented experience in RL.
To solve the problem of poor generalization and sparse data, apopular approach is to fabricate virtual experiences and train the RL agent to learn with them.
Domain randomization is a technique to produce such experiences from a simulator of a targetedsystem, [32, 24, 26].
Accurate simulators of practical systems are difficult to obtain, and it haslimited applicability.
However, visual augmentation has no such limit because the method uses simpleimage transformations such as cropping, tilting, and color jitter, although applications require acareful understanding of the targeted system to guide the design of an appropriate image transformer.
A method of curriculum learning for domain randomization, in which the difficulty is graduallyincreasing [26] provided insights that coincide with some of our findings.
However, we provide afurther understanding of the types of visual augmentation that should be early or late during training.
Regularization from augmented data in vision-based RL has been implemented in various learningframeworks, including but not limited to representation [14, 31], self-supervised [25], and contrast[30].
One proposed algorithm [25] applies the UCB algorithm [3] to automatically select the mosteffective augmentations over RL training, where each augmentation is considered as an arm and thenevaluate the effectiveness of augmentation by using a sliding window average.
The idea of adaptingaugmentation concurs with our main message regarding the timing of augmentation.
In [25], ’notaugmenting’ is not an option, whereas our findings indicate that it should be.
In addition, [25] doesnot consider post augmentation followed by RL training, as in ExDA.
Other time-sensitivity in deep learning.
During deep learning, the early stage of training often hasa significant effect [7, 1].
Therefore, we devised time-sensitive methods that adapt to the progress oftraining, such as learning rate decay [35] and curriculum learning [33].
Golatkar et al .
[10] studiedsuch a time-sensitivity of regularization techniques for SL, where the effect of data augmentationin different time does not change much.
We find that the time-sensitivity of augmentation can besignificant in RL.
This contrast may occur because of the non-stationary nature of RL, which SLdoes not have.
Although a set of techniques originally developed for SL such as convolutional neuralnetwork, weight decay, batch normalization, dropout and self-supervised learning improve deep RL[16, 6, 22, 8, 30, 34, 13], a thorough study should be conducted before introducing a method fromdifferent learning frameworks, because we find the contrasting time-sensitivities of data augmentation.
This spirit is also shared with an application [17] of implicit bias in SL [12, 2, 9] to RL.
Background
Notation.
We consider a standard agent-environment interface of vision-based reinforcementlearning in a discrete Markov decision process of state space S , action space A , and kernel P =
P ( s t +1 , r t | s t , a t ) which determines the state transition and reward distribution.
The goal of the RLagent is to find a policy that maximizes the expectation of cumulative reward (cid:80) t ′ − 1 t =
γ t r t , wheret ′ is terminating time and γ ∈ [0 , 1] is discount factor.
At each timestep t , the agent selects anaction a t ∈ A and receives a reward r t and an image o t +
= O
( s t +1 )
∈ R k × k ′ as an observationof the next state s t +1 .
Data augmentation can be described by an image transformation functionϕ : R k × k ′ (cid:55)→ R k × k ′ of which output has the same or similar semantics of the input.
Baseline RL algorithm.
Throughout this paper, we use Proximal Policy Optimization (PPO)
[27]as a baseline, although we believe our findings and methods can be easily adjusted to others.
PPO is arepresentative on-policy RL algorithm to learn policy π θ ( a | o ) and value function V θ of neural agentparameterized by θ .
Storing a set of recent transitions τ t :=
( o t , a t , r t , o t +1 ) in experience buffer D ,the agent is updated to minimize the following loss function:
[Equation]
where α is a hyperparameter and some regularization terms are omitted.
The clipped policy objectivefunction L π and value loss function L V are defined as:
[Equation]
where the expectation ˆ E is taken with respect to τ ∼ D , we denote by θ old the parameter before theupdate, ρ t ( θ ) is the importance ratio π θ ( a t | o t )
π θ old ( a t | o t ) , and ˆ A t is the generalized advantage estimation [27].
Method
In what follows, we present our methods: D istillation with A ugmented observations (DA; Section 4.1),
In tra DA (InDA; Section 4.2), Ex tra DA (ExDA; Section 4.3), and then the adaptive schedulingframeworks based on UCB (UCB-InDA and UCB-ExDA; Section 4.4).
DA is a stand-alone knowledge distillation method, which can be used at any time to instill the underlying prior of augmentationinto a given RL agent.
InDA and ExDA conduct either DA or PPO in each epoch but have differentschedules (Figure 1), where InDA interleaves PPO and DA, whereas ExDA performs PPO first then
DA.
UCB-InDA adaptively decides which or no augmentation to be used in each DA epoch of InDAbased on the UCB of estimated gain from each option.
UCB-ExDA performs ExDA preceded by
UCB-InDA.
4.1 Distillation with augmented observations (DA)
The underlying prior of augmentation can be infused by minimizing a measure of inconsistencybetween the agent’s responses to original and augmented inputs (resp.
o t and ϕ ( o t ) ).
For instance,
[Figure]
with PPO agent learning policy π θ and value V θ , Raileanu et al .
[25] proposes the following measure:
[Equation]
which uses Kullback-Leibler divergence (first term) and mean squared deviation (second term) forpolicy and value inconsistencies, respectively.
Noting that L dis ( θ ; ϕ ) can be minimized to be zero bya constant response to all inputs, the distillation with Eq (4) can distort the RL agent, in particular,when applying it outside of RL training.
We hence devise a network distillation technique (DA) which explicitly preserves the RL agent’sresponse to original input and thus is applicable even after RL training.
DA distills the knowledge of
RL agent θ old into θ using not only original but also augmented observations.
More formally, the lossof DA is written as:
[Equation]
We here denote the identity transformation by I such that I ( o ) =
o for all o , and extend the definitionof L dis in Eq (4) as follows:
[Equation]
In Eq (5), the first term ensures that θ and θ old behave identically for the original inputs, and thesecond one infuses the consistency prior.
In Figure 2, L dis ( θ, I ; θ old ) , L dis ( θ, ϕ ; θ old ) , and L dis ( θ, ϕ ; θ )graphically correspond to (A), (B), and (C), respectively.
From this, it follows that minimizing L DAin Eq (5) eventually reduces L dis ( θ, ϕ ; θ ) in Eq (4).
In addition, the minimization of L DA secures theresponses of θ to the original inputs (which can be pre-computed to reduce computation cost) to thethose of θ old , while the alternatives (e.g., (A)+(C): L dis ( θ, I ; θ old ) + L dis ( θ, ϕ ; θ ) ) does not and thusmay generate interference with RL training.
Our experiments (Section 5.2; Figure 3) justifies thedesign of DA by showing substantial advantage compared to the other alternatives such as DrAC[25] using L dis ( θ, ϕ ; θ ) in Eq (4).
We note that the advantage becomes much clearer when a wrongaugmentation is given, c.f., the supplementary material.
4.2 Intra distillation with augmented observations
InDA (Algorithm 1) alternates between minimizing L PPO and L DA , i.e., PPO and DA are explicitlyseparated, whereas they are often executed simultaneously in other methods [25].
Such a clearseparation reduces the interference [14].
We can control the frequency and timing of applyingdistillation with hyperparameters I , S and T , where we perform DA after each I rounds of RLtraining only if the current epoch n is in the interval of [ S, T ] , i.e., S and T are the epochs to beginand terminate DA, respectively.
We vary S and T to study the time-dependency of data augmentation.
We denote InDA [ S, T ] to indicate the period to apply DA, while we omit the indication when DA isapplied over the entire period.
We provide further details on InDA in the supplementary material.
Require: N, I, ϕ , S , T 1: Initialize θ close to origin.
2: for n = 1 , 2 , . . .
, N do 3: //
RL training 4:
Store sampled transitions to D ; 5:
Optimize RL objective L PPO ( θ ) with D ; 6: //
Distillation 7: if n ∈ [ S, T ] and mod ( n − 1 , I ) =
then 8:
Store θ old ← θ ; 9:
Minimize L DA ( θ ) for D , θ old and ϕ ; 10: end if 11: end for
Require: N , M , ϕ 1: Initialize θ close to origin.
2: //Pre-training phase with RL algorithm 3: for n = 1 , 2 , . .
. , N do 4:
Store sampled transitions to D ; 5:
Optimize RL objective L PPO ( θ ) with D ; 6: end for 7: 8: Store θ old ← θ ; 9: //
Distillation at the end of RL training 10: for m = 1 , 2 , . .
, M do 11:
Minimize L DA ( θ ) for D , θ old and ϕ ; 12: end for
4.3 Extra distillation with augmented observations
ExDA (Algorithm 2) performs the distillation after the end of RL training, where the lengths of DAand RL training are parameterized by M and N , respectively.
We note that computational cost canbe reduced by removing the value inconsistency measure ˆ E o t ∼D (cid:2) ( V θ old ( o t ) − V θ ( ϕ ( o t ))) 2 (cid:3) from
L dis in Eq (6) because the value consistency is not necessary for RL training nor distillation in theactor-critic framework and including it has a potential risk of generating additional interference.
In thesupplementary material, it is empirically verified that this reduction does not degrade RL performance.
We leave more interesting details in the supplementary.
For instance, one can consider re-initializingθ before starting DA as a part of exploiting the implicit bias [12, 17] to improve generalization.
However, test performance is often dropped.
This is mainly because the dataset to distill π θ old hasmuch less diversity than that observed along the trajectory.
Thus, we use no re-initialization for theexperiments in the main paper.
4. UCB-based adaptive scheduling frameworks
The training benefit by augmentation differs depending on the task.
This dependency complicates thechoice of whether to use InDA or ExDA for augmentation.
Hence, we devise an auto-augmentationmethod, called UCB-InDA, inspired by UCB-DrAC [25], where each augmentation is correspondedto an arm in multi-armed bandit (MAB) problem and assessed its potential gain in training withupper confidence bound (UCB) [3].
More formally, in UCB-InDA, the set of arms is the set ofaugmentations, Φ = { ϕ 1 , . . .
, ϕ K } , which must include the identity function I , i.e., the option not toaugment.
The inclusion of identity function is small but makes crucial difference than UCB-DrAC[25] since we observe that using augmentation sometimes needs to be postponed after RL training forthe sake of better sampling complexity and test performance.
A round of MAB corresponds to every I epoch of InDA, where we let ϕ k ( s ) ∈ Φ be the augmentationselected at the s -th round of DA.
Let G ( s ) be the average return, the sum of estimated advantage ˆ
Aand predicted value V θ , over ( I − 1) epochs of PPO followed by the s -th DA.
UCB algorithm assumesthat each arm generates reward independently drawn from a fixed distribution, and estimates theempirical mean over the entire sampling process.
However, in RL, the return G ( s ) is non-stationary,so UCB-InDA computes moving average gain ¯ G k ( s ) , instead, taken over a certain number (chosento be 3 in our experiment) of most recent rounds selecting ϕ k as Raileanu et al .
[25] proposed.
Then,inspired by UCB1 algorithm [3], UCB-InDA selects action k ( s ) at round s as follows:
[Equation]
where c is the UCB exploration coefficient, and N k ( s ) is the number of times selecting ϕ k up to rounds .
We refer to the supplementary material for the hyperparameter choice.
We remark that compared to
UCB-DrAC [25], the proposed UCB-InDA has subtle but important differences, summarized in twofolds: (i) the inclusion of identity transformation (i.e., no augmentation) and (ii) the distillation with augmentation via InDA.
The gain of each component is numerically studied in Section 5.
Finally, wenote that UCB-ExDA is nothing but UCB-InDA followed by ExDA.
Experiment
5.1 Setups
Train and test tasks.
We use the OpenAI Procgen benchmark of 16 video games [5], where amain character tries to achieve a specific goal, e.g., finding exit (Maze) or collecting coins (Coinrun),while avoiding enemies given a 2D map.
At each time t , visual observation o t is given as an imageof size 64 × 64 .
A train or test task is to achieve a high score on a set of environments configuredby game and mode, where a mode describes predefined sets of levels (e.g., complexity of map) andbackgrounds.
Cobbe et al .
[5] provide easy mode for each game, consisting of 200 levels and acertain set of backgrounds.
We simplify easy mode and train agents in easybg mode, of which theonly difference from easy mode [5] is showing only a single background. This is useful to investigatethe case that using visual augmentation is helpful for testing but not for training.
We denote thetask configuration by game_name( mode ), e.g., Coinrun( easybg ).
Then, we evaluate generalizationcapabilities using two modes: test-bg and test-lv , which contain unseen backgrounds and levels,respectively, in addition to easybg mode that we use for training.
Types of augmentation.
For clarity, we mainly focus on two types of visual augmentation, each ofwhich conveys distinguishing inductive bias:
(a) Random color transforms an image by passing through either color jitter layer or randomconvolutional layer proposed in [21].
From this, we can impose the consistency to color
changes, which may provide a strong generalization to diverse backgrounds of test-bg mode.
(b) Random crop leaves a randomly-selected rectangle and pads zeros to the rest of the image
This augmentation is particularly useful in fully-observable games (e
g., Chaser and
Heist), because it imposes an efficient attention mechanism.
We also report the result with other augmentations including color jitter, random convolution, gray,and cutout color in the supplementary material, from which the same messages can be interpreted.
Baseline methods for RL with data augmentation.
We mainly compare the proposed methods(InDA and ExDA) to the following baselines:
(a) RAD [20] simply feeds PPO with experiences of original and augmented observations.
(b) DrAC [25] is a method to simultaneously minimizing L PPO in Eq (1) and L dis in Eq (4).
(c) DrAC+PAGrad is a variant of DrAC, which we devise to investigate another mechanism
to relieve the interference between RL training and augmentation.
PCGrad (projecting conflicting gradients) [36] mitigates the interference in multi-task learning.
Noting the
distillation is an auxiliary task for RL
, we devise DrAC +PAGrad (projecting auxiliary
gradient
) which is a gradient-based method towards the negative of the following direction:
[Equation]
(8) where we remove the components of ∇ L dis opposite to ∇ L PPO which may disturb optimizingthe main objective L PPO .
The supplementary material presents further details and experiments, which we omitted for simplicity.
All results in the main paper are averaged over five runs.
[Figure]
5.2 Benefit of separating distillation from RL training.
In Figure 3, after training PPO agent up to 20M timesteps, we start to suddenly apply one of thedifferent distillation methods with random crop.
We report averaged scores over 6 environments(Bigfish, Dodgeball, Plunder, Chaser, Heist, Maze) after normalized by the highest train score of PPOon each environment.
After 20M timesteps, ExDA and ExDrAC have no RL training but minimize L DA in Eq (5) and L dis in Eq (4), respectively.
The substantial gap between ExDA and ExDrACjustifies the design of DA explicitly preserving the knowledge from RL when distilling the prior.
More importantly, it is remarkable that ExDA promptly learns the generalization ability once it startsto distill the prior.
This validates the idea of postponing the distillation RL training
We now compare the distillation methods (InDA, DrAC, and DrAC+PAGrad) concurrently optimizingthe RL objective and distilling the prior in Figure 3.
Each method has performance degeneration dueto the interference generated by distillation.
However, InDA and DrAC+PAGrad have clearly smallerdegeneration than DrAC which is the only one without any separation of optimizing the RL objectiveand distillation loss.
We note that DrAC+PAGrad has more interference than InDA, and it seems tofail to impose the prior since there is not much difference to PPO in testing.
Hence, this verifies thesuperiority of InDA which enables distilling the prior while alleviating the interference.
5.3 Effective timings of distillation
In what follows, we aim to identifying effective timings of distillation.
To this end, in Figure 5, wetest several schedules of applying DA on two representative environments of distinguishing traits.
The supplementary material presents the result on more environments, which is similar one of therepresentatives.
An effective timing: InDA.
Chaser with random crop (Figure 5(a) and 5(b)) represents the casewhen augmentation improves the sample efficiency of training and thus the generalization abilityin training.
To compare the generalization ability, we measure policy distance between originaland augmented observations using Jenson-Shannon divergence (Figure 5(c)).
InDA[0, 5]’s policydistance is increased after it stops using DA.
Thus, the generalization ability is degraded, if we donot continue to use DA.
In this case, it is clear that InDA should be applied during the entire RLtraining, as InDA[0, 25] shows the best performance in both training and testing.
In addition, it isalso important to apply DA as soon as possible since the effect of InDA[0,5] applying DA in thebeginning is relatively prompt and significant comparing to that of InDA[20,25].
This suggests thatthe automatic framework needs to explore more in the early stage.
An effective timing: ExDA.
Heist with random crop (Figure 5(d) and 5(e)) shows the oppositeuse of data augmentation to what Chaser case suggests, i.e., postponing data augmentation as muchas possible.
Random crop generates a slight interference, although it immediately improves thegeneralization ability.
We remark that the inductive bias from the random crop is easily forgotten, [Figure] as the test performance of InDA[0, 5] is saturated right after stopping the distillation.
This can beexplained by the time-varying nature of sample distribution and objective in RL.
Interestingly, itis in contrast to the data augmentation in SL, where an early distillation is sufficient to impose theprior [10].
On the other hand, the test performance curve of InDA[20, 25] soars right after DA.
Furthermore, Figure 5(f) shows that InDA[20, 25] narrows the gap between two policies on theoriginal and augmented observation.
Recalling the interference between RL training and distillation,this suggests postponing the distillation at the end of RL training, and motivates our ExDA.
Performance benchmark on InDA and ExDA.
In Table 1, we summarize the train and testperformances of InDA and ExDA on a set of games with different augmentations and modes.
ExDAoutperforms other baselines with random color on test-bg .
Moreover, we note that ExDA consumesonly 0.5M timesteps to inject the prior at the end of RL training, whereas the others use all the trainingdata.
ExDA in both sample efficiency and generalization ability with random crop on test-lv .
It isworth noting that DrAC+PAGrad is slightly better than DrAC, while there is a substantial gap between
InDA and each DrAC-based algorithm.
This again confirms the benefit from the separated distillationof
InDA observed in Figure 3.
These results demonstrate that each combination of environment andaugmentation has a suitable time at which to apply augmentation, and the gain from using the rightdistillation timing, i.e., online (InDA, DrAC, or DrAC+PAGrad) or offline (ExDA), is rigid regardlessof the choice of algorithms.
5.4 Adaptive scheduling methods
Adaptive selecting of timings: UCB-InDA and UCB-ExDA.
It is hard to know in advancewhether a certain augmentation helps the training or not [25].
We hence employ UCB-InDA whichestimates the gain or damage from each augmentation from trial and error and identifies the onewith most help.
Recall Table 2 where PPO (without augmentation) shows much better trainingperformance than InDA in Heist.
As shown in Figure 4(b), UCB-InDA is able to identify that noaugmentation is best for training in Heist.
This implies that ExDA is more appropriate than InDA.
Conversely, random crop is selected on Chaser (Figure 4(a)).
As the result, we can automaticallyselect InDA or ExDA appropriately for each task. Augmentation Task PPO
RAD
DrAC
DrAC+PAGrad InDA ExDA Randcolor
Rand conv
Train 1.
0.98 Test-bg 1.
2.11 Test-lv 1.
Color jitter Train 1. 0.98 Test-bg 1. 1.48 Test-lv 1.
Rand crop Train 1. 0.91 Test-bg 1. 0.95 Test-lv 1.
[Table]
[Figure] [Table] with both augmentation. Fully exploitation of augmentation In Table 2, when both random color and random crop are usedto improve generalization on both test-bg and test-lv , we report numerical evaluation of UCB-InDAand UCB-ExDA with other baselines on train and test tasks.
Decreased train performance of DrACand InDA compare to PPO show the difficulty of simultaneous training with several augmentations.
Train performance of UCB-InDA and UCB-DrAC are improved by adaptive selecting, especially,
UCB-InDA is better than UCB-DrAC.
The gap is made due to the robustness about the change ofaugmentation during training.
In terms of generalization, UCB-ExDA clearly surpasses UCB-InDAthanks to ExDA to extract all the priors from the complete set of data augmentations at the end of RLtraining.
Discussion
We have identified two most effective yet simple timings (InDA and ExDA) of data augmentation for
RL, and proposed UCB-ExDA framework to adaptively select the best scheduling augmentations.
Wenote that the effectiveness of this framework is restricted but specialized for RL with the unique nonstationary nature.
Indeed, in SL without shift of data distribution and objective, it is sufficient to applydata augmentation at the beginning [10].
Our framework employs the most basic multi-armed banditalgorithm with a finite set of data augmentation.
It is interesting to investigate a room to improve byfurther optimizing continuous parameters of data augmentation for RL, c.f., an auto-augmentationtechnique to optimize continuous augmentation parameter per sample for SL [11].
Another promisingdirection is to accelerate the distillation process of DA by data condensation with augmentation [37].
This is possible with our framework clearly separating between RL training and distillation, and maybe particularly useful to train distributed RL agents since a condensed data for an agent’s distillationis usable for the other.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We claim our three main contributions in 1 and targeton DeepRL.
(b) Did you describe the limitations of your work?
[Yes] We describe the limitations of our work about a limited set of data augmentation for UCB methods in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
We
believe that our research will not have a negative impact on society.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We read the ethics review guidelines.2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Our works are
based on experimental results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
Our works are based on experimental results.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We describeour training environments and include code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We describe training details in the supplementary material.
(c) Did you report error bars (e.g.,
with respect to the random seed after running exper-
iments multiple times)?
[Yes] We indicate standard deviation with score in Figure 5and Table 2 except normalized score with several environments such as Figure 3 and
Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We explain about training time in
the supplementary material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Our experiment
environments are baed on OpenAI Procgen [5].
(b) Did you mention the license of the assets?
[Yes] We include the license of the asssets
in the code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]