DNA: Proximal Policy Optimization with a Dual Network Architecture
Abstract
This paper explores the problem of simultaneously learning a value function and policy in deep actor-critic reinforcement learning models.
We find that the common practice of learning these functions jointly is sub-optimal due to an order-ofmagnitude difference in noise levels between the two tasks.
Instead, we show that learning these tasks independently, but with a constrained distillation phase, significantly improves performance.
Furthermore, we find that policy gradient noise levels decrease when using a lower variance return estimate.
Whereas, value learning noise level decreases with a lower bias estimate.
Together these insights inform an extension to Proximal Policy Optimization we call Dual Network
Architecture (DNA), which significantly outperforms its predecessor.
DNA also exceeds the performance of the popular Rainbow DQN algorithm on four of the five environments tested, even under more difficult stochastic control settings.
Introduction
Combining deep neural networks with reinforcement learning has produced impressive results on challenging problems, such as playing Chess [32], Atari games [25], and robotics tasks [31].
However, results have bifurcated between two competing approaches: Q-learning-based approaches that require learning an (action conditioned) value estimate and actor-critic policy gradient (AC-PG, or just PG)approaches that learn both a policy and value estimate.
PG offers many theoretical advantages over
Q-learning, such as natural support for continuous control problems and the ability to learn stochastic policies.
Until recently, PG approaches [24, 31, 11], while strong on continuous control problems, have under-performed Q-learning approaches on vision-based tasks [25, 16, 4], restricting the use of
PG in this domain.
This paper aims to close the gap between PG and Q-learning methods by showing that the common practice of jointly learning value and policy with shared features negatively affects the algorithm’s performance.
2 We also demonstrate an order-of-magnitude difference in noise levels between these two tasks and argue that this makes these two tasks poorly aligned.
In light of this result, we introduce a new algorithm, based on Proximal Policy Optimization (PPO) [31], called Dual Network
Architecture (DNA).
We test DNA empirically on a subset of the Arcade
Learning Environment (ALE) [6], Atari-5 [2].
Our results show a strong increase in performance on this benchmark compared to both PPO and another dual network model Phasic Policy Gradient (PPG) [11].
Our model also outperforms Rainbow DQN [16] on four of the five games tested, even under more challenging environmental settings.
We summarize our contributions.
First, we provide empirical results showing an order-of-magnitude difference in the noise scale between the policy gradient and the value loss gradient.
Second, we give evidence for the benefit of using low bias value estimates for value learning, but low variance estimates for advantage estimates.
Finally, we introduce and justify our dual network constrained distillation algorithm DNA with empirical results on ALE.
Preliminaries and Related Work
Proximal Policy Optimization Our algorithm builds on the well-established Proximal Policy
Optimization (PPO) algorithm [31].
PPO is a policy gradient, reinforcement learning algorithm, with many of the advantages of its predecessor Trust Region Policy Optimization [29], while being much simpler to implement.
Various aspects of PPO have been studied in recent years, such as the impact of implementation choices [12, 19], the ability to generalize [11, 27], and performance under multi-agent settings [37].
There has also been a growing understanding that the value and policy learning tasks involved in actor-critic models like PPO have important asymmetries [27, 11].
Our work builds on this by investigating how differences in value and policy noise levels can be accommodated to improve PPO’s performance on video-based discrete action tasks.
TD( λ ) Return Estimation Policy gradient algorithms often make use of a value estimate as a baseline [24], as well as for estimating the value of truncated trajectories [9].
To better facilitate control over the noise levels for policy and value learning, our work makes use of two different return estimations, both using TD( λ ) [33].
Given value estimates V ( · ) from the value network we define the n-step value estimate for some state s t taken at time t , and their exponentially weighted sum as
[Equation]
For values of λ close to 1 , more weight is assigned to longer n-step return estimates, and less to shorter ones.
There has been a long-standing belief that shorter n-step returns generate more biased estimates, whereas longer n-step estimates, due to summing over many stochastic rewards, have higher variance.
For a more thorough discussion on this topic see [20].
Noise Scale.
Noise scale is a measure of noise-to-signal in stochastic gradient descent (SGD).
While SGD provides unbiased gradient estimates, these estimates are typically very noisy.
Each sampled gradient estimate ˆ G can be thought of as a sum of the true gradient ⃗G and some noise vector ⃗σ .
A method for estimating the ratio of the magnitude of this implied noise vector to the magnitude of the true gradient was proposed by [23] who show that their efficient-to-calculate simple noise scale is a good match for the noise scale.
They also show that the noise scale provides useful information about the choice of mini-batch size to use when estimating gradients.
Phasic Policy Gradient.
Most similar to our work is Phasic Policy Gradient (PPG) [11].
Like DNA,
PPG has three distinct phases during training and uses two independent networks.
However, there are several important distinctions in our work.
First, we forgo the large replay buffer, relying instead on learning entirely from recent experience.
This significantly reduces the memory requirements of our algorithm.
Second, we make use of a distillation phase rather than an auxiliary phase.
Finally, our work reduces gradient noise by using two different return estimators calibrated for the characteristics of each task.
We assess the impact of these differences on the performance of the agent in Section 6.
The Noise Properties of Value Learning and Policy Gradient
Here we examine the noise properties of the policy gradient and value learning tasks and show that the policy gradient has a much higher noise level than value learning.
This implies different challenges in the optimization problem.
Specifically, that policy should be learned with a much larger mini-batch size than value learning and that return estimates should be adapted appropriately.
3.1 Motivating Example
To motivate our investigation into noise levels, we consider the task of learning two independent functions F 1 ( x ) := sin(5 x ) + N (0 , σ 21 ) and F 2 ( x ) :
= cos(5 x ) + N (0 , σ 22 ) , where N ( µ, σ 2 ) is
Gaussian noise with mean µ and standard deviation σ , over the domain [ − π, π ] .
We fix σ 2 = 1, and vary σ 1 on a log scale from 0 .
1 to 100 .
We trained two 3-layer multi-layer perceptions (MLPs) on this problem.
The first was a joint model, using a first hidden layer of 1024 units and a second hidden layer of 2048 units, followed by two output heads.
Our second model was a dual network consisting of two independent MLPs with 1024 units on each hidden layer.
6 All models used ReLU activations in between linear layers, and error was measured as mean-squared-error (MSE) between the predicted value and the noise free true value.
The results are presented in Figure 1.
At low noise levels, the tasks did not interfere, but as the noise of the first task increased, performance on the second task eventually degraded for the joint network, but not the dual network.
While simple, this experiment demonstrates the impact one noisy task can have on another when learned jointly by a deep neural network.
[Figure]
3.2 Noise Scale in Reinforcement Learning
To assess the differences in noise levels in our experiments, we measure the gradient noise scale of the policy gradient and value loss gradient.
We did this by calculating an estimate of the noise scale, developed by [23] called simple noise scale defined as
[Equation]
where Σ is the gradient covariant matrix, and ⃗G is the true gradient.
For convenience we also use the notation σ := (cid:112) B simple , which can interpreted as the ratio of the length of the implied noise vector to the length of the true gradient vector.
That is, a kind of noise-to-signal ratio.
As [23] have shown, a low bias estimate of B simple can be found efficiently by generating gradient estimates ⃗G B small using a small mini-batch of size B small as well as the gradient
⃗G B big using a large mini-batch of size B big , then calculating
[Figure]
[Equation]
where E [ |G| 2 ] = | ⃗G | 2 and E [ S ] = tr (Σ) .
To evaluate the noise scale of value learning and policy gradient we trained our dual network model (described fully in section 4) on the three Atari games from the Atari-3 validation set [2].
Like [23], we found it necessary to smooth out noise by maintaining an exponential moving average of | ⃗G B big | 2 .
We used B small = 32 and B big = 16 , 384 in all our experiments.
Pseudocode for this procedure is given in Appendix N.
The results are presented in Figure 2.
We found that policy noise ( σ π ) reduced by about half during training and that value noise ( σ v ) remained constant.
Both noise levels varied very little between the three environments.
A large variation was observed between the value loss and policy gradient, consistent between environments.
The noise levels, measured at the end of training and averaged over all three environments, were found to be σ π = 220 . 5 and σ V = 17 . 5 , representing a 12 .
6 times difference.
Dual Network Architecture
Based on our experimental results in Section 3, we propose an architecture which takes into account the large difference in noise levels between the two tasks.
This architecture consists of three improvements to PPO.
First, to reduce negative interference from the noisy policy gradient, policy and value should be learned by independent networks with different hyperparameters (e.g. training epochs and mini-batch size).
Second, the variance/bias trade-off in the return estimations should be calibrated to each task.
Finally, a constrained distillation phase should be applied to take advantage of any constructive interference between the two tasks.
4.1 Independent Networks
Important to DNA is the use of a dual network architecture.
8 Not only does this setup allow for learning of the value and policy without destructive interference, it also enables a specialized set of hyperparameters to be calibrated for the distinct tasks of policy and value learning.
Specifically, the calibration of the mini-batch size, which it has been suggested is best set roughly proportional to σ [23].
The policy network outputs a policy π and a value estimate V π , whereas the value network only outputs a value V V , as depicted in Figure 3.
4.2 Decoupled Return Estimation
To better facilitate control over the noise levels for policy and value learning, DNA makes use of two different return estimations, both using TD( λ ) [33] as follows
[Equation]
where λ V and λ π are hyperparameters controlling the variance / bias trade-off of each estimate.
The
V targ estimates are used as targets for training the value function, whereas V adv are used for advantage estimates used for policy gradient estimates given by
[Equation]
This formulation of the advantages is equivalent to the general advantage estimation (GAE) [30], that is, for any λ ∈ [0 .. 1] , γ ∈ [0 .. 1) we have,
[Equation]
Because of this, it is common when using PPO to generate value targets, V targ , for the value function by adding the network’s value estimates to the (unnormalized) advantages, implicitly settings λ V = λ π .
We hypothesize that due to bootstrapping, estimates used for V targ would benefit from being low bias , conversely, because estimates used for advantage estimation contribute to the noise of the (already very noisy) policy gradient, they would benefit from being low variance .
That is, the bootstrapping process may cause symmetrically distributed error, but not biased error, to cancel out.
This suggests setting λ π < λ V , a thought which we test empirically in section 6.2.
4.3 Distillation
Because value learning is a simpler task in terms of noise level, it makes sense to try to transfer some of the knowledge learned by the value network to the policy network.
If the value network learns to identify important features in the environment, we would like the policy network to identify those features also.
To facilitate this, we employ a constrained distillation update [17].
Distillation between two identical networks has the property that the student’s target function (since an identical network generated it) is guaranteed to be in the model’s solution space.
Also, because the distillation targets are deterministic, 11 the process is also very low noise, which we verify in Appendix D.
We contrast this with the auxiliary update of PPG which trains both the value function and the policy on an auxiliary task.
This auxiliary task equates to learning 1 N (cid:80) N − 1 i =
V π − i targ where V π − i targ is a value estimate for the policy from i updates prior.
For quickly changing policies, and large N , these updates may cause tension with the value estimates of the current policy.
We use V V ( s ) as the distillation targets, with the input states s being taken from the current rollout.
However, unlike policy and value training, distillation state inputs need not be on-policy.
We explored other distillation targets, covered in Appendix J, but found V V ( s ) to be the best of those tried.
Distillation is performed, like PPG’s auxiliary task, using mean squared error, and under a soft constraint on the policy network’s policy, specifically
[Equation]
where β is the policy constraint coefficient, and π old is a copy of the policy before the distillation update.
During updates gradients are only propagated through the policy network, and not the value network.
4.4 Training
Like Phasic Policy Gradient (PPG) [11], DNA splits training into three distinct phases (policy, value, and distil), but unlike PPG, rather than using a large replay buffer, we perform all updates on-policy on the current batch of rollout data.
Each of the three phases optimizes a single objective, for some number of epochs, using its own optimizer, with a unique set of hyperparameters.
The optimization objective for the policy phase of DNA is the clipped surrogate object from PPO [31] including the entropy bonus
[Equation]
where S is the entropy in nats, ρ t is the ratio π ( a t | s t ) π old ( a t
| s t ) at time t , ϵ is the clipping coefficient, and c eb is the entropy bonus coefficient.
For the value phase, we we use the squared-error value loss,
[Equation]
In summary, the DNA algorithm separates the tasks of policy learning and value learning into a network to handle the high noise of policy learning and a separate network to handle the lower noise task of value learning.
Knowledge from the value learning network is transferred to the policy network through a separate constrained distillation phase, which allows for constructive interference between the learning tasks while minimizing the destructive.
We formalize the algorithm as follows.
1: procedure PPO-DNA 2:
Input N ∈ Z + rollout horizon 3:
Input A ∈ Z + number of agents 4:
Input π the initial policy.
5: for i = 1 to ... do 6: for a = 1 to A do 7:
Run policy π in environment a for N timesteps 8:
Compute V targ ← TD ( γ,λ V ) 9:
Compute V adv ← TD ( γ,λ π ) 10:
Compute ˆ A ← TD ( γ,λ π ) − V adv 11: for j = 1 to E π do 12:
Optimize L CLIP wrt θ π 13: for j = 1 to E V do 14:
Optimize L V F wrt θ V 15: π old ← π 16: for j = 1 to E D do 17:
Optimize L D wrt θ π 18:
Output π
Evaluation
To evaluate our algorithm’s performance, we used the Atari-5 benchmark [2].
Scores in Atariare generated using a weighted geometric average over five specific games and produce results that correlate well with the median score if all 57-games had been evaluated.
This allowed us to perform multiple seeded runs and defined a clear training and test split between the games.
In all cases, we fit hyperparameters to the 3-game validation set and only used the 5-game test set for final evaluations.
We opted for the more difficult stochastic ALE settings recommended as best practice by [22].
However, to better understand our results in the context of prior work, we also provide results under the simpler deterministic settings in Appendix F, and additionally provide for reference a single evaluation on the full 57-game set in Appendix M. Unless otherwise specified, agents were scored according to their average performance over the previously completed 100-episodes at the end of training.
A coarse hyperparameter sweep found initial hyperparameters for our model on the Atari-3 validation set.
Notably, we found the optimal mini-batch size for value and distillation to be the minimum tested (256), while the optimal mini-batch size for policy was the largest tested (2048).
For optimization, we used Adam [21], over the standard 200 million frames.
13 Full hyperparameter details for our experiments are given in Appendix B.
In order to understand the impact of the hyperparameters introduced by our algorithm, namely
E π , E V , E D and λ π , λ V , we performed the following experiments on the Atari-3 validation set.
We started by setting E π = E V = E D = 2 , then searched over E V ∈ [1 , 2 , 3 , 4] , and selected the best
E V .
We then searched over E π and E D while keeping the other two parameters constant.
We also performed a similar experiment on the impact of λ V and λ π by setting λ V to 0.95 then sweeping across λ π ∈ [0 . 6 , 0 . 8 , 0 . 9 , 0 .
95 , 0 . 975] , then setting λ V to the best λ π and repeating.
This process allowed us to verify if the best settings occur at λ V =
λ π or at λ V ̸ = λ π .
During our λ tuning, we also recorded noise levels for one of the seeds, the results of which we discuss in Section 6.2.
We evaluated our algorithm DNA, against PPO, and PPG.
All models used the ‘NatureCNN’ encoder from [25].
However, because DNA and PPG use two networks and thus twice the parameters, we doubled the number of channels of the PPO encoder which we refer to as PPO (2x) and which is very similar to the encoder used by [4].
All models have have approximately 3.5 million parameters in total.
For fairness we repeated the same sweep on PPO for the number of training epochs ( E PPO ) and λ used for GAE ( λ PPO ).
To verify that our settings did not inadvertently degrade the performance of 13 That is million environment interactions.
[Figure]
[Figure]
PPO we also tested against the settings used by [31] and refer to this as PPO (original).
These results are included in the ablation study in Appendix C.
Main Experimental Results
In this section we present the results of our main study comparing DNA to PPO and PPG on Atari-5.
We also include supplementary experiments on MuJoCo [34] and Procgen [10] in Appendix K, and L respectively.
6.1 Impact of Training Epochs
We start by investigating the impact of the E π , E V , and E D hyperparameters.
We found that using less than three policy epochs for DNA dramatically increased the performance of the agent while also decreasing the computation required to train the agent (Figure 4).
We also found that DNA was robust to the choice of value and distillation epochs but that overtraining on value ( E V = 4 ) marginally decreased performance.
This was not the case for distillation updates.
The distillation phase demonstrated some benefit as indicated by the E D = 0 results under-perform the others.
We selected E π = 2 , E V = 1 , E D = 2 as the optimal settings and used these for the remainder of the experiments.
These settings require a total of four updates for the policy network, but only one update for the value network (we discuss this further in Section 7).
The training curves also suggest DNA may benefit from training beyond the standard 200 million frames.
6.2 Return Estimation
We found that the choice of λ π made a large difference to the performance of our agent.
As was hypothesised the optimal settings for λ π differed to that of λ V with λ π preferring a lower variance, higher bias value of 0 . 8 , while the value targets, λ V preferred a lower bias, higher variance setting of 0 .
95 (Figure 5).
Notable is that the non-homogeneous setting λ V = 0 .
95 , λ π = 0 .
8 outperformed both homogeneous choices
λ π = λ V = 0 .
8 and λ π = λ V = 0 .
We found, as expected, that policy gradient noise can be reduced by selecting a low value for λ π .
However, against our expectations, setting λ V lower actually increased, not decreased the noise level of the value learning task.
Once we
[Figure]
Rainbow
[Figure]
discovered this we ran addition experiments for a broader choice of λ V , and include the results in
Figure 5.
6.3 Comparison to PPO and PPG
We found DNA to outperform both PPO and PPG on the Atari-5 dataset by a wide margin (Figure 6).
We give Atari-5 scores, as well as training plots for each individual game.
We also include references for Rainbow DQN, although we note that these scores were generated under the simpler deterministic settings.
Discussion
Reduced Training as an Alternative to Experience Replay We found that less training, both on policy and value updates, typically improves our agent’s performance.
One explanation for this could be, as [11] have suggested, that in the case of value learning, this could be due to overfitting to recent experience.
In terms of policy updates, where the largest performance degradation was observed, this could be due to increased training epochs causing a large change in the policy, which may suggest a better constraint than PPO’s clipping is required.
15 That being said, limiting the training to one or two epochs provides straightforward solution to the problem while having the additional benefit of decreasing the computational resources required to train an agent.
Choice of λ
Our results give evidence for the benefit of selecting λ π < λ V .
As expected, picking lower values of λ π reduces the policy gradient noise.
However, counterintuitively, noise on value learning reduces with higher values of λ V .
That is, policy learning prefers low variance , but value learning prefers low bias .
This result may explain why clipping, which introduces bias, performs poorly on value learning but quite well on policy learning.
Our results also suggest that an even higher choice of λ V than the 0 .
95 we used in our experiments may be appropriate.
Update-to-Data Ratio Our work also raises an interesting question about the update-to-data (UTD) ratio differences between value and policy learning.
We found DNA to work best with only a single value network update but four policy network updates.
This asymmetry could be due to the extremely high noise levels for policy updates.
The single value update contrasts with other work where much higher value learning UTD ratios was found to be preferred [1, 8].
Limitations Our paper’s primary focus was improving PG results on discrete action vision-based problems, as this is where PG has traditionally unperformed Deep Q-learning.
However, we do also provide an initial look into continuous control problems on the MuJoCo [34] dataset in Appendix K.
Broader Impact In our work, we have shown that deep actor-critic reinforcement learning models can effectively solve complex vision-based, discrete-action environments, even without the use of large replay buffers.
Surprisingly this can be achieved using less not more policy and value network updates.
Because Q-learning approaches generally produce deterministic policies, algorithms based on them may ‘lock in’ to a decision based on minimal differences in outcome.
This is not always the case with stochastic policies, which are able to randomly split over actions of roughly equal quality, which may lead to fairer outcomes.
We do not foresee any direct negative societal impacts from this research.
Future work Our algorithm is already highly competitive with Rainbow DQN, and could likely be extended with orthogonal improvements such as adding recurrence [14], distributional value learning [5], improved feature encoders [13], better exploration bonuses [7], and making use of a learned model [28].
Conclusion
In this paper, we have highlighted noise level as a key difference between policy and value learning.
We have introduced an algorithm that accounts for this order-of-magnitude difference by limiting negative interference through the training of two independent networks for value and policy but retaining constructive interference through a constrained distillation process.
Furthermore, we have shown that the variance/bias tradeoff differs for value learning and policy gradient and that return estimation should cater for this.
Together these changes result in a novel algorithm, DNA, that outperforms its predecessor PPO, and even surpasses the popular Q-learning approach Rainbow
DQN, while under more challenging environmental settings.
Acknowledgments and Disclosure of Funding
This research was supported by an Australian Government Research Training Program (RTP) Scholarship.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix H. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Implementation Details
The implementation details often matter with reinforcement learning [12].
For this reason, full source code is provided in the supplementary material for all experiments.
This section details some of the more important implementation decisions made, most of which match those found in [3].
We ran all experiments (PPO, DNA, PPG) using these same implementation choices to confirm that performance differences were not due differences in implementation.
Reward normalization We normalized rewards such that returns have unit variance, as is common with PPO.
Even though we used two separate models, and therefore had less reason to balance the magnitude of the value and policy loss, we still kept reward normalization so that distillation loss, and its interaction with the policy constraint, would be of a similar scale between environments.
Observation normalization We also adopted observation normalization.
Each state s was normalized by s ′ =
clip (( s − s µ ) /s σ , − 3 , 3) where s µ was the element wide mean over states seen by the agent so far, and s σ was the standard deviation.
Normalization constants were shared between the policy and value networks.
Repeat action penalty We found that our policy would occasionally get stuck repeating a single action, causing the game to freeze until the time limit occurred.
This could occur if the agent mistakenly thought it would get a slightly negative score for continuing and therefore acted to postpone that reward as long as possible.
18 Q-learning algorithms, such as DQN, which make use of ϵ -greedy, do not experience this problem so long as ϵ > 0 .
To address this, we implemented a reward penalty of 0 .
25 (normalized) if the agent repeated the same action more than 100 times.
We leave finding a better solution to this problem for future work.
Integrating time and action information We added a watermark to the least recent frame in the 4-frame stack indicating the proportion of time which has occurred as a ‘progress bar’ as well as markers on each frame indicating which action the agent selected on the previous frame.
Inclusion of time is necessary to avoid violation of the Markovian property in time-limited environments [26].
Action indicators were added to allow the agent to understand when it repeated the same action multiple times, which is not always possible to determine from the state itself (due to multiple actions causing identical outcomes).
Warmup / desyncing environments When initializing our environments, we ran each of the parallel 128 environments for t ∼ U (1 , 1000) interactions with actions selected uniformly over the action space.
This served two purposes: to provide initial normalization parameters and to desynchronize the environments.
We found that if we did not do this, agents would terminate around the same time on some environments, causing parallel rollouts to become correlated.
While we found this made very little difference to the agent’s performance, it removed oscillating scoring artefacts found early in training in some environments.
B Hyperparameters and Environmental Settings
We selected initial hyperparameters from an initial coarse hyperparameter search on the Atarivalidation set.
In some cases, where only small differences in performances were observed, we prefered settings that had been used in previous papers or were likely to be more efficient.
For example, our search found a mini-batch size of 256 optimal for value and distil updates.
However, we selected 512 instead, as the difference was not large and found this a more computationally efficient mini-batch size when trained on a GPU.
A full list of hyperparameters are given in Table 1.
We also provide hyperparameters for our PPG experiments in Table 2.
The environmental settings we used are given in Table 3.
[Table]
[Table] [Table]
[Figure] [Figure]
S c
o r e ( A t a r i -)
DNA (ref)
DNA (mb=512)
DNA (no distil)
DNA ( V = =0.95)
PPO (basic)
Rainbow
C Ablation Study
This appendix quantifies the performance contribution of several important components of DNA.
We considered the following changes:
• DNA (mb=512)
Our analysis of noise levels suggested a much larger mini-batch size for policy updates than for value updates.
We measure the impact of this change by evaluating DNA with mini-batch sizes for all three training objectives set to 512.
• DNA (no distil)
Validation scores indicated an improvement in performance using distillation over no distillation.
We verify that this result is replicated on our test set.
• DNA ( λ V = λ π = 0 .
We measured the impact of using non-homogeneous values for λ V and λ π by testing with these both set to 0 .
We also include the reference run from the main study and a "PPO (basic)" run, which was a single network with the ‘Nature-CNN’ encoder, and a mini-batch size of 512, and can be thought of as
DNA with all novel components turned off.
Results are provided in Figure 7, along with the score, and performance regression in Table 4.
We found non-homogeneous values for λ V and λ π , and a larger policy mini-batch size, to be the most significant changes, with distillation also providing some benefit.
[Table]
D Noise Scale for Distillation Learning
In this appendix, we present the noise scale results for distillation learning.
We expected distillation to have a low noise level because the targets are drawn from the relatively noise-free value network estimations and not from the higher variance value targets.
Our results in Figure 8 confirm this hypothesis.
Of note is that the difference in noise scale between environments was much more significant for distillation loss than it was for the value or policy loss.
These results indicate that distillation may benefit from smaller mini-batch sizes.
However, the decreased efficiency of processing these smaller batches on GPU hardware may out-weigh any potential advantages.
E Proof of Relationship between GAE and TD( λ )
We provide a proof that calculating the General Advantage Estimate [30] is equivalent to calculating
TD( λ ) returns, then subtracting the state value estimate.
Concretely, for some λ ∈ [0 .. 1) and γ ∈ [0 .. 1] we have from [30]
[Equation]
[Equation]
The GAE advantage estimate is defined as an exponentially weighted sum of these A ( k ) ’s as follows
[Equation] [Equation]
[Equation] [Equation]
(18) as required.
F Results under Rainbow DQN Style Environmental Settings.
In our main study, we compared DNA to PPO on the Atari-5 benchmark under the recommended settings given by [22].
Many prior results have been generated using the simpler, non-stochastic version of the environments and with domain-specific knowledge, such as loss of life as a terminal state and a custom clipped reward modifier.
We evaluated DNA and our implementation of PPO here in the simplified environments and found a modest improvement under these settings.
We provide these results for better comparison against previous works.
In these experiments we did not use reward clipping.
Reward clipping is a domain-specific reward modification that reduces all positive rewards to +1 and all negative rewards to -1.
We were concerned that clipping rewards would bias the algorithm, as the agent is optimizing a reward structure that may not match the true rewards of the game.
It could be that reward clipping may be necessary for Deep
Q-learning approaches to reduce high variance returns.
However, we have not found this to be an advantage over reward normalization for PPO or DNA.
We found that DNA outperformed Rainbow DQN on all five environments, and obtained a better
Atari-5 score after just 49M environmental frames.
Proximal Policy Optimization, with a single policy update, also outperformed Rainbow DQN on this task.
[Figure]
G Supplementary Results
We investigated some supplementary questions.
Specifically, we wanted to validate that the performance improvement of DNA compared to PPO and PPG did not result solely from the hyperparameter choices.
We, therefore, evaluated PPO and PPG under a range of alternative hyperparameters on
Atari-5 and note the results here.
We tested a variety of alternatives for PPO as described below.
We found that none of the alternative settings resulted in improved performance (Figure 10 left ).
Or squashing the value see Appendix A
[Figure]
• PPO ( E π = 2 )
Our tuning process found a single epoch optimal for PPO, but two epochs optimal for DNA, would PPO have performed better if it was given two epochs instead of one?
We found that while initial performance was stronger, this change ultimately regressed the performance.
• PPO ( λ V = 0 .
95 λ π = 0 . 8 ) Using separate return estimations for advantages and value targets does not require a dual network setup.
Therefore we checked if the performance of PPO can be improved by using the non-homogeneous λ values used in our DNA experiment.
We found these settings regressed performance.
• PPO (1x)
In our experiments DNA and PPO used different network encoders (PPO used twice as many channels).
It is possible that increasing the parameters made training more difficult for PPO?
We tested PPO with the standard NatureCNN encoder, and found this change regressed performance.
• PPO (orig)
Our settings for PPO deviated from those used by [31].
For completeness we also provide results using these settings.
We found that while the these original settings performed well initially, they eventually unperformed the reference by a large margin.
We also note that these settings took much longer to train (see Appendix H).
We also evaluated PPG with alternative settings
Figure
10 right
• PPG (tuned)
In our main experiment we evaluated PPG using E π = 1 , E V = 1 , E aux = 6 taken from [
These differ significantly from those used for DNA.
We therefore reevaluated PPG using E π = 2 , E V = 1 , E aux = 2 which more closely match the settings used by DNA.
We found this change had little impact on the performance of the algorithm.
H Training Time
DNA makes use of two independent networks and three training phases, which may have a negative effect on training time.
We examine this here.
All times are approximate and for comparative purposes only.
Rainbow DQN times are on different hardware and using a different codebase.
Our implementation of DNA ran very quickly and is faster than PPO when PPO is configured as per [31].
This is due to our use of more parallel agents (128 vs 8) coupled with a larger mini-batch size.
We give approximate training times in Table 5 which were taken from a 24-core machine with four 2080-TIs.
We found we could train 8 DNA models in 8-hours on our 4-GPU machine, giving a rate of 4 GPU hours per game learned.
I Tuning for Proximal Policy Optimization
We repeated the same hyperparameter sweep on PPO as we did for DNA for a fair comparison.
We found that, like DNA, PPO also benefited greatly from reduced epochs during training.
We present
[Table]
the results here for 1,2,3 and 4 epochs, along with a search over the choice for λ used in the General
Advantage Estimate.
Results from the main study used the best performing model, which was found to be λ = 0 .
95 , and E π = 1 .
These hyperparameters differ from those used by [31], and achieve a significant improvement in performance with less computation (see Appendix F, H).
We found the performance of PPO to plateau after a point in training which decreases with the number of training epochs.
This is consistent with [31], who trained for 40M frames and whose results show performance levelling off around 20M. However, when fewer epochs are used, performance continues to increase after these points.
We also performed some quick experiments using partial epochs (0. and 0.75) but found these under-performed a single epoch and have not included the results here.
We also found that the single network setup of PPO did not benefit as much as DNA from tuning the λ parameter and that the commonly used λ = 0 .
95 was optimal for our training set.
[Figure]
J Distillation Targets
We evaluated two distillation targets for our distillation phase: a random projection into R 16 , and the value networks value estimates as well as a third strategy, feature matching .
22 Random projection and value estimate outperformed the no distillation baseline, and feature matching underperformed the baseline.
Results (Figure 12) are from a single seed on the Atari-3 validation dataset.
In all cases, distillation was trained on two passes of trajectories sampled from the rollout and used a policy constraint.
K Additional Results on MuJoCo
We applied DNA to the robotics task MuJoCo
We used hyperparameters based on the work of [31].
However, we used the slightly different ‘v2’ versions of the environments rather than the ’v1’ versions used in their experiments.
Scores are taken during training.
PPO and DNA learned a per action standard deviation independent of the state.
[Figure] [Figure]
DNA shows an improvement over PPO on five of the eight of the environments tested ( Ant , HalfCheetah , Reacher , Swimmer , and Walker2d )
(see Figure 13).
In the remaining three environments,
DNA and PPO produce similar results.
We performed only basic hyperparameter tuning, using the
Walker2D environment.
Hyperparameters for these experiments are given in Table
[Figure]
Unlike in our Atari experiments, we found setting λ π to 0 .
8 to produce poor results
[Figure]
and so reverted this setting back to 0 .
We removed the KL divergence penalty for distillation and replaced it with the mean-squared error between the center of the Gaussian distribution outputs for the policy and value networks.
[Table]
L Additional Results on Procgen
We ran additional experiments on the Procgen benchmark [10].
Each environment within this benchmark requires learning across a set of 200 procedurally generated environments.
Because of this, algorithms benefit from large replay buffers able to capture the diverse conditions under which the agent must act.
Unlike PPG, DNA does not make use of replay buffer and so is not likely to perform well at this task.
Despite this, we found DNA to be competitive with PPG on many of the environments tested ( coinrun , fruitbot , leaper , maze , miner , ninja , dodgeball and heist ), as shown in Figure 15.
When scores across all environments are normalized, DNA produces an average normalized score of 0 .
65 compared to 0 .
49 for PPO and 0 .
75 for PPG (see Figure 14).
[Figure]
We kept hyperparameters close to that of [11] and used the bigfish environment to select one policy, one value, and two distil epochs as optimal for DNA.
We note that these settings require the observations to be forwarded through a network only four times, compared to fourteen times in PPG.
This makes DNA similar in computational requirements to PPO.
We generated results for DNA and PPO and used the results supplied by [11] for PPG.
24 We also noticed a modest decline in the performance of our implementation of PPO compared to that of [11] (0.49 vs 0.58).
This could be due to our use of observation normalization.
The difference in performance is especially apparent in the environment plunder (for both DNA and PPO).
This might indicate that performance could be improved further by adopting the fixed scaling preprocessing procedure used in PPG.
We give hyperparameters for our experiments in Table
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
off
M Additional Results on ALE
Our main study used Atari-5 to allow enough time for seeded runs and because it provides an established training/test split.
Because Atari-5 is a new benchmark, we thought it important to validate our algorithm’s performance on the full 57-game suite.
We, therefore, provide results for both PPO (2x) and DNA on all 57 games in the ALE using both the ‘easy’ settings similar to [16] and the more difficult settings used in our main study.
We measured the median score as the median human-normalized score over the past 100-episodes and report the final median scores as the average for this measure over the last 10M frames (5% of training frames).
Individual game scores are also reported as the average over the final 10M frames.
We found, under both the hard and the easy settings, DNA outperformed PPO by a wide margin (see Table 8).
DNA also outperformed Rainbow DQN on the easy settings after just 85.5M training frames (Figure 16).
Training plots are provided in Figures 17, 18.
Results for each game are given in
Tables 9, 10.
Most surprising is that when trained with only a single policy epoch, a larger batch size, and more parallel agents, PPO becomes comparable to Rainbow DQN on the easy settings despite being a much simpler algorithm and being 80-times faster to train.
[Table]
[Figure]
c o r e
N Pseudocode for Generating Noise Scale Estimates.
In their paper, McCandlish et al.
[23] define the noise scale as,
[Equation]
(19) where at parameter values θ , G is the true gradient, H is the true Hessian, and Σ is the per-example covariance matrix.
For large models, the calculation of the Hessian is not practical.
We, therefore, use their simplified measure
[Equation]
Even though this formulation makes the unrealistic assumption that the Hessian is a multiple of the identity matrix, empirical studies by McCandlish et al. have shown it to provide a surprisingly good approximation for B noise .
Our method for generating estimates of B simple follows closely that of [23] and is formalized in
Algorithm 2.
While the estimates for tr (Σ) , and | ⃗G | 2 are both unbiased their ratio may not be.
We mitigate this by averaging over multiple samples when calculating ⃗G B small and applying smoothing to our estimate for |G| 2 .
Algorithm 2 Estimate
1: function E STIMATE N OISE S CALE (
D , a batch of data.
L , a loss function.
θ , the model parameters.
N samples , the number of small mini-batches to use.
B big , B small , the big and small mini-batch sizes.
|G| 2 old , the previous smoothed |G| 2 value.
α , smoothing factor to use for |G| 2 ) 2:
Define SAMPLE(x,y), to draw y samples from x without replacement.
D big ← SAMPLE
( D, B big ) 4: | ⃗G B big | 2 ← |∇ θ L D big ( θ ) |5: | ⃗G B small | 2 ←6: for i = 1 ..
N samples do 7:
D small ← SAMPLE ( D, B small ) 8: | ⃗G B small | 2 ← | ⃗G B small | 2 + (1 /N samples ) |∇ θ L D small ( θ ) |9: |G| 2 new ← ( B big | ⃗G B big | 2 − B small | ⃗G B small | 2 ) /
( B big − B small ) 10:
S ← ( | ⃗G B small | 2 − |
⃗G
B big | 2 ) / (1 /B small − 1 /B big ) 11: |G| 2 ← α |G| 2 old + (1 − α ) |G| 2 new ▷ Perform smoothing over |G| 2 , to reduce variance 12:
B simple ← S / |G|13: return B simple
▷ the (squared) noise scale estimate.
14: return |G|▷ pass to next call of EstimateNoiseScale
[Figure] [Figure]
1e
Alien
[Figure]
6 1e
Amidar
[Figure]
1e
Assault
[Figure]
[Figure] [Figure]
1e
Asterix
[Figure] [Figure]
1e
Asteroids
[Figure]
Atlantis
[Figure] [Figure]
[Table]
Riverraid
[Table]
Robotank
[Table]
Seaquest
[Table] [Table]
655 Skiing -17098.
[Table] [Table]
29974.5 Solaris
[Table] [Table]
[Table] [Table]
976 SpaceInvaders
[Table]
[Table] [Table]
StarGunner
[Table] [Table]
[Table]
Surround -10.
[Table]
Tennis -23.
[Table]
Tutankham
[Table] [Table]
[Table]
2, 1, 2,059 FishingDerby -91.
Freeway 0.
33.0 Frostbite 65.
4, 9, 4, 320 Gopher 2, 70, 67, 80,
Gravitar 3, 1, 2, 2,190 Hero 1, 30, 55, 11, 24,904 IceHockey -11.
7.2 Jamesbond 29.
19, 14, 14,102 Kangaroo 52.
3, 14, 11, 14,373 Krull 1, 2, 8, 11, 10,956 KungFuMaster 22, 52, 67, 110,
MontezumaRevenge 0.
0.0 MsPacman 6, 5, 8, 5,894 NameThisGame 2, 8, 13, 20, 20,
Phoenix 7, 108, 220, 391,
Pitfall -229.
0.0 Pong -20.
19.7 PrivateEye 24.
100 Qbert 13, 33, 8, 52,
Riverraid 1, 17, 22, 17, 16,789 RoadRunner 11.
7, 62, 17, 61,713 Robotank 2.
Seaquest 68.
42, 15,  4,146 Skiing -17098.
Solaris 1, 12, 3, 2, 2,225 SpaceInvaders 1, 18, 2, 2,731 StarGunner 10, 127, 70, 104,125 Surround -10.
5.3 Tennis -23.
-10.9 TimePilot 3, 5, 12, 8, 12,774 Tutankham 11.
127 UpNDown 11, 103, 247, 291,
Venture 0.
0.0 VideoPinball 0.
17, 533, 359, 505,392 WizardOfWor 4, 17, 11, 20,
YarsRevenge 3, 54, 102, 166, 564,
Zaxxon 32.