DNA: Proximal Policy Optimization with a Dual Network Architecture
Abstract
This paper explores the problem of simultaneously learning a value function andpolicy in deep actor-critic reinforcement learning models.
We ﬁ nd that the commonpractice of learning these functions jointly is sub-optimal, due to an order-ofmagnitude difference in noise levels between these two tasks.
Instead, we showthat learning these tasks independently, but with a constrained distillation phase,signi ﬁ cantly improves performance.
Furthermore, we ﬁ nd that the policy gradientnoise levels can be decreased by using a lower variance return estimate.
Whereas,the value learning noise level decreases with a lower bias estimate.
Together theseinsights inform an extension to Proximal Policy Optimization we call Dual Network
Architecture (DNA), which signi ﬁ cantly outperforms its predecessor.
DNA alsoexceeds the performance of the popular Rainbow DQN algorithm on four of theﬁ ve environments tested, even under more dif ﬁ cult stochastic control settings.
Introduction
Combining deep neural networks with reinforcement learning has produced impressive results onchallenging problems, such as playing Chess [31], Atari games [24], and robotics tasks [30].
However,results have bifurcated between two competing approaches: Q-learning-based approaches that requirelearning an (action conditioned) value estimate and actor-critic policy gradient (AC-PG, or just
PG) 1 approaches that learn both a policy and value estimate.
PG offers many theoretical advantagesover Q-learning, such as natural support for continuous control problems and the ability to learnstochastic policies.
However, until recently, PG approaches [23, 30, 10], while strong on continuouscontrol problems, have under-performed Q-learning approaches on complex vision-based problems[24, 15, 3], restricting the use of PG in this domain.
This paper aims to close the gap between PG and Q-learning methods by showing that the commonpractice of jointly learning value and policy with a shared representation negatively affects thealgorithm’s performance.
2 We also demonstrate an order-of-magnitude difference in noise levelsbetween these two tasks and argue that this makes these two tasks poorly aligned.
In light of this result, we introduce a new algorithm, based on Proximal Policy Optimization (PPO)[30], called Dual Network Architecture (DNA).
We test DNA empirically on a subset of the Arcade
Learning Environment (ALE) [5] called Atari-5. 3 , Our results show a strong increase in performancecompared to both PPO and another dual network model Phasic Policy Gradient (PPG) [10].
Our model also outperforms Rainbow DQN [15] on four of the ﬁ ve games tested, even under morechallenging environmental settings.
We summarize our contributions.
First, we provide empirical results showing an order-of-magnitudedifference in the noise scale between policy gradient and value learning.
Second, we give evidencefor the bene ﬁ t of using low bias value estimates for value learning, but low variance estimates foradvantage estimates.
Finally, we introduce and justify our dual network constrained distillationalgorithm DNA with empirical results on ALE.
Preliminaries and Related Work
Proximal Policy
Our algorithm builds on the well-established Proximal Policy Optimization (PPO) algorithm [30]. PPO is a policy gradient, reinforcement learning algorithm,with many of the advantages of its predecessor Trust Region Policy Optimization [28], yet withoutthe complexity.
Various aspects of PPO have been studied in recent years, such as the impact ofimplementation choices [11, 18], the ability to generalize [10, 26], and performance under multi-agentsettings [36].
There has also been a growing understanding that the value and policy learning tasksinvolved in actor-critic models like PPO have important asymmetries [26, 10]. Our work buildson this by investigating how differences in value and policy noise levels can be accommodated toimprove PPO’s performance on discrete action tasks.
TD( λ ) Return Estimation Policy gradient algorithms often make use of a value estimate as abaseline [23], as well as for estimating the value of truncated trajectories [8].
To better facilitatecontrol over the noise levels for policy and value learning, our work makes use of two different returnestimations, both using TD( λ ) [32].
Given value estimates V ( · ) from the value network we de ﬁ nethe n-step value estimate for some state s t taken at time t , and their exponentially weighted sum as
[Equation]
For values of λ close to 1 , more weight is assigned to longer n-step return estimates, and less toshorter ones.
There has been a long-standing belief that shorter n-step returns generate more biasedestimates, whereas longer n-step estimates, due to summing over many stochastic rewards, havehigher variance.
For a more thorough discussion on this topic see [19].
Noise Scale.
Noise scale is a measure of ‘signal to noise’ in stochastic gradient descent (SGD).
While SGD provides unbiased gradient estimates, these estimates are typically very noisy.
Eachsampled gradient estimate ˆ G can be thought of as a sum of the true gradient ⃗ G and some noise vector⃗σ .
A method for estimating the ratio of the magnitude of this implied noise vector to the magnitudeof the true gradient was proposed by [22] who show that their ef ﬁ cient-to-calculate simple noise scaleis a good match for the noise scale.
They also show that the noise scale provides useful informationabout the choice of mini-batch size to use when estimating gradients.
Phasic Policy Gradient.
Most similar to our work is Phasic Policy Gradient (PPG) [10].
Like
DNA, PPG has three distinct phases during training and uses two independent networks.
However,there are several important distinctions in our work.
First, we forgo the large replay buffer, relyinginstead on learning entirely from recent experience, signi ﬁ cantly reducing the memory requirementsof our algorithm.
Second, we make use of a distillation phase rather than an auxiliary phase.
Finally, our work reduces gradient noise by using two different return estimators calibrated for the propertiesof each task.
We assess the impact of these differences on the performance of the agent in Section 6.
The Noise Properties of Value Learning and Policy Gradient
Here we examine the noise properties of the loss functions for policy gradient and value learning andshow that policy gradient has a much higher noise level than value learning.
This implies differentchallenges in the optimization problem.
Speci ﬁ cally, that policy should be learned with a largermini-batch size than value learning and that return estimates should be adapted appropriately.
3.1 Motivating Example
To motivate our investigation into noise levels, we consider the task of learning two independentfunctions F 1 ( x ) := sin(5 x ) + N (0 , σ 21 ) and F 2 ( x ) :
= cos(5 x ) + N (0 , σ 22 ) , where N ( µ, σ 2 ) is
Gaussian noise with mean µ and standard deviation σ , over the domain [ − π , π ] .
We ﬁ x σ 2 = 1, andvary σ 1 on a log scale from 0 .
1 to 100 .
We trained two 3-layer multi-layer perceptions (MLPs) onthis problem.
The ﬁ rst was a joint model, using a ﬁ rst hidden layer of 1024 units and a secondhidden layer of 2048 units, followed by two output heads.
Our second model was a dual networkconsisting of two independent MLPs with 1024 units on each hidden layer.
7 All models used ReLUactivations in between linear layers, and error was measured as mean-squared-error (MSE) betweenthe predicted value and the noise free true value.
The results are presented in Figure 1.
At low noiselevels, the tasks did not interfere, but as the noise of the ﬁ rst task increased, performance on thesecond task eventually degraded for the joint network, but not the dual network.
While simple, thisexperiment demonstrates the impact one noisy task can have on another when learned jointly by adeep neural network.
�� � �� � �� � �� � � ��� ��� ��� ��� ��� ��� � � � ������ �� ������ � �� �� � �� � �� � �� � � ����� ����� ����� ����� ����� ����� ����� ����� ����� � � � ������ �� ������� � �
3.2 Noise Scale in Reinforcement Learning
To assess the differences in noise levels in our experiments, we measure the gradient noise scale ofthe policy gradient and value loss gradient.
We did this by learning an estimate of the noise scale,developed by [22] called simple noise scale de ﬁ ned as
[Equation]
where Σ is the gradient covariant matrix, and ⃗ G is the true gradient if the entire batch was used.
Forconvenience we also use the notation σ := p
B simple , which can interpreted as the ratio of the length 95 of the implied noise vector to the length of the true gradient vector.
That is, a kind of noise-to-signalratio.
As [22] have shown, an unbiased estimate of B simple can be found ef ﬁ ciently by generatinggradient estimates ⃗ G B small using a small mini-batch of size B small as well as the gradient ⃗ G B big using alarge mini-batch of size B big , then calculating
� �� �� �� ��� ��� ��� ��� ��� ��������� ��� ��� ��� ��� ��� ��� ��� � � � �� � � � � �������������������������������������������������������������������������������������������������������
[Equation] [Equation]
(5) where E [ |G| 2 ] =
| ⃗ G | 2 and E [ S ] = tr ( Σ ) .
To evaluate the noise scale of value learning and policy gradient we trained our dual network model(described fully in section 4) on the three Atari games from the Atari-3 validation set.
8 Like [22], wefound it necessary to smooth out noise by maintaining an exponential moving average of | ⃗ G B big | 2 .
We used B small = 16 and B big = 16 , 384 in all our experiments.
The results are presented in Figure 2.
We found that policy noise ( σ π ) reduced by about half duringtraining and that value noise ( σ v ) remained constant.
Both noise levels varied very little betweenthe three environments.
However, a large variation was observed between the value loss and policygradient, consistent between environments.
The noise levels, measured at the end of training andaveraged over all three environments, were found to be σ π = 220 . 5 and σ V = 17 . 5 , representing a12 .
6 times difference.
Dual Network Architecture
Based on our experimental results in Section 3 we propose an architecture which takes into accountthe large difference in noise levels between the two tasks.
This architecture consists of threeimprovements to PPO.
First, to reduce negative interference from the noisy policy gradient, policyand value should be learned by independent networks with different hyperparameters (e.g. trainingepochs and mini-batch size).
Second, the variance/bias trade-off in the return estimations should becalibrated to the properties of each task.
Finally, a constrained distillation phase should account forany constructive interference between the tasks.
4.1 Independent Networks
Important to DNA is the use of a dual network architecture.
9 Not only does this setup allow forlearning of the value and policy without destructive interference, it also enables a specialized set ofhyperparameters to be calibrated for the distinct tasks of policy and value learning.
Speci ﬁ cally, thecalibration of the mini-batch size, which it has been suggested is best set roughly proportional to σ[22].
The policy network outputs a policy π and a value estimate V π , whereas the value network onlyoutputs a value V V , as depicted in Figure 3.
4.2 Decoupled Return Estimation
To better facilitate control over the noise levels for policy and value learning, DNA makes use of twodifferent return estimations, both using TD( λ ) [32] as follows [Equation]
[Equation]
) where λ V and λ π are hyperparameters controlling the variance / bias trade-off of each estimate.
The
V targ estimates are used as targets for training the value function, whereas V adv are used for advantageestimates used for policy gradient estimates given by
[Equation]
[Equation] This formulation of the advantages is equivalent to the general advantage estimation (GAE) [29], thatis, for any λ ∈ [0 .. 1] , γ ∈ [0 .. 1) we have,
[Equation]
Because of this, it is common when using PPO to generate value targets, V targ , for the value function byadding the network’s value estimates to the (unnormalized) advantages, implicitly settings λ V = λ π .
We hypothesize that due to bootstrapping, estimates used for V targ bene ﬁ t from being low bias ,conversely, because estimates used for advantage estimation contribute to the noise of the policygradient, they would bene ﬁ t from being low variance .
That is, the bootstrapping process may causesymmetrically distributed errors, but not biased errors, to cancel out.
This suggests λ π < λ V , athought which we test empirically in section 6.2.
4.3 Distillation
Because value learning is a simpler task in terms of noise level, it makes sense to try to transfer someof the knowledge learned by the value network to the policy network.
If the value network learns toidentify important features in the environment, we would like the policy network to identify thosefeatures also.
To facilitate this, we employ a constrained distillation update [16].
Distillation betweentwo identical networks has the property that the student’s target function since an identical networkgenerated it, is guaranteed to be in the model’s solution space.
Also, because the distillation targetsare deterministic, 12 the process is also very low noise, which we verify in Appendix D.
We contrast this with the auxiliary update of PPG which trains both the value function and the policyon an auxiliary task.
This auxiliary task equates to learning 1 N
P
Ni =
V π − i targ where V π − i targ is a valueestimate for the policy from i updates prior.
For quickly changing policies, and large N , these updatesmay cause tension with the value estimates of the current policy.
R is a (noisy) return
V ( s t ) is a deterministic function produces an of E R ( s t )
A better way to do auxiliary tasks would be to train a separate value network output head, dedicated to the auxiliary task.
This way the agent could learn the value of the current policy, along with the moving average
We use V V ( s ) as the distillation targets, with the input states s being taken from the current rollout.
However, unlike policy and value training, distillation state inputs need not be generated on-policy.
We explored other distillation targets, covered in Appendix J, but found V V ( s ) to be the best of thosetried.
Distillation is performed, like PPG’s auxiliary task, using mean squared error, and under a softconstraint on the policy network’s policy, speci ﬁ cally
[Equation]
where β is the policy constraint coef ﬁ cient, and π old is a copy of the policy before the distillationupdate.
During updates gradients are only propagated through the policy network, and not the valuenetwork.
4.4 Training
Like Phasic Policy Gradient (PPG) [10], DNA splits training into three distinct phases, but unlike
PPG, rather than using a large replay buffer, we perform all updates on-policy on the current batch ofrollout data.
Each of the three phases optimizes a single objective, for some number of epochs, usingits own optimizer, with a unique set of hyperparameters.
The optimization objective for the policynetwork of DNA is the clipped surrogate object from PPO [30] including the entropy bonus [Equation] t :
[Equation]
where S is the entropy in nats, ρ t is the ratio π ( a t | s t ) π old ( a t
| s t ) at time t , ϵ is the clipping coef ﬁ cient, and c ebis the entropy bonus coef ﬁ cient.
For value loss we use the squared-error loss,
L
[Equation]
 [Equation] In summary, the DNA algorithm separates the tasks of policy learning and value learning into anetwork designed to handle the high noise of policy learning and a separate network designedto handle the lower noise task of value learning.
Knowledge from the value learning network istransferred to the policy network through a separate constrained distillation phase, which allows forconstructive interference between the learning tasks while minimizing the destructive.
We formalizethe algorithm as follows.
1: procedure PPO-DNA 2: 3: for t = 1 to N do 4: for i = 1 to A do 5:
Run policy π t in environment i for T timesteps 6:
Compute V targ ← TD ( γ , λ V ) 7:
Compute ˆ A ← TD ( γ , λ π ) − V adv 8: for i = 1 to E π do 9:
Optimize L CLIP wrt θ π 10: for i = 1 to E V do 11:
Optimize L V F wrt θ V 12: π old ← π t 13: for i = 1 to E D do 14:
Optimize L D wrt θ π
Evaluation
To evaluate our algorithm’s performance, we used the Atari-5 benchmark.
Scores in Atari-5 are generated using a weighted geometric average over ﬁ ve speci ﬁ c games and produce results that correlate well with the median score if all 57-games had been evaluated.
This allowed us to performmultiple seeded runs and de ﬁ ned a clear training and test split between the games.
In all cases, we ﬁ t hyperparameters to the 3-game validation set and only used the 5-game test set forﬁ nal evaluations.
To verify the statistical signi ﬁ cance of our results, we performed multiple seededruns.
We opted for the more dif ﬁ cult stochastic ALE settings recommended as best practice by [21].
However, to better understand our results in the context of prior work, we also provide detailed resultsunder the simpler deterministic settings in Appendix F, and additionally provide for reference a singleseed evaluation on the full 57-game set in Appendix M.
Unless otherwise speci ﬁ ed, agents werescored according to their average performance over the previously completed 100-episodes at the endof training.
A coarse hyperparameter sweep found initial hyperparameters for our model on the Atari-3 validationset.
Notably, we found the optimal mini-batch size for value and distillation to be the minimumwe tested (256), while the optimal mini-batch size for policy was the largest tested (2048).
Foroptimization, we used Adam [20], over the standard 200 million frames.
Full hyperparameter detailsfor our experiments are given in Appendix B.
In order to understand the impact of the hyperparameters introduced by our algorithm, namely
E π , E V , E D and λ π , λ V , we performed the following experiments on the Atari-3 validation set.
Westarted by setting E π = E V = E D
= 2 , then searched over E V ∈ [1 , 2 , 3 , 4] , and selected the best
E V .
We then searched over E π and E D while keeping the other two parameters constant.
We alsoperformed a similar experiment on the impact of λ V and λ π by setting λ V to 0.95 then sweepingacross λ π ∈ [0 . 6 , 0 . 8 , 0 . 9 , 0 .
95 , 0 . 975] , then setting λ V to the best λ π and repeating.
This processallowed us to verify if the best settings occur at λ V = λ π or at λ V ̸ = λ π .
During our λ tuning, wealso recorded noise levels for one of the seeds, the results of which we discuss in Section 6.2.
We evaluated our algorithm DNA, against PPO, and PPG.
All models used the ‘NatureCNN’ encoderfrom [24].
However, because DNA and PPG use two networks and thus twice the parameters, wedoubled the number of channels of the PPO encoder which we refer to as PPO (2x) and which is verysimilar to the encoder used by [3].
All models have have approximately 3.5 million parameters intotal.
For fairness we repeated the same sweep on PPO for the number of training epochs ( E PPO ) andλ used for GAE ( λ PPO ).
To verify that our settings did not inadvertently degrade the performance of
PPO we also tested against the settings used by [30], and refer to this as PPO (original).
These resultsare included in the ablation study in Appendix C.
Main Experimental Results
In this section we present the results of our main study comparing DNA to PPO and PPG on Atari-5.
We also include supplementary experiments on MuJoCo [33] and Procgen [9] in the Appendix.
6.1 Impact of Training Epochs
We start by investigating the impact of the E π , E V , and E D hyperparameters.
We found that usingless than three policy epochs for DNA dramatically increased the performance of the agent whilealso decreasing the computation required to train the agent (Figure 4).
We also found that DNAwas robust to the choice of value and distillation epochs but that overtraining on value ( E V = 4 )marginally decreased performance.
This was not the case for distillation updates.
The distillationphase demonstrated some bene ﬁ t as indicated by the E D = 0 results under-perform the others.
Weselected E π = 2 , E V = 1 , E D = 2 as the optimal settings and used these for the remainder of theexperiments.
These settings require a total of four updates for the policy network, but only one updatefor the value network.
The training curves also suggest that DNA may bene ﬁ t from training beyondthe standard 200-million frames.
6.2 Return Estimation
We found that the choice of λ π made a large difference to the performance of our agent.
Aswas hypothesised the optimal settings for λ π differed to those of λ V with λ π preferring a lowervariance, higher bias value of 0 . 8 , while the value targets, λ V preferred a lower bias, higher variance
� �� �� �� ��� ��� ��� ��� ��� ��������� � �� ��� ��� ��� ��� ��� � � � � � � � � � � � � � � � � � � � � � �� � � �� � � �� � � �� � �� �� �� ��� ��� ��� ��� ��� ��������� � �� ��� ��� ��� ��� ��� � � � � � � � � � � � � � � � � � � � � �� � �� � �� � �� � �� �� �� ��� ��� ��� ��� ��� ��������� � �� ��� ��� ��� ��� ��� � � � � � � � � � � � � � � � � � � � � � �� � � �� � � �� � � ��
��� ��� ��� ���� ����� ��� ��� ��� ��� ��� ��� ��� � � � � � � � � � � � � � � � � � � � � ���� � ����� ��� ��� ��� ���� ����� � ���� ���� ���� ���� ���� ���� ���� ���� � � � � � � � � � ����� ��� ��� ��� ���� ����� ���� ���� ���� ���� ���� ���� ���� ���� � � � � � � � � � � � ����� setting of 0 .
Our results are presented in Figure 5.
Notable is that the non-homogeneoussetting λ V
=
95 , λ π = 0 .
8 outperformed both homogeneous choices
λ π = λ V = 0 .
8 andλ π = λ V = 0 .
We found, as expected, that noise scale can be reduced by selecting a low valuefor λ π .
However, against our expectations, setting λ V lower actually increased, not decreased thenoise level of the value learning task.
Once we discovered this we ran addition experiments for abroader choice of λ V , and include the results in Figure 5.
6.3 Comparision to PPO and PPG
We found DNA to outperform both PPO and PPG on the Atari-5 dataset by a wide margin (Figure 6).
We give Atari-5 summary scores, as well as training plots for each individual game.
We also includereferences for Rainbow DQN, although note that these scores were generated under the simplerdeterministic settings.
Discussion
Reduced Training as an Alternative to Experience Replay We found that less training, bothon policy and value updates, typically improves the agent’s performance.
One explanation for thiscould be, as [10] have suggested, that in the value case, this might be due to over ﬁ tting to recentexperience.
In terms of policy updates, where the largest performance degradation was observed,this could be due to increased training epochs causing large changes in policy, which may suggest abetter constraint than PPO’s clipping is required.
15 That being said, limiting the training epochs toone or two provides straightforward solution to the problem while having the additional bene ﬁ t ofdecreasing the computational resources required to train an agent.
Even in Deep Q-learning, othershave found that, beyond some point, increasing the number of times an example is processed has anegative effect unless the learning rate is reduced [1].
� �� �� �� ��� ��� ��� ��� ��� �������� � �� ��� ��� ��� ��� � � � � � � � � � � � � � � � ��� ��� �������� ������� � ��� ��� ��� ��� ��� ��� ���������� � ��� ��� � � ��� ���������� � ��� ��� � � � ��� ������������ � ��� ��� ��� ��� ��� ��� ������� � ��� ��� � � � ��� �����
Choice of λ
Our results have shown strong evidence for the bene ﬁ t of selecting λ π < λ V .
Asexpected, picking lower values of λ π reduces the policy gradient noise.
However, counterintuitively,noise on value learning reduces with higher choices of λ V .
That is, policy learning prefers lowvariance , but value learning prefers low bias .
This result may explain why clipping, which introducesbias, performs poorly on value learning but quite well on policy learning.
This result also suggeststhat an even higher choice of λ V than the 0 .
95 we used in our experiments may be appropriate.
Limitations Our paper’s primary focus was improving PG results on discrete action vision-basedproblems, as this is where PG has traditionally unperformed Deep Q-Learning.
However, do we alsoprovide an initial look into continuous control problems on the MuJoCo [33] dataset in Appendix K.
Broader Impact In our work, we have shown that deep actor-critic reinforcement learning modelscan effectively solve complex vision-based, discrete-action environments, even without the use oflarge replay buffers.
Surprisingly this can be achieved using less not more policy and value networkupdates.
Because Q-learning approaches generally produce deterministic policies, algorithms basedon them may ‘lock in’ to a decision based on minimal differences in outcome.
This is not always thecase with stochastic policies, which are able to randomly split over actions of roughly equal quality.
This may lead to fairer outcomes.
We do not foresee any direct negative societal impacts from thisresearch.
Future work Our algorithm is already highly competitive with Rainbow DQN, and could likelybe extended with orthogonal improvements such as adding recurrence [13], distributional valuelearning [4], residual encoders [12], better exploration bonuses [6], and making use of a learnedmodel [27].
Our work also raises an interesting question about the update-to-data ratio (UTD).
Whereour experiments ﬁ nd a low (1-2) UTD to be critical, in contrast to others who have found high values(20-40) to be preferred [7].
Conclusions
In this paper, we have highlighted noise level as a key difference between policy and value learning.
We have introduced an algorithm that accounts for this order-of-magnitude difference by limitingnegative interference through two independent networks for value and policy but retaining constructive interference through a constrained distillation process.
Furthermore, we have shown thatthe variance/bias tradeoff differs for value learning and policy gradient and that return estimationshould cater for this.
Together these changes result in a novel algorithm, DNA, that outperforms itspredecessor PPO, and even surpasses the popular Q-learning approach Rainbow DQN, while undermore challenging environmental settings.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justi ﬁ cation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately re ﬂ ect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 7.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 7.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix H.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identi ﬁ ableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]