DASCO: Dual-Generator Adversarial Support Constrained Offline Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
In offline RL, constraining the learned policy to remain close to the data is essentialto prevent the policy from outputting out-of-distribution (OOD) actions with erroneously overestimated values.
In principle, generative adversarial networks (GAN)can provide an elegant solution to do so, with the discriminator directly providinga probability that quantifies distributional shift.
However, in practice, GAN-basedoffline RL methods have not outperformed alternative approaches, perhaps becausethe generator is trained to both fool the discriminator and maximize return – twoobjectives that are often at odds with each other.
In this paper, we show that theissue of conflicting objectives can be resolved by training two generators: one thatmaximizes return, with the other capturing the “remainder” of the data distributionin the offline dataset, such that the mixture of the two is close to the behavior policy.
We show that not only does having two generators enable an effective GAN-basedoffline RL method, but also approximates a support constraint, where the policydoes not need to match the entire data distribution, but only the slice of the datathat leads to high long term performance.
We name our method DASCO, for
D ual-Generator
A dversarial S upport C onstrained O ffline RL.
On benchmark tasksthat require learning from sub-optimal data, DASCO significantly outperformsprior methods that enforce distribution constraint.
Introduction
Offline reinforcement learning (RL) algorithms aim to extract policies from datasets of previouslylogged experience.
The promise of offline RL is to enable the extraction of decision making enginesfrom existing data [24].
Such promise is especially appealing in domains where data collection isexpensive or dangerous, but large amounts of data may already exists (e g., robotics, autonomousdriving, task-oriented dialog systems).
Real-world datasets often consist of both expert and suboptimal behaviors for the task of interest and also include potentially unrelated behavior correspondingto other tasks.
While not all behaviors in the dataset are relevant for solving the task of interest, evensub-optimal trajectories can provide an RL algorithm with some useful information.
In principle, ifoffline RL algorithms can combine segments of useful behavior spread across multiple sub-optimaltrajectories together, they can then perform better than any behavior observed in the dataset.
Effective offline RL requires estimating the value of actions other than those that were taken in thedataset, so as to pick actions that are better than the actions selected by the behavior policy.
However,this requirement introduces a fundamental tension: the offline RL method must generalize to newactions, but it should not attempt to use actions in the Bellman backup for which the value simplycannot be estimated using the provided data.
These are often referred to in the literature as out-ofdistribution (OOD) actions [20].
While a wide variety of methods have been proposed to constrainoffline RL to avoid OOD actions [18, 8, 1], the formulation and enforcement of such constraints canbe challenging, and might introduce considerable complexity, such as the need to explicitly estimate the behavior policy [38] or evaluate high-dimensional integrals [22].
Generative adversarial networks(GANs) in principle offer an appealing and simple solution: use the discriminator as an estimator forwhether an action is in-distribution, and train the policy as the “generator” in the GAN to fool thisdiscriminator.
Although some prior works have proposed variants on this approach [38], it has beenproven difficult in practice as GANs can already suffer from instability when the discriminator is toopowerful.
Forcing the generator (i.e., the policy) to simultaneously both maximize reward and foolthe discriminator only exacerbates the issue of an overpowered discriminator.
We propose a novel solution that enables the effective use of GANs in offline RL, in the process notonly mitigating the above challenge but also providing a more appealing form of support constraintthat leads to improved performance.
Our key observation is that the generative distribution in
GANs can be split into two separate distributions, one that represents the “good parts” of the datadistribution and becomes the final learned policy, and an auxiliary generator that becomes the policy’scomplement, such that the mixture of the two is equal to the data distribution.
This formulationremoves the tension between maximizing rewards and matching the data distribution perfectly: aslong as the learned policy is within the support of the data distribution, the complement will pick upthe slack and model the “remainder" of the data distribution, allowing the two generators together toperfectly fool the discriminator.
If however the policy ventures outside of the support of the data,the second generator cannot compensate for this mistake, and the discriminator will push the policyback inside the support.
We name our method DASCO, for D
ual-Generator
A dversarial S upport C onstrained O ffline RL.
Experimentally, we demonstrate the benefits of our approach, DASCO, on standard benchmarktasks.
For offline datasets that require a combination of expert and sub-optimal data to obtain goodperformance, our method outperforms distribution-constrained offline RL methods by a large margin.
Related Work
Combining behaviors from sub-optimal trajectories to obtain high-performing policies is a centralpromise of offline RL.
During offline training, querying the value function on unseen actions oftenleads to value over-estimation and unrecoverable collapse in learning progress.
To avoid queryingthe value functions on out-of-distribution actions, existing methods encourage the learned policiesto match the distribution of the dataset generation policies.
This principle has been realized with avariety of practical algorithms [15, 38, 31, 32, 38, 21, 18, 17, 36, 7, 4, 9, 14, 28, 5, 26].
For example,by optimizing the policies with respective to a conservative lower bound of the value function estimate[22], only optimizing the policies on actions contained in the dataset [18], or jointly optimizingthe policy on the long-term return and a behavior cloning objective [7].
While explicitly enforcingdistribution constraint by adding the behavior cloning objective allows for good performance onnear-optimal data, this approach fails to produce good trajectories on sub-optimal datasets [18].
Methods that implicitly enforce distribution constraints, such as CQL and IQL, have seen moresuccesses on such datasets.
However, they still struggle to produce near-optimal trajectories when theactions of the dataset generation policies are corrupted with noise or systematic biases (a result wedemonstrate in Section 5).
However, enforcing distribution constraints to avoid value over-estimation may not be necessary.
It is sufficient to ensure the learned policies do not produce actions that are too unlikely underthe dataset generation policy, but it is not necessary for them to fully cover the data distribution,only to remain in-support [21, 19, 24, 38, 39, 3].
Unfortunately, previous methods that attempt toinstantiate this principle into algorithms have not seen as much empirical success as algorithmsthat penalize the policies for not matching the action distribution of the behavior policies.
In thispaper, we propose a new GAN-based offline RL algorithm whose use of dual generators naturallyinduce support constraint and has competitive performance with recent offline RL methods.
In anumber of prior works, GANs have been used in the context of imitation learning to learn from expertdata [13, 25, 12, 27].
In this work, we show that dual-generator GANs can be used to learn fromsub-optimal data in the context of offline RL.
Background
Let M =
( S , A , P, R, γ ) define a Markov decision process (MDP), where S and A are state andaction spaces, P : S ×
A × S → R + is a state-transition probability function, R : S ×
A → R is a reward function and γ is a discount factor.
Reinforcement learning methods aim at finding a policyπ ( a | s ) that maximizes the expected discounted reward R ( τ )
=
(cid:80) Tt =
γ t R ( s t , a t ) over trajectoriesτ =
( s 0 , a 0 , . . .
, s T , a T ) with time horizon T induced by the policy π .
In this work, we concentrate on the offline or off-policy RL setting, i.e. finding an optimal policygiven a dataset D of already previously collected experience τ ∼ D by a behavior policy π β .
Aparticularly popular family of methods for offline learning are based on training a Q-function throughdynamic programming using temporal-difference (TD) learning [37, 34].
Such methods train a Q-function to satisfy the Bellman equation:
[Equation]
In Q-learning, the policy is replaced with a maximization, such that π ( a | s ) =
arg max a Q θ ( s, a ) ,while actor-critic methods optimize a separate parametric policy π ϕ ( a | s ) that maximizes the Qfunction.
In this work, we extend the Soft Actor-Critic (SAC) method [11] for learning from diverseoffline datasets.
Generative Adversarial Networks (GANs) [10] enable modeling a data distribution p D through anadversarial game between a generator G and a discriminator D :
[Equation]
For this two player zero-sum game, [10] shows that for a fixed generator G , the optimal discriminatoris D ∗ G ( x ) =
p D ( x ) p D ( x ) + p G ( x ) and the optimal generator matches the data distribution p ∗ g ( x ) = p D .
GAN has been extended to the offline RL setting by interpreting the discriminator function as ameasure of how likely an action is under the behavior policy, and jointly optimizing the policy tomaximize an estimate of the long-term return and the discriminator function [38]:
[Equation]
where Q ( s, a ) is trained via the Bellman operator to approximate the value function of the policyπ ( a | s ) .
This leads to iterative policy evaluation and policy improvement rules for the actor and thepolicy [38]:
[Equation]
where the log D ( a | s ) term in the policy objective aims at regularizing the learnt policy to preventit from outputting OOD actions.
In practice, maximizing both Q and discriminator might lead toconflicting objectives for the policy and thus poor performance on either objective.
This can happenwhen the data contains a mixture of good and bad actions.
Maximizing the value function wouldmean avoiding low-reward behaviors, while on the other side maximizing discriminator would requiretaking all in-distribution actions, including sub-optimal ones.
Our approach alleviates this conflict andenables in support maximization of the value function when learning from mixed-quality datasets.
Dual-Generator Adversarial Support Constraint Offline RL
We now present our algorithm, which uses a novel dual-generator GAN in combination with aweighting method to enable GAN-based offline RL that constrains the learned policy to remainwithin the support of the data distribution.
We call our method Dual-generator Adversarial Support
Constraint Offline RL (DASCO) .
We will first introduce the dual-generator training method generically,for arbitrary generators that must optimize a user-specified function f ( x ) within the support of thedata distribution in Section 4.1.
We will then show this method can be incorporated into a completeoffline RL algorithm in Section 4.2 in combination with our proposed weighting scheme, and thensummarize the full resulting actor-critic method in Section 4.3.
4.1 Dual generator in-support optimization
In this section, we will develop an approach for performing a joint optimization of adversarial andsecondary objectives of the generator in a GAN framework, which we will then apply to offline RL.
This is a necessary component for performing the joint optimization in Eq.
2 without introducing aconflict of these objectives.
All proofs for theorems presented in this section are in Appendix A.
Let’s consider a general objective that requires training a generator G to fool the discriminator Dwhile also optimizing the expected value of some other function f :
[Equation]
Here, we have added an additional term E z ∼ p ( z ) [ f ( G ( z ))] , where f is a mapping from the generatoroutput to a scalar value.
Theorem 4.1 The optimal generator of Eq.
4 induces a distribution p ∗ g ( x ) = p D ( x ) e
[Equation]
where τ > 0 is the Lagrange multiplier that ensures that p ∗ g ( x ) is normalized to 1.
We can see that by adding a secondary objective function for the generator, in general, the optimalgenerator does not attempt to match the data distribution p D ( x ) anymore, but instead tries to matchthe data distribution weighted by e 2( f ( x ) −
[Equation]
2 − e 2( f ( x ) − τ ) .
We expect that in such case, the discriminatorclearly has an advantage in the two player zero-sum game and will be able to distinguish betweenreal samples and sample generated by the generator.
To allow the generator to specialize in optimizing the secondary objective function, we propose tointroduce a second auxiliary generator that matches the portion of the data distribution that is notwell captured by the primary generator.
Let p mix =
p G + p aux, consider the min-max problem:
[Equation]
where we mix samples from the primary generator G and the auxiliary generator G aux to generatesamples that can fool the discriminator.
Theorem 4.2 (Informal)
Given that f is appropriately bounded in its ∞ -norm, || f || ∞ ≤ C 0 < ∞ ,the primary generator p G performs in-support optimization of f ( x ) .
We first note that the optimal solution of the mixed distribution from Eq.
5 is the real data distribution:
[Equation]
Accordingly, the auxiliary generator distribution can be expressed as
[Equation]
We define x 0 to be the element inside the support of the data distribution p D that maximizes f , i.e.
[Equation]
p [Equation] ) f ( x ) . When optimizing the secondary objective f ( x ) , the primary generator willmaximize the probability mass of in-support samples that maximize f ( x ) .
However, Eq.
7 introducesa constraint that enforces 2 p D ( x ) − p ∗ g ( x ) ≥ 0 for p ∗
aux ( x ) ≥ 0 to remain a valid distribution.
Thisleads us to conclude that the optimal primary generator p ∗ g assigns the following probability to x 0 :
[Equation]
otherwise (8)
Interestingly, if the global maximum x 0 is not taking the full probability mass, the rest of theprobability mass is redistributed to the next best in-support maxima, which we can define recursively:
[-In the first case, p ∗ g ( x i ) = 2 p D ( x i ) if(cid:80) ij =
 p ∗ g ( x j ) < 1 . That is, if the optimal solution for the primary generator p ∗ g can assign theprobability 2 p D (-]For x i [-) to the i th in support minima of f ( x ) without the total sum of probability assigned 186 (cid:80) ij =
-]∈ arg max
[-In the second case, p ∗ g ( x i ) = 1 − (cid:80) i − 1 j =
 p ∗ g ( x j ) if (cid:80) ij =
 p ∗ g ( x j ) > 1 . That is, if by assigning theprobability 2 p D ( x i ) to the i th in support minima of f ( x ) , the total sum of probability assigned 189 (cid:80) ij =
-][Equation]
[-p ∗ g ( x j ) goes over 1 , then the primary generator p ∗ g will assign the remaining probability1 − (cid:80) i − 1 j =
-]
We note that by introducing an auxiliary generator and mixing it with the primary generator, not onlydoes the optimal solution for the mixed distribution match the real data distribution, but also theprimary generator optimizes the secondary objective f only on the part of the domain of f that iswithin the support of p D .
4.2 Update rules for offline reinforcement learning
We will now incorporate the dual-generator method to train policies for offline RL, based on optimizing the joint objective from Eq. 5.
The updates for the actor and the critic are generally similar to
Eq. 3.
However, simply combining Eq.
5 and Eq.
3 can still allow the policy to exploit errors in thevalue function during the policy improvement step.
We therefore augment the policy improvementstep with an adaptive weight on the Q-value.
More concretely, as the policy improvement stepsamples actions from the current policy iterate π k to optimize the policy objective, there is a non-zeroprobability that the sampled actions will exploit spurious maxima in the value function and havetheir probability of being sampled again in the future increased. If the same actions are sampledduring the policy evaluation step, the errors in the value functions from the next states are backedup into the preceding states, leading to divergent value functions, as we observe in our experiments.
To alleviate this issue, we use the probability assigned to the sampled actions to weight the valuefunction estimates in the policy objective, leading to the following updates:
[Equation]
11) where a D ( s ) is the action from the offline dataset.
The term D k ( s, a ) down-weights the contributionof the gradient of the value function to the policy update if the discriminator deems the sampledaction too unlikely.
We further calibrate the probability
D k ( s, a ) by dividing it with the probability
D k ( s, a D ( s )) that the discriminator assigns to the dataset action a D ( s ) .
It should be noted that duringoptimization the gradients are not propagated into these weights.
Next, we define the update rules for the auxiliary generator and the discriminator.
We mix the samplesfrom the k th iterate of the policy π k and the distribution p aux induced by the k th iterate of theauxiliary generator G kaux , that is, let p mix =
π k + p aux.
At every iteration k , we update the k thiterate of the auxiliary generator G kaux and discriminator D k using the objectives:
[Equation]
[Equation] [Equation]
4.3 Algorithm summary
Algorithm 1 provides a step-by-step description of our algorithm.
At every training step, we samplea batch of transitions from the offline dataset and proceed to update the parameters of the valuefunction, the policy, the auxiliary generator and the discriminator in that order. policy π ϕ using the augmented objective in Eq. [Table]
ψ k +1 ← Update auxiliary generator G aux,ψ using the objective in Eq.7: ω k +1 ← Update discriminator D ω using mixed samples from π ϕ and G aux,ψ as in Eq.
Experiments
Our experiments aim at answering the following questions: 1.
When learning from offline datasetsthat require combining actions from sub-optimal trajectories, does DASCO outperform existingmethods that are based on distribution constraints?
On standard benchmarks such as D4RL [6],how does DASCO compare against recent methods? 3.
Are both the dual generator and the probabilityratio weight important for the performance of DASCO?
5.1 Comparisons on standard benchmarks and new datasets
For our first set of experiments, we introduce four new datasets to simulate the challenges onemight encounter when using offline RL algorithms on real world data.
These datasets also introduceadditional learning challenges and require the algorithm to combine actions in different trajectoriesto obtain good performance.
We use the existing AntMaze environments from the D4RL suite [6]:antmaze-medium and antmaze-large.
In these two environments, the algorithm controls an 8-DoF“Ant" quadruped robot to navigate a 2D maze to reach desired goal locations.
The D4RL benchmarkgenerates the offline datasets for these two environments using two policies: 1.
a low-level goalreaching policy that outputs torque commands to move the Ant to a nearby goal location and
ahigh-level waypoint generator to provide sub-goals that guide the low-level goal-reaching policy tothe desired location.
We use the same two policies to generate two new classes of datasets.
For thenoisy dataset, we add Gaussian noise to the action computed by the low-level goal-reaching policy.
The noise variance depends on the current 2D location of the Ant in the maze – larger in some 2Dregions than others.
We intend this dataset to be representative of situations where the data generationpolicies are more deterministic in some states than others [23] – a robot picking up an object hasmany good options to approach the object, but when the robot grasps the object, its behavior shouldbe more deterministic to ensure successful grasp without damaging or dropping the object [29].
For a biased dataset, in addition to adding Gaussian noise to the actions as it is done in the noisydataset, we also add bias to the actions computed by the low-level policy.
The values of the bias alsodepend on the current 2D location of the Ant in the maze.
This setting is meant to simulate learningfrom relabelled data, where the dataset was generated when the data generation policies were performing a different task than the tasks we are using the dataset to learn to perform.
Relabelling offlinedata is a popular method for improving the performance of offline RL algorithms [35, 33], especiallywhen we have much more data for some tasks than others [16].
In the AntMaze environment, offline
RL algorithms must combine data from sub-optimal trajectories to learn behaviors with high returns.
In addition, noisy and biased datasets present a more challenging learning scenarios due to theadded noise and systematic bias which vary non-uniformly based on the 2D location of the Ant.
Table 1 illustrates the performance comparison of our method and representative methods that enforcedistribution constraints, either through optimizing a conservative lower bound of the value estimates(CQL) or only optimizing the policy on actions in the dataset using Advantage Weighted Regression[31] (IQL).
Our method outperforms both CQL and IQL.
In these tasks, to ensure a fair comparisonbetween different methods, we perform oracle offline policy selection to obtain the performanceestimates for CQL, IQL, and our method.
We also compare the performance on standard AntMazetasks without modifications in Table 2.
Our method outperforms IQL by a large margin two
[Table]
By comparing the results in Table 1 (learning from noisy and biased datasets) and Table 2 (learningfrom existing offline datasets in D4RL), we also note that our proposed algorithm outperforms
Our algorithm outperforms the baselines when learning from two diverse datasets
[Table]
[Table]
distribution-constraint offline RL algorithms (CQL, IQL) more consistently when tested on the noisyand biased datasets.
For the results in these two tables, the definition of the antmaze-mediumand antmaze-large environments are the same.
The only axis of variation in the learning setup isthe noise and systematic bias added to the actions of the dataset generation policies.
We thereforeconclude that our algorithm is more robust to the noise and systematic bias added to the actions thandistribution-constrained offline RL algorithms.
Next, we evaluate our approach on Gym locomotion tasks from the standard D4RL suite.
Theperformance results on these tasks are illustrated in Table 3.
Our method is competitive with BC,one-step offline RL methods [2], and multi-step distribution-constraint RL methods [18, 22].
This isnot surprising because in these tasks, the offline dataset contains a large number of trajectories withhigh returns.
In terms of total amount of compute and type of resources used, we use an internal cluster that allowsfor access up to 64 preemptive Nvidia RTX 2080 Ti GPUs. For each experiment of learning from anoffline dataset, we use half a GPU and 3 CPU cores. Each experiment generally takes half a day tofinish. We implemented our algorithms in Pytorch [30] and obtained results for baselines from thepublicly available implementations released by the original authors.
5.2 Ablations
We conduct three different sets of experiments to gain more insights into our algorithm.
The firstexperiment measures the importance of having an auxiliary generator.
We recall that there are twobenefits to having the auxiliary generator.
Firstly, without the auxiliary generator, the generator doesnot in general match the data distribution (Theorem 4.1).
As such, the discriminator has an unfairadvantage in learning how to distinguish between real and generated examples.
Secondly, the auxiliarygenerator plays the role of a support player and learns to output actions that are assigned non-zeroprobability by the data distribution, but have low Q values.
The support player allows the policyto concentrate on in-support maximization of the Q-function (Theorem 4.2).
Table 4 demonstratesthat having an auxiliary generator clearly leads to a performance improvement across different taskfamilies, from Gym locomotion tasks to AntMaze tasks and even dexterous manipulation tasks.
The second experiment compares the performance of the policy and the auxiliary generator on a subsetof the Gym locomotion and AntMaze tasks (Table 5).
The difference in the performance of the policyand auxiliary generator illustrates their specialization of responsibility: the policy learns to outputactions that lead to good performance, while the auxiliary generator learns to model the “remainder" of the data distribution.
If this “remainder" also contains good action, then the auxiliary generatorwill have non-trivial performance.
Otherwise, the auxiliary generator will have poor performance.
In the Gym locomotion tasks, the auxiliary generator has non-trivial performance, but it is still worsethan the policy.
This demonstrates that: 1.
By optimizing the policy to maximize the long-termreturn and the discriminator function, the policy can outperform the auxiliary generator, which onlymaximizes the discriminator function, 2.
The dataset contains a large fraction of medium performancelevel actions contained in continuous trajectories, which the auxiliary generator has learnt to output.
In contrast, in the bias and noisy AntMaze tasks, the auxiliary generator fails to obtain non-zeroperformance while the policy has strong performance.
This reflects the necessity of carefully pickinga subset of the in-support actions to obtain good performance.
[Table] [Table] The third set of experiments illustrates the importance of weighing the value function in the policyobjective by the probability computed by the discriminator, as described in Eq.
Doing so providesa second layer of protection against exploitation of errors in the value function by the policy.
Table 6illustrates that this is very important for the AntMaze tasks, which require combining optimal andsub-optimal trajectories to obtain good performance.
Perhaps this is because learning from suchtrajectories necessitates many rounds of offline policy evaluation and improvement steps, with eachround creating an opportunity for the policy to exploit the errors in the value estimates.
On the otherhand, the dynamic weight is less important in the Gym locomotion tasks, presumably because asignificant fraction of the corresponding offline datasets has high returns and therefore incorporatingsub-optimal data is less criticial to obtain high performance.
[Table]
Conclusions
In this paper, we introduced DASCO, a GAN-based offline RL method that addresses the challengesof training policies as generators with a discriminator to minimize deviation from the behavior policy by means of two modifications: an auxiliary generator to turn the GAN loss into a support constraint,and a value function weight in the policy objective.
The auxiliary generator makes it possible forthe policy to focus on maximizing the value function without needing to match the entirety of thedata distribution, only that part of it that has high value, effectively turning the standard distributionalconstraint that would be enforced by a conventional GAN into a kind of support constraint.
Thistechnique may in fact be of interest in other settings where there is a need to maximize some objectivein addition to fooling a discriminator, and applications of this approach outside of reinforcementlearning are an exciting direction for future work.
Further, since our method enables GAN-basedstrategies to attain good results on a range of offline RL benchmark tasks, it would also be interestingin future work to consider other types of GAN losses that induce different divergence measures.
Wealso plan to explore robust methods for offline policy and hyper-parameter selection in the future.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]1.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes] Please see the experimental
results in Section 5 and avenues for future work in the Conclusion
(c) Did you discuss any potential negative societal impacts of your work?
[No] Because
our focus is on developing a generic offline RL algorithm and not its application in anyparticular domain. Our method therefore inherits the potential negative societal impacts of an offline RL algorithm, such as data bias and lack of robustness to distributional shift.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We will include
the proofs for all theorems in the Appendix
(b) Did you include complete proofs of all theoretical results?
[Yes] We will include the
proofs for all theorems in the Appendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will include
these results in the Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We will include this information in the Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We will include this information in the Appendix
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please see end of section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a ) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d ) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e ) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...(
) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
N/ A](
) Did you describe any potential participant risks, with
links to
Institutional
Review
Board
(IRB) approvals, if applicable?
[N/A
Did you
include
the estimated
hourly
wage
paid
to[-x • Case 2: If p ∗ g assigns non-zero probability to x , then p ∗ g = 2 p data ( x ) • Case 3: There exists an x where 2 p data ( x ) > p ∗ g ( x ) > 
-]
[-[Equation] Let { x 0 , . . . , x i } be the set of x where p ∗ g ( x ) > 0 , then we also require that (cid:80) ij =
 p ∗ g ( x ) = 1 .-]participants
and the total amount
spent
[-, x i } be the set of x where p ∗ g ( x ) > 0 , we also require (cid:80) ij =
-]
on participant compensation?
[N/A]