S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Offline reinforcement learning (Offline RL) suffers from the innate distributionalshift as it cannot interact with the physical environment during training.
To alleviatesuch limitation, state-based offline RL leverages a learned dynamics model fromthe logged experience and augments the predicted state transition to extend the datadistribution.
For exploiting such benefit also on the image-based RL, we firstlypropose a generative model, S2P (State2Pixel), which synthesizes the raw pixelof the agent from its corresponding state.
It enables bridging the gap between thestate and the image domain in RL algorithms, and virtually exploring unseen imagedistribution via model-based transition in the state space.
Through experiments,we confirm that our S2P-based image synthesis not only improves the imagebased offline RL performance but also shows powerful generalization capability onunseen tasks.
Introduction
[Figure]
Image Distribution
Virtual Exploration
Out of Distribution
Extended Distribution
Deep learning algorithms have shown significant development thanks to the large pre-collected dataset, such as
SQuAD [47] in natural language processing (NLP), and
ImageNet [4] in computer vision.
On the contrary, reinforcement learning (RL) requires an online trial-and-errorin training process to collect the data by interacting withthe environment, which hinders its utilization in manyreal-world applications.
Due to this intrinsic property ofcurrent online RL algorithms, there exist some approachesthat try to deploy large and diverse pre-recorded datasetswithout online interaction with the environment, which iscalled offline RL.
However, recent studies have observed that the currentonline RL algorithms [10, 35] perform poorly in an offlinesetting.
It is primarily attributed to the large extrapolationerror when the Q-function is evaluated on out-of-distribution actions, which is called the distributionalshift [25, 22, 8].
That is, due to the offline setting that limits online data collection, the offline
RL has struggled to generalize beyond the given offline dataset.
Even though some offline RL methods [26, 24, 54] achieve reasonable performances in some settings, their training is still limitedto behaviors within the given offline dataset distribution, and detours the evaluation on out-of- distribution data rather than directly addressing such an empty space of the offline dataset.
Thus,there exists a growing need for the development of algorithms specialized to directly address suchout-of-distribution data by extending the offline dataset distribution’s support.
To alleviate the fixed dataset distribution problem, recent studies propose some data augmentationstrategies.
In state-based RL, model-based algorithms [2, 1, 27, 16] which learn a dynamics modelfrom the pre-recorded dataset and augment the dataset with generated state transitions have emergedas a promising paradigm.
As the model-based approach trains dynamics models in a supervisedmanner, it allows a stable training process and generates reliable state transition data for augmentation.
Thus, it can be a plausible choice that it enables the generalization into the unseen state-action byperforming dynamics-consistent planning on unseen state distribution.
When it comes to the image domain, however, there is still no augmentation strategy to mitigate theaforementioned distribution shift.
Even though some model-based image RL methods [12, 11] thatpropose to learn latent dynamics using reconstruction error from ELBO objective [18, 51, 23] can beexploited for generating image transition data in a similar manner to the state-based methods, thequality of the output images from these approaches is not satisfactory because 1) their focus is onlearning latent representation suitable for the RL network’s inputs rather than generating high-qualityand accurate images, and 2) ELBO-based objective cannot generate photo-realistic images comparedto other generative models such as GAN [9] or diffusion [14, 41] and it usually produces blurryoutputs.
Above all, 3) model-based image RL only exploits image input and it makes the generativemodel fail to capture the accurate dynamics and the details of the agent’s posture or objects in theimage [40, 42, 3].
These undesired properties discourage offline RL algorithms from adding thereconstruction output of model-based image RL to their training data as an augmentation strategy.
Therefore, we propose S2P (State2Pixel) which utilizes multi-modal input (the state, and the previousimage observation of the agent) to synthesize the realistic image from its corresponding state.
The keyelement of S2P is a multi-modal affine transformation (MAT) which effectively exploits both stateand image cross-modality information.
Unlike previous learned affine transformation [19–21, 44, 34],which leverages a single domain input, MAT fuses the cross-modal representation from the stateand the image to produce the scale and the bias modulation parameters.
This multi-modality of S2Pmakes it possible to generate the dynamics-consistent images from the reliable state transition whilepreserving high-quality image generation capability.
To sum up, our work makes the following key contributions.
• We propose a state-to-pixel generative model (S2P) which generates dynamics-consistent
image and multimodal affine transformation (MAT) module for aggregating cross-modal
inputs.
• To the best of the author’s knowledge, this work is the first to propose image augmentation
for offline image RL and overcome innate fixed distribution problem by implicitly leveragingreliable state transition.•
We evaluate our S2P on the DMControl [52] benchmark environments with the offline
setting, and it results in 1 . 0 − 3 .
0 x higher offline RL performance by augmenting thegenerated synthetic image transition data.• Even with the state distributions of the unseen tasks, S2P can generalize to unseen imagedistribution, and we show that the agent can be trained by offline RL with these generatedimages only.
Related Work2.1 Image Synthesis
Generative Adversarial Networks (GAN) [9] based deep generative models enjoy huge success insynthesizing high-resolution photo-realistic images via style mapping function and the learned affine transformation [19–21, 44, 34].
StyleGAN [19] firstly utilizes a style vector w and Adaptive Instance
Normalization (AdaIN) [5, 15] in the generative networks to disentangle the latent space and controlthe scale-specific synthesis.
Following studies [20] pinpoint that the AdaIN operation which leads toinformation loss in the feature magnitude makes undesired droplet-like artifacts in the synthesizedimages and proposes weight demodulation by assuming the variance of input features.
SPADE [44]proposes an architecture to synthesize the image using its corresponding semantic masks and spatiallylearned affine transformation.
ManiGAN [34] suggests Text-Image Affine Combination Module(ACM) which enables the network to manipulate the images using text descriptions given by users.
The difference between our proposed MAT and the previous studies is that we leverage cross-modaldata, state and image, to estimate the modulation parameters for the learned affine transformationwhereas other studies use a single data type such as text or image.
2.2 Offline Reinforcement Learning & Data Augmentation
Offline RL [6, 29, 33] is the task of learning policies from a given static dataset, which is different fromonline RL that learns useful behaviors through trial-and-error in the environment.
Prior offline RLalgorithms are designed to constrain the policy to the behavior policy used for offline data collectionvia direct state or action constraints [8, 37], maximum mean discrepancy [25], KL divergence[54, 59, 17], or learning conservative critics [26, 24].
However, most of these methods are limited toexploiting the state-action distribution of the given static dataset, rather than exploring and extendingthe distribution.
As the offline setting prohibits online interaction with the environment, we suggestthe synthetic data generation method to enable the offline RL agent to virtually explore and extendthe distribution by leaving the support of the dataset.
Recent works in model-based state RL that involves learning a policy with a dynamics model[22, 2, 1, 28, 57, 58] suggest the need to augment the data with generated transitions from the modelfor extending the data distribution.
Also, augmentation strategies in image domain [30, 49, 31, 55, 56]emphasize the importance of image augmentation for sample efficiency and robust representationlearning.
But, these works focus on purely image manipulation techniques on the given image suchas cropping rather than generating image transition.
Some image-based methods [45, 11, 12] that usethe variational model to train the latent dynamics are studied in a similar concept to the state-basedones.
However, the generated image from these methods is the byproduct of learning the effectiveimage representation rather than the main purpose of these works, and as these methods only utilizethe image input, it leads to missing objects or inaccurate dynamics of the agent in the reconstructedimage.
Therefore, to bridge the gap between the model-based state transition data augmentationtechniques and image augmentation, we propose the method that generates dynamics-consistentimage transition data along timesteps with multi-modal inputs.
Method
3.1 S2P Generator
Architecture.
The goal of S2P (State2Pixel) generator is to synthesize the image ˆ I t which perfectlyrepresents all the information of its corresponding state s t .
Unfortunately, a state-sole conditioncannot formulate a single deterministic rendered image because, in most cases, state does not providethe agent’s position from the global coordinate, but rather from an egocentric coordinate.
Also,image-based RL algorithms utilize sequential images to capture the agent’s velocity using the changeof the background, e.g. ground checkerboard, between input images.
It means that the image of thecurrent step I t is dependent not only on the current state s t , but also on the image of the previousstep I t − 1 .
We, therefore, build the generator G to synthesize the image ˆ I t from both the state s t andthe previous image I t − 1 so that the generated image ˆ I t can preserve the dynamics-consistency in thephysical environment.
[Equation]
At the first layer of S2P, the input image I t − 1 and the state s t are projected to the feature space viaconvolution and MLP encoders respectively.
Both features are then fed to the hierarchical generator
[Figure]
block which consists of several residual connections [13] and the upsampling layer.
After passingthrough each generator block, the spatial size of the feature map is doubled while the channeldimension is halved.
The image features are converted to the RGB images at the last layer of thegenerator with a single convolution layer.
We observe that the input signals, i.e. state s t and image I t − 1 , become attenuated as they pass throughdeeper generating layers and the network produces images with poor quality.
So, similar to recentstyle-based image synthesis algorithms [19–21, 44, 34], we adopt a learned affine transformationarchitecture to inject auxiliary signals to the generator.
The difference between the previous stylebased generative models and S2P is that we propose a multimodal affine transformation (MAT) toproduce the learnable modulation parameters, γ and β , with the cross-modality representation via amultimodal feature extractor and state-to-latent mapping function.
The overall architecture of ourproposed S2P is depicted in Figure 2.
A non-linear latent mapping function g : S → W which is implemented as an 8-layer MLP producesa latent code w ∈ W from the given state s in the state space S .
The latent code w is spatiallyexpanded as the same size of the input feature of MAT module x , and the conditioned image I t − 1 isalso linearly interpolated to make its size same as x .
The spatially expanded w and the resized
I t − 1are channel-wisely concatenated and fed to the multimodal feature extractor to fuse the state andimage cross-modality representation.
Each estimator then produces the learnable scale γ and bias βfor effective cross-modal affine transformation.
Finally, our proposed MAT operation is defined as:
[Equation]
where µ c ( · ) and σ c ( · ) are the channel-wise mean and the standard deviation of the input feature of
MAT x i from the i th block of the generator, and ⊙ denotes Hadamard element-wise product.
Thedesign of MAT is illustrated in Figure 3.
In addition, it is shown that the neural network is biased toward learning a low frequency mappingand has difficulty in representing a high frequency information [46, 39].
To mitigate such undesiredtendency, we do not use naïve state vector s as input, but employ a positional encoding with the highfrequency function ψ : R → R 2 L which is defined as:
[Equation]
where x indicates each component of state vector s .
[Figure] [Figure]
We utilize multi-scale discriminators in [53] toincrease the receptive field without deeper layers or larger convolution kernels for alleviatingthe overfitting.
Two discriminators with iden-
[Figure]
tical architecture are adopted during training,and the synthesized and real images which areresized to several spatial sizes are fed to eachdiscriminator.
Loss Function.
Our S2P generator is trainedby linearly combined multiple objectives.
First,we leverage a pixel-wise L 1 loss between theoutput of the generator ˆ I t and the real image I t ,
[Equation]
In addition to the pixel-wise loss, we also utilize the ImageNet [48] pre-trained VGG19 [50] networkto calculate the perceptual similarity loss L per between two images, [Equation]
[Equation]
where ϕ ( i ) denotes the i-th layer of VGG19 network.
We implement the adversarial objective L adv for both S2P generator and multi-scale discriminatorssame as pix2pixHD [53].
The difference is that we replace the least square loss with the hinge-basedloss [36] and we condition state information to the discriminator D so that the generator is induced toproduce the dynamics-consistent outputs.
[Equation] [Equation]
The total loss function to optimize the S2P generator can be defined as:
[Equation]
where λ 1 , λ 2 , and λ 3 are the hyperparameters to balance among the objectives.
3.2 Offline reinforcement learning with synthetic data
We consider the Markov decision process (MDP) M
= ( I , S , A , T, r, ρ 0 , γ ) , where I denotes theimage space, S the state space corresponding to I , A the action space, T ( s ′ | s, a ) the transitiondynamics, r ( s, a ) the reward function, ρ 0 the initial distribution, and γ the discount factor.
We denotethe discounted image visitation distribution of a policy π using d π M
( I ) :
= (1 − γ ) (cid:80) ∞
t =0 γ t P ( I t =
I | π ) , where P ( I t
= I | π ) is the probability of reaching image observation I at time t by using π in
M .
Similarly, we denote the image-action visitation distribution with d π M ( I, a ) :
= d π M ( I ) π ( a | I ) .
The objective of RL is to optimize a policy π ( a | I ) that maximizes the expected discounted return
J ( π ) =
− γ E ( I,a ) ∼ d π M ( I,a )
[ r ( s, a )] .
In the offline RL setting, the algorithm has access to a static dataset D ={ ( I t , s t , a t , r t , I t +1 , s t +1 ) }
Nt =0 collected by unknown behavior policy π β .
In other words,the dataset D is obtained from d π β M ( I, a ) :
= d π β M ( I ) π β ( a | I ) and the goal is to find the best possiblepolicy using the static dataset without online interaction with the environment.
To utilize the dynamics-consistent transition data for augmentation in offline RL, we take the modelbased approach that trains an ensemble of dynamics and reward model
ˆ T θ ( s ′ , r | s, a ) , which outputsthe predicted next state, and reward
ˆ r
( s, a ) .
Once a model has been learned, we can constructthe learned MDP (cid:99)
M
=
( I , S , A , ˆ T, ˆ r, ρ 0 , γ ) , which has the same spaces, but uses the learneddynamics and reward function.
Naively optimizing the RL objective with the (cid:99) M is known tofail in the offline RL setting, both in theory and practice, due to the distribution shift and modelbias.
To overcome these, we take an uncertainty estimation algorithm like bootstrap ensembles [43, 38], and obtain u ( s, a ) , an estimate of uncertainty in dynamics.
Then we could utilize theuncertainty penalized reward ˜ r
( s, a )
=
ˆ r
( s, a ) − λu ( s, a ) , where λ is a hyperparameter.
We considerthe following uncertainty quantification method that uses the maximum learned variance over theensemble, u ( s, a ) = max i =1 ,...
N || Σ iθ ( s, a ) || F [57, 22].
As a final process, we train offline RL with the following hybrid objective : J ( π ) =11 − γ E
( I,a ) ∼ d f ( I,a ) [ r ( s, a )] , where d f ( I, a ) =
fd π β M ( I, a ) + (1 − f )
d η (cid:102)
M ( I, a ) , f ∈ [0 , 1] is theratio of the datapoints drawn from the offline dataset D , and η ( ·| s ) is the state rollout distributionused with the trained dynamics ensemble model ˆ T θ , and (cid:102)
M is same as (cid:99) M except that the reward is˜ r ( s, a ) instead of ˆ r ( s, a ) .
Samples from d η (cid:102) M ( I, a ) can be obtained by rollout η in (cid:102) M and convert theobtained state transitions { ( s t , a t , r t , s t +1 ) }
Nt =0 into { ( I t , a t , r t , I t +1 ) }
Nt =0 by using the trained S2Pgenerator in Section 3.1.
For implementation, we collect synthetic image transition data in separatereplay buffer D model and train the agent by any offline RL algorithms with the sampled mini-batchesfrom D and D model by the ratio of f and 1 − f .
The overall algorithm and more training details aresummarized in Appendix B.
Experiments4.1 Environments & Data collection
We evaluate our method on a large subset of the dataset from the DeepMind Control (DMControl) suite[52].
It includes 6 environments, which were typically used for online image-based RL benchmarks.
However, to the best of our knowledge, all of these environments have never been properly evaluatedin an offline image-based RL setting.
The datasets in these benchmarks are generated as follows:
random : rollout by a random policy that samples actions from a uniform distribution.
mixed : traina policy using state-based SAC [10] until 500k steps for finger, cheetah and 100k steps for the others,then randomly samples trajectories from the replay buffer.
500k, 100k steps are minimum stepsrequired to reach the expert level performance for each task.
expert : train a policy using state-basedSAC.
After convergence, we collect trajectories from the converged policy.
4.2 Offline Reinforcement Learning
To validate whether the S2P can help improve the offline RL performance, we evaluate our algorithmwith recent offline RL algorithms like CQL [26] which utilizes conservative training of the critic, and
IQL [24] which trains critic by implicitly querying actions near the distribution of the dataset, and
SLAC-off [32] which is state-of-the-art online image-based RL algorithm.
We use this SLAC-offwith the offline setting.
We also compare the policy constraint-based offline RL algorithms like
BEAR [25], and behavior cloning, BC .
The results for these two algorithms are in Appendix B. Toextend the offline RL into the image-based setting, we follow the image encoder architecture from[32] and train a variational model using the offline data.
Then, we train in the latent space of thismodel.
We include the offline RL results on different types of environments and data in Table 1.
The resultson the originally given offline dataset (50k) are shown in the left column of each algorithm, and theresults on the S2P-based augmented dataset are shown in the right column of each algorithm.
The
S2P augments the same amount of the original offline dataset (50k).
As it physically has more data(100k) than the original dataset (50k), for a reference, we also include the results on the 100k offlinedataset in the parenthesis in Table 1.
Overall, the S2P-based method achieves better performancethan the 50k dataset, even exceeding the 100k dataset’s score in some tasks.
As the S2P answers how to generate the image transition data, we also have to consider where togenerate the image transition data through S2P. Specifically, we use a random policy as η ( ·| s ) inthe mixed, expert dataset, and a policy trained by the state-based offline RL as η ( ·| s ) in the randomdataset.
These strategies are considered due to the following two assumptions.
Firstly, as the randomdataset only has random behavior, the dataset may not have any meaningfully rewarded states even
[Table]
with the augmentation by the random policy, especially in the locomotion environment (e.g. cheetah,walker cannot leave the initial states by the random policy).
As the S2P’s objective is to extend thedistribution of the given dataset, the policy trained in an offline manner can help leave the support ofthe dataset compared to the naive random policy.
Secondly, as the non-random datasets may haverelatively biased state-action distributions that receive meaningful rewards compared to the randomdataset, it is difficult to get out of the support of these datasets by the trained policy as most of thestate-action induced by the trained policy are included in the similar distribution of these datasets.
However, the random policy can be effective as it can bring some exploration effects like noiseinjection or increasing entropy in [35, 10].
[Table]
To verify such assumptions, we analyze the effect of different types of rollout distributionη ( ·| s ) in offline RL performance (Table 2).
Wedenote 50k dataset as the results of the givenoriginal offline dataset, and +S2P(random η )as the results of the data augmented by rolloutwith the random actions, and +S2P(offRL η ) asthe results of the data augmented by rollout withthe state-based offline RL policy.
As expected,the random policy is more effective in the expert dataset.
Also, we could find the oppositephenomenon in the random dataset, and trade-offs between these two strategies in the mixed dataset.
4.3 Comparison with model-based image RL
To show why the multi-modal inputs are necessary for augmenting the image transition data in offlineimage RL, we compare our S2P with the previous model-based image RL algorithm, Dreamer
[12],as it can also reconstruct the images of the agent by training the reconstruction error from ELBOobjective only using uni-modal inputs (previous images of the agent).
For comparison, we trained
Dreamer in an offline manner with the same dataset used for training S2P, and predicted future imageswith the episode context obtained from 5 consecutive ground truth images.
Even though the Dreamersaw more previous steps’ images compared to S2P (only single the previous image), it still hasdifficulty in generating accurate posture, which supports the S2P’s advantages (Fig 4).
It is becausethe priority of the model-based image RL algorithms is learning the effective latent representation for RL tasks, and they do not utilize a broader source of supervision from the state inputs.
To prove that state-inconsistent images from the model-based image RL method cannot improveoffline RL performance at the S2P level, we perform the same experiment in Table 1, but replace theaugmented images from S2P with images from Dreamer.
We observe that the image augmentationfrom Dreamer even degrades the original RL performance in several tasks, and the performanceimprovements with S2P excels the augmentation from Dreamer by a large margin in all baselines(Table 3).
Thus, we could say that the inaccurate posture and quality of the images generated by themodel-based method trained with the uni-modal inputs (images) are not sufficient for augmentingimage transition data in the offline setting.
More experiments and details of the augmentation are inthe Appendix.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
4.4 Ablation
To observe how each component of the S2P contributes to the quality of the synthesized images, weperform an ablation study on the model architecture of the S2P and show its qualitative results on
Figure 5(a).
A baseline architecture without any contribution of our proposed method, i.e. positionalencoding (PE) and multimodal affine transformation (MAT), shows the worst image quality.
Wereport that a simple application of the high frequency function ψ to the input state s t (PE) results in anoticeable increase in image quality.
We also address the necessity of the input image I t − 1 to estimatethe modulation parameters, γ , β , for the learned affine transformation.
We ablate the input I t − 1 in
MAT and utilize only the spatially expanded input state which is called State Affine Transformation(SAT).
The generator which replaces the MAT module with SAT has difficulty in exploiting dynamicsinformation from the previous image I t − 1 and the translation error is accumulated as the generatorrecurrently synthesizes the long-horizon trajectory.
We can observe such dynamics inconsistencyespecially in the locomotion tasks, e.g. cheetah, and walker, which expresses the velocity in imagesby the change of the checkerboard in the ground.
Compared to SAT, our proposed MAT whichleverages both the state s t and the previous image I t − 1 shows better performance in reconstructingnot only the posture of the agent but also its dynamics-consistent background.
4.5 Zero-shot Task Adaptation
To validate whether the S2P can help the offline RL process in settings that require generalization totasks that are different from the given dataset, we construct two environments cheetah-jump andwalker-run .
In cheetah-jump , which is referred from [57], the agent should solve a task that isdifferent from the original purpose of the behavior policy.
Specifically, we relabel the rewards in thecheetah-run-mixed dataset to reward the cheetah to jump as high as possible.
Then, we generatethe image transition data from the states whose z positions are bigger than a threshold to validatethe S2P-based augmentation’s advantage in tasks that require generalization.
By training with therelabeled reward, the agent with the S2P achieves a higher return and learns to bounce back and forthto take a leap higher (Figure 5(b)), even though the batch data contain little jumping motion.
To investigate whether the S2P can generate unseen image distributions from unseen state distribution,we collect walker-run state dataset by the same way of other mixed datasets in Section 4.1.
Then, wegenerate images from these unseen states by recurrently using the S2P generator and we apply offline
t =t = 3 t = 1 (a)
[Figure]
[Figure]
Cheetahjump
[Figure] t =t =
t = 3 t =
RL on these generated image transition data.
Even though the state distribution is totally differentfrom walker-walk dataset as the agent should run instead of walk, not only the S2P successfullygeneralizes to the unseen image distributions (Figure 5(b)), but also the agent can be trained to run by offline RL only with these synthesized images beyond the expertise of the dataset (Table 4).
More details are in Appendix B.
[Figure]
We include mean undiscounted return of the episodes in the batch data for comparison.
We attribute such satisfactory task generalization to the model architecture of the S2P whichleverages both I t − 1 and s t for synthesizing the
I t .
The posture of the agent is deterministic withthe sole-state condition and the background ofthe image such as the ground checkerboard isdependent both on the state and the previous image.
Therefore, our S2P exploits the state inputto generate the posture of the agent and exploitsthe image input to generate the background.
Itis well shown in Figure 5(c) where we intentionally replace all the image input with the zero matrix O H × W × 3 during the inference phase.
Weobserve that S2P still perfectly reconstructs the posture of the agent with its corresponding state onlywhile it fails to recover the background and the ground checkerboard as we expected.
Conclusion
We firstly present the state-to-pixel (S2P) algorithm that synthesizes the raw pixel from its corresponding state and the previous image.
As the augmentation paradigm of the S2P is generatingdynamics-consistent image transition data, we demonstrate that S2P not only improves the imagebased offline RL performance but also shows powerful generalization capability on unseen tasks.
For the future work, we plan to further extend other state-based applications to the image-based RLalgorithms.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[No] This work
inherits the potential negative societal impacts of reinforcement learning.
We do not
anticipate any additional negative impacts that are unique to this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We included
code in the supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 3 and Appendix
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section
[Table]
(b) Did you mention the license of the assets?
[Yes] See Appendix.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
We include the code & model in the supplemental materials.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]