Robust Imitation of a Few Demonstrations with a Backwards Model
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Behavior cloning of expert demonstrations can speed up learning optimal policiesin a more sample-efﬁcient way over reinforcement learning.
However, the policycannot extrapolate well to unseen states outside of the demonstration data, creatingcovariate shift (agent drifting away from demonstrations) and compounding errors.
In this work, we tackle this issue by extending the region of attraction around thedemonstrations so that the agent can learn how to get back onto the demonstratedtrajectories if it veers off-course.
We train a generative backwards dynamics modeland generate short imagined trajectories from states in the demonstrations.
Byimitating both demonstrations and these model rollouts, the agent learns boththe demonstrated paths and how to get back on to these paths.
With optimal ornear-optimal demonstrations, the learned policy will be both optimal and robustto deviations, with a wider region of attraction.
On continuous control domains,we evaluate the robustness when starting from different initial states unseen in thedemonstration data.
While both our method and other imitation learning baselinescan successfully solve the tasks for initial states in the training distribution, ourmethod exhibits considerably more robustness to different initial states.
Introduction
While reinforcement learning (RL) has shown remarkable success in many challenging domains[18, 35, 33], tasks with sparse rewards and long horizons still remain extremely difﬁcult to solve.
Insuch tasks, a positive reward is only encountered after the RL agent reaches the goal after a longsequence of actions, meaning that it cannot learn any useful signals until this occurs (typically theagent must reach the goal several times to learn reliably as well).
Furthermore, the learning signaldecreases exponentially with the horizon, which combined with slow gradient-based updates, canlead to catastrophic forgetting even after the agent learns to reach the goal.
Expert demonstrations can help RL agents solve difﬁcult tasks [27, 39, 19].
These demonstrationscan be used in the supervised setting where the agent imitates the expert’s behavior, termed imitationlearning (IL).
However, naively cloning (BC) of the expert’s trajectories suffers from covariate shift:the agent’s policy drifts away from the expert’s, which leads to compounding errors due to RL’ssequential nature.
Furthermore, the distribution of states given in the demonstrations often haslow-dimensional support with respect to the entire state space.
As such, the agent cannot extrapolatecorrectly when outside of the demonstration data.
Many approaches to solve this issue have beenproposed [30, 36, 13], but such approaches require an interactive expert to query the correct actions.
Another way to use demonstrations is to combine imitation learning with reinforcement learning in aform of learning from demonstrations (LfD)
In this case, demonstrations do not simplyact as supervised labels and can guide the agent’s exploration, and also act as augmentations to gooddata samples.
These LfD approaches either use demonstrations to pretrain the policy [32, 7], use an
[Figure]
auxiliary imitation loss in conjuction with the policy update [27, 19], or modify the reward functionsuch that the agent is rewarded when it imitates the demonstrations [45, 28].
One primary concern of relying on demonstrations is that they are costly to obtain, especially inreal-world applications.
Requiring an operator providing corrections in-the-loop to handle covariateshift is often prohibitive as well.
Given only a few ofﬂine trajectories demonstrating successful taskcompletion, an agent ought to be able to replicate the behavior from similar starting conditions, evenif there are small perturbations and variations along the way, and correct itself when necessary.
We areprimarily interested in this setting.
In this work, we aim to minimize the number of demonstrationsnecessary for sparse-reward tasks, while preserving successful task completion.
To tackle covariateshift, we seek an approach that will be robust , in the sense of Figures 1a and 1b: the agent isonly trained with a few demonstrations from a single starting state, but at evaluation time, it mustgeneralize its behavior to a variety of unseen starting states.
If the agent can successfully completethe task from states adjacent to the demonstrated trajectory, then it can recover from small deviationsfrom the demonstration.
Inspired by the notion of “funnels” in robotics [17] and feedback control [2, 16], we introducea reverse-time generative model that can generate possible paths leading the agent back onto thedemonstrations.
These reverse rollouts provide useful information because every rollout ends withinthe support of the demonstration data.
Assuming that the demonstrations lead to the goal, imitatingthese generated reverse rollouts and the original demonstration data allow the agent to reach the goalfrom more starting conditions, including unseen ones.
As illustrated in Figures 1c and 1d, typicalbehavior-cloning (BC) methods focus learning on the small number of demonstrated states, whereasour proposed approach, Backwards Model-based Imitation Learning (BMIL), uses the reverse rolloutsto learn a wider region of attraction around the demonstration.
We validate our approach on a numberof long-horizon, sparse-reward continuous-control tasks.
Even from 5 – 20 demonstrations, BMILprovides a signiﬁcant increase in the region of attraction and robustness on many domains comparedto BC, or to using a forward dynamics model.
We summarize our contributions as follows:
• We propose an imitation learning method that pairs a backwards dynamics model with apolicy and train on both demonstrations and imagined model rollouts.•
In the restrictive setting of an ofﬂine expert and no access to environment interactions, we show that a backwards model can improve robustness over behavior cloning.
• Our experiments on a variety of long-horizon, sparse-reward domains demonstrate that
BMIL can noticeably extend the region of attraction around the demonstration data, evenwhen trained on a very small subsets of the state space.
Related Work
Imitation learning has a long history [24, 32] and is well-studied, as documented in various surveys [1,22, 9].
The challenges of covariate shift and compounding errors are also well-known [24, 29].
Mostsolutions involve on-policy imitation learning, where environment interaction and interactive queryingof the expert allow for the agent’s distribution to match that of the expert [30, 36].
More closelyrelated to our approach are methods that modify or augment the demonstrations to increase robustness.
Laskey et al.
[13] inject noise into the supervisor’s policy during training to force the demonstrator toprovide corrections.
Luo et al.
[15] learn value function and dynamics model from demonstrations, using this to conservatively extrapolate a value function that encourages the agent to return to theexpert data distribution.
Generative approaches have also been used in imitation learning [8, 42], buttheir focus is not on robustly following a few goal-reaching demonstrations.
Time-reversibility has been explored in RL, often as a form of regularization [37, 20, 44, 26, 31].
Reverse-time dynamics models, also called predecessor models, have also been used in RL [3, 4, 34,12, 14, 43, 5].
However, in all cases, the reverse-time dynamics model is used as either an alternativeto the forward-time dynamics model, or as an auxiliary model in addition to the standard forward-timedynamics model, in order to mitigate model-compounding errors.
The result is that the reverse-timedynamics model can accelerate RL and enable greater sample-efﬁciency.
In this work, we take adifferent perspective where the reverse-time dynamics model used to generate possible, unseen pathsthat can lead back to the expert demonstration and thus to the goal, thereby improving robustness infollowing the demonstration.
Our work is also similar to [41], where a reverse-time dynamics modelis used to generate possible trajectories; however, their focus is on ofﬂine RL, where the generatedtrajectories are used to connect distinct sets of states in the ofﬂine dataset.
Method3.1 Preliminaries
We model the setting as a Markov Decision Process (MDP) with a continuous state space S and acontinuous action space A .
T : S ⇥ A 7! S is the transition function that deﬁnes the distributions of 96 the next state s t + 2 S , given the current state s t 2 S and action a t 2 A taken at timestep t .
The objective of imitation learning is to learn a policy ⇡ ✓ parameterized by ✓ , that matches the expert’spolicy ⇡ E .
We assume that the expert generates demonstrations D demo by rolling out its policy ⇡ Ein the environment.
Note that we consider the more restrictive case of where demonstrations onlyconsist of transitions ( s, a, s 0 ) and not rewards, where s 0 is the next state and r is the reward.
Behavior cloning (BC) is a form of imitation learning where the policy ⇡ ✓ learns to imitate expertactions via supervised learning.
The policy learned by behavior cloning is found by minimizing thenegative log likelihood over the demonstration data
[Equation]
Note that BC does not require more interactions with the environment and can be considered to beofﬂine.
Furthermore, the expert behavior is inferred only from demonstrations, without access to theexpert policy itself.
Compounding errors in behavior cloning As pointed out in [29, 40], behavior cloning suffersfrom covariate shift, where errors in the policy can compound and lead the agent to states where itcannot recover.
Intuitively, this occurs as states in the training data D demo is a small subset of theentire state space S
and it is difﬁcult for the policy ⇡ ✓ to learn the optimal action for states outside ofthe training data.
If during a rollout, once the policy ⇡ ✓ makes an error and leaves the D demo , it mayencounter completely new states, leading to the compounding of errors.
Furthermore, as the agentmoves farther away from D demo , there is very little hope for it to make the correct action and moveback onto the distribution of the training data.
3.2 Problem Setting
We consider the same setting as BC, where we do not assume access to the environment or the expertpolicy during training and that only expert demonstrations with no rewards are given.
Furthermore,we consider the setting where the expert demonstrations are given in the MDP where the set of initialstates s 0 2 S 0 and goals g 2 G is restricted to a very small subset of the state space.
In other words,the expert only provides demonstrations for a limited number of initial states and goals.
An exampleof this scenario is where the environment is a maze and the agent starts from the same initial stateand the goal is ﬁxed.
We assume that the expert policy is optimal or nearly optimal, and that alldemonstrations reach a goal.
Formally, we deﬁne the demonstration trajectories ⌧ demo as coming from a probability density
[Equation]
where s 0 2 S 0 , and s T 2 G .
We assume that the set of initial states and goals is very low-dimensionalmanifold of the entire state space: in all of our experiments, S 0 and G consist of a single start or goalstate and the " -ball of its neighborhood.
As such, only a few demonstrations are required to learn astable optimal policy.
Our objective is to learn a policy that is robust to policy errors when imitating expert behavior andcan learn to reach the goal from a variety of initial states.
This is different than the multi-goal ormulti-task setting, where the agent learns to solve multiple goals or tasks, usually from a smallnumber of initial states.
More formally, robustness of the policy R ( ⇡ ✓ ) is deﬁned as
[Equation]
where the expectation is taken over S R that is strictly greater than the initial states given in D demo ,|S R | > | S 0 | .
Note that our measure of robustness is somewhat coarse, in that we do not consider the 135 shortest path to reach the goal from every start state (which would probably require more informationsuch as diverse trajectories or environment interations).
Instead, we seek to extend the region ofattraction around D demo such that the learned policy can still reach the goal.
As we consider continuous states in our work, we measure the robustness R using samples from S R ,either by sampling randomly or by using stratiﬁed random sampling over a discretized grid of thestate space.
Furthermore, we are concerned with being able to learning robustness in terms of spatialcoordinates such that the agent can learn to go back onto the optimal path, rather than other statedimensions such as joint velocities.
Thus we restrict the expectation to only the spatial dimensions.
If the policy ⇡ ✓ can learn to reach the goal from different initial positions not contained in thedemonstrations, then it can also perform the correct action when an error is made during a rollout,making the policy robust to deviations.
Throughout, we assume that there exists an action a 2 A thatallows the policy to go towards D demo when in a state s 62 D demo .
This scenario is true for many 147 navigation and physics-based domains if we ignore the rare circumstances such as irrecoverableunsafe states or the breakdown of the agent.
3.3 Backwards Model-based Imitation Learning
In our work, we use a backwards dynamics model to provide more synthetic training data to thepolicy and therefore increase the policy’s robustness.
We call our method backwards model-basedimitation learning or BMIL.
Backwards model The backwards model is a probabilistic generative model deﬁned as B =p ( s t , a t | s t +1 ) .
This model estimates the conditional distribution of the reverse time dynamics.
It takes in the next state and outputs the previous state and previous action.
As we consider onlycontinuous state and action spaces, we implement B as a conditional Gaussian, parameterized by  .
The backwards model is decomposed into two functions B = B A · B S = p ( a t | s t +1 ) · p ( s t | a t , s t +1 ) , 158 an action generator and previous state generator.
The action generator M A predicts which actionwas taken in order to land to the next state.
There may be several such actions from different statesthat can lead to the next state.
Thus the action generator implicitly encodes a backwards policy.
It isimportant for this backwards policy to closely match the learned forward policy ⇡ but be differentenough to generate diverse new traces such that the policy can become robust by imitating thesetraces.
The previous state generator B S predicts the previous state given the next state and previous actiontaken.
The goal of this generator is to accurately predict the backwards dynamics, which are learnedusing rollouts from the policy ⇡ .
Unlike the action generator, this generator should be as accurate aspossible in order to produce accurate traces.
As we consider the setting with no access to the expert or the environment, the reverse time model Bis trained only on the demonstrations.
The action generator and previous state generator is jointly trained by maximum likelihood
[Equation]
where s t , a t , s t
+1 is the state, action, and next state, respectively, at timestep t .
Model rollouts Given expert demonstrations ⌧ E , we use the backwards model to generate possibleseveral short reverse rollouts or traces ⌧ B , starting from every state in ⌧ E .
Note that all of these tracesend on states within the demonstration data s 2 D demo and as the demonstration reaches the goal,following these traces will eventually lead to the goal.
For all s 1 , . . .
, s T in ⌧ E , we generate K traces⌧ B by repeatedly rolling out previous state-action pairs from B until the desired horizon length H isreached.
These traces are collected into a buffer D M .
If the reverse time model B is accurate and theprevious action generator B A gives sufﬁciently diverse actions, the traces ⌧ B are then samples fromthe region of attraction or “funnels” around every state along the demonstration.
As we assume alldemonstrations reach the goal, these samples from the funnels can be used to learn a robust policy ⇡ ✓ ,as it can follow the traces onto the optimal path.
Action selection strategy for B A
As the backwards model B is trained only on expert data with alimited number of demonstrations, it is very likely that the backwards model can only learn accuratereverse-time dynamics for states contained within or close to the demonstration data D demo .
Thusrepeatedly rolling out B A would only generate traces whose ( s, a ) pairs are contained within D demoand would not help with learning robust policies.
We thus need to choose an action selection strategyfor B A when generating these traces in order to provide synthetic transitions to robustify ⇡ ✓ . A goodaction selection strategy would be similar to the reversed version of the expert policy ⇡ E , but wouldchoose put more weight into less probable actions.
In other words, the action selection strategy for
B A should “cover” the probability distribution p ( a E | s ) , but have enough of a wide tail to providediverse rollouts.
As our backwards model B is probabilistic (implemented as a conditional Gaussianwith diagonal covariance), we can easily perturb p ( a E | s ) by increasing the distribution’s variance.
Let a ⇠ N ( µ,  2 ) =
B A  A ( a | s 0 ) .
We consider two possible ways to generate action a : 1) simplescaling of the  and 2) resampling a noisy version of a , a 0 ⇠ N ( µ,  2 ) , a =
U [ a 0  k, a + k ] , wherek is a ﬁxed hyperparameter.
For the scaling strategy, we further scale  by entropy of the probabilitydensity as we wish to make the distribution “wider” for more peaky distributions.
As we do not wishto deviate too much from the demonstration data, we perturb only the ﬁrst action when s 0 2 D demo .
As the backwards model B is only trained on D demo , it is very likely that B becomes increasinglyinaccurate the farther away state s 0 is from D demo (though some generalization may occur statessufﬁciently close to D demo ).
However, we show that even in this constrained scenario, we ﬁndthat having short reverse rollouts from the backwards model still helps to increase robustness inour experiments.
In addition to using relatively short rollouts that limits compounding errors, wehypothesize that this is these traces do not necessarily need to be accurate in order to beneﬁt thepolicy.
It may be that having the general correct direction contained in the traces is sufﬁcient for thepolicy to eventually reach states in the demonstration data.
Algorithm Our overall method BMIL is outlined in Algorithm 1.
Given expert demonstrations ofthe form ( s t , a t ) , we train the backwards model using Eqn. 1 to estimate the reverse-time dynamicsp ( s t , a t | s t +1 ) .
We train our policy ⇡ ✓ on both the demonstration data D demo and the model traces 209 ⌧ B by sampling from both at a ﬁxed ratio and using maximum likelihood,
[Equation]
where p d is the probability of sampling from D demo .
As our aim is to learn a robust policy while stillsucceeding at the original start states and goals, we sample from the demonstrations a higher ratiothan the model traces.
As we consider the restrictive case of an ofﬂine expert and demonstrations with no rewards, we pairbehavior cloning with the backwards model.
Note that BMIL does not depend on the type of imitationlearning policy; any algorithm can potentially be used as long as the policy can train with samples ofthe form ( s, a ) .
Experiment Design
4.1 Environments
We validate our approach on several continuous control domains: 1) the Fetch robotics environment[23] and 2) maze navigation with two different agents (see Figure 6 and Figure 7 in Appendix A).
For the Fetch robotics environments, we consider the “Push” and “PickAndPlace” tasks.
For themaze environments, we consider 3 mazes of increasing difﬁculty and use two agents.
All domainsuse the Mujoco simulator [38] for a total of 8 distinct domains.
All environments have sparse rewardstructures, where either only every step has a constant negative reward until the goal is reached(Fetch) or only the goal has a non-zero reward (Maze).
More details are provided in Appendix A.
4.2 Demonstrations
For all environments, we obtain demonstrations by ﬁrst training an expert policy. The expert policy istrained by concatenating the goal position to the state, as in goal-oriented learning.
For Fetch, weuse pre-deﬁned settings in [25] to train the expert policy. For the maze environments, we predeﬁne aseries of subgoals, and the position of the next subgoal is concatenated to the state. The reward isdense, using the negative Euclidean distance to the next subgoal and subsequent subgoals, and weuse soft actor-critic [6] as the expert policy. We rollout the expert policy to obtain 5 and 10 expert demonstrations for the Push and PickAndPlaceenvironments, respectively.
As each episode is of length 50 , this amounts to a total of a couplehundred training samples.
We thus make 10 identical copies of each demonstration in order to stablytrain the policy and backwards model (we do the same for all other methods). This has a similareffect of performing more gradient updates, but we found it is computationally cheaper. For Maze,we use 20 demonstrations for all 6 environments and do not repeat the demonstrations as the episodesare sufﬁciently long. 4.3 Implementation Details
For the policy, we use neural networks with 3 fully connected hidden layers with 256 neurons and
ReLU activations.
For the backwards model, we use 4-layer MLPs with 256 hidden units for boththe action predictor B A and previous state predictor B S and use diagonal Gaussian distributions.
Totrain the policy, we use p d = 0 .
5 for the Fetch environments and at higher ratios p d = 0 .
8 ⇠ 0 .
95 forthe Maze environments.
We ﬁnd that higher demo ratios are necessary for longer-horizon and morecomplex domains.
For the model rollouts, we use the variance scaling action selection strategy forthe ﬁrst action only and use an increasing rollout length of 1 to 10 for all domains, similar to [10].
All other hyperparameters and details are provided in the appendix.
4.4 Evaluation
We evaluate BMIL against behavior cloning (BC) and VINS [15].
VINS speciﬁcally aims to learnvalue functions robust to perturbations using negative sampling and the induced policy learns selfcorrective behavior.
VINS was chosen as it is most relevant to our setting; other methods such as
DART [13] or SQIL [28] require either an online expert or environment interactions.
We use the same number of demonstrations for all methods and also keep the same policy networkarchitecture and the total number of policy gradient steps equal across all methods. We train both [Table]
the policy and backwards model until the backwards model loss converges.
Note that our goal isnot to solve the training task faster but rather to robustify the policy using the backwards model.
Additionally, we wish to do solve the task at a various starting conditions while still being able tosucceed at the original initial start-goal states.
To evaluate the robustness of the learned policy, we change the initial positions to random positionsin the state space.
For Fetch, we ﬁx the initial gripper, object, and goal position during training andvary the gripper’s x and y position within the table boundaries during evaluation.
We use 10000samples during evaluation.
For Maze, we initialize the agent to a random start position within adiscretized grid of the maze.
We sample 100 for each discrete grid cell and compute the success rate.
By sampling points for each grid cell, this gives us an idea of which positions are easy for the agentto reach the goal.
Intuitively, such positions would be those near the goal and the demonstrated path.
Results
Our experiments aim to answer the following questions: 1) how robust of a policy does BMIL learn?and 2) what components of BMIL are important to improve robustness?
5.1 Robustness evaluation
The robustness results are shown in Table 1.
We note that the absolute robustness percentages aregenerally low for all methods because it is very difﬁcult to extrapolate from limited demonstrations( 5  20 ) from a single pair of initial start and goal states.
We therefore also include the relativeimprovement over BC.
On the Fetch environments, BMIL substantially increases robustness over BC and VINS.
In particular,our method has an approximately 20% and 330% improvement over BC on Push and PickAndPlace,respectively.
VINS on the other hand performs similarly to BC.
For the Maze environments, BMILgenerally outperforms BC for the Point agent, while the robustness is decreased for the Ant agent.
Somewhat surprisingly, BC performs quite well on the long-horizon Maze domains.
It may be that as BC has some built-in extrapolation capabilities or that the backwards model may need a better latentrepresentation using an encoder.
The success rates during training are shown in Table 3 in the Appendix.
BMIL achieves success ratesclose to 1 for most domains, suggesting that increased robustness does not necessarily come at thecost of decreased performance on demonstrated start-goal states.
On the other hand, VINS cannotconsistently succeed during training, even though its robustness is similar to BC.
Visualization of robustness Figure 2 shows which starting
[Figure]
[Figure]
larger region of attraction over BC and even succeeds at points that are much farther away from
D demo .
We hypothesize that instead of perturbing a single state within D demo as done in VINS,learning a short reverse rollout from this state allows BMIL to learn optimal paths from states muchfarther away from D demo , leading to higher robustness values.
Figure 3 shows a similar visualizationfor some Point maze environments where the agent’s position is discretized into a grid.
BMIL learnsa region of attraction similar in size to that of BC, but has a higher rate of success at each cell.
5.2 Additional Experiments
Forwards vs Backwards Dynamics We ﬁrst analyze the utility of a backwards vs a forwarddynamics model.
On the Fetch environments, we train BMIL with a forward dynamics modelp ( s 0 | s, a ) and compare against the original backwards model p ( s, a | s 0 ) .
To generate modelrollouts from demonstration states, we use the action from the policy a = ⇡ ✓ ( s ) .
As shown in Table 2,the forwards model offers little to no beneﬁt over BC for both Push and PickAndPlace, suggestingthat the backwards model is needed to produce a robust policy.
Action selection strategy We ablate different action selection strategies in trace generation in
Figure 4a (and Figure 8 in the appendix).
Compared to no perturbation (None), we see that either
[Table] [Figure]
Number of demonstrations We also studied our method with varying numbers of demonstrations.
As shown in Figure 4b, both BC and BMIL improve in robustness with more demonstrations, but BCplateaus at a much lower level.
On the other hand, BMIL requires slightly more ( 10 ) demonstrationsthan needed ( 3 demonstrations are sufﬁcient for BC to succeed during training) in order to train thebackwards model (Figure 9 in the appendix).
Computation budget As BMIL trains both the policy and the backwards model, it requires moretotal gradient updates than BC.
On Fetch domains, BMIL uses approximately 5 x more computationthan BC.
We train BC for more steps to match or exceed BMIL’s computation budget, as shown in
Figure 5 (BC given 1 x– 20 x computation budget).
However, more training for BC does not improverobustness and in fact seems to have a harmful effect on robustness.
[Figure]
BMIL has roughly 5 times more total gradient steps than BC(1x) due to the backwards model update.
More BC gradient steps does not increase robustness
Discussion
This work proposed a method to tackle the issue of covariate shift in imitation learning.
Weconsidered the restrictive setting where the expert is ofﬂine, where its behavior can only be inferredfrom demonstrations, and no access to additional environment interactions.
Speciﬁcally, we showthat pairing a generative backwards model with behavior cloning can allow a policy to learn a widerregion of attraction around the demonstration data.
By rolling out imagined traces from states withinthe demonstration and perturbing actions to generate diverse traces, BMIL learns a wider funnel thannaive BC.
Through experiments on several long-horizon, sparse-reward, continuous control domains,
BMIL noticeably improves robustness when trained on a narrow set of initial start and goal states andevaluated at random starting positions.
Future work will include applying BMIL to more difﬁculttasks and extending BMIL to allow for environment interactions and pairing with other imitationlearning algorithms.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code and
data will be provided in the supplementary material, and also released prior to ﬁnal
publication.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Brieﬂy in Section 4, and in full in the supplementary material.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary material.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board ) approvals, if applicable?
[N/A]
[Table]
) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]