An Invisible Issue of Task Underspeciﬁcation in Deep Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Performance evaluations of Deep Reinforcement Learning (DRL) algorithms arean integral part of the scientiﬁc progress of the ﬁeld.
However, standard practices in DRL performance evaluations can be unreliable and misleading if not careful.
Animportant source of possible error lies in the reliance of the reported outcomes onoften arbitrarily selected point Markov decision processes (point MDPs), stemmingfrom task underspeciﬁcation.
A large class of DRL tasks, particularly in real-worlddecision problems, induce a family of MDPs, which—perhaps confusingly—eachhas the same high-level problem deﬁnition.
As a demonstrative example, considerthat a classic pendulum control task could be represented by a family of possible MDPs, each with a different pendulum mass, but is typically represented as asingle MDP.
This article argues that for reliable downstream decision-making,performance evaluations on a task in DRL should be carried out over a family of MDPs rather than a point MDP, which may be subject to bias.
This article ﬁrstillustrates the pitfalls of point MDP-based evaluations through benchmark DRLcontrol tasks and a real-world case study in trafﬁc signal control.
Then, signiﬁcantinconsistencies between conclusions derived from point-MDP-based evaluationsand MDP family-based evaluations are presented.
Subsequently, to overcomethe prohibitive cost of training DRL models on entire families of MDPs, a seriesof recommendations is provided to perform accurate yet efﬁcient performanceevaluations under a computational budget.
This work contributes to bolsteringthe empirical rigor of reinforcement learning, especially as the outcomes of DRLtrickle into downstream decision-making in real-world contexts.
Introduction
Deep reinforcement learning (DRL) research has progressed rapidly in recent years, achieving superhuman level performance in many applications.
At the core of DRL research lies the need to performrigorous quantitative performance evaluations.
Performance evaluations (PE) not only disclose thestrengths and limitations of a method but also impact the progress of the ﬁeld at large.
PE of DRLmethods are typically carried out by solving the corresponding Markov Decision Process (MDP) andbuilding performance proﬁles of the method according to some performance metrics.
However, weﬁnd that if not careful, typical practice can lead to unreliable PE.
For example, consider the classic pendulum control task, which is typically represented as a single
MDP.
At the same time, the task is underspeciﬁed and could easily be represented by a family of MDPs, each with a different pendulum mass or length (speciﬁed using some environment parametersdenote by  , which induces different MDPs).
Standard practice for performance evaluation in DRLis to consider a variety of benchmark tasks [5, 26, 29], commonly including the pendulum example. More speciﬁcally, the process typically involves training and evaluating the method on a point MDP
(or a subset of MDPs) chosen from an implicit family of MDPs (e.g., selecting a speciﬁc  in theexample).
We refer to such evaluation approaches as point MDP-based evaluations .
Unfortunately,such point MDPs often appear to be selected arbitrarily or in a way that inadvertently simpliﬁesthe task.
It is well known that the performance of a DRL method is sensitive to the underlying MDP [22, 13, 2].
In this work, we hypothesize that the coupling of an arbitrary selection of point MDPs for evaluation and the general sensitivity of DRL methods can result in substantial errorin the true performance of DRL methods over the implicit MDP family of a task.
This, in turn,harms empirical rigor in DRL.
As DRL trickles into real-world implementation and critical use cases,these errors could have profound implications, ranging from the opportunity cost of employing alower quality method to hurting conﬁdence in the ﬁeld.
While evaluating DRL methods on a sufﬁciently large and carefully chosen family of MDPs can alleviate the issues and enable reliable PE, it often can be prohibitively costly to execute computationally.
Many of the problems can have an excessive amount of possible MDPs that one needs to evaluate.
Moreover, multiple runs per single MDP are needed to achieve statistical conﬁdence.
Especiallywhen considering a suite of benchmark tasks, even training a single model for multiple MDPs can beprohibitively expensive, let alone multiple runs per MDP.
In this work, we take a deep dive into this matter.
In particular, we reveal potential pitfalls ofpoint MDP based PE using classic and modern control tasks frequently used in the DRL literature.
Furthermore, we conduct a case study on a benchmark DRL trafﬁc signal control suite to highlightthe limitations of these pitfalls in real-world applications.
Afterward, we provide a series of recommendations to avoid drawbacks of point MDP based evaluations by evaluating on a family of MDPs while subjected to a computational budget.
Finally, we re-evaluate the trafﬁc signal controlbenchmark accommodating the recommendations and show discrepancies in what has been reportedpreviously.
This article contributes to bolstering the practical accuracy of DRL, when employed for acomplex downstream decision (e.g., whether to adopt an DRL method for a real-world system) andcalls for a change on how PE are performed at present.
Important distinctions: First, this paper focuses on sequential decision-making problems in whichthere is an inherent requirement to devise methods that perform well on the entire family of MDPsthan only on a single MDP in the family.
Such requirements can be frequently seen in real-worldproblems where there is a necessity to achieve economies-of-scale in order to deploy the methodcost-effectively in practice.
Second, we focus on problems where a separate individual DRL modelcan be trained for each MDP ( i.e., to achieve overall better performance).
Another parallel line ofwork is in which a single agent is designed to perform well on all MDPs of the family ( e.g., domainrandomization in robotics).
However, in the scope of this work, we do not consider such problems.
Pitfalls of Point MDP-based Performance Evaluations
We illustrate the pifalls of point MDP-based PE by considering an experiment derived from popular
DRL benchmarks.
We use three popular control tasks (Pendulum, Quad [14], Swimmer) as exampleunderspeciﬁed tasks.
For each task, we augment the nominal task to induce a family of MDPs.
Asummary of the tasks is given in Table 1, along with its augmentation.
For each task family, we thenspecify ﬁve point MDPs.
More details of the tasks and their corresponding modiﬁcations are givenin the Appendix B.1).
Visual representations of the ﬁnal MDP families are given in Appendix B.2.
Since different instances within a task family will reasonably yield different scores, due to changes indifﬁculty and other factors, we will consider the relative performance evaluation between multiple
DRL methods, rather than absolute PE. We consider three popular DRL algorithms (PPO [24],
TRPO [23] and TD3 [12]) for performance evaluation on the tasks.
We have veriﬁed that eachmodiﬁed point MDP is not under-actuated and that given actuator limits, all are solvable.
Observation 1: Reporting performance evaluations based on point MDPs can be misleading .
Previous works commonly use three selection criteria for selecting a point MDP for PE. 1) random
MDP : a random MDP is used to model a somewhat arbitrary selection of an MDPs from the family( e.g., a random selection of a signalized intersection for training a trafﬁc signal control agent [4]) 2)generic MDP : an MDP that represents the key characteristics the MDPs the family ( e.g., the use of ageneric cancer progression model for training a chemotherapy designing agent [31]) and 3) simpliﬁed
MDP : an MDP that simpliﬁes the modeling ( e.g., zero-error instrument modeling when training anagent for chemical reaction optimization [33]).
[Table]
•2D quadcopter in an obstacle course.
• Goal : maneuver the 2D quadcopter through an obstacle course using its vertical acceleration as the control action.
•MDPs with varying the obstacle course lengths (upper obstacle length and lower obstacle length).
[Table]
•A pendulum that can swing.
• Goal : swing up the pendulum.
•MDPs with varying masses and lengths of the pendulum.
[Table]
•MuJoCo 3-link swimming robot in a viscous ﬂuid.
• Goal : make the robot swim forward as fast as possible by actuating the two joints.
[Table] capsule sizes for the segments comprising the robot swimmer.
In Figure 1, we report the PE of the DRL methods on each point MDP described in Table 1 for thethree control tasks.
That is, each method is trained and evaluated on each individual point MDP.
Given a MDP family, all child MDPs are considered possible random MDPs.
For each task, ?
denotesthe generic MDP, and † represents the simpliﬁed MDP.
The simpliﬁed MDP is chosen as the “easiest”
MDP, i.e. simpliﬁes the transition dynamics.
The generic MDP is the parametric mean MDP of other
MDPs of the family.
We observe a few interesting phenomena: (1) the same method yield signiﬁcant differences inperformance under different point MDPs for all three tasks, (2) simpliﬁed MDPs generally achievebetter performance than other point MDPs in the family, and (3) comparing DRL methods based onpoint MDPs can provide conﬂicting conclusions (e.g., conﬂicting relative performance beneﬁts) basedon which point MDP is used for evaluations.
These observations highlight the variability of point MDP-based evaluations and the uncertainties involved.
Such PE could mislead the community toincorrectly conclude that one method is better than another and thereby hinder the scientiﬁc progressof the ﬁeld
[Figure] Observation 2: DRL training can be sensitive to the selected point MDP properties .
It isgenerally known that DRL training is sensitive to the underlying MDP.
Therefore, selection of a point
MDP from a family can demonstrate a training impact that does not generalize to other MDPs in thefamily.
Subsequently, the performance of the DRL method could be overestimated or underestimated.
We demonstrate this phenomenon using our three control tasks in Figure 2 when using the PPOalgorithm for training 1 .
First, in Quad (Figure 2a), reaching the goal means achieving a normalizedscore of 1.
However, only under a few of the point MDPs, the agent reaches the goal during training.
Speciﬁcally, 2 out of 5 settings converge to a local minimum (crashing on the same obstacle astraining progress).
Similarly, in Figure 2b for the pendulum, under one of the MDPs the agent doesnot achieve the goal (stuck at the 200 steps mark) while under the other four the agent achieves thegoal.
Finally, we see similar behavior in Swimmer task in Figure 2c where under one point MDP theagent fails to achieve the goal (stuck at the 5000 steps mark) during training and succeed under others.
Further examples of training complications under other DRL algorithms are given in Appendix B.3.
[Figure]
We hypothesize one of the root causes of this phenomenon is the complexity differences in point
MDPs.
For example, in Quad, making the obstacle-free course narrower requires the DRL agent toexplore some speciﬁc actions different from what it would otherwise need to explore when the courseis wider.
This effectively makes the training process harder for point MDPs with the narrower paths.
Another potential cause is MDP designers over-ﬁtting MDP design to point MDPs.
For example, inpendulum, the default reward function penalizes higher torque values while encouraging the reachingthe goal with a weighted composite reward.
MDP designers may over-ﬁt the the weighting values toa selected point MDP which does not generalize to the entire family.
This can result in point MDPswith higher pendulum mass to fail if the weight on the torque limits are higher.
Notation and Formalism
Given a sequential decision-making task T and a reinforcement learning method R to be evaluated,we consider the setting where there are M possible point MDPs in the family of MDPs.
In general,despite the illustrative examples in the previous section, M can be quite large.
Let N denote acomputational budget in terms of the number of models that can be trained 2 .
We assume that
M  N ; that is, we cannot evaluate the given RL method on all MDPs in the family.
As we demonstrated in Section 2, the performance of a DRL method R can signiﬁcantly depend onthe choice of the point MDP.
As suggested by previous work [1], it is therefore warranted to modelthe performance of R on a given point MDP as a real-valued random variable X .
This means aperformance score s R,i for a given R and a point
MDP i is a realization of the random variable X .
Given a family of MDPs, the a point MDP i may be more important or common than another point
MDP j .
This importance score p T,i of a point MDP on task T can be considered as a real-valued random variable Y .
This means depending on the importance scores of each point MDP, a distributioncan be generated for the MDP family.
For ⌧ 2 R n where ⌧ represents the point MDP characterizationfeatures, we therefore deﬁne the point MDP distribution as F ( ⌧ ) =
P ( Y ) .
Assumption : Given a task, we assume that MDP designers are aware of the importance scores foreach MDP according to some predeﬁned criterion for building the point MDP distribution.
Deﬁnition
The overall performance of a DRL method R on task T is deﬁned as E TR =
P | U | i =1 s R,i ⇥ 146 p
T,i where U is the set of point MDPs.
However, obtaining E TR is not always possible because of the budget constraint M  N .
Therefore,we focus on selecting a subset of point MDPs V from the MDP family to perform a performanceevaluation.
Accordingly, the estimated performance of the method R on task T is ˆ E TR =
P | V | i
=
1 s R,i ⇥ 150 p
T,i .
Clearly, if not careful, selecting different subsets can greatly affect the accuracy of the PE.
An intuitive approach to selecting a subset of point MDPs is to assess
the contribution c
i of each point
MDP i to overall performance evaluation.
Contribution depends both on the importance score, whichis given, and performance score, which incurs a computational cost to evaluate, and can be deﬁned asc i
=
s R,i ⇥ p T,i .
Thus, we seek to ﬁnd the set of point MDPs that has the highest contributions tothe overall performance evaluation.
Given that we wish to estimate the contribution of individualpoint MDPs without assessing the overall performance E TR , this poses a chicken-and-egg problem.
In the subsequent sections, we propose approximation techniques that one can employ to identify asubset V based on the approximated contributions.
Case Study: Trafﬁc Signal Control
To illustrate the pitfalls of point MDP-based evaluations in already established benchmark taskswhere the presence of a large number of point MDPs are clearly visible and considered to besigniﬁcant, we use trafﬁc signal control with reinforcement learning as a case study.
We use a recentlyproposed RESCO DRL benchmark for trafﬁc signal control [4]. RESCO provides a standardizedimplementation of state-of-the-art DRL algorithms for trafﬁc signal control that have become popularin recent years. It also provides non-RL methods that the trafﬁc engineering community presume tobe of value.
We use six algorithms from RESCO in our case study, namely: (1) IDQN : a deep Q-learningapproach, (2) IPPO [3]: same as IDQN with a modiﬁed output layer, (3) MPLight [9]: scalable
FRAP model [32] approach using the pressure concept, (4) MPLight ⇤ : MPLight implementationwith the addition of sensing information, (5) Fixed time : a non-RL pre-deﬁned controller wherephases are enabled for a ﬁxed duration following a ﬁxed cycle, (6) Max pressure [28]: a non-RLcontroller where phase selection is based on the maximal joint pressure.
The intersection layout and vehicle ﬂow rates mainly govern the family of point MDPs in trafﬁcsignal control- each intersection poses as one point MDP.
We base our importance score p T,i ofeach intersection on the frequency of occurrence within a region and the performance score s R,ias the per vehicle average delay.
The importance scores and the intersections used to build the(MDP) distribution of the intersections are taken from Salt Lake City in Utah.
We use 164 uniqueintersections out of 345 signalized intersections as our point MDPs. Refer to Appendix C.1 for moredetails on building this distribution.
In Figure 3, we report the performance of each of six methods on 164 unique point MDPs and thereported performance as per [4] 3 .
First, we observe signiﬁcant variations in the performance basedon point MDP used for evaluations.
Speciﬁcally, all DRL methods demonstrate a signiﬁcant variationwhile non-DRL methods demonstrate comparatively low yet considerable variations.
Second, weobserve that the reported performances in related literature can be signiﬁcantly biased.
As an example,
IDQN and IPPO performances are clearly overestimated.
To quantitatively analyze the potentialpitfalls, we denote the performance of each method over the entire MDP family in Figure 4.
Here, wecap the per vehicle average delay to maximum of 300s to minimize the impact of outliers.
Interestingly, we see a signiﬁcant result change from previously reported results.
Although the Fixed time controller is the least performing method as per reported performance in Figure 3, we ﬁnd that under the family of MDP-based evaluations, the non-RL Fixed time controller and Max pressure controller perform better than all the four DRL controllers.
The Fixed time controller is not ﬁne-tuned for the trafﬁc settings of each intersection but instead conﬁgured to use a default ﬁxed time cycle to be fair with other DRL approaches.
Separately, we also plot the ﬁned-tuned Fixed time controller and observe even more performance beneﬁts.
We note that ﬁne-tuning a non-RL Fixed time controller is simple enough that it does not pose a computational burden and can be done easily, even on a regular computer.
While reported performances ranked IPPO as a well-performing model with an average delay of lessthan 20s, our analysis shows that it, in fact, is the lowest-performing method and that the averagedelay is almost one minute.
It is thus clear that point MDP-based PE can be misleading and may poseperformance beneﬁts that do not generalize to the MDP family.
Recommendations for Reliable Evaluation
Section 2 and the case study in Section C demonstrate the pitfalls of point MDP based evaluationsand requirement for computationally tractable but reliable performance evaluation methods.
In thissection, we present recommendations to alleviate the pointed pitfalls and perform reliable PE under abudget.
Speciﬁcally, we provide three main recommendations to the research community. A summaryof the recommendations are given in Table 2.
[Table]
Lack of benchmarks
•Create benchmark control tasks that depict MDP families.
•Publish datasets of MDP families of control tasks including point-MDP distributions.
•Incentivize publication of such datasets and control task at leading conferences.
Large families of MDPs with limited computational budgets
•Adopt performance approximations using clustering and random sampling under a computational budget.
•Standardize the evaluations by making the selected pointMDPs public.
[Table]
•Use performance proﬁles to show a detailed view of how overall performance changes with point MPDs
5.1 Create data sets of MDP families for benchmark RL tasks for performance evaluations
The use of benchmark datasets for PE is well established in supervised learning.
Ranging fromcomputer vision and biological data to natural language processing, standard datasets are being usedfor PE of deep learning methods.
In reinforcement learning, the use of datasets takes a different form.
Instead of published datasets, researchers use standardized control tasks like Atari 100k [16] and DeepMind control suite [25] for PE.
However, we recognize that current state-of-the-art standard control suits alone are not sufﬁcient tostandardize the performance evaluations in DRL.
The main limitation of current suits is that they onlyprovide an arbitrary-created (or simpliﬁed) point MDPs and do not consider the family of possible MDPs.
As many real-world applications inherently demonstrate a family of MDPs, the current DRLstandardization of PE may steer the research community in a vacuum, while realistically useful DRLmethods may get rejected or not even developed.
Therefore, our ﬁrst recommendation is to create new standard control tasks which demonstrate thenecessity of incorporating a family of MDPs in the DRL method design process.
Second, we alsoencourage the research community to actively create MDP families for existing control tasks (e.g.,different signalized intersections as trafﬁc signal control MDP family) and publish them publically.
Finally, we also encourage main artiﬁcial intelligence conferences with datasets and benchmarktracks like NeurIPS to encourage the community to publish such datasets and tasks and to includenecessary check-ins in the paper submission {+checklists.5.
+} Evaluate DRL methods on a family of
MDPs instead of point MDPs Evaluating performance over an entire family of MDPs can often be computationally expensive tocarry out in practice due to the large cardinality of the family.
Therefore, it is often will require toconduct performance approximations.
In this section, we analyze three techniques that can be used toconduct a performance approximation on a large MDP family.
While more sophisticated methods like active learning approaches are possible, we resort to effectiveyet straightforward techniques. We note that the simplicity of the techniques alone with effectivenessshould be given precedence when designing performance approximations algorithms as it bolsters theadoptability of the techniques within a wider community.
Our three techniques are: (1) M1 : random sampling with replacements from the point MDP distribution, (2) M2 : random sampling without replacements, and (3) M3 : clustering point MDPs usingk-means and assigning probability mass of all point MDPs that belong to same cluster to its centroid.
More details of each method can be found in Appendix D.1.
In Figure5, we denote the approximate PE of IDQN, IPPO and Fixed Time in comparison to theground-truth performance while varying the budget size.
Extra plots for other DRL methods can befound in Appendix 11.
In general, k-means clustering-based approximation produces better estimatesthan the other two techniques with smaller standard deviations.
Speciﬁcally, if the budget size ishalf the total MDP family size, k-means clustering demonstrates a reasonably accurate performance estimates even for methods like IPPO which has a high variations in its performance. However,k-means clustering can be inefﬁcient when the point MDP probability distribution has higher numberof dimensions. If that is the case, we recommend using random sampling with replacements.
Ineither case, we recommend researchers to publish their selection of point MDPs as data sets to enablereproducibility and standardization of the PE of the tasks.
Moreover, a sensitivity analysis of theproposed techniques to the underlying point MDP distribution is also given in Appendix D.2. [Figure]
Remarks.
We acknowledge that there are other factors to consider for a computational budget when conducting DRL studies, including hyperparameter tuning [15] and training using multiple randomseeds within each point MDPs [1].
In this article, we focus on the computational budget allotted forthe task underspeciﬁcation issue. [Figure]
5.3 Use performance proﬁle of the MDP family instead of point MDP performance
[Figure] The typical practice of current DRL research in presenting the performance of a method is to use a point MDP and record performance mean and standard deviation over multiple runs. However, this practice can be misleading and unreliable, as we have argued. Inspired by the idea of performance proﬁling for
DRL evaluations by Agarwal et al. [1] and in optimization software in Dolan et al. [10], our ﬁnal recommendation is to report the performance of a DRL method over a family of MDPs as a performance proﬁle.
Although the overall performance of a DRL method over a family of MDPs can yield more reliableevaluations than evaluating on a point MDP, it may encapsulate further insights into the method’sperformance.
By representing the performance of a method as a performance proﬁle, a more detailedvisualization of the performance over the point MDPs can be illustrated.
Inspired by the Paretodiagrams in statistics, we introduce a proﬁling diagram which we illustrate through an example in
Figure 6. A performance proﬁle illustrates what point MDPs are most probable in the distribution and how eachof the point MDP contributes to the ﬁnal estimate of the performance.
It enables direct comparison ofmethods.
In particular, if the cumulative performance curve of method A is strictly above method
B, method A is said to stochastically dominate method B [1].
Furthermore, point MDPs in which amethod seemingly under-performs is easily visible, giving a clear overview of where the limitationsand strengths are originating from.
Related Work
Deep reinforcement learning (DRL) algorithms have notoriously high variance [1, 15, 8, 8, 6],resulting in reliability and reproducibility issues when applying them to real world applications.
While standardized control benchmarks [5, 26] are proposed to benchmark existing and newlyproposed algorithms, often times ﬂawed evaluation metrics, such as point estimates, are applied thatdo not properly account for the uncertainty in the results and lead to inconsistency in performancerankings among different algorithms.
In light of this, Agrawal et al.
[1] advocates for using betterevaluation metrics such as interval estimates, performance proﬁles, and more robust interquantilemean while Jordan et al.
[15] proposes a performance percentiles measure which normalizes eachalgorithm’s performance by its cumulative density function.
Additionally, Chan et al.
[8] studiesthe behaviors of DRL algorithms by focusing on variability and risk both during and after training.
Besides distribution-aware metrics, a few papers study the effect of architecture design, weightinitialization, and optimization techniques on DRL’s variance and instability [21, 6, 17].
A few papers study DRL evaluation under a budget-constraint setting. A recent work [7] sharesimilar motivation by urging researchers to consider smaller-scale environments in their investigationto reduce the barriers to entry for underprivileged communities.
A related direction is domainrandomization in Sim2Real transfer [20, 30], which samples stochastic training environments insimulation in the hope of covering the true real-world test distribution. It shares similar spirit as ourwork that subsamples a set of environments to approximate the original distribution.
However, workin domain randomization typically assume a pamametric form of the noise (e.g. Gaussian) to samplethe environment from, are not subject to budget constraint (i.e. how many stochastic environmentscan they sample), and train a single model on all training environments.
On the other hand, we takethe nonparametric approach to directly subsample from the probability distribution, are subject to abudget constraint, and allow training a different model for each different environment.
Conclusion and Future Work
In this work, we identify an important yet overlooked issue of task underspeciﬁcation in DRL—inparticular, the reliance of reporting outcomes on somewhat arbitrarily selected point MDPs.
Inparticular, we detail the pitfalls of point-MDP based evaluations and what negative outcomes wouldarise due to this mostly unintentional yet undesired practice.
We provide series of recommendationsto alleviate this issue.
An exciting avenue of future work is to explore the design of efﬁcient yeteffective methods that can produce good approximate performance estimates.
We intend for ourﬁndings to raise awareness in issues of task underspeciﬁcation that impact the empirical rigor of DRL,and aim to help move the needle toward a more disciplined science overall.
Broader Impact
Although there is not a single deﬁnition of responsible machine learning, the Institute for Ethical AI& Machine Learning had developed a series of eight principles to guide the responsible developmentof machine learning systems.
Practical accuracy is one of the principles, which emphasizes that“accuracy and cost metric functions are aligned to the domain-speciﬁc applications.”
This articlecontributes to bolstering the practical accuracy of RL, when employed for a complex downstreamdecision (e.g. whether to adopt an RL method for a societal system), by highlighting some limitationsof current standard RL research practices and proposing to explicitly consider a family of MDPsthat constitutes the complex decision.
On the other hand, even with more empirically rigorous MLpractices, there are still subjective aspects of the task distribution (analogous to family of MDPs in RL), so it remains important to not over-index on RL-based PE; such examples of subjectivity seen inthis work include the task distribution pre-processing approach and permitted error tolerances.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 7.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Table 3 and
Section C in Appendix.
Code and data used are separately submitted as supplementarymaterials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Provided the details in the respective sections where they arediscussed.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes] Attribution-NonCommercial-
ShareAlike 2.0 Generic (CC BY-NC-SA 2.0)
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]