Low-Rank Modular Reinforcement Learning via Muscle Synergy
Abstract
Modular Reinforcement Learning (RL) decentralizes the control of multi-jointrobots by learning policies for each actuator.
Previous work on modular RL hasproven its ability to control morphologically different agents with a shared actuator policy.
However, with the increase in the Degree of Freedom (DoF) of robots,training a morphology-generalizable modular controller becomes exponentiallydifﬁcult.
Motivated by the way the human central nervous system controls numerous muscles, we propose a Synergy-Oriented LeARning (S OLAR ) framework thatexploits the redundant nature of DoF in robot control.
Actuators are grouped intosynergies by an unsupervised learning method, and a synergy action is learned tocontrol multiple actuators in synchrony.
In this way, we achieve a low-rank control at the synergy level.
We extensively evaluate our method on a variety of robotmorphologies, and the results show its superior efﬁciency and generalizability, especially on robots with a large DoF like Humanoid ++ and UNIMALs.
Introduction
Deep reinforcement learning (RL) has contributed signiﬁcantly to the sensorimotor control of bothsimulated [Heess et al., 2017, Zhu et al., 2020] and real-world
[Levine et al., 2016, Mahmoodet al., 2018] robots.
Monolithic learning is a popular paradigm for learning control policies.
In thisparadigm, a policy inferring a joint action for all limb actuators based on a global sensory state islearned.
Although monolithic learning has made impressive progress [Chen et al., 2020, Kuznetsovet al., 2020], it has two major shortcomings.
First, the input and output space is large.
For robotswith more joints, learning controlling policies puts a heavy burden on the representational capacity of neural networks.
Second, the input and output dimensions are ﬁxed, making it inﬂexible totransfer the learned control policies to robots with different morphologies.
Modular reinforcement learning provides an elegant solution to these problems.
In this learningparadigm [Wang et al., 2018], the control policy is decentralized [Peng et al., 2021], and each limbactuator is controlled by a local policy.
Recent research efforts show that the local policies can learnhigh-performance and transferable control strategies by sharing parameters [Huang et al., 2020],communicating to each other by message passing [Huang et al., 2020], and adaptively paying attention to other actuators via graph neural networks [Kurin et al., 2020].
By exploiting the ﬂexibilityand generalizability provided by modularity, a modular policy can now control robots of up to thousands of morphologies [Gupta et al., 2021a].
Despite the signiﬁcant progress, modular reinforcement learning is still limited in terms of the complexity of morphological structures that can be controlled and struggles on robots with many jointslike Humanoid [Kurin et al., 2020].
The large degree of control freedom presents a major challenge for learning control policies.
A question is why humans can control hundreds of muscles withdexterity while the most advanced RL policy can only control less than ﬁfteen actuators.
Studies on muscle synergies [d’Avella et al., 2003] may provide an answer.
A human central nervous system decreases the control complexity by producing a small number of electrical signalsand activating muscles in groups [Ting and McKay, 2007].
Muscle synergy is the coordination of muscles that are activated in synchrony.
With muscle synergies, the human nervous system achieves low-rank control over its actuators.
In this paper, we aim to use the inspiration of muscle synergiesto reduce the control complexity and improve the learning performance of modular RL.
The ﬁrst challenge of incorporating muscle synergies into modular RL is to discover a synergy structure that can promote policy learning.
Neuroscience researchers factorize electrical signals [Saitoet al., 2018, Falaki et al., 2017, Kieliba et al., 2018] to analyze the synergy structure, but policysignals are sub-optimal or even absent during reinforcement learning.
We thus exploit the functionalsimilarity and morphological context of actuators and use a clustering algorithm to identify actuatorsin the same synergy.
The intuition is that muscles in a synergy typically serve the same functionalpurpose and have similar morphological contexts.
We quantify the functional similarity by the inﬂuence of an actuator’s action on the global value function, and the morphological structure is encodedas a distance matrix.
To use the two types of information simultaneously, we adopt the afﬁnity propagation algorithm
[Frey and Dueck, 2007].
The synergy structure is updated periodically duringlearning to promptly reﬂect changes in value functions.
To exploit the discovered synergy structure, we design a synergy-aware architecture for policy learning.
The major novelty here is that the policy learns action selection for each synergy, and thesynergy actions are transformed linearly to get actuator actions.
Since the number of synergies istypically much smaller than actuators, we actually learn a low-rank control policy where the physicalactions are a linear mapping from a low-dimensional action space.
Moreover, for better processingstate information, the synergy-aware policy adopts a two-level transformer structure, which ﬁrstaggregates information within each synergy and then processes information across synergies.
We evaluate our Synergy-Oriented LeARning (S OLAR ) framework on two MuJoCo
[Todorov et al.
,2012] locomotion benchmarks [Huang et al., 2020, Gupta et al., 2021b] and in multi-task to zeroshot learning, single-task settings.
S OLAR signiﬁcantly outperforms previous state-of-the-art algorithms in terms of both sample efﬁciency and ﬁnal performance on all tested settings, especially onrobots with a large DoF like Humanoid ++ [Huang et al., 2020] and UNIMALs [Gupta et al., 2021b].
Performance comparison and the visualization of learned synergy structures strongly support the effectiveness of our synergy discovery method and synergy-aware transformer-based policy learningapproach.
Our experimental results reveal the low-rank nature of multi-joint robot control signals.
Background
Modular RL .
Modular Reinforcement Learning decentralizes the control of multi-joint robots bylearning policies for each actuator.
Each joint has its controlling policy and they coordinate witheach other via various message passing schemes.
Modular RL usually needs to deal with agents
withdifferent morphologies.
To do so, Wang et al.
[2018] and Pathak et al.
[2019] represent the robot’smorphology as a graph and use GNNs as policy and message passing networks.
Huang et al.
[2020]uses both bottom-up and top-down message passing scheme through the links between joints forcoordinating.
All of these GNN-like works show the beneﬁts of modular policies over a monolithicpolicy in tasks tackling different morphologies.
However, recently, Kurin et al.
[2020] validated ahypothesis that any beneﬁt GNNs can extract from morphological structures is outweighed by the difﬁculty of message passing across multiple hops.
They further propose a transformer-based method,
A MORPHEUS , that utilizes self-attention mechanisms as a message passing approach.
A MORPHEUSoutperforms prior works and our work is based on A MORPHEUS .
Previous works mainly focusedon effective message passing schemes, while our work aims at reducing learning complexities whenthe DoF of the robot is large.
Muscle Synergy .
How the human central nervous system (CNS) coordinates the activation of alarge number of muscles during movement is still an open question.
According to numerous studies,the CNS activates muscles in groups to decrease the complexity required to control each individualmuscle [d’Avella et al., 2003, Ting and McKay, 2007].
According to muscle synergy theory, the
CNS produces a small number of signals.
The combinations of these signals are distributed to themuscles [Wojtara et al., 2014].
Muscle synergy is the term for the coordination of muscles that activate at the same time [Ferrante et al., 2016].
A synergy can include multiple muscles, and a musclecan belong to multiple synergies.
Synergies produce complicated activation patterns for a set ofmuscles during the performance of a task, which is commonly measured using electromyography(EMG)
[Tresch et al., 2002, Singh et al., 2018].
EMG signals are typically recorded as a matrixwith a column for activation signals for a moment and a row for activation of a muscle [Rabbi et al., 2020].
Factorisation methods on the matrix are used to extract muscle synergies from muscle activation patterns.
Four most commonly used factorization methods are non-negative matrixfactorisation [Steele et al., 2015, Schwartz et al., 2016, Lee and Seung, 1999, Rozumalski et al.
,2017, Shuman et al., 2016, Saito et al., 2018] , principal component analysis [Ting and Macpherson,2005, Ting et al., 2015, Danion and Latash, 2010, Falaki et al., 2017], independent component analysis [Hyvärinen and Oja, 2000, Hart and Giszter, 2013], and factor analysis [Kieliba et al., 2018,
Saito et al., 2015].
Afﬁnity propagation [Frey and Dueck, 2007] is a state-of-the-art clustering algorithm based onmulti-round message passing between data points.
It does not need to pre-deﬁne the number of clusters beforehand and proceeds by ﬁnding for each instance the exemplar input that are representativeof a cluster.
Input data points that choose the same exemplar input belongs to the same cluster.
Suppose { x i } ni =1 is a set of data points.
Deﬁne S ∈ R n × n as a similarity matrix with the element s i,jat i th row and j th column being the similarity between x i and x j . The similarity can be measuredas, for example, the negative squared distance of two data points.
The diagonal of S is particularlyimportant because it represents how likely the corresponding instance is to become an exemplar.
The vector of diagonal elements, ( s 11 , s 22 , . . .
, s nn ) , is called preference .
Non-diagonal elementsin S constitute the afﬁnity matrix.
Afﬁnity propagation proceeds by updating two matrices: theresponsibility matrix R whose values r i,j represent whether x j is well-suited to be the exemplarfor x i ; the availability matrix A whose values a i,j quantify the appropriateness for x i picking x j asits exemplar.
These two matrices are initialized to be zeroes and can be regarded as log-probabilitytables.
The algorithm then alternatives between two message-passing steps.
First, the responsibilitymatrix is updated:
[Equation]
Then, the availability matrix is updated:
[Equation]
Messages are passed until the clusters stabilize or the pre-determined number of iterations is reached.
The exemplars are those whose responsibility value plus availability value ( r i,i + a i,i ) is positive.
Method
In this section, we present our Synergy-Oriented LeARning (S OLAR ) scheme that incorporates themuscle synergy mechanism into modular reinforcement learning to reduce its learning complexity.
Our method has two major components.
The ﬁrst one is an unsupervised learning module thatutilizes the morphological structure and value information to discover the synergy hierarchy.
Thesecond is a novel attention-based policy architecture that supports synergy-aware learning.
Both ofthe components are specially designed to enable the control of robots with different morphologies.
We ﬁrst introduce the problem settings and then describe the details of the two components.
Problem settings.
We consider N robots, each with a unique morphology.
Agent n contains K nlimb actuators that are connected together to constitute its overall morphological structure.
Examples of such robots that are studied in this paper include Humanoid ++ and UNIMALs.
At eachdiscrete timestep t , actuator k ∈ { 1 , 2 , . . .
, K n } of a robot n ∈ { 1 , 2 , . .
, N } receives a local sensory state s n,kt as input and outputs individual torque values a n,kt for the corresponding actuator.
Then the robot n executes the joint action a nt = { a n,kt } K n k =1 at time t , after which the environmentreturns the next state s
nt +
= { s n,kt +1 } K n k
=1 corresponding to all limbs of the agent n and a collectivereward for the whole morphology r n t ( s n t , a n t ) .
We learn a policy π θ to generate actions based onstates.
The learning objective of the policy is to maximize the expected return on all the tasks:
[Equation]
where γ is a discount factor.
We adopt an actor-critic framework for policy learning.
The critic isshared among all tasks and estimates the Q-function for each robot n :
[Equation] [Equation]
I n t r a - S y n e r g y S e l fA tt e n t i o n
I n t e r - S y n e r g y S e l f - A tt e n t
i o n
A c t i o n T r a n s f o r m a t i o n
S y ne r g y S t r u c t u r e
P e r a c t ua t
o r a c t i on
3.1 Discovering synergy structure
In Neuroscience, the muscle synergies are usually discovered by a factorization of electrical musclesignals during performing tasks [Todorov and Ghahramani, 2004, Rabbi et al., 2020].
Factorizationis a method in hindsight – it statistically analyzes the optimal control policies of animals embodiedin the electrical signals.
By contrast, in reinforcement learning, we do not have the optimal control policies in advance.
The synergy hierarchy learned from non-optimal policies is more likelyincorrect, which will hamper policy learning.
Therefore, we propose learning the synergy hierarchyby an unsupervised learning method that incorporates morphological information besides learninginformation.
In this section, we describe our synergy hierarchy discovery methods.
Intuitively, actuators in the same synergy are activated simultaneously and they together ﬁnish amotion of an end effector.
This gives us a hint that actuators with similar functions are supposed tobe in a synergy.
Formally, the function of an actuator (the k th actuator in robot n ) can be modelledby its inﬂuence to the value function:
[Equation]
where b n,k is a default action of agent i ( b n,k = 0 in MuJoCo environments) and a n, - k is the actionsof actuators on robot n except the k th limb.
In practice, we use a SoftMax function to regularize
[Equation]
Ideally, ∆ ˜ Q n,k contains actuator clustering information, but the Q-values may be inaccurate duringlearning.
We propose to treat ( ∆ ˜ Q n, 1 , ∆ ˜ Q n, 2 , · · · , ∆ ˜ Q n,K n ) as the preference vector of the afﬁnitypropagation clustering algorithm
(Sec. 2) and we further encode stable morphological informationin the afﬁnity matrix of afﬁnity propagation.
The advantage of this design is that (1) we can simultaneously consider functional similarity and morphological context when using clustering to discoversynergies and (2) static morphological information may help stabilize the clustering results.
We now discuss how to encode morphological information in the afﬁnity matrix.
A robot morphology is usually organized in a tree structure with its torso as the root node [Huang et al., 2020].
Wedeﬁne the adjacency matrix A d ∈ { 0 , 1 } K n × K n to represent robot’s morphology, where A i,j = 1 if actuators i, j are directly linked.
To measure the connectiveness of every actuator pair, we use
Floyd-Warshall algorithm
[Cormen et al., 2022] to compute the shortest distances between actuatorsbased on A d , resulting in a distance matrix D ∈ N K n × K n .
The matrix exp( − D ) will serve asthe afﬁnity matrix A of afﬁnity propagation.
After deﬁning the preference and afﬁnity for afﬁnitypropagation, the clustering algorithms run n max rounds of message passing to determine actuatorclusters.
We treat each cluster as a synergy.
In practice, we update the synergy structure periodically during the learning process.
While Dremains unchanged, ∆ ˜ Q n,k changes as the Q-function is updated.
Each time we run the afﬁnitypropagation with the new preference vector and then freeze the synergy structure for a while.
3.2 Learning synergy-aware low-rank policies
We now describe how we design a synergy-aware actor-critic learning framework that exploits thesynergy hierarchy discovered in Sec. 3.1.
The core idea is that the policy generates control signals (actions) for synergies instead of actuators.
As the number of synergies is much fewer than actuators, the learning complexity is reduced.
Thesynergy actions will be combined linearly to generate physical actions for actuators.
In this way, welearn a low-rank control policy – the physical actions are actually a linear combination of actions ina low-dimensional space.
Suppose that we ﬁnd L n synergies { S n,i } L n i =1 for the n th robot.
We use a mask M n ∈ { 0 , 1 } K n × K nto represent the synergy structure.
The element at the i th row, j th column of M n is 1 if actuator i andj are in the same synergy.
The policy π θ consists of three components: an intra-synergy attentionmodule, an inter-synergy attention module, and an action transformation matrix as in Figure 1.
The ﬁrst module aggregates information of actuators in a synergy.
Input to this module is the jointstate s n , and this module outputs a hidden representation h n,i ∈
R d for each synergy S n,i .
Inpractice, we use a transformer with a two-head self-attention layer where the attentions betweenactuators belonging to different synergies are masked out.
The output of the self-attention layer is ofthe size K n × d .
We then use mean pooling to aggregate information of actuators in the same synergy.
In this way, we get a hidden state h n ∈ R L n × d , with each row being the aggregated information ofa synergy.
The second module aggregates information from different synergies and outputs the synergy actions.
It has two multi-head self-attention layers.
We feed h n into this these layers and get real-valuedsynergy actions a ns ∈ R L n .
The synergy actions are then transformed by a learnable matrix T ∈ R K n × L n , and we obtain actions for actuators by a n = T a ns .
One question about the transformationmatrix T is that L n may change during the learning process, requiring the dimension of T to bedynamic.
In Appendix A, we discuss how to deal with this problem.
The critic uses the same intra- and inter-synergy attention structure as the policy for processingstate information.
The outputs of the inter-synergy attention module are fed into a fully-connectednetwork to obtain the Q-value estimation.
In practice, we use TD3 [Fujimoto et al., 2018] for thetraining of the policy and the critic.
The intra- and inter-synergy transformer and the transformationmatrix are updated in an end-to-end manner.
We provide more details of our learning framework in
Appendix A.
Experiments
In this section, we benchmark our method S OLAR on various MuJoCo
[Todorov et al., 2012] locomotion tasks.
we evaluate the effectiveness of S OLAR by asking the following questions: (1) Canour S OLAR outperforms other modular RL approaches when simultaneously trained on a large number of diverse agents with different morphologies?
(Sec. 4.2) (2) Can the learned policy in multi-taskenvironments generalize to new tasks with unseen morphologies?
(Sec. 4.3) (3) How does S OLARlearn synergy clusters and how do synergy clusters facilitate learning?
(Sec. 4.4) (4) Can S OLARscale to single-tasks with numerous joints?
(Sec. 4.5).
For qualitative results, please refer to thevideos on our project website 1 .
4.1 Experiment setup
We run experiments on two benchmarks, and report several results in this section.
For additionalresults, please refer to Appendix B. We test all methods with 4 random seeds and show the meanperformance as well as 95% conﬁdence intervals.
Environments .
For multi-task and zero-shot evaluation, we adopt the widely-used modular MTRLbenchmarks [Huang et al., 2020, Kurin et al., 2020, Hong et al., 2021], which are created basedon Gym MuJoCo locomotion tasks by Huang et al.
In this section, we report results on 3in-domain settings: (1) 6 variants of walker
[ Walker ++] (2) 8 variants of humanoid [ Humanoid ++](3) 3 variants of hopper [ Hopper ++]; and 1 cross-domain settings: all 6 variants of walker, all 8variants of humanoids, and all 3 variants of hopper [ Walker - Humanoid - Hopper ++].
For single-taskevaluation, we sample 3 morphologies from UNIMALS [Gupta et al., 2021b,a], which are obtainedfrom evolution and have numerous joints, making it suitable to evaluate the performance of modular
RL policies in single-tasks.
For a detailed description of the environments please refer to Guptaet al.
[2021b] and Appendix A.
Baselines .
In this section, we compare our method S OLAR against state-of-the-art modular RLmethods SMP [Huang et al., 2020] and A MORPHEUS
[Kurin et al., 2020].
SMP passes messagesalong the limbs using a bottom-up and top-down scheme.
S OLAR and A MORPHEUS use transformerbased actors and critics, whose message passing schemes are realized by self-attention.
Please referto Appendix A for more details about baselines.
Ablations .
There are two contributions that characterize our method.
(1) An unsupervised learningmodule that utilizes the value information and the morphological contextual structure to discoverthe synergy hierarchy.
(2) A novel attention-based synergy-aware policy architecture that supportssynergy-aware learning.
Our novelties are mainly about the discovery and utilization of synergies,and these two contributions closely rely on each other.
If we totally ablate synergies, we will get
A MORPHEUS .
Therefore, the effectiveness of synergy-based learning can be demonstrated by comparing S OLAR against A MORPHEUS .
We further design the following ablations: (1) A MORPHEUSw/ synergy mask .
Incorporate synergy-aware inter-cluster masks into A MORPHEUS to test the contribution of synergy hierarchy.
(2) S OLAR w/o preference .
Remove value information (the preferencevector) from afﬁnity propagation of S OLAR .
In this case, afﬁnity propagation is based solely onmorphological information, and we can test the effects of value information.
Implementations .
We use TD3 [Fujimoto et al., 2018] as the underlying reinforcement learningalgorithm for training the policy over all baselines, ablations and our method for fairness.
Weimplement S OLAR in the A MORPHEUS codebase.
And like A MORPHEUS , there is no weight sharingbetween actor and critic.
S OLAR uses a traversal-based embedding for action transformation, whichmay incorporate structural information.
For a fair comparison, we concatenate these embeddingsinto the original observation vectors of each environment which is applied to all tested algorithms.
4.2 Multi-task with different morphologies
[Table]
We summarize the multi-task results in Figure 2.
S OLAR outperforms the previous state-of-theart algorithms
A MORPHEUS and SMP in all settings.
Furthermore, the performance gap between
S OLAR and A MORPHEUS is notably larger in Humanoid ++ which has the maximum number ofjoints among the tested settings.
We also plot the learning curves of six training variants (Figure 3)and two testing variants (Figure 4) of Humanoid ++.
These results consistently demonstrate theeffectiveness of S OLAR in training robots with a large degree of freedom (DoF). We speculate that
[Table]
the reason of the effectiveness is that S OLAR controls robots in a lower-rank action space than theoriginal space and the action spaces of robots with a large DoF are more likely to be reduced viathe synergy mechanism.
To further validate this speculation, we conduct extra evaluations on robotswith much more joints in Sec. 4.5.
As for ablations, A MORPHEUS w/ synergy mask outperforms A MORPHEUS in most tasks in Figure 2and Figure 3, which indicates the effectiveness of the unsupervised learning method that discoverssynergy clusters and synergy-aware masks in the policy.
However, S OLAR w/o preference performscomparable to or even worse than A MORPHEUS , which reﬂects the prominent usefulness of valueinformation.
4.3 Zero-shot generalization
[Figure]
[Table]
[Figure] In this section [Figure] we benchmark S OLAR in a zero-shot learning setting where the policy is trained inmultiple tasks and then generalized to unseen tasks.
We compare S OLAR with A MORPHEUS , SMP,
A MORPHEUS w/ synergy mask and S OLAR
w/o preference in Humanoid ++, which has the largestnumber of joints among the tested environments.
Humanoid ++ has eight variants of humanoids,where six of them are used as training tasks (see Figure 3) and the other two are used as testingtasks (Figure 4).
To obtain the results of testing tasks at each time step, we periodically load policynetwork parameters when training and evaluate the policy in these tasks.
The trajectories on testingtasks will be discarded and will not be used for learning.
As shown in Figure 4, S OLAR outperforms all the baselines by a large margin in these unseen tasks.
S OLAR learns faster than other baselines and shows higher sample efﬁciency in Figure 3, showingthat the knowledge obtained from training tasks is effectively used in unseen tasks.
4.4 Analysis of synergies
To investigate why S OLAR performs better than baselines, in this section, we visualize the synergyclusters learned by S OLAR and the evolutionary process of synergy clusters in Humanoid ++.
[Figure]
Afﬁnity propagation which we used as the method to discover synergy clusters can naturally outputsclustering results and the cluster centers in each cluster.
We ﬁrst visualize the synergy clusters of
S OLAR in Humanoid ++ at the end of training in Figure 5.
Here we use different colors and twoshapes to mark each joint in robots.
Different colors represent different synergy clusters, and jointsmarked with the same color are in the same cluster.
Joints marked with triangles are the centersof their corresponding clusters.
For example, the Full humanoid in the lowermost right corner has4 synergy clusters, and the joints between the torso and two thighs and two arms are the clustercenters.
To further investigate how these clusters evolve, we visualize the learning process of Left Knee
Humanoid in Figure 6.
According to the change of synergy clusters, the learning process can bedivided into three phases.
In Phase 1, the episode reward is very low and ∆ ˜ Q may only providenoisy information, which results in all joints being clustered in a same cluster.
In Phase 2, S OLARﬁnds that the right shin is important and divides it into a separate cluster.
Then in Phase 3, S OLARﬁnally obtains a suitable clustering solution and the episode reward increases correspondingly.
The visualization results reﬂect two facts: (1) close joints are more likely to be in the same synergycluster, and (2) joints near the torso may be more inﬂuential than those who are far from the torso,and are thus selected as the cluster centers.
These clusters can reduce learning and controlling complexities but are hard to discover through a standard learning mechanism.
Instead, S OLAR makes useof early, less accurate learning information ∆ ˜ Q and structural information via unsupervised learningto discover synergy clusters.
The synergy clusters then further facilitate learning.
[Table]
[Figure]
Mean Episode Reward of Amorpheus
[Figure]
[Table]
To consolidate the speculation in Sec.
4.2 that S OLAR is suitable for tasks with numerous joints, wetest S OLAR in three single-robot tasks sampled from UNIMALS [Gupta et al., 2021b].
The averagenumber of joints is about twice as many as that in Humanoid ++.
For more details about these tasks,please refer to Appendix A. We showcase the results in Figure 7.
S OLAR learns faster than state-ofthe-art A MORPHEUS , showing much higher sample efﬁciency, and again outperforms it in averageepisode rewards.
Conclusions
In this paper, we use the inspiration of muscle synergies from neuroscience to reduce the controlcomplexity of modular reinforcement learning in tasks with a large number of degree of freedom.
We present S OLAR that leverages both structural information and policy learning information tolearn synergy clusters.
Moreover, a stacked-transformer architecture is proposed for learning synergy actions which are combined linearly to produce low-rank actuator actions.
The evaluations of
S OLAR on various tasks and settings showcase its effectiveness.
The authors do not see obviousnegative societal impacts of the proposed method.
Checklist
For all authors
(a) Do the main claims made in the abstract
and introduction
accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In Appendices.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
In Appendicesand supplemental materials. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes] In Appendices.(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] All results have conﬁdence intervals.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] In Appendices.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In Sec. 4.1. (b) Did you mention the license of the assets? [Yes] In Appendices. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Website for qualitative results.
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]