Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs
Abstract
A core goal in systems neuroscience and neuroethology is to understand howneural circuits generate naturalistic behavior.
One foundational idea is that complexnaturalistic behavior may be composed of sequences of stereotyped behavioralsyllables, which combine to generate rich sequences of actions.
To investigate this,a common approach is to use autoregressive hidden Markov models (ARHMMs) tosegment video into discrete behavioral syllables.
While these approaches have beensuccessful in extracting syllables that are interpretable, they fail to account for otherforms of behavioral variability, such as differences in speed, which may be betterdescribed as continuous in nature.
To overcome these limitations, we introduce aclass of warped ARHMMs (WARHMM).
As is the case in the ARHMM, behavioris modeled as a mixture of autoregressive dynamics.
However, the dynamics undereach discrete latent state (i.e. each behavioral syllable) are additionally modulatedby a continuous latent “warping variable.”
We present two versions of warped
ARHMM in which the warping variable affects the dynamics of each syllable eitherlinearly or nonlinearly.
Using depth-camera recordings of freely moving mice, wedemonstrate that the failure of ARHMMs to account for continuous behavioralvariability results in duplicate cluster assignments.
WARHMM achieves similarperformance to the standard ARHMM while using fewer behavioral syllables.
Further analysis of behavioral measurements in mice demonstrates that WARHMMidentiﬁes structure relating to response vigor.
Introduction
A fundamental question in systems neuroscience is how neural activity generates complex behavior [1–3].
Speciﬁcally, a key goal is to understand how changes in neural activity determine which actionsare selected or executed on a moment-by-moment basis.
To make progress towards this goal, it isessential to study ethologically relevant, naturalistic behavior.
A common way of studying naturalisticbehavior is to observe animals as they freely explore an environment [4–6].
Such unconstrained,spontaneous behavior offers a rich setting for studying behavioral variability.
However, the hours ofraw video data required to capture naturalistic behavior in detail are high-dimensional and difﬁcultto use for follow-up analyses.
Thus, a key goal in behavioral neuroscience research is to developdata-analysis strategies that can extract interpretable lower-dimensional summaries of behavior[4, 5, 7, 7–10].
Ultimately, such descriptions will facilitate relating complex naturalistic behaviors tothe underlying neural activity patterns that generate them [3, 11].
This will provide insight into howand why behavior may differ across environmental contexts [2, 3, 11, 12], under pharmacologicalmanipulations [6] or across health and disease [13, 14].
One hypothesis is that the brain generates complex behaviors by concatenating a series of simpler,stereotyped actions [4, 5, 11, 15, 16].
Just like syllables form the building blocks of spoken language,
[Figure]
behavioral syllables may be composed to perform complex sequences of behavior.
A large focus ofprevious work has been to discover such behavioral syllables in an unsupervised manner, therebyobtaining a low-dimensional description of high-dimensional behavior [4, 5, 12, 16–19].
Autoregressive Hidden Markov Models (ARHMMs) are well-suited to this task [5, 6].
For example,
Wiltschko et al.
[5] used depth video to capture the posture of freely moving mice (Fig.
1A).
In thiswork, authors projected the video frames onto the top D principal components (PCs) to obtain a
D -dimensional time series of behavior.
The authors then used an ARHMM to segment the behavioraltime series into discrete syllables (Fig. 1B).
Each discrete syllable corresponds to vector autoregressivedynamics in PC space, which can be conceptualized as a vector ﬁeld (Fig.
1C).
Each instance ofa syllable corresponds to a short, stereotyped trajectory following this vector ﬁeld (red-to-bluetrajectories in Fig.
1C).
Empirically, these often correspond to stereotyped patterns of movement likerearing, darting, or grooming (Fig.
1D).
However, current modeling approaches have focused on clustering discrete behavioral syllables whilefailing to account for other, continuous forms of behavioral variability.
For example, the same type ofbehavior (e.g. a dart) could be performed more or less vigorously.
While these two actions mightbe identiﬁed by a human observer as the same behavior under different speeds (fast vs. slow darts),current models lack the ability to allow for such structured continuous variability within states, andthus might assign these actions to entirely separate behavioral states.
As a result, current approachesoften over-segment video data, by allocating distinct clusters to the same movement type.
Thedata might be better described by a model which incorporates a continuous spectrum of structuredvariability within a speciﬁc behavior.
We extend the ARHMM by incorporating a latent warping variable .
With this Warped Autoregressive
Hidden Markov Model (WARHMM), we are able to capture continuous variability within a discretesyllable.
Thus, we are able to disentangle variability due to (discrete) movement type from otherforms of (continuous) variability, such as movement speed.
We consider two types of warping:time-warping , which captures changes in the speed of evolution of the autoregressive dynamics,and Gaussian process warping , which allows for nonlinear changes in dynamics.
We develop anefﬁcient inference algorithm for the WARHMM and show that it can correctly identify the underlyingstates and parameters in simulated data.
Then, using behavioral measurements of freely behavingmice, we demonstrate that accounting for continuous sources of behavioral variability can resolveissues of over-segmentation commonly observed using previous approaches.
Furthermore, usingbehavioral data of mice treated with either saline solution or amphetamine, we demonstrate that the
WARHMM identiﬁes differences in the distribution of latent warping variables across both groups ofmice.
This result reﬂects potential modulations of movement vigor and speed distributions due to thepharmacological intervention.
Background
We ﬁrst review related work and then present a brief description of two classes of Hidden Markov
Models that have been used for time series segmentation and behavioral modeling.
The key featuresof each model form the basis for our warped extension to the ARHMM model, which represents themain contribution of our work.
Related work in unsupervised behavioral segmentation.
We build upon a rich body of work inthe unsupervised behavioral segmentation space.
Our model is inspired by the ARHMM approach tounsupervised behavioral segmentation (MoSeq) proposed by Wiltschko et al.
[5], which is describedin the introduction.
Berman et al.
[4] addressed a similar task in fruit ﬂy video using MotionMapper,which identiﬁes discrete behavioral states as peaks in a non-linear, two-dimensional embedding ofpostural spectrograms.
Hsu and Yttri [17] have proposed B-SOiD, which uses a random forest toclassify non-linear postural feature embedding clusters into multiple behavioral classes.
Harris et al.[18] ﬁt an autoregressive linear model to time-windowed postural features using low-rank tensordecomposition, and interpret clusters in the ﬁt model parameters as discrete behavioral states.
Luxemet al.
[19] identify behavioral states by clustering the latent vectors produced by training a variationalautoencoder on input from markerless pose estimation [9].
For a more thorough treatment of workin unsupervised behavioral quantiﬁcation, see McCullough and Goodhill [20] for a recent review.
The majority of these related approaches have focused on segmenting behavioral video based onmovement type or on tracking animal pose.
The model we introduce in section 3 extends on this workby explicitly taking other forms of behavioral variability, such as movement speed, into account.
Autoregressive Hidden Markov Models.
An ARHMM (Fig.
2A, top) consists of a discrete latentstate variable z t ∈ { 1 , 2 , ..., K } and observations x t ∈ R D .
Transitions between the discrete statevalues over time are governed by a transition matrix P , where the entry P k,k (cid:48) indicates the probabilityof advancing from state z t =
k to state z t
+1 = k (cid:48) .
Given the discrete state value, the observationsare modeled to evolve according to linear dynamics, where a dynamics matrix A z t and bias term b z tdetermine the mapping from x t to x t +1 in the presence of Gaussian noise with covariance matrix
Q z t .
Thus, the model represents an extension of classic Gaussian mixture models to time series withautoregressive dynamics.
The ARHMM model can be summarized as
[Equation]
In the context of behavioral modeling [5], the discrete latent state reﬂects the behavioral syllable(such as a dart, leftward turn, rear, etc.) and determines which dynamics are used to describe thetemporal evolution of the observed posture x t .
Factorial Hidden Markov Models.
In Factorial Hidden Markov Models (FHMMs) [21], multiplediscrete hidden state variables inﬂuence the distribution of observed states.
The FHMM model canbe summarized as
[Equation]
FHMMs are useful in cases when aspects of data can be distributed across multiple states.
Forexample, to ﬁt data that varies according to M binary variables, a standard HMM would need 2 Mdiscrete states, while a FHMM would only need M binary states [21].
In the context of behavior, adding additional latent variables is an appealing way to provide structuredvariability within discrete syllables.
Assuming this variability is included in a reasonable way,
an
FHMM has the potential to describe a dataset with fewer discrete latent states than an HMM.
Warped Autoregressive Hidden Markov Models
ARHMMs have been successful in clustering video measurements of behavior into discrete sets ofbehavioral syllables.
However, in practice they are prone to over-segmentation , where behaviors thatappear similar to a human expert are split into distinct clusters.
We hypothesize that oversegmentation discretestates
ARHMM
[Figure]
T i m e - W a r ped
A RH
MM
A.
B.
G PW a r ped
A RH
MM arises because the ARHMM conﬂates discrete sources of behavioral variability — the expression of distinct behavioral syllables with continuous sources of variability that cannot be captured bylinear autoregressive dynamics with Gaussian noise.
To address this limitation, we develop Warped Autoregressive Hidden Markov Models (WARHMMs).
WARHMMs extend the ARHMM with an additional latent variable, like in a Factorial HMM.
Inaddition to the discrete state variable z t , WARHMM includes a latent warping state τ t ∈ [ − 1 , 1] at each time step (Fig.
2A, bottom).
The state the with
While z t can model rapid switches in dynamics, τ t can account for additional variability in the dynamics associated with a given latent state.
The general form of this model class can be summarized as follows, where A k ( τ t ) , b k ( τ t ) , and Q k ( τ t )are functions of τ t :
[Equation]
Below, we consider two possibilities for how exactly the warping variable modulates the dynamics.
3.1 Time-Warped ARHMM: linear modulation of autoregressive dynamics
Our ﬁrst speciﬁc instance of WARHMM has a direct motivation in terms of time-warping, and wethus refer to it as a time-warped ARHMM (T-WARHMM).
Speciﬁcally, we aim to capture continuouschanges in how quickly trajectories move through observation space.
If we consider the change incurrent state ∆ x t = x t +1 − x t due to the dynamics within a given state z t = k and let τ be a logstep-size parameter , we can write
[Equation]
When τ =
the step size C τ is always one, so equation (5) is equivalent to the classic ARHMM.
For nonzero τ , however, the warping variable modulates how far a single update can move the state,akin to a time-constant in ordinary differential equations.
The constant C determines the maximummultiplicative factor by which dynamics can be scaled; in our experiments we set C = 2 .
In terms ofthe functional mapping, T-WARHMM corresponds to the following mapping between the warpingstate τ t and the parameters determining the autoregressive dynamics of the observed states x t :
[Equation] [Equation]
Fig.
2B (top) shows how an example syllable’s dynamics are sped up or slowed down by changing τ .
As τ increases, the trajectories traverse a greater distance in the same number of time steps.
In the context of behavioral modeling, the time-warping variable may capture modulations in movementspeed.
As we will see in section 5, the inferred warping variables correlate with other intuitive notionsof vigor, like centroid velocity in darting syllables, while also offering a quantiﬁcation of vigor instationary syllables such as grooming.
3.2 Gaussian Process-WARHMM: Nonlinear modulation of autoregressive dynamics
The model formulation for T-WARHMM is interpetable and intuitive, but also makes strong parametric assumptions about how continuous variability affects the dynamics of behavior.
As a point ofcomparison, we consider a more ﬂexible model, in which the inﬂuence of τ t via A z t ( τ t ) and b z t ( τ t )is speciﬁed nonparametrically in terms of Gaussian processes.
This Gaussian Process WARHMM(GP-WARHMM) moves beyond using τ t to describe speed or vigor, as τ t could have effects beyondmodulation of time-constants of the dynamics.
Here τ t can modulate entries of the dynamics matrixaccording to a smooth nonparametric function.
In particular, the dynamics are modeled as follows:
[Equation]
(7) where A kij ( τ ) denotes the ( i, j ) -th entry of the matrix A k ( τ ) and b ki ( τ ) is the i -th entry of thevector b k ( τ ) , each modeled as a Gaussian Process with mean zero and covariance function K θ ( τ, τ (cid:48) ) .
Thus, each coordinate varies smoothly and independently as a function of τ , a priori .
The prior mean
A kij = b ki = 0 corresponds to a pause with the state x t remaining stationary in expectation.
In ourexperiments, we choose a squared exponential kernel with kernel hyperparameters θ = ( ρ, σ ) .
[Equation]
Fig.
2B (bottom) shows how an example syllable’s dynamics are modulated by changing τ in a
GP-WARHMM.
The central dynamics ( A k ( τ = 0) ) are the same as in the T-WARHMM above, butnow τ can have nonlinear effects on the dynamics that do more than simply slow them down orspeed them up.
This extra ﬂexibility could be useful for capturing more general types of continuousvariability within discrete syllables, but it may also come at the cost of interpretability.
3.3 Inference and Learning
Though we have presented τ t ∈ [ − 1 , 1] as a continuous latent variable, in practice we ﬁnd it sufﬁcientto discretize τ t over a ﬁne grid of J evenly spaced points, since it is a one-dimensional, boundedrandom variable.
Once discretized, we model the warping variable dynamics as a Markov processwith a banded transition matrix to encourage small, local changes over time.
Similarly, the GPprior in eq.
(7) reduces to a multivariate normal after discretization.
Full details are provided in the
Supplementary Material.
As in standard ARHMMs, we estimate the states and parameters with the Expectation-Maximizationalgorithm (EM) [22].
We perform exact inference over the discrete syllables and warping variablesusing the forward-backward algorithm, which runs in O ( T ( K 2 + J 2 )) time and uses O ( TKJ )memory.
For both the time-warped and GP-warped ARHMMs, the parameter updates have closedform solutions.
For large datasets, we used stochastic EM [23] to speed convergence.
For the
GP-WARHMM, we learned the kernel hyperparameters through optimization of the variational lowerbound.
Again, complete details are in the Supplementary Material.
Synthetic data validation
We begin by generating two-dimensional synthetic data x t ∈ R 2 with K
= 2 discrete states.
Theautoregressive dynamics under each discrete state are chosen to be a clockwise and counterclockwiserotation.
We modulated the dynamics with a time-warped ARHMM, setting A k ( τ ) = 2 τ A k , asillustrated in Fig.
3A. For this simple experiment, we limited the true time-warping variables to
J = 5 values of τ evenly spaced on [ − 1 , 1] .
A snippet of simulated data and the corresponding timeconstants and states is shown in Fig.
3B. We ﬁt a GP-WARHMM and T-WARHMM with K = {+
+}
z = 2 τ = 0 .
75 2 τ = 1 .
2 τ = 1 .z =
A. z =2 τ = 1 .xx
B. z =
z =t r u e t w gp
A z ( τ ) for increasing τ 0.
C. t r u e t w gp     time time
D.
E. z =z =
The speed of the rotation is modulated via different values of τ , where A z ( τ ) =
2 τ
A z
Illustration of one instance of the 2-dimensional state trajectory ( x 1 ( t ) , x 2 ( t )) as the generative dynamics are modulated by both state switches in z (discrete variability, blue versus pink) and changes in rotation angle via τ (continuous variability, shading of color
A =
A to variations the autoregressive learns the structure this example, but the increased
GP makes it harder for it to and continuous contributions to states and J = 5 warping variables to data simulated from this model, and additionally compareresults in terms of log-likelihood performance to a classic ARHMM with K = 2 , 4 , 10 discrete states.
The results from this experiment are summarized in Fig.
3C-E and Table 1. Panel C shows thetrue and learned values of entries of the autoregressive dynamics matrix A z ( τ ) as both z and τvary.
We see that both the T-WARHMM and GP-WARHMM learn dynamics that reﬂect the overallrotational structure of the matrix.
However, only T-WARHMM is able to recover the clockwise andcounterclockwise variation with z and correct speed modulation with τ .
[Table]
The increased ﬂexibility of GP-WARHMM, allowsit to achieve a test log likelihood which outperformsthe K = 2 , 4 state ARHMMs (Table 1), withoutcorrectly disentangling discrete and continuous contributions to the observed data variability.
Fig.
3Dand E show the true state changes in z t and τ t , together with the mode of the inferred posterior distributions under each model.
T-WARHMM is able torecover the correct segmentation of the data, while GP-WARHMM’s increased ﬂexibility allows it toﬁt the data, but in a less interpretable way.
Modeling depth video of freely behaving mice
For the remainder of the paper, we are interested in extracting discrete and continuous structure inbehavior from posture data x t extracted from depth-imaging recordings of freely behaving mice.
Todo this, we reanalyze data from Wiltschko et al.
[5], which represents the original application of the
ARHMM to clustering mouse behavior.
In the context of this [Figure]
In the MoSeq dataset, the observations x t are taken to be the ﬁrst 10 principal components of depthcamera video data of mice exploring an open ﬁeld.
The dataset consists of 20-minute depth camerarecordings of 24 mice.
In preprocessing, the videos are cropped and centered around the mousecentroid, and then ﬁltered to remove recording artifacts.
Finally, the preprocessed video is projectedonto the top principal components to obtain a 10-dimensional time series.
5.1 Comparison of model performances on the MoSeq dataset
To provide a direct comparison of performance between the ARHMM, T-WARHMM, and GPWARHMM, we trained each model using 50 epochs of stochastic EM.
The ARHMM is equivalent tosetting the number of τ -values in T-WARHMM to J = 1 .
T-WARHMM and GP-WARHMM eachhad J = 31 evenly spaced values of τ on the interval [ − 1 , 1] .
Fig.
4A shows the log-likelihood of each model on held-out test data for a range of K values.
For a given value of K , both T-WARHMM and GP-WARHMM outperform the classic ARHMMin terms of their generalization performance on unseen data.
However, we achieve similar testlog-likelihoods for an ARHMM with K = 40 syllables, a T-WARHMM with K = 25 syllables, anda GP-WARHMM with K = 10 syllables.
This further illustrates that the ARHMM has to account forcontinuous variability by creating a larger number of discrete states, while the factorial structure of the
WARHMMs enables syllables to be merged when the difference in their dynamics can be explainedby changes in τ .
The GP-WARHMM outperforms the T-WARHMM in terms of test log-likelihood,which can be attributed to the fact that the GP-WARHMM is more ﬂexible than the T-WARHMMand can capture additional structure in the data.
However, the GP-WARHMM accomplishes this atthe cost of interpretability.
The T-WARHMM warping constant is physiologically motivated and, aswe will show in the later results, has direct correlations to centroid velocity and other notions of vigor.
While the GP-WARHMM warping variable is allowed to vary the syllable in whichever way willincrease data log-likelihood, the T-WARHMM is restricted to modulating syllables in a way that isbiologically interpretable.
We believe that this interpretability of T-WARHMM makes it more usefulfor behavioral analysis purposes and thus focus on the utility of T-WARHMM in the remainder.
5.2 Interpreting the time-warping variable
Which syllables are “warped”?
To begin our analysis of the T-WARHMM results, we ﬁt a modelwith K = 20 discrete states and J = 31 time constants.
As shown in Fig.
4A, this model attainssimilar test log-likelihood to an ARHMM with K = 35 discrete states.
We then determine whichtypes of syllables are over-represented as multiple distinct discrete states in the ARHMM by analyzingthe syllable labels from each model. After generating sample videos of behavior from both models, [Figure]
C en t r o i d v e l o c i t y we asked expert neuroethologists and MoSeq users to label the syllables produced.
We then gatheredthe labels under three general behaviors: darts, grooms/pauses, and rears.
The results are shownin Fig.
4B. Both models have similar numbers of darting states, while the numbers of grooms andrears are reduced in T-WARHMM, with T-WARHMM halving the number of rear states.
While
T-WARHMM does not completely resolve the oversegmentation issue, it is able to perform as wellas the ARHMM with fewer syllables.
More details and video examples of behaviors identiﬁed by
T-WARHMM are provided in the Supplementary Material.
Connections to centroid velocity.
From our formulation of the time warping variable, it is expectedthat the time warping variable and centroid velocity of the mouse would be highly correlated.
In Fig.5 we plot measured centroid velocity (in pixels/ms) vs. inferred vigor ( 2 τ ) for four representativestates.
We see a strong relationship between centroid velocity and vigor for the darting state, wherewe would expect such a relationship to occur.
The other states also show this relationship, butless consistently.
In these states, the warping variable may be accounting for additional forms oftiming-related variability that are not well accounted for by centroid speed.
These analyses validatethat the time warping parameter τ is able to extract speed-related information also contained incentroid velocity for behavioral syllables such as darting motions.
5.3 Identifying drug-induced changes in behavior
Finally, we analyze video recordings of two groups of mice treated with either saline solution oramphetamine.
As shown in Fig.
6A, the distribution of centroid speed varies slightly between thegroups, with the amphetamine-treated mice performing actions at higher speeds more frequently thanthe saline mice.
We ﬁt a T-WARHMM with J = 31 time constants and K = 20 syllables to bothsets of data.
Fig.
6B shows the the inferred vigor ( 2 τ ) distributions from two representative syllables,demonstrating that there is a clear rightward shift between amphetamine and saline treated mice.
Thisindicates that amphetamine mice use faster time warping variables more frequently than saline mice.
The difference between means of the τ index distributions are shown in Fig.
6C. Our model showsa signiﬁcant difference ( p < 0.05, independent t-test) between the means of the τ distributions forall except one of the inferred syllables.
Thus, T-WARHMM is able to detect and dynamically trackdrug-induced differences in behavior across both groups of mice.
Conclusion
We have introduced an approach for unsupervised behavioral modeling which allows for the identiﬁcation of continuous variability within discrete behavioral syllables.
Our work builds on a priorstate-of-the-art technique, which uses a discrete hidden state in combination with autoregressivelinear dynamics to cluster behavior.
In our proposed model extension (WARHMM), underlying thebehavioral measurement at each time point are both a discrete state variable and a warping variable.
Both of these latent variables modulate the dynamics of the syllable, either through discrete switches(via the discrete state variable) or through more continuous forms of modulation (via the warpingvariable).
We found that WARHMM achieved similar performance in terms of log-likelihood on heldout test data while utilizing fewer discrete latent states than the ARHMM. [Figure]
Our analyses on mouse behavioral data focused on an interpretable subclass of warped ARHMM,where the warping variable interacts linearly with the autoregressive dynamics of the observeddata (the T-WARHMM).
Notably, the warping variable in this simple model can be viewed asimplementing a local rescaling of time.
Similar time warping models have been fruitfully appliedto trial-structured neural data [24, 25]
[Figure]
and our work illustrates how these ideas can be extendedto naturalistic time series data that lack repeated trial-structure.
Our work on mouse behavioraldata illustrates that continuous variability in behavior may be induced through pharamacologicalinterventions and can be extracted directly from video recordings.
Limitations.
For simplicity, we discretized τ on a ﬁne grid instead of explicitly performing inference over the continuous variable.
This was done in favor of computational efﬁciency and easeof inference.
While we believe the ﬁne grid we’ve used for τ offers a good approximation of acontinuous function, future work could capture continuous variability by including a truly continuousposterior, e.g. via a Gaussian Process [24].
Furthermore, our model assumes that the behavioral
PCs evolve according to linear dynamics, and though the GP-WARHMM allows these dynamics tobe modulated in nonlinear ways, the basic assumption could be relaxed to more general nonlineardynamics in future work.
Future work.
This work establishes multiple directions for future research.
The ﬁrst involvesfurther extending the ARHMM model by adding in additional forms of structured, interpretablevariability.
What aspects of behavior, in addition to speed, may vary within syllables?
In the context ofmouse behavior, variables such as turning radius or depth-camera pixel height may provide additionalcontinuous axes along which behaviors can be modulated.
Another future direction lies in using
T-WARHMM to analyze neural correlates of behavior.
Similar behavioral segmentation approacheshave been used to simultaneously analyze the evolution of behavioral and neural recordings [8, 26–28].
A future research area of particular interest lies in understanding how the dorsolateral striatumencodes action selection and vigor [13, 29–32].
Neural activity in this midbrain nucleus is known to bedisrupted in motor diseases like Parkinson’s and Huntington’s
The T-WARHMM approachallows us to simultaneously extract descriptions of movement type (how frequently actions areselected) and movement speed (how vigorously actions are performed) from behavioral measurements.
We expect that the ability to disentangle the contributions of these two factors could provide animportant basis for better understanding how variations in neural activity determine changes in actionselection and vigor.
Finally, the models we have proposed here could be applied more widely toanalyze other types of behavioral data. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] The limitations section iscontained in the discussion, on page 9.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Discussedin Supplemental Material
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Discussed in
Supplemental Material
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Discussed in Supplemental Material
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Error bars are reported Fig.
5, and p-values are reported
in section 5.3.
Due to computational strain we did not report error bars for Fig.
4A.
(d) Did you include the total amount of compute and the type of resources used (e
g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] Discussed in Supplemental
Material
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Our work uses an
existing dataset from [5].
(b) Did you mention the license of the assets?
[N/
A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]