Sparsity in Continuous-Depth Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural Ordinary Differential Equations (NODEs) have proven successful in learning dynamical systems in terms of accurately recovering the observed trajectories.
While different types of sparsity have been proposed to improve robustness, thegeneralization properties of NODEs for dynamical systems beyond the observeddata are underexplored.
We systematically study the influence of weight and feature sparsity on forecasting as well as on identifying the underlying dynamicallaws.
Besides assessing existing methods, we propose a regularization technique tosparsify “input-output connections” and extract relevant features during training.
Moreover, we curate real-world datasets consisting of human motion capture andhuman hematopoiesis single-cell RNA-seq data to realistically analyze differentlevels of out-of-distribution (OOD) generalization in forecasting and dynamicsidentification respectively.
Our extensive empirical evaluation on these challengingbenchmarks suggests that weight sparsity improves generalization in the presenceof noise or irregular sampling.
However, it does not prevent learning spuriousfeature dependencies in the inferred dynamics, rendering them impractical for predictions under interventions, or for inferring the true underlying dynamics.
Instead,feature sparsity can indeed help with recovering sparse ground-truth dynamicscompared to unregularized NODEs.
Introduction
Extreme over-parameterization has shown to be part of the success story of deep neural networks(Zhang et al., 2016; Allen-Zhu et al., 2018; Li & Liang, 2018).
This has been linked to evidencethat over-parameterized models are easier to train, perhaps because they develop “convexity-like”properties that help convergence of gradient descent (Brutzkus et al., 2017; Du et al., 2018).
However,over-parameterization comes at the cost of additional computational footprint during training andinference (Denil et al., 2013).
Therefore, motivation for sparse neural networks are manifold andinclude: (i) computational efficiency (speed up, memory reduction, scalability) (Hoefler et al., 2021),(ii) imitation of human learning, where neuron activity is typically sparse (Ahmad & Hawkins, 2016),as well as (iii) interpretability (Xu et al., 2022), and (iv) avoiding overfitting or improving robustness(Bartoldson et al., 2020).
Enforcing sparsity in model weights, also called model pruning, and its effects for standard predictivemodelling tasks have been extensively studied in the literature (Hoefler et al., 2021).
However, sparsityin continuous-depth neural nets for modeling dynamical systems and its effects on generalizationproperties is still underexplored.
Neural Ordinary Differential Equations (NODEs) (Chen et al.
, 2018)have been introduced as the limit of taking the number of layers in residual neural networks to infinity,resulting in continuous-depth (or continuous-time) neural nets.
These networks have shown greatpromise for modeling noisy and irregular time-series.
In this case, the trained NODE model mightnot represent just some dynamics serving the predictive task (e.g., regression or classification via afinal linear layer).
It can rather be thought of as learning the underlying dynamics (i.e., the ODE)that actually govern(s) the evolution of the observed trajectory.
In the prior scenario (predictive performance), a large body of literature has focused on regularizing the number of model evaluationsrequired for a single step of the ODE solver for improving the efficiency Finlay et al.
Kellyet al.
Ghosh et al.
; Pal et al.
(2021); Grathwohl et al.
These regularizationtechniques rely on learning one out of many possible equivalent dynamical systems giving rise to thesame ultimate performance, but are easier and faster to solve.
Crucially, these regularizers do notexplicitly target sparsity in the model weights or in the features used by the model.
[Figure]
In the latter scenario (inferring dynamical laws), we assume the existence of a ground truth ODE that governsthe dynamics of the observed trajectories.
Here, two typesof sparsity may be of interest.
First, motivated by standard network pruning, sparsity in model weights ( weightsparsity or model sparsity ) can reduce computational requirements for inference (Liebenwein et al., 2021), see
Figure 1(top).
Second, it has been argued that featuresparsity , i.e., reducing the number of inputs a given outputof the NODE relies on, can improve identifiability of theunderlying dynamical law (Aliee et al., 2021; Bellot et al.
,2021; Liu et al., 2022), see Figure 1(bottom).
The motivation for feature sparsity often relies on an argument fromcausality, where it is common to assume sparse and modular causal dependencies between variables (Schölkopfet al., 2021; Zheng et al., 2019).
In this paper, we empirically assess the impact of varioustypes of sparsity-enforcing methods on the performance of
NODEs on different types of OOD generalization for bothprediction and for inferring dynamical laws on real-worlddatasets.
While most existing methods target weight sparsity, we highlight that feature sparsity canimprove interpretability and dynamical system inference by proposing a new regularization techniquefor continuous-time models.
Specifically, our main contributions are the following:
• We propose PathReg, a differentiable L0-based regularizer that enforces sparsity of “input-outputpaths” and leads to both feature and weight sparsity.•
We extend L0 regularization (Louizos et al., 2018) and LassoNet (Lemhadri et al., 2021) to NODEsand perform extensive comparisons with existing models including vanilla NODE (Chen et al.,2018), C-NODE (Aliee et al., 2021), GroupLasso for NODEs (Bellot et al., 2021), as well as ourown PathReg.• We define a useful metric for evaluating feature sparsity, demonstrate the differences betweenweight and feature sparsity, and explore whether and when one implies the other.• To cover at least part of the immense diversity of time-series problems, each of which implyingdifferent types of OOD challenges, we curate large, real-world datasets consisting of human motioncapture ( mocap.cs.cmu.edu ) as well as human hematopoiesis single-cell RNA-seq (Lueckenet al., 2021) data for our empirical evaluations.
We will release these datasets openly and hope they benefit further research on inferring dynamical laws from observed trajectories.
Background2.1 Continuous-depth neural nets
Among the plethora of deep learning based methods to estimate dynamical systems from data (Lim &
Zohren, 2021), we focus on Neural Ordinary Differential Equations (NODEs).
In NODEs, a neuralnetwork with parameters θ is used to learn the function f θ ≈ f from data, where f defines a (firstorder) ODE in its explicit representation ˙
X = f
( X, t ) (Chen et al., 2018).
Starting from the initialobservation X ( a ) at some time t = a , an explicit iterative ODE solver is deployed to predict X ( t )for t ∈ ( a, b ] using the current derivative estimates from f θ .
The parameters θ are then updatedvia backpropagation on the mean squared error (MSE) between predictions and observations.
Asdiscussed extensively in the literature, NODEs can outperform traditional ODE parameter inferencetechniques in terms of reconstruction error, especially for non-linear dynamics f
(Chen et al., 2018;
Dupont et al., 2019).
In particular, one advantage of NODEs over previous methods for inferringnon-linear dynamics such as SINDy (Brunton et al., 2016) is that no dictionary of non-linear basisfunctions has to be pre-specified.
A variant of NODEs for second order systems called SONODE exploits the common reduction ofhigher-order ODEs to first order systems (Norcliffe et al., 2020).
The second required initial condition,the initial velocity ˙ X ( a ) , is simply learned from X ( a ) via another neural network in an end-to-endfashion.
Our experiments build on the public NODE and SONODE implementations.
However, ourframework is readily applicable to most other continuous-depth neural nets including Augmented
NODE (Dupont et al., 2019), latent NODEs (Rubanova et al., 2019), and neural stochastic DEs (Liet al., 2020; Oganesyan et al., 2020).
2.2 Sparsity and generalization
Two prominent motivations for enforcing sparsity in over-parameterized deep learning models are(a) pruning large networks for efficiency (speed, memory), (b) as a regularizer that can improveinterpretability or prevent overfitting.
In both cases, one aims at preserving (as much as possible)i.i.d.
generalization performance, i.e., performance on unseen data from the same distribution as thetraining data, compared to a non-sparse model (Hoefler et al., 2021).
Another line of research has explored the link between generalizability of neural nets and causallearning (Schölkopf et al., 2021), where generalization outside the i.i.d.
setting, is conjectured torequire an underlying causal model .
Deducing true laws of nature purely from observational datacould be considered an instance of inferring a causal model of the world.
Learning the correct causalmodel enables accurate predictions not only on the observed data (next observation), but also underdistribution shifts for example under interventions (Schölkopf et al., 2021).
A common assumption incausal modeling is that each variable depends on (or is a function of) few other variables (Schölkopfet al., 2021; Zheng et al., 2019; Ng et al., 2020; Bolstad et al., 2011).
In the ODE context, we interpretthe variables (or their derivatives) that enter a specific component
f i as causal parents, such that wecan write ˙
X i = f ( pa ( X i ) , t ) (Mooij et al., 2013).
Thus, feature sparsity translates to each variable
X i having only few parents pa ( X i ) , which can also be interpreted as asking for “simple” dynamics.
Since weight sparsity as well as regularizing the number of function evaluations in NODEs can alsobe considered to bias them towards “simpler dynamics”, these notions are not strictly disjoint, raisingthe question whether one implies the other.
In terms of feature sparsity, Aliee et al. (2021); Bellot et al.
(2021) study system identification andcausal structure inference from time-series data using NODEs.
They suggest that enforcing sparsityin the number of causal interactions improves parameter estimation as well as predictions underinterventions.
Let us write out f θ as a fully connected net with L hidden layers parameterized byθ :
=
( W l , b l ) L +1 l =1 as
[Equation]
with element-wise activation function σ , l -th layer weights W l , and biases b l .
Aliee et al.
(2021) thenseek to reduce the overall number of parents of all variables by attempting to cancel all contributionsof a given input on a given output node through the neural net.
In a linear setting, where σ ( x ) = x ,the regularization term is defined by
[Equation]
where A ij = 0 if and only if the i -th output is constant in the j -th input.
In the non-linear setting, forcertain σ ( x ) ̸
= x , the regularizer ∥
A ∥ 1 , 1 = ∥| W L +1 | · · · | W 1 |∥ 1 , 1 (with entry wise absolute valueson all W l ) is an upper bound on the number of input-output dependencies, i.e., for each output isumming up all the inputs j it is not constant in.
Regularizing input gradients (Ross et al., 2017; Ross& Doshi-Velez, 2017; Slavin, 2018) is another alternative to train neural networks that depend onfewer inputs, however it is not scalable to high-dimensional regression tasks.
Bellot et al.
(2021) instead train a separate neural net f
θ
i : R n → R for each variable X
i and penalize
NODEs using GroupLasso on the inputs via
[Equation]
where W 1 i is the weight matrix in the input layer of f i and [ W 1 i ] · ,k refers to the k th column of
W 1 i that should simultaneously (as a group) be set to zero or not.
While enforcing strict featuresparsity (instead of regularizing an upper bound), parallel training of multiple NODE networks can becomputationally expensive (Figure 1, bottom).
While the findings of both works suggest that sparsityof causal interactions helps system identification, their empirical evaluation predominantly focuseson synthetic data settings, leaving performance on real data underexplored.
Another recent work suggests that standard weight or neuron pruning improves generalization for
NODEs (Liebenwein et al., 2021).
The authors show that pruning lowers empirical risk in densityestimation tasks and decreases Hessian’s eigenvalues, thus obtaining better generalization (flatminima).
However, the effect of sparsity on identifying the underlying dynamics as well as thegeneralization properties of NODEs to forecast future values are not assessed.
Sparsification of neural ODEs
In an attempt to combine the strengths of both weight and feature sparsity, we propose a newregularization technique, called PathReg, which compares favorably to both C-NODE (Aliee et al.,2021) and GroupLasso (Bellot et al., 2021).
Before introducing PathReg, we describe how to extendexisting methods to NODEs for an exhaustive empirical evaluation.
3.1 Methods
L0 regularization.
Inspired by (Louizos et al., 2018), we use a differentiable L0 norm regularizationmethod that can be incorporated in the objective function and optimized via stochastic gradientdescent.
The L0 regularizer prunes the network during training by encouraging weights to get exactlyzero using a set of non-negative stochastic gates z .
For an efficient gradient-based optimization,
Louizos et al.
(2018) propose to use a continuous random variable s with distribution q ( s ) andparameters ϕ , where z is then given by
[Equation]
Gate z is a hard-sigmoid rectification of s 2 that allows the gate to be exactly zero.
The regularizationterm is then defined as the probability of s being positive
[Equation]
where Q is the cumulative distribution function of s .
Minimizing the regularizer pushes many gatesto be zero, which implies weight sparsity as these gates are multiplied with model weights.
L0regularization can be added directly to the NODE network f θ .
LassoNet is a feature selection method (Lemhadri et al., 2021) that uses an input-to-output skip(residual) connection that allows a feature to participate in a hidden unit only if its skip connection isactive.
LassoNet can be thought of as a residual feed-forward neural net Y = S T X
+ h θ ( X ) , whereh θ denotes another feed-forward network with parameters θ , S T ∈ R n × n refers to the weights inthe residual layer, and Y are the responses.
To enforce feature sparsity, L1 regularization is appliedto the weights of the skip connection, defined as ||S|| 1 , 1 (where ∥ · ∥ 1 , 1 denotes the element-wise L1 norm).
A constraint term with factor ρ is then added to the objective to budget the non-linearityinvolved for feature k relative to the importance of X k
[Equation]
subject
[Equation]
where [ W 1 ] · ,k denotes the k th column of the first layer weights of h θ , and [ S ] · ,k represents the k thcolumn of S .
When ρ = 0 , only the skip connection remains (standard linear Lasso), while ρ → ∞corresponds to unregularized network training.
LassoNet regularization can be extended to NODEs by adding the skip connection either before orafter integration (the ODE solver).
If added before integration, which we call Inner LassoNet, a linearfunction of X is added to its time derivative
[Equation]
Adding the skip connection after integration (and the predictor o), called Outer LassoNet, yields
[Equation]
ODESolver [Equation] [Equation]
PathReg (ours).
While L0 regularization minimizes the number of non-zero weights in a network, itdoes not necessarily lead to feature sparsity meaning that the response variables can still depend onall features including spurious ones.
To constrain the number of input-output paths , i.e., to enforcefeature-sparsity, we regularize the probability of any path throughout the entire network contributingto an output.
To this end, we use non-negative stochastic gates z
= g ( s ) similar to Eq.
4, where theprobability of an input-output path P being non-zero is given by
[Equation]
and we constrain
[Equation]
to minimize the number of paths that yield non-zero contributions from inputs to outputs.
This isequivalent to regularizing the gate adjacency matrix A z = G L +1 · . . .
· G 1 .
Where G l is a probabilitymatrix corresponding to the probability of the l -th layer gates being positive.
Then, A z ij representsthe sum of the probabilities of all paths between the i -th input and the j -th output.
Ultimately, wethus obtain our PathReg regularization term
[Equation]
G
[Equation]
q
[Equation]
Regularizing ∥
A z ∥ 1 , 1 , minimizes the number of paths between inputs and outputs and induces noshrinkage on the actual values of the weights.
Therefore, we can utilize other regularizers on θ suchas ∥ A ∥ 1 , 1 in conjunction with PathReg similar to Eq. 2.
In this work, we consider the followingoverall loss function
[Equation]
where L corresponds to the loss function of the original predictive task, L E the overall predictive loss,o is the NODE-based model with f modeling time derivatives, g ( · ) :
= min(1 , max(0 , · )) , λ 0 , λ 1 ≥ 0are regularization parameters, and ⊙ is entry-wise multiplication.
L E measures how well the modelfits the current dataset.
While ∥ A ∥ 1 , 1 shrinks the actual values of weights θ in an attempt to zero outentire paths, ∥
A z ∥ 1 , 1 enforces exact zeros for entire paths at once.
When A z ij = 0 the output i isconstant in input j .
In practice, during training the expectation in Eq.
12 is estimated as usual via
Monte Carlo sampling.
Unlike GroupLasso (Bellot et al., 2021), PathReg does not require training of multiple networks inparallel.
Moreover, unlike C-NODE (Aliee et al., 2021), PathReg leads to exact zeros in the weightmatrices and requires no choice of threshold for deciding which paths are considered as zeros (see
Figure 2).
3.2 Training sparse models
Sparsity can generally be enforced during or after model training.
In this paper, we study two mainapproaches (Hoefler et al., 2021) to train sparse models: (i) sparse training , where we add regularizers already during model training; (II) training and iterative sparsification , where we iteratively increaseregularization parameters only after the model has been trained (close to) to convergence.
A detaileddescription of our training procedures and architecture choices for each experiment is provided in
Appendix A.
3.3 Assessing feature sparsity
For models regularized with L0 or PathReg the natural measure for feature sparsity is
[Equation]
(13) where absolute values are taken entry-wise.
LassoNet also allows for a natural feature sparsitymetric by counting the non-zero input weights (cid:80) nj =1 ∥ W (1) j ∥ 2 > ϵ of the first hidden layer, wheren denotes the number of input features and ϵ is a small tolerance parameter.
To evaluate featuresparsity for other feed-forward continuous-depth models, we count the number of non-zero elementsin A = | W L +1 | · · · | W 1 | again with entry wise absolute values on all W and within some tolerance ϵ(set to 1 e − 5 in our experiments).
This amounts to an upper bound on the number of input-outputdependencies: while we can have A i,j ̸ = 0 even though X j / ∈ pa f θ ( X i ) , A i,j = 0 always implies
X j / ∈ pa f θ ( X i ) .
Results
In this section, we present empirical results on several datasets from different domains includinghuman motion capture data and large-scale single-cell RNA-seq data.
We demonstrate the effect of thedifferent sparsity methods including L0 (Louizos et al., 2018), C-NODE (Aliee et al., 2021), LassoNet(Lemhadri et al., 2021), GroupLasso (Bellot et al., 2021), and our PathReg on the generalizationcapability of continuous-depth neural nets.
We address different aspects of generalization by assessingthe accuracy of these models for time-series forecasting and the identification of the governingdynamical laws.
In all examples, we assume that the observed motion can be (approximately)described by a system of ODEs where variables correspond directly to the observables of thetrajectories.
4.1 Sparsity for system identification
We motivate sparsity for NODEs starting with a simple second-order NODE as used by Aliee et al.(2021).
The differential equations explaining this system are presented in Appendix B.1.
Thequestions that we try to answer here are whether different sparsity methods can (i) learn the trueunderlying system, and (ii) forecast into the future (extrapolate).
For this linear system, the adjacencymatrix represents the exact coefficients in Eq.
The results summarized in Tables 1 and 2 showthat PathReg, C-NODE, LassoNet, and GroupLasso perform well with respect to reconstruction error.
Moreover, PathReg, C-NODE, and GroupLasso lead to higher sparsity and outperform other methodsfor extrapolation.
Among these methods, only PathReg and C-NODE are able to learn the true systemwith the lowest error 2 .
[Table]
No regularizationL
Model
C-NODE
Feature
PathReg
Model + Feature
LassoNet
Feature
Method
N a m e
R e g u l a r
i z a t
i o n a pp li e d o n
Overall
Performance
Sparsities
O v e r a ll S c
o
r e
T r a i n W a l k
T e s t G o l f
W e i g h t
F ea t u r e
Worst
Best 8.5e-3 2.0ePerformance (MSE)
T e s t W a l k T r a
i n W a v e T r a i n G o l f
Method-Ranking
Best
Worst
O v e r a ll
P e
r
f
o r m a n ce
O v e r a ll
S p a r s i t i e s
Min
Max 0.
Sparsity (%)
4.2 Sparsity improves time-series forecasting
We next demonstrate the robustness of sparse models for both reconstructing and extrapolatinghuman movements using real motion capture data from mocap.cs.cmu.edu .
Each datapoint is93-dimensional (31 joint locations with three dimensions each) captured over time.
We select threedifferent movements including walking, waving, and golfing, and use 100 frames each for training.
After training, we query all models to extrapolate the next 100 frames.
For walking and golfing,where multiple trials are available, we also test the model on unseen data in the sense that it has notseen any subset of those specific sequences (despite having trained on other sequences depicting thesame type of movement).
Generally, for time-series forecasting, it is not straight forward to giveprecise definitions of in-, and out-of-distribution.
In Table 7 in Appendix C, we show a grid of plotsillustrating loss and sparsity as a function of the regularization parameter λ of each method on thisdataset.
We show detailed results in Table 3 in Appendix C and summarize them concisely in Figure 3.
We did not manage to scale GroupLasso to this 93-dimensional dataset.
While all other modelsshow comparative performance (in terms of MSE error), only PathReg achieves strong levels of bothweight and feature sparsity (quantitative comparison is in Table 3).
The results show that PathRegoutperforms other methods with respect to MSE for both train and test datasets while resulting in thehighest model and feature sparsity.
[Figure]
We further examine the extrapolation (in time)capability of these models.
We observe that allsparsity regularization techniques improve extrapolation over unregularized NODEs.
Figure 4qualitatively illustrates the reconstruction andextrapolation performance of both the baselineand PathReg for the walking dataset (visually,the results are similar to PathReg for other regularizers).
While these models perform well forextrapolation and generalization to unseen data,we show that only PathReg and C-NODE areable to avoid using spurious features for forecasting.
Figures 7 and 8 in Appendix C show that PathReg and C-NODE learn sensible and arguablycorrect adjacency matrices (derived from joint connectedness), whereas L0 and LassoNet fail tosparsify the input-output interactions appropriately.
Since we use second-order NODEs for thisdataset, the adjacency matrix described in Section 3.3 represents how the acceleration predicted byf θ of each joint point (e.g., X i ) depends on the positions as well as the velocities
[Figure]
E
P a t
h R e g
4.3 Sparsity improves dynamic identification for single-cell data
Finally, we explore how pruning (i) improves real-world generalization and (ii) helps to learngene-gene interactions in human hematopoiesis single-cell multiomics data (Luecken et al., 2021)(GEO accession code = GSE194122).
The regulatory (or causal) dependencies between genes canexplain cellular functions and help understanding biological mechanisms (Aalto et al., 2020).
Geneinteractions are often represented as a gene regulatory network (GRN) where nodes correspond togenes and directed edges indicate regulatory interactions between genes.
Reliable GRN inferencefrom observational data is exceptionally challenging and still an open problem in genomics (Qiuet al., 2020).
As human regulatory genes known as transcription factors ( ∼ 1,600 (Lambert et al.,2018)), do not individually target all genes (potentially up to ∼ 20,000), the inferred GRN is expectedto be sparse .
Thus, predicting the expression of a gene should depend on few other genes that can berestricted using sparsity-enforcing regularizers.
For assessing real-world generalization of various sparsity-enforcing methods, we use human bonemarrow data of four diverse donors, generated at three different sites using five sequencing batches.
In this experiment, we train NODE models with one batch (S1D2), and later test those using otherfour batches (S1D1, S2D1, S2D4, and S3D6).
Notably, time-resolved ground truth dynamics ofcells throughout their lifetime are not available, because all existing genome-wide technologies aredestructive to the cells (Luecken et al., 2021).
Therefore, scRNA-seq only offers static snapshotsof data and the temporal order of cell states is commonly estimated based on certain distancesbetween their measured transcriptomes.
Following that, we first apply the diffusion pseudo-time(DPT) ordering approach (Haghverdi et al., 2016), a diffusion map based manifold learning technique,to rank the cells.
This allows to generate a cell fate mapping from hematopoietic stem cells (HSC)into three biological lineages that are used for training: Monocyte, Erythroid, and B cells (Figure 5).
NODEs are then trained using highly-variable genes (529 genes) with time taken to be the inferredpseudo-time (Figure 9).
The train and test MSE across 5 batches and three pruning methods are presented in Table 5 and
Table 7 in Appendix C and summarized in Figure 6.
While all regularized models achieve decent
No regularizationL
Model
C-NODE
Feature
PathReg
Model + Feature
LassoNet Feature
Method
N a m e
R e g u l a r
i z a t
i o n a pp li e d o n
Overall
Performance
Sparsities
O v e r a ll S c
o
r e
S 1 D
S 3 D
W e i g h t
F ea t u r e
Worst
Best 1.
Performance (MSE)
S 2 D 4 S 2 D 1 S 1 D
Method-Ranking
Best
Worst
O v e r a ll
P e
r
f
o r m a n ce
O v e r a ll
S p a r s i t i e s
S p u r i o u s F ea t u r e s
Min
Max 0.
Sparsity (%) predictive performance, PathReg outperforms the other methods in most scenarios.
We furtherstudy the transcription-factor-to-target-gene interactions via the adjacency matrices filtered fortranscription factors (columns) and target genes (rows) (see Appendix A for more details).
Theadjacency matrix
A demonstrates what transcription factors (on columns) are used for predicting theexpression value of a target gene (on rows).
| A ij | > ϵ for a small threshold ϵ = 1 e − 5 can then beinterpreted as an interaction between gene i and gene j being present in the data (see Figure 11 foran illustrating example).
For visualisation purposes, Figure 10 only shows the adjacency matricesacross the Erythroid lineage.
However, the results are comparable for other lineages.
The used listof transcription factors and target genes are extracted using chromatin and sequence-specific motifinformation (we again refer to Appendix A for more details).
Since ground truth dynamics are notknown for GRNs, validating the inferred interactions for multiple cell types is a challenging task.
However, we observe that PathReg and L0 are able to remove the interactions between pairs known tonot directly causally influence each other, including target genes (called spurious features in Table 5),while the other models fail in that regard.
Assessed by the literature, we also observe that knownerythroid transcription factors are inferred as important by both PathReg and L0 (RUNX1 (Kuvardinaet al., 2015),
ETV6 (Wang et al., 1998), NFIA (Micci et al., 2013), MAZ (Deen et al., 2021), GATA1(Zheng et al., 2006).
Conclusion and discussion
We have shown the efficacy of various sparsity enforcing techniques on generalization of continuousdepth NODEs and proposed PathReg, a regularizer acting directly on entire paths throughout aneural network and achieving exact zeros.
We curate real-world datasets from different domainsconsisting of human motion capture and single-cell RNA-seq data.
Our findings demonstrate thatsparsity improves out-of-distribution generalization (for the types of OOD considered) of NODEswhen our objective is system identification from observational data, extrapolation of a trajectory,or real-world generalization to unseen biological datasets collected from diverse donors.
Finally,we show that unlike weight sparsity, feature sparsity as enforced by PathReg can indeed helpin identifying the underlying dynamical laws instead of merely achieving strong in-distributionpredictive performance.
This is particularly relevant for applications like gene-regulatory networkinference, where the ultimate goal is not prediction and forecasting, but revealing the true underlyingregulatory interactions between genes of interest.
We hope that our empirical findings as well ascurated datasets can serve as useful benchmarks to a broader community and expect that extensions ofour framework to incorporate both observational and experimental data will further improve practicalsystem identification for ODEs.
Finally, our path-based regularization technique may be of interest toother communities that aim at enforcing various types of shape constraints or allowed dependenciesinto deep learning based models 3 .
Checklist
For all authors..
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
[Table]
mental results (either in the supplemental material or as a URL)?
[Yes] The codes areincluded in the supplementary files.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Explained in Section A and included in the codes.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Included in Appendix A.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] (Chen et al., 2018;
[Table]
Norcliffe et al., 2020)
[Table]
(b) Did you mention the license of the assets?
[Yes]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A] All the assets used in this paper are publicly accessible.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
A Models
All neural networks used in this work are fully connected, feed-forward neural networks.
First-order
NODEs are used for single-cell data, while second NODEs are used for the synthetic example aswell as the motion capture data.
In the second-order NODEs, the initial velocities are predictedusing a neural network with two hidden layers with 20 or 100 neurons depending on the datasetwith ELU activation function.
The main architecture to infer velocities (or accelerations) alsocontains two hidden layers of sizes 20 or 100 depending on the size of the input and ELU (or for thesynthetic experiments linear) activation function.
As an ODE solver, we use an explicit 5 -th order
Dormand-Prince solver commonly denoted by dopri5 .
All models are optimized using Adam (Kingma & Ba, 2017) with an initial learning rate of 0 . 01 .
Weuse PyTorch’s default weight initialization scheme for the weights.
All models can be trained entirelyon CPUs on consumer grade Laptop machines within minutes or hours.
B DatasetsB.1 Synthetic second-order ODE
We use a second-order, homogeneous, linear system of ODEs with constant coefficients to assesshow pruning affects system identification using NODEs.
Our system is denoted as:
[Equation]
[Equation]
[Equation]
We also set X (0) =
8) and ˙ X (0) =
as initial positions and velocities,respectively.
B.2 Motion capture data
The Carnegie Mellon University Motion Capture Database (Shell, 2012) is a large collection ofhuman motion data.
We are only interested in those records where activities are performed by asingle person who does not interact with other people and has minimal contact with objects withinthe environment.
The data is presented as 31 joint points in 3D space (in total of 93 variables) and agraph links them.
The data is preprocessed as the following:
• Centering the frames: The root point is fixed at the origin (0 , 0 , 0) for the whole duration
of the movement.
We observe that this method is effective particularly for walking data.
• Centering the first frame: For golfing and waving, the root point of the first frame is
moved to the origin (0 , 0 , 0) .
• Setting the floor at z = 0 : We find the minimum value of the Z-coordinate throughout the
trial and among all joint points.
We consider this level to be the “floor” and move all points
so that the minimum value of z during the whole trial is at 0 on the Z-axis.
B.3 Single-Cell RNA-seq
To map putative transcription factor (TF) and target gene relationships, we use as a referencea regulatory network generated using the gene expression and chromatin accessibility featuresavailable in the human immune cells dataset.
Briefly, using the function rank_genes_groups from
SCANPY (Wolf et al., 2018), we select the top 1,000 genes and 25,000 chromatin peaks, usingcell type label as a covariate.
Then, for each chromatin peak we map sequence-specific TF motifsusing as reference the TF archetypes from ENCODE (n=236), Peaks are mapped to the closest geneusing the R package ChIPseeker, with default parameters for the genome build hg38.
All chromatinpeaks linked to a gene as an enhancer annotation are considered, as long as the mapped distancebetween peak and promoter is below 200 Kbp.
Promoter/intron/exon peaks are always mapped to theharboring gene.
Our rule for successfully mapping a TF to a target gene through a chromatin peak isthat all TF, chromatin peak, and target gene, have to be simultaneously in the list of features selected
[Table]
in the rank_genes_groups function for cell type of interest, and there have to be TF motifs linked tothat transcription factor in the chromatin peak.
This gives us a total of 113 TFs and 365 potentialtargets genes across all cell types, that we use for evaluating the models.
C Extended results
[Table]
C.1 Synthetic second-order ODE
[Table]
Table 2 shows the inferred adjacency matrices, reconstructed and extrapolated samples.
The valuesin the adjacency matrices show the coefficients of the learned differential equations with the truecoefficients presented in Eq.
14, 15,and 16.
Only C-NODE and PathReg are able to learn the truedynamics.
C.2 Motion capture data
[Table]
Tables 3 and 4 show train and test MSE as well as model and feature sparsity for motion capture data.
The inferred adjacency matrix for walking motion using PathReg is illustrated in Figure 7.
The jointpoints corresponding to each leg or hand are highlighted with dashed lines.
The extended resultsacross different methods and motions are also presented in Figure 8.
[Table]
[Table]
C.3 Single-cell data
Tables 5 and 6 compare MSE and sparsity of various models.
Figure 9 presents the real and predictedexpression of genes over time for three lineages including Monocyte, Erythroid
[Table]
and B using PathReg.
The inferred adjacency matrices are also illustrated in Figure 10.
C.4 Analysis of the effect
The plots in Table 7 show the effects that different regularization weights have on performance andmodel as well as feature sparsity.
[Figure]
l e f t l e g r
i g h t
l e g l e f t h a nd r
i g h t
h a nd l e f t l e g r
i g h t
l e g l e f t h a nd r
i g h t
h a nd
[Table]
[Figure]
C - N
O D E
L
L a ss o N
[Figure]
[Table]
r e a l e x p r e ss i o n s t i m e genes p r e d i c t e d e x p r e ss i o n s
M o n o c
y t e li n e a g e r e a l e x p r e ss i o n s p
r e d i c t e
d e x p r e ss
i o n s
E r y t h r
o i
d li n e a g e r e a l e x p r e ss i o n s p
r e d i c t e
d e x p r e ss
i o n s
B li n e a g e
[Figure]
[Figure]
[Table]
[Table]