Iterative Structural Inference of Directed Graphs
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we propose a variational model, Iterative Structural Inference of
Directed Graphs (iSIDG), to infer the existence of directed interactions fromobservational agents’ features over a time period in a dynamical system.
First, theiterative process in our model feeds the learned interactions back to encourage ourmodel to eliminate indirect interactions and to emphasize directional representationduring learning.
Second, we show that extra regularization terms in the objectivefunction for smoothness, connectiveness, and sparsity prompt our model to infer amore realistic structure and to further eliminate indirect interactions.
We evaluateiSIDG on various datasets including biological networks, simulated fMRI data,and physics simulations to demonstrate that our model is able to precisely infer theexistence of interactions, and is significantly superior to baseline models
Introduction
A wide range of dynamical systems in the real world can be viewed as the composition of interactingagents, including physical systems [24, 13], multi-agent systems [5, 27] and biological systems [48,35].
Knowing the interactions of dynamical systems can promote our understanding of the intrinsicmechanisms of these systems, and further facilitate accurate prediction and control of dynamicalsystems.
However, in many cases, unlike the states of the agents, the interactions are usually notdirectly observable or measurable.
Therefore, it is necessary to uncover the underlying interactionsof dynamical systems based on the observational states of the agents.
In dynamical systems, thestates of an agent are affected by the interactions, and the states are usually recorded as a set ofcontinuous variables, which make it difficult to uncover the interactions based on the similaritybetween the agents.
It is also worth mentioning that most dynamical systems are characterized bydirected interactions , such as citation networks [44], social ego networks [31], and gene regulatorynetworks [35].
We address the revealing of the existence of interactions in the dynamical systems asthe problem of structural inference, and model the dynamical system as a directed graph, in whichthe nodes represent the agents of the system, and the directed edges denote the interactions betweenthe agents.
Thus, the structural inference problem is transformed into the inference of the asymmetricadjacency matrix of the graph based on observational nodes’ features (dynamics).
While several models have been proposed to address this problem with unsupervised variationalgenerative frameworks [22, 51, 30], most of them only deal with undirected interactions, and almostnone of them explains why the variational method succeeds in inferring the structure.
In this paper,based on the theory of Information Bottleneck (IB) [47, 1], which states the training of variationalauto-encoder (VAE) follows the two-step procedure of deep learning: label-fitting and representationcompression, we explain the intrinsic mechanism of structural inference with VAEs, and proposea novel VAE based machine learning approach, namely i terative S tructural
I nference of D irected
G raphs (iSIDG), for iterative inference of the directed graph structure based on observational nodefeatures.
Our iSIDG model utilizes a graph neural network (GNN) as encoder and infers adjacencymatrix of the graph in its latent space.
The key rationale of our iSIDG model is the iterative process for structural inference, which isdesigned based on the Variational Information Bottleneck [1].
The iterative process not only feedsthe learned structure backward with direction information, but also creates tighter bounds andrelaxation during the training process.
The direction information contributes to a more diverse edgerepresentations in the encoder and latent space.
The iterative process connects several rounds oftraining a VAE into a loop, and joints the second phase: representation-compression with the labelfitting phase of the next round, which creates a tighter bound and a relaxation, to learn a more precisedisentanglement between indirect connections and direct ones.
Unlike previous works [22, 51, 30] thattotally rely on the bottleneck structure of VAE to infer structure, iSIDG has extra terms to regularizethe learned features in the latent space, such as sparsity and connectiveness, which carefully takesthe properties of (asymmetric) adjacency matrix into account.
Experimental results show that iSIDGoutperforms or matches the state-of-the-art baselines on datasets with directed or undirected graphs.
More importantly, iSIDG can distinguish indirect connections from direct ones more precisely.
Related Work
Neural relational inference (NRI) [22] is the first to utilize a VAE to address the problem of structuralinference based on observational node features.
NRI conducts message-passing operations on a fixedfully connected graph structure in its encoder and infers graph structure in its latent space.
Based on NRI, Webb et al.
[51] propose factorized neural relational inference (fNRI), which extends NRI tomulti-interaction systems.
However, fNRI relies on ground-truth graph structure to classify results foreach type of interaction.
To deal with more complex systems, Li et al.
[29] and Chen et al.
[9] takevarious prior knowledge into account, such as sparsity and the distribution of node degrees, to increasethe accuracy of structural inference.
Unfortunately, it is quite difficult to acquire prior knowledge inmany problem settings.
Alet et al.
[2] propose a modular meta-learning-based framework that jointlyinfers the structure with higher data efficiency.
However, the framework has to be trained on variousdatasets beforehand to reach its best performance.
From the aspect of Granger-causality, Löwe et al.[30] propose amortized causality discovery (ACD) method, which infers a latent posterior graph fromtemporal conditional dependence.
Despite its high accuracy, the ACD method suffers from indirectinteractions.
Compared with this line of work, iSIDG introduces an iterative structural inferenceprocess to create a tighter bound by updating the fully connected graph structure in the encoder, andencourages edge representations to be diverse with the integrated direction information.
In addition to the aforementioned models to infer the structure of static graphs, several frameworkscan also infer the structure of dynamic graphs.
Ivanovic and Pavone [15] present a graph-structuredmodel which combines recurrent sequence modeling and variational deep generative modeling to inferdynamic structure and predict future trajectories.
Li et al.
[26] propose a generic trajectory predictionforecasting network to provide multi-modal trajectory hypotheses and auxiliary structural inference.
The framework in [12] utilizes sequential latent variable models to infer and predict separate relationgraphs for every single time step.
Moreover, the generic generative neural system proposed in [27]predicts trajectories based on dynamic graph representation and scene context information.
Theframeworks that can infer dynamic graph structures are mostly application-focused, and the structuralinference is auxiliary to the task of trajectory prediction.
Besides the models based on variational interface, we would like to mention a few works whichdeal with the problem of structural inference as well, but based on different methodologies.
ARNIframework [6] infers the latent structure based on regression analysis and a careful choice of basisfunctions.
Mutual information is also utilized to determine the existence of causal links and thuscan infer the structure of dynamical systems [52, 40].
Some approaches fit a dynamics model andthen produce a causal graph estimate of the model by using recurrent models [45, 19], or infer thestructure by generating edges sequentially [17, 28] and others independently prune the generatededges from an over-complete graph [41].
Xue and Bogdan [54] reconstruct network structuresunder unknown adversarial interventions.
Although these methods can provide insights about thecorrelations between adjacent nodes, such as mutual information and causal relation, they eitherrequire prior knowledge, or demand the comparison of features from all possible node-pairs, or aresensitive to indirect interactions in the graphs.
There exists another branch of research called Graph Structure Learning, which aims to jointly learn anoptimized graph structure and corresponding graph representations for downstream tasks [60, 11, 16].
However, these methods adopt similarity-based methods to optimize the noisy graph structure, which cannot be applied to dynamical systems.
Furthermore, the main focus of these methods is thedownstream task, not the reconstruction of the graph structure.
Besides that, there are also a seriesof work to reconstruct the structure of directed acyclic graphs [59, 57, 38, 58].
However, theseworks have difficulties to work on graphs with cycles (and self-loops), which are commonly observedamong dynamical systems.
Last but not least, in the research field of biology, there are some workwhich can infer the structure of gene regulatory networks [25, 20, 7].
But these works either requireprior assumption or cannot be extended to dynamical systems whose agents have multi-dimensionalfeatures.
Unlike these works, iSIDG focuses on the structural inference of dynamical systems, andcan deal with cycles in the system and high-dimensional node features.
Preliminaries
Notations and problem formulation.
We view a dynamical system as a directed graph, where theagents of the system are represented as the nodes of the graph, and the directed interactions betweenthe agents as the edges.
We note the directed graph as G =
( V , E ) , in which V represents the set of nnodes: { v i , 1 ≤ i ≤ n } , and E represents the set of edges: ( v i , v j
) ∈ E ⊆
V × V .
Based on the setof edges, we derive to an asymmetric adjacency matrix A ∈ R n × n , in which a ij ∈ { 0 , 1 } indicatesthe presence ( a ij = 1 ) of the edge between v i and v j or not ( a ij = 0 ).
The edges in our work areassumed to be static and do not evolve with time.
The feature recordings of the nodes over a timeperiod are represented as V =
{ V t , 0 ≤ t ≤ T } , where T is the total number of time steps , and
V t is the set of features of all the n nodes at time step t : V t = { v t 0 , v t 1 , . .
. , v tn } .
Given M sets offeature recordings {V k , 0 ≤ k ≤ M } , the structural inference problem we consider in this paper is toreconstruct the asymmetric adjacency matrix A of the graph in an unsupervised way.
We state the problem of structural inference as searching for a combinatorial distribution to describethe existence of the edges between any of the node pairs in the graph.
As shown in Figure 1, weutilize a GNN based VAE [21] as the cornerstone, which can be trained to approximate complicatedand high-dimensional probability distributions from the sampled node features [37].
We design aniterative process based on the IB theory, and feed the direction information, which is computed fromthe learned structure in the past training procedures, back to the input side, to encourage a morediverse learned representations, and to create a tighter bound as well as a relaxation of the training.
Graph neural networks.
Graph neural networks (GNNs) are powerful deep learning algorithmsto learn the representations of nodes and graphs.
Most modern GNNs are designed based on theneighbourhood aggregation strategy [14], where the representation of a node is updated by aggregatingthe representations of its neighbours.
In brief, we denote h ( k ) i as the representation of node v i at thek -th layer, and the mechanism of these GNNs can be described as:
[Equation]
[Equation]
where N ( v i ) denotes the set of chosen nodes related to v i (e.g., its neighbours), UPDATE denotesthe arbitrary function in the layer to update node representation, and AGGREGATE represents theaggregation function in the layer.
GCN [23] updates node representation based on one-hop neighbours and also its own representationin the previous layer, and calculates new representations with ReLU.
GIN [53] aggregates therepresentations of adjacent nodes and node’s own representation in the previous layer as a weightedsummation, and utilizes multilayer perceptron to update node representations.
GAT [49] introducesattention mechanism to implicitly specify different weights to adjacent nodes, and then updatesrepresentations upon the weighted summation.
GNNs are efficient in learning rich representations fornodes and graphs.
As a result, we utilize GNNs in the encoder to firstly learn representations of thenodes, and later on learn edge representations from the node representations in iSIDG.
Information bottleneck.
The theory of Information Bottleneck (IB) provides an explanation andunderstanding of learning with deep neural networks [47, 46, 42].
In general, for the input data Xand its label Y , the IB aims to learn the minimal sufficient representation Z = arg min
Z I ( Z ; X ) −u · I ( Z ; Y ) , where u is the Lagrangian multiplier to balance sufficiency and minimality.
The trainingepochs of standard deep learning can be divided into two phases: label-fitting and representationcompression.
These two phases can be characterized by the mutual information between the inputfeatures and the learned representation of a layer
I ( X ; Z ) , as well as between learned representationand the label variables I ( Z ; Y ) .
In the label-fitting phase, both I ( X ; Z ) and I ( Z ; Y ) increase, but
[Figure]
Alemi et al.
[1] present a variational approximation to the IB theory (VIB), and prove VAE objectiveis a special case of the variational approximation.
Under a fully unsupervised setting, VAEs learn theminimal sufficient statistics from the input features, which are sufficient to derive the output features,and keep the most compressed and abstractive representations in their latent space.
We utilize a VAEto infer the structure of the graph in its latent space, which is minimally sufficient to predict the futurestates of the nodes together with present node
It is suggested that IB iterations with stochastic relaxation methods may boost training [42].
It isalso mentioned that a relaxation process is helpful to reduce local ambiguity and achieve globalconsistency [56], as already proved in previous work [8, 55].
Therefore, we design an iterative processwith stochastic gradient descent (SGD) based optimizer to create a tighter bound and a relaxation forthe training, and to encourage iSIDG to further disentangle dynamics from the adjacency matrix.
Inorder to verify the effectiveness of this iterative process, we conduct a case study in Section D.3.
Model Design
4.1 Structural Inference by Information Bottleneck
Together with node features, graph structure characterizes the basic information of a graph, whichrepresents a dynamical system.
The goal of structural inference is to accurately reconstruct theadjacency matrix A of the directed graph based on the recordings of node features: A ← { V t , 0 ≤t ≤ T } .
Interestingly, for dynamical systems, the future features of the nodes are deterministic by theirprevious features and the connectivity: V t +1 ← { V 0 , V 1 , . .
, V t ; A } .
We assume the Markovianassumption works in our case, and thus previous statement can be simplified to: V t +1 ← { V t ; A } .
Thus, we derive to an assumption: the information about the adjacency matrix is implicitly includedin the node features over a time period, which also conforms to the IB theory.
Based on VIB, we derive the Information Bottleneck for Structural Inference , which aims at inferringthe adjacency matrix A in the latent space Z :
[Equation]
Interestingly, A is also required in the first term in Equation 2, which is unrealistic in our problemsetting.
Thus, we assume a fully connected graphs structure A (0) as input, which is not lessinformative than the actual sparse adjacency matrix:
[Equation]
Following the statement in VIB [1], we adopt a VAE [21] to deal with the problem describedin Equation 3, where an encoder q ϕ ( Z | V t , A (0) ) approximates I ( Z ; V t , A (0) ) , and a decoder p
θ
( ˆ V t +1 | Z , V t ) approximates I ( Z ; V t +1 ) .
ϕ
and θ represent the parameters of encoder and decoder, respectively.
We set the latent space of the VAE with the size of n × n .
The fixed dimension ofthe latent space restricts the size of the search space for the adjacency matrix.
We utilize a weighted
ELBO with multiple integrated regularization terms to optimize model parameters { ϕ, θ } .
4.2 Encoder
The encoder in our model is tasked with inferring the existence of directed interactions z ij betweenall node-pairs by extracting the most compressed representation given the node features V .
The encoder q ϕ ( Z | V t , A (0) ) returns a set { z ij | v i , v j ∈ V} , which features the distribution to describehow probable the edge from node j to node i exists: p ( z ij ) ∈ [0 , 1] .
The prior p ( Z ) = (cid:81) i,j ∈V p ( z ij )is a factorized uniform distribution.
We utilize GNNs to extract features in the encoder.
But the GNNslearn new representation based on both node features and the topology of the graph f GNN ( V , A ) ,in which the adjacency matrix A is our learning target.
To encounter this problem, we impose aniterative learning process on the adjacency matrix, and we discuss the details in Section 4.4.
Forsimplicity, in this section we use ˆ A to represent the adjacency matrix.
In general, the encoder:
[Equation]
(4) where f enc represents the encoder.
The basic setup of our encoder is similar to that in [22], whichapplies two rounds of node-to-edge and one round of edge-to-node message passing operations tolearn the edge embeddings h ij :
[Equation]
where f (1) embed is the embedding network for input features, f 1 e and f 2 e are node-to-edge messagepassing networks, and f v is the edge-to-node operation.
We observe that Equations 5 - 7 is a round ofmessage passing for node features in most GNNs, as a result, we substitute Equations 5 - 8 as:
[Equation]
so we can integrate state-of-the-art GNNs such as GCN, GAT and GIN into encoder. We chose GIN.
As a consequence, our encoder is designed to learn informative features from input node features andcompress them to fit the restricted search space, which has the dimension of a fully connected graph.
By extracting information from the high-dimensional but low-level node features to the restricted andlow-dimensional latent space, our encoder disentangles the dynamics and approximates the minimallysufficient statistics, the adjacency matrix.
After that, we model the existence of edges as Z ∈ R n × n ,where its component z ij comes from q ϕ ( z ij | V t , ˆ A ) =
softmax ( h ij ) .
4.3 Decoder
The task of the decoder is to predict the future states V t +1 based on the samplings from latent space
Z and the present node features V t .
From the IB theory, if the encoder derives minimally sufficientstatistics as the adjacency matrix, the decoder can precisely predict the future node features.
However,there remains the problem of how to design the decoder, so that it can further encourage the encoderto disentangle dynamics and to infer the graph structure only.
In order to deal with this problem,we design the decoder to imitate a single message-passing operation.
We feed our decoder with thepresent node features besides the samplings from the latent space to predict the future node features.
As a consequence, the node features are disentangled from the compressed information in the latentspace during training.
In short, the decoder deals with the following problem: p θ ( V t +1 | V t , Z ) .
Weuse a two-stage decoder, which consists of a message-passing layer and an embedding layer:
[Equation]
where f (2) embed and f (3) embed are multilayer perceptrons (MLPs) to embed the features.
4.4 Iterative Learning with Direction Information
The motivation of integrating iterative learning process into the structural inference consists of threeaspects: (1) to create a tighter bound for Equation 3; (2) to create a relaxation for the training process;(3) to emphasize direction information.
In order to formulate the ideas more precisely, we would liketo firstly describe the iterative process with direction information in our model.
From the very beginning, we have no prior knowledge about the structure of the graph.
In order toutilize GNNs to extract and compress the features into a restricted search space for adjacency matrix,we feed the encoder with a fully connected graph A (0) = 1 ∈ R n × n .
After η rounds of training,we obtain an asymmetric adjacency matrix A ( p ) from the latent space, which is calculated from theaveraged result of the entire training samples:
[Equation]
where m is the total number of training samples, and M is the set of training data.
The full-batchaverage operation not only preserves the most information from the training set, but also makes the
A ( p ) to be invulnerable to outliers.
We then normalize A ( t ) in both row-wise and column-wise:
[Equation]
12) where D − 1 in and D − 1 out are in-degree and out-degree matrices which are calculated based on A ( p ) ,respectively.
We name Equations 12 as direction information .
After that, we combine the directioninformation matrices with the normalized initial fully-connected adjacency matrix, respectively:
[Equation]
[Equation]
where L is the normalized adjacency matrix of the initial fully-connected graph A (0) , and λ isthe combination coefficient.
Since A (0) is symmetric, we can use either column-wise or row-wisenormalization.
We map the results in Equation 13 to binary sets with the characteristic function:
[Equation]
where κ ∈ { row, col } , and ξ is the threshold value.
Finally, we feed ˜ R ( p ) κ back to the input side ofencoder mentioned in Section 4.2 by substituting the ˆ A in Equations 6 and 8.
More formally, the twofeature vectors which are concatenated in Equations 6 and 8 represent the sender and receiver of theedge, respectively.
We rewrite the equations for the iterative process according to ˜ R ( p ) κ :
[Equation]
where k ij ∈ { e ij , h ij } , f n e ∈ { f 1 e , f 2 e } , and k n ∈ { e 1 n , e 2 n } in Equations 6 and 8, respectively.
Wethen train iSIDG with the updated adjacency matrices ˜ R ( p ) κ .
The stop condition for iteration is:
[Equation]
16) where superscript [ i ] represents the rounds within current iteration.
Once this condition is satisfied,the adjacency matrix at the input side of the encoder will be updated according to Equations 11 - 15.
Tighter bound.
During training, the initial fully connected adjacency matrix A (0) is updated tosparser matrices { ˆ A ( p ) κ , κ ∈ { row , col }} .
(We omit the description for κ in this paragraph forclearer statement).
The model is able to roughly detect which edges are not beneficial to the futurestate prediction, and eliminates these edges in the new representation.
As a consequence, any ofthe matrices in { ˆ A ( p ) κ } is sparser than A (0) .
Thus the first term in Equation 3 is re-framed as
I ( Z ; V t , { ˆ A ( p ) κ } ) , which obviously supports: I ( Z ; V t , { ˆ A ( p ) κ } ) ≤ I ( Z ; V t , A (0) ) , and results in atighter bound for Z in Equation 3.
Relaxation.
Besides that, our iterative training process imposes relaxation to connect the two phases,i.e., label-fitting and representation-compression, of training a deep learning network as a cycle.
Inthe first phase, our model for structural inference tries to fit the target for prediction, when the trainingerrors become small, it turns to the second phase and compresses the representation of any layer tobe minimally sufficient to fit the target.
In the second phase, the compression process affects theadjacency matrix in latent space as well, and due to the difficulty of training in an unsupervisedmanner, it is very apt to reach a local optimal.
The iterative process, which feeds the learned adjacencymatrix back to the encoder, stops the phase of representation-compression, and starts the next round oflabel-fitting and representation-compression.
At this round, the combination of direction informationand the initial adjacency matrix together creates a different adjacency matrix for the encoder comparedwith the previous round, which will relax the learning of adjacency matrix back from the local optimal.
We verify the effectiveness of the iterative process through mutual information in Section D.3.
Direction information.
One of the main differences between undirected and directed graphs is thatthe former can be described with a unified degree matrix, but the latter requires two matrices forin-degree and out-degree.
As a result, we introduce direction information by integrating the in- andout-degrees of every node with the new adjacency matrices, which contributes to a bifurcated trainingprocess and encourages the learned embeddings for edges to be more diverse.
The integration ofdirection information is beneficial to the structural inference for directed graphs, which is validatedexperimentally in Section 5.1.
4.5 Training with a Hybrid Loss
Although the iterative process is an effective way to learn the adjacency matrix of directed graphs,adequate loss function can promote the training process as well.
Firstly, the following is the lossfunction for prediction:
[Equation]
Then the Kullback–Leibler ( KL ) divergence acts as a regularization for the latent space:
[Equation]
We omit the superscripts of vectors V and v in Equations 17 - 21, the actual correspondence refers to
Sections 4.2 and 4.3.
In addition to the aforementioned terms which are often utilized as the objective function for VAEs,we introduce extra regularization terms.
The regularization term of signal smoothness takes bothinferred structure and node features into consideration to eliminate any connection which does notfulfill the smoothness assumption of adjacent node features, such as indirect interactions.
We considerinput features V as graph signals, and the assumption for graph signals also works here: valueschange smoothly across adjacent nodes [10].
Therefore, we introduce Dirichlet energy [3] to measurethe smoothness between the signals:
[Equation]
It is worth mentioning that in most cases the adjacency matrix is asymmetric, we cannot use Laplacianmatrices to simplify the formulation.
Moreover, we introduce more regularization terms [3, 18] to feature the learned features in latentspace with the properties of realistic adjacency matrices, such as connectiveness L d and sparsity L s .
Training the network with these regularization terms encourages our model to further disentanglenode representations from the latent space and distinguish indirect interactions from direct ones:
[Equation]
where the logarithmic barrier in L d forces the model to learn a connected adjacency matrix, but lacksthe regularization of sparsity.
In consequence, we introduce L s to regularize its sparsity.
[Table]
To conclude, we train our iSIDG with the hybrid loss L :
[Equation]
where { µ, α, β, γ } are hyperparameters.
The first two terms in Equation 21 formulate a weighted
ELBO, which is the training objective for VAEs.
The rest of the terms are regularization for thelearned adjacency matrix in latent space.
Together with the training cycles of fitting and compression,which are connected with relaxations, the hybrid loss function further encourages our iSIDG model toreconstruct the adjacency matrix precisely.
We give a detailed discussion on the design methodologyof iSIDG on how to effectively eliminate indirect connections in Section A.
Experiments
We test our iSIDG model on three different datasets, and present its ablation study on one of thedatasets.
Implementation details and further experimental results can be found in Sections B - G.
5.1 Structural Inference on Different Systems
Datasets.
We test our model on the six directed synthetic biological networks [35], namely Linear (LI),
Linear Long (LL), Cycle (CY), Bifurcating (BF), Trifurcating (TF), and Bifurcating Converging (BFCV) networks.
These networks are essential components that lead to a variety of different trajectoriesthat are commonly observed in differentiating and developing cells [39].
We use BoolODE [35]to simulate the process of developing cells with these synthetic networks, and obtain six groups oftrajectories.
The features at every node are one-dimensional, i.e., the level of mRNA expression.
We also test our model on NetSim datasets [43] of simulated fMRI data.
The NetSim datasets consistof simulated blood-oxygen-level-dependent imaging data across different regions within the humanbrain, which leads to asymmetric relation networks.
The node features are one-dimensional.
The last datasets to be mentioned here are three physical simulations mentioned in [22], namelysprings, charged particles and phase-coupled oscillators (Kuramoto model).
We keep the symmetricinteractions in the setting of these networks.
However, different from the setting mentioned in thatwork, we sample the trajectories with fixed interactions between the agents in the system, but withdifferent initial conditions, in order to simulate the real application scenario on the observational datafrom a specific unknown system.
The features at every node are 4-dimensional.
Baselines and metrics.
We compare iSIDG with the state-of-the-art models for structural inference:
• NRI [22]: a VAE-based model for unsupervised relational inference.
• fNRI [51]: an NRI-based model with additional latent space for every factorized interaction type.
• MPIR [52] a model based on minimum predictive information regularization. • ACD [30]: a variational model that leverages shared dynamics to infer causal relations.
We describe the implementation details of the baseline methods in Section B.2.
We demonstrate ourevaluation results with the following metrics: the area under the receiver operating characteristic(AUROC), the area under the precision-recall curve (AUPRC) and the Jaccard similarity index (JI).
We also provide results of two DAG-based structure learning methods on our datasets in Section F.
Results.
The experimental results of our iSIDG model and baseline methods are summarized in
Tables 1 and 2 (also Tables 5 - 8 in Appendix), which consist of mean values and 95% intervals of
AUROC, AUPRC and JI from 15 experiments on each, respectively.
Due to the alternative simulationpipelines between ours and other literature, we obtain different results of ACD and MPIR on physicalsimulations from other literature.
iSIDG outperforms or matches the baseline methods.
In thedatasets of synthetic networks and three NetSim datasets, which consist of directed graphs and low
[Table]
[Figure]
dimensional input features, iSIDG outperforms baseline models by at most 13.2 percent in termsof AUROC, showing that it can infer structure more precisely than the baseline methods.
Theseresults highlight the validity of integrating the iterative training process and the direction informationon structural inference problem for directed graphs.
It is also remarkable that iSIDG matches or isslightly inferior to the baseline methods in the datasets of physical simulations, especially fNRI and
ACD.
The reason may be that high dimensional input features (4-dimensional features for physicalsimulation datasets) contain richer dynamics for fNRI and other methods to infer from.
It is worthmentioning that fNRI method utilizes ground truth to distinguish every latent space and match themto the edge type with the highest accuracy, which results in higher AUROC, AUPRC and JI scores.
Moreover, since ACD relies on the information shared by input features, the richer informationcontained in high dimensional features promotes a more promising result.
5.2 Ablation Study
We conduct ablation studies on the effectiveness of regularization terms in objective function: signalsmoothness Ω , connectiveness L d and sparsity L s , and the reconstruction results are shown in termsof heat maps in Figure 2.
The heat maps visualize the average results of 15 experiments of iSIDGwith different objective functions by deleting one regularization term, respectively.
As shown in thesecond plot in Figure 2, without considering signal smoothness, the indirect interactions emerge againin the results of structural inference with the correctly inferred edges.
The third plot in Figure 2 showsthat without considering the connectiveness, the model correctly infers only a few direct interactionsbut as isolated “islands", and connects these islands with numerous indirect interactions.
The lastplot in Figure 2 demonstrates the results that without the terms to regularize the sparsity, many directinteractions are substituted by the indirect ones.
Therefore, the extra regularization terms in theobjective function of iSIDG effectively encourage the model to infer realistic graph structures and toeliminate the influence of indirect interactions.
More ablation studies can be found in Section D.4.
Future Work
We so far only consider the structural inference for static and (relatively) small graphs, though wetested the scalability of iSIDG on graphs of 100 nodes (see Section E for the results).
Future researchincludes introducing dynamic interactions and more scalable model design which remains as a criticalchallenge in structure learning for dynamical systems.
Furthermore, we also want to integrate priorknowledge such as partial structural information to further promote the performance of iSIDG.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See abstract and last paragraph of Section 1.
(b) Did you describe the limitations of your work?
[Yes]
Due to page limitation, we
describe the limitation of our work with a few words in Section 6, and in Section E.(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection H.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] We have read the ethics review guidelines and ensured that our paper conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sections 3and 4.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs are quite intuitive, see Section 4.1.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We attach the
link to our code and data in the supplementary documents.
We also give instructions
about how to reproduce the main experimental results in Section B.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sections 5 and B.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Instead of error bars, we report our experimental results
with the mean value and 95% intervals of the results from running the experimentsseveral rounds.
We describe the method to obtain the data in Section 5, and the results
in Tables 1, 2 and 5- 8.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section B.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.1.
(b) Did you mention the license of the assets?
[Yes] We mention the license in our code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include new assets, and they can be found via a URL.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] The datasets are generated with simulations, which has nothing
to do with personal data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The data is neutral to biological human being.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Discussion
A.1 Indirect connections
Originally, iSIDG starts training with a fully connected graph structure:
A (0) .
Based on the problemsetting of dynamical systems, we assume A (0) can be decomposed as: A (0) =
{ A D , A IN , A U } ,where A D represents the actual edges in the graph (directed connections), A IN denotes the indirectconnections between the two actual connected nodes, and A U is the set of non-connections.
[Figure]
Previous structural inference methods, such as NRI, fNRI and ACD, are good at eliminating A Uin the inference results.
However, as shown in Figure 3, these methods may falsely reconstruct thestructure with indirect connections.
It is interesting that the indirect connections resulted from thetransmission of signals between nodes.
In Figure 3, there exists a chain connection from node v 1to v 3 through v 2 , and in a dynamical system, the changes in v 1 will finally affect v 3 , which wouldmislead the structural inference methods to falsely infer an edge from v 1 to v 3 .
Moreover, for a sparsegraph, it is possible that A IN ∩ A U ̸ = ∅ , which challenges structural inference methods, such as
NRI, fNRI and ACD.
A.2 Solution for Eliminating Indirect Connections
Our solution to eliminate indirect connections in the inference results consists of three methodologies:(1) VAE ; (2) Iterative process ; and (3) Regularization terms .
VAE.
As shown in previous works [22, 51, 30], the bottleneck structure of VAEs can correctly identifymany non-connections in the reconstructed adjacency matrix.
Thus, we also rely on the VAE to firstlyidentify as many non-connections as possible before the first round of iterative process, and sample{ ˆ R ( p ) row ,
ˆ R ( p ) col } according to operations shown in Equations 11 - 14.
Iterative process.
The iterative process follows the methodology of VIB, and is mentioned in
Section 4.4 with a new objective function:
[Equation] [Equation]
The iterative process is characterized by a main advantage: the first term in Equation
I ( Z ; V t , { ˆ A ( p ) κ } ) <
I ( Z ; V t , A (0) ) , which results in a tighter bound than Equation 3.
In otherwords, after feeding the direction information back to the input side of the encoder, the new searchspace excludes the non-connections, which are found by VAE in the previous training epochs.
With asmaller search space, iSIDG can focus on the differentiation between misleading indirect connectionsand the real connections.
Regularization terms.
Although feeding the learned adjacency matrix back to the encoder caneliminate the indirect connections, we need regularization terms to secure and accelerate this process.
The sparsity term L S encourages iSIDG to choose the more solid path from every node, to meet theminimal sufficient statistics assumption of IB.
For example, in Figure 3, at node 1 , iSIDG has tochoose between the path 1 → 3 and 1 → 2 → 3 based on the sparsity regularization terms and theminimal sufficient information for dynamics prediction in decoder.
One may argue that it is also possible that iSIDG will falsely think there are edges from node 1 tonode 2 , and from node 1 to node 3 .
However, this does not conform to the future state prediction.
The task of the decoder is trying to approximate:
[Equation]
[Equation]
where f ( · ) denotes the interaction from node v j to node v i .
We may imagine the interaction could be
RNA activation level in GRN, spring force in spring-balls system, and the phase-couple of oscillators.
As a result, the interactions between nodes are assumed to be the same, and described with the samefunction.
Yet node 1 can only affect node 3 through node 2 , which results in a superposition offunctions: f ( f ( · )) .
Therefore, both edges from node 1 to node 2 , and from node 1 to node 3 cannotpredict the future state of node 3 correctly, and VAE will search for other possible edges.
B Further Implementation Details
[Figure]
B.1 Implementation details of iSIDG
We summarize the described architecture of iSIDG and present the pipeline of training iSIDG in
Algorithm 1.
( V T , A ( i ) , ϕ ) 12: ˆ V T +1 ← Decoder ( V T , Z , θ ) 13: if i > η and StopCond then 14:
˜ R
( i ) κ ← { Z , A (0) , ξ } using Eqs.
A ( i ) ← ˜ R ( i ) κ 16:
L p ← { ˆ V T +1 , V T +1 , δ } 17:
L KL , L d , L s ← { Z , n } 18:
L ( i ) ← L p + µ L KL + α Ω( A ( t ) , V ) + β L d + γ L s 19:
L ← L ( i ) /T 20:
Back-propagate L to update model weights ϕ
and θ 21: i ← i +22: end if 23: end while 24: Return: A ( t )
Algorithm 2 A MLP block.
1: Input: features input 2: dropout 6: Return: out
1: Input: features input 2: x = mlp1( input ) 3: x = node2edge(x) 4: x = mlp2(x) 5: x_skip = x 6: x = edge2node(x) 7: x = mlp3(x) 8: x = node2edge(x) 9: x = concatenation([x, x_skip]) 10: x = mlp4(x) 11: out = fully connected out(x) 12: Return: out
Basic settings.
We implement our iSIDG model in PyTorch [33] with the help of Scikit-Learn [34]to calculate various metrics.
We run experiments on a single NVIDIA Tesla V100 SXM2 graphiccard, which has 32 GB graphic memory and 5120 NVIDIA CUDA Cores.
We attach our pseudocodeand implementation as the supplementary document to this paper.
During training, we set batch sizeas 128 for datasets which have less than 10 nodes, for those more than 10 nodes, we set batch size as
We train our iSIDG model with 2000 epochs on every dataset, which takes around 30 - 42 hoursfor every training session.
Loss Functions.
The hybrid loss function shown in Equation 21 consists of five different terms tobe calculated: {L p , L KL , Ω( Z , V ) , L d , L s } .
We would like to describe the actual implementationsof L p and L KL .
Since the target of the decoder is the prediction of future node features based onpresent states, L p is estimated by:
[Equation]
where the variance σ is set as a fixed value, and the const is e − 16 .
And the calculation of KL-divergence L KL over a uniform prior can be simplified as the calculationof entropy H :
[Equation]
Hyper parameters.
The choice of most hyper parameters in our iSIDG model (such as thosementioned in Algorithm 1) may refer to Table 3.
In addition to the hyper parameters mentioned in thetable, we set the combination coefficient λ in Equation 13 as 0.3 for the training on all of the datasets.
We argue that λ can be set as varying values according to the training epochs, and decrease duringtraining, which may be helpful for faster converge.
We would like to carry on more research ontothis hypothesis in the future.
Besides that, we describe the choice of time step length T in Section C.
Time step length T decides the amount of samplings along a trajectory.
[Table]
[Table] [Table]
[Table]
Ablation study with different GNNs.
The main difference between the utilization of different GNNstakes place in the encoder of iSIDG.
For the implementation of MLP encoder, we use three stackedmulti-layer perceptron blocks, with each block consists of the units shown in Algorithm 2.
We place aconcatenation operation between Block 1 and 2 to get edge representations.
The last block is attachedwith another Linear layer with its dimension matching the number of edges in a full-graph setting.
[Table]
The number of units in the Linear layers in the MLP encoder is shown in Table 4.
The dropout ratesin all of the blocks are set as 0 . 5 .
For the simple message-passing mechanism encoder ( MPM encoder), we follow the design of “MLP
Based on the design of Graph Identity Networks and the MPM encoder, we combine these ideatogether into the GIN encoder, by replacing the operations from line 3 to line 6 in Algorithm 3 withthe GIN operation mentioned in [53], but still keep the skip connection for edges representations,and the pseudocode is shown in Algorithm 4.
Similarly, if we replace the GIN operation in line 5of
Algorithm 4 with GCN operation, we obtain the GCN encoder.
Moreover, the design of GATencoder is similar to that of GIN encoder and is shown as the pseudocode shown in Algorithm 5.
B.2 Implementation details of baselines
NRI.
We use the official implementation code by the author from https://github.com/ethanfetaya/NRI with customized data loader for our chosen datasets: synthetic networks, NetSims and physical simulations.
We add our metric-evaluation in “test" function, after the calculationof accuracy in the original code.fNRI.
We use the official implementation code by the author from https://github.com/ekwebb/fNRI with customized data loader for our chosen three datasets.
We add our metric-evaluation in “test"function, after the calculation of accuracy and the selection of correct order for the representations inlatent spaces in the original code.
MPIR.
We use the official implementation code from https://github.com/tailintalent/causal as the code for MPIR.
We run the code by customized data loader for the chosen threedatasets.
After the obtain of the results, we run another script to calculate the metrics.
ACD.
We follow the official implementation code by the author as the framework for ACD( https://github.com/loeweX/AmortizedCausalDiscovery ).
We run the code with customized data loader for the chosen three datasets.
We implement the metric-calculation pipeline inthe “forward_pass_and_eval()" function.
C Further Details about DatasetsC.1 Synthetic networks
The six directed Boolean networks (LI, LL, CY, BF, TF, BF-CV) are the most often observedfragments in many gene regulatory networks, each has 7, 18, 6, 7, 8 and 10 nodes, respectively.
Thusby carrying out experiments on these networks, we can get acknowledge about the performanceof the chosen methods on the structural inference of real-world biological networks.
We collectthe six ground-truth directed Boolean networks from [35] and simulate the single cell evolvingtrajectories with BoolODE [35] https://github.com/Murali-group/BoolODE with defaultsettings mentioned in that paper for every network.
We sample 12000 trajectories and group theminto three datasets: for training, for validation, and for testing, each with the number of 8000, 2000and 2000, respectively.
Then we sample 49 snapshots with equal time
C.2 NetSim datasets
The NetSim datasets simulate blood-oxygen-level-dependent imaging data across different regionswithin the human brain and is described in [43] and https://www.fmrib.ox.ac.uk/datasets/netsim/ .
We target at inferring the existence of directed connections between different brain areas.
Among the total 28 datasets in NetSim, we choose the first three datasets (NetSim1, NetSim2 and
NetSim3) which have 5, 10, and 15 nodes, respectively.
We sample 49 snapshots on each trajectorywith equal interval and randomly group them into three sets for training, validation and testing withthe ratio of 8: 2: 2, respectively.
C.3 Physical simulations
To generate these physical simulations (springs, charged particles and phase-coupled oscillators), wefollow the description of the data in [22] but with fixed interactions.
To be specific, at the beginningof the data generation for each physical simulation, we randomly generate a ground truth graph andthen simulate 12000 trajectories on the same ground truth graph, but with different initial conditions.
The rest settings for the simulations are the same as that mentioned in [22].
It is worth mentioningthat the connections in the physical simulations are indirected, which are different from those in thesynthetic networks and NetSim datasets.
We collect the trajectories and randomly group them intothree sets for training, validation and testing with the ratio of 8: 2: 2, respectively.
D Further Details about ExperimentsD.1 Metrics
We choose the three most representative metrics from 0-1 classification for the evaluation of structuralinference results, namely the area under the receiver operating characteristic (AUROC), the areaunder the precision-recall curve (AUPRC) and Jaccard index (JI).
All of the metrics are calculatedfrom the inferred structure and the full-patch of ground-truth.
AUROC.
The area under the receiver operating characteristic (AUROC) is a performance metric toevaluate the result of classification tasks.
It demonstrates the model’s ability to discriminate betweencases (positive examples) and non-cases (negative examples).
For our case, it is used to make clearthe model’s ability to distinguish actual edges and empty interactions.
It is calculated as the areaunder the receiver operating characteristic curve (ROC), which sets false positive rate (FPR) as x-axisand true positive rate (TPR) as y-axis at various threshold settings.
In the ROC space, the bestpossible prediction method would yield a point in the upper left corner or coordinate (0,1) of the
ROC space, representing 100% no false negatives and 100% no false positives.
Then a model outputsmore precise results with many TPR yields a higher AUROC value, and vice versa.
AUPRC.
The area under the precision-recall curve (AUPRC) is a useful performance metric forimbalanced data in a problem setting since we care a lot about finding the existence of interactionsbetween nodes.
It shows the ability of the model to find the existence of all ground-truth edgeswithout accidentally marking any non-interaction as existence.
The AUPRC is calculated as the areaunder the Precision-recall (PR) curve.
The x-axis of a PR curve is the TPR and the y-axis is theprecision.
This is in contrast to ROC curves, where the y-axis is the TPR and the x-axis is FPR.
Similar to calculate AUROC, AUPRC is also calculated from various threshold settings.
Jaccard index.
The Jaccard index (sometimes called the Jaccard similarity coefficient) (JI)compares the inferred edges and ground-truth to see the shared results and the distinct results.
Jaccard index is a measure of similarity for the two sets of data, with a range from 0% to100%.
The higher the percentage, the more similar the two populations.
It is calculated as"Intersection of two sets / Union of two sets".
Although JI is easy to calculate and interpret, theresult is extremely sensitive to small samples sizes and may produce erroneous results, especiallywhen
[Table] [Table]
[Table] [Table]
D.2 Further experimental results
In this section, we demonstrate additional experimental results as the supplement to Section 5.
Wepresent AUPRC and JI values as well as a case study about the structural inference results of iSIDGand baseline methods.
AUPRC and JI results.
We present the AUPRC and JI results of iSIDG and baseline methods onsynthetic networks, NetSim datasets and physical simulations in Tables 5 - 8.
The results are the meanvalues and 95% intervals of 15 experiments, respectively.
Similar to the AUROC results presentedin Section 5.1, iSIDG performs the best among all of the chosen methods in datasets with directedgraphs, and can match the best baseline method in datasets with indirected graphs.
Besides that, we conduct a case study of iSIDG with different objective functions, and calculate theaverage accumulated path lengths based on 15 experiments of each on Linear network dataset.
Theresults are shown in Figure 4.
It is clear that our choice of objective function with regularizationterms of smoothness, connectiveness and sparsity successfully encourages iSIDG model to infer thestructure of the system with less indirect interactions and more accurate results.
N
Full
No Ω
[Figure] [Figure]
Case study.
To gain an intuitive understanding of the results of the structural inference of directedgraphs, we conduct a case study on the Linear network dataset.
The reconstructed adjacency matricesof our iSIDG, fNRI and ACD methods on Linear network dataset are visualized in Figure 5, aswell as the ground-truth.
The results are the mean results of 15 experiments of each method andare shown in terms of distributions of reconstructed relations.
In the figure, 0 .
at a point [ x, y ]denotes the absence of interaction from x to y , and 1 . 0 represents in an opposite way.
As shown in
Figure 5, compared with the ground-truth, the structure reconstructed by ACD contains many indirect
[Figure]
connections.
It is possibly the result from the fact that ACD infers a latent posterior graph from theinput features based on hidden confounding [50].
For Linear network dataset, which has a “chain"structure, it may confuse ACD method as it contains many indirect interactions, and thus leads toinaccurate inference results.
Moreover, fNRI utilizes a factorized latent space for every possibleinteraction types.
For the case here, it even reserves a latent space for “Non-interaction" type, whichfurther disentangles possible indirect interactions from direct ones, and therefore slightly promotesits accuracy.
In contrast, the iterative training process in iSIDG creates relaxation of the gradientsduring training.
As a result, it can distinguish most indirect interactions from the direct ones andreconstruct the adjacency matrix more precisely than the other methods on Linear network dataset.
D.3 Mutual information
We conduct another case study to demonstrate the difference of the mutual information between ouriSIDG with and without the iterative process on Linear network dataset with 10 experiments of everymodel with 1400 training epochs.
We utilize nearest-neighbor MI estimator [36] and the plots areshown in Figure 6.
The plots show the mutual information between input features and embeddings inlatent space MI_XZ for every edge, as well as the mutual information between embeddings in latentspace and the output features MI_ZY for every edge.
Since the input consists of node features, weconcatenate them according to [ sender, receiver ] to represent each edge for the calculation of mutualinformation, respectively.
As shown in Figure 6, in both methods, the two types of mutual information increase in the first phase,and then after around 800 epochs of training, the mutual information between embeddings in latentspace and output features decreases, while the other one keeps growing.
These phenomena verify thetwo procedures in the training of deep neural networks as mentioned in the IB theory [42, 1], namelylabel-fitting and representation-compression.
In the phase of representation-compression, when the stop condition is reached, the method withthe iterative process stops the current training round, feeds the obtained adjacency matrix fromlatent space back to the input side of encoder, and starts another round of training with the obtainedadjacency matrix.
The phase-transition is demonstrated by the decrease of both mutual informationand start with another round of label-fitting and representation-compression.
Therefore, the iterativetraining process creates several rounds of relaxation during training, and encourages our iSIDGmodel to learn a more compressed representation in the latent space.
As shown in Figure 5, thecompressed representation in the latent space eliminates most of the indirect interactions and promotesthe structural inference accuracy of our iSIDG model.
D.4 Ablation studies on different encoders and decoders
We conduct another series of ablation studies to demonstrate the different choices of encodersand decoders in our iSIDG model.
We choose four methods as encoders for our ablation study,
[Table] [Table]
namely: multi-layer-perceptrons (MLP), simple message-passing mechanism (MPM), GCN, GIN,and GAT.
We also take three different decoders into account: MLP, avoiding degenerate decoder(MULTI)
[Table] and recurrent decoder (RNN) [22].
The results are shown in Table
[Table] where therows denote the choice of encoder, the columns for decoder [Table] and the results are the average valuesand 95% intervals of 15 experiments with every combination of encoder and decoders.
As shown in
Table 9, the methods with GNNs as encoder are greatly more accurate than the ones with simple MLPencoder.
The only exception are the ones with GAT encoder, which are only marginally superior tothe ones with MLP encoder.
The reason may be that GAT is prone to over-fitting and the attentionweights lack supervision in the cases of structural inference.
When it comes to GIN as encoder, theresults are superior to methods with other encoders by at least 3.2 percent of AUROC.
Surprisingly,methods with MLP decoder, which is the simplest decoder among the three, output the most accuratereconstruction results.
We argue that it may be due to the simple task of the iSIDG: only focusing onstructural inference instead of both structural inference and dynamic prediction (as that of NRI).
E Experiments on Scalability and Real-world Datasets
Although iSIDG works well on small datasets, such as synthetic networks, physical simulations and
NetSim datasets, we are curious about the performance of iSIDG on large datasets.
For this reason,we create another “springs" dataset with 100 nodes (we name it as “Springs100"), and with the samesettings as the details mentioned in Section C.3.
Besides that, we also test our model and baselineson two GRN datasets: ESC dataset [4] and HSC dataset [32]. ESC has 96 nodes and 409 edges,while HSC has 33 nodes and 126 edges.
We ran the experiments for ten rounds, and summarize the AUROC results in Table 10.
As shown in the table, our iSIDG still outperforms baseline methods onthese datasets.
However, we think that iSIDG cannot prove its scalability in a broader view.
The reasons are thefollowing:
Despite the iSIDG can work better than baseline methods on “Springs100" dataset, the AUROConly values 72 .
7% , which makes the inferred results to be highly unauthentic for downstream
tasks.
On ESC dataset, iSIDG can only work better than guessing the existence of edges with equal probability (either exist or not).
So that the results are not reliable.
Compared with the results shown in Section 5, we observe a significant decline, which suggests
that we can extrapolate an even worse inference result on datasets with more nodes.
Because the latent space in iSIDG has a fixed size, which means that we have to adopt a larger latent space for larger graphs.
Thus the scalability of the iSIDG is limited by VAE as well.
[Table] [Table] [Table] Therefore, we suggest that iSIDG can be utilized to infer the structure of dynamical systems with tensof nodes.
We will investigate the problem of scalability in structural inference in our future work,which is also interesting and of practical value.
F Do DAG-structure Learning Methods Work in Our Problem Setting?
We observe that there is a series of works, which deal with the directed acyclic graphs (DAG)structure learning [59, 57, 58].
These works formulate the problem as a continuous optimization witha structural constraint that ensures acyclicity [59], with explicit structural constraints [57], or withimplicit mechanism to force the acyclicity [58].
Although it is claimed by the authors that these workdeal with DAG structure learning, which is different from our problem setting, we are still curiousabout the performance of these works on the problem of structural inference of dynamical systems.
We choose DAG-GNN [57] and DAG-NoCurl [58], which are the representative in the research fieldof DAG structure learning.
We follow the official implementations of these models:
• DAG-GNN: https://github.com/fishmoon1234/DAG-GNN ;• DAG-NoCurl: https://github.com/fishmoon1234/DAG-NoCurl ;
and we only change the data loader modules to load physical simulations, synthetic networks and
NetSims datasets, respectively.
We ran the experiments for ten rounds, and summarize the AUROCresults in Tables 11 and 12, where we note DAG-GNN* as the DAG-GNN with acyclicity constraintset as zero.
As shown in the tables, DAG-GNN, DAG-GNN* and DAG-NoCurl fail to infer theexistence of interactions in dynamical systems.
The reason may be that the existence of cycles in thesystems, which is common observed among dynamical systems, violates the acyclicity assumptionsof these methods.
In particularly, although the acyclicity constraint in DAG-GNN* is set as zero,
DAG-GNN* is unable to correctly infer the structure of our datasets, and (un)surprisingly performseven worse than DAG-GNN.
G Time and Memory Efficiency
We summarized the training time of iSIDG and baseline methods on all of the datasets mentioned inthis work in Tables 13 - 15.
All of the results are the averaged training time of 10 rounds of each
[Table]
[Table]
[Table] method, and is summarized in hours. Unfortunately yet unsurprisingly, iSIDG performs the worstamong all of the methods in almost all of the datasets. On one hand, the iterative process surely has anegative effect on the time efficiency, on the other hand, iterative process encourages our methodto learn the adjacency matrix more accurate than baseline methods on most of the datasets.
It isnotable that ACD has comparable training time to iSIDG, and has comparable accuracy with iSIDGon Kuramoto dataset.
We summarized the memory efficiency of iSIDG and baseline models in Table 16.
As shown inthe table, the number of total parameters of iSIDG is 0.6 M larger than NRI and fNRI, but is lessthan half of the number of ACD, which indicates the moderate memory efficiency of iSIDG amongbaseline methods.
H Broader Impact
Methods such as iSIDG for structural inference of dynamical systems allow for numerous researchersin the field of physics, chemistry and biology to study the interactions inside the systems.
We haveshown that iSIDG works well on either one-dimensional node features or multi-dimensional features,where the features are continuous variables, which proves its wide application.
While the emergenceof the structural inference technology may be extremely helpful for many, it has the potential formisuse.
Potentially, iSIDG can be extended to infer the online social connections via measuringmutual information, which could erode privacy.