Iterative Structural Inference of Directed Graphs
Abstract
In this paper, we propose a variational model, Iterative Structural Inference of
Directed Graphs (iSIDG), to infer the existence of directed interactions from observational agents’ features over a time period in a dynamical system.
First, the iterative process in our model feeds the learned interactions back to encourage our model to eliminate indirect interactions and to emphasize directional representation during learning.
Second, we show that extra regularization terms in the objective function for smoothness, connectiveness and sparsity prompt our model to infer a more realistic structure and to further eliminate indirect interactions.
We evaluate iSIDG on various datasets including biological networks, simulated fMRI data, and physical simulations to demonstrate that our model is able to precisely infer the existence of interactions, and is significantly superior to baseline models
Introduction
A wide range of dynamical systems in the real world can be viewed as the composition of interacting agents, including physical systems [24, 13], multi-agent systems [5, 27] and biological systems [48, 35].
Knowing the interactions of dynamical systems can promote our understanding of the intrinsic mechanisms of these systems, and further facilitate accurate prediction and control of dynamical systems.
However, in many cases, unlike the states of the agents, the interactions are usually not directly observable or measurable.
Therefore, it is necessary to uncover the underlying interactions of dynamical systems based on the observational states of the agents.
In dynamical systems, the states of an agent are affected by the interactions, and the states are usually recorded as a set of continuous variables, which make it difficult to uncover the interactions based on the similarity between the agents.
It is also worth mentioning that most dynamical systems are characterized by directed interactions , such as citation networks [44], social ego networks [31], and gene regulatory networks [35].
We address the revealing of the existence of interactions in the dynamical systems as the problem of structural inference, and model the dynamical system as a directed graph, in which the nodes represent the agents of the system, and the directed edges denote the interactions between the agents.
Thus, the structural inference problem is transformed into the inference of the asymmetric adjacency matrix of the graph based on observational nodes’ features (dynamics).
While several models have been proposed to address this problem with unsupervised variational generative frameworks [22, 51, 30], most of them only deal with undirected interactions, and almost none of them explains why the variational method succeeds in inferring the structure.
In this paper, based on the theory of Information Bottleneck (IB) [47, 1], which states the training of variational auto-encoder (VAE) follows the two-step procedure of deep learning: label-fitting and representationcompression, we explain the intrinsic mechanism of structural inference with VAEs, and propose a novel VAE based machine learning approach, namely i terative S tructural I nference of D irected
G raphs (iSIDG), for iterative inference of the directed graph structure based on observational node features.
Our iSIDG model utilizes a graph neural network (GNN) as encoder and infers adjacency matrix of the graph in its latent space.
The key rationale of our iSIDG model is the iterative process for structural inference, which is designed based on the Variational Information Bottleneck [1].
The iterative process not only feeds the learned structure backward with direction information, but also creates tighter bounds and relaxation during the training process.
The direction information contributes to a more diverse edge representations in the encoder and latent space.
The iterative process connects several rounds of training a VAE into a loop, and joints the second phase: representation-compression with the labelfitting phase of the next round, which creates a tighter bound and a relaxation, to learn a more precise disentanglement between indirect connections and direct ones.
Unlike previous works [22, 51, 30] that totally rely on the bottleneck structure of VAE to infer structure, iSIDG has extra terms to regularize the learned features in the latent space, such as sparsity and connectiveness, which carefully takes the properties of (asymmetric) adjacency matrix into account.
Experimental results show that iSIDG outperforms or matches the state-of-the-art baselines on datasets with directed or undirected graphs.
More importantly, iSIDG can distinguish indirect connections from direct ones more precisely.
Related Work
Neural relational inference (NRI) [22] is the first to utilize a VAE to address the problem of structural inference based on observational node features.
NRI conducts message-passing operations on a fixed fully connected graph structure in its encoder and infers graph structure in its latent space.
Based on
NRI, Webb et al.
[51] propose factorized neural relational inference (fNRI), which extends NRI to multi-interaction systems.
However, fNRI relies on ground-truth graph structure to classify results for each type of interaction.
To deal with more complex systems, Li et al.
[29] and Chen et al.
[9] take various prior knowledge into account, such as sparsity and the distribution of node degrees, to increase the accuracy of structural inference.
Unfortunately, it is quite difficult to acquire prior knowledge in many problem settings.
Alet et al.
[2] propose a modular meta-learning-based framework that jointly infers the structure with higher data efficiency.
However, the framework has to be trained on various datasets beforehand to reach its best performance.
From the aspect of Granger-causality, Löwe et al. [30] propose amortized causality discovery (ACD) method, which infers a latent posterior graph from temporal conditional dependence.
Despite its high accuracy, the ACD method suffers from indirect interactions.
Compared with this line of work, iSIDG introduces an iterative structural inference process to create a tighter bound by updating the fully connected graph structure in the encoder, and encourages edge representations to be diverse with the integrated direction information.
In addition to the aforementioned models to infer the structure of static graphs, several frameworks can also infer the structure of dynamic graphs.
Ivanovic and Pavone [15] present a graph-structured model which combines recurrent sequence modeling and variational deep generative modeling to infer dynamic structure and predict future trajectories.
Li et al.
[26] propose a generic trajectory prediction forecasting network to provide multi-modal trajectory hypotheses and auxiliary structural inference.
The framework in [12] utilizes sequential latent variable models to infer and predict separate relation graphs for every single time step.
Moreover, the generic generative neural system proposed in [27] predicts trajectories based on dynamic graph representation and scene context information.
The frameworks that can infer dynamic graph structures are mostly application-focused, and the structural inference is auxiliary to the task of trajectory prediction.
Besides the models based on variational interface, we would like to mention a few works which deal with the problem of structural inference as well, but based on different methodologies.
ARNI framework [6] infers the latent structure based on regression analysis and a careful choice of basis functions.
Mutual information is also utilized to determine the existence of causal links and thus can infer the structure of dynamical systems [52, 40].
Some approaches fit a dynamics model and then produce a causal graph estimate of the model by using recurrent models [45, 19], or infer the structure by generating edges sequentially [17, 28] and others independently prune the generated edges from an over-complete graph [41].
Xue and Bogdan [54] reconstruct network structures under unknown adversarial interventions.
Although these methods can provide insights about the correlations between adjacent nodes, such as mutual information and causal relation, they either require prior knowledge, or demand the comparison of features from all possible node-pairs, or are sensitive to indirect interactions in the graphs.
There exists another branch of research called Graph Structure Learning, which aims to jointly learn an optimized graph structure and corresponding graph representations for downstream tasks [60, 11, 16].
However, these methods adopt similarity-based methods to optimize the noisy graph structure, which cannot be applied to dynamical systems.
Furthermore, the main focus of these methods is the downstream task, not the reconstruction of the graph structure.
Besides that, there are also a series of work to reconstruct the structure of directed acyclic graphs [59, 57, 38, 58].
However, these works have difficulties to work on graphs with cycles (and self-loops), which are commonly observed among dynamical systems.
Last but not least, in the research field of biology, there are some work which can infer the structure of gene regulatory networks [25, 20, 7].
But these works either require prior assumption or cannot be extended to dynamical systems whose agents have multi-dimensional features.
Unlike these works, iSIDG focuses on the structural inference of dynamical systems, and can deal with cycles in the system and high-dimensional node features.
Preliminaries
Notations and problem formulation.
We view a dynamical system as a directed graph, where the agents of the system are represented as the nodes of the graph, and the directed interactions between the agents as the edges.
We note the directed graph as G =
( V , E ) , in which V represents the set of n nodes: { v i , 1 ≤ i ≤ n } , and E represents the set of edges: ( v i , v j
) ∈ E ⊆
V × V .
Based on the set of edges, we derive to an asymmetric adjacency matrix A ∈ R n × n , in which a ij ∈ { 0 , 1 } indicates the presence ( a ij = 1 ) of the edge between v i and v j or not ( a ij = 0 ).
The edges in our work are assumed to be static and do not evolve with time.
The feature recordings of the nodes over a time period are represented as V =
{ V t , 0 ≤ t ≤ T } , where T is the total number of time steps , and
V t is the set of features of all the n nodes at time step t : V t = { v t 0 , v t 1 , . .
. , v tn } .
Given M sets of feature recordings {V k , 0 ≤ k ≤ M } , the structural inference problem we consider in this paper is to reconstruct the asymmetric adjacency matrix A of the graph in an unsupervised way.
We state the problem of structural inference as searching for a combinatorial distribution to describe the existence of the edges between any of the node pairs in the graph.
As shown in Figure 1, we utilize a GNN based VAE [21] as the cornerstone, which can be trained to approximate complicated and high-dimensional probability distributions from the sampled node features [37].
We design an iterative process based on the IB theory, and feed the direction information, which is computed from the learned structure in the past training procedures, back to the input side, to encourage a more diverse learned representations, and to create a tighter bound as well as a relaxation of the training.
Graph neural networks.
Graph neural networks (GNNs) are powerful deep learning algorithms to learn the representations of nodes and graphs.
Most modern GNNs are designed based on the neighbourhood aggregation strategy [14], where the representation of a node is updated by aggregating the representations of its neighbours.
In brief, we denote h ( k ) i as the representation of node v i at the k -th layer, and the mechanism of these GNNs can be described as:
[Equation]
[Equation]
m ( k
[Equation]
where N ( v i ) denotes the set of chosen nodes related to v i (e.g., its neighbours), UPDATE denotes the arbitrary function in the layer to update node representation, and AGGREGATE represents the aggregation function in the layer.
GCN [23] updates node representation based on one-hop neighbours and also its own representation in the previous layer, and calculates new representations with ReLU.
GIN [53] aggregates the representations of adjacent nodes and node’s own representation in the previous layer as a weighted summation, and utilizes multilayer perceptron to update node representations.
GAT [49] introduces attention mechanism to implicitly specify different weights to adjacent nodes, and then updates representations upon the weighted summation.
GNNs are efficient in learning rich representations for nodes and graphs.
As a result, we utilize GNNs in the encoder to firstly learn representations of the nodes, and later on learn edge representations from the node representations in iSIDG.
Information bottleneck.
The theory of Information Bottleneck (IB) provides an explanation and understanding of learning with deep neural networks [47, 46, 42].
In general, for the input data X and its label Y , the IB aims to learn the minimal sufficient representation Z = arg min
Z I ( Z ; X ) − u · I ( Z ; Y ) , where u is the Lagrangian multiplier to balance sufficiency and minimality.
The training epochs of standard deep learning can be divided into two phases: label-fitting and representationcompression.
These two phases can be characterized by the mutual information between the input features and the learned representation of a layer
I ( X ; Z ) , as well as between learned representation and the label variables I ( Z ; Y ) .
In the label-fitting phase, both I ( X ; Z ) and I ( Z ; Y ) increase, but
[Figure]
Alemi et al.
[1] present a variational approximation to the IB theory (VIB), and prove VAE objective is a special case of the variational approximation.
Under a fully unsupervised setting, VAEs learn the minimal sufficient statistics from the input features, which are sufficient to derive the output features, and keep the most compressed and abstractive representations in their latent space.
We utilize a VAE to infer the structure of the graph in its latent space, which is minimally sufficient to predict the future states of the nodes together with present node states.
It is suggested that IB iterations with stochastic relaxation methods may boost training [42].
It is also mentioned that a relaxation process is helpful to reduce local ambiguity and achieve global consistency [56], as already proved in previous work [8, 55].
Therefore, we design an iterative process with stochastic gradient descent (SGD) based optimizer to create a tighter bound and a relaxation for the training, and to encourage iSIDG to further disentangle dynamics from the adjacency matrix.
In order to verify the effectiveness of this iterative process, we conduct a case study in Section D.3.
Model Design 4.1 Structural Inference by Information Bottleneck
Together with node features, graph structure characterizes the basic information of a graph, which represents a dynamical system.
The goal of structural inference is to accurately reconstruct the adjacency matrix A of the directed graph based on the recordings of node features: A ← { V t , 0 ≤ t ≤ T } .
Interestingly, for dynamical systems, the future features of the nodes are deterministic by their previous features and the connectivity: V t +1 ← { V 0 , V 1 , . .
, V t ; A } .
We assume the Markovian assumption works in our case, and thus previous statement can be simplified to: V t +1 ← { V t ; A } .
Thus, we derive to an assumption: the information about the adjacency matrix is implicitly included in the node features over a time period, which also conforms to the IB theory.
Based on VIB, we derive the Information Bottleneck for Structural Inference , which aims at inferring the adjacency matrix A in the latent space Z :
[Equation]
Interestingly, A is also required in the first term in Equation 2, which is unrealistic in our problem setting.
Thus, we assume a fully connected graphs structure A (0) as input, which is not less informative than the actual sparse adjacency matrix:
[Equation]
Following the statement in VIB [1], we adopt a VAE [21] to deal with the problem described in Equation 3, where an encoder q ϕ ( Z | V t , A (0) ) approximates I ( Z ; V t , A (0) ) , and a decoder p
θ
( ˆ V t +1 | Z , V t ) approximates I ( Z ; V t +1 ) .
ϕ
and θ represent the parameters of encoder and decoder, respectively.
We set the latent space of the VAE with the size of n × n .
The fixed dimension of the latent space restricts the size of the search space for the adjacency matrix.
We utilize a weighted
ELBO with multiple integrated regularization terms to optimize model parameters { ϕ, θ } .
4.2 Encoder
The encoder in our model is tasked with inferring the existence of directed interactions z ij between all node-pairs by extracting the most compressed representation given the node features V .
The encoder q ϕ ( Z | V t , A (0) ) returns a set { z ij | v i , v j ∈ V} , which features the distribution to describe how probable the edge from node j to node i exists: p ( z ij ) ∈ [0 , 1] .
The prior p ( Z ) = (cid:81) i,j ∈V p ( z ij ) is a factorized uniform distribution.
We utilize GNNs to extract features in the encoder.
But the GNNs learn new representation based on both node features and the topology of the graph f GNN ( V , A ) , in which the adjacency matrix A is our learning target.
To encounter this problem, we impose an iterative learning process on the adjacency matrix, and we discuss the details in Section 4.4.
For simplicity, in this section we use ˆ A to represent the adjacency matrix.
In general, the encoder:
[Equation]
where f enc represents the encoder.
The basic setup of our encoder is similar to that in [22], which applies two rounds of node-to-edge and one round of edge-to-node message passing operations to learn the edge embeddings h ij :
[Equation]
where f (1) embed is the embedding network for input features, f 1 e and f 2 e are node-to-edge messagepassing networks, and f v is the edge-to-node operation.
We observe that Equations 5 - 7 is a round of message passing for node features in most GNNs, as a result, we substitute Equations 5 - 8 as:
[Equation]
so we can integrate state-of-the-art GNNs such as GCN, GAT and GIN into encoder.
In this work, we integrate GIN into the encoder.
As a consequence, our encoder is designed to learn informative features from input node features and compress them to fit the restricted search space, which has the dimension of a fully connected graph.
By extracting information from the high-dimensional but low-level node features to the restricted and low-dimensional latent space, our encoder disentangles the dynamics and approximates the minimally sufficient statistics, the adjacency matrix.
After that, we model the existence of edges as Z ∈ R n × n , where its component z ij comes from q ϕ ( z ij | V t , ˆ A ) =
softmax ( h ij ) .
4.3 Decoder
The task of the decoder is to predict the future states V t +1 based on the samplings from latent space
Z and the present node features V t .
From the IB theory, if the encoder derives minimally sufficient statistics as the adjacency matrix, the decoder can precisely predict the future node features.
However, there remains the problem of how to design the decoder, so that it can further encourage the encoder to disentangle dynamics and to infer the graph structure only.
In order to deal with this problem, we design the decoder to imitate a single message-passing operation.
We feed our decoder with the present node features besides the samplings from the latent space to predict the future node features.
As a consequence, the node features are disentangled from the compressed information in the latent space during training.
In short, the decoder deals with the following problem: p θ ( V t +1 | V t , Z ) .
We use a two-stage decoder, which consists of a message-passing layer and an embedding layer:
[Equation]
where f (2) embed and f (3) embed are multilayer perceptrons (MLPs) to embed the features.
4.4 Iterative Learning with Direction Information
The motivation of integrating iterative learning process into the structural inference consists of three aspects: (1) to create a tighter bound for Equation 3; (2) to create a relaxation for the training process; (3) to emphasize direction information.
In order to formulate the ideas more precisely, we would like to firstly describe the iterative process with direction information in our model.
From the very beginning, we have no prior knowledge about the structure of the graph.
In order to utilize GNNs to extract and compress the features into a restricted search space for adjacency matrix, we feed the encoder with a fully connected graph A (0) = 1 ∈ R n × n .
After η rounds of training, we obtain an asymmetric adjacency matrix A ( p ) from the latent space, which is calculated from the averaged result of the entire training samples: [Equation]
[Equation]
where m is the total number of training samples, and M is the set of training data.
The full-batch average operation not only preserves the most information from the training set, but also makes the
A ( p ) to be invulnerable to outliers.
We then normalize A ( t ) in both row-wise and column-wise:
[Equation]
and [Equation] (12) where D − 1 in and D − 1 out are in-degree and out-degree matrices which are calculated based on A ( p ) , respectively.
We name Equations 12 as direction information .
After that, we combine the direction information matrices with the normalized initial fully-connected adjacency matrix, respectively:
[Equation]
where L is the normalized adjacency matrix of the initial fully-connected graph A (0) , and λ is the combination coefficient.
Since A (0) is symmetric, we can use either column-wise or row-wise normalization.
We map the results in Equation 13 to binary sets with the characteristic function:
[Equation]
where κ ∈ { row, col } , and ξ is the threshold value.
Finally, we feed ˜ R ( p ) κ back to the input side of encoder mentioned in Section 4.2 by substituting the ˆ A in Equations 6 and 8.
More formally, the two feature vectors which are concatenated in Equations 6 and 8 represent the sender and receiver of the edge, respectively.
We rewrite the equations for the iterative process according to ˜ R ( p ) κ :
[Equation]
where k ij ∈ { e ij , h ij } , f n e ∈ { f 1 e , f 2 e } , and k n ∈ { e 1 n , e 2 n } in Equations 6 and 8, respectively.
We then train iSIDG with the updated adjacency matrices ˜ R ( p ) κ .
The stop condition for iteration is:
[Equation]
[Equation] where superscript [ i ] represents the rounds within current iteration.
Once this condition is satisfied, the adjacency matrix at the input side of the encoder will be updated according to Equations 11 - 15.
Tighter bound.
During training, the initial fully connected adjacency matrix A (0) is updated to sparser matrices { ˆ A ( p ) κ , κ ∈ { row , col }} .
(We omit the description for κ in this paragraph for clearer statement).
The model is able to roughly detect which edges are not beneficial to the future state prediction, and eliminates these edges in the new representation.
As a consequence, any of the matrices in { ˆ A ( p ) κ } is sparser than A (0) .
Thus the first term in Equation 3 is re-framed as
I ( Z ; V t , { ˆ A ( p ) κ } ) , which obviously supports: I ( Z ; V t , { ˆ A ( p ) κ } ) ≤ I ( Z ; V t , A (0) ) , and results in a tighter bound for Z in Equation 3.
Relaxation.
Besides that, our iterative training process imposes relaxation to connect the two phases, i.e., label-fitting and representation-compression, of training a deep learning network as a cycle.
In the first phase, our model for structural inference tries to fit the target for prediction, when the training errors become small, it turns to the second phase and compresses the representation of any layer to be minimally sufficient to fit the target.
In the second phase, the compression process affects the adjacency matrix in latent space as well, and due to the difficulty of training in an unsupervised manner, it is very apt to reach a local optimal.
The iterative process, which feeds the learned adjacency matrix back to the encoder, stops the phase of representation-compression, and starts the next round of label-fitting and representation-compression.
At this round, the combination of direction information and the initial adjacency matrix together creates a different adjacency matrix for the encoder compared with the previous round, which will relax the learning of adjacency matrix back from the local optimal.
We verify the effectiveness of the iterative process through mutual information in Section D.3.
Direction information.
One of the main differences between undirected and directed graphs is that the former can be described with a unified degree matrix, but the latter requires two matrices for in-degree and out-degree.
As a result, we introduce direction information by integrating the in- and out-degrees of every node with the new adjacency matrices, which contributes to a bifurcated training process and encourages the learned embeddings for edges to be more diverse.
The integration of direction information is beneficial to the structural inference for directed graphs, which is validated experimentally in Section 5.1.
4.5 Training with a Hybrid Loss
Although the iterative process is an effective way to learn the adjacency matrix of directed graphs, adequate loss function can promote the training process as well.
Firstly, the following is the loss function for prediction:
[Equation]
Then the Kullback–Leibler ( KL ) divergence acts as a regularization for the latent space:
[Equation]
We omit the superscripts of vectors V and v in Equations 17 - 21, the actual correspondence refers to
Sections 4.2 and 4.3.
In addition to the aforementioned terms which are often utilized as the objective function for VAEs, we introduce extra regularization terms.
The regularization term of signal smoothness takes both inferred structure and node features into consideration to eliminate any connection which does not fulfill the smoothness assumption of adjacent node features, such as indirect interactions.
We consider input features V as graph signals, and the assumption for graph signals also works here: values change smoothly across adjacent nodes [10].
Therefore, we introduce Dirichlet energy [3] to measure the smoothness between the signals:
[Equation]
It is worth mentioning that in most cases the adjacency matrix is asymmetric, we cannot use Laplacian matrices to simplify the formulation.
Moreover, we introduce more regularization terms [3, 18] to feature the learned features in latent space with the properties of realistic adjacency matrices, such as connectiveness L d and sparsity L s .
Training the network with these regularization terms encourages our model to further disentangle [Table] node representations from the latent space and distinguish indirect interactions from direct ones:
[Equation]
where the logarithmic barrier in L d forces the model to learn a connected adjacency matrix [Table] but lacks the regularization of sparsity.
In consequence [Table] we introduce L s to regularize its sparsity. To conclude
[Table]
we train our iSIDG with the hybrid loss L :
[Equation]
(21) where { µ, α, β, γ } are hyperparameters.
The first two terms in Equation 21 formulate a weighted
ELBO, which is the training objective for VAEs.
The rest of the terms are regularization for the learned adjacency matrix in latent space.
Together with the training cycles of fitting and compression, which are connected with relaxations, the hybrid loss function further encourages our iSIDG model to reconstruct the adjacency matrix precisely.
We give a detailed discussion on the design methodology of iSIDG on how to effectively eliminate indirect connections in Section A.
Experiments
We test our iSIDG model on three different datasets, and present its ablation study on one of the datasets.
Implementation details and further experimental results can be found in Sections B - F.
5.1 Structural Inference on Different Systems
Datasets.
We test our model on the six directed synthetic biological networks [35], namely Linear (LI),
Linear Long (LL), Cycle (CY), Bifurcating (BF), Trifurcating (TF), and Bifurcating Converging (BFCV) networks.
These networks are essential components that lead to a variety of different trajectories that are commonly observed in differentiating and developing cells [39].
We use BoolODE [35] to simulate the process of developing cells with these synthetic networks, and obtain six groups of trajectories.
The features at every node are one-dimensional, i.e., the level of mRNA expression.
We also test our model on NetSim datasets [43] of simulated fMRI data.
The NetSim datasets consist of simulated blood-oxygen-level-dependent imaging data across different regions within the human brain, which leads to asymmetric relation networks.
The node features are one-dimensional.
The last datasets to be mentioned here are three physical simulations mentioned in [22], namely springs, charged particles and phase-coupled oscillators (Kuramoto model).
We keep the symmetric interactions in the setting of these networks.
However, different from the setting mentioned in that work, we sample the trajectories with fixed interactions between the agents in the system, but with different initial conditions, in order to simulate the real application scenario on the observational data from a specific unknown system.
The features at every node are 4-dimensional.
Baselines and metrics.
We compare iSIDG with the state-of-the-art models for structural inference:
• NRI [22]: a VAE-based model for unsupervised relational inference.
• fNRI [51]: an NRI-based model with additional latent space for every factorized interaction type.
• MPIR [52] a model based on minimum predictive information regularization.
• ACD [30]: a variational model that leverages shared dynamics to infer causal relations.
We describe the implementation details of the baseline methods in Section B.2.
We demonstrate our evaluation results with the following metrics: the area under the receiver operating characteristic (AUROC), the area under the precision-recall curve (AUPRC) and the Jaccard similarity index (JI).
We also provide results of two DAG-based structure learning methods on our datasets in Section E. [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
Results. The experimental results of our iSIDG model and baseline methods are summarized in
Tables 1 and 2 (also Tables 6 - 9 in Appendix), which consist of mean values and 95% intervals of
AUROC, AUPRC and JI from 15 experiments on each, respectively.
Due to the alternative simulation pipelines between ours and other literature, we obtain different results of ACD and MPIR on physical simulations from other literature.
iSIDG outperforms or matches the baseline methods.
In the datasets of synthetic networks and three NetSim datasets, which consist of directed graphs and low dimensional input features, iSIDG outperforms baseline models by at most 13.2 percent in terms of AUROC, showing that it can infer structure more precisely than the baseline methods.
These results highlight the validity of integrating the iterative training process and the direction information on structural inference problem for directed graphs.
It is also remarkable that iSIDG matches or is slightly inferior to the baseline methods in the datasets of physical simulations, especially fNRI and
ACD.
The reason may be that high dimensional input features (4-dimensional features for physical simulation datasets) contain richer dynamics for fNRI and other methods to infer from.
It is worth mentioning that fNRI method utilizes ground truth to distinguish every latent space and match them to the edge type with the highest accuracy, which results in higher AUROC, AUPRC and JI scores.
Moreover, since ACD relies on the information shared by input features, the richer information contained in high dimensional features promotes a more promising result.
5.2 Ablation Study
We conduct ablation studies on the effectiveness of regularization terms in objective function: signal smoothness Ω , connectiveness L d and sparsity L s , and the reconstruction results are shown in terms of heat maps in Figure 2.
The heat maps visualize the average results of 15 experiments of iSIDG with different objective functions by deleting one regularization term, respectively.
As shown in the second plot in Figure 2, without considering signal smoothness, the indirect interactions emerge again in the results of structural inference with the correctly inferred edges.
The third plot in Figure 2 shows that without considering the connectiveness, the model correctly infers only a few direct interactions but as isolated “islands", and connects these islands with numerous indirect interactions.
The last plot in Figure 2 demonstrates the results that without the terms to regularize the sparsity, many direct interactions are substituted by the indirect ones.
Therefore, the extra regularization terms in the objective function of iSIDG effectively encourage the model to infer realistic graph structures and to eliminate the influence of indirect interactions.
More ablation studies can be found in Section D.4.
5.3 Experiments on Scalability and Real-world Datasets
Although iSIDG works well on small datasets from simulations, such as synthetic networks, physical simulations and NetSim datasets, we are curious about the performance of iSIDG on large datasets and real-world datasets. For this reason, we create another “springs" dataset with 100 nodes (we name it as “Springs100"), and with the same settings as the details mentioned in Section C.3. Besides that, we also test our model and baselines on two gene regulatory network (GRN) datasets: ESC dataset [4] and HSC dataset [32].
As a brief description about the GRNs: ESC has 96 nodes andedges, while HSC has 33 nodes and 126 edges.
We ran the iSIDG and baseline methods on three datasets for ten rounds, and summarize the averaged AUROC results in Table 3.
As shown in the table, our iSIDG still outperforms baseline methods on all of the three datasets.
[Table]
However, we think that iSIDG cannot prove its scalability in a broader view: 1.
Despite the iSIDG performing better than baselines on “Springs100" dataset, the AUROC only values 72 .
7% , which makes the inferred results to be highly unauthentic for downstream tasks.
On ESC dataset, iSIDG can only work better than guessing the existence of edges with equal probability (either exist or not).
So that the results are not reliable
Compared with the results shown in Section 5, we observe a significant decline, which suggests that we can extrapolate an even worse inference result on datasets with more nodes.
Because the latent space in iSIDG has a fixed size, which means that we have to adopt a larger latent space for larger graphs.
Thus the scalability of the iSIDG is limited by VAE as well.
Therefore, we suggest that iSIDG can be utilized to infer the structure of dynamical systems with tens of nodes.
We will investigate the problem of scalability in structural inference in our future work, which is also interesting and of practical value.
We so far only consider the structural inference for static and (relatively) small graphs, though we tested the scalability of iSIDG on graphs of 100 nodes.
Future research includes introducing dynamic interactions and more scalable model design which remains as a critical challenge in structure learning for dynamical systems.
Furthermore, we also want to integrate prior knowledge such as partial structural information to further promote the performance of iSIDG.
Acknowledgments and Disclosure of Funding
This work is partially supported by the Audacity project GENERIC of Institute for Advanced Studies (IAS) at University of Luxembourg.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See abstract and last paragraph of Section 1.
(b) Did you describe the limitations of your work?
[Yes] Due to page limitation, we describe the limitation of our work with a few words in Section 5.3, and in Section 6. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section G. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the ethics review guidelines and ensured that our paper conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sectionsand 4.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs are quite intuitive, see Section 4.1.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We attach the link to our code and data in the supplementary documents.
We also give instructions about how to reproduce the main experimental results in Section B. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sections 5 and B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Instead of error bars, we report our experimental results with the mean value and 95% intervals of the results from running the experiments several rounds.
We describe the method to obtain the data in Section 5, and the results in Tables 1, 2 and 6- 9.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section B.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.1.
(b) Did you mention the license of the assets?
[Yes] We mention the license in our code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We include new assets, and they can be found via a URL.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] The datasets are generated with simulations, which has nothing to do with personal data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] The data is neutral to biological human being.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Discussion A.1 Indirect connections
Originally, iSIDG starts training with a fully connected graph structure:
A (0) .
Based on the problem setting of dynamical systems, we assume A (0) can be decomposed as: A (0) =
{ A D , A IN , A U } , where A D represents the actual edges in the graph (directed connections), A IN denotes the indirect connections between the two actual connected nodes, and A U is the set of non-connections.
[Figure]
Previous structural inference methods, such as NRI, fNRI and ACD, are good at eliminating A U in the inference results.
However, as shown in Figure 3, these methods may falsely reconstruct the structure with indirect connections.
It is interesting that the indirect connections resulted from the transmission of signals between nodes.
In Figure 3, there exists a chain connection from node vto v 3 through v 2 , and in a dynamical system, the changes in v 1 will finally affect v 3 , which would mislead the structural inference methods to falsely infer an edge from v 1 to v 3 .
Moreover, for a sparse graph, it is possible that A IN ∩ A U ̸ = ∅ , which challenges structural inference methods, such as
NRI, fNRI and ACD.
A.2 Solution for Eliminating Indirect Connections
Our solution to eliminate indirect connections in the inference results consists of three methodologies: (1) VAE ; (2) Iterative process ; and (3) Regularization terms .
VAE.
As shown in previous works [22, 51, 30], the bottleneck structure of VAEs can correctly identify many non-connections in the reconstructed adjacency matrix.
Thus, we also rely on the VAE to firstly identify as many non-connections as possible before the first round of iterative process, and sample { ˆ R ( p ) row ,
ˆ R ( p ) col } according to operations shown in Equations 11 - 14.
Iterative process.
The iterative process follows the methodology of VIB, and is mentioned in
Section 4.4 with a new objective function:
[Equation]
The iterative process is characterized by a main advantage: the first term in Equation[-
-]
I ( Z ; V t , { ˆ A ( p ) κ } ) <
I ( Z ; V t , A (0) ) , which results in a tighter bound than Equation 3.
In other words, after feeding the direction information back to the input side of the encoder, the new search space excludes the non-connections, which are found by VAE in the previous training epochs.
With a smaller search space, iSIDG can focus on the differentiation between misleading indirect connections and the real connections.
Regularization terms.
Although feeding the learned adjacency matrix back to the encoder can eliminate the indirect connections, we need regularization terms to secure and accelerate this process.
The sparsity term L S encourages iSIDG to choose the more solid path from every node, to meet the minimal sufficient statistics assumption of IB.
For example, in Figure 3, at node 1 , iSIDG has to choose between the path 1 → 3 and 1 → 2 → 3 based on the sparsity regularization terms and the minimal sufficient information for dynamics prediction in decoder.
One may argue that it is also possible that iSIDG will falsely think there are edges from node 1 to node 2 , and from node 1 to node 3 .
However, this does not conform to the future state prediction.
The task of the decoder is trying to approximate:
[Equation]
where f ( · ) denotes the interaction from node v j to node v i .
We may imagine the interaction could be
RNA activation level in GRN, spring force in spring-balls system, and the phase-couple of oscillators.
As a result, the interactions between nodes are assumed to be the same, and described with the same function.
Yet node 1 can only affect node 3 through node 2 , which results in a superposition of functions: f ( f ( · )) .
Therefore, both edges from node 1 to node 2 , and from node 1 to node 3 cannot predict the future state of node 3 correctly, and VAE will search for other possible edges.
B Further Implementation Details B.1 Implementation details of iSIDG
We summarize the described architecture of iSIDG and present the pipeline of training iSIDG in
Algorithm 1.
( V T , Z , θ ) 13: if i > η and StopCond then 14: ˜ R ( i ) κ ← { Z , A (0) , ξ } using Eqs.
A ( i ) ← ˜ R ( i ) κ 16:
L p ← { ˆ V T +1 , V T +1 , δ } 17:
L KL , L d , L s ← { Z , n } 18:
L ( i ) ← L p + µ L KL + α Ω( A ( t ) , V ) + β L d + γ L s 19:
L ← L ( i ) /T 20:
Back-propagate L to update model weights ϕ
and θ 21: i ← i +22: end if 23: end while 24: Return: A ( t )
1: Input: features input 2: x = elu(Linear1( input )) 3: x = dropout(x) 4: x = elu(Linear2(x)) 5: out = batch_norm(x) 6: Return: out
1: Input: features input 2: x = mlp1( input ) 3: x = node2edge(x) 4: x = mlp2(x) 5: x_skip = x 6: x = edge2node(x) 7: x = mlp3(x) 8: x = node2edge(x) 9: x = concatenation([x, x_skip]) 10: x = mlp4(x) 11: out = fully connected out(x) 12: Return: out
Basic settings.
We implement our iSIDG model in PyTorch [33] with the help of Scikit-Learn [34] to calculate various metrics.
We run experiments on a single NVIDIA Tesla V100 SXM2 graphic card, which has 32 GB graphic memory and 5120 NVIDIA CUDA Cores.
During training, we set batch size as 128 for datasets which have less than 10 nodes, for those more than 10 nodes, we set batch size as
We train our iSIDG model with 2000 epochs on every dataset, which takes around 30 - 42 hours for every training session.
The code can be found at https://github.com/AoranWANGRalf/iSIDG .
Loss Functions.
The hybrid loss function shown in Equation 21 consists of five different terms to be calculated: {L p , L KL , Ω( Z , V ) , L d , L s } .
We would like to describe the actual implementations of L p and L KL .
Since the target of the decoder is the prediction of future node features based on present states, L p is estimated by:
[Equation]
where the variance σ is set as a fixed value, and the const is e − 16 .
And the calculation of KL-divergence L KL over a uniform prior can be simplified as the calculation of entropy H :
[Equation]
Hyper parameters.
The choice of most hyper parameters in our iSIDG model (such as those mentioned in Algorithm 1) may refer to Table 4.
In addition to the hyper parameters mentioned in the table, we set the combination coefficient λ in Equation 13 as 0.3 for the training on all of the datasets.
We argue that λ can be set as varying values according to the training epochs, and decrease during training, which may be helpful for faster converge.
We would like to carry on more research onto this hypothesis in the future.
Besides that, we describe the choice of time step length T in Section C.
Time step length T decides the amount of samplings along a trajectory.
[Table]
[Table] [Table]
Particles
[Table]
Ablation study with different GNNs.
The main difference between the utilization of different GNNs takes place in the encoder of iSIDG.
For the implementation of MLP encoder, we use three stacked multi-layer perceptron blocks, with each block consists of the units shown in Algorithm 2.
We place a concatenation operation between Block 1 and 2 to get edge representations.
The last block is attached with another Linear layer with its dimension matching the number of edges in a full-graph setting.
[Table]
The number of units in the Linear layers in the MLP encoder is shown in Table 5.
The dropout rates in all of the blocks are set as 0 . 5 .
For the simple message-passing mechanism encoder ( MPM encoder), we follow the design of “MLP
Encoder" in NRI [22].
And can be 10: out connected out 11: Return: out Based on the design of Graph Identity Networks and the MPM encoder, we combine these idea together into the GIN encoder, by replacing the operations from line 3 to line 6 in Algorithm 3 with the GIN operation mentioned in [53], but still keep the skip connection for edges representations, and the pseudocode is shown in Algorithm 4.
Similarly, if we replace the GIN operation in lineof Algorithm 4 with GCN operation, we obtain the GCN encoder.
Moreover, the design of GAT encoder is similar to that of GIN encoder and is shown as the pseudocode shown in Algorithm 5.
B.2 Implementation details of baselines
NRI.
We use the official implementation code by the author from https://github.com/ ethanfetaya/NRI with customized data loader for our chosen datasets: synthetic networks, NetSims and physical simulations.
We add our metric-evaluation in “test" function, after the calculation of accuracy in the original code. fNRI.
We use the official implementation code by the author from https://github.com/ekwebb/ fNRI with customized data loader for our chosen three datasets.
We add our metric-evaluation in “test" function, after the calculation of accuracy and the selection of correct order for the representations in latent spaces in the original code.
MPIR.
We use the official implementation code from https://github.com/tailintalent/ causal as the code for MPIR.
We run the code by customized data loader for the chosen three datasets.
After the obtain of the results, we run another script to calculate the metrics.
ACD.
We follow the official implementation code by the author as the framework for ACD ( https://github.com/loeweX/AmortizedCausalDiscovery ).
We run the code with customized data loader for the chosen three datasets.
We implement the metric-calculation pipeline in the “forward_pass_and_eval()" function.
C Further Details about Datasets C.1 Synthetic networks
The six directed Boolean networks (LI, LL, CY, BF, TF, BF-CV) are the most often observed fragments in many gene regulatory networks, each has 7, 18, 6, 7, 8 and 10 nodes, respectively.
Thus by carrying out experiments on these networks, we can get acknowledge about the performance of the chosen methods on the structural inference of real-world biological networks.
We collect the six ground-truth directed Boolean networks from [35] and simulate the single cell evolving trajectories with BoolODE [35] https://github.com/Murali-group/BoolODE with default settings mentioned in that paper for every network.
We sample 12000 trajectories and group them into three datasets: for training, for validation, and for testing, each with the number of 8000,and 2000, respectively.
Then we sample 49 snapshots with equal time interval in every trajectory and save them as “.npy" files for data loading.
C.2 NetSim datasets
The NetSim datasets simulate blood-oxygen-level-dependent imaging data across different regions within the human brain and is described in [43] and https://www.fmrib.ox.ac.uk/datasets/ netsim/ .
We target at inferring the existence of directed connections between different brain areas.
Among the total 28 datasets in NetSim, we choose the first three datasets (NetSim1, NetSim2 and
NetSim3) which have 5, 10, and 15 nodes, respectively.
We sample 49 snapshots on each trajectory with equal interval and randomly group them into three sets for training, validation and testing with the ratio of 8: 2: 2, respectively.
C.3 Physical simulations
To generate these physical simulations (springs, charged particles and phase-coupled oscillators), we follow the description of the data in [22] but with fixed interactions.
To be specific, at the beginning of the data generation for each physical simulation, we randomly generate a ground truth graph and then simulate 12000 trajectories on the same ground truth graph, but with different initial conditions.
The rest settings for the simulations are the same as that mentioned in [22].
It is worth mentioning that the connections in the physical simulations are indirected, which are different from those in the synthetic networks and NetSim datasets.
We collect the trajectories and randomly group them into three sets for training, validation and testing with the ratio of 8: 2: 2, respectively.
D Further Details about Experiments D.1 Metrics
We choose the three most representative metrics from 0-1 classification for the evaluation of structural inference results, namely the area under the receiver operating characteristic (AUROC), the area under the precision-recall curve (AUPRC) and Jaccard index (JI).
All of the metrics are calculated from the inferred structure and the full-patch of ground-truth.
AUROC.
The area under the receiver operating characteristic (AUROC) is a performance metric to evaluate the result of classification tasks.
It demonstrates the model’s ability to discriminate between cases (positive examples) and non-cases (negative examples).
For our case, it is used to make clear the model’s ability to distinguish actual edges and empty interactions.
It is calculated as the area under the receiver operating characteristic curve (ROC), which sets false positive rate (FPR) as x-axis and true positive rate (TPR) as y-axis at various threshold settings.
In the ROC space, the best possible prediction method would yield a point in the upper left corner or coordinate (0,1) of the
ROC space, representing 100% no false negatives and 100% no false positives.
Then a model outputs more precise results with many TPR yields a higher AUROC value, and vice versa.
AUPRC.
The area under the precision-recall curve (AUPRC) is a useful performance metric for imbalanced data in a problem setting since we care a lot about finding the existence of interactions between nodes.
It shows the ability of the model to find the existence of all ground-truth edges without accidentally marking any non-interaction as existence.
The AUPRC is calculated as the area under the Precision-recall (PR) curve.
The x-axis of a PR curve is the TPR and the y-axis is the precision.
This is in contrast to ROC curves, where the y-axis is the TPR and the x-axis is FPR.
Similar to calculate AUROC, AUPRC is also calculated from various threshold settings.
Jaccard index.
The Jaccard index (sometimes called the Jaccard similarity coefficient) (JI) compares the inferred edges and ground-truth to see the shared results and the distinct results.
Jaccard index is a measure of similarity for the two sets of data, with a range from 0% to 100%.
The higher the percentage, the more similar the two populations.
It is calculated as "Intersection of two sets / Union of two sets".
Although JI is easy to calculate and interpret, the result is extremely sensitive to small samples sizes and may produce erroneous results, especially when the number of samples is small or the data sets contain missing observations.
[Table] [Table]
[Table] [Table]
D.2 Further experimental results
In this section, we demonstrate additional experimental results as the supplement to Section 5.
We present AUPRC and JI values as well as a case study about the structural inference results of iSIDG and baseline methods.
AUPRC and JI results.
We present the AUPRC and JI results of iSIDG and baseline methods on synthetic networks, NetSim datasets and physical simulations in Tables 6 - 9.
The results are the mean values and 95% intervals of 15 experiments, respectively.
Similar to the AUROC results presented in Section 5.1, iSIDG performs the best among all of the chosen methods in datasets with directed graphs, and can match the best baseline method in datasets with indirected graphs.
Besides that, we conduct a case study of iSIDG with different objective functions, and calculate the average accumulated path lengths based on 15 experiments of each on Linear network dataset.
The results are shown in Figure 4.
It is clear that our choice of objective function with regularization terms of smoothness, connectiveness and sparsity successfully encourages iSIDG model to infer the structure of the system with less indirect interactions and more accurate results. [Figure]
N
Full
No Ω
[Figure] [Figure]
Case study.
To gain an intuitive understanding of the results of the structural inference of directed graphs [Figure] we conduct a case study on the Linear network dataset.
The reconstructed adjacency matrices of our iSIDG, fNRI and ACD methods on Linear network dataset are visualized in Figure [Figure] as well as the ground-truth.
The results are the mean results of 15 experiments of each method and are shown in terms of distributions of reconstructed relations.
In the figure, 0 .
at a point [ x, y ] denotes the absence of interaction from x to y [Figure] and 1 . 0 represents in an opposite way.
As shown in
Figure 5, compared with the ground-truth, the structure reconstructed by ACD contains many indirect
[Figure]
connections.
It is possibly the result from the fact that ACD infers a latent posterior graph from the input features based on hidden confounding [50].
For Linear network dataset, which has a “chain" structure, it may confuse ACD method as it contains many indirect interactions, and thus leads to inaccurate inference results.
Moreover, fNRI utilizes a factorized latent space for every possible interaction types.
For the case here, it even reserves a latent space for “Non-interaction" type, which further disentangles possible indirect interactions from direct ones, and therefore slightly promotes its accuracy.
In contrast, the iterative training process in iSIDG creates relaxation of the gradients during training.
As a result, it can distinguish most indirect interactions from the direct ones and reconstruct the adjacency matrix more precisely than the other methods on Linear network dataset.
D.3 Mutual information
We conduct another case study to demonstrate the difference of the mutual information between our iSIDG with and without the iterative process on Linear network dataset with 10 experiments of every model with 1400 training epochs.
We utilize nearest-neighbor MI estimator [36] and the plots are shown in Figure 6.
The plots show the mutual information between input features and embeddings in latent space MI_XZ for every edge, as well as the mutual information between embeddings in latent space and the output features MI_ZY for every edge.
Since the input consists of node features, we concatenate them according to [ sender, receiver ] to represent each edge for the calculation of mutual information, respectively.
As shown in Figure 6, in both methods, the two types of mutual information increase in the first phase, and then after around 800 epochs of training, the mutual information between embeddings in latent space and output features decreases, while the other one keeps growing.
These phenomena verify the two procedures in the training of deep neural networks as mentioned in the IB theory [42, 1], namely label-fitting and representation-compression.
In the phase of representation-compression, when the stop condition is reached, the method with the iterative process stops the current training round, feeds the obtained adjacency matrix from latent space back to the input side of encoder, and starts another round of training with the obtained adjacency matrix.
The phase-transition is demonstrated by the decrease of both mutual information and start with another round of label-fitting and representation-compression.
Therefore, the iterative training process creates several rounds of relaxation during training, and encourages our iSIDG model to learn a more compressed representation in the latent space.
As shown in Figure 5, the compressed representation in the latent space eliminates most of the indirect interactions and promotes the structural inference accuracy of our iSIDG model.
D.4 Ablation studies on different encoders and decoders
We conduct another series of ablation studies to demonstrate the different choices of encoders and decoders in our iSIDG model.
We choose four methods as encoders for our ablation study,
[Table]
[Table]
[Table]
namely: multi-layer-perceptrons (MLP), simple message-passing mechanism (MPM), GCN, GIN, and GAT.
We also take three different decoders into account: MLP, avoiding degenerate decoder (MULTI)
[22][51], and recurrent decoder (RNN) [22].
The results are shown in Table 10, where the rows denote the choice of encoder, the columns for decoder, and the results are the average values and 95% intervals of 15 experiments with every combination of encoder and decoders.
As shown in
Table 10, the methods with GNNs as encoder are greatly more accurate than the ones with simple MLP encoder.
The only exception are the ones with GAT encoder, which are only marginally superior to the ones with MLP encoder.
The reason may be that GAT is prone to over-fitting and the attention weights lack supervision in the cases of structural inference.
When it comes to GIN as encoder, the results are superior to methods with other encoders by at least 3.2 percent of AUROC.
Surprisingly, methods with MLP decoder, which is the simplest decoder among the three, output the most accurate reconstruction results.
We argue that it may be due to the simple task of the iSIDG: only focusing on structural inference instead of both structural inference and dynamic prediction (as that of NRI).
E Do DAG-structure Learning Methods Work in Our Problem Setting?
We observe that there is a series of works, which deal with the directed acyclic graphs (DAG) structure learning [59, 57, 58].
These works formulate the problem as a continuous optimization with a structural constraint that ensures acyclicity [59], with explicit structural constraints [57], or with implicit mechanism to force the acyclicity [58].
Although it is claimed by the authors that these work deal with DAG structure learning, which is different from our problem setting, we are still curious about the performance of these works on the problem of structural inference of dynamical systems.
We choose DAG-GNN [57] and DAG-NoCurl [58], which are the representative in the research field of DAG structure learning.
We follow the official implementations of these models:
• DAG-GNN: https://github.com/fishmoon1234/DAG-GNN ; • DAG-NoCurl: https://github.com/fishmoon1234/DAG-NoCurl ;
and we only change the data loader modules to load physical simulations, synthetic networks and
NetSims datasets, respectively.
We ran the experiments for ten rounds, and summarize the AUROC results in Tables 11 and 12, where we note DAG-GNN* as the DAG-GNN with acyclicity constraint set as zero.
As shown in the tables, DAG-GNN, DAG-GNN* and DAG-NoCurl fail to infer the existence of interactions in dynamical systems.
The reason may be that the existence of cycles in the
[Table]
[Table] [Table] [Table] systems, which is common observed among dynamical systems, violates the acyclicity assumptions of these methods.
In particularly, although the acyclicity constraint in DAG-GNN* is set as zero,
DAG-GNN* is unable to correctly infer the structure of our datasets, and (un)surprisingly performs even worse than DAG-GNN.
F Time and Memory Efficiency
We summarized the training time of iSIDG and baseline methods on all of the datasets mentioned in this work in Tables 13 - 15.
All of the results are the averaged training time of 10 rounds of each method, and is summarized in hours.
Unfortunately yet unsurprisingly, iSIDG performs the worst among all of the methods in almost all of the datasets.
On one hand, the iterative process surely has a negative effect on the time efficiency, on the other hand, iterative process encourages our method to learn the adjacency matrix more accurate than baseline methods on most of the datasets.
It is notable that ACD has comparable training time to iSIDG, and has comparable accuracy with iSIDG on Kuramoto dataset.
We summarized the memory efficiency of iSIDG and baseline models in Table 16.
As shown in the table, the number of total parameters of iSIDG is 0.6 M larger than NRI and fNRI, but is less than half of the number of ACD, which indicates the moderate memory efficiency of iSIDG among baseline methods.
G Broader Impact
Methods such as iSIDG for structural inference of dynamical systems allow for numerous researchers in the field of physics, chemistry and biology to study the interactions inside the systems.
We have shown that iSIDG works well on either one-dimensional node features or multi-dimensional features, where the features are continuous variables, which proves its wide application.
While the emergence of the structural inference technology may be extremely helpful for many, it has the potential for misuse.
Potentially, iSIDG can be extended to infer the online social connections via measuring mutual information, which could erode privacy.