S ECURE N ETWORK R ELEASE WITH L INK P RIVACY
Anonymous authors
Paper under double-blind review
A BSTRACT
Many data mining and analytical tasks rely on the abstraction of networks (graphs) to summarize relational structures among individuals (nodes).
Since relational data are often sensitive, we aim to seek effective approaches to release utilitypreserved yet privacy-protected structured data.
In this paper, we leverage the differential privacy (DP) framework to formulate and enforce rigorous privacy constraints on deep graph generation models, with a focus on edge-DP to guarantee individual link privacy.
In particular, we enforce edge-DP by injecting Gaussian noise to the gradients of a link reconstruction based graph generation model, while ensuring data utility by improving structure learning with structure-oriented graph comparison.
Extensive experiments on two real-world network datasets show that our proposed DPGG AN model is able to generate networks with effectively preserved global structure and rigorously protected individual link privacy.
I NTRODUCTION
Nowadays, open data of networks play a pivotal role in data mining and data analytics (Tang et al., 2008; Sen et al., 2008; Blum et al., 2013; Leskovec & Krevl, 2014).
By releasing and sharing structured relational data with research facilities and enterprise partners, data companies harvest the enormous potential value from their data, which beneﬁts decision-making on various aspects, including social, ﬁnancial, environmental, through collectively improved ads, recommendation, retention, and so on (Yang et al., 2017; 2018; Sigurbjörnsson & Van Zwol, 2008; Kuhn, 2009).
However, network data usually encode sensitive information not only about individuals but also their interactions, which makes direct release and exploitation rather unsafe.
More importantly, even with careful anonymization, individual privacy is still at stake under collective attack models facilitated by the underlying network structure (Zhang et al., 2019;
Cai et al., 2018).
Can we ﬁnd a way to securely release network data without drastic sanitization that essentially renders the released data useless?
In dealing with such tension between the need to release utilizable data and the concern of data owners’ privacy, quite a few models have been proposed recently, focusing on grid-based data like images, texts and gene sequences (Frigerio et al., 2019; Papernot et al., 2018; Triastcyn & Faltings, 2018; Narayanan & Shmatikov, 2008; Xie et al., 2018; Chen et al., 2018; Boob et al., 2018; Dy & Krause, 2018; Lecuyer et al., 2018; Zhang et al., 2018).
However, none of the existing models can be directly applied to the network (graph) setting.
While a secure generative model on grid-based data apparently aims to preserve high-level semantics ( e.g. , class distributions) and protect detailed training data ( e.g. , exact images or sentences), it remains obtuse what to be preserved and what to be protected for network data, due to its modeling of complex interactive objects.
Motivating scenario.
In Figure 1, a bank aims to encourage public studies on its customers’ community structures.
It does so by ﬁrstly anonymizing all customers and then sharing the network ( i.e. , (a) in Figure 1) to the public.
However, an attacker interested in knowing the ﬁnancial interactions ( e.g. , money transfer) between particular customers in the bank may happen to have access to another network of a similar set of customers ( e.g. , a malicious employee of another ﬁnancial company).
The similarity of simple graph properties like node degree distribution and triangle count between the two networks can then be used to identify speciﬁc customers with high accuracy in the released network ( e.g. , customer A as the only node with degree 5 and within 1 triangle, and customer B as the only node with degree 2 and within 1 triangle).
Thus, the attacker conﬁdently knows the A and B’s identities and the fact that they have ﬁnancial interactions in the bank, which seriously harms customers’ privacy and poses potential crises.
[Figure]
As the ﬁrst contribution in this work, we deﬁne and formulate secure network release goals as preserving global network structure while protecting individual link privacy .
Continue with the toy example, the solution we propose is to train a graph neural network model on the original network and release the generated networks ( e.g. , (b) in Figure 1).
Towards the utility of generated networks, we require them to be similar to the original networks from a global perspective, which can be measured by various graph global properties ( e.g. , network (b) has very similar degree distribution and the same triangle count as (a)).
In this way, we expect many downstream data-mining and analytical tasks on them to produce similar results as on the original networks.
As for privacy protection, we require that the information in the generated networks cannot conﬁdently reveal the existence or absence of any individual links in the original networks ( e.g. , the attacker may still identify customers A and B in network (b), but their link structure has changed).
Subsequently, there are two unique challenges in learning such structure-preserved and privacyprotected graph generation models, which have not been explored by existing literature so far.
Challenge 1: Rigorous protection of individual link privacy.
The rich relational structures in graph data often allow attackers to recover private information through various ways of collective inference (Zhang et al., 2014; Narayanan & Shmatikov, 2009;
Backstrom et al., 2007).
Moreover, graph structure can always be converted to numerical features such as spectral embedding, after which most attacks on grid-based data like model inversion (Fredrikson et al., 2015) and membership inference (Shokri et al., 2017) can be directly applied for link identiﬁcation.
How can we design an effective mechanism with rigorous privacy protection on links in networks against various attacks?
Challenge 2: Effective preservation of global network structure.
To capture the global network structure, the model has to constantly compare the structures of the input graphs and currently generated graphs during training.
However, unlike images and other grid-based data, graphs have ﬂexible structures, and thus they lack efﬁcient universal representations (Dong et al., 2019).
How can we allow a network generation model to effectively learn from the structural difference between two graphs, without conducting very time-costly operations like isomorphism tests all the time?
Present work.
In this work, for the ﬁrst time, we draw attention to the secure release of network data with deep generative models.
Technically, towards the aforementioned two challenges, we develop Differentially Private Graph Generative Nets (DPGG AN ), which imposes DP training over a link reconstruction based network generation model for rigorous individual link privacy protection, and further ensures structure-oriented graph comparison for effective global network structure preservation.
In particular, we ﬁrst formulate and enforce edge-DP via Gaussian gradient distortion by injecting designed noise into the sensitive modules during model training.
Then we leverage graph convolutional networks (Kipf & Welling, 2017) through a variational generative adversarial network architecture (Gu et al., 2019; Larsen et al., 2016) to enable structure-oriented network comparison.
To evaluate the effectiveness of DPGG AN , we conduct extensive experiments on two real-world network datasets.
On one hand, we evaluate the utility of generated networks by computing a suite of commonly concerned graph properties to compare the global structure of generated networks with the original ones.
On the other hand, we validate the privacy of individual links by evaluating links predicted from the generated networks on the original networks.
Consistent experimental results show that DPGG AN is able to effectively generate networks that are similar to the original ones regarding global network structure, while at the same time useless towards individual link prediction.
R ELATED W ORK
Differential Privacy (DP).
Differential privacy is a statistical approach in addressing the paradox of learning nothing about an individual while learning useful information about a population (Dwork et al., 2006).
Recent advances in deep learning have led to the rapid development of DP-oriented learning schemes.
Among them, the Gaussian Mechanism (Dwork et al., 2014), deﬁned as follows, provides a neat and compatible framework for DP analysis over machine learning models.
Deﬁnition 1 (Gaussian Mechanism (Dwork et al., 2014)) .
For a deterministic function f with its (cid:96) 2 -norm sensitivity as ∆ 2 f =
[Equation]
we have:
[Equation]
where N (0 , ∆ 2 f 2 σ 2 ) is a random variable obeying the Gaussian distribution with meanand standard deviation ∆ 2 fσ .
The randomized mechanism M f ( G ) is ( ε, δ ) -DP if σ ≥ ∆ 2 f (cid:112) 2 ln(1 .
25 /δ ) /ε
and ε < 1 .
Following this framework, (Abadi et al., 2016) proposes a general training strategy called DPSGD, which looses the condition on the overall privacy loss than that in Deﬁnition 1 by tracking detailed information of the SGD process to achieve an adaptive Gaussian Mechanism.
DP learning has also been widely adapted to generative models (Frigerio et al., 2019; Papernot et al., 2018; Triastcyn & Faltings, 2018; Narayanan & Shmatikov, 2008; Mohammed et al., 2011; Xie et al., 2018; Chen et al., 2018; Boob et al., 2018; Dy & Krause, 2018; Lecuyer et al., 2018; Zhang et al., 2018).
For example, (Frigerio et al., 2019; Chen et al., 2018; Boob et al., 2018; Zhang et al., 2018) share the same spirit by enforcing DP on the discriminators, and thus inductively on the generators, in a generative adversarial network (GAN) scheme.
However, none of them can be directly applied to graph data due to the lack of consideration of structure generation.
For graphs’ structural data, two types of privacy constraints can be applied, i.e. , node-DP (Kasiviswanathan et al., 2013) and edge-DP (Blocki et al., 2012), which deﬁne two neighboring graphs to differ by at most one node or edge.
In this work, we aim at the secure release of network data, and particularly, we focus on edge privacy because it is essential for the protection of object interactions unique for network data compared with other types of data.
Several existing works have studied the protection of edge-DP.
For example, (Sala et al., 2011) generates graphs based on the statistical representations extracted from the original graphs blurred by designed noise, whereas (Wang & Wu, 2013) enforces the parameters of dK-graph models to be private.
However, based on shallow graph generation models, they do not ﬂexibly capture global network structure that can support various unknown downstream analytical tasks (Zhang et al., 2019; Wasserman & Zhou, 2010).
Graph Generation (GGen).
GGen has been studied for decades and is widely used to synthesize network data used to develop various collective analysis and mining models (Evans & Lambiotte, 2009; Hallac et al., 2017).
Earlier works mainly use probabilistic models to generate graphs with certain properties (Erd˝os & Rényi, 1960; Watts & Strogatz, 1998; Barabási & Albert, 1999; Newman, 2001), which are manually designed based on sheer observations and prior assumptions.
Thanks to the surge of deep learning, many advanced GGen models have been developed recently, which leverage different powerful neural networks in a learn-to-generate manner (Kipf & Welling, 2016; Bojchevski et al., 2018;
You et al., 2018b; Simonovsky & Komodakis, 2018; Li et al., 2018;
You et al., 2018a; Jin et al., 2018; Grover et al., 2017; De Cao & Kipf, 2018; Zou & Lerman, 2018;
Ma et al., 2018).
For example, NetGAN (Bojchevski et al., 2018) converts graphs into biased random walks, learns the generation of walks with GAN, and assembles the generated walks into graphs;
GraphRNN (You et al., 2018b) regards the generation of graphs as node-and-edge addition sequences, and models it with a heuristic breadth-ﬁrst-search scheme and hierarchical RNN.
These neural network based models can often generate graphs with much richer properties, and ﬂexible structures learned from real-world graphs.
To the best of our knowledge, no existing work on deep GGen has looked into the potential privacy threats laid during the learning and releasing of the powerful models.
Such concerns are rather urgent in the network setting, where sensitive information can often be more easily compromised in a collective manner (Dai et al., 2018; Backstrom et al., 2007; Zhang et al., 2014) and privacy leakage can easily further propagate (Narayanan & Shmatikov, 2009; Zügner et al., 2018).
DPGG AN
In this work, we propose DPGG AN for the secure release of generated networks, whose global graph structures are similar to the original sensitive networks, but the individual links (edges) between objects (nodes) are safely protected.
To provide robust privacy guarantees towards various graph attacks, we propose to leverage the well-studied technique of differential privacy (DP) (Dwork et al., 2014) by enforcing the edge-DP deﬁned as follows.
Deﬁnition 2 (Edge Differential Privacy (Blocki et al., 2012)) .
A randomized mechanism M satisﬁes ( ε, δ ) -edge-DP if for any two neighboring graphs G 1 , G 2 ∈ G , which differ by at most one edge,
Pr[ M ( G 1 ) ∈ S ] ≤ exp ( ε ) × Pr[ M ( G 2 ) ∈ S ] + δ , where S ⊂ range ( M ) .
Our key insight is, a graph generation model M satisfying the above edge-DP should learn to generate similar graphs given the input of two neighboring graphs that differ by at most one edge; as a consequence, the information in the generated graph does not conﬁdently reveal the existence or absence of any one particular edge in the original graph, thus protecting individual link privacy.
To ensure DP on individual links, we exploit the existing link reconstruction based graph generation model GVAE (Kipf & Welling, 2016), and design a training algorithm to dynamically distort the gradients of its sensitive model parameters by injecting proper amounts of Gaussian noise based on the framework of DPSGD (Abadi et al., 2016).
We provide theoretical analysis on applying
DPSGD to achieve edge-DP with GVAE based on the nature of graph data and the link reconstruction loss.
Moreover, to improve the capturing of global graph structures, we replace the direct binary cross-entropy (BCE) loss on graph adjacency matrices in GVAE with a structure-oriented graph discriminator based on GCN (Kipf & Welling, 2017) and the framework of VAEGAN (Gu et al., 2019; Larsen et al., 2016).
We further prove the improved model to maintain the same edge-DP.
Backbone GVAE.
Recent research on graph models has been primarily focused around GCN (Kipf &
Welling, 2017), which is shown to be promising in calculating universal graph representations (Maron et al., 2019; Xu et al., 2019; Chen et al., 2019; Keriven & Peyré, 2019).
In this work, we harness the power of GCN under the consideration of edge-DP by adapting the link reconstruction based graph variational autoencoder (GVAE) (Kipf & Welling, 2016) as our backbone graph generation model.
Notably, we are given a graph G = { V , E } , where V is the set of N nodes (vertices), and E is the set of M links (edges), which can be further modeled by a binary adjacency matrix A .
As a common practice (Hamilton et al., 2017), we set the node features X simply as the one-hot node identity matrix.
The autoencoder architecture of GVAE consists of a GCN-based graph encoder to guide the learning of a feedforward neural network (FNN) based adjacency matrix decoder, which can be trained to directly reconstruct graphs with similar links as in the input graphs.
A stochastic latent variable Z is further introduced as the latent representation of A as
[Equation]
where µ = g µ ( X , A ) is the matrix of mean vectors µ i , and σ =
g σ ( X , A ) is the matrix of standard deviation vectors σ i .
g • ( X , A ) =
˜
A ReLU ( ˜ AXW 0 )
W 1 is a two-layer GCN model.
g µ and g σ share the ﬁrst-layer parameters W 0 .
˜
A
= D − 12 AD − 12 is the symmetrically normalized adjacency matrix of G , with degree matrix D ii =
(cid:80) Nj =
1 A ij .
g µ and g σ form the encoder network.
To generate a graph G (cid:48) , a reconstructed adjacency matrix A (cid:48) is computed from Z by an FNN decoder
[Equation]
where σ ( z ) = 1 / (1 + e − z ) , f is a two-layer FNN appended to Z before the logistic sigmoid function.
The whole model is trained through optimizing the following variational lower bound
[Equation]
where L rec is implemented as the sum of an element-wise binary cross entropy (BCE) loss between the adjacency matrices of the input and generated graphs, and L prior is a prior loss based on the
Kullback-Leibler divergence towards the Gaussian prior p ( Z ) =
(cid:81) N i =1 p ( z i ) = (cid:81) N i N ( z i | 0 , I ) .
Enforcing DP.
The probabilistic nature of Z allows the model to be generative, meaning that after training the model with an input graph G , we can detach and disregard the encoder, and then freely generate an unlimited amount of graphs G (cid:48) with similar links to G , by solely drawing random samples of Z from the prior distribution N ( 0 , I ) and computing A (cid:48) with the learned decoder network w.r.t.
Eq.
However, as shown in (Kurakin et al., 2017; Gondim-Ribeiro et al., 2018), powerful neural network models like VAE can easily overﬁt training data, so directly releasing a trained GVAE model poses potential privacy threats, as links in its generated graphs may be highly indicative towards links in the training graphs.
In this work, we care about the generation model’s rigorously protecting the privacy of individual links in the training data, i.e. , ensuring edge-DP.
Particularly, in Deﬁnition 2, the inequality guarantees that the distinguishability of any one edge in the graph will be restricted to the privacy leak level proportional to ε , quantifying the absolute value of privacy information possibly to be leaked by a graph generation model.
According to Eq.
(3), GVAE essentially takes a graph G , in particular, the links E among the nodes
V in G , as input and generates a new graph G (cid:48) by reconstructing the links E (cid:48) among the same set of nodes V .
Therefore, if we regard GVAE as the mechanism M , as long as its model parameters are properly randomized, the framework satisﬁes edge-DP.
To be speciﬁc, any two input graphs G 1 and
G 2 differing by at most one link in principle lead to similar generated graphs G (cid:48) , so information in
G (cid:48) does not conﬁdently reveal the existence or absence of any particular link in G 1 or G 2 .
To exploit the well-structured graph generation framework of GVAE, we leverage the Gaussian mechanism (Deﬁnition 1)
(Dwork et al., 2014) and DPSGD (Abadi et al., 2016) to enforce edge-DP on it.
In our setting, G is the original training graph.
Then Eq.
(1) tells us that a link reconstruction based graph generation model M can be randomized to ensure ( ε, δ ) -edge-DP with properly parameterized
Gaussian noise.
Prominently, we follow DPSGD (Abadi et al., 2016) to inject a designed Gaussian noise to the gradients of our decoder network clipped by a hyper-parameter C as follows.
[Equation]
where L is the loss function of a link reconstruction based graph generation model, C is the clipping hyper-parameter for the model’s original gradient to bound the inﬂuence of each link, and σ is the noise scale hyper-parameter.
In the following theorem, we analyze and prove that the noised clipped gradient ˜ g θ, L applied as above guarantees the learned graph generation model to be edge-DP, with a different condition from that in Deﬁnition 1 due to the learning process of link based graph generation model.
Theorem 1.
In training a link reconstruction based graph generation model on a graph with N nodes with batch size B , given the sampling probability q = B/N , and the number of steps T , there exist explicit constants c 1 and c 2 that for any ε < c 1 q 2 T , iteratively updating the model T times with ˜
g θ, L attains it with ( ε, δ ) -edge-DP for any δ > 0 if we choose
[Equation]
where c 1 ≥ 1 c 0 log 1 qσ , c 2 ≤ 1 / (cid:112) c 0 (1 − c 0 ) for any c 0 ∈ (0 , 1) . The proofs of Theorem 1 are detailed in Appendix A. For the training of the DPGV AE decoder, L in Eq.
(5) is speciﬁed as L rec in Eq. Due to the link reconstruction nature of DPGV AE , we derive Corollary 1.1 from Theorem 1 as follows. Corollary 1.1 (DPGV AE edge-DP) . Under the same conditions in Theorem 1, iteratively updating the decoder in DPGV AE for T times with ˜ g θ, L rec attains it with ( ε, δ ) -edge-DP.
In the generation stage, we can disregard the encoder and only use the decoder to generate an unlimited amount of graphs from randomly sampled vectors from the prior distribution N ( 0 , I ) .
Due to the randomness of the normal Gaussian distribution, the sampling process can be regarded as (0 , 0) -DP.
By the composability property of DP (Dwork et al., 2014), generating graphs from random noises with the DPGV AE decoder satisﬁes ( ε, δ ) -edge-DP, whose release in principle does not disclose sensitive information regarding individual links in the original sensitive networks.
Since we do not release the encoder network, we do not need to clip and perturb its gradients during training to induce minimum interruptions.
Improving structure learning.
Besides individual link privacy, we also aim to preserve the global network structure to ensure the utility of released data.
As we discuss before, original GVAE computes the reconstruction loss between input and generated graphs based on the element-wise BCE between their adjacency matrices.
Such a computation is speciﬁed on each link, rather than the graph structure as a whole.
To improve the global graph structure learning, we leverage GCN again, which has been shown universally powerful in capturing graph-level structures (Maron et al., 2019; Xu et al., 2019; Chen et al., 2019; Keriven & Peyré, 2019).
In particular, we borrow the framework of VAEGAN from recent research (Gu et al., 2019; Larsen et al., 2016; Yang et al., 2019), and compute a structure-oriented generative adversarial network (GAN) loss as
[Equation]
where g (cid:48) and f (cid:48) are GCN and FNN networks similar as deﬁned before, besides at the end of g (cid:48)
the node-level representations are summed up as the graph-level representation, which resembles the recently proposed GIN model for graph-level representation learning (Xu et al., 2019).
In this
DPGG AN framework, the decoder also serves as the generator, while D = f (cid:48) · g (cid:48) is the discriminator.
Following (Gu et al., 2019), the encoder is trained w.r.t.
L rec + λ 1 L prior , the generator w.r.t.
L rec − λ 2 L gan , and the discriminator w.r.t.
λ 2 L gan , where λ 1 and λ 2 are hyper-parameters.
To enforce DP constraints and complete our proposed DPGG AN framework, Eq.
(5) with L rec substituted by L rec − λ 2 L gan is applied to distort the gradients of the generator and guarantee edge-DP, which can be used to securely generate networks with the other parts disregarded after training.
The overall framework of DPGG AN is shown in Figure 2, and the training process is detailed in Appendix B.
The intuition behind the novel design of DPGG AN is, the GCN encodings g (cid:48)
( A ) and g (cid:48)
( A (cid:48) ) capture the graph structures of G and G (cid:48) , so a reconstruction loss L rec =
(cid:107) g (cid:48)
( A ) − g (cid:48)
( A (cid:48) )
(cid:107) 22 captures the intrinsic structural difference between G and G (cid:48) instead of the simple sum of the differences over their individual links.
Note that the effectiveness of our structure-oriented discriminator is critical not only because it can directly enforce effective training of the graph generator through the minimax game in Eq.
(6), but also because it can learn to relax the penalty on certain individual links through ﬂexible and diverse conﬁgurations of the whole graph as long as the global structures remain similar, which exactly fulﬁlls our goals of secure network release.
The beneﬁts of such diversity enabled by the VAEGAN have also been discussed in image generation (Gu et al., 2019; Larsen et al., 2016).
Compared with DPGV AE , DPGG AN does not directly compute the link reconstruction loss based on BCE in Eq.
(4), but rather computes it based on the graph discriminator D . However, the link reconstruction based graph generator of DPGG AN is exactly the same as DPGV AE . Since we also do not release D after training, we can simply retrieve Corollary 1.2 from Theorem 1 as follows. Corollary 1.2 (DPGG AN edge-DP) .
Under the same conditions in Theorem 1, iteratively updating the generator in DPGG AN for T times with ˜ g θ, ( L rec − λ 2 L gan ) attains it with ( ε, δ ) -edge-DP.
With Corollary 1.2, we attain DPGG AN with the same ( ε, δ ) -edge-DP protection of DPGV AE . For both DPGV AE and DPGG AN , the decoder/generator networks only get exposed to the noised and clipped gradients, representing the partial sensitive information within the training graphs. Hence, it prevents the inference of training graphs from both learned model parameters and generated graphs.
E XPERIMENTAL E VALUATIONS
We conduct two sets of experiments to evaluate the effectiveness of DPGG AN in preserving global network structure and protecting individual link privacy .
All code and data are also in the submission.
Experimental settings.
To provide a side-to-side comparison between the original networks and generated networks, we use two standard datasets of real-world networks, i.e. , DBLP, and IMDB.
DBLP includes 72 networks of author nodes and co-author links, where the average numbers of nodes and links are 177.2 and 258; IMDB includes 1500 networks of actor/actress nodes and co-star links, with average node and link numbers 13 and 65.9.
To show that DPGG AN effectively captures global network structures, we compare it and DPGV AE under different privacy budgets (controlled by ε in Eq.
(32)), regarding a suite of graph statistics commonly used to evaluate the performance of graph generation models, especially from a global perspective (Bojchevski et al., 2018;
You et al., 2018b; Yang et al., 2019).
1 In particular, we train all models from scratch to convergence for K times, where K is the number of networks in the datasets.
Each time, the trained model is used to generate one network, which is compared with the original network regarding the suite of graph statistics.
Then we average the absolute differences between the generated networks and the original networks, ensuring that the positive and negative differences do not cancel out.
The results are summarized in Table 1.
Beyond the single value statistics, we also compare the generated graph regarding degree distribution and motif counts.
For degree distribution, we convert each graph into a 50-dim vector (all nodes with degree larger than 50 are binned together); for motif counts, we enumerate all 29 undirected motifs with 3-5 nodes and convert each graph into a 29-dim vector by motif matching.
We compute the average cosine similarity between pairs of original graphs and generated graphs.
Furthermore, we use the most widely studied graph-level downstream task, i.e. , graph classiﬁcation, to evaluate the global utilities of generated graphs. In particular, we evaluate the accuracy of the state-of-the-art graph classiﬁcation model, i.e. , GIN (Xu et al., 2019), with the default parameter setting and 4: training-testing ratio. The results are summarized in Table 2. To facilitate a better understanding towards how the graph statistics reﬂect the global network structure captured by the models, we also provide results of two recent deep network generation methods, i.e. , NetGAN (Bojchevski et al., 2018) and GraphRNN (You et al., 2018b), with default parameter settings and no DP constraints at all.
In this experiment, we expect to see the more effective structure-preserving models generate networks that are more similar to the original ones regarding various global and distributional graph properties and achieve high graph classiﬁcation accuracy , thus maintaining high network data utility.
To show that DPGG AN effectively guarantees individual link privacy, we train all models for another
K times on each dataset.
Instead of complete networks, we randomly sample 80% of the original networks’ links to train the models.
After training and generation, we use degree distribution to align the nodes in the generated networks with those in the original networks.
Then we evaluate the standard AUC metric on the task of individual link prediction by comparing links predicted in the generated networks and links hidden during training in the original networks.
In this experiment, we expect to see the more effective privacy-protecting models generate networks that are less useful for predicting individual links in the original networks, thus guaranteeing network data privacy.
[-99 , learning rate η to 10 − 3 , and the loss weighing parameters λ 
-]
All experiments are done with four GeForce GTX 1080 GPUs and a 12-core 2.2GHz CPU.
The training time of DP-enforced models is often slightly shorter due to early stops when the privacy budget runs out, ( e.g. , a typical train of GVAE, DPGV AE , and DPGG AN takes 60, 42 and 53 seconds on average on DBLP).
The generation times of the three models are roughly the same ( e.g. , 0.
second on average on DBLP).
As a direct comparison, NetGAN and GraphRNN take longer times under the same settings, especially for the generation ( e.g. , 89, and 4.5 seconds for NetGAN to train and generate, and 75 and 2.4 seconds for GraphRNN, on DBLP).
Although efﬁciency is not our primary concern, short runtimes (especially for generation) are favorable for efﬁcient data share.
Due to space limitation, detailed settings of the neural architectures and hyper-parameters of our models are put into Appendix C.
[Table]
[Table]
Preserving global structures.
In Table 1, our strictly DP-constrained models constantly yield highly competitive and even better results compared with the strongest DP-free baselines regarding global network structural similarity between generated and original networks on both datasets, clearly showing the effectiveness of our models on global network structure preservation.
As we gradually increase the privacy budget ε , our two models (especially DPGG AN ) apparently perform better, showing the effectiveness of our privacy constraints and a clear trade-off between privacy and utility.
Furthermore, as in Table 2, the graphs generated by DPGG AN are competitively similar to the original graphs regarding both degree distributions and motif counts, while achieving satisfactory graph classiﬁcation accuracy.
The improvements of DPGG AN all passed t-tests with p-value 0.01, which corroborates our novel design of the structure-oriented graph generation framework.
Protecting individual links.
For both datasets, links predicted on the networks generated by DPGG AN are much less accurate than those predicted on the original networks (26%-35% and 15%-20%
AUC drops on DBLP and IMDB, respectively) as well as the networks generated by all baselines.
This means even if the attackers identify nodes in the generated (released) networks of DPGG AN , they cannot leverage the information there to accurately infer the existence or absence of links between particular pairs of nodes on the original networks.
This directly corroborates our claim that DPGG AN is effective in protecting individual link privacy.
Due to space limit, more details and discussions regarding the experimental results are put into Appendix D. In Appendix E, we also provide graph visualizations for qualitative visual inspections.
C ONCLUSION
Due to the recent development of deep graph generation models, synthetic networks are generated and released for granted, without the concern about possible privacy leakage over the original networks used for model training.
In this work, for the ﬁrst time, we pay attention to the task of secure network release and formulate its goals as preserving global network structure while protecting individual link privacy .
Subsequently, we adopt the well-studied DP framework and develop DPGG AN , which protects individual link privacy by enforcing edge-DP on the graph generation model while preserving global network structure with a structure-oriented graph discriminator.
Comprehensive experiments show that DPGG AN is advantageous in generating networks that are globally similar to the original ones (thus effectively maintaining network data utility), and at the same time, useless for predicting individual links in the original network (thus rigorously protecting network data privacy).
A APPENDIX: P ROOFS FOR T HEOREM
In this appendix, we provide proofs for Theorem 1, and derive Corollary 1.1 and Corollary 1.2.
Theorem 1 indicates the link privacy protection achieved through updating model’s parameters with clipped and noised gradient (latter referred to as DP learning) for link reconstruction based graph generation models. Corollary 1.1 and Corollary 1.2 derived from Theorem 1 support us to guarantee ( (cid:15), δ ) -edge-DP for DPGV AE and DPGG AN with DP learning in Theorem 1. The proof for Theorem 1 is divided into three steps.
We ﬁrst brieﬂy introduce the deﬁnition of the moment accountant privacy analysis and respective properties in Section A.1, for it being the fundamentals of our proof.
Note that in (Abadi et al., 2016), DPSGD is originally designed for classical machine learning tasks, such as image classiﬁcation.
Therefore, in Section A.2, we leverage moment accountant to conduct the extended privacy analysis of DPSGD for general types of data and loss functions.
Then in Section A.3, we apply the conclusion from Section A.2 on graph data and the link reconstruction loss function to derive the theoretical analysis over edge-DP achieved by link reconstruction based graph generation models and ﬁnish our proof for Theorem 1.
Following the conclusion in Theorem 1, we tune gradient representations to certain gradient functions leveraged in training DPGV AE decoder and DPGG AN generator to derive Corollary 1.1 and Corollary 1.2, as the theoretical support for the ( ε, δ ) -edge-DP held by respective models.
A. M OMENT A CCOUNTANT
Our proof for Theorem 1 is mainly based on moment accountant (Abadi et al., 2016).
The deﬁnition of moment accountant and the properties leveraged in our proof are listed below.
Deﬁnition 3.
Let M : D → R be a randomized mechanism and d, d (cid:48) a pair of adjacent databases.
Let aux denote an auxiliary input.
For an outcome o ∈ R , the privacy loss at o is deﬁned as:
[Equation]
The privacy loss random variable C ( M , aux, d, d (cid:48) ) is deﬁned as c ( M ( d ); M , aux, d, d (cid:48) ) , i.e.the random variable deﬁned by evaluating the privacy loss at an outcome sampled from M ( d ) .
Deﬁnition 4.
Let M : D → R be a randomized mechanism and d, d (cid:48) a pair of adjacent databases.
Let aux denote an auxiliary input.
The moments accountant is deﬁned as:
[Equation]
where α M ( λ ; aux, d, d (cid:48) )
(cid:44) log E [exp ( λC ( M , aux, d, d (cid:48) ))] is the moment generating function of the privacy loss random variable.
The following properties of the moments accountant are proved in (Abadi et al., 2016).
Property 4.1.
[Composability] Suppose that a mechanism M consists of a sequence of adaptive mechanisms
M 1 , . .
, M k where M i : (cid:81) i − 1 j =1 R j × D
→ R i .
Then, for any output sequence o 1 , . . .
, o k − 1 and any λ , we have
[Equation]
[Equation]
where α M is conditioned on M i ’s output being o i for i < k .
Property 4.2.
[Tail bound]
For any ε > 0 , the mechanism M is ( ε, δ ) -DP for
[Equation]
A. T HE G ENERALIZED P RIVACY A NALYSIS OF DPSGD
To achieve ( (cid:15), δ ) -edge-DP for graph data, we exploit DPSGD (Abadi et al., 2016) with necessary adaptions according to the special nature of graph data compared to other types of data ( e.g. , images), for which DPSGD was originally designed.
The original DPSGD only provides DP proof for gradient function f clipped by C with its (cid:96) 2 -norm sensitivity as ∆ 2 f = 1 · C = C .
For classical tasks of machine learning like image classiﬁcation, ∆ 2 f = C is obvious.
However, in a more complex task like graph learning, a minor change in the training dataset can probably induce a different gap according to the chosen measurement.
To explore the potential of DPSGD with customized machine learning tasks, we further prove the privacy performance of DPSGD with a gradient function f with (cid:96) 2 -norm sensitivity ∆ 2 f = s .
Therefore, to prepare for the proof for Theorem 1, we ﬁrst leverage moments accountant to derive the upper bound of privacy loss for a Gaussian Mechanism as below.
Lemma 1.
Suppose that f : D → R p with (cid:107) f ( · ) (cid:107) 2 ≤ s .
Let σ ≥ s and let J be a sample from [ n ] where each i ∈ [ n ] is chosen independently with probability q < s 16 σ .
Then for any positive integer λ ≤ σ 2 s 2 ln sqσ , the
Gaussian Mechanism M ( d ) =
(cid:80) i ∈
J f ( d i
[Equation]
+ N (cid:0)
[Equation]
[Equation]
satisﬁes
[Equation]
Proof.
Fix d (cid:48) and let d
= d (cid:48) ∪ { d n } .
Without loss of generality, we assume f ( d n ) = s · e 1 and (cid:80) i ∈ J \ [ n ] f ( d i )
= 0 .
Thus M ( d ) and M ( d (cid:48) ) are distributed identically except for the ﬁrst coordinate and hence we have a one-dimensional problem.
Let µ 0 denote the pdf of N (cid:0) 0 , σ 2 (cid:1) and let µ s denote the pdf of
N (cid:0) s, σ 2 (cid:1) .
We have
[Equation]
We want to show that
[Equation]
and
[Equation]
where α is a value to be determined.
We will use the same method as in (Abadi et al., 2016) to prove both bounds.
Assume we have two distributions ν 0 and ν s , and we wish to bound
[Equation]
Leveraging binomial expansion, we obtain
[Equation]
The ﬁrst term in Eq.
(15) is 1 , and the second term is
[Equation]
[Equation]
Regarding conditions stated in the lemma, for both cases, where ν 0 = µ, ν 1 = µ 0 and ν
= µ 0 , ν 1 = µ , the third term is bounded by q 2 λ ( λ + 1) / (1 − q ) σ 2 and this bound dominates the sum of the remaining terms.
We provide the proof for the case of ( ν 0 = µ 0 , ν
To upper bound the third term in 15, we note that µ ( z ) ≥ (1 − q ) µ
( z ) , and write
[Equation]
Recalling the deﬁnition of µ 0 and the normal distribution [Equation] we have E z
[Equation]
For the second term in Eq.
[Equation] [Equation] E
[Equation]
σ 2 (cid:17)(cid:105) , we have
[Equation]
For the third term in Eq.
(18), we have
[Equation]
Thus, for Eq.
(18), we have
[Equation]
Hence, the third term in the binomial expansion of Eq.
(15) is
[Equation]
[Equation]
For σ ≥ s , it is easy to get exp( s 2 σ 2 ) − 1 ≤ 2 s 2 σ 2 .
Therefore, we retrieve that
[Equation]
By standard calculus, we get | µ 0 ( z ) − µ s ( z ) | = (cid:12)(cid:12)(cid:12)(cid:82)
z z − s µ {+(cid:48) 
+}
( z )d z (cid:12)(cid:12)(cid:12) .
Note that µ (cid:48)
( z ) is monotonically decreasing in ( −∞ , + ∞ ) .
Thus, to bound the remaining terms, we derive
[Equation]
We can then write
[Equation]
We consider these terms individually.
We repeatedly make use of three observations: (1) µ 0 − µ = q ( µ 0 − µ s ) ,(2) µ ≥ (1 − q ) µ 0 , (3) µ ≥ qµ s , and (4) E µ 0 (cid:2) | z | t (cid:3) ≤ σ t ( t − 1)!! .
The ﬁrst term can then be bounded by
[Equation]
Then the second term is at most
[Equation] [Equation]
Similarly, the third term is at most
[Equation]
[Equation]
Under the assumptions on q, σ, and λ, it is easy to check that the three terms, and their sum, drop off geometrically fast in t for t > 3 .
Hence the binomial expansion (5) is dominated by the t = 3 term, which is O (cid:0) s 3 q 3 λ 3 /σ 3 (cid:1) .
Therefore, the lemma is proved.
With Lemma 1, we retrieve the upper bound of privacy loss of the Gaussian Mechanism.
Hence, based on Lemma 1 and Property 4.1, we provide the generalized privacy analysis of DPSGD with different learning tasks, which iteratively performs multiple times of the Gaussian Mechanism.
Lemma 2.
Suppose that f : D → R p with (cid:107) f ( · ) (cid:107) 2 ≤ s . Let J be a sample from [ N ] that each i ∈ [ N ] is chosen independently in probability q = | J | /N , given the number of steps T , for any c 0 ∈ (0 , 1) , there exist explicit constants c 1 and c 2 that with any ε < c 1 q 2 T , iteratively computing T times of M ( d ) in Lemma 1 attains it with ( ε, δ ) -DP for any δ > 0 if we choose
[Equation]
where c 1 ≥ 1 c 0 log sqσ and c 2 ≤ 1 √ c 0 (1 − c 0 ) for any c 0 ∈ (0 , 1) .
Proof.
Assume for now that σ, λ satisfy the conditions in Lemma 1.
After T times of iteration, with Property 4.1 we derive that α ( λ ) ≤
Tq 2 s 2 λ 2 /σ 2 .
In order to to guarantee the whole training process to be ( ε, δ ) -DP, combining α ( λ ) with Property 4.2, for any c 0 ∈ (0 , 1) , we choose
[Equation]
Plugging the condition λ ≤ σ 2 s 2 log sqσ into Eq.
(30), we derive the bound for ε as ε < 1 c 0 log sqσ q 2 T to accomplish ( ε, δ ) -DP by setting
[Equation]
where c 0 ∈ (0 , 1) .
A. P RIVACY A NALYSIS FOR THE L INK R ECONSTRUCTION B ASED G RAPH G ENERATION M ODELS WITH DPSGD
In this section, we conduct the theoretical privacy analysis for link reconstruction based graph generation model based on Lemma 2 and obtain the conclusion of Theorem 1.
Theorem 1.
In training a link reconstruction based graph generation model on a graph with N nodes with batch size as B , given the sampling probability q = B/N , and the number of steps T , there exist explicit constants c 1 and c 2 that for any ε < c 1 q 2 T , iteratively updating the model T times with ˜
g θ, L attains it with ( ε, δ ) -edge-DP for any δ > 0 if we choose
[Equation]
where c 1 ≥ 1 c 0 log 1 qσ , c 2 ≤ 1 / (cid:112) c 0 (1 − c 0 ) for any c 0 ∈ (0 , 1) .
Proof.
Recall the expression of ˜ g
θ, L as in Eq. [Equation]
where L is the loss function for a link reconstruction based graph generation model, C is the clipping hyper-
parameter for the model’s original gradient to bound the inﬂuence of each node, and σ is the noise scale hyper-parameter.
According to Deﬁnition 1, ˜ g θ, L is a Gaussian mechanism.
Therefore, we ﬁrst analyze the (cid:96) 2 -norm sensitivity of the clipped gradient function ˜
g θ, L , and then plug the sensitivity value to Lemma 2 and conclude the privacy cost of training DPGV AE , thus ﬁnishing the proof for Thereom 1.
Following the graph reconstruction procedure in (Simonovsky & Komodakis, 2018), a single value in the adjacency matrix is sufﬁcient to represent one edge in the respective graph for both directed and undirected graphs.
Therefore, referring to Deﬁnition 2, though changing an edge in the graph affects 2 nodes for node classiﬁcation tasks, for a structural inference task, i.e. , graph reconstruction, as our work targeting at, adding or removing an edge only results to at most 1 record difference.
Together with ∇ v i , f L being clipped as its (cid:96) 2 -norm no more than C , we obtain the sensitivity of (cid:80) Ni =1 ∇ v i , f L / max(1 , (cid:107)∇ vi, f L(cid:107) 2 C ) as C .
With plugging in the clipped ∇ v i , f L ’s sensitivity ( s = C ) into Lemma 2, we derive Theorem 1.
We prove that, given the sampling probability q = B/N and the number of steps T , with explicit constants c 1 ≥ 1 c 0 log 1 qσ and c 2 ≤ 1 √ c 0 (1 − c 0 ) , where c 0 ∈ (0 , 1) , through iteratively updating model T times with Eq. (5), the outcome generation model achieves ( ε, δ ) -edge-DP for any ε < c 1 q 2 T , and δ > 0 when we choose
[Equation]
Recall the training process for the decoder in DPGV AE and the generator in DPGG AN in Section 3.
L in ˜
g θ, L is substituted with L rec and L rec + λ 2 L gan , respectively. For both L rec and L rec + λ 2 L gan , their gradients are clipped with C and adding Gaussian noises during the training process.
Based on Theorem 1, we derive
Corollary 1.1 and 1.2 for the decoder in DPGV AE and the generator in DPGG AN respectively as below. Corollary 1.1 (DPGV AE edge-DP) . Under the same conditions in Theorem 1,iteratively updating the decoder in DPGV AE T times with ˜ g θ, L rec attains it with ( ε, δ ) -edge-DP.
Corollary 1.2 (DPGG AN edge-DP) . Under the same conditions in Theorem 1, iteratively updating the generator in DPGG AN T times with ˜ g θ, ( L rec − λ 2 L gan ) attains it with ( ε, δ ) -edge-DP.
With Corollary 1.1 and 1.2, under speciﬁed conditions, the public model (either the decoder in DPGV AE or the generator in DPGG AN ) is guaranteed with ( ε, δ ) -edge-DP by the DP training process.
For both DPGV AE decoder and DPGG AN generator updated with noised and clipped representations of the sensitive training graph, they only record noised and partial sensitive information. DPGV AE decoder and DPGG AN generator’s link reconstruction procedures, reﬂecting its training information, only allude to the desensitize information rather than the true sensitive training information.
Thus, DPGV AE decoder and DPGG AN generator not only prevent privacy leakage from their inner parameters with DP learning but also preserve the raw private training graphs from being accurately inferred through the respective outputs.
B APPENDIX: D ETAILED T RAINING A LGORITHM
The overall architecture of DPGG AN is shown in Figure 2 in the main content.
In Figure 2, the original graph is fed into the GCN-based encoder network g to compute node embeddings, which is then compared with the prior distribution N ( 0 , I ) and fed into the FNN-based decoder/generator network f to produce the reconstructed graph and generated graph.
After going through the GCN-based discriminator network part 1 ( g (cid:48) ), a reconstruction loss is computed between the reconstructed graph and the original graph, and a discrimination loss is computed for the generated graph and original graph after the FNN-based discriminator network part 2 ( f (cid:48) ).
Sample z i
A )
Reconstruct adjacent matrix A
← f
T f
D KL
1 − D node i do
end
Clip the generator
22 end
Here, we give more details towards its training procedures in Algorithm 1.
For a better description, we shorten g θ, ( L rec − λ 2 L gan ) ( ˜ g θ, ( L rec − λ 2 L gan ) ) as g θ ( ˜ g θ ).
In the algorithm, for proper gradient distortion, we devise gradient clipping in Line 15 and noise injection in Line 16, which is only applied to the generator network f in
Line 18.
In Line 21, we gradually reduce the clipping hyper-parameter C as the volume of gradients decreases along training.
In the experimental analysis, existing works often ﬁx δ as a dataset-speciﬁc value like 10 − 5 , and then analyze the [Figure] C APPENDIX: M ORE DETAILS OF EXPERIMENTAL SETTINGS
For GVAE and our models, we use two-layer GCNs with sizes 32 → 16 for both g µ and g σ of the encoder network, where the ﬁrst layer is shared, and we use two-layer FNNs with sizes 16 → 32 for f of the decoder (generator) network.
For DPGG AN , we use another two-layer GCN with the same sizes for g (cid:48) and a three-layer FNN with sizes 16 → 32 → 1 for f (cid:48) . For DP-related hyper-parameters, we follow existing works (Abadi et al., 2016; Shokri & Shmatikov, 2015) to ﬁx δ to 10 − 5 , σ to 5, and q to 0.01 (which determines the batch size B as B = qN with N as the graph size). Then we vary ε from 0.1 to 10 to see how much graph-level utilities are preserved under different privacy budgets.
According to Eq. (32), we terminate the training of DPGG AN at T iterations when ε is depleted. Other than the essential parameters in Eq.
(32), we empirically set the clipping parameter C to 5 , decay ratio γ to 0 .
99 , learning rate η to 10 − 3 , and the loss weighing parameters λ 1 and λboth to 0.1.
We do not observe the model to be very sensitive to the setting of these non-essential parameters.
D APPENDIX: M ORE DETAILS OF EXPERIMENTAL RESULTS
In this work, we deﬁne the goals of secure network release as preserving global network structure while protecting individual link privacy .
In the main content, we have presented experimental results to support the effectiveness of DPGG AN in both perspectives.
That is, for global network structure preservation, we show that the generated graphs of DPGG AN are competitively similar to the original graphs in comparison with the DP-free state-of-the-art graph generative models regarding a suite of commonly concerned global graph statistics, and for individual link privacy protection, we show that the links predicted in the generated graphs of
DPGG AN are useless (with low accuracy) when evaluated in the original graphs.
The suite of statistics measures the global network structure from different perspectives. As can be inferred from TC, CPL, and GINI, the IMDB networks are in general smaller, tighter, and likely more structurally complex than the DBLP networks, which favors link generation models ( e.g. , GVAE) over sequence generation models ( e.g. , NetGAN, and GraphRNN). Consequently, DPGG AN also performs better on the IMDB networks, indicating its advantages in modeling complex link structures as a whole.
In addition to the graph statistics, we further demonstrate the data utility of networks generated by DPGG AN with graph classiﬁcation, which is the most widely studied graph-level downstream task.
We deem this task important towards evaluating network data utility, especially under our consideration of global network structure preservation, because correct graph classiﬁcation requires the generated graphs to share essential structural properties with the original graphs.
As we can see from Table 2 in the main paper, the data utilities evaluated with graph classiﬁcation are consistent with those evaluated with global graph statistics, as shown in Tablein the main paper.
Our two DP-constrained models yield highly competitive performance compared with the DP-free state-of-the-art graph generative models.
As for privacy protection, we conduct more detailed inspections of the performance of individual link prediction. In particular, we vary two of the major hyper-parameters, i.e. , the privacy budget ε , and sampling ratio q .
Consistently with the results in Table 1, larger privacy budgets lead to more privacy leakage, which allows attackers to infer individual links in the original networks with higher accuracy.
While some DP-constrained deep learning models are observed to be sensitive to the sampling ratio during training (Abadi et al., 2016; Shokri & Shmatikov, 2015), the privacy protection of DPGG AN is robust when q is changed in large ranges in practice. E APPENDIX: Q UALITATIVE G RAPH V ISUALIZATIONS
To understand the behaviors and performances of compared algorithms, we conduct visualizations between the original graphs and graphs generated graphs by different algorithms. We mainly focus on the visualization of DBLP author networks, since they are in general smaller, sparser and thus easier to visually inspect. We draw the graphs with NetworkX 2 .
In general, as we can observe in Figure 4-10:
The graphs generated by algorithms without DP constraints like NetGAN and GraphRNN are more similar to the original graphs, which is consistent with our results in Table 1 and 2 in the main paper.
After enforcing the DP constraints, the graphs generated by DPGV AE are signiﬁcantly different from the graphs generated by GVAE, especially regarding local structures around individual nodes.
The graphs generated by DPGG AN , while also having different local structures compared with those generated by GVAE, have more similar global structures with the original graphs.