P 2 IR: U NIVERSAL D EEP N ODE R EPRESENTATION VIA P ARTIAL P ERMUTATION I NVARIANT S ET F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph node representation learning is a central problem in social network analysis, aiming to learn the vector representation for each node in a graph.
The key problem is how to model the dependence of each node to its neighbor nodes since the neighborhood can uniquely characterize a graph.
Most existing approaches rely on deﬁning the speciﬁc neighborhood dependence as the computation mechanism of representations, which may exclude important subtle structures within the graph and dependence among neighbors.
Instead, we propose a novel graph node embedding method (namely P 2 IR) via developing a novel notion, namely partial permutation invariant set function to learn those subtle structures.
Our method can 1) learn an arbitrary form of the representation function from the neighborhood, without losing any potential dependence structures, 2) automatically decide the signiﬁcance of neighbors at different distances, and 3) be applicable to both homogeneous and heterogeneous graph embedding, which may contain multiple types of nodes.
Theoretical guarantee for the representation capability of our method has been proved for general homogeneous and heterogeneous graphs.
Evaluation results on benchmark data sets show that the proposed P 2 IR outperforms the state-of-the-art approaches on producing node vectors for classiﬁcation tasks.
I NTRODUCTION
Graph node representation learning (or graph embedding in some literature) is to learn the numerical representation of each node in a graph by vectors in a Euclidean space, where the geometric relationship reﬂects the structure of the original graph.
Nodes that are “close” in the graph are embedded to have similar vector representations (Cai et al., 2017).
The learned node vectors beneﬁt a number of graph analysis tasks, such as node classiﬁcation (Bhagat et al., 2011), link prediction (LibenNowell & Kleinberg, 2007), community detection (Fortunato, 2010), and many others (Hamilton et al., 2017b).
In order to preserve the node geometric relations in an embedded space, the similarity/proximity/distance of a node to its neighborhood is generally taken as input to different graph embedding approaches.
For example, matrix-factorization approaches work on pre-deﬁned pairwise similarity measures (e.g., different order of adjacency matrix)
(Zhang et al., 2018).
Deepwalk (Perozzi et al., 2014), node2vec (Grover & Leskovec, 2016) and other recent approaches (Dong et al., 2017) consider ﬂexible, stochastic measure of node similarity by the node co-occurrence on short random walks over the graph (Goyal & Ferrara, 2017).
Neighborhood autoencoder methods compress the information about a node’s local neighborhood that is described as a neighborhood vector containing the node’s pairwise similarity to all other nodes in the graph (Wang et al., 2016; Cao et al., 2016;
Bojchevski & Günnemann, 2018).
Neural network based approaches such as graph convolutional networks (GCN) and GraphSAGE apply convolution like functions on its surrounding neighborhood for aggregating neighborhood information (Kipf & Welling, 2016a; Hamilton et al., 2017a;
Tu et al., 2018).
However, most existing methods either explicitly or implicitly restrict the dependence form of each node to its neighbors and also the depth of neighbors.
Therefore, some important topology structures within the graph and subtle dependence between neighbors may not be captured in the embedded space.
For example, the methods in the family of stochastic walk lose the control of inﬂuence of further neighbors in a network since they usually utilize a ﬁxed hyper-parameter as a window size, or need to specify the degree of neighborhood proximity; GCN losses the ﬂexibility of neighborhood information since it ﬁxes the two-layer aggregation function for the representation computation.
In this work, we propose a Partial Permutation Invariant Representation method (P 2 IR) by developing a new notion of partial permutation invariant set function , that can
• learn node representation via a universal graph embedding function f , without pre-deﬁning pairwise similarity, specifying random walk parameters, or choosing aggregation functions among element-wise mean, a max-pooling neural network, or LSTMs; • capture the arbitrary relationship of each node to its neighbors; • automatically decide the signiﬁcance of nodes at different distances; • be generally applied to any graphs from simple homogeneous graphs to heterogeneous graphs with complicated types of nodes.
Evaluation results on benchmark data sets show that the proposed P 2 IR outperforms the state-of-theart approaches on producing node vectors for classiﬁcation tasks.
R ELATED W ORK
The main difference among various graph embedding methods lies in how they deﬁne the “closeness” between two nodes (Cai et al., 2017).
First-order proximity, second-order proximity or even high-order proximity have been widely studied for capturing the structural relationship between nodes (Tang et al., 2015b; Yang et al., 2017; Zhu et al., 2018).
Comprehensive reviews of graph embedding have been done by Cai et al.
Hamilton et al. (2017b); Goyal & Ferrara (2017); Yang et al.
In this section, we discuss the relevant graph embedding approaches in terms of how node closeness to neighboring nodes is measured, for highlighting our contribution on utilizing neighboring nodes in a most general manner.
Matrix Analysis on Graph Embedding As early as 2011, a spectral clustering method (Tang &
Liu, 2011) was proposed to take the eigenvalue decomposition of a normalized Laplacian matrix of a graph as an effective approach to obtain the embeddings of nodes.
Other similar approaches choose different similar matrix (from Laplacian matrix) to make a trade-off between modeling the “ﬁrst-order similarity” and “higher-order similarity” (Cao et al., 2015; Ou et al., 2016; Rossi et al., 2018).
Node content information can also be easily fused in the pairwise similarity measure, e.g., in TADW (Yang et al., 2015), as well as node label information, which resulting in semi-supervised graph embedding methods, e.g., MMDW (Tu et al., 2016).
Recently, an arbitrary-order proximity preserved network embedding method is introduced in (Zhang et al., 2018) based on matrix eigen-decomposition, which is applied to a pre-deﬁned high-order proximity matrix.
For heterogeneous networks, Huang et al.
(2017) proposed a label involved matrix analysis to learn the classiﬁcation result of each vertex within a semi-supervised framework.
Random Walk on a Graph to Node Representation Learning Both deepwalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016) are outstanding graph embedding methods to solve the node representation learning problem.
They convert the graph structures into a sequential context format with random walk (Lovász, 1993).
Thanks to the invention of Mikolov et al.
(2013) for word representation learning of sentences, deepwalk inherited the learning framework for words representation learning in paragraphs to generate the representation of nodes in random walk context.
And then node2vec evolved such the idea with additional hyper-parameters tuning for the trade-off between DFS and WFS to control the direction of random walk.
Struc2vec
(Ribeiro et al., 2017) also utilizes the multilayer graph to construct the graph node representations.
Gao & Huang (2018) propose a self-paced network embedding by introducing a dynamic negative sampling method to select difﬁcult negative context nodes in the training process.
Planetoid (Yang et al., 2016) proposed a semi-supervised learning framework by guiding random walk with available node label information.
The heterogeneity of graph nodes is often handled by a heterogeneous random walk procedure (Dong et al., 2017), or selected relation pairs (Chang et al., 2015).
Tang et al.
(2015a) considered the predictive text embedding problem on a large-scale heterogeneous text network and the proposed method is also based on pre-deﬁned heterogeneous random walks.
to Graph Embedding There are also methods focusing on aggregating or encoding the neighbors’ information to generate node embeddings.
DNGR (Cao et al., 2016) and
SDNE (Wang et al., 2016) introduce the autoencoder to construct the similarity function between the neighborhood vectors and the embedding of the target node.
DNGR deﬁnes neighborhood vectors based on random walks and SDNE introduces adjacency matrix and Laplacian eigenmaps to the deﬁnition of neighborhood vectors.
GraphWave (Donnat et al., 2018) learns the representation of each node’s neighborhood via leveraging heat wavelet diffusion patterns.
Although the idea of autoencoder is a great improvement, these methods are painful when the scale of the graph is up to millions of nodes.
Therefore, methods with neighborhood aggregation and convolutional encoders are involved to construct a local aggregation for node embedding, such as GCN (Kipf & Welling, 2016a;b; Schlichtkrull et al., 2018; van den Berg et al., 2017), FastGCN (Chen et al., 2018), column networks (Pham et al., 2017), the GraphSAGE algorithm (Hamilton et al., 2017a), GAT (Velickovic et al., 2017) and a recent DRNE (Tu et al., 2018) method using layer normalized LSTM to approximate the embedding of a target node by the aggregation of its neighbors’ embeddings.
The main idea of these methods is involving an iterative or recursive aggregation procedure e.g., convolutional kernels or pooling procedures to generate the embedding vectors for all nodes and such aggregation procedures are shared by all nodes in a graph.
The above-mentioned methods work differently on using neighboring nodes for node representation learning.
They require on pre-deﬁning pairwise similarity measure between nodes, or specifying random walk parameters, or choosing aggregation functions.
In practice, it takes non-trivial effort to tune these parameters or try different measures, especially when graphs are complicated with nodes in multiple types, i.e., heterogeneous graphs.
This work hence targets on making neighboring nodes play their roles in a most general manner such that their contributions are learned but not user-deﬁned.
The resultant embedding method has the ﬂexibility to work on any types of
homogeneous and heterogeneous graph.
Our proposed method P 2 IR has a natural advantage on avoiding any manual manipulation of random walking strategies or designs for the relationships between different types of nodes.
To the invention of set functions by Zaheer et al.
(2017), all existing valid mapping strategies from neighborhood to the target nodes can be represented by the set functions which are learnt by P 2 IR automatically.
T HE P ROPOSED P 2 IR METHOD
In this section, we ﬁrst formally deﬁne the problem, followed by introducing a new notion - partial permutation invariant function set function.
This section ends up with the overall P 2 IR method.
3. P ROBLEM SETUP
We target on designing graph embedding models for the most general graph that may include K different types of nodes ( k =1 for homogeneous graphs).
Formally, a graph G = {V , E} , where the node set V = ∪ Kk
=1 V k , i.e., V is composed of K disjoint types of nodes.
One instance of such a graph is the academic publication network, which includes different types of nodes for papers, publication venues, author names, author afﬁliations, research domains etc.
Given such a graph G , our goal is to learn the embedding vector for each node in this graph.
3. T HE PROPOSED UNIVERSAL GRAPH EMBEDDING MODEL
As we know, the position of a node in the embedded space is collaboratively determined by its neighboring nodes.
Therefore, the key issue to address for graph embedding is how to model the dependence of each node to its neighbors .
Most existing approaches need to pre-specify the neighbors (usually via the distance on the graph), and either explicitly or implicitly deﬁne a speciﬁc form to characterize the dependence between each node and its neighbors.
We propose a universal graph embedding model that neither requires to pre-deﬁne neighbors nor to specify the dependence form between each node and its neighbors.
The embedding vector x v of node v ∈ V k can be represented by its neighbors’ embedding vectors via a function f
[Equation]
, K } where X vk is a matrix with column vectors corresponding to the embedding of node v ’s neighbors in type k .
Note that the neighbors can include step-1 (or immediate) neighbors, step-2 neighbors, and even further neighbors.
How to characterize an arbitrary dependence to neighbors?
The key observation is that all neighboring nodes reachable from a target node at the same step are not distinguishable from the view of the target node.
Therefore, function f ( · ) should be a partially permutation invariant function .
That is, if we swap any elements in a X vk , the function value remains the same.
Unfortunately, the set function is not simply learn-able because the permutation property is hard to guarantee directly.
One straightforward idea to represent the partially permutation invariant function is to deﬁne it in the following form
[Equation]
where P n denotes the set of n -dimensional permutation matrices.
X v 1 P 1 is to permute the columns in X v 1 .
It is easy to verify that the function deﬁned in equation 1 is partially permutation invariant, but it is intractable because it involves (cid:81)
Nk =
( |V k | !)
“sum” items.
Our solution of learning function f is based on the following important theorem, which gives a neat way to represent any partial permutation invariant function.
The proof is provided in the Appendix.
Theorem 3.1.
[Partial permutation invariant function representation theorem] Let f be a continuous real-valued function deﬁned on a compact set X with the following form
[Equation]
If function f is partially permutation invariant, that is, any permutations of the values within the group G k for any k does not change the function value, then there must exist functions h ( · ) and { g k ( · ) } Kk =1 to approximate f with arbitrary precision in the following form
[Equation]
The rigorous proof is provided in Appendix.
This result essentially suggests a neat but universal way to represent any partially permutation invariant function.
For instance, a popular permutation invariant function widely used in deep learning max( · ) can be approximated in an arbitrary precision by
[Equation]
with g ( z ) =
[exp( kz ) z, exp( kz )] , and h ([ z, z (cid:48) ])
=
z/z (cid:48) , as long as k is large enough.
This is because
[Equation]
Therefore, based on Theorem 3.1, we only need to parameterize h ( · ) and { g k ( · ) } Kk =1 to learn the node embedding function, which makes the partial permutation function ﬁnally learn-able and tractable.
To make equation 2 approximate an arbitrary partial permutation invariant function, we only need to ensure that functions h and all g ’s can represent (or approximate) arbitrary real functions.
As we know, deep neural network provides an efﬁcient way to approximate an arbitrary continuous real function, because 3-layer neural networks is able to approximate any function (Cybenko, 1989;
Hornik, 1991; Haykin, 1994).
We next formulate the embedding model when considering different order of neighborhood.
How to automatically learn the importance of neighbors at different distances?
( 1-step neighbors )
From Theorem 3.1, any mapping function of a node v ∈ V k can be characterized by appropriately deﬁned functions ψ 1 , ψ 2 , · · · , ψ K , and φ k :
[Equation]
where Ω vn,k := Ω vn ∩ V k denotes the step- n neighbors of node v in node type k .
( Multi-step neighbors )
High order proximity has been shown to be beneﬁcial on generating high quality embedding vectors (Yang et al., 2017).
Extending the 1-step neighbor model, we can have the more general model where the representation of each node could depend on immediate (1-step) neighbors, 2-step neighbors, 3-step neighbors, and even inﬁnite-step neighbors.
[Equation]
where α 1 , α 2 , · · · , α ∞ are the weights for neighbors at different steps.
Let A ∈ { 0 , 1 } |V|×|V| be the adjacent matrix indicating all edges by 1 .
If we deﬁne the polynomial matrix function B ( · ) on the adjacent matrix A as B ( A ) = (cid:80) ∞ n =
α n A n , we can cast equation 3 into its matrix form
[Equation]
where X k denotes the representation matrix for nodes in type k , [ B ( A )]
V k ,v denotes the sub-matrix of B ( A ) indexed by column v and rows in V k , and function ¯ ψ (with ¯ · on the top of function ψ ( · ) ) is deﬁned as the function extension
[Equation]
Note that the embedding vectors for different type of nodes may be with different dimensions.
Homogeneous graph is a special case of the heterogeneous graph with K = 1 .
The above proposed model is thus naturally usable on a homogeneous graph.
To avoid optimizing inﬁnite number of coefﬁcients, we propose to use a 1-dimensional NN function ρ ( · ) :
R → R to equivalently represent the function B ( · ) to reduce the number of parameters based on the following observations
[Equation]
where A = U diag ( σ 1 , · · · , σ N ) U (cid:62) is the singular value decomposition.
We parameterize ρ : R (cid:55)→ R using 1-dimensional NN function, which allows us easily controlling the number of variables to optimize in ρ by choosing the number of layers and the number of nodes in each layer.
3. T HE OVERALL P 2 IR MODEL
For short, we denote the representation function for x v in equation 4 by
[Equation]
To fulﬁll the requirement of a speciﬁc learning task, we propose the complete P 2 IR model by involving a supervised component
[Equation]
k
[Equation] [Equation]
(5) where V label ⊂
V denotes the set of labeled nodes, and λ k > 0 balances the representation error and prediction error.
The ﬁrst unsupervised learning component restricts the representation error between the target node and its neighbors with L 2 norm since it is allowed to have noise in a practical graph.
And the supervised component is ﬂexible to be replaced with any designed learning task on the nodes in a graph.
For example, to a regression problem, a least square loss can be chosen to replace (cid:96) ( x , y ) and a cross entropy loss can be used to formulate a classiﬁcation learning problem.
To solve the problem in equation 5, we apply a stochastic gradient descent algorithm (SGD) to compute the effective solutions for the learning variables simultaneously.
3. C OMPLEXITY A NALYSIS
The most time consuming parts of P 2 IR are matrix decomposition and neural network backpropagation.
Hence, at the beginning, we discuss the complexity of matrix decomposition.
In general, the full spectral decomposition such as SVD takes the cubic computation complexity O ( N 3 ) where N is the total number of nodes in the network and in fact, the computation is pretty expensive when N is large.
However, since our adjacency matrix of the network is not a dense matrix (the number of edges are far less than O ( N 2 ) ), the complexity of the matrix decomposition can be reduced from O ( N 3 )
to
O ( N × # e ) where # e is the number of edges in the network.
And furthermore, with a very large
N (e.g., 10K), we only need to pick up a few top k eigenvalues to approximate the original large sparse matrix.
Therefore, the approximated version of the matrix decomposition can be completed in
O ( k × # e ) .
Then we talk about the backpropagation complexity of the whole computation network.
Once the optimization problem is fully deﬁned with a given graph, we only update the neural network parameters and a batch of node representations in the SGD procedure.
Therefore, the complexity of the backpropagation is in O ( t ( w + b × d )) where t is the number of iterations, w is the amount of parameters, b is the batch size and d is the representation dimension for each node.
In conclusion, the overall complexity of P 2 IR is O
( k × # e + t ( w + b × d )) when N is large.
E XPERIMENTS
This section reports experimental results to validate the proposed method, comparing to state-of-theart algorithms on benchmark datasets including both homogeneous and heterogeneous graphs, for evaluating the applicability of our proposed method on embedding problem of general graphs.
4. C OMPARISON ON HOMOGENEOUS GRAPHS
We consider the multi-class classiﬁcation problem over the homogeneous graphs.
Given a graph with partially labeled nodes, the goal is to learn the representation for each node for predicting the class for unlabeled nodes.
Datasets We evaluate the performance of P 2 IR and methods for comparison on ﬁve datasets.
• Cora (McCallum et al., 2000) is a paper citation network.
Each node is a paper.
There are 2708 papers and 5429 citation links in total.
Each paper is associated with one of 7 classes.
• CiteSeer (Giles et al., 1998) is another paper citation network.
CiteSeer contains 3,312 papers and 4,732 citations in total.
All these papers have been classiﬁed into 6 classes.
• Pubmed (Sen et al., 2008) is a larger and more complex citation networks compared to previous two datasets.
There are 19,717 vertexes and 88,651 citation links.
Papers are classiﬁed into 3 classes.
• Wikipedia (Sen et al., 2008) contains 2,405 online web-pages.
The 17,981 links are undirected between pairs of them.
All these pages are from 17 categories.
• Email-eu (Leskovec et al., 2007) is an Email communication network which illustrates the email communication relationships between researchers in a large European research institution.
There are 1,005 researchers and 25,571 links between them.
Department afﬁliations (42 in total) of the researchers are considered as labels to predict.
Baseline methods The compared baseline algorithms are listed below:
• Deepwalk (Perozzi et al., 2014) is an unsupervised graph embedding method which relies on the random walk and word2vec method.
For each vertex, we take 80 random walks with length 40, and set window size as 10.
Since deepwalk is unsupervised , we apply a logistic regression on the generated embeddings for node classiﬁcation.
• Node2vec (Grover & Leskovec, 2016) is an improved graph embedding method based on deepwalk.
We set the window size as 10, the walk length as 80 and the number of walks for each node is set to 100.
Similarly, the node2vec is unsupervised as well.
We apply the same evaluation procedure on the embeddings of node2vec as what we did for deepwalk.
• Struc2vec
(Ribeiro et al., 2017) applies the window size as 10, the
[Table]
• GraphWave
(Donnat et al., 2018) applies the heat coefﬁcient as 1000, the number of characteristic functions as 50, the number of Chebyshev approximations as 100 and the number of steps as 20.
• WYS (Watch-Your-Step) (Abu-El-Haija et al., 2017) applies the learning rate as 0.2, the highest power of normalized adjacency matrix as 5, the regularization co-efﬁcient as 0.1 and uses the “Log Graph Likelihood” as objective function.
• MMDW
(Tu et al., 2016) is a semi-supervised learning framework of graph embedding which combines matrix decomposition and SVM classiﬁcation.
We tune the method multiple times and take 0.01 as the hyper-parameter η in the method which is recommended by the authors.
• Planetoid (Yang et al., 2016) is a semi-supervised learning framework.
We set the batch size as 200, learning rate as 0.01, the batch size for label context loss as 200, and mute the node attributes as input while using softmax for the model output.
• GCN (Graph Convolutional Networks) (Kipf & Welling, 2016a) applies the convolutional neural networks into the semi-supervised embedding learning of graph.
We eliminate the node attributes for fairness as well.
• GAT (Graph Attention Networks) (Velickovic et al., 2017) applies the learning rate as 0.005, the co-efﬁcient of the regularization as 0.0005 and the number of hidden units as 64.
To make the comparison fair, we mute the node attributes in the training of GAT as well.
Experiment setup and results For fair comparison, the dimension of representation vectors is chosen to be the same for all algorithms (the dimension is 64 ).
The hyper-parameters are ﬁne-tuned for all of them.
The details of P 2 IR for multi-class case are as follows.
• Supervised Component: Softmax function is chosen to formulate our supervised component in equation 5 which is deﬁned as σ : R K
→ { z ∈ R K | (cid:80)
Ki =
1 z i = 1 , z i > 0 } .
For an arbitrary embedding x ∈ R d , we have the probability term as P ( y =
j | x ) =
[Equation]
for such node to be predicted as class j , where ( w j ∈ R d , b j ∈ R ) is a classiﬁer for class j .
Therefore, the whole supervised component in equation 5 is − (cid:80) v ∈V label (cid:80)
Kj y j log P ( y = j | x v )+ λ w R ( w ) , where R ( w ) is an L 2 regularization for w and λ w is chosen to be 10 − 3 .
Unsupervised embedding mapping Component: We design a two-layer NN with hidden dimension 64 to form the mapping from embeddings of neighbors to the target node and we also form a two-layer 1-to-1 NN with a 3 dimensional hidden layer to construct the matrix function for the adjacency matrix.
We pre-process the matrix A with an eigenvalue decomposition where the procedure preserve the highest 1000 eigenvalues in default.
The balance hyper-parameter λ 1 is set to be 10 − 3 .
[Table] [Table]
We take experiments on each data set and compare the performance among all baseline methods mentioned above.
Since it is the multi-class classiﬁcation scenario
[Table]
we use Accuracy as the evaluation criterion.
The percentage of labeled samples is chosen from 10% to 90% and the remaining samples are used for evaluation.
All experiments are repeated for ﬁve times and we report the mean and standard deviation of their performance in the Table 1 .
We highlight the best performance for each dataset with bold font style and the second best results with a “*”.
We can observe that in most cases, our method outperforms other methods.
4. C OMPARISON ON HETEROGENEOUS GRAPHS
We next conduct evaluation on heterogeneous graphs, where learned node embedding vectors are used for multi-label classiﬁcation.
Datasets The used datasets include
• BlogCatalog (Wang et al., 2010) is a social media network with 55,814 users and according to the interests of users, they are classiﬁed into multiple overlapped groups.
We take the ﬁve largest groups to evaluate the performance of methods.
Users and tags are two types of nodes.
The 5,413 tags are generated by users with their blogs as keywords.
Therefore, tags are shared with different users and also have connections since some tags are generated from the same blogs.
The number of edges between users, between tags and between users and tags are about 1.4M, 619K and 343K respectively.
• DBLP
(Ji et al., 2010) is an academic community network.
Here we obtain a subset of the large network with two types of nodes, authors and key words from authors’ publications.
The generated subgraph includes 27 K (authors) + 3 .
7 K (key words) vertexes.
The link between a pair of author indicates the coauthor relationships, and the link between an author and a word means the word belongs to at least one publication of this author.
There are 66,832 edges between pairs of authors and 338,210 edges between authors and words.
Each node can have multiple labels out of four in total.
Baseline Methods To illustrate the validation of the performance of P 2 IR on heterogeneous graphs, we conduct the experiments on two stages: (1) comparing P 2 IR with Deepwalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016) on the graphs by treating all nodes as the same type (P 2 IR with K = 1 in a homogeneous setting); (2) comparing P 2 IR with the state-of-the-art heterogeneous graph embedding method, metapath2vec (Dong et al., 2017), in a heterogeneous setting.
The hyper-parameters of the method are ﬁne-tuned and metapath2vec++ is chosen as the option for the comparison.
Experiment Setup and Results For fair comparison, the dimension of representation vectors is chosen to be the same for all algorithms (the dimension is 64).
We ﬁne-tune the hyper-parameter for all of them.
The details of P 2 IR for multi-label case are as follows
• Supervised Component: Since it is a multi-label classiﬁcation problem, each label can be treated as a binary classiﬁcation problem.
Therefore, we apply logistic regression for each label and for an arbitrary instance x
and the i -th label y i , the supervised component is formulated as l ( x , y i )
=
log(1+exp( w (cid:62) i x + b i ))
− y
i ( w (cid:62) i x + b i ) , where ( w i ∈ R d , b i ∈ R ) is the classiﬁer for the i -th label.
Therefore, the supervised component in equation 5 is deﬁned as (cid:80) v ∈V label (cid:80)
i l ( x , y i )
+ λ w R ( w i ) and R ( w i ) is the regularization component for w i , where λ w is chosen to be 10 − 4 .
Unsupervised Embedding Mapping Component:
We design a two-layes NN with a 64-dimensional hidden layer for each type of nodes with the types of nodes in its neighborhood to formulate the mapping from embedding of neighbors to the embedding of the target node.
We also form a two-layer 1-to-1 NN wth a 3 dimensional hidden layer to construct the matrix function for the adjacency matrix A for the whole graph.
We pre-process the matrix A with an eigenvalue decomposition by preserving the highest 1000 eigenvalues of magnitude in default.
We denote the nodes to be classiﬁed as typeand the other type as type 2.
The balance hyper-parameter [ λ 1 , λ 2 ] is set to be [0.2, 200].
For the datasets DBLP and BlogCatalog, we carry out the experiments on each of them and compare the performance among all methods mentioned above.
Since it is a multi-label classiﬁcation task, we take f1-score(macro, micro) as the evaluation score for the comparison.
The percentage of labeled samples is chosen from 10% to 90%, while the remaining samples are used for evaluation.
We repeat all experiments for three times and report the mean and standard deviation of their performance in the
Table 2.
We can observe that in most cases, P 2 IR in heterogeneous setting has the best performance.
P 2 IR in homogeneous setting is better than deepwalk and node2vec in the same homogeneous setting, and is even better than metapath2vec++ in heterogeneous setting (achieving the second best results).
Overall, the superior performance of P 2 IR in Table 1 and 2 demonstrates the validity of our proposed universal graph embedding mechanism.
[Table]
C ONCLUSION AND F UTURE W ORK
To summarize the whole work, we propose P 2 IR, a general graph embedding solution with the novel notion of partial permutation invariant set function, in principle that it can capture an arbitrary dependence between neighbors and automatically decide the signiﬁcance of neighbor nodes at different distance for both homogeneous and heterogeneous graphs.
We provide a theoretical guarantee for the effectiveness of the whole model.
Through conducting extensive experimental evaluation, we show P 2 IR has better performance on both homogeneous and heterogeneous graphs, comparing to the stochastic trajectories based, matrix analytics based and graph neural network based state-of-the-art algorithms.
For the future work, our model can be extended to more general cases, e.g., involving the rich content information out of graph neighborhood structures.
Appendix
We provide the proof to Theorem 3.1 in the appendix.
The proof in our paper borrows some idea from the proof sketch for a special case of our theorem in (Zaheer et al., 2017).
While (Zaheer et al., 2017) only provides a small paragraph to explain the raw idea on how to prove a special case, we provide complete and rigorous proof and nontrivial extension to the more general case.
Deﬁnition 5.1 (Power Sum Symmetric Polynomials) .
For every integer k ≥ 0 , the k -th power sum symmetric polynomial in variables x 1 , x 2 , · · · , x n is deﬁned as
[Equation]
Deﬁnition 5.2 (Monomial Symmetric Polynomials) .
For λ =
[ λ 1 , λ 2 , . . .
, λ n ] (cid:62) ∈ R n where λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ 0 , let Ω λ be the set of all permutations of the entries in λ .
For λ , the monomial symmetric polynomials is deﬁned as
[Equation]
Lemma 5.3.
If f : X → R is a continuous real-valued function deﬁned on the compact set
X ⊂ R n , then ∀ (cid:15) > 0 , there exists a polynomial function p : X → R , such that sup x ∈ X | f ( x ) − p ( x ) | < (cid:15) .
Proof.
This lemma is a direct application of Stone-Weierstrass Theorem (Stone, 1937; 1948) for the compact Hausdorff space.
Corollary 5.3.1.
For any continuous real-valued function f : X → R of the form deﬁned in
Theorem 3.1,
[Equation]
there exist polynomial functions p :
X → R also permutation invariant within each G k to closely approximate f .
Proof.
Let S k be the set of all possible permutations of { 1 , · · · , N k } for the group G k , where k ∈ { 1 , 2 , . .
. , K } .
Suppose σ k =
[ σ k 1 , σ k 2 , . . .
, σ kN k ] ∈ S k is some permutation for G k .
Then we know that ∀ σ k ∈ S k with k ∈ { 1 , 2 , . .
, K } , we have
[Equation]
where we let x k to denote [ x k, 1 , x k, 2 , · · · , x k,N k ] for G k and [ x k ] σ k to denote the permuted x k for simplicity.
There are in total (cid:81)
Kk =1 ( N k !)
permutations.
By lemma 5.3, we can know that ∀ (cid:15) > 0 , there exists a polynomial function q : X → R such that sup | f ( x 1 , x 2 , · · · , x K ) − q ( x 1 , x 2 , · · · , x K ) | ≤ (cid:15) .
This further implies that ∀ σ k ∈ S k with k ∈ { 1 , 2 , . . .
, K } , we have
[Equation]
, · · · , [ x K ]
[Equation]
We let p ( x 1 , x 2 , · · · , x K ) =
[Equation]
) and by the property of permutation invariant within each G k of the function f , we have that
[Equation]
Therefore
[Equation]
we can see that the polynomial function p ( x 1 , x 2 , · · · , x K ) is permutation invariant within each group and can closely approximate the function f ( x 1 , x 2 , · · · , x K )
[Equation]
Lemma
[Equation]
Suppose p : X → R is a real-valued polynomial in the form
[Equation]
which is permutation invariant within each group G k , k ∈ { 1 , 2 , .
. . , K } .
Then, p can be represented as
[Equation]
where c i is the rational coefﬁcient, m λ iK k denotes a monomial symmetric polynomial of variables
[Equation]
N k within group G k and λ iK ∈ R n are some certain exponents for the monomial symmetric polynomial.
Proof.
Suppose that the polynomial is expressed as a summation of monomials
[Equation]
where α rk,j is the exponent of x k,j for the r -th monomial.
We consider the term
[Equation]
for a certain r in the above summation.
Since p ( · ) is partially symmetric within the group G 1 , then there must exist terms having exponents of permuted α r 1 ,j , ∀ j ∈ { 1 , . .
N 1 } on x 1 ,j while the other factors c r (cid:81) N 2 j =1 x α r 2 ,j 2 ,j · · · (cid:81) N K j =1 x α rK,j K,j remain the same.
(Otherwise, the polynomial is not permutationinvariant w.r.t.
group G 1 .)
By summing up those terms together, we get a term with a factor of monomial symmetric polynomial as c r m α
[Equation]
(cid:81)
[Equation]
N K j =1 x α rK,j K,j .
Based on this term, we consider that the polynomial p ( · ) is also permuted invariant within G 2 , which implies that there must exist terms having exponents of permuted α r 2 ,j , ∀ j ∈ { 1 , . . .
, N 2 } on x 2 ,j while the other factors c r m α r 1 1 (cid:81) N 3 j =1 x α r 3 ,j 3 ,j · · · (cid:81) N K j =1 x α rK,j K,j remain the same.
Therefore, adding up all those terms together, we have c r m α r 1 1 m α r 2
(cid:81) N 3 j =1 x α r 3 ,j 3 ,j · · · (cid:81)
N K j =1 x α rK,j K,j .
Carrying out the above procedures recursively, we can eventually have c r m α r 1 1 m α r 2 2 . . .
m α rK K .
For all the remaining terms in the polynomial p ( · ) , performing the same steps will lead to completion of our proof.
Example 5.5.
p ( x 11 , x 12 , x 21 , x 22 ) is a polynomial that is permutation invariant within each group,
G 1 = { x 11 , x 12 } and G 2 = { x 21 , x 22 } .
We let p ( x 1 , 1 , x 1 , 2 , x 2 , 1 , x 2 , 2 ) =
x 1 , 1 x 21 , 2 x 2 , 1 x 22 , 2 + x 21 , 1 x 1 , 2 x 22 , 1 x 2 , 2 + x 1 , 1 x 21 , 2 x 22 , 1 x 2 , 2 + x 21 , 1 x 1 , 2 x 2 , 1 x
[Equation]
[Equation] [Equation]
[Equation]
It is easy to observe that p ( x 1 , 1 , x 1 , 2 , x 2 , 1 , x 2 , 2 ) can be rewritten as p
[Equation]
Lemma 5.6.
For a symmetric polynomial p ( x 1 , . . .
, x n ) of n variables, it can be expressed by a polynomial in the power sum symmetric polynomials p k ( x 1 , . . .
, x n ) for 1 ≤ k ≤ n with rational coefﬁcients.
Proof.
This lemma is a direct result of the fact that p 1 , p 2 , . .
. , p n are algebraically independent and the ring of symmetric polynomials with rational coefﬁcients can be generated as a Q -algebra, i.e. Q [ p 1 , p 2 , . .
p n ] .
(Macdonald, 1998; Stanley, 2001)
This lemma can also be easily proved by the combination of the fundamental theorem of symmetric polynomials and newton’s identities.
Proof of Theorem 3.
Proof.
By Corollary 5.3.1, for any function f : X → R that are permutation-invariant w.r.t.
the variables within each group G k , ∀ k ∈ { 1 , 2 , . .
, K } , which is in the form
[Equation]
we can always ﬁnd a polynomial function p : X → R that are also permutation-invariant w.r.t.
the variables within each group G k to approximate f closely in a given small error tolerance (cid:15) .
Here we ﬁrst deﬁne the function g k : R → R N k in the following form,
[Equation]
 which thus leads to
[Equation]

Therefore, we generate a sequence of power sums basis by (cid:80) N k n =1 g k ( x k,n ) .
By Lemma 5.4, the polynomial function p ( x 1 , 1 , x 1 , 2 , · · · , x 1 ,N 1 , · · · , x K, 1 , x K, 2 , · · · , x K,N K ) can be expressed as (cid:80) i =1 c
i m λ
i 1 1 m λ i 2 2 . . .
m λ iK K .
Note that each m λ ik k is a symmetric polynomial, which thus can be rewritten as a polynomial expression of power sum basis,
[Equation]
which has been generated by (cid:80) N k n =1 g k ( x k,n ) .
Hence the polynomial p ( x 1 , 1 , x 1 , 2 , · · · , x 1 ,N 1 , · · · , x K, 1 , x K, 2 , · · · , x K,N K ) is also a function of (cid:80) N k n =1 g k ( x k,n ) , ∀ k ∈ { 1 , 2 , . .
, K } , which will be expressed as p ( x 1 , 1 , x 1 , 2 , · · · , x 1 ,N 1 , · · · , x K, 1 , x K, 2 , · · · , x K,N K )
=
h (
N 1 (cid:88) n = g 1 ( x 1 ,n ) ,
N 2 (cid:88) n = g 2 ( x 2 ,n ) , · · · ,
N K (cid:88) n = g K ( x K,n )) .
Thus, the function f could be approximate in any given error tolerance (cid:15) by a polynomial h ( (cid:80) N 1 n =1 g 1 ( x 1 ,n ) , (cid:80) N 2 n =1 g 2 ( x 2 ,n ) , · · · , (cid:80) N K n =1 g K ( x K,n )) , which ﬁnishes our proof.