MONET: D EBIASING G RAPH E MBEDDINGS VIA THE M ETADATA -O RTHOGONAL T RAINING U NIT
Anonymous authors
Paper under double-blind review
A BSTRACT
Are Graph Neural Networks (GNNs) fair?
In many real world graphs, the formation of edges is related to certain node attributes (e.g. gender, community, reputation).
In this case, standard GNNs using these edges will be biased by this information, as it is encoded in the structure of the adjacency matrix itself.
In this paper, we show that when metadata is correlated with the formation of node neighborhoods, unsupervised node embedding dimensions learn this metadata.
This bias implies an inability to control for important covariates in real-world applications, such as recommendation systems.
To solve these issues, we introduce the Metadata-Orthogonal Node Embedding
Training (MONET) unit, a generalizable neural network architecture for performing training-time linear debiasing of graph embeddings.
MONET achieves this by ensuring that the node embeddings are trained on a hyperplane orthogonal to that of the node metadata.
This effectively organizes unstructured embedding dimensions into an interpretable topology-only, metadata-only division with no linear interactions.
We illustrate the effectiveness of MONET though our experiments on a variety of real world graphs, which shows that our method can learn and remove the effect of arbitrary covariates in tasks such as preventing the leakage of political party afﬁliation in a blog network, and thwarting the gaming of embedding-based recommendation systems.
I NTRODUCTION
Graph embeddings – continuous, low-dimensional vector representations of nodes – have been eminently useful in network visualization, node classiﬁcation, link prediction, and many other graph learning tasks (10).
While graph embeddings can be estimated directly by unsupervised algorithms using the graph’s structure (e.g. 24; 28; 15; 25), there is often additional (non-relational) information available for each node in the graph.
This information, frequently referred to as node attributes or node metadata , can contain information that is useful for prediction tasks including demographic, geo-spatial, and/or textual features.
The interplay between a node’s metadata and edges is a rich and active area of research.
Interestingly, in a number of cases, this metadata can be measurably related to a graph’s structure (21), and in some instances there may be a causal relationship (the node’s attributes inﬂuence the formation of edges).
As such, metadata can enhance graph learning models (31; 20), and conversely, graphs can be used as regularizers in supervised and semi-supervised models of node features (32; 11).
Furthermore, metadata are commonly used as evaluation data for graph embeddings (8).
For example, node embeddings trained on a Flickr user graph were shown to predict user-speciﬁed Flickr “interests" (24).
This is presumably because users (as nodes) in the Flickr graph tend to follow users with similar interests, which illustrates a potential causal connection between node topology and node metadata.
However, despite the usefulness and prevalence of metadata in graph learning, there are instances where it desirable to design a system to avoid the effects of a particular kind of sensitive data.
For instance, the designers of a recommendation system may want to make recommendations independent of a user’s demographic information or location.
At ﬁrst glance, this may seem like an artiﬁcial dilemma – surely one could just avoid the problem by not adding such sensitive attributes to the model.
However, such an approach (ignoring a sensitive attribute) does not control for any existing correlations that may exist between the sensitive metadata and the edges of a node.
In other words, if the edges of the graph are correlated with sensitive metadata, then any algorithm which does not explicitly model and remove this correlation will be biased as a result of it.
Surprisingly, almost all of the existing work in the area (31; 35) has ignored this important realization.
In this work, we seek to refocus the discussion about graph learning with node metadata.
To this end, we propose a novel, general technique for extending graph representations with metadata embedding dimensions while debiasing the remaining (topology) dimensions.
Speciﬁcally, our contributions are the following:
The Metadata-Orthogonal Node Embedding Training (MONET) unit, a novel GNN algorithm which jointly embeds graph topology and graph metadata while enforcing linear decorrelation between the two embedding spaces.
Analysis which proves that a naive approach (adding metadata embeddings without MONET) leaks metadata information into topology embeddings, and that the MONET unit does not.
Experimental results on real world graphs which show that MONET can successfully “debias" topology embeddings while relegating metadata information to separate metadata embeddings.
P RELIMINARIES
Early graph embedding methods involved dimensionality reduction techniques like multidimensional scaling and singular value decomposition (8).
In this paper we use graph neural networks trained on random walks, similarly to DeepWalk (24).
DeepWalk and many subsequent methods ﬁrst generate a sequence of random walks from the graph, to create a “corpus" of node “sentences" which are then modeled via word embedding techniques (e.g. word2vec (19) or GloVe (23)) to learn low dimensional representations that preserve the observed co-occurrence similarity.
Let W be a d -dimensional graph embedding matrix, W ∈ R n × d , which aims to preserve the lowdimensional structure of a graph ( d << n ).
Rows of W correspond to nodes, and node pairs i, j with large dot-products W Ti W j should be structurally or topologically close in the graph.
As a concrete example, in this paper we consider the debiasing of a recently proposed graph embedding using the
GloVe model (6).
Its training objective is:
[Equation]
where U, V ∈ R n × d are the “center" and “context" embeddings, a, b ∈ R n × 1 are the biases, C is the walk-distance-weighted context co-occurrences, and f α is the loss smoothing function (23).
We use the GloVe model in the next section to illustrate topology/metadata embeddings and metadataorthogonal training. However, the MONET unit we propose is broadly generalizable.
To illustrate this, we also describe a MONET unit for DeepWalk (24), a popular graph embedding algorithm.
Notation.
In this paper, given a matrix A ∈ R n × d and an index i ∈ 1 , . . . , n , A i denotes the d ×i -th row vector of A .
Column indices will not be used.
0 n × d denotes the n × d zero matrix, and || · || F denotes the Frobenius norm.
M ETADATA E MBEDDINGS AND O RTHOGONAL T RAINING
In this section we present MONET, our proposed method for separating and controlling the effects of metadata on topology embeddings.
First, we begin by outlining the straightforward extension of metadata to traditional embedding models in Section 3.1.
Next, in Section 3.2, we prove that such a simple model will leak information from the metadata to the topology (structural) embeddings.
Then, in Section 3.3 we present MONET, our proposed approach for training embeddings of a graph’s structure which are not correlated with metadata.
Finally, we conclude with some analysis of MONET in Section 3.
3. J OINTLY M ODELING M ETADATA & T OPOLOGY
A natural ﬁrst approach to modeling the effects of metadata on the graph is to explicitly include the node metadata as part of a node embedding model.
For instance, to extend Eq.
(1), in addition to U and V (the “topology embeddings"), we can consider the node metadata M directly ( M ∈ R n × m , row vector M
i is the metadata for node u i ).
We then can deﬁne metadata embeddings X = MT 1 ,
Y = MT 2 , where T 1 , T 2 are trainable transformations, and propose the concatenations [ U, X ] and [ V, Y ] as full-graph representations.
The GloVe loss with metadata embeddings is:
[Equation]
While in this paper we demonstrate metadata embeddings within the GloVe model, they can be incorporated in any dot-product-based graph neural network.
For instance, the well-known DeepWalk (24) loss, which is based on word2vec (19), would incorporate metadata embeddings as follows:
[Equation]
Above, W is the set of context pairs from random walks, and K i is a set of negative samples associated with node i .
For GloVe, DeepWalk, and many other GNNs, this approach augments the overall graph representation by concatenating metadata-learned dimensions.
However, this naïve approach does not guarantee that the topology embeddings converge to be decorrelated from the metadata embeddings.
Suppose that the metadata (like demographic information) are indeed associated with the formation of links in the graph.
In this case, any algorithm which does not explicitly model and remove the association will be biased as a result of it.
In the next section we formalize this concept, which we call metadata leakage .
3. M ETADATA L EAKAGE IN G RAPH N EURAL N ETWORKS
Here, we formally deﬁne metadata leakage for general topology and metadata embeddings, and show how it can occur even in embedding models with separate metadata embeddings.
All proofs appear in the Appendix.
Deﬁnition 1.
The metadata leakage of metadata embeddings Z ∈ R n × d Z into topology embeddings
W ∈ R n × d is deﬁned ML ( Z, W ) :
= ||
Z T W || 2 F .
We say that there is no metadata leakage if and only if ML ( Z, W ) = 0 .
Without a more nuanced approach, metadata leakage can occur even in embedding models that explicitly include the metadata, like Eqs.
(2) and (3).
To demonstrate this, we consider for simplicity a reduced metadata-aware GloVe loss with W :=
U = V ∈ R n × d as the sole topology embedding and T :
= T 1 = T 2 ∈ R m ×
d Z as the sole metadata transformation parameter.
With Z := MT , the reduced loss is:
[Equation]
We now show that under a random update of the GloVe ∗ meta model in Eq.
(4), the expected metadata leakage is non-zero.
Speciﬁcally, let ( i, j ) be a node pair from C , and deﬁne δ W ( i, j ) as the incurred
Stochastic Gradient Descent update W (cid:48) ← W + δ W
i, j ) .
Suppose there is a “ground-truth" metadata transformation B ∈ R m × d B , and deﬁne ground-truth metadata embeddings
˜ Z := MB , which represent the “true" dimensions of the metadata effect on the co-occurrences C .
Deﬁne Σ B := BB T and Σ T :=
TT T .
With expectations taken with respect to the sampling of a pair ( i, j ) for Stochastic Gradient Descent, deﬁne µ W :=
E [ W i ] and Σ W :=
E [ W i W Ti ] .
Deﬁne µ M , Σ M similarly.
Then our main Theorem is as follows:
Given: topology embedding W , metadata embedding Z 1: procedure F ORWARD P ASS DEBIASING ( W , Z ) 2:
Compute Z left-singular vectors Q Z and projection P Z ← I n × n − Q Z Q TZ 3:
Compute orthogonal topology embedding W ⊥ ← P Z W 4: return debiased graph representation [ W ⊥ , Z ] 5: procedure B ACKWARD P ASS DEBIASING ( δ W ) 6:
Compute orthogonal topology embedding update δ ⊥ W ← P Z δ W 7:
Apply update W ⊥ ← W ⊥ + δ ⊥ W 8: return debiased topology embedding W ⊥
Theorem 1.
Assume Σ W =
σ W I d for σ W > 0 , µ W =
d × 1 , and µ M = 0 m × 1 .
Suppose for some ﬁxed θ ∈ R we have log( C ij )
= θ + ˜ Z Ti ˜
Z j .
Let ( i, j ) be a randomly sampled co-occurrence pair and W (cid:48)
the incurred update.
Then if E [ M i W Ti ] = β ∈ R m × d , we have
[Equation]
Importantly, Σ T and σ W are neural network hyperparameters, so we give a useful Corollary:
Corollary 1.
Under the assumptions of Theorem 1, E [ ML ( Z, W )]
= Ω( n || T T β || 2 F ) as n → ∞ .
Note that under reasonable GNN initialization schemes, T and β are random perturbations.
Thus,
Corollary 1 implies the surprising result that incorporating feed-forward metadata embeddings is not sufﬁcient to prevent metadata leakage in practical settings.
3. MONET: M ETADATA -O RTHOGONAL N ODE E MBEDDING T RAINING
Here, we introduce the Metadata-Orthogonal Node
Embedding Training (MONET) unit for training joint topology-metadata graph representations [ W, Z ] without metadata leakage.
MONET explicitly prevents the correlation between topology and metadata, by using the Singular Value Decomposition (SVD) of Z to orthogonalize updates to W during training.
MONET.
The MONET unit is a two-step algorithm applied to the training of a topology embedding in a neural network, and is detailed in Algorithm 1.
The input to a MONET unit is a metadata embedding Z ∈ R n × d z and a target topology embedding W ∈ R n × d for debiasing.
Then, let Q Z be the left-singular vectors of Z , and deﬁne the projection
P Z := I n × n − Q Z Q TZ .
In the forward pass procedure, debiased topology weights are obtained by using the projection W ⊥ =
P Z W .
Similarly,
W ⊥ is used in place of W in subsequent GNN layers.
In the backward pass, MONET also debiases the backpropagation update to the topology embedding, δ W , using δ ⊥ W = P Z δ W .
Figure 1 illustrates a geometric interpretation of the MONET algorithm.
Straightforward properties of the SVD show that MONET directly prevents metadata leakage:
Theorem 2.
Using Algorithm 1, ML ( Z, W ⊥ )
= 0 and ML ( Z, δ ⊥ )
= 0 .
We note that in this work we have only considered linear metadata leakage; debiasing nonlinear topology/metadata associations is an area of future work.
Implementation (MONET G and MONET D ).
We demonstrate MONET in our experiments by applying Algorithm 1 to Eq.
(2) and Eq.
We denote these models respectively by MONET G and
MONET D , for MONET “GloVe" and “DeepWalk".
For MONET G , we orthogonalize the input and output topology embeddings U, V with the summed metadata embeddings Z := X + Y .
By linearity, this implies Z -orthogonal training of the summed topology representation
W
=
U + V .
We note that working with the sums of center and context embeddings is the standard way to combine these matrices (23).
Figure 2 shows an illustration of MONET G .
MONET D is implemented similarly, and is fully described in the Appendix Section A.3.
Both prediction and training for W occur on a hyperplane orthogonal to Z .
In the forward pass,
W is projected onto the Z -orthogonal plane.
When an update δ W is proposed, it too is projected, resulting in the best metadata-orthogonal update.
This allows W to explore the space of unknown latent structure without bias from Z .
U and V are topology embeddings.
The
MONET unit adds a feed-forward transformation of the metadata, resulting in metadata embeddings X and Y .
Z =
X + Y gives the combined metadata representation, used to debias
U and V via P Z .
Dotted lines indicate stopped gradient ﬂow during backpropagation. 3. A NALYSIS
Here we address some brief remarks about the algorithmic complexity of MONET, and the interpretation of its parameters. Algorithmic Complexity. The bottleneck of MONET occurs in the SVD computation and orthogonalization. In our setting, the SVD is O ( nd 2 z ) (29).
The matrix P Z need not be computed to perform orthogonalization steps, as P Z W = W − Q Z ( Q TZ W ) , and the right-hand quantity is O ( ndd z ) to compute.
Hence the general complexity of the MONET unit is O ( nd z max { d, d z } ) .
In Table 3 we compare the wall clock time of MONET and baselines, showing only about a 10% overall time increase from standard GloVe.
Metadata Parameter Interpretation.
The i, j terms in the sum of the loss for GloVe models with metadata (GloVe meta and MONET G ) involve the dot product X Ti Y j
= M Ti T 1 T T 2 M j .
That expansion suggests that the matrix Σ T :
= T 1 T T 2 contains all pairwise metadata dimension relationships.
In other words, Σ T gives the direction and magnitude of the raw metadata effect on log co-occurrence, and is therefore a way to measure the extent to which the model has captured metadata information.
We will refer to this interpretation in the experiments that follow.
An important experiment will show that applying the MONET algorithm increases the magnitude of Σ T entries.
Connection to Adversarial Methods.
There are now many methods for supervised learning that use adversarial networks to produce data representations that are invariant to given factors (e.g.
Because we are introducing MONET in the unsupervised graph learning setting, none of these approaches (to our knowledge) apply out-of-the-box to produce a baseline.
To give an idea for how an adversarial approach might work as an alternative to MONET, we craft a adversarial version of GloVe which attempts to predict the metadata from the topology embeddings.
This method, which to our knowledge is novel, is fully described in Section 4.1 and the Appendix.
M ETADATA D EBIASING E XPERIMENTS
Here we empirically demonstrate Theorems 1 and 2 by conﬁrming the following hypotheses:
1. H1.
The MONET unit can remove leakage of metadata information from topology embeddings, so that the topology embeddings cannot predict the metadata.
H2.
The MONET unit can make recommender systems more robust to abuse by removing malicious user directions from rating graphs.
For all embedding models, we use the center-context embedding sum of topology embeddings
W :=
U + V as the graph representation for task evaluation.
Note that some standard baselines (e.g. DeepWalk) do not incorporate metadata and therefore only train topology embeddings.
All
GloVe-based models are trained with TensorFlow (1) using the AdaGrad optimizer (12) with initial learning rate 0.05.
DeepWalk models were trained using the gensim software (26).
4. Q UANTITATIVE E XPERIMENT : P OLITICAL B LOGS N ETWORK
To address H1, illustrating Theorem 1 and the effect of MONET debiasing, we embed the effect of political ideology on a blogger network (3).
The political blog network 2 has has 1,107 nodes corresponding to blog websites, 19,034 hyperlink edges between the blogs (after converting the graph to be undirected), and two clearly deﬁned, equally sized communities of liberal and conservative bloggers.
Methods and Design .
In this experiment all graph neural network models were trained on 5 iterations across 80 random walks per node of length 40 with context window size 10 (MONET D was trained on 20 iterations and used 5 negative samples per positive).
Topology embeddings had dimension 16, and metadata embeddings had dimension 2.
As one baseline, we use a random embedding generated from a 16-dimensional multivariate Normal. As our adversarial baseline, we apply the ideas introduced in (14) by adding a 2-layer MLP adversary to the GloVe model, referred to as GloVe adversary . The adversary is trained to predict political party from the topology embeddings (more detail given in Appendix A.4).
We measure embedding bias by the Macro-F1 score of a linear SVM predicting political party from the embeddings, using a LIBLINEAR implementation (7).
For each embedding set, we compute the mean and standard deviation Macro-F1 over 10 independent classiﬁcation repetitions, each trained using half of the node labels sampled at random.
To assess metadata information leakage, we also track the metadata dimension importance matrix Σ T :
= T 1 T T 2 , recalling its interpretation from Section 3.
Results .
Table 1 shows that the baselines DeepWalk and GloVe are highly effective at predicting political party, and therefore biased.
This is unsurprising, as these methods are trained without metadata information, and were originally intended to encode low-dimensional structure like that present in this data set.
The bias in DeepWalk and GloVe embeddings is further seen in their metadata leakage values, computed using political party one-hot vectors as metadata embeddings.
Considering the embedding models with metadata embeddings, we ﬁnd that, interestingly,
GloVe meta ’s topology embeddings are still able to predict political party with 88.3% Macro-F1.
Also, as predicted by Corollary 1, GloVe meta ’s metadata leakage remains O ( n ) .
This shows that simply concatenating metadata embeddings is not sufﬁcient to isolate the metadata effect.
In contrast,
MONET G and MONET D achieve random Macro-F1 and no metadata leakage (under machine precision), demonstrating that on this data, the MONET unit is necessary to debias the blog embeddings from political party.
Surprisingly, the MONET-enhanced models show signiﬁcantly less bias than random embeddings, reﬂecting the fact that even random embeddings will not be perfectly linearly de-correlated from any given sensitive attribute.
The contrast between GloVe meta and MONET G /MONET D is seen in two other ways.
First, there is a noticeable increase in Σ T magnitude when MONET is used, implying that GloVe meta metadata embeddings are not capturing all possible metadata information.
Second, as seen in Fig.
3, the 2-dimensional PCA plots of the GloVe meta embeddings still show political party separation, whereas the MONET G PCA dimensions reveal strong mixing.
In addition to the metrics in Table 1, we show additional metrics from this experiment in Appendix
Table 3.
In particular, that table contains results from a non-linear SVM applied to all embeddings.
Accuracy from that classiﬁer is high even on MONET embeddings, which emphasizes the fact that MONET only performs linear debiasing.
We also report wall times for each method, and the embdding pairwise distance correlation to the GloVe model.
These metrics (respectively) show that the SVD correction does not add substantial runtime, and that it does not overly corrupt the GloVe embedding signal. [Table] [Figure] [Table]
4. E XPERIMENT 2: T HWARTING A TTACKS ON G RAPH - BASED R ECOMMENDATION S YSTEMS
In this experiment we address H [Table] investigating the effectiveness of MONET to defend against a shilling attack (9) against graph-embedding based recommender systems (33) [Table]
In a shilling attack
[Table]
a number of users act together to artiﬁcially increase the likelihood that a particular inﬂuenced item will be recommended for a particular target item
[Table]
Data.
In a single repetition of this experiment, we inject an artiﬁcial shilling attack into the MovieLens 100k dataset 3 .
The raw data is represented as a bipartite graph with 943 users, 1682 items, and a total of 100,000 ratings (edges).
Each user has rated at least 20 items.
At random, we sampleitems into an inﬂuence set S I , and a target item i t to be attacked.
We take a random sample of 5% of the existing users to be the set of attackers, S A .
We then create a new graph, G attacked which in addition to all the existing ratings, contains new ratings from each attacker ∈ S A to each item
∈ S I as well as the target video.
(Note that this corresponds to several varieties of behavior including both incentivizing formerly good users, and account takeover.)
Design and Methods.
For each embedding method, we perform random walks through the new bipartite graph G attacked .
As we wish to study item recommendation, in the random walks, we simply remove user nodes each time they are visited (so the walks contain only pairwise co-occurrence information over items).
With any given network embedding, we measure its bias by the number of inﬂuence items in S I in top-20 embedding-nearest-neighbor list of i t .
As metadata, we allow MONET models to know the per-movie attacker rating count for each attacked movie.
However, to better demonstrate real-world performance, we only allow 50% (randomly sampled) attackers from the original 5% sample to be “known" when constructing these metadata.
As non-debiasing baselines, we compare against DeepWalk and Glove.
As debiasing baselines, we applied a generalized correlation removal framework developed for removing word embedding bias (27; 4).
Speciﬁcally, we tried two approaches to “debias” the GloVe embedding of the MovieLens graph – as the “gender" embedding direction, we tried both (a) the most attacked movie vector and (b) the sum of attacked movie vectors.
All methods use 128 dimensional topology embeddings and are trained on 100 random walks per node, each walk of length 5.
Results.
As seen in Table 2, the topology embeddings from MONET G are the least biased by a large margin, letting on average only 1.2 inﬂuence items in the top-20 neighbors of t i .
Interestingly, we note that this behavior occurs even though the majority of observed co-occurrences for the algorithm had nothing to do with the attack in question, and only the known 50% of attackers were used to construct the metadata.
MONET D had comparably less efﬁcacy in this experiment.
We speculate that this is due to a harmful effect of negative sampling on the learning of the metadata direction from the continuous attacker metadata - which would affect MONET’s ability to debias the DeepWalk embeddings.
Further research could investigate appropriate hyperparameter settings for MONET D in this case.
All other baselines (including those that explicitly model the attacker metadata) left at least around half of the attacked items in the top-20 list.
To measure the extent to which debiased embeddings retain the original recommendation signal, we compute the pair-wise embedding distances of each method, and compute their Pearson correlation to the standard GloVe embeddings.
We ﬁnd that MONET embedding distances achieve high correlation (0.83) to the original distances, showing that with MONET it is possible to nearly nullify a shilling attack while preserving most of the signal from the true, un-attacked ratings.
We note that the max-attack-embedding baseline higher embedding distance correlation, but this method let many more attacked items (on average) into higher ranks.
This reveals a trade-off between embedding debiasing and prediction efﬁcacy which has also been observed in other contexts (33).
R ELATED W ORK
Though graph learning is an immense ﬁeld, a minority of unsupervised graph embedding techniques involve graph metadata.
To our knowledge, none of these techniques involve either metadata orthogonalization or the capacity to learn arbitrary metadata transformations.
(36) is a matrix factorization approach which uses a shared node embedding matrix to factor both the graph adjacencies and the raw metadata in a joint loss, with a tunable parameter to control the inﬂuence of the metadata loss.
Similarly, (18) pre-computes a metadata similarity matrix and trains shared center-context embedding matrices on the metadata similarities and random walk similarities.
In contrast, we learn the direction and effect of metadata as neural network parameters, and we separate those parameters into unique embedding dimensions.
(31) and (35) are matrix factorization approaches which factor an approximation to the co-occurrence matrix into equally-sized metadata and topology embeddings, and were built mainly for text metadata.
Their approaches enforce metric space similarity and dimensional homogeneity between metadata and topology representations, restrictions that we do not rely on and are ill-suited to the setting with multiple types of arbitrarily-sized metadata.
(16) constructs random walks that traverse between the original graph and the metadata freely, an approach which runs counter to our ability to separate out the effects of metadata on graph adjacencies.
(20) introduce a version of the stochastic block model with metadata priors, and show that the estimated posteriors yield insight into the inﬂuence of metadata on the graph.
However, this model estimates a community partition and in/out-community probabilities - it does not yield embeddings either of the node topology or the node metadata.
There has been work in Natural Language Processing on removing gender bias from word embeddings (e.g. 4), but these methods operate with pre-computed embeddings and rely on identiﬁcation of gendered terminology.
Additionally, there has been a wealth of work studying semi-supervised learning with graphs (e.g.
32) and graph convolutional networks (e.g. 2; 17; 11), which use graph metadata as features.
While most semi-supervised and supervised neural networks for graphs indirectly produce embeddings that in some cases can be identiﬁed with feature and topology dimensions, they are trained as part of prediction or label propagation tasks.
Therefore, the topology embeddings are free to correlate with features to the extent that this serves the loss function - there is no explicit separation of topology and metadata dimensions.
In this paper, we have studied the beneﬁts of metadata orthogonalization in the unsupervised setting, and we leave the exploration of our techniques in the semi-supervised and supervised settings to future work.
As described at the end of Section 3, there are many approaches to data representation learning that use adversarial networks to produce attribute-invariant embeddings.
For instance, (30) and (13) use an adversary to allow feature embeddings to forget differences in data sources.
Similar techniques have been applied to debias word embeddings from gender information (34) and recommender graph embeddings from demographic information (5).
MONET differs from these approaches in a few key and consequential ways. First, (to our knowledge) none apply out-of-the-box to the unsupervised setting, which motivated our introduction of a baseline adversarial version of GloVe. Second, whereas MONET can accept any metadata in an appropriate design matrix M , different types of metadata require different adversary losses, which can require cumbersome tuning.
Third, adversarial approaches induce a trade-off between accuracy on the main task and debiasing.
In contrast, our work aims to minimize training error subject to perfect (linear) debiasing.
C ONCLUSION
In this work, we have shown that unsupervised training of graph embeddings induces bias from important graph metadata.
We proposed a novel solution to address this problem – the MetadataOrthogonal Node Embedding Training (MONET) unit.
The MONET unit is the ﬁrst graph learning technique for training-time debiasing of embeddings, using orthogonalization.
Our experimental results using real datasets showed that MONET is able to encode the effect of graph metadata in isolated embedding dimensions, and simultaneously remove the effect from other dimensions.
This has immediate practical applications, which we illustrate by mitigating a simulated shilling attack on a real dataset of movie ratings.
We note that, because MONET only performs linear debiasing, the method is simply a ﬁrst step in this area, and does not completely solve the problem of exact metadata independence.
That being said, we argue that this is not a major limitation for many practical uses.
As we show in the shilling attack experiment, MONET seems to greatly reduce bias when embeddings are used for simple nearest-neighbor lookups, which is a common application in graph-based recommender systems.
Furthermore, advanced non-linear classiﬁers are not always scalable to graphs commonly found in industrial applications.
This work was meant to introduce the basic principles underlying the need for the MONET technique, and show its utility in a shallow graph neural network (GloVe).
While we used a shallow network for instructional purposes, we note that MONET is generalizable, and MONET units can be used to debias any set of embeddings from another set during training.
Subsequent research can explore the use of MONET in deeper networks and potentially semi-supervised models or graph convolutional networks.
As MONET’s SVD calculation can be expensive with large graphs and large embedding dimensions, future research could be in assessing the effect of SVD approximations, or training algorithms that utilize caching of previous metadata embedding SVDs to speed up training.
A A PPENDIX
Proposition 1.
Under the assumptions of Theorem 1, we have
[Equation]
Proof.
Derivatives of GloVe ∗ meta yield that the i -th row of δ W ( i, j ) is d ij W Tj , where
[Equation]
Similarly the j -th row is d ij W Ti , and all other rows are zero vectors.
Hence
[Equation]
We derive the second term on the right-hand side of Equation 8; the ﬁrst term follows by symmetry.
Note ﬁrst that E Z i ( θ − a i − b j ) W Tj = 0 by independence and centering assumptions.
Second:
[Equation]
E [ W
[Equation] W [Equation] d = T T
[Equation]
W [Equation] by independence. Third:
[Equation]
by independence, and similarly E [ Z i ˜
Z Ti ˜ Z j W Tj ] = T T Σ M Σ B β .
Combining these with Equation 7, we have
[Equation]
Applying symmetry to the second term in Equation 8 completes the proof.
A. P ROOF OF T HEOREM
Proof.
Proposition 1 gives E [ Z T δ W ( i, j )]
= 2 T T [Σ M (Σ B − Σ T ) +
σ W
I m ] β .
Second, note that E [ M i W Ti ] = β ⇒ M T W = nβ and thus Z W = T T M T W =
nT T β .
Recalling that W (cid:48)
=
W + δ W
( i, j ) we have
[Equation]
Applying Jensen’s Inequality completes the proof.
A. P ROOF OF T HEOREM
Proof.
Consider metadata embeddings Z ∈ R n × d Z and, as in the MONET algorithm, deﬁne the projection P Z
= I
d Z − Q Z Q TZ , where Q Z are the left-singular vectors of Z .
By properties of the
SVD, Z T Q Z Q TZ =
Z T , and hence Z T P Z = 0 n × d Z .
This means that Z T W ⊥ =
Z T δ ⊥ W =
d Z × d , which completes the proof by deﬁnition of metadata leakage.
[Table] A. D ESCRIPTION OF MONET D : MONET IMPLEMENTED IN D EEP W ALK The implementation of MONET D follows the general procedure laid out in Algorithm 1. As with
MONET G , for DeepWalk the MONET unit adds a feed-forward transformation of the metadata to the graph representation, resulting in metadata embeddings X and Y (see Eq. 3).
Z =
X + Y gives the combined metadata representation, used to debias U and V via P Z (see Algorithm 1 and surrounding description).
MONET does not affect the negative sampling component of DeepWalk’s loss (Eq.
MONET debiases the topology embeddings as described, which are then used throughout the standard DeepWalk model (along with the metadata embeddings)
[Table]
A. A DVERSARIAL B ASELINE
As our adversarial baseline, we implemented a 2-layer MLP discriminator following the framework of (14).
The MLP had ReLU activations and an 8 dimensional hidden layer.
The MLP was trained to predict political party from the topology vectors of each batch of input nodes, using cross-entropy loss (discriminator task).
Then, the topology vectors were fed through the discriminators, but their negative logits were used for prediction (topology task).
The discrimnator task and topology task were evaluated and optimized after each optimization of the GloVe loss.
A. A DDITIONAL R ESULTS F ROM P OLITICAL B LOGS E XPERIMENT
In Table 3 we give the following additional metrics computed from the political blogs experiment, averaged over thirty repetitions:
Wall Time (sec) : user wait time in seconds from the beginning to end of each method.
The signiﬁcant increase seen from MONET D is due to DeepWalk’s negative sampling loss.
SVM Accuracy : to emphasize the fact that MONET only performs linear debiasing, we trained a non-linear SVM with an RBF kernel using a randomly-chosen 50% of the nodes as training points.
Embedding Distance Correlation w/GloVe : This metric was also used in the shilling experiment.
In our experiments, GloVe meta and MONET G were implemented into the GloVe model, so we use the correlation between the pairwise embedding distances between MONET models and GloVe to measure the amount that metadata embedding and SVD correction has “corrupted" the embeddings.
As the results show, among non-random methods, MONET G and MONET D are most corrupted.
This is because most of the community signal in the political blogs network is due to the political afﬁliation attribute, and MONET models explicitly removes linear correlation with that attribute.
That said, we see that MONET models still preserve a statistically signiﬁcant amount of signal from the GloVe embeddings above the random baseline.
Additionally, we perform a variant of the political blogs experiment to test the inductive performance of MONET.
Speciﬁcally, we aim to answer the question: if some nodes are not used to train the
MONET model, can we still use the MONET model to debias other embeddings for those nodes?
Given a subset of nodes N a , we train a MONET model on the induced subgraph G a := G ( N a ) .
We then attempt to debias the DeepWalk embeddings of held-out nodes N b := N \ N a .
To do this, we apply the MONET metadata transformation T 1 , T 2 , trained on G a , to the metadata of N b .
This yields inductive metadata embeddings Z b for the nodes N b .
We then apply the de-biasing projection given in Algorithm 1 to the DeepWalk embeddings for N b , and compute the linear SVM Macro/Micro-F scores to measure embddings bias.
Figure 4 shows the average Macro/Micro-F1 over 5 repetitions per proportion of held-out nodes.
For reference, we also give the accuracy scores for the un-corrected DeepWalk embeddings for N b .
Note that the linear SVM was trained on a 50% training-test split across N b only.
We ﬁnd that the MONET transformation learned from the given nodes is able to properly generalize to and debias the new data
N b .