Zero-shot Transfer Learning within a Heterogeneous Graph via Knowledge Transfer Networks
Abstract
Data continuously emitted from industrial ecosystems such as social or e-commerce platforms are commonly represented as heterogeneous graphs (HG) composed of multiple node/edge types.
State-of-the-art graph learning methods for HGs known as heterogeneous graph neural networks (HGNNs) are applied to learn deep context-informed node representations.
However, many HG datasets from industrial applications suffer from label imbalance between node types.
As there is no direct way to learn using labels rooted at different node types, HGNNs have been applied on only a few node types with abundant labels.
We propose a zeroshot transfer learning module for HGNNs called a Knowledge Transfer Network (KTN) that transfers knowledge from label-abundant node types to zero-labeled node types through rich relational information given in the HG.
KTN is derived from the theoretical relationship, which we introduce in this work, between distinct feature extractors for each node types given in a HGNN model.
KTN improves performance of 6 different types of HGNN models by up to 960% for inference on zero-labeled node types and outperforms state-of-the-art transfer learning baselines by up to 73% across 18 different transfer learning tasks on HGs.
Introduction
Large technology companies commonly maintain large relational datasets, derived from their internal logs, that can be represented as or joined into a massive heterogeneous graph (HG) composed of nodes and edges with multiple types [30].
For instance, in e-commerce networks, there are product, user and review nodes, all interconnected by many edge types that represent forms of interactions such as spending (user-product), reviewing (user-review) and reviews-of (product-review).
To learn powerful features representing the complex multimodal structure of HGs, various heterogeneous graph neural networks (HGNN) have been proposed [15, 26, 35, 43].
A common issue in these industrial applications of HGNNs is the label imbalance among different node types.
For instance, publicly available content nodes – such as those representing video, text, and image content – are abundantly labelled, whereas labels for other types (such as user or account nodes) may be much more expensive to collect (or even not available, e.g. due to privacy restrictions).
This means that in most standard training settings, HGNN models can only learn to make good inferences for a few label-abundant node types, and can usually not make any inferences for the remaining node types, given the absence of any labels for them.
If there is a pair of label-abundant and zero-labeled node types which share an inference task, could we transfer knowledge between them?
One body of work has focused on transferring knowledge between nodes of the same type from two different HGs (i.e., graph-to-graph transfer learning) [16, 40].
However, these approaches are not applicable in many real-world scenarios for three reasons.
First, any external large-scale HG that could be used in a graph-to-graph transfer learning setting would almost surely be proprietary.
Second, even if practitioners could obtain access to an external industrial
HG, it is unlikely the distribution of that (source) graph would match their target graph well enough to apply transfer learning.
Finally, node types suffering label scarcity are likely to suffer the same issue on other HGs (e.g. user nodes).
In this paper, we introduce a zero-shot transfer learning approach for a single HG (assumed to be fully-owned by the practitioners), transferring knowledge from labelled to unlabelled node types.
This setting is distinct from any graph-to-graph transfer learning scenarios, since the source and target domains exist in the same HG dataset, and are assumed to have different node types.
Our model utilizes the shared context between source and target node types; for instance, in the e-commerce network, the latent (unknown) labels of user nodes can be strongly correlated with spending/reviewing patterns that are encoded in the cross-edges between user nodes and product/review nodes.
We propose a novel zero-shot transfer learning problem for this HG learning setting as follows:
Informal Problem Definition 1.
Zero-shot cross-type transfer learning running on a HG:
Given a heterogeneous graph G with node types { s , t , · · · } with abundant labels for source type s but no labels for target type t , can we train HGNNs to infer the labels of target-type nodes?
A naïve solution to this problem would be to re-use an HGNN pre-trained on the source nodes for target node inference, given that both domains exist in the same HG.
However, as we show in our paper, HGNNs have distinct parameter sets for each node type [15], edge type [26], and meta-path type [8, 35].
These facts cause HGNNs to learn entirely different feature extractors for nodes and edges of different types – in other words, the final embeddings for source and target nodes are computed by different sets of parameters in HGNNs.
Thus, a classifier pre-trained on source nodes will fail to perform well on inference tasks for target nodes.
The field of domain adaptation (DA) targets this setting, seeking to transfer knowledge from a source domain with abundant labels to a target domain which lacks them [9, 19, 20, 27].
However, distinct feature extractors across node types in HGNNs break a standard assumption of DA setting, namely that source and target domains share the same feature extractors (e.g., CNNs for both source and target image domains).
As we demonstrate in this paper, in our problem setting, DA approaches fail to achieve the outstanding performance they are known for in computer vision and NLP.
In our work, we first dissect the gradient path of HGNN models to see how feature extractors are designed independently for each node type, and some empirical consequences.
Then we theoretically analyze how feature extractors across node types relate to each other and how their output distributions could be represented in terms of each other.
We model this theoretical relationship between two feature extractors as a Knowledge Transfer Network (KTN) which can be optimized to transform target embeddings to fit the source domain distribution.
We perform an extensive evaluation of our method on 18 different transfer learning tasks on HGs where we compare against state-of-the-art domain adaptation baselines.
Additionally, in order to understand which environments are ideal for transferring knowledge between different node types for HGs, we formulate a synthetic heterogeneous graph generator that allows us to study the sensitivity of these methods.
Our main contributions are:
• Novel and practical problem definition:
To the best of our knowledge, KTN is the first zero-shot cross-type transfer learning method running on a heterogeneous graph — transfer knowledge across different node types within a heterogeneous graph.
• Generality: KTN is a principled approach analytically induced from the architecture of HGNNs, thus applicable to any HGNN models, showing up to 960% performance improvement for zerolabeled node inference across 6 different HGNN models.
• Effectiveness: We show that KTN outperforms state-of-the-art domain adaptation methods, being up to 73 .
3% higher in MRR on 18 different transfer learning tasks on HGs.
• Sensitivity Analysis: We provide a HG generator model to analyze how the node attribute and edge distributions of HGs affect the performance of KTN and other methods on the task.
Related Work
Various transfer learning problems have been defined on the graph domain.
[21, 22, 38, 42] construct synthetic graphs from unstructured data and transfer knowledge over the graphs using GNNs.
On the other hand, [13, 14, 24, 37] focus on extracting knowledge from the existing graph structures.
They pretrain a GNN model on a source graph and re-use the model on a target graph.
While these methods focus on homogeneous graphs, [16, 40] transfer HGNNs across different HGs.
However, none of them can be directly applied to our cross-type transfer learning problem running on a single
HG.
Here we cover two classes of learning approaches that are related to our problem.
As HGNNs are the models to which our method can be applied, we cover them extensively in Section 3.
Zero-shot domain adaptation (DA) transfers knowledge from a source domain with abundant labels to a target domain which lacks them.
Zero-shot DA can be categorized into three groups — MMD-based methods, adversarial methods, and optimal-transport-based methods.
MMD-based methods [18, 20, 29] minimize the maximum mean discrepancy (MMD) [11] between the mean embeddings of two distributions in reproducing kernel Hilbert space.
Adversarial methods [9, 19] are motivated by theory in [2, 3] suggesting that a good cross-domain representation contains no discriminative information about the origin of the input.
They learn domain-invariant features by a min-max game between the domain classifier and the feature extractor.
Optimal transport-based methods [27] estimate the empirical Wasserstein distance [25] between two domains and minimizes the distance in an adversarial manner.
All three categories rely on two networks — a feature extractor network and a task classifier network.
Adversarial and OT-based methods use an additional domain classifier network.
Due to the assumption that source and target domains have the same modality 2 , the standard DA setting assumes identical feature extractors across domains.
More descriptions can be found in Appendix A.9.
Label propagation (LP) approaches (e.g., [45]) use message-passing to pass each node’s label to its neighbors according to normalized edge weights.
LP relies on only a graph’s edges, and is therefore easily applied to a heterogeneous graph – labels are simply propagated across edges, regardless of type.
In this paper we also evaluate a similarly-simple baseline, embedding propagation (EP).
Similar to LP, EP recursively propagates source embeddings (computed using source labels) until they reach the target domain.
EP exploits both node attribute information and the node adjacencies, but only uses the source node embeddings.
Preliminaries
In this section we review heterogeneous graphs and heterogeneous graph neural networks (HGNNs).
3.1 Heterogeneous graph
Heterogeneous graphs (HGs) are an important abstraction for modeling the relational data of multimodal systems.
Formally, a heterogeneous graph is defined as G =
( V , E , T , R ) where the node set
V ; the edge set E consisting of ordered tuples e ij :=
( i, j ) with i, j ∈ V , where e ij ∈ E iff an edge exists from i to j ; the set of node types T with associated map τ : V (cid:55)→ T ; the set of relation types
R with associated map ϕ : E (cid:55)→ R .
This flexible formulation allows directed, multi-type edges.
We additionally assume the existence of a node attribute vector x i ∈
X τ ( i ) for each i ∈ V , where X t is an attribute matrix specific to nodes of type t .
3.2 Heterogeneous Graph Neural Networks (HGNN)
Various HGNN models have been proposed [15, 26, 35, 41, 43].
Fully-specified HGNN models have specialized parameters for each node type [15], edge type [26], and meta-path type [8] to most effectively utilize the complex relationships encoded in the HG data structure.
In this paper, we use a flavor of HGNN known as a Heterogeneous Message-Passing Neural Network (HMPNN) as our base model on which to demonstrate KTN (though KTN can be implemented in almost any HGNN, as we show in experiments in Section 6).
The HMPNN merely extends the standard MPNN [10] by specializing all transformation and message matrices in each node/edge type.
With its generality,
HMPNN is itself a base model for RGCN [26] and HGT [15], and is also widely used as a default
HGNN model in popular GNN libraries (e.g., pyG [7], TF-GNN [6], DGL [34]).
In a HMPNN, for any node j , the embedding of node j at the l - th layer is obtained with the following generic formulation:
[Equation]
where E ( j ) =
{ ( i, j
) ∈
E : i, j ∈ V} denotes all the edges which connect (directionally) to j .
The above operations typically involve type-specific parameters to exploit the inherent multiplicity of modalities in heterogeneous graphs.
First, we define a linear Message function:
[Equation]
i
[Equation]
where M ( l ) r are the specific message passing parameters for each edge type r ∈ R and each of L
HMPNN layers.
Then defining E r ( j ) as the set of edges of type r pointing to node j , the Aggregate function mean-pools messages by edge type, and concatenates:
[Equation]
Finally, the Transform function maps the message into a type-specific latent space:
[Equation]
where W ( l ) t are the specific transformation parameters for each node type t ∈ T and each of L
HMPNN layers.
By stacking L layers, HMPNN outputs highly contextualized final node representations, and the final node representations can be fed into another model to perform downstream heterogeneous network tasks, such as node classification or link prediction.
3.3 Problem definition
Using notations defined above, we formalize our novel transfer learning problem on HGs.
Problem Definition 1.
Zero-shot cross-type transfer learning running on a HG:
In a given heterogeneous graph G =
( V , E , T , R ) with node attributes X = ∪ t ∈T X t , assume node types s and t share a classification task { ( i, y i ) : i ∈ V s , V t } .
During the training phase, using labels { ( i, y i ) : i ∈ V s } only for source-type nodes, we train an HGNN model f : f ( G , X ) =
h i to get node embeddings
h i for all nodes i ∈ V and a classifier g : g ( h i )
=
ˆ y i to predict labels ˆ y i from the node embeddings h i .
During the test phase, our task is to predict labels { ( j, y j ) : j ∈ V t } of target-type nodes where none of labels of target-type nodes were used for training.
Cross-Type Feature Extractor Transformations in HGNNs
We define f t : G (cid:55)→ R d to be the “feature extractor" of a HGNN, which is composed of parameters participating to map input node attributes of type t into a shared feature space R d .
In this section, we derive a strict transformation between feature extractors within a HMPNN.
Specifically, for any two nodes i, j with types τ ( i ) =
s and τ ( j )
=
t , we derive an expression for f s in terms of f t , and use that expression to inspire a trainable transfer learning module called KTN in the following section.
For simplicity, throughout this section we ignore the activation α ( · ) within the Transform function in Equation (4).
4.1 Feature extractors in HMPNNs
We first reason intuitively about the differences between f s and f t when s ̸ = t , using a toy heterogeneous graph shown in Figure 1(a).
Consider nodes v 1 and v 2 , noticing that τ (1) ̸
= τ (2) .
Using
HMPNN’s equations (2)-(4) from Section 3.2, for any l ∈ { 0 , . . .
, L − 1 } we have
[Equation]
where h (0) j = x j .
From these equations, we see that h ( l ) 1 and h ( l ) 2 , which are features of different types, are extracted using disjoint sets of model parameters at l -th layer.
In a 2-layer HMPNN, this creates unique gradient backpropagation paths between the two node types, as illustrated in
Figures 1(b)-1(c).
In other words, even though the same HMPNN is applied to node types s and t , the feature extractors f s and f t have different computational paths.
Therefore they project node features into different latent spaces, and have different update equations during training.
[Figure]
[Figure]
4.2 Empirical gap between f s and f t
Here we study the experimental consequences of the above observation via simulation.
We first construct a synthetic graph extending the 2-type graph in Figure 1(a) to have multiple nodes per-type, and multiple classes.
To maximize the effects of having different feature extractors, we sample source and target nodes from the same feature distributions and each classes are well-separated in the both the graph and feature space (more details available in Appendix A.7.1)
On such a well-aligned heterogeneous graph, without considering the observation in Section 4.
there may seem to be no need for domain adaptation from f t to f s .
However, when we train the
HMPNN model solely on s -type nodes, as shown in Figure 2(a) we find the test accuracy for s -type nodes to be high ( 90% , blue line) and the test accuracy for t -type nodes to be quite low ( 25% , green line).
Now if instead we make the t -type nodes use the source feature extractor f s , much more transfer learning is possible ( ∼ 65% , orange line).
This shows that the different feature extractors present in the HMPNN model result in the significant performance drop, and simply matching input data distributions can not solve the problem.
To analyze this phenomenon at the level of backpropagation, in Figures 2(b)-2(c) we show the magnitude of gradients passed to parameters of source and target node types.
As illustrated in
Figures 1(b)-1(c)
, we find that the final-layer Transform parameter W (2) t for type- t nodes have zero gradients (Figure 2(b)), and similarly for the final-layer Message parameters (Figure 2(c)).
Additionally, those same parameters in the first-layer for t -type nodes have much smaller gradients than their s -type counterparts: W (1) t (blue line in Figure 2(b)), M (1) st and M (1) tt (blue and orange lines in Figure 2(c)) appear below than other lines.
This is because they contribute to f s less than f t
This case study shows that even when an HGNN is trained on a relatively simple, balanced, and class-separated heterogeneous graph, a model trained only on the source domain node type cannot transfer to the target domain node type.
4.3 Relationship between feature extractors in HMPNNs
We show that a HMPNN model provides different feature extractors for each node type.
However, still, f s and f t are built inside one HMPNN model and interchange intermediate feature embeddings with each other.
Both H ( L ) s and H ( L ) t (
the output of f s and f t ) are computed using the previous layer’s intermediate embeddings H ( L − 1) s , H ( L − 1) t , and any other connected node type embeddings
H ( L − 1) x at the L -th HMPNN layer.
Therefore H ( L ) s and H ( L ) t can be mathematically presented by each other using the ( L − 1) -th layer embeddings as connecting points, so do f s and f t .
Based on this intuition, we derive a strict transformation between f s and f t , which will motivate the core domain adaptation component of our proposed KTN model.
Theorem 1.
Given a heterogeneous graph G = {V , E , T , R} .
For any layer l > 0 , define the set of ( l − 1) - th layer HMPNN parameters as
[Equation]
Let A be the total n × n adjacency matrix.
Then for any s, t ∈ T there exist matrices A ∗ ts = a ts
[Equation]
A ) and Q ∗ ts =
q ts ( Q ( l − 1) )
such that
[Equation]
where a ts ( · ) and q ts ( · ) are matrix functions that depend only on s, t .
The full proof of Theorem 1 can be found in Appendix A.1.
Notice that in Equation 8, Q ∗ ts acts as a macro- Transform operator that maps H ( L ) t into the source domain, then A ∗ ts aggregates the mapped embeddings into s -type nodes.
In other words, Q ∗ ts acts as a mapping matrix from the target domain to the source domain.
To examine the implications, we run the same experiment as described in
Section 4.2, while this time mapping the target features H ( L ) t into the source domain by multiplying with Q ∗ ts in Equation 8 before passing over to a task classifier.
We see via the red line in Figure 2(a) that, with this mapping, the accuracy in the target domain becomes much closer to the accuracy in the source domain ( ∼ 70% ).
Thus, we use this theoretical transformation as a foundation for our trainable
HGNN domain adaptation module, introduced in the following section.
4.4 Generalized cross-type transformations for HGNNs
In this section we showed that a HMPNN feature extractor on the (label-abundant) source node type can be expressed in terms of the (label-scarce) target node type feature extractor, and this transformation enables cross-type zero-shot learning for the target node type.
As most HGNNs have distinct feature extractors for each node types (even single-layer HGNNs, which have specialized parameters for each node/edge attribute projection layer), they will suffer from the under-trained target embeddings phenomena we showed in Section 4.2.
For instance, in the meta-path based MAGNN model [8], meta-paths directing toward the target node types are generally less engaged in the source node feature computation and thus receive smaller gradients.
While we cannot derive the exact cross-type transformation for all possible HGNNs, the core intuition in the HMPNN theory holds, namely that H ( L ) s and H ( L ) t are both computed using the previous layer’s intermediate embeddings (see Section 4.3) across all HGNN models.
This observation allows us to extend our KTN and apply it to almost any HGNN.
We illustrate this by applying KTN to 6 different HGNN models in Section 6, where we see greatly increased target-type accuracy.
KTN: Trainable Cross-Type Transfer Learning for HGNNs
Inspired by these derivations we introduce our primary contribution, Knowledge Transfer Networks .
We begin by noting Equation 8 in Theorem 1 has a similar form to a single-layer graph convolutional network [17] with a deterministic transformation matrix ( Q ∗ ts ) and a combination of adjacency matrices directing from target node type t to source node type s ( A ∗ ts ).
Instead of hand-computing the mapping function Q ∗ ts for arbitrary HGs and HGNNs (
which would be intractable), we learn the mapping function by modelling Equation 8 as a trainable graph convolutional network, named the
Require:
heterogeneous graph G =
( V , E , T , R ) , node feature matrices X , source node type s , target node type t , adjacency matrix A ts , source node label matrix Y s .
Ensure: HGNN f , classifier g , KTN t KTN 1: H ( L ) s , H ( L ) t = f
( G , H (0) =
X ) 2: H ∗ t = t KTN ( H ( L ) t ) =
A ts H ( L ) t T ts 3: L KTN =
(cid:13)(cid:13)(cid:13) H ( L ) s − H ∗ t (cid:13)(cid:13)(cid:13)4:
L = L CL ( g ( H ( L ) s ) , Y s )
+ λ L KTN 5:
Update f , g , t using ∇L
Require: pretrained HGNN f , classifier g , KTN t
KTN Ensure: target node label matrix Y t 1: H ( L ) t = f
( G , H (0) =
X ) 2: H ∗ t =
H ( L ) t T ts 3: return g ( H ∗ t )
Knowledge Transfer Network, t KTN ( · ) .
KTN replaces Q ∗ ts and A ∗ ts in Equation 8 as follows:
[Equation]
[Equation]
where A ts is an adjacency matrix from node type t to s , and T ts is a trainable transformation matrix.
By minimizing L KTN , T ts is optimized to a mapping function of the target domain into the source domain.
5.1 Algorithm
We minimize a classification loss L CL and a transfer loss L KTN jointly with regard to a HGNN model f , a classifier g , and a knowledge transfer network t KTN as follows:
[Equation]
where λ is a hyperparameter regulating the effect of L KTN ; and f ( G , X ) s and f ( G , X ) t denote H ( L ) s and H ( L ) t , respectively.
Algorithm 1 describes a training step on the source domain.
After computing the node embeddings H ( L ) s and H ( L ) t , we map H ( L ) t to the source domain using t KTN and compute
L KTN .
Then, we update the models using gradients of L CL (computed using only source labels) and
L KTN .
Algorithm 2 describes the test phase on the target domain.
After we get node embeddings H ( L ) t from the trained HGNN model, we map H ( L ) t into the source domain using the trained transformation matrix T ts .
Finally we pass the transformed target embeddings H ∗ t into the classifier which was trained on the source domain.
Indirect Connections We note that in practice, the source and target node types can be indirectly connected in HGs via other node types (i.e., there is no A ts ).
Appendix A.2 describes how we can easily extend KTN to cover domain adaption scenarios in this case.
Experiments 6.1 Datasets
Open Academic Graph (OAG).
A dataset introduced in [44] composed of five types of nodes: papers (P), authors (A), institutions (I), venues (V), fields (F) and their corresponding relationships.
Paper and author nodes have text-based attributes, while institution, venue, and field nodes have text- and graph structure-based attributes.
Paper, author, and venue nodes are labeled with research fields in two hierarchical levels, L1 and L2.
We construct three field-specific subgraphs from OAG: computer science, computer networks, and machine learning academic graphs.
[Table] [Table]
PubMed.
[39] A network composed of of four types of nodes: genes (G), diseases (D), chemicals (C), and species (S), and their corresponding relationships.
Gene and chemical nodes have graph structure-based attributes, while disease and species nodes have text-based attributes.
Each gene and disease is labeled with a set of diseases they belong to or cause.
Synthetic heterogeneous graphs.
We generate stochastic block models [1] with multiple node/edge types.
We label each node types with the same set of classes.
Then we control feature/edge distributions within/between node types by manipulating feature/edge signal-to-noise ratio within/between classes.
A complete definition of the generative model is given in Appendix A.7.
6.2 Baselines
We compare KTN with two MMD-based DA methods (DAN [18], JAN [20]), three adversarial
DA methods (DANN [9], CDAN [19], CDAN-E [19]), one optimal transport-based method (WDGRL [27]), and two traditional graph mining methods (LP and EP [45]).
For KTN and DA methods, we use HMPNN as their base HGNN model.
More information of each method is in Appendix A.9.
6.3 Zero-shot transfer learning
We run 18 different zero-shot transfer learning tasks across three OAG and PubMed graphs.
We run each experiment 3 times and report the average value.
Due to the space limitation, we report the standard deviations and results on OAG-computer networks and OAG-machine learning in
Appendix A.3.
Each heterogeneous graph has the same node classification task for both source and target node types.
During training, we are given 1) the heterogeneous graph structure information (i.e., adjacency matrices), 2) input node attribute matrices for all node types, and 3) labels on source-type nodes for the classification task.
During the test phase, we predict labels on target-type nodes for the same classification task.
The performance is evaluated by NDCG and MRR — widely adopted ranking metrics [14, 15].
In Tables 1 and 2, our proposed method KTN consistently outperforms all baselines on all tasks and graphs by up to 73 .
3% higher in MRR (P-A(L1) task in OAG-CS, Table 1).
When we compare with the base accuracy using the model pretrained on the source domain without any domain adaptation ( 3 rd column, Base ), the results are even more impressive.
We see our method KTN provides relative gains of up to 340% higher MRR without using any labels from the target domain.
These results show the clear effectiveness of KTN on zero-shot transfer learning tasks on a heterogeneous graph.
We mention that venue and author node types are not directly connected in the OAG graphs (Figure 5(b) in Appendix), but KTN successfully transfer knowledge by passing intermediate node types.
[Table]
Baseline Performance.
Among baselines, MMD-based models (DAN and JAN) outperform adversarial-based methods (DANN, CDAN, and CDAN-E) and optimal transport-based method (WDGRL), unlike results reported in [19, 27].
These reversed results are a consequence of HGNN’s unique feature extractors for each domains.
Adversarial- and optimal transport-based methods define separate losses for source and target feature extractors (which are not separated in their shared feature extractor assumption), resulting in divergent gradients between different feature extractors and poor domain adaption performance.
This shows again the importance of considering different feature extractors in HGNNs.
More analysis can be found in Appendix A.4.
6.4 Generality of KTN
Here, we use 6 different HGNN models, R-GCN [26], HAN [35], HGT [15], MAGNN [8],
MPNN [10], and HMPNN.
MPNN maps all node types to the shared embedding space using projection matrices at the beginning then applies MPNN layers designed for homogeneous graphs.
In
Table 3, KTN improves accuracy on the target nodes across all HGNN models by up to 960% .
This shows the strong generality of KTN.
More results and analysis can be found in Appendix A.5.
6.5 Sensitivity analysis
Using our synthetic heterogeneous graph generator, we generate non-trivial 2-type heterogeneous graphs to examine how the feature and edge distributions affect the performance of KTN and other baselines.
We generate a range of test-case scenarios by manipulating (1) signal-to-noise ratio σ e of within-class edge distributions and (2) signal-to-noise ratio σ f of within-class feature distributions across all of the (a) source-source ( s ↔ s ), (b) target-target ( t ↔ t ), and (c) source-target ( s ↔ t ) relationships.
For instance, in Figure 3, for each edge type ( s ↔ s , t ↔ t , and s ↔ t , differentiated by colors), there are two different types of edges, edges within the same class (plain line) and edges across different classes (dotted line).
For each edge type, we manipulate σ e by changing the ratio of within-class and cross-class edges, and σ f by diverging feature distributions between classes.
Thus there will besignal-to-noise ratios in total.
A higher signalto-noise ratio for a particular data dimension (edges or features) across a particular relationship r ∈ { s ↔ s, t ↔ t, s ↔ t } means that classes are more separable in that data dimension, when comparing within r , and hence easier for
HGNNs.
Note that while tuning one σ ( · ) on the range [1 . 0 , 10 . 0] , the remaining five σ ( · ) are held at 10 . 0 .
Additionally, we vary σ ( · ) across two scenarios: (I) “easy": source and target node types have same number of clusters and same feature dimensions, (II) “hard" source and target node types have different number of clusters and feature dimensions.
Note that clusters and classes are different concepts in this experiment; several clusters could have the same class label.
[Figure]
Figures 4(a) and 4(c) show results from changing σ e across the three relation types.
We see that KTN is affected only by σ e across the s ↔ t (cross-types) relationship, which accords with our theory, since KTN exploits the between-type adjacency matrix.
Surprisingly, as seen in Figures 4(b) and 4(d), we do not find a similar dependence of KTN on σ f , which shows that KTN is robust by learning purely from edge homophily in the absence of feature homophily.
Regarding the performance of other baselines, EP shows similar tendencies as KTN— only affected by cross-type σ e — because
EP also relies on cross-type propagation along edges.
However, its accuracy is bounded above due to the fact that it does not exploit the (unlabelled) target features.
DAN and DANN, which do not exploit cross-type edges, are not affected by cross-type σ e .
However, they show either low or unstable performance across different scenarios.
DAN shows especially poor performance in the “hard" scenarios (Figure 4(c) and 4(d)), failing to deal with different feature spaces for source and target domains.
Conclusion
In this work, we proposed the first cross-type zero-shot transfer learning method for heterogeneous graphs.
Our method, Knowledge Transfer Networks (KTN) for Heterogeneous Graph Neural
Networks, transfers knowledge from label-abundant node types to label-scarce node types.
We illustrate KTN handily improves HGNN performances up to 960% for zero-labeled node types across 6 different HGNN models and outperforms many challenging baselines up to 73% higher in MRR.
Future work in the area includes filtering noisy edges between source and target domains and making
KTN more robust and less dependent on structure of given noisy heterogeneous graphs.
Limitation Statement Our transfer learning method is limited to node types sharing the same task (i.e., the same classifier).
We plan to extend our work to transfer knowledge between different tasks running on different node types on a heterogeneous graph.
Societal Impact Statement
KTN allows organizations to learn better from their own graph data, leveraging its structure without requiring external information.
We believe this has a number of positive applications (preserving model quality without needing extra datasets).
However like all modeling improvements, its true impact depends on what modeling tasks the technique is applied to.
Acknowledgement
MY gratefully acknowledges support from Amazon Graduate Research Fellowship.
GPUs are partially supported by AWS Cloud Credit for Research program.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We describe it in Section 7.
We also show which environments are ideal for our method via sensitivity test in Section 6.5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We describe it in Section 7.
We also mention about possible private information leakage in graph-to-graph transfer learning in Section 1.
By proposing transfer learning between node types in one heterogeneous graph, we can alleviate this issue.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We mention the full set of assumptions for Theorem 1 in Section 4.3.
(b) Did you include complete proofs of all theoretical results?
[Yes] The proof for Theorem 1 is described in Appendix A.1.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide a URL to our code in Appendix A.11.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] They are specified in Appendix A.8 and A.11.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] They are specified in Appendix A.3 and A.5.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] They are specified in Appendix A.11.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite public domain adaptation library ADA and heterogeneous graph neural network library OpenHGNN that we have used in our experiment in Appendix A.11.
(b) Did you mention the license of the assets?
[N/A] All codes and data we used are public.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We provide a URL to our code in Appendix A.11.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We do not use any personal data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
We do not use any personal data.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A.1 Proof of Theorem
In this proof, we adopt a simplified version of our message-passing function that ignores the skipconnection:
[Equation]
The HGNN trained in the experimental results shown in Figure 2 also does not use skip-connections and hence represents a theoretically-exact KTN component.
In the real experiments, we use (1) skip-connections, exploiting their usual benefits [12], and (2) the trainable version of KTN.
Proof.
Without loss of generality, we prove the result for the case where R = { ( s, t ) : s, t ∈ T } , meaning the type of an edge is identified with the (ordered) types of the neighbor nodes.
In other words, there is only one edge modality possible, such as a social networks with multiple node types (e.g. “users", “groups") but only one edge modality (“friendship").
In the case of multiple edge modalities (e.g. “friendship" and “message"), the result is extended trivially (through with more algebraically-dense forms of a ts and q ts ).
Throughout this proof, we use the following notation for the set of all j -adjacent edges of relation type r :
[Equation]
We write A x 1 x 2 to denote the sub-matrix of the total n x 1 × n x 2 adjacency matrix A corresponding to node types x 1 , x 2 ∈ T , and ¯ A x 1 x 2 to denote the same matrix divided by its column sum.
H ( l ) x is the (row-wise) n x × d l embedding matrix of x -type nodes at layer l .
We first compute the l - th output g ( l ) j of the Aggregate step defined for HGNNs in Equation 3, for any node j ∈ V such that τ ( j ) = s .
The output of Aggregate is a concatenation of edge-type-specific aggregations (see Equation 3).
Note that at most T = |T
| elements of this concatenation are non-zero, since the node j only participates in T out of T 2 relation types in R .
Thus we can write g ( l ) j as
[Equation]
where ¯ A ( j ) xs denotes the j - th column of ¯ A xs .
Notice that
[Equation]
and (again) at most T elements of the concatenation g ( l ) j are non-zero.
Therefore let W ( l ) xs be the columns of W ( l ) s that select the concatenated element of g ( l ) j corresponding to node type x .
Then we can write
[Equation]
Require:
heterogeneous graph G =
( V , E , T , R ) , node feature matrices X , adjacency matrices A xy where ∀ ( x, y ) ∈ R , source node type s , target node type t , source node label matrix Y s .
Ensure: HGNN f , classifier g , KTN t KTN 1: H ( L ) s , H ( L ) t = f
( H (0) =
X, G ) , H ∗ t =2: for each meta-path p =
t → s do 3: x = t , Z = H ( L ) t 4: for each node type y ∈ p do 5:
Z =
A xy ZT xy 6: x = y 7: end for 8:
H ∗ t =
H ∗ t + Z 9: end for 10: L KTN =
(cid:13)(cid:13)(cid:13) H ( L ) s − H ∗ t (cid:13)(cid:13)(cid:13)11:
L = L CL ( g ( H ( L ) s ) , Y s )
+ λ L KTN 12: Update f , g , t KTN using ∇L
Require: pretrained HGNN f , classifier g , KTN t
KTN Ensure: target node label matrix Y t 1: H ( L ) t = f
( H (0) =
X, G ) , H ∗ t =2: for each meta-path p =
t → s do 3: x = t , Z = H ( L ) t 4: for each node type y ∈ p do 5:
X =
ZT xy 6: x = y 7: end for 8:
H ∗ t =
H ∗ t + Z 9: end for 10: return g ( H ∗ t )
Defining the operator Q ( l ) xs := (cid:16) W ( l ) xs M ( l ) xs (cid:17) ′ , this implies that
H ( l ) s = (cid:88)
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
 
[Equation]
( l [Equation] [Equation] Q ( l
[Equation] [Equation]
  = ¯ A · s H ( l − 1) · Q ( l
[Equation]
1) · s Similarly we have H ( l ) t
[Equation]
Q ( l
[Equation]
1) · t .
Since H ( l ) s and H ( l ) t share the term H ( l − 1) · , we can write
[Equation]
where X − 1 denotes the pseudo-inverse.
■
A.2 Indirectly Connected Source and Target Node Types
When source and target node types are indirectly connected by another node type x , we can simply extend t KTN ( H ( L ) t ) to ( A xs ( A tx H ( L ) t T tx ) T xs ) where T tx T xs becomes a mapping function from target to source domains.
Algorithms 3 and 4 show how to extend KTN.
For every step ( x → y ) in a meta-path ( t → · · · → s ) connecting target node type t to source node type s , we define a transformation matrix T xy , run a convolution operation with an adjacency matrix A xy , then map the transformed embedding to the source domain.
We run the same process for all meta-paths connecting from target node type t to source node type s , and sum up them to match with the source embeddings.
In the test phase, we run the same process to get the transformed target embeddings, but this time, without adjacency matrices.
We run Algorithm 3 and 4 for transfer learning tasks between author and venue nodes which are indirectly connected by paper nodes in OAG graphs (Figure 5(b)).
As shown
[Table]
in Tables 4, 6, and 7, we successfully transfer HGNN models between author and venue nodes (A-V and V-A) for both L1 and L2 tasks.
Will lengths of meta-paths affect the performance?
We examine the performance of KTN varying the length of meta-paths between source and target node types.
In Table 8, accuracy decreases with longer meta-paths.
When we add additional meta-paths than the minimum path, it also brings noise in every edge types.
Note that author and venue nodes are indirectly connected by paper nodes; thus the minimum length of meta-paths in the A-V (L1) task is 2 .
The accuracy in the A-V (L1) task with a meta-path of length 1 is low because KTN fails to transfer anything with a meta-path shorter than the minimum.
Using the minimum length of meta-paths is enough for KTN.
A.3 More results for Zero-shot Transfer Learning in Section 6.
We show the zero-shot transfer learning results with error bars on OAG-computer science and Pubmed in Tables 4 and 5.
We also present the results with error bars on OAG-computer networks and OAGmachine learning in Tables 6 and 7, respectively.
Across all tasks and graphs, our proposed method KTN consistently outperforms all baselines.
A.4 Analysis for Baselines in Section 6.
Among baselines, MMD-based models (DAN and JAN) outperform adversarial based methods (DANN, CDAN, and CDAN-E) and optimal transport-based method (WDGRL), unlike results reported in [19, 27].
These reversed results are a consequence of HGNN’s unique feature extractors for source and target domains.
When f s and f t denote feature extractors for source and target domains, respectively, DANN and CDAN define their adversarial losses as a cross entropy loss ( E [log f s ] − E [log f t ] ) where gradients of the subloss E [log f s ] are passed only back to f s , while gradients of the subloss E [log f t ] are passed only back to f t .
Importantly, source and target feature extractors do not share any gradient information, resulting in divergence.
This did not occur in their original test environments where source and target domains share a single feature extractor.
Similarly,
WDGRL measures the first-order Wasserstein distance as an adversarial loss, which also brings the same effect as the cross-entropy loss we described above, leading to divergent gradients between source and target feature extractors.
On the other hand, DAN and JAN define a loss in terms of higher-order MMD between source and target features. Then the gradients of the loss passed to
[Table] [Table]
each feature extractor contain both source and target feature information, resulting in a more stable gradient estimation.
This shows again the importance of considering different feature extractors in
HGNNs.
JAN, CDAN, and CDAN-E often show out of memory issues in Tables 4, 6, and 7.
These baselines consider the classifier prediction whose dimension is equal to the number of classes in a given task.
That is why JAN, CDAN, and CDAN-E fail at the L2 field prediction tasks in OAG graphs where the number of classes is 17 , 729 .
LP performs worst among the baselines, showing the limitation of relying only on graph structures.
LP maintains a label vector with the length equal to the number of classes for each node, thus shows out-of-memory issues on tasks with large number of classes on large-size graphs (L2 tasks with 17 , 729 labels on the OAG-CS graph).
EP performs moderately well similar to other DA methods, but lower than KTN up to 60% absolute points of MRR, showing the limitation of not using target node attributes.
A.5 More results for Generality of KTN in Section 6.
We show KTN performance on 6 different types of HGNN models across 4 different zero-shot domain adaptation tasks on the OAG-computer science dataset in Table 9.
Descriptions of each
HGNN model can be found in Appendix A.10.
While KTN consistently improves all HGNN models’ performance on zero-labeled node types using labels rooted at other node types, the magnitude of improvements varies.
While HAN sees up to 4958% (V-A (L1) task in Table 9), MAGNN is improved by up to 47% (P-A(L1) task) or sees no improvement (A-V(L1) task).
This gap stems from how many parameters each HGNN model shares across node types.
HAN does not share any parameters during message-passing operations (every parameters are specialized to each meta-path), while MAGNN shares the transformation matrices across all node types at every layer.
By sharing more parameters with other node types, the gradients are more likely passed to target node type-specific parameters, leaving less room for improvement by KTN.
However, KTN is still necessary for any HGNN models.
MPNN who shares all parameters except the projection matrices that map different input attributes into the same embedding space at the beginning still sees improvements by up to 311% .
Again, these experimental results show the impact of having different feature extractors for each node type in HGNN models.
[Table] [Table]
A.6 Effect of trade-off coefficient λ
We examine the effect of λ on transfer learning performance.
In Table 10, as λ decreases, target accuracy decreases as expected.
Source accuracy also sees small drops since L KTN functions as a regularizer; by removing the regularization effect, source accuracy decreases.
When λ becomes large, both source and target accuracy drop significantly.
Source accuracy drops since the effect of L KTN becomes larger than the classification loss L CL .
Even the effect of transfer learning become larger by having larger λ , since the source accuracy which will be transferred to the target domain is low, the target accuracy is also low.
Thus we set λ to 1 throughout the experiments.
A.7 Synthetic Heterogeneous Graph Generator
Our synthetic heterogeneous graph generator is based on attributed Stochastic Block Models (SBM) [32, 33], using blocks (clusters) as the node classes.
In the attributed SBM, graphs exhibit within-type cluster homophily at the edge-level (nodes most-frequently connect to other nodes in the same cluster), and at the feature-level (nodes are closest in feature space to other nodes in the same cluster).
To produce heterogeneous graphs, we additionally introduce between-type cluster homophily, which allows us to model real-world heterogeneous graphs in which knowledge can be shared across node types.
The first step in generating a heterogeneous SBM is to decide how many clusters will partition each node type.
Assume within-type cluster counts k 1 , . .
. , k T .
We allow for between-type cluster homophily with a K T :=
min t { k t } -partition of clusters such that each cluster group has at least one corresponding cluster from other node types.
Secondly, edge-level homophily is controlled by signal-to-noise ratios σ e = p/q where nodes withincluster are connected with probability p and nodes between-cluster are connected with probability q .
Additionally, edges within one cluster group across different types (see previous paragraph) is controlled together with edges between different cluster groups across different types using some σ e .
In Section 6.5, we describe the manipulation of multiple σ e parameters within-and-between types.
Finally, node attributes are generated by a multivariate Normal mixture model, using the cluster partition as the mixture groups.
Thus feature-level homophily is controlled by increasing the variance of the cluster centers σ f , while keeping the within-cluster variance fixed.
Cross-type feature
[Table] [Table]
[Figure]
[Table] [Table]
homophily is controlled by setting a center of cluster centers within-type with linear combinations of centers (of cluster centers) of other types. Note that features of different types are allowed to have different dimensions, as we generate different mixture-model cluster centers for each cluster within each type
[Table]
A.7.1 Toy Heterogeneous Graph in Section 4.
Using the synthetic graph procedure described above
[Table]
we used the following hyperparameters to simulate the toy heterogeneous graph shown in Figure 2.
We generate the graph with 2 node types and 4 edge types as described in Figure 1(a), then we divide each node type into 4 classes of 400 nodes.
To generate an easy-to-transfer scenario, signal-to-noise ratio σ f between means of feature distributions are all set to 10 .
The ratio σ e of the number of intra-class edges to the number of inter-class edges is set to 10 among the same node types and across different node types.
The dimension of features is set to 24 for both node types.
A.7.2 Sensitivity test in Section 6.
Figure 5(a) shows the structures of graphs we used in Section 6.5.
The dimension of features are set to 24 for both node types for the "easy" scenario, and 32 and 48 for types s and t , respectively, for the "hard" scenario.
Additionally, for the "hard" scenario, we divide the t nodes into 8 clusters instead of 4 .
The other hyperparameters σ e and σ f are described in Section 6.5.
For each unique value of σ ( · ) across the six ( σ ( · ) , r ) pairs, we generate 5 heterogeneous graphs.
A.8 Real-world Dataset Open Academic Graph (OAG)
[28, 31, 44] is the largest publicly available heterogeneous graph.
It is composed of five types of nodes: papers, authors, institutions, venues, fields and their corresponding relationships.
Papers and authors have text-based attributes, while institutions, venues, and fields have text- and graph structure-based attributes.
To test the generalization of the proposed model, we construct three field-specific subgraphs from OAG: the Computer Science (OAG-CS),
Computer Networks (OAG-CN), and Machine Learning (OAG-ML) academic graphs.
Papers, authors, and venues are labeled with research fields in two hierarchical levels, L1 and L2.
OAG-CS has both L1 and L2 labels, while OAG-CN and OAG-ML have only L2 labels (their L labels are all "computer science").
Transfer learning is performed on the L1 and L2 field prediction tasks between papers, authors, and venues for each of the aforementioned subgraphs.
Note that paper-author (P-A) and paper-venue (P-V) are directly connected, while author-venue (A-V) are indirectly connected via papers.
[Figure]
The number of classes in the L1 task is 275 , while the number of classes in the L2 task is 17 , 729 .
The graph statistics are listed in Table 11, in which P–A, P–F, P–V,
A–I, P–P, and F-F denote the edges between paper and author, paper and field, paper and venue, author and institute, the citation links between two papers, and the hierarchical links between two fields.
The graph structure is described in Figure 5(b).
For paper nodes, features are generated from each paper’s title using a pre-trained XLNet [36].
For author nodes, features are averaged over features of papers they published.
Feature dimension of paper and author nodes is 769 .
For venue, institution, and field node types, features of dimensionare generated from their heterogeneous graph structures using metapath2vec [5].
PubMed [39] is a novel biomedical network constructed through text mining and manual processing on biomedical literature.
PubMed is composed of genes, diseases, chemicals, and species.
Each gene or disease is labeled with a set of diseases (e.g., cardiovascular disease) they belong to or cause.
Transfer learning is performed on a disease prediction task between genes and disease node types.
The number of classes in the disease prediction task is 8 .
The graph statistics are listed in Table 12, in which G, D, C, and S denote genes, diseases, chemicals, and species node types.
The graph structure is described in Figure 5(c).
For gene and chemical nodes, features of dimension 200 are generated from related PubMed papers using word2vec [23].
For diseases and species nodes, features of dimension 50 are generated based on their graph structures using TransE [4].
A.9 Baselines
Zero-shot domain adaptation can be categorized into three groups — MMD-based methods, adversarial methods, and optimal-transport-based methods.
MMD-based methods [18, 20, 29] minimize the maximum mean discrepancy (MMD) [11] between the mean embeddings of two distributions in reproducing kernel Hilbert space.
DAN [18] enhances the feature transferability by minimizing multi-kernel MMD in several task-specific layers.
JAN [20] aligns the joint distributions of multiple domain-specific layers based on a joint maximum mean discrepancy (JMMD) criterion.
Adversarial methods [9, 19] are motivated by theory in [2, 3] suggesting that a good cross-domain representation contains no discriminative information about the origin of the input.
They learn domain invariant features by a min-max game between the domain classifier and the feature extractor.
DANN [9] learns domain invariant features by a min-max game between the domain classifier and the feature extractor.
CDAN [19] exploits discriminative information conveyed in the classifier predictions to assist adversarial adaptation.
CDAN-E [19] extends CDAN to condition the domain discriminator on the uncertainty of classifier predictions, prioritizing the discriminator on easy-totransfer examples.
Optimal transport-based methods [27] estimate the empirical Wasserstein distance [25] between two domains and minimizes the distance in an adversarial manner. Optimal transport-based method are based on a theoretical analysis [25] that Wasserstein distance can guarantee generalization for domain adaptation.
WDGRL [27] estimates the empirical Wasserstein distance between two domains and minimizes the distance in an adversarial manner.
A.10 HGNN models
We briefly describe 6 heterogeneous graph neural networks (HGNN) models we used in the experiments.
MPNN (message passing neural networks) [10] is originally designed for homogeneous graphs.
We extend MPNN to process heterogeneous graphs by adding projection matrices that project input attributes of different node types into the same feature space before running the original MPNN.
R-GCN [26] extends MPNN by specializing message matrices in each edge type, while HMPNN specializes all transformation and message matrices in each node/edge type in MPNN.
HGT [15] extends HMPNN by adding attention modules.
The attention modules have node-type-specific key/query projection matrices and edge-type-specific key-query similarity matrices, following the transformer architecture.
HAN [35] is a meta-path-based model who specializes parameters in each meta-path.
HAN exploits meta-path-specific attention modules to aggregate features of neighboring nodes connected by each meta-path.
Then HAN aggregates embeddings of different meta-paths with semantic-level attention modules.
MAGNN [8] is another meta-path-based HGNN model.
MAGNN aggregates features of all nearby nodes sitting on each meta-path using intra-meta-path attention modules.
Then MAGNN aggregates features of different meta-paths using inter-meta-path attention modules.
All HGNN models we describe above have layer-wise parameters.
As all HGNN models have parameters specialized in either node/edge/meta-path types, they all have distinct feature extractors for each node types, thus, they will suffer from the under-trained target node phenomena we showed in Section 4.2.
Also, because the core intuition in KTN — namely that embeddings of any node types at the last layer are computed using the same set of the previous layer’s intermediate embeddings (see
Section 4.3) — holds across all HGNN models, KTN can be applied to any HGNN models and show greatly increased target-type accuracy.
A.11 Experimental Settings
All experiments were conducted on the same p2.xlarge Amazon EC2 instance.
Here, we describe the structure of HGNNs used in each heterogeneous graph.
Open Academic Graph: We use a 4 -layered HGNN with transformation and message parameters of dimension 128 for KTN and other baselines.
Learning rate is set to 10 − 4 .
PubMed: We use a single-layered HGNN with transformation and message parameters of dimension 10 for KTN and other baselines.
Learning rate is set to 5 × 10 − 5 .
Synthetic Heterogeneous Graphs: We use a 2 -layered HGNN with transformation and message parameters of dimension 128 for KTN and other baselines.
Learning rate is set to 10 − 4 .
We implement LP, EP and KTN using Pytorch.
For the domain adaptation baselines (DAN, JAN,
DANN, CDAN, CDAN-E, and WDGRL), we use a public domain adaptation library ADA 3 .
We match the numbers of layers and dimensions of hidden embeddings across all HGNN models.
We implement MPNN and HMPNN using Pytorch.
For other HGNN models (R-GCN, HAN, HGT, and
MAGNN), we use an open-source toolkit for Heterogeneous Graph Neural Network (OpenHGNN) 4 .
Our code is publicly available 5 .