Zero-shot Transfer Learning within a Heterogeneous Graph via Knowledge Transfer Networks
Abstract
Data continuously emitted from industrial ecosystems such as social or e-commerce platforms are commonly represented as heterogeneous graphs (HG) composed of multiple node/edge types.
State-of-the-art graph learning methods for HGs known as heterogeneous graph neural networks (HGNNs) are applied to learn deep context-informed node representations.
However, many HG datasets from industrial applications suffer from label imbalance between node types.
As there is no direct way to learn using labels rooted at different node types, HGNNs have been applied on only a few node types with abundant labels.
We propose a zeroshot transfer learning module for HGNNs called a Knowledge Transfer Network (KTN) that transfers knowledge from label-abundant node types to zero-labeled node types through rich relational information given in the HG.
KTN is derived from the theoretical relationship, which we introduce in this work, between distinct feature extractors for each node types given in a HGNN model.
KTN improves performance of 6 different types of HGNN models by up to 960% for inference on zero-labeled node types and outperforms state-of-the-art transfer learning baselines by up to 73% across 18 different transfer learning tasks on HGs.
Introduction
Large technology companies commonly maintain large relational datasets, derived from their internal logs, that can be represented as or joined into a massive heterogeneous graph (HG) composed of nodes and edges with multiple types (30).
For instance, in e-commerce networks, there are product, user and review nodes, all interconnected by many edge types that represent forms of interactions such as spending (user-product), reviewing (user-review) and reviews-of (product-review).
To learn powerful features representing the complex multimodal structure of HGs, various heterogeneous graph neural networks (HGNN) have been proposed (15; 26; 35; 43).
A common issue in these industrial applications of HGNNs is the label imbalance among different node types.
For instance, publicly available content nodes – such as those representing video, text, and image content – are abundantly labelled, whereas labels for other types (such as user or account nodes) may be much more expensive to collect (or even not available, e.g. due to privacy restrictions).
This means that in most standard training settings, HGNN models can only learn to make good inferences for a few label-abundant node types, and can usually not make any inferences for the remaining node types, given the absence of any labels for them.
If there is a pair of label-abundant and zero-labeled node types which share an inference task, could we transfer knowledge between them?
One body of work has focused on transferring knowledge between nodes of the same type from two different HGs (i.e., graph-to-graph transfer learning) (16; 40).
However, these approaches are not applicable in many real-world scenarios for three reasons.
First, any external large-scale HG that could be used in a graph-to-graph transfer learning setting would almost surely be proprietary.
Second, even if practitioners could obtain access to an external industrial
HG, it is unlikely the distribution of that (source) graph would match their target graph well enough to apply transfer learning.
Finally, node types suffering label scarcity are likely to suffer the same issue on other HGs (e.g. user nodes).
In this paper, we introduce a zero-shot transfer learning approach for a single HG (assumed to be fully-owned by the practitioners), transferring knowledge from labelled to unlabelled node types.
This setting is distinct from any graph-to-graph transfer learning scenarios, since the source and target domains exist in the same HG dataset, and are assumed to have different node types.
Our model utilizes the shared context between source and target node types; for instance, in the e-commerce network, the latent (unknown) labels of user nodes can be strongly correlated with spending/reviewing patterns that are encoded in the cross-edges between user nodes and product/review nodes.
We propose a novel zero-shot transfer learning problem for this HG learning setting as follows:
Informal Problem Definition 1.
Zero-shot cross-type transfer learning running on a HG:
Given a heterogeneous graph G with node types { s , t , · · · } with abundant labels for source type s but no labels for target type t , can we train HGNNs to infer the labels of target-type nodes?
A naïve solution to this problem would be to re-use an HGNN pre-trained on the source nodes for target node inference, given that both domains exist in the same HG.
However, as we show in our paper, HGNNs have distinct parameter sets for each node type (15), edge type (26), and meta-path type (8; 35).
These facts cause HGNNs to learn entirely different feature extractors for nodes and edges of different types – in other words, the final embeddings for source and target nodes are computed by different sets of parameters in HGNNs.
Thus, a classifier pre-trained on source nodes will fail to perform well on inference tasks for target nodes.
The field of domain adaptation (DA) targets this setting, seeking to transfer knowledge from a source domain with abundant labels to a target domain which lacks them (9; 19; 20; 27).
However, distinct feature extractors across node types in HGNNs break a standard assumption of DA setting, namely that source and target domains share the same feature extractors (e.g., CNNs for both source and target image domains).
As we demonstrate in this paper, in our problem setting, DA approaches fail to achieve the outstanding performance they are known for in computer vision and NLP.
In our work, we first dissect the gradient path of HGNN models to see how feature extractors are designed independently for each node type, and some empirical consequences.
Then we theoretically analyze how feature extractors across node types relate to each other and how their output distributions could be represented in terms of each other.
We model this theoretical relationship between two feature extractors as a Knowledge Transfer Network (KTN) which can be optimized to transform target embeddings to fit the source domain distribution.
We perform an extensive evaluation of our method on 18 different transfer learning tasks on HGs where we compare against state-of-the-art domain adaptation baselines.
Additionally, in order to understand which environments are ideal for transferring knowledge between different node types for HGs, we formulate a synthetic heterogeneous graph generator that allows us to study the sensitivity of these methods.
Our main contributions are:
• Novel and practical problem definition:
To the best of our knowledge, KTN is the first zero-shot cross-type transfer learning method running on a heterogeneous graph — transfer knowledge across different node types within a heterogeneous graph.
• Generality: KTN is a principled approach analytically induced from the architecture of HGNNs, thus applicable to any HGNN models, showing up to 960% performance improvement for zerolabeled node inference across 6 different HGNN models.
• Effectiveness: We show that KTN outperforms state-of-the-art domain adaptation methods, being up to 73 .
3% higher in MRR on 18 different transfer learning tasks on HGs.
• Sensitivity Analysis: We provide a HG generator model to analyze how the node attribute and edge distributions of HGs affect the performance of KTN and other methods on the task.
Related Work
Various transfer learning problems have been defined on the graph domain.
(21; 22; 38; 42) construct synthetic graphs from unstructured data and transfer knowledge over the graphs using GNNs.
On the other hand, (13; 14; 24; 37) focus on extracting knowledge from the existing graph structures.
They pretrain a GNN model on a source graph and re-use the model on a target graph.
While these methods focus on homogeneous graphs, (16; 40) transfer HGNNs across different HGs.
However, none of them can be directly applied to our cross-type transfer learning problem running on a single
HG.
Here we cover two classes of learning approaches that are related to our problem.
As HGNNs are the models to which our method can be applied, we cover them extensively in Section 3.
Zero-shot domain adaptation (DA) transfers knowledge from a source domain with abundant labels to a target domain which lacks them.
Zero-shot DA can be categorized into three groups — MMD-based methods, adversarial methods, and optimal-transport-based methods.
MMD-based methods (18; 20; 29) minimize the maximum mean discrepancy (MMD) (11) between the mean embeddings of two distributions in reproducing kernel Hilbert space.
Adversarial methods (9; 19) are motivated by theory in (2; 3) suggesting that a good cross-domain representation contains no discriminative information about the origin of the input.
They learn domain-invariant features by a min-max game between the domain classifier and the feature extractor.
Optimal transport-based methods (27) estimate the empirical Wasserstein distance (25) between two domains and minimizes the distance in an adversarial manner.
All three categories rely on two networks — a feature extractor network and a task classifier network.
Adversarial and OT-based methods use an additional domain classifier network.
Due to the assumption that source and target domains have the same modality 2 , the standard DA setting assumes identical feature extractors across domains.
More descriptions can be found in Appendix A.9.
Label propagation (LP) approaches (e.g., (45)) use message-passing to pass each node’s label to its neighbors according to normalized edge weights.
LP relies on only a graph’s edges, and is therefore easily applied to a heterogeneous graph – labels are simply propagated across edges, regardless of type.
In this paper we also evaluate a similarly-simple baseline, embedding propagation (EP).
Similar to LP, EP recursively propagates source embeddings (computed using source labels) until they reach the target domain.
EP exploits both node attribute information and the node adjacencies, but only uses the source node embeddings.
Preliminaries
In this section we review heterogeneous graphs and heterogeneous graph neural networks (HGNNs).
3.1 Heterogeneous graph
Heterogeneous graphs (HGs) are an important abstraction for modeling the relational data of multimodal systems.
Formally, a heterogeneous graph is defined as G =
( V , E , T , R ) where the node set
V ; the edge set E consisting of ordered tuples e ij :=
( i, j ) with i, j ∈ V , where e ij ∈ E iff an edge exists from i to j ; the set of node types T with associated map τ : V (cid:55)→ T ; the set of relation types
R with associated map ϕ : E (cid:55)→ R .
This flexible formulation allows directed, multi-type edges.
We additionally assume the existence of a node attribute vector x i ∈
X τ ( i ) for each i ∈ V , where X t is an attribute matrix specific to nodes of type t .
3.2 Heterogeneous Graph Neural Networks (HGNN)
Various HGNN models have been proposed (15; 26; 35; 41; 43).
Fully-specified HGNN models have specialized parameters for each node type (15), edge type (26), and meta-path type (8) to most effectively utilize the complex relationships encoded in the HG data structure.
In this paper, we use a flavor of HGNN known as a Heterogeneous Message-Passing Neural Network (HMPNN) as our base model on which to demonstrate KTN (though KTN can be implemented in almost any HGNN, as we show in experiments in Section 6).
The HMPNN merely extends the standard MPNN (10) by specializing all transformation and message matrices in each node/edge type.
With its generality,
HMPNN is itself a base model for RGCN (26) and HGT (15), and is also widely used as a default
HGNN model in popular GNN libraries (e.g., pyG (7), TF-GNN (6), DGL (34)).
In a HMPNN, for any node j , the embedding of node j at the l - th layer is obtained with the following generic formulation:
[Equation]
where E ( j ) =
{ ( i, j
) ∈
E : i, j ∈ V} denotes all the edges which connect (directionally) to j .
The above operations typically involve type-specific parameters to exploit the inherent multiplicity of modalities in heterogeneous graphs.
First, we define a linear Message function:
[Equation]
i
[Equation]
where M ( l ) r are the specific message passing parameters for each edge type r ∈ R and each of L
HMPNN layers.
Then defining E r ( j ) as the set of edges of type r pointing to node j , the Aggregate function mean-pools messages by edge type, and concatenates:
[Equation]
Finally, the Transform function maps the message into a type-specific latent space:
[Equation]
where W ( l ) t are the specific transformation parameters for each node type t ∈ T and each of L
HMPNN layers.
By stacking L layers, HMPNN outputs highly contextualized final node representations, and the final node representations can be fed into another model to perform downstream heterogeneous network tasks, such as node classification or link prediction.
3.3 Problem definition
Using notations defined above, we formalize our novel transfer learning problem on HGs.
Problem Definition 1.
Zero-shot cross-type transfer learning running on a HG:
In a given heterogeneous graph G =
( V , E , T , R ) with node attributes X = ∪ t ∈T X t , assume node types s and t share a classification task { ( i, y i ) : i ∈ V s , V t } .
During the training phase, using labels { ( i, y i ) : i ∈ V s } only for source-type nodes, we train an HGNN model f : f ( G , X ) =
h i to get node embeddings
h i for all nodes i ∈ V and a classifier g : g ( h i )
=
ˆ y i to predict labels ˆ y i from the node embeddings h i .
During the test phase, our task is to predict labels { ( j, y j ) : j ∈ V t } of target-type nodes where none of labels of target-type nodes were used for training.
Cross-Type Feature Extractor Transformations in HGNNs
We define f t : G (cid:55)→ R d to be the “feature extractor" of a HGNN, which is composed of parameters participating to map input node attributes of type t into a shared feature space R d .
In this section, we derive a strict transformation between feature extractors within a HMPNN.
Specifically, for any two nodes i, j with types τ ( i ) =
s and τ ( j )
=
t , we derive an expression for f s in terms of f t , and use that expression to inspire a trainable transfer learning module called KTN in the following section.
For simplicity, throughout this section we ignore the activation α ( · ) within the Transform function in Equation (4).
4.1 Feature extractors in HMPNNs
We first reason intuitively about the differences between f s and f t when s ̸ = t , using a toy heterogeneous graph shown in Figure 1(a).
Consider nodes v 1 and v 2 , noticing that τ (1) ̸
= τ (2) .
Using
HMPNN’s equations (2)-(4) from Section 3.2, for any l ∈ { 0 , . . .
, L − 1 } we have
[Equation]
where h (0) j = x j .
From these equations, we see that h ( l ) 1 and h ( l ) 2 , which are features of different types, are extracted using disjoint sets of model parameters at l -th layer.
In a 2-layer HMPNN, this creates unique gradient backpropagation paths between the two node types, as illustrated in
Figures 1(b)-1(c).
In other words, even though the same HMPNN is applied to node types s and t , the feature extractors f s and f t have different computational paths.
Therefore they project node features into different latent spaces, and have different update equations during training.
[Figure]
[Figure]
4.2 Empirical gap between f s and f t
Here we study the experimental consequences of the above observation via simulation.
We first construct a synthetic graph extending the 2-type graph in Figure 1(a) to have multiple nodes per-type, and multiple classes.
To maximize the effects of having different feature extractors, we sample source and target nodes from the same feature distributions and each classes are well-separated in the both the graph and feature space (more details available in Appendix A.7.1)
On such a well-aligned heterogeneous graph, without considering the observation in Section 4.
there may seem to be no need for domain adaptation from f t to f s .
However, when we train the
HMPNN model solely on s -type nodes, as shown in Figure 2(a) we find the test accuracy for s -type nodes to be high ( 90% , blue line) and the test accuracy for t -type nodes to be quite low ( 25% , green line).
Now if instead we make the t -type nodes use the source feature extractor f s , much more transfer learning is possible ( ∼ 65% , orange line).
This shows that the different feature extractors present in the HMPNN model result in the significant performance drop, and simply matching input data distributions can not solve the problem.
To analyze this phenomenon at the level of backpropagation, in Figures 2(b)-2(c) we show the magnitude of gradients passed to parameters of source and target node types.
As illustrated in
Figures 1(b)-1(c)
, we find that the final-layer Transform parameter W (2) t for type- t nodes have zero gradients (Figure 2(b)), and similarly for the final-layer Message parameters (Figure 2(c)).
Additionally, those same parameters in the first-layer for t -type nodes have much smaller gradients than their s -type counterparts: W (1) t (blue line in Figure 2(b)), M (1) st and M (1) tt (blue and orange lines in Figure 2(c)) appear below than other lines.
This is because they contribute to f s less than f t
This case study shows that even when an HGNN is trained on a relatively simple, balanced, and class-separated heterogeneous graph, a model trained only on the source domain node type cannot transfer to the target domain node type.
4.3 Relationship between feature extractors in HMPNNs
We show that a HMPNN model provides different feature extractors for each node type.
However, still, f s and f t are built inside one HMPNN model and interchange intermediate feature embeddings with each other.
Both H ( L ) s and H ( L ) t (
the output of f s and f t ) are computed using the previous layer’s intermediate embeddings H ( L − 1) s , H ( L − 1) t , and any other connected node type embeddings
H ( L − 1) x at the L -th HMPNN layer.
Therefore H ( L ) s and H ( L ) t can be mathematically presented by each other using the ( L − 1) -th layer embeddings as connecting points, so do f s and f t .
Based on this intuition, we derive a strict transformation between f s and f t , which will motivate the core domain adaptation component of our proposed KTN model.
Theorem 1.
Given a heterogeneous graph G = {V , E , T , R} .
For any layer l > 0 , define the set of ( l − 1) - th layer HMPNN parameters as
[Equation]
Let A be the total n × n adjacency matrix.
Then for any s, t ∈ T there exist matrices A ∗ ts = a ts
[Equation]
A ) and Q ∗ ts =
q ts ( Q ( l − 1) )
such that
[Equation]
where a ts ( · ) and q ts ( · ) are matrix functions that depend only on s, t .
The full proof of Theorem 1 can be found in Appendix A.1.
Notice that in Equation 8, Q ∗ ts acts as a macro- Transform operator that maps H ( L ) t into the source domain, then A ∗ ts aggregates the mapped embeddings into s -type nodes.
In other words, Q ∗ ts acts as a mapping matrix from the target domain to the source domain.
To examine the implications, we run the same experiment as described in
Section 4.2, while this time mapping the target features H ( L ) t into the source domain by multiplying with Q ∗ ts in Equation 8 before passing over to a task classifier.
We see via the red line in Figure 2(a) that, with this mapping, the accuracy in the target domain becomes much closer to the accuracy in the source domain ( ∼ 70% ).
Thus, we use this theoretical transformation as a foundation for our trainable
HGNN domain adaptation module, introduced in the following section.
4.4 Generalized cross-type transformations for HGNNs
In this section we showed that a HMPNN feature extractor on the (label-abundant) source node type can be expressed in terms of the (label-scarce) target node type feature extractor, and this transformation enables cross-type zero-shot learning for the target node type.
As most HGNNs have distinct feature extractors for each node types (even single-layer HGNNs, which have specialized parameters for each node/edge attribute projection layer), they will suffer from the under-trained target embeddings phenomena we showed in Section 4.2.
For instance, in the meta-path based MAGNN model (8), meta-paths directing toward the target node types are generally less engaged in the source node feature computation and thus receive smaller gradients.
While we cannot derive the exact cross-type transformation for all possible HGNNs, the core intuition in the HMPNN theory holds, namely that H ( L ) s and H ( L ) t are both computed using the previous layer’s intermediate embeddings (see Section 4.3) across all HGNN models.
This observation allows us to extend our KTN and apply it to almost any HGNN.
We illustrate this by applying KTN to 6 different HGNN models in Section 6, where we see greatly increased target-type accuracy.
KTN: Trainable Cross-Type Transfer Learning for HGNNs
Inspired by these derivations we introduce our primary contribution, Knowledge Transfer Networks .
We begin by noting Equation 8 in Theorem 1 has a similar form to a single-layer graph convolutional network (17) with a deterministic transformation matrix ( Q ∗ ts ) and a combination of adjacency matrices directing from target node type t to source node type s ( A ∗ ts ).
Instead of hand-computing the mapping function Q ∗ ts for arbitrary HGs and HGNNs (
which would be intractable), we learn the mapping function by modelling Equation 8 as a trainable graph convolutional network, named the
Require:
heterogeneous graph G =
( V , E , T , R ) , node feature matrices X , source node type s , target node type t , adjacency matrix A ts , source node label matrix Y s .
Ensure: HGNN f , classifier g , KTN t KTN 1: H ( L ) s , H ( L ) t = f
( G , H (0) =
X ) 2: H ∗ t = t KTN ( H ( L ) t ) =
A ts H ( L ) t T ts 3: L KTN =
(cid:13)(cid:13)(cid:13) H ( L ) s − H ∗ t (cid:13)(cid:13)(cid:13)4:
L = L CL ( g ( H ( L ) s ) , Y s )
+ λ L KTN 5:
Update f , g , t using ∇L
Require: pretrained HGNN f , classifier g , KTN t
KTN Ensure: target node label matrix Y t 1: H ( L ) t = f
( G , H (0) =
X ) 2: H ∗ t =
H ( L ) t T ts 3: return g ( H ∗ t )
Knowledge Transfer Network, t KTN ( · ) .
KTN replaces Q ∗ ts and A ∗ ts in Equation 8 as follows:
[Equation]
[Equation]
where A ts is an adjacency matrix from node type t to s , and T ts is a trainable transformation matrix.
By minimizing L KTN , T ts is optimized to a mapping function of the target domain into the source domain.
5.1 Algorithm
We minimize a classification loss L CL and a transfer loss L KTN jointly with regard to a HGNN model f , a classifier g , and a knowledge transfer network t KTN as follows:
[Equation]
where λ is a hyperparameter regulating the effect of L KTN ; and f ( G , X ) s and f ( G , X ) t denote H ( L ) s and H ( L ) t , respectively.
Algorithm 1 describes a training step on the source domain.
After computing the node embeddings H ( L ) s and H ( L ) t , we map H ( L ) t to the source domain using t KTN and compute
L KTN .
Then, we update the models using gradients of L CL (computed using only source labels) and
L KTN .
Algorithm 2 describes the test phase on the target domain.
After we get node embeddings H ( L ) t from the trained HGNN model, we map H ( L ) t into the source domain using the trained transformation matrix T ts .
Finally we pass the transformed target embeddings H ∗ t into the classifier which was trained on the source domain.
Indirect Connections We note that in practice, the source and target node types can be indirectly connected in HGs via other node types (i.e., there is no A ts ).
Appendix A.2 describes how we can easily extend KTN to cover domain adaption scenarios in this case.
Experiments 6.1 Datasets
Open Academic Graph (OAG).
A dataset introduced in (44) composed of five types of nodes: papers (P), authors (A), institutions (I), venues (V), fields (F) and their corresponding relationships.
Paper and author nodes have text-based attributes, while institution, venue, and field nodes have text- and graph structure-based attributes.
Paper, author, and venue nodes are labeled with research fields in two hierarchical levels, L1 and L2.
We construct three field-specific subgraphs from OAG: computer science, computer networks, and machine learning academic graphs.
[Table] [Table]
PubMed.
A network composed of of four types of nodes: genes (G), diseases (D), chemicals (C), and species (S), and their corresponding relationships.
Gene and chemical nodes have graph structure-based attributes, while disease and species nodes have text-based attributes.
Each gene and disease is labeled with a set of diseases they belong to or cause.
Synthetic heterogeneous graphs.
We generate stochastic block models (1) with multiple node/edge types.
We label each node types with the same set of classes.
Then we control feature/edge distributions within/between node types by manipulating feature/edge signal-to-noise ratio within/between classes.
A complete definition of the generative model is given in Appendix A.7.
6.2 Baselines
We compare KTN with two MMD-based DA methods (DAN (18), JAN (20)), three adversarial
DA methods (DANN (9), CDAN (19), CDAN-E (19)), one optimal transport-based method (WDGRL (27)), and two traditional graph mining methods (LP and EP (45)).
For KTN and DA methods, we use HMPNN as their base HGNN model.
More information of each method is in Appendix A.9.
6.3 Zero-shot transfer learning
We run 18 different zero-shot transfer learning tasks across three OAG and PubMed graphs.
We run each experiment 3 times and report the average value.
Due to the space limitation, we report the standard deviations and results on OAG-computer networks and OAG-machine learning in
Appendix A.3.
Each heterogeneous graph has the same node classification task for both source and target node types.
During training, we are given 1) the heterogeneous graph structure information (i.e., adjacency matrices), 2) input node attribute matrices for all node types, and 3) labels on source-type nodes for the classification task.
During the test phase, we predict labels on target-type nodes for the same classification task.
The performance is evaluated by NDCG and MRR — widely adopted ranking metrics (14; 15).
In Tables 1 and 2, our proposed method KTN consistently outperforms all baselines on all tasks and graphs by up to 73 .
3% higher in MRR (P-A(L1) task in OAG-CS, Table 1).
When we compare with the base accuracy using the model pretrained on the source domain without any domain adaptation ( 3 rd column, Base ), the results are even more impressive.
We see our method KTN provides relative gains of up to 340% higher MRR without using any labels from the target domain.
These results show the clear effectiveness of KTN on zero-shot transfer learning tasks on a heterogeneous graph.
We mention that venue and author node types are not directly connected in the OAG graphs (Figure 5(b) in Appendix), but KTN successfully transfer knowledge by passing intermediate node types.
[Table]
Baseline Performance.
Among baselines, MMD-based models (DAN and JAN) outperform adversarial-based methods (DANN, CDAN, and CDAN-E) and optimal transport-based method (WDGRL), unlike results reported in (19; 27).
These reversed results are a consequence of HGNN’s unique feature extractors for each domains.
Adversarial- and optimal transport-based methods define separate losses for source and target feature extractors (which are not separated in their shared feature extractor assumption), resulting in divergent gradients between different feature extractors and poor domain adaption performance.
This shows again the importance of considering different feature extractors in HGNNs.
More analysis can be found in Appendix A.4.
6.4 Generality of KTN
Here, we use 6 different HGNN models, R-GCN (26), HAN (35), HGT (15), MAGNN (8),
MPNN (10), and HMPNN.
MPNN maps all node types to the shared embedding space using projection matrices at the beginning then applies MPNN layers designed for homogeneous graphs.
In
Table 3, KTN improves accuracy on the target nodes across all HGNN models by up to 960% .
This shows the strong generality of KTN.
More results and analysis can be found in Appendix A.5.
6.5 Sensitivity analysis
Using our synthetic heterogeneous graph generator, we generate non-trivial 2 -type heterogeneous graphs to examine how the feature and edge distributions affect the performance of KTN and other baselines.
We generate a range of test-case scenarios by manipulating (1) signal-to-noise ratio σ e of within-class edge distributions and (2) signal-to-noise ratio σ f of within-class feature distributions across all of the (a) source-source ( s ↔ s ), (b) target-target ( t ↔ t ), and (c) source-target ( s ↔ t ) relationships.
For instance, in Figure 3, for each edge type ( s ↔ s , t ↔ t , and s ↔ t , differentiated by colors), there are two different types of edges, edges within the same class (plain line) and edges across different classes (dotted line).
For each edge type, we manipulate σ e by changing the ratio of within-class and cross-class edges, and σ f by diverging feature distributions between classes.
Thus there will besignal-to-noise ratios in total.
A higher signalto-noise ratio for a particular data dimension (edges or features) across a particular relationship r ∈ { s ↔ s, t ↔ t, s ↔ t } means that classes are more separable in that data dimension, when comparing within r , and hence easier for
HGNNs.
Note that while tuning one σ ( · ) on the range [1 . 0 , 10 . 0] , the remaining five σ ( · ) are held at 10 . 0 .
Additionally, we vary σ ( · ) across two scenarios: (I) “easy": source and target node types have same number of clusters and same feature dimensions, (II) “hard" source and target node types have different number of clusters and feature dimensions.
Note that clusters and classes are different concepts in this experiment; several clusters could have the same class label.
[Figure]
Figures 4(a) and 4(c) show results from changing σ e across the three relation types.
We see that KTN is affected only by σ e across the s ↔ t (cross-types) relationship, which accords with our theory, since KTN exploits the between-type adjacency matrix.
Surprisingly, as seen in Figures 4(b) and 4(d), we do not find a similar dependence of KTN on σ f , which shows that KTN is robust by learning purely from edge homophily in the absence of feature homophily.
Regarding the performance of other baselines, EP shows similar tendencies as KTN— only affected by cross-type σ e — because
EP also relies on cross-type propagation along edges.
However, its accuracy is bounded above due to the fact that it does not exploit the (unlabelled) target features.
DAN and DANN, which do not exploit cross-type edges, are not affected by cross-type σ e .
However, they show either low or unstable performance across different scenarios.
DAN shows especially poor performance in the “hard" scenarios (Figure 4(c) and 4(d)), failing to deal with different feature spaces for source and target domains.
Conclusion
In this work, we proposed the first cross-type zero-shot transfer learning method for heterogeneous graphs.
Our method, Knowledge Transfer Networks (KTN) for Heterogeneous Graph Neural
Networks, transfers knowledge from label-abundant node types to label-scarce node types.
We illustrate KTN handily improves HGNN performances up to 960% for zero-labeled node types across 6 different HGNN models and outperforms many challenging baselines up to 73% higher in MRR.
Future work in the area includes filtering noisy edges between source and target domains and making
KTN more robust and less dependent on structure of given noisy heterogeneous graphs.
Limitation Statement Our transfer learning method is limited to node types sharing the same task (i.e., the same classifier).
We plan to extend our work to transfer knowledge between different tasks running on different node types on a heterogeneous graph.
Societal Impact Statement
KTN allows organizations to learn better from their own graph data, leveraging its structure without requiring external information.
We believe this has a number of positive applications (preserving model quality without needing extra datasets).
However like all modeling improvements, its true impact depends on what modeling tasks the technique is applied to.
Acknowledgement
MY gratefully acknowledges support from Amazon Graduate Research Fellowship.
GPUs are partially supported by AWS Cloud Credit for Research program.