Zero-shot Transfer Learning within a Heterogeneous Graph via Knowledge Transfer Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Data continuously emitted from industrial ecosystems such as social or commerceplatforms are commonly represented as heterogeneous graphs (HG) composedof multiple node/edge types.
State-of-the-art graph learning methods for HGsknown as heterogeneous graph neural networks (HGNNs) are applied to learndeep context-informed node representations.
However, many HG datasets fromindustrial applications suffer from label imbalance between node types.
As thereis no direct way to learn using labels rooted at different node types, HGNNs havebeen applied on only a few node types with abundant labels.
We propose a zeroshot transfer learning module for HGNNs called a Knowledge Transfer Network(KTN) that transfers knowledge from label-abundant node types to zero-labelednode types through rich relational information given in the HG.
KTN is derivedfrom the theoretical relationship, which we introduce in this work, between distinctfeature extractors for each node types given in a HGNN model.
KTN improvesperformance of 6 different types of HGNN models by up to 960% for inference onzero-labeled node types and outperforms state-of-the-art transfer learning baselinesby up to 73% across 18 different transfer learning tasks on HGs.
Introduction
Large technology companies commonly maintain large relational datasets, derived from their internallogs, that can be represented as or joined into a massive heterogeneous graph (HG) composed of nodes and edges with multiple types [30].
For instance, in e-commerce networks, there are product,user and review nodes, all interconnected by many edge types that represent forms of interactionssuch as spending (user-product), reviewing (user-review) and reviews-of (product-review).
To learnpowerful features representing the complex multimodal structure of HGs, various heterogeneousgraph neural networks (HGNN) have been proposed [15, 26, 35, 43]
A common issue in these industrial applications of HGNNs is the label imbalance among differentnode types.
For instance, publicly available content nodes – such as those representing video, text,and image content – are abundantly labelled, whereas labels for other types (such as user or accountnodes) may be much more expensive to collect (or even not available, e.g. due to privacy restrictions).
This means that in most standard training settings, HGNN models can only learn to make goodinferences for a few label-abundant node types, and can usually not make any inferences for theremaining node types, given the absence of any labels for them.
If there is a pair of label-abundant and zero-labeled node types which share an inference task, couldwe transfer knowledge between them?
One body of work has focused on transferring knowledgebetween nodes of the same type from two different HGs (i.e., graph-to-graph transfer learning) [16, 40].
However, these approaches are not applicable in many real-world scenarios for three reasons
First,any external large-scale HG that could be used in a graph-to-graph transfer learning setting would almost surely be proprietary.
Second, even if practitioners could obtain access to an external industrial
HG, it is unlikely the distribution of that (source) graph would match their target graph well enoughto apply transfer learning.
Finally, node types suffering label scarcity are likely to suffer the sameissue on other HGs (e.g. user nodes).
In this paper, we introduce a zero-shot transfer learning approach for a single HG (assumed to befully-owned by the practitioners), transferring knowledge from labelled to unlabelled node types.
This setting is distinct from any graph-to-graph transfer learning scenarios, since the source and targetdomains exist in the same HG dataset, and are assumed to have different node types.
Our modelutilizes the shared context between source and target node types; for instance, in the e-commercenetwork, the latent (unknown) labels of user nodes can be strongly correlated with spending/reviewingpatterns that are encoded in the cross-edges between user nodes and product/review nodes.
We proposea novel zero-shot transfer learning problem for this HG learning setting as follows:
Informal Problem Definition 1.
Zero-shot cross-type transfer learning running on a HG:
Given a heterogeneous graph G with node types { s , t , · · · } with abundant labels for source type s butno labels for target type t , can we train HGNNs to infer the labels of target-type nodes?
A naïve solution to this problem would be to re-use an HGNN pre-trained on the source nodes fortarget node inference, given that both domains exist in the same HG.
However, as we show in ourpaper, HGNNs have distinct parameter sets for each node type [15], edge type [26], and meta-pathtype [8, 35].
These facts cause HGNNs to learn entirely different feature extractors for nodes andedges of different types – in other words, the final embeddings for source and target nodes arecomputed by different sets of parameters in HGNNs.
Thus, a classifier pre-trained on source nodeswill fail to perform well on inference tasks for target nodes.
The field of domain adaptation (DA)targets this setting, seeking to transfer knowledge from a source domain with abundant labels to atarget domain which lacks them [9, 19, 20, 27].
However, distinct feature extractors across nodetypes in HGNNs break a standard assumption of DA setting, namely that source and target domainsshare the same feature extractors (e.g., CNNs for both source and target image domains).
As wedemonstrate in this paper, in our problem setting, DA approaches fail to achieve the outstandingperformance they are known for in computer vision and NLP.
In our work, we first dissect the gradient path of HGNN models to see how feature extractors aredesigned independently for each node type, and some empirical consequences.
Then we theoreticallyanalyze how feature extractors across node types relate to each other and how their output distributionscould be represented in terms of each other.
We model this theoretical relationship between twofeature extractors as a Knowledge Transfer Network (KTN) which can be optimized to transformtarget embeddings to fit the source domain distribution.
We perform an extensive evaluation of ourmethod on 18 different transfer learning tasks on HGs where we compare against state-of-the-artdomain adaptation baselines.
Additionally, in order to understand which environments are ideal fortransferring knowledge between different node types for HGs, we formulate a synthetic heterogeneousgraph generator that allows us to study the sensitivity of these methods.
Our main contributions are:
• Novel and practical problem definition:
To the best of our knowledge, KTN is the first zero-shotcross-type transfer learning method running on a heterogeneous graph — transfer knowledgeacross different node types within a heterogeneous graph.
• Generality: KTN is a principled approach analytically induced from the architecture of HGNNs,thus applicable to any HGNN models, showing up to 960% performance improvement for zerolabeled node inference across 6 different HGNN models.
• Effectiveness: We show that KTN outperforms state-of-the-art domain adaptation methods, beingup to 73 .
3% higher in MRR on 18 different transfer learning tasks on HGs.•
Sensitivity Analysis: We provide a HG generator model to analyze how the node attribute and edge distributions of HGs affect the performance of KTN and other methods on the task.
Related Work
Various transfer learning problems have been defined on the graph domain.
[21, 22, 42, 38] constructsynthetic graphs from unstructured data and transfer knowledge over the graphs using GNNs.
Onthe other hand, [13, 14, 24, 37] focus on extracting knowledge from the existing graph structures.
They pretrain a GNN model on a source graph and re-use the model on a target graph.
While thosemethods focus on homogeneous graphs, [16, 40] transfer HGNNs across different HGs. However, none of them can be directly applied to our cross-type transfer learning problem running on a single HG. Here we cover two classes of learning approaches that are related to our problem.
As HGNNsare the models to which our method can be applied, we cover them extensively in Section 3.
Zero-shot domain adaptation (DA) Approaches in the traditional zero-shot domain adaptationspace transfer knowledge from a source domain with abundant labels to a target domain which lacksthem.
Zero-shot DA can be categorized into three groups — MMD-based methods, adversarialmethods, and optimal-transport-based methods.
MMD-based methods [18, 20, 29] minimize themaximum mean discrepancy (MMD) [11] between the mean embeddings of two distributions inreproducing kernel Hilbert space.
Adversarial methods [9, 19] are motivated by theory in [2, 3]suggesting that a good cross-domain representation contains no discriminative information about theorigin of the input.
They learn domain-invariant features by a min-max game between the domainclassifier and the feature extractor.
Optimal transport-based methods [27] estimate the empirical
Wasserstein distance [25] between two domains and minimizes the distance in an adversarial manner.
All three categories rely on two networks — a feature extractor network and a task classifier network.
Adversarial and OT-based methods use an additional domain classifier network.
Due to the assumptionthat source and target domains have the same modality 1 , the standard DA setting assumes identicalfeature extractors across domains.
Label propagation (LP) Label propagation approaches [e.g. 45] use message-passing to pass eachnodes’ labels to their neighbors according to normalized edge weights.
LP relies on only a graph’sedges, and is therefore easily applied to a heterogeneous graph – labels are simply propagated acrossedges, regardless of type.
In this paper we also evaluate a similarly-simple baseline, embeddingpropagation (EP).
Similar to LP, EP recursively propagates source embeddings (computed usingsource labels) until they reach the target domain.
EP exploits both node attribute information and thenode adjacencies, but only uses the source node embeddings.
Preliminaries
In this section we review heterogeneous graphs and heterogeneous graph neural networks (HGNNs).
3.1 Heterogeneous graph
Heterogeneous graphs (HGs) are an important abstraction for modeling the relational data of multimodal systems.
Formally, a heterogeneous graph is defined as G =
( V , E , T , R ) where the node set
V ; the edge set E consisting of ordered tuples e ij :=
( i, j ) with i, j ∈ V , where e ij ∈ E iff an edgeexists from i to j ; the set of node types T with associated map τ : V (cid:55)→ T ; the set of relation types
R with associated map ϕ : E (cid:55)→ R .
This flexible formulation allows directed, multi-type edges.
Weadditionally assume the existence of a node attribute vector x i ∈
X τ ( i ) for each i ∈ V , where X t is anattribute matrix specific to nodes of type t .
This allows G to represent nodes with different attributemodalities such as images, text, locations, or booleans.
Note that these modalities are not necessarilyexclusive (e.g. two node types s, t might share the same attribute space).
3.2 Heterogeneous Graph Neural Networks (HGNN)
Various HGNN models have been proposed [26, 35, 43, 15, 41].
Fully-specified HGNN modelshave specialized parameters for each node type [15], edge type [26], and meta-path type [8] to mosteffectively utilize the complex relationships encoded in the HG data structure.
In this paper, we use aflavor of HGNN known as a Heterogeneous Message-Passing Neural Network (HMPNN) as our basemodel on which to demonstrate KTN (though KTN can be implemented in almost any HGNN, aswe show in experiments in Section 6).
The HMPNN merely extends the standard MPNN [10] byspecializing all transformation and message matrices in each node/edge type.
With its generality,
HMPNN is itself a base model for RGCN [26] and HGT [15], and is also widely used as a default
HGNN model in popular GNN libraries (e.g., pyG [7], TF-GNN [6], DGL [34]).
In a HMPNN, for any node j , the embedding of node j at the l - th layer is obtained with the followinggeneric formulation:
[Equation]
where E ( j ) =
{ ( i, j
) ∈
E : i, j ∈ V} denotes all the edges which connect (directionally) to j .
Theabove operations typically involve type-specific parameters to exploit the inherent multiplicity ofmodalities in heterogeneous graphs.
First, we define a linear Message function:
[Equation]
where M ( l ) r are the specific message passing parameters for each r ∈ R and each of L GNNlayers.
Then defining E r ( j ) as the set of edges of type r pointing to node j , the Aggregate functionmean-pools messages by edge type, and concatenates:
[Equation]
Finally, the Transform function maps the message into a type-specific latent space:
[Equation]
By stacking L layers, HMPNN outputs highly contextualized final node representations, and the finalnode representations can be fed into another model to perform downstream heterogeneous networktasks, such as node classification or link prediction.
3.3 Problem definition
Using notations defined above, we formalize our novel transfer learning problem on HGs. Problem Definition 1. Zero-shot cross-type transfer learning running on a HG: In a given heterogeneous graph G =
( V , E , T , R ) with node attributes X = ∪ t ∈T X t , assume nodetypes s and t share a classification task { ( i, y i ) : i ∈ V s , V t } .
During the training phase, using labels{ ( i, y i ) : i ∈ V s } only for source-type nodes, we train an HGNN model f : f ( G , X ) =
h i to get nodeembeddings
h i for all nodes i ∈ V and a classifier g : g ( h i )
=
ˆ y i to predict labels ˆ y i from the nodeembeddings
h i .
During the test phase, our task is to predict labels { ( j, y j ) : j ∈ V t } of target-typenodes (where none of labels of target-type nodes were used for training).
Cross-Type Feature Extractor Transformations in HGNNs
We define f t : G (cid:55)→ R d to be the “feature extractor" of a HGNN, which is composed of parametersparticipating to map input node attributes of type t into a shared feature space R d .
In this section,we derive a strict transformation between feature extractors in HMPNNs.
Specifically, for any twonodes i, j with types τ ( i ) =
s and τ ( j )
=
t , we derive an expression for f s in terms of f t , and usethat expression to inspire a trainable transfer learning module called KTN in the following section.
For simplicity, throughout this section we ignore the activation α ( · ) within the Transform functionin Equation (4).
4.1 Feature extractors in HMPNNs
We first reason intuitively about the differences between f s and f t when s ̸ = t , using a toy heterogeneous graph shown in Figure 1(a).
Consider nodes v 1 and v 2 , noticing that τ (1) ̸
= τ (2) .
Using
HMPNN’s equations (2)-(4) from Section 3.2, for any l ∈ { 0 , . . .
, L − 1 } we have
[Equation]
where h (0) j = x j .
From these equations, we see that h ( l ) 1 and h ( l ) 2 , which are features of differenttypes, are extracted using disjoint sets of model parameters at l -th layer.
In a 2-layer HMPNN,this creates unique gradient backpropagation paths between the two node types, as illustrated in
Figures 1(b)-1(c).
In other words, even though the same HMPNN is applied to node types s and t , thefeature extractors f s and f t have different computational paths.
Therefore they project node featuresinto different latent spaces, and have different update equations during training. [Figure] [Figure]
4.2 Empirical gap between f s and f t
Here we study the experimental consequences of the above observation via simulation.
We firstconstruct a synthetic graph extending the 2-type graph in Figure 1(a) to have multiple nodes per-type,and multiple classes.
To maximize the effects of having different feature extractors, we sample sourceand target nodes from the same feature distributions and each classes are well-separated in the boththe graph and feature space (more details available in Appendix A.7.1)
On such a well-aligned heterogeneous graph, without considering the observation in Section 4.1,there may seem to be no need for domain adaptation from f t to f s .
However, when we train the
HMPNN model solely on s -type nodes, as shown in Figure 2(a) we find the test accuracy for s -typenodes to be high ( 90% , blue line) and the test accuracy for t -type nodes to be quite low ( 25% , greenline).
Now if instead we make the t -type nodes use the source feature extractor f s , much moretransfer learning is possible ( ∼ 65% , orange line).
This shows that the different feature extractorspresent in the HMPNN model result in the significant performance drop, and simply matching inputdata distributions can not solve the problem.
To analyze this phenomenon at the level of backpropagation, in Figures 2(b)-2(c) we show themagnitude of gradients passed to parameters of source and target node types.
As illustrated in
Figures 1(b)-1(c)
, we find that the final-layer Transform parameter W (2) t for type- t nodes havezero gradients (Figure 2(b)), and similarly for the final-layer Message parameters (Figure 2(c)).
Additionally, those same parameters in the first-layer for t -type nodes have much smaller gradientsthan their s -type counterparts: W (1) t (blue line in Figure 2(b)), M (1) st and M (1) tt (blue and orange linesin Figure 2(c)) appear below than other lines.
This is because they contribute to f s less than f t
This case study shows that even when an HGNN is trained on a relatively simple, balanced, andclass-separated heterogeneous graph, a model trained only on the source domain node type cannottransfer to the target domain node type.
4.3 Relationship between feature extractors in HMPNNs
We show that a HMPNN model provides different feature extractors for each node type.
However,still, f s and f t are built inside one HMPNN model and interchange intermediate feature embeddingswith each other.
Both H ( L ) s and H ( L ) t (
the output of f s and f t ) are computed using the previouslayer’s intermediate embeddings H ( L − 1) s , H ( L − 1) t , and any other connected node type embeddings
H ( L − 1) x at the L -th HMPNN layer.
Therefore H ( L ) s and H ( L ) t can be mathematically presented byeach other using the ( L − 1) -th layer embeddings as connecting points, so do f s and f t .
Based on thisintuition, we derive a strict transformation between f s and f t , which will motivate the core domainadaptation component of our proposed KTN model.
Theorem 1.
Given a heterogeneous graph G = {V , E , T , R} .
For any layer l > 0 , define the set of( l − 1) - th layer HMPNN parameters as
[Equation]
Let A be the total n × n adjacency matrix.
Then for any s, t ∈ T there exist matrices A ∗ ts = a ts ( A )and Q ∗ ts
=
q ts ( Q ( l − 1) ) such that
[Equation]
8) where a ts ( · ) and q ts ( · ) are matrix functions that depend only on s, t .
The full proof of Theorem 1 can be found in Appendix A.1.
Notice that in Equation 8, Q ∗ ts acts as amacro- Transform operator that maps H ( L ) t into the source domain, then A ∗ ts aggregates the mappedembeddings into s -type nodes.
In other words, Q ∗ ts acts as a mapping matrix from the target domainto the source domain.
To examine the implications, we run the same experiment as described in
Section 4.2, while this time mapping the target features H ( L ) t into the source domain by multiplyingwith Q ∗ ts in Equation 8 before passing over to a task classifier.
We see via the red line in Figure 2(a)that, with this mapping, the accuracy in the target domain becomes much closer to the accuracy in thesource domain ( ∼ 70% ).
Thus, we use this theoretical transformation as a foundation for our trainable
HGNN domain adaptation module, introduced in the following section.
4.4 Generalized cross-type transformations for HGNNs
In this section we showed that a HMPNN feature extractor on the source (label-abundant) nodetype can be expressed in terms of the target (label-scarce) node type feature extractor, and thistransformation enables cross-type zero-shot learning for the target node type.
As most HGNNs havedistinct feature extractors for each node types (even single-layer HGNNs, which have specializedparameters for each node/edge attribute projection layer), they will suffer from the under-trainedtarget node phenomena we showed in Section 4.2.
For instance, in the meta-path based MAGNNmodel [8], meta-paths directing toward the target node types are generally less engaged in the sourcenode feature computation and thus receive smaller gradients.
While we cannot derive the exactcross-type transformation for all possible HGNNs, the core intuition in the HMPNN theory holds,namely that H ( L ) s and H ( L ) t are both computed using the previous layer’s intermediate embeddings(see Section 4.3) across all HGNN models.
This observation allows us to extend our KTN and applyit to almost any HGNN.
We illustrate this by applying KTN to 6 different HGNN models in Section 6,where we see greatly increased target-type accuracy.
KTN: Trainable Cross-Type Transfer Learning for HGNNs
Inspired by these derivations we introduce our primary contribution, Knowledge Transfer Networks .
We begin by noting Equation 8 in Theorem 1 has a similar form to a single-layer graph convolutionalnetwork [17] with a deterministic transformation matrix ( Q ∗ ts ) and a combination of adjacencymatrices directing from target node type t to source node type s ( A ∗ ts ).
Instead of hand-computingthe mapping function Q ∗ ts for arbitrary HGs and HGNNs (which would be intractable), we learn themapping function by modelling Equation 8 as a trainable graph convolutional network, named the Require: heterogeneous graph G =
( V , E , A , R ) , node feature matrices X , source node type s , target node type t , adjacency matrix A ts , source node label matrix Y s .
Ensure: HGNN f , classifier g , KTN t KTN 1: H ( L ) s , H ( L ) t = f ( G , H (0) = X ) 2: H ∗ t = t KTN ( H ( L ) t ) = A ts H ( L ) t T ts 3: L KTN =
(cid:13)(cid:13)(cid:13) H ( L ) s − H ∗ t (cid:13)(cid:13)(cid:13)4:
L = L CL ( g ( H ( L ) s ) , Y s )
+ λ L KTN 5:
Update f , g , t using ∇L
Require: pretrained HGNN f , classifier g , KTN t
KTN Ensure: target node label matrix Y t 1: H ( L ) t = f
( G , H (0) =
X ) 2: H ∗ t =
H ( L ) t T ts 3: return g ( H ∗ t )
Knowledge Transfer Network, t KTN ( · ) .
KTN replaces Q ∗ ts and A ∗ ts in Equation 8 as follows:
[Equation]
[Equation]
where A ts is an adjacency matrix from node type t to s , and T ts is a trainable transformation matrix.
By minimizing L KTN , T ts is optimized to a mapping function of the target domain into the sourcedomain.
5.1 Algorithm
We minimize a classification loss L CL and a transfer loss L KTN jointly with regard to a HGNN modelf , a classifier g , and a knowledge transfer network t KTN as follows:
[Equation]
where λ is a hyperparameter regulating the effect of L KTN ; and f ( G , X ) s and f ( G , X ) t denote H ( L ) sand H ( L ) t , respectively.
Algorithm 1 describes a training step on the source domain.
After computingthe node embeddings H ( L ) s and H ( L ) t , we map H ( L ) t to the source domain using t KTN and compute L KTN .
Then, we update the models using gradients of L CL (computed using only source labels) and L KTN .
Algorithm 2 describes the test phase on the target domain.
After we get node embeddings H ( L ) tfrom the trained HGNN model, we map H ( L ) t into the source domain using the trained transformationmatrix T ts .
Finally we pass the transformed target embeddings H ∗ t into the classifier which wastrained on the source domain.
Indirect Connections We note that in practice, the source and target node types can be indirectlyconnected in heterogeneous graphs via other node types (i.e., there is no A ts ).
Appendix A.2 describeshow we can easily extend KTN to cover domain adaption scenarios in this case.
Experiments
6.1 Datasets
Open Academic Graph (OAG).
A dataset introduced in [44] composed of five types of nodes:papers (P), authors (A), institutions (I), venues (V), fields (F) and their corresponding relationships.
Paper and author nodes have text-based attributes, while institution, venue, and field nodes havetext- and graph structure-based attributes.
Paper, author, and venue nodes are labeled with researchfields in two hierarchical levels, L1 and L2.
We construct three field-specific subgraphs from OAG:computer science, computer machine [Table] [Table]
PubMed.
[39] A network composed of of four types of nodes: genes (G), diseases (D), chemicals(C), and species (S), and their corresponding relationships.
Gene and chemical nodes have graphstructure-based attributes, while disease and species nodes have text-based attributes.
Each gene ordisease is labeled with a set of diseases they belong to or cause.
Synthetic heterogeneous graphs.
We generate stochastic block models [1] with multiple node/edgetypes.
We label each node types with the same set of classes.
Then we control feature/edge distributions within/between node types by manipulating feature/edge signal-to-noise ratio within/betweenclasses.
A complete definition of the generative model is given in Appendix A.7.
6.2 Baselines
We compare KTN with two MMD-based DA methods (DAN [18], JAN [20]), three adversarial
DA methods (DANN [9], CDAN [19], CDAN-E [19]), one optimal transport-based method (WDGRL [27]), and two traditional graph mining methods (LP and EP [45]).
For KTN and DA methods,we use HMPNN as their base HGNN model.
More information of each method is in Appendix A.9.
6.3 Zero-shot transfer learning
We run 18 different zero-shot transfer learning tasks across three OAG and PubMed graphs.
We runeach experiment 3 times and report the average value.
Due to the space limitation, we report thestandard deviations and results on OAG-computer networks and OAG-machine learning in Appendix
A.3.
Each heterogeneous graph has node classification tasks for both source and target node types.
During training, we are given 1) the heterogeneous graph structure information (i.e., adjacencymatrices), 2) input node attribute matrices for all node types, and 3) labels on source-type nodes forthe classification task. During the test phase, we predict labels on target-type nodes for the sameclassification task.
The performance is evaluated by NDCG and MRR — widely adopted rankingmetrics [14, 15].
In Tables 1 and 2, our proposed method KTN consistently outperforms all baselines on all tasks andgraphs by up to 73 .
3% higher in MRR (P-A(L1) task in OAG-CS, Table 1).
When we compare withthe base accuracy using the model pretrained on the source domain without any domain adaptation( 3 rd column, Base ), the results are even more impressive.
We see our method KTN provides relativegains of up to 340% higher MRR without using any labels from the target domain.
These resultsshow the clear effectiveness of KTN on zero-shot transfer learning tasks on a heterogeneous graph.
We mention that venue and author node types are not directly connected in the OAG graphs (Figure4(b) in Appendix), but KTN successfully transfer knowledge by passing the intermediate nodes. [Table]
Baseline Performance.
Among baselines, MMD-based models (DAN and JAN) outperformadversarial-based methods (DANN, CDAN, and CDAN-E) and optimal transport-based method(WDGRL), unlike results reported in [19, 27].
These reversed results are a consequence of HGNN’sunique feature extractors for each domains.
Adversarial- and optimal transport-based methods defineseparate losses for source and target feature extractors (which are not separated in their shared featureextractor assumption), resulting in divergent gradients between different feature extractors and poordomain adaption performance.
This shows again the importance of considering different featureextractors in HGNNs.
More analysis can be found in Appendix A.4.
6.4 Generality of KTN
Here, we use 6 different HGNN models, R-GCN [26], HAN [35], HGT [15], MAGNN [8], MPNN [10], and HMPNN.
MPNN maps all node types to the shared embedding space usingprojection matrices at the beginning then applies MPNN layers designed for homogeneous graphs. In
Table 3, KTN improves accuracy on the target nodes across all HGNN models by up to 960% .
Thisshows the strong generality of KTN.
More results and analysis can be found in Appendix A.5.
6.5 Sensitivity analysis
[Figure] Using our synthetic heterogeneous graph generator,we generate non-trivial 2-type heterogeneous graphsto examine how the feature and edge distributionsaffect the performance of KTN and other baselines.
We generate a range of test-case scenarios by manipulating (1) signal-to-noise ratio σ e of within-classedge distributions and (2) signal-to-noise ratio σ f ofwithin-class feature distributions across all of the (a)source-source ( s ↔ s ), (b) target-target ( t ↔ t ), and(c) source-target ( s ↔ t ) relationships.
For instance,in Figure 3, for each edge type ( s ↔ s , t ↔ t , ands ↔ t , differentiated by colors), there are two different types of edges, edges within the same class (plain line) and edges across different classes (dottedline).
For each edge type, we manipulate σ e by changing the ratio of within-class and cross-classedges, and σ f by diverging feature distributions between classes.
Thus there will be 6 signal-to-noiseratios in total.
A higher signal-to-noise ratio for a particular data dimension (edges vs features) acrossa particular relationship r ∈ { s ↔ s, t ↔ t, s ↔ t } means that classes are more separable in thatdata dimension, when comparing within r , and hence easier for HGNNs.
Note that while tuning oneσ on the range [1 . 0 , 10 . 0] , the remaining five σ are held at 10 . 0 .
Additionally, we vary σ across twoscenarios: (I) “easy": source and target node types have same number of clusters and same featuredimensions, (II) “hard" source and target node types have different number of clusters and featuredimensions.
Figures 4(a) and 4(c) show results from changing σ e across the three relation types.
We see that
KTN is affected only by σ e across the s ↔ t relationship, which accords with our theory, since KTNexploits the between-type computation (adjacency) matrix.
Surprisingly, as seen in Figures 4(b) and4(d), we do not find a similar dependence of KTN on σ f , which shows that KTN is robust by learning [Figure] purely from edge homophily in the absence of feature homophily.
Regarding the performance ofother baselines, EP shows similar tendencies as KTN— only affected by cross-type σ e — because
EP also relies on cross-type propagation along edges.
However, its accuracy is bounded abovedue to the fact that it does not exploit the (unlabelled) target features.
DAN and DANN, which donot exploit cross-type edges, are not affected by cross-type σ e .
However, they show either low orunstable performance across different scenarios.
DAN shows especially poor performance in the“hard" scenarios (Figure 4(c) and 4(d)), failing to deal with different feature spaces for source andtarget domains.
Conclusion
In this work, we proposed the first cross-type zero-shot transfer learning method for heterogeneousgraphs.
Our method, Knowledge Transfer Networks (KTN) for Heterogeneous Graph Neural
Networks, transfers knowledge from label-abundant node types to label-scarce node types.
Weillustrate KTN handily improves HGNN performances up to 960% for zero-labeled node types across6 different HGNN models and outperforms many challenging baselines up to 73% higher in MRR.
Future work in the area includes filtering noisy edges between source and target domains and making
KTN more robust and less dependent on structure of given noisy heterogeneous graphs.
Limitation Statement Our transfer learning method is limited to node types sharing the same task(i.e.
, the same classifier).
We plan to extend our work to transfer knowledge between different tasksrunning on different node types on a heterogeneous graph.
Societal Impact Statement
KTN allows organizations to learn better from their own graph data,leveraging its structure without requiring external information.
We believe this has a number ofpositive applications (preserving model quality without needing extra datasets).
However like allmodeling improvements, its true impact depends on what modeling tasks the technique is applied to.