M ETA GL: E VALUATION -F REE S ELECTION OF G RAPH L EARNING M ODELS VIA M ETA -L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Given a graph learning task, such as link prediction, on a new graph, how can we select the best method as well as its hyperparameters (collectively called a model) without having to train or evaluate any model on the new graph?
Model selection for graph learning has been largely ad hoc.
A typical approach has been to apply popular methods to new datasets, but this is often suboptimal.
On the other hand, systematically comparing models on the new graph quickly becomes too costly, or even impractical.
In this work, we develop the ﬁrst meta-learning approach for evaluation-free graph learning model selection , called M ETA GL, which utilizes the prior performances of existing methods on various benchmark graph datasets to automatically select an effective model for the new graph, without any model training or evaluations.
To quantify similarities across a wide variety of graphs, we introduce specialized meta-graph features that capture the structural characteristics of a graph.
Then we design G-M network, which represents the relations among graphs and models, and develop a graph-based meta-learner operating on this G-M network, which estimates the relevance of each model to different graphs.
Extensive experiments show that using M ETA GL to select a model for the new graph greatly outperforms several existing meta-learning techniques tailed for graph learning model selection (up to 47% better), while being extremely fast at test time ( ∼ 1 sec ).
I NTRODUCTION
Given a graph learning (GL) task, such as link prediction, for a new graph dataset, how can we select the best method as well as its hyperparameters (HPs) (collectively called a model) without performing any model training or evaluations on the new graph ?
GL has received increasing attention recently (Xia et al., 2021;
Zhang et al., 2022), achieving successes across a large array of applications, such as recommendation and ranking (Fan et al., 2019; Park et al., 2020), trafﬁc forecasting (Jiang &
Luo, 2021), bioinformatics (Su et al., 2020; Li et al., 2017a), anomaly detection (Cai et al., 2021), and question answering (Park et al., 2022).
However, as GL methods continue to be developed for various tasks, it becomes increasingly difﬁcult to determine which model to use for the given graph.
Model selection ( i.e. , selecting a method and its conﬁguration such as HPs) for graph learning has been largely ad hoc to date.
A typical approach, called “no model selection”, is to simply apply popular methods to new graphs, often with the default HP values.
However, it is well known that there is no universal learning algorithm that performs best on all problem instances (Wolpert & Macready, 1997), and such consistent model selection is often suboptimal.
At the other extreme lies “naive model selection” (Fig. 1b), where all candidate models are trained on the new graph, evaluated on a hold-out validation graph, and then the best performing model for the new graph is selected.
This approach is very costly as all candidate models are trained when a new graph arrives.
Recent methods on neural architecture search (NAS) and hyperparameter optimization (HPO) of GL methods, which we review in Section 3, adopt smarter and more efﬁcient strategies, such as Bayesian optimization (Snoek et al., 2012;
Tu et al., 2019), which carefully choose a relatively small number of HP settings to evaluate.
However, they still need to evaluate multiple conﬁgurations of each GL method on the new graph.
Evaluation-free model selection is yet another paradigm, which aims to tackle the limitations of the above approaches by attempting to simultaneously achieve the speed of no model selection and the accuracy of exhaustive model selection.
Recently, a seminal work by Zhao et al.
(2021) proposed a technique for outlier detection (OD) model selection, which carries over the observed performance of
OD methods on benchmark datasets for selecting OD methods.
However, it does not address the
[Figure]
Proposed
Existing
[Figure]
unique challenges of GL model selection, and cannot be directly used to solve the problem.
Inspired by this work, we systematically tackle the model selection problem for graph learning, especially link prediction.
We choose link prediction as it is a key task for graph-structured data: It has many applications ( e.g. , recommendation, knowledge graph reasoning, and entity resolution), and several inference and learning tasks can be cast as a link prediction problem ( e.g. , Fadaee & Haeri (2019)).
In this work, we develop M ETA GL, the ﬁrst meta-learning framework for evaluation-free selection of graph learning models , which ﬁnds an effective GL model to employ for a new graph without training or evaluating any GL model on the new graph, as Figure 1a illustrates.
M ETA GL satisﬁes all of desirable features for GL model selection listed in Table 1, while no existing paradigms satisfy all.
The high-level idea of meta-learning based model selection is to estimate a candidate model’s performance on the new graph based on its observed performances on similar graphs.
Our meta-learning problem for graph data presents a unique challenge of how to model graph similarities, and what characteristic features ( i.e. , meta-features) of a graph to consider.
Note that this step is often not needed for traditional meta-learning problems on non-graph data, as features for non-graph objects ( e.g. , location, age of users) are often readily available.
Also, the high complexity and irregularity of graphs ( e.g. , different number of nodes and edges, and widely varying connectivity patterns among different graphs) makes the task even more challenging.
To handle these challenges, we design specialized meta-graph features that can characterize major structural properties of real-world graphs.
Then to estimate the performance of a candidate model on a given graph, M ETA GL learns to embed models and graphs in the shared latent space such that their embeddings reﬂect the graph-to-model afﬁnity.
Speciﬁcally, we design a multi-relational graph called G-M network , which captures multiple types of relations among models and graphs, and develop a meta-learner operating on this G-M network, based on an attentive graph neural network that is optimized to leverage meta-graph features and prior model performance into producing model and graph embeddings that can be effectively used to estimate the best performing model for the new graph.
M ETA GL greatly outperforms existing metalearners in GL model selection (Fig. 1c).
In sum, the key contributions of this work are as follows
• Problem Formulation.
We formulate the problem of selecting effective GL models in an evaluation-free manner ( i
e. , without ever having to train/evaluate any model on the new graph), To the best of our knowledge, we are the ﬁrst to study this important problem.
• Meta-Learning Framework and Features.
We propose M ETA GL, the ﬁrst meta-learning framework for evaluation-free GL model selection.
For meta-learning on various graphs, we design metagraph features that quantify graph similarities by capturing structural characteristics of a graph.
• Effectiveness.
Using M ETA GL for GL model selection greatly outperforms existing meta-learning techniques (up to 47% better, Fig. 1c), with negligible
runtime overhead at test time ( ∼ 1 sec ).
Benchmark Data/Code: To facilitate further research on this important new problem, we release code and data, including performances of 400+ models on 300+ graphs, and 300+ meta-graph features.
model selection (MS) paradigms, all of which fail to satisfy some of desirable properties for GL MS
[Table]
See Sec 3 and Fig.
1b)
[Table]
P ROBLEM F ORMULATION
Given a new unseen graph, our goal is to select the best model from a heterogeneous set of graph learning models, without requiring any model evaluations and user intervention.
In comparison to traditional meta-learning problems where a model denotes a single method and its hyperparameters, a model in the graph meta-learning problem is more broadly deﬁned to be model M = { ( graph embedding method , hyperparameters ) , ( predictor , hyperparameters ) } , (1) as graph learning tasks usually involve two steps: (1) embedding the graph using a graph representation learning method, and (2) providing node embeddings to the predictor of a downstream task like link prediction.
Both steps require learning a method with speciﬁc hyperparameters.
Thus, there can be many models with the same embedding method and predictor, which have different hyperparameters.
Also, the set M of models may contain many different graph representation learning methods ( e.g. , node2vec (Grover & Leskovec, 2016), GraphSAGE (Hamilton et al., 2017), DeepGL (Rossi et al., 2020) to name a few), as well as multiple task-speciﬁc predictors, making M heterogeneous .
Given a training meta-corpus of n graphs G = { G i } ni =1 , and m models M = { M j } mj
=1 for GL tasks, we derive performance matrix P ∈ R n × m where P ij is the performance ( e.g. , accuracy) of model j on graph i .
Our meta-learning problem for evaluation-free GL model selection is deﬁned as follows.
Problem 1 (Evaluation-Free Graph Learning Model Selection) .
Given
• an unseen test graph G test / ∈ G , and • a potentially sparse performance matrix P ∈ R n × m of m heterogeneous graph learning models
M = { M 1 , . . .
, M m } on n graphs G = { G 1 , . .
. , G n } , Select • the best model M ∗ ∈ M to employ on G test without evaluating any model in M on G test .
R ELATED W ORK
A majority of works on GL focus on developing new algorithms for certain graph tasks and applications (Xia et al., 2021; Zhang et al., 2022).
In comparison, there exist relatively few recent works that address the GL model selection problem (Zhang et al., 2021).
They mainly focus on neural architecture search (NAS) and hyperparameter optimization (HPO) for GL models, especially graph neural networks (GNNs).
Toward efﬁcient and effective NAS and HPO in GL, they investigated several approaches, such as Bayesian optimization (AutoNE by Tu et al.
, reinforcement learning (GraphNAS by Gao et al.
AGNN by Zhou et al.
Policy-GNN by Lai et al.
, hypernets (ST-GCN by Zhu et al.
(2021)), and evolutionary algorithms (Bu et al.
(2021)), as well as techniques like subgraph sampling (AutoNE by Tu et al.
, graph coarsening (JITuNE by Guo et al. (2021)), and hierarchical evaluation (HESGA by Yuan et al.
However, as summarized in Table 1, these methods cannot perform evaluation-free GL model selection (Problem 1), since they need to evaluate multiple conﬁgurations of each GL method on the new graph for model selection.
Further, they are limited to ﬁnding the best conﬁguration of a single algorithm, and thus cannot select a model from a heterogeneous model set M with various GL models, as Problem 1 requests.
An earlier work on GNN design space (You et al., 2020) is somewhat relevant, as it proposes an approach to quantify graph similarities, which can be used to ﬁnd an observed graph similar to the test graph, and select a model that performed best on it.
However, their approach evaluates a set of anchor models on all graphs, and computes similarities between two graphs based on anchor models’ performance on them.
As it needs to run anchor models on the new graph, it is inapplicable to Problem 1.
For the ﬁrst time, M ETA GL enables evaluation-free model selection from a heterogeneous set of GL models.
[Figure]
𝐆
F RAMEWORK
In this section, we present M ETA GL, our meta-learning based framework that solves Problemby leveraging prior performances of existing methods.
M ETA GL consists of two phases: (1) ofﬂine meta-training phase (Section 4.1) that trains a meta-learner using observed graphs G and model performances P , and (2) online model prediction phase (Section 4.2), which selects the best model for the new graph.
A summary of notations used in this work is provided in Table 3 in the Appendix.
4. O FFLINE M ETA -T RAINING
Meta-learning leverages prior experience from related learning tasks to do a better job on the new task.
When the new task is similar to some historical learning tasks, then the knowledge from those similar tasks can be transferred and applied to the new task.
Thus effectively capturing the similarity between an input task and observed ones is fundamentally important for successful meta-learning.
In meta-learning, the similarity between learning tasks is modeled using meta-features , i.e. , characteristic features of the learning task that can be used to quantify the task similarity.
Meta-Graph Features.
Given the graph learning model selection problem (where new graphs correspond to new learning tasks), M ETA GL captures the graph similarity by extracting meta-graph features such that they reﬂect the structural characteristics of a graph.
Notably, since graphs have irregular structure, with different number of nodes and edges, M ETA GL designs meta-graph features to be of the same size for any arbitrary graph such that they can be easily compared using meta-graph features.
We use the symbol m ∈ R d to denote the ﬁxed-size meta-graph feature vector of graph G .
We defer the details of how M ETA GL computes m to Section 4.3.
Model Performance Estimation.
To estimate how well a model would perform on a given graph,
M ETA GL represent models and graphs in the latent k -dimensional space, and captures the graphto-model afﬁnity using the dot product similarity between the two representations h G i and h M j of the i -th graph G i and j -th model M j , respectively, such that p ij ≈ (cid:104)
h G i , h M j (cid:105) where p ij is the performance of model M j on graph G i .
Then to obtain the latent representation h , we design a learnable function f ( · ) that takes in relevant information on models and graphs from the meta-graph features m and the prior knowledge ( i.e. , model performances P and observed graphs G ).
Below in this section, we focus on the inputs to the function f ( · ) , and defer the details of f ( · ) to Section 4.4.
We ﬁrst factorize performance matrix P into latent graph factors U ∈ R n × k and model factors
V ∈ R m × k , and take the model factor V j ∈ R k (the j -th row of V ) as the input representation of model M j .
Then, M ETA GL obtains the latent embedding h M j of model M j by h M j = f ( V j ) .
For graphs, more information is available since we have both meta-graph features m and meta-train graph factors U .
However, while we have the same number of models during training and inference, we observe new graphs during inference, and thus cannot obtain the graph factor U test for the test graph as for the train graphs since matrix factorization (MF) is transductive by construction ( i.e. , existing models’ performance on the test graph is needed to get latent factors for the test graph directly via MF).
To handle this issue, we learn an estimator φ : R d (cid:55)→ R k that maps the meta-graph features m into the latent factors of meta-train graphs obtained via MF above ( i.e. , for graph G i with m , φ ( m )
= (cid:98) U i
≈ U i )
and use this estimated graph factor.
We then combine both inputs ( [ m ; φ ( m )]
∈ R d + k ), and apply linear transformation to make the input representation of graph G i to be of the same size as that of model M j , obtaining the latent embedding of graph G i to be h G i =
f ( W [ m ; φ ( m )]) where W ∈ R k × ( d + k ) is a weight matrix.
Thus in M ETA GL, the performance p ij of model M j on graph G i with meta-graph features m is estimated as
[Equation]
Meta-Learning Objective.
For tasks where the goal is to estimate real values, such as accuracy, the mean squared error (MSE) is a typical choice for the loss function.
While MSE is easy to optimize and effective for regression, it does not directly take the ranking quality into account.
By contrast, in our problem setup, accurately ranking models for each graph dataset is more important than estimating the performance itself, which makes MSE a suboptimal choice.
In particular, Problem 1 focuses on ﬁnding the model with the best performance on the given graph.
Therefore, we consider rank-based learning objectives, and among them, we adapt the top-1 probability to the proposed Problem 1 as follows.
Let (cid:98) P i ∈
R m be the i -th row of (cid:98) P ( i.e. , estimated performance of all m models on graph
G i ).
Given (cid:98) P i , the top-1 probability p (cid:98) P i top1 ( j ) of j -th model M j in the model set M is deﬁned to be
[Equation]
where π ( · ) is an increasing, strictly positive function, which we deﬁne to be an exponential function.
Theorem 1 (Cao et al.
Given the performance (cid:98) P i of all models on graph G i , p (cid:98) P i top1 ( j ) represents the probability of model M j to be ranked at the top of the list ( i.e. , all models in M ).
Top-1 probabilities p (cid:98) P i top1 ( j ) for all j = 1 , . . .
, m form a probability distribution over m models.
Based on Theorem 1, we obtain two probability distributions by applying top-1 probability to the true performance P i and estimated performance (cid:98) P i of m models, and optimize M ETA GL such that the distance between the two resulting distributions gets decreased.
Using the cross entropy as the distance metric, we obtain the following loss over all n meta-train graphs G :
[Equation]
When P is sparse, meta-training can be performed via slightly modiﬁed Eqs.
(3) and (4) in App.
G.2.
O NLINE M ODEL P REDICTION
In the meta-training phase, M ETA GL learns estimators f ( · ) and φ ( · ) , as well as weight matrix W and latent model factors V .
Given a new graph G test , M ETA GL ﬁrst computes the meta-graph features m test ∈ R d as we discuss in Section 4.3.
Then m test is regressed to obtain the (approximate) latent graph factors (cid:98) U test =
φ ( m test ) ∈ R k .
Recall that the model factors V learned in the meta-training stage can be directly used for model prediction.
Then model M j ’s performance on test graph G test can be estimated by applying Equation (2) with m test and φ ( m test ) .
Finally, the model that has the highest estimated performance is selected by M ETA GL as the best model M ∗ , i.e. ,
[Equation]
[Equation]
Note that model selection using Equation (5) depends only on the meta-graph features m test of the test graph and other pretrained estimators and latent factors that M ETA GL learned in the meta-training phase.
As no model training or evaluation is involved, model prediction by M ETA GL is much faster than training and evaluating different models multiple times, as our experiments show in Section 5.4.
Further, model prediction process is fully automatic since it does not require users to choose or ﬁne-tune any values at test time.
Figure 2 shows an overview of the model prediction process, and Algorithm 1 in the Appendix lists steps for ofﬂine meta-training and online model prediction.
4. S TRUCTURAL M ETA -G RAPH F EATURES
[Equation]
( ψ ) ]
[ 𝒎
MetaGraphFeatures
[Figure]
See Section 4.3 for
Meta-graph features are a crucial component of our metalearning approach M ETA GL since they capture important structural characteristics of an arbitrary graph.
Meta-graph features enable M ETA GL to quantify graph similarities, and utilize prior experience with observed graphs for GL model selection.
It is important that a sufﬁcient and representative set of meta-graph features are used to capture the important structural properties of graphs from a wide variety of domains, including biological, technological, information, and social networks to name a few.
In this work, we are not able to leverage the commonly used simple statistical meta-features used by previous work on model selection-based meta-learning, as they cannot be used directly over irregular and complex graph data.
To address this problem, we introduce the notion of meta-graph features and develop a general framework for computing them on any arbitrary graph.
Meta-graph features in M ETA GL are derived in two steps, which is shown in Figure 3.
First, we apply a set of structural meta-feature extractors Ψ
= { ψ 1 , . . .
, ψ q } to the input graph G , obtaining Ψ( G ) =
{ ψ 1 ( G ) , . . .
, ψ q ( G ) } .
Applying ψ ∈ Ψ to G yields a vector or a distribution of values for the nodes (or edges) in the graph, such as degree distribution and PageRank scores.
That is, in Figure 3, ψ 1 can be a degree distribution, ψ 2 can be PageRank scores of all nodes, and so on.
Speciﬁcally, we use both local and global structural feature extractors.
To capture the local structural properties around a node or an edge, we compute node degree, number of wedges ( i.e. , a path of length 2), triangles centered at each node, and also frequency of triangles for every edge.
To capture global structural properties of a node, we derive eccentricity, PageRank score, and k-core number of each node.
Appendix D summarizes meta-feature extractors used in this work.
Let ψ denote the local structural extractors for nodes.
Given a graph G i
=
( V i , E i ) and ψ , we obtain an | V i | -dimensional node vector ψ ( G i ) .
Since any two graphs G i and G j are likely to have a different number of nodes and edges, the resulting structural feature matrices ψ ( G i ) and ψ ( G j ) for these graphs are also likely to be of different sizes as the rows of these matrices correspond to nodes or edges of the corresponding graph.
Thus, in general, these structural feature-based representations of the graphs cannot be used directly to derive similarity between graphs.
Now, to address this issue, we apply the set Σ of global statistical meta-graph feature extractors to every ψ i ( G ) , ∀ i = 1 , . . .
, q , which summarizes each ψ i ( G ) to a ﬁxed-size vector.
Speciﬁcally, Σ( ψ i ( G )) applies each of the statistical functions in Σ ( e.g. , mean, kurtosis, etc) to the distribution ψ
i ( G ) , which computes a real number that summarizes the given feature distribution ψ i ( G ) from different statistical point of view, producing a vector Σ( ψ i ( G ))
∈ R | Σ | .
Then we obtain the metagraph feature vector m of graph G by concatenating the resulting meta-graph feature vectors:
[Equation]
Table 5 in Appendix D lists the global statistical functions Σ used in this work to derive metagraph features.
Further, in addition to the node- and edge-level structural features, we also compute global graph statistics (scalars directly derived from the graph, e.g. , density and degree assortativity coefﬁcient), and append them to m , i.e. , the node- or edge-level structural features obtained above.
Most importantly, given any arbitrary graph G (cid:48) , the proposed approach is guaranteed to output a ﬁxed d -dimensional meta-graph feature vector characterizing G (cid:48) .
Hence, the structural similarity of any two graphs G and G (cid:48) can be quantiﬁed using a similarity function over m and m (cid:48) , respectively.
4. E MBEDDING M ODELS AND G RAPHS
Given an informative context ( i.e. , input features) of models and graphs that M ETA GL learns from model performances P and meta-graph features M (Sections 4.1 and 4.3), how can we use it to effectively learn model and graph embeddings that capture graph-to-model afﬁnity?
We note that similar entities can make each other’s context more accurate and informative.
For instance, in our problem setup, similar models tend to have similar performance distributions over graphs, and likewise similar graphs are likely to exhibit similar afﬁnity to different models.
With this consideration, we model the task as a graph representation learning problem, where we construct a graph called G-M network that connects similar graphs and models, and learn the graph and model embeddings over it.
G-M Network.
We deﬁne G-M network to be a multi-relational graph with two types of nodes ( i.e. , models and graphs) where edges connect similar model nodes and graph nodes.
To measure similarity among graphs and models, we utilize the latent graph and model factors ( U and V , respectively) obtained by factorizing P , as well as the meta-graph features M .
More precisely, we use the estimated graph factor (cid:98) U instead of U to let the same graph construction process work for new graphs.
Note that this gives us two types of features for graph nodes ( i.e. , (cid:98) U and M ), and one type of features for model nodes ( i.e. , V ).
To let different features inﬂuence the embedding step differently as needed, we connect graph nodes and model nodes using ﬁve type of edges: M-g2g, P-g2g, P-m2m, P-g2m, P-m2g where g and m denote the type of nodes that an edge connects (graph and model, respectively), and
M and P denote that the edge is based on meta-graph features and model performance, respectively.
For example, M-g2g and P-g2g edges connect two graph nodes that are similar in terms of M and (cid:98) U , respectively.
Then for each edge type, we construct a k -NN graph by connecting nodes to their top- k similar nodes, where node-to-node similarity is deﬁned as the cosine similarity between the corresponding node features.
For instance, for P-g2m edge type, graph nodes and model nodes are linked based on the similarity between (cid:98) U and V .
Fig.
7 in the Appendix illustrates the G-M network.
Learning Over G-M Network.
Given the G-M network G train with meta-train graphs and models, graph neural networks (GNNs) provide an effective framework to embed models and graphs via (weighted) neighborhood aggregation over G train .
However, since the structure of G-M network is induced by simple k -NN search, some of the neighbors may not provide the same amount of information as others, or may even provide noisy information.
We found it helpful to perform attentive neighborhood aggregation, so more informative neighbors can be given more weights.
To this end, we choose to use attentive GNNs designed for multi-relational networks, and speciﬁcally use HGT (Hu et al., 2020).
Then the embedding function f ( · ) in Section 4.1 is deﬁned to be f ( x ) =
HGT ( x , G train ) during training, which transforms the input node feature x into an embedding via attentive neighborhood aggregation over G train .
Further details of HGT are provided in Appendix G.4.
Inference Over G-M Network.
For inference at test time, we extend G train to be a larger G-M network G test that additionally contains test graph nodes, and edges between test graph nodes, and existing graphs and models in G train .
The extension is done in the same way as in the training phase, by ﬁnding top-
k similar nodes.
Then the embedding at test time can be done by f ( x ) =
HGT ( x , G test ) .
E XPERIMENTS 5. E XPERIMENTAL S ETTINGS
Models and Evaluation.
A model in our problem (Eqn. 1) consists of two components.
The ﬁrst component performs graph representation learning (GRL), and the other component leverages the learned embeddings for a downstream task of interest.
In this work, we focus on link prediction, which is a key task for graph-structured data as we discuss in Section 1.
We evaluate the performance of selecting a link prediction model for new graphs without any model evaluation .
For the ﬁrst component, we use 12 popular GRL methods, and for the second component for link scoring, we use a simple estimator that computes the cosine similarity between two node embeddings.
This results in a model set M with 423 models.
The full list of models is given in Table 4 in Appendix C.
For evaluation, we create a testbed containing benchmark graphs, meta-graph features, and a performance matrix.
We construct the performance matrix by evaluating each link prediction model in M on the graphs in the testbed, in terms of mean average precision score.
Then we evaluate M ETA GL and baselines via 5-fold cross validation where the benchmark graphs are split into meta-train G train and meta-test G test graphs for each fold, and meta-learners trained over the meta-train graph data are evaluated using the meta-test graph datasets.
Thus, model performances over the meta-test graphs
G test and the meta-graph features of G test were unseen during training, but used only for testing.
[Table]
Since model selection aims to accurately predict the best model for a new graph, we evaluate the top-1 prediction performance in terms of MRR (Mean Reciprocal Rank), AUC, and NDCG (Normalized Discounted Cumulative Gain).
To apply
MRR and AUC, we label models such that the top1 model ( i.e. , the model with the best performance for the given graph) is labeled as 1, while all others are labeled as 0.
For NDCG, we report NDCG@1, which evaluates the relevance of top-1 predicted model.
All metrics range from 0 to 1, with larger values indicating better performance.
Baselines.
Being the ﬁrst work for evaluation-free model selection in GL, we do not have immediate baselines for comparison.
Instead, we adapt baselines used for OD model selection (Zhao et al., 2021) and collaborative ﬁltering for our problem setting.
In Appendix A, we describe baselines in detail.
Baselines are grouped into two categories: (a)
Simple meta-learners select a model that performs generally well, either globally or locally: Global Best (GB)-AvgPerf , GB-AvgRank , ISAC (Kadioglu et al., 2010), and ARGOSMART (AS) (Nikoli ´ c et al., 2013); (b) Optimization-based meta- learners learn to estimate the model performance by modeling the relation between meta features and model performances: Supervised Surrogates (S2) (Xu et al., 2012), ALORS (Misir & Sebag, 2017), NCF (He et al., 2017), and MetaOD (Zhao et al., 2021).
We also include Random Selection (RS) as a baseline to see how these methods compare to random scoring.
Note that, except the simplest meta-learners RS and GB, no baselines can handle graph data, and thus they cannot estimate model performance on the new graph on their own.
In that sense, they are not direct competitors of M ETA GL.
We enable them to be used for GL model selection by providing the proposed meta-graph features.
MetaOD, which was originally designed for OD model selection, is also given the same meta-graph features to perform GL model selection.
We denote baselines either by combining M ETA GL with their names ( e.g. , M ETA GL_S2) to clearly show that they use
M ETA GL’s meta-graph features, or using their name alone ( e.g. , S2) for simplicity.
5. M ODEL S ELECTION A CCURACY
Fully Observed Performance Matrix.
In this setup, meta-learners are trained using a full performance matrix P with no missing entries.
The model selection accuracy of all meta-learners in this setup is reported in Table 2, where M ETA GL achieves the best performance in all metrics, with 18% higher MRR than the best baseline (AS).
• Among simple meta-learners, Global Best meta-learners, which simply average model performance or rank over all observed graphs, are outperformed by more sophisticated meta-learners AS and ISAC, which leverage dataset similarities for model selection using meta-graph features.
• For optimization-based meta-learners, it is important to be aware of how models and graphs relate to each other, and have high ﬂexibility to capture that complex relationship.
In methods like ALORS and MetaOD, relations between models and datasets ( i.e. , relative position of models and datasets in the embedding subspace) are learned rather indirectly via reconstructing the performance matrix.
• M ETA GL, in contrast, directly captures graph-to-model afﬁnity by modeling their relations via employing ﬂexible GNNs over the G-M network, as well as reconstructing the performance matrix.
As a result, M ETA GL consistently outperforms other optimization-based meta-learners.
Partially Observed Performance Matrix.
In this setup, meta-
learners
are trained using a sparse performance
matrix P
, obtained
by randomly
masking out a
full P
Figure
4 reports results
obtained
with
varying sparsity
, ranging
up
to 0.
In this more challenging setup
M ETA GL
consistently performs the best across
all
levels of
sparsity
, achieving
up
to 47%
higher MRR than the best
baseline.
• With increased sparsity, nearly all meta-learners perform increasingly worse, as one might expect.
• While AS was the best baseline given a full P , its accuracy decreased rapidly as sparsity increased.
Since AS selects a
model
based on the 1NN meta-train graph, it is
highly
sensitive to P ’s sparsity.
• Baselines such as the Global Best baselines are more stable as they average across multiple graphs.
• Optimization-based methods like M ETA GL and S2 perform favorably to simple meta-learners in sparse settings as they learn to reconstruct P by modeling the relation between graphs and models.
[Figure]
[Table]
[Figure]
[Table]
ALORS
[Table]
[Figure]
[Table]
[Figure]
M ETA GL and all baseline meta-learners used the same meta-graph features proposed in this work
5. E FFECTIVENESS OF M ETA -G RAPH F EATURES
In Figure 5, we evaluate how the performance of meta-learners obtained with the proposed meta-graph feature (Section 4.3) compares to that obtained with existing graph-level embedding (GLE) techniques,
GL2Vec (Chen & Koga, 2019), Graph2Vec (Narayanan et al., 2017), and GraphLoG (Xu et al., 2021).
[Figure] [Figure]
• Most points are below the diagonal in Figure 5,
i
[Figure]
e. , all meta-learners nearly consistently perform better when they use M ETA GL’s meta-graph features than when they use existing GLE methods.
This shows the effectiveness of M ETA GL’s features for the proposed task of GL model selection.
• M ETA GL performs the best, whether M ETA GL’s features or existing GLE methods are used.
5. M ODEL S ELECTION E FFICIENCY
[Figure]
To evaluate how efﬁcient M ETA GL’s model selection is, we measure its runtime ( i.e. , the time to create metagraph features for the new graph at test time, plus the time to predict the best model), and compare it with the time to train a GL model.
Figure 6 shows the distribution in box plots, where red and green lines denote the median and mean, respectively.
Results show that M ETA GL is fast, and incurs negligible runtime overhead: its runtime is just around 1 seconds or less in most cases (Figure 6a).
Notably, compared to training each GL model for only 5% of its available model conﬁgurations, M ETA GL takes considerably less time, i.e. , a median of 5% and a mean of 11% of the time required for model training (Figure 6b).
Given large-scale test graphs in practice, the speed-up enabled by M ETA GL will be greater than that reported in Figure 6b, due to the increased training time on such graphs.
Also, M ETA GL’s model selection process can be further streamlined, e.g. , by parallelizing meta-feature generation process.
We provide additional results on the runtime of M ETA GL and naive model selection in Appendix I.2.
5. A DDITIONAL R ESULTS
We present ablation study in Appendix I.3, which shows the effectiveness of M ETA GL’s proposed components, e.g. , the meta-learning objective, G-M network, and the graph encoder used by M ETA GL.
We evaluate the sensitivity of model selection approaches to the variance of performance matrix P in Appendix I.4, and compare the predicted model performance with the actual best performance in
Appendix I.5.
C ONCLUSION
As more and more GL models are developed, selecting which one to use is becoming increasingly hard.
Toward near-instantaneous, automatic GL model selection, we make the following contributions.
• Problem Formulation.
We present the ﬁrst problem formulation to select effective GL models in an evaluation-free manner ( i.e. , without ever having to train/evaluate any model on the new graph).
• Meta-Learning Framework and Features.
We propose M ETA GL, the ﬁrst meta-learning framework for evaluation-free GL model selection, and meta-graph features to quantify graph similarities.
• Effectiveness.
Using M ETA GL for model selection greatly outperforms existing meta-learning techniques (up to 47% better), while incurring negligible runtime overhead at test time ( ∼ 1 sec ).
A B ASELINES
Being the ﬁrst work for evaluation-free model selection in graph learning (GL), we do not have immediate baselines for comparison.
Instead, we adapt baselines used in MetaOD (Zhao et al., 2021) for outlier detection (OD) model selection as well as collaborative ﬁltering for our problem setting.
The baselines used in experiments can be organized into the following two categories.
(a) Simple meta-learners select a model that performs generally well, either globally or locally.
• Global Best (GB)-AvgPerf selects the model that has the largest average performance over all meta-train graphs.
• Global Best (GB)-AvgRank computes the rank of all models (in percentile) for each graph, and selects the model with the largest average ranking over all meta-train graphs.
• ISAC (Kadioglu et al., 2010)
ﬁrst clusters meta-train datasets using meta-graph features, and at test time, ﬁnds the cluster closest to the test graph, and selects the model with the largest average performance over all graphs in that cluster.
• ARGOSMART (AS) (Nikoli´c et al., 2013) ﬁnds the meta-train graph closest to the test graph ( i.e. , 1NN) in terms of meta-graph feature similarity, and selects the model with the best result on the 1NN graph.
(b) Optimization-based meta-learners learn to estimate the model performance by modeling the relation between meta-graph features and model performances.
• Supervised Surrogates (S2) (Xu et al., 2012) learns a surrogate model (a regressor) that maps meta-graph features to model performances.
• ALORS (Misir & Sebag, 2017) factorizes the performance matrix into latent factors on graphs and models, and estimates the performance to be the dot product between the two factors, where a non-linear regressor maps meta-graph features into the latent graph factors.
• NCF
(He et al., 2017) replaces the dot product used in ALORS with a more general neural architecture that estimates performance by combining the linearity of matrix factorization and non-linearity of deep neural networks.
• MetaOD
(Zhao et al., 2021) pioneered the ﬁeld of unsupervised OD model selection by designing meta-features specialized to capture the outlying characteristics of datasets, as well as improving upon ALORS with the adoption of an NDCG-based meta-training objective.
We enable MetaOD to be applicable to our problem setting, by applying our proposed meta-graph features to MetaOD.
In addition, we also include Random Selection (RS) as a baseline, to see how meta-learners perform in comparison to random scoring.
Note that among the above approaches, only GB-AvgPerf and
GB-AvgRank do not rely on meta-features for model selection.
All other meta-learners make use of the proposed meta-graph features to estimate model performances on an unseen test graph.
B N OTATIONS
Table 3 provides a list of notations frequently used in this work.
C M ODEL S ET
A model in the model set M refers to a graph representation learning (GRL) method along with its hyperparameters settings, and a predictor that makes a downstream task-speciﬁc prediction given the node embeddings from the GRL method.
In this work, we use a link predictor which scores a given link by computing the cosine similarity between the two nodes’ embeddings.
Table 4 shows the
G set of graphs { G 1 , . . . , G n } in the training set n number of graphs in training set n = | G |
G test new unseen test graph G test (cid:54)∈ G
M model set { M 1 , . . .
, M m } to search over m number of models to search over m = | M | Ψ set of structural meta-node/edge feature extractors Σ set of meta-graph feature extractors
M meta-graph feature matrix where M ∈ R n × d d number of meta-graph features m test meta-graph feature vector for the new unseen test graph G test
[Table]
embedding size
P performance matrix of m models on n graphs
U latent graph factors obtained by factorizing P ( P ≈ UV (cid:124) )
V latent model factors obtained by factorizing P ( P ≈ UV (cid:124) )
f ( · ) learnable embedding function for models and graphs
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
complete list of 12 popular GRL methods and their speciﬁc hyperparameter settings
[Table]
which compose 412 unique models in the model set M .
Note that the link predictor is omitted from Table 4 since we employ the same link predictor based on cosine similarity to all GRL methods
[Table]
D M ETA -G RAPH F EATURES
Structural Meta-Feature Extractors.
To capture the local structural properties around a node or an edge, we compute the distribution of node degrees, number of wedges ( i.e. , a path of length 2), triangles centered at each node, as well as the frequency of triangles for each edge.
To capture the global structural properties of a node, we derive the eccentricity, PageRank score, and k-core number of each node.
We also capture the global graph-level statistics ( i.e. , different from local node/edge-level structural properties), such as the density of A and AA T where A is the adjacency matrix, and also the degree assortativity coefﬁcient r .
Global Statistical Functions.
For each of the structural property distributions (degree, k-core numbers, and so on) derived by the above structural meta-feature extractors, we apply the set Σ of global statistical functions (Table 5) over it to obtain a ﬁxed-length vector representation for the node/edge/graph-level structural feature distribution.
After obtaining a set of meta-graph features
[Table]
we concatenate all of them together to create the ﬁnal meta-graph feature vector m for the graph.
[Table]
[Table]
| x | − 1 (cid:81)
i x i
[Table]
| x | / (cid:80) i 1 x i
[Table]
µ x , σ x , σ 2 x
[Table]
E ( x − µ x ) 3 / σ 3 x
[Table]
E ( x − µ x ) 4 / σ 4 x
[Table]
Q 3 − Q 1 Q 3 + Q
[Table]
med ( |
[Table] − med [Table] x ) | )
[Table]
1 | x | e T | x − µ x |
[Table]
σ x / µ x
Efﬁciency
[Table]
σ 2 x / µ 2 x
[Table]
σ 2 x / µ x
[Table]
µ 2 x / σ 2 x
Entropy
H ( x ) = − (cid:80) i x i log x
i
[Table]
H ( x ) / log 2 | x |
[Table]
−
[Table]
max gap max( Q i +1 − Q i )
[Table]
max gap max ij |
c i − c
j |
[Table]
prob.
dist.
p h
= hh T e (with ﬁxed # of bins)
E G RAPH D ATASETS
The testbed used in this work comprises 301 graphs that have widely varying structural characteristics.
Table 6 provides a list of all graph datasets in the testbed.
All graph data are from (Rossi & Ahmed, 2015); they are publicly available under the Creative Commons Attribution-ShareAlike License
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
KPGM
[Table] [Table]
[Table]
KPGM
[Table]
[Table] [Table] [Table]
KPGM
[Table]
[Table]
[Table]
KPGM
[Table]
[Table]
[Table]
KPGM
[Table]
[Table] [Table]
KPGM-
[Table] [Table]
[Table]
KPGM
[Table]
[Table] [Table]
Continued on the next page
[Table]
DD_g    road-minnesota
DD_g    road-usroads
DD_g    rt-retweet
DD_g    rt-twitter-copen
DD_g    rt_alwefaq
DD_g    rt_assad
DD_g    rt_bahrain
DD_g    rt_barackobama 
 DD_g    rt_damascus
DD_g    rt_dash
DD_g    rt_gmanews
DD_g    rt_gop
DD_g    rt_http
DD_g    rt_islam
DD_g    rt_israel 
 DD_g    rt_lebanon
DD_g    rt_libya
DD_g    rt_lolgop
DD_g    rt_obama
DD_g    rt_occupy 
 DD_g    rt_occupywallstnyc
DD_g    rt_oman
DD_g    rt_onedirection
DD_g    rt_p
DD_g    rt_saudi
DD_g    rt_tcot 
 DD_g    rt_tlot
DD_g    rt_uae
DD_g    rt_voteonedirection
DD_g    sc-nasasrb 54870
DD_g    soc-advogato
DD_g    soc-dolphins
DD_g    soc-ﬁrm-hi-tech
DD_g    soc-gplus
DD_g    soc-hamsterster
DD_g    soc-highschool-moreno
DD_g    soc-physicians
DD_g    soc-sign-bitcoinalpha
DD_g    soc-student-coop
DD_g    soc-wiki-Vote
DD_g    socfb-Amherst
DD_g    socfb-Bowdoin
DD_g    socfb-Caltech
DD_g    socfb-Hamilton  96394eco-everglades   socfb-Haverford  59589eco-ﬂorida   socfb-nips-ego 2981eco-foodweb-baydry   socfb-Oberlin  89912eco-foodweb-baywet   socfb-Reed  18812eco-mangwet   socfb-Simmons  32988eco-stmarks   socfb-Smith  97133email-dnc-corecipient   socfb-Swarthmore  61050email-dnc-leak   socfb-Trinity  111996email-enron-only   socfb-USFCA  65252email-EU   socfb-Vassar  119161email-radoslaw   socfb-Villanova  314989email-univ   socfb-Wellesley  94899enzymes_g    socfb-Williams  112986enzymes_g    tech-routers-rf
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table]
F E XPERIMENTAL D ETAILS F. E XPERIMENTAL S ETTINGS
Software.
We used PyTorch 1 for implementing the training and inference pipeline, and used the
DGL’s implementation of HGT 2 .
For MetaOD (Zhao et al., 2021), we used the implementation provided by the authors 3 .
We used the Karate Club library (Rozemberczki et al., 2020) for the implementations of the following graph-level embedding (GLE) methods, Graph2Vec (Narayanan et al., 2017),
GL2Vec (Chen & Koga, 2019), WaveletCharacteristic (Wang et al., 2021), SF (de Lara & Pineau, 2018), and LDP (Cai & Wang, 2019).
For GraphLoG (Xu et al., 2021), we used the authors’ implementation 4 .
We used open source libraries, such as NetworkX 5 and NumPy 6 , for implementing meta-graph feature extractors.
Hyperparameters.
We set the embedding size k to 32 for M ETA GL and other meta-learners that learn embeddings of models and graphs.
For M ETA GL, we created the G-M network by connecting nodes to their top-30 similar nodes.
As an the embedding function f ( · ) in M ETA GL, we used
HGT (Hu et al., 2020) with 2 layers and 4 heads per layer.
HGT is included in the Deep Graph
Library (DGL), which is licensed under the Apache License 2.0.
For training, we used the Adam optimizer with a learning rate of 0.0005 and a weight decay of 0.0001.
For GLE approaches, we used the default hyperparameter settings speciﬁed in the corresponding library and GitHub repository.
Link Prediction Model Training.
Given a graph G , we ﬁrst hold out 10% of the edges in graph G to be used for evaluation, and train GL models with the resulting subgraph for link prediction.
The training of GL models was performed by sampling 20 negative edges per positive edge, computing the link score by applying a dot product between the two corresponding node embeddings, followed by a sigmoid function, and then optimizing a binary cross entropy loss for the positive and negative edge scores.
For evaluation, we randomly sampled the same number of negative edges as the positive edges, and evaluated the predicted link scores in terms of mean average precision.
F. E VALUATION OF M ODEL S ELECTION A CCURACY (S ECTION 5.2)
In our evaluation involving a partially observed performance matrix, we extended baseline metalearners as follows so they can operate in the presence of missing entries in the performance matrix.
• Global Best-AvgPerf averaged observed performance entries alone, ignoring missing values.
If an average performance cannot be computed for some model (which is the case when a model has no observed performance entries for all graphs), we use the mean of averaged performance for other models in its place.
• Global Best-AvgRank computed the model rankings for each graph in percentile, as the number of observed model performances may be different for different graphs, and averaged the rank percentiles for observed cases only, as in Global Best-AvgPerf.
• ISAC handled the sparse performance matrix in the same way as Global Best-AvgPerf, except that only a subset of graphs, which is similar to the test graph, is considered in ISAC.
• ARGOSMART (AS) computed the mean of observed performance entries of the 1NN graph, and used this quantity in place of missing values.
• ALORS factorized the sparse performance matrix using a missing value-aware non-negative matrix factorization algorithm.
• Supervised Sur.
(S2), NCF, and MetaOD performed optimization by only considering observed performance values in the loss function, while skipping over missing entries.
Early stopping based on the validation performance was also done with respect to the observed performances alone.
F. E VALUATION OF M ETA -G RAPH F EATURES (S ECTION 5.3)
Except for WaveletCharacteristic and GraphLoG, we applied graph-level embedding (GLE) approaches to all graphs in our testbed, and meta-learners were trained and evaluated using the representations of all graphs via 5-fold cross validation.
Since WaveletCharacteristic and GraphLoG could not scale up to some of the largest graphs in the testbed ( e.g. , due to out-of-memory error), we excluded 9% and 27% largest graphs for GraphLoG and WaveletCharacteristic, respectively, and evaluated meta-learners using the resulting subset of graphs.
Note that, in these cases, M ETA GL was also trained and evaluated using the same subset of graphs.
G A DDITIONAL D ETAILS AND A NALYSIS OF M ETA GL G. M ETA GL A LGORITHM
Algorithm 1 provides detailed steps of M ETA GL, for both ofﬂine meta-training (top) and online model selection (bottom).
[-18 Create meta-test graph G test by extending G train with new edges between test graph node and existing nodes in G train (Sec. 4.4) 
-]
G. M ETA -L EARNING O BJECTIVE FOR S PARSE P ERFORMANCE M ATRIX
Given a sparse performance matrix P , meta-training of M ETA GL can be performed by modifying the top-1 probability (Equation (3)) and the loss function (Equation (4)), such that the missing entries in
P are ignored as follows:
[Equation]
where I p ij ( · ) is deﬁned as
[Equation]
the observed performance matrix
[Equation]
is missing in the observed performance matrix P
[Equation] Thus the supervision signal for each graph comes only from the model performances observed on it.
If an entire row in P is empty, the loss terms for the corresponding graph are dropped from Equation (8).
G. G-M N ETWORK
Figure 7 illustrates the G-M network (graph-model network)
(Section 4.4), which is a multi-relational bipartite network between graph nodes and model nodes.
In the G-M network, model and graph nodes are connected via ﬁve types of edges ( e.g. , P-m2m, P-g2m, M-g2g), which is shown as edges with distinct line styles and colors.
Note that while Figure 7 shows only one edge per edge type, in the G-M network, each node is connected to its top- k similar nodes.
[Figure]
# 𝑔 $ models graphs ... ...
M-g2g
P-g2g P-m2m P-g2m P-m2g G. A TTENTIVE G RAPH N EURAL N ETWORKS AND H ETEROGENEOUS G RAPH T RANSFORMER
The embedding function f ( · ) in M ETA GL (Section 4.4) produces embeddings of models and graphs via weighted neighborhood aggregation over the multi-relational G-M network.
Speciﬁcally, we deﬁne f ( · ) using Heterogeneous Graph Transformer (HGT)
(Hu et al., 2020), which is a relationaware graph neural network (GNN) that performs attentive neighborhood aggregation over the G-M network.
Let z (cid:96)t denote the node t ’s embedding produced by the (cid:96) -th
HGT layer, which becomes the
/* (Ofﬂine) Meta-Learner Training (
*/ 1 Train & evaluate models
M on get performance matrix P 2 Extract meta-graph features M for each graph G i in G (Sec. 4.3)
3 Factorize P to obtain latent graph factors U and model factors V , i.e. , P ≈ UV (cid:124) 4 Learn an estimator φ ( · ) such that φ ( m ) =
(cid:98) U i ≈ U i 5 Create meta-train graph G train (Sec 4) 6 while not converged for i = 1 , . .
. , n do
Get embeddings f ( W [ m ; φ ( m )]) of train graph G i on G train for j = 1 , . . .
, m do
Get embeddings f ( V j ) of each model M j on G train
Estimate (cid:98) p ij = (cid:104) f
( W [ m ; φ ( m )]) , f ( V j ) (cid:105) (Eqn. 2) end end
Compute meta-training loss L ( P , (cid:98) P ) (Eqn. 4) and optimize parameters 15 end
Input:
Output: selected
/* (Online) Model Selection (
*/ 16 Extract meta-graph features m test =
ψ ( G test ) 17 Estimate latent factor (cid:98) U test = φ ( m test ) for test graph G test 18 Create the test G-M network G test by extending G train with new edges between test graph node and existing nodes in G train (Sec. 4.4)
19 Get embeddings f ( W [ m test ; (cid:98) U test ]) of test graph on G test 20 Get embeddings f ( V j ) of each model M j on G test 21 Return the best model M ∗ ← arg max M j ∈ M (cid:10) f
( W [ m test ; (cid:98) U test ]) , f ( V j ) (cid:11) input of the ( (cid:96) + 1) -th layer.
Given L total layers, the ﬁnal embedding h t of node t is obtained to be the output from the last layer, i.e. , z Lt .
In general, node embeddings z (cid:96)t produced by the (cid:96) -th layer in an attention-based GNN, such as HGT, can be expressed as:
[Equation]
where s and t are source and target nodes, respectively; N ( t ) denotes all the source nodes of node t ; and E ( s, t ) denotes all edges from node s to t .
There are three basic operators: Attention , which assigns different weights to neighbors based on the estimated importance of node s with respect to target node t ; Message , which extracts the message vector from the source node s ; and Aggregate , which aggregates the neighborhood messages by the attention weight.
HGT effectively processes multi-relational graphs, such as the proposed G-M network, by designing all of the above three operators to be aware of node types and edge types, e.g. , by employing distinct set of projection weights for each type of nodes and edges, and utilizing node- and edge-type dependent attention mechanisms.
We refer the reader to (Hu et al., 2020) for the details of how HGT deﬁnes the above three operators.
In summary, M ETA GL computes the embedding function f ( x t ) by providing node t ’s input features x t as the initial embedding ( i.e. , z 0 t ) to HGT, and returning z Lt , the output from the last layer, which is computed over the G-M network via relation-aware attentive neighborhood aggregation. G. T IME C OMPLEXITY A NALYSIS We now state the time complexity of our approach for inferring the best model given a new unseen graph G (cid:48) =
( V (cid:48) , E (cid:48) ) .
Let G = ( V , E ) be the G-M network, which is comprised of model nodes and graph nodes, and induced by c -NN (nearest neighbor) search (Section 4.4 and Appendix G.3). Let k denote the embedding size, and h be the number of attention heads in HGT (Appendix G.4). The time complexity of M ETA GL is
[Equation]
where q is the number of meta-graph feature extractors, and | E (cid:48) | is the number of edges in the new unseen graph G (cid:48) .
Note that both q and ∆ are small and thus negligible.
Hence, M ETA GL is fast and efﬁcient.
Meta-Graph Feature Extraction: The ﬁrst term of the above time complexity includes the time required to estimate the frequency of all network motifs with { 2 , 3 , 4 } -nodes, which is O ( | E (cid:48) | ∆) in the worst case where ∆ is a small constant representing the maximum sampled degree which can be set by the user.
See Ahmed et al.
(2016) for further details.
The other structural meta-feature extractors such as PageRank all take at most O ( | E (cid:48) | ) time.
Furthermore, our approach is ﬂexible and supports any set of meta-graph feature extractors.
Thus, it is straightforward to see that we can achieve a slightly better time by restricting the set of such meta-graph feature extractors to those that can be computed in time that is linear in the number of edges of any arbitrary graph.
Hence, in this case, the ∆ term is dropped and we have simply O ( q | E (cid:48) | + |V| ck 2 /h ) .
Also, note that feature extractors are independent of each other, and thus can be run in parallel.
Embedding Models and Graphs: To augment the G-M network G given a new graph G (cid:48) , we ﬁnd a ﬁxed number of nearest neighbors for G (cid:48) , which takes O ( |V| k ) time.
Then we embed models and graphs by applying HGT over the G-M network G =
( V , E ) .
Assuming an HGT with h attention heads, the time to employ HGT over G is O ( |V| k 2 + |E| k 2 /h ) .
More speciﬁcally, the time taken for Attention ( · ) and Message ( · ) functions is O ( |V| k 2 + |E| k 2 /h ) , where O ( |V| k 2 ) is for feature transformation by h heads for all nodes, and O ( |E| k 2 /h ) is for message transformation/attention computation over each edge.
Similarly, Aggregate ( · ) step takes O ( |V| k 2 + |E| k ) time.
Assuming k 2 /h > k , the time for Aggregate ( · ) can be absorbed into the time for other steps.
Further, given that the G-M network is induced by c -NN search, we have that O ( |E| ) =
O ( c |V| ) , and thus the time complexity for embedding models and graphs is O ( |V| ck 2 /h ) .
H A DDITIONAL R ELATED W ORK H. M ODEL S ELECTION IN M ACHINE L EARNING
In this section, we provide a further review of model selection in machine learning, which we group into two categories.
Evaluation-Based Model Selection :
A majority of model selection methods belong to this category.
Representative techniques used by these methods include grid search (Liashchynskyi & Liashchynskyi, 2019), random search (Bergstra & Bengio, 2012), early stopping-based (Golovin et al., 2017) and bandit-based (Li et al., 2017b) approaches, and Bayesian optimization (BO) (Snoek et al., 2012; Wu et al., 2019b; Falkner et al., 2018).
Among them, BO methods are more efﬁcient than grid or random search, requiring fewer evaluations of hyperparameter conﬁgurations (HCs), as they determine which
HC to try next in a guided manner using prior experience from previous trials.
Since these methods perform model training or evaluation multiple times using different HCs, they are much less efﬁcient than the following group of methods.
Evaluation-Free Model Selection : Methods in this category do not require model evaluation for model selection.
A simple approach (Abdulrahman et al., 2018) identiﬁes the best model by considering the models’ rankings observed on prior datasets.
Instead of ﬁnding the globally best model, ISAC (Kadioglu et al., 2010)
and AS (Nikoli´c et al., 2013) select a model that performed well on similar datasets, where the dataset similarity is modeled in the meta-feature space via clustering (Kadioglu et al., 2010) or k -nearest neighbor search (Nikoli´c et al., 2013).
A different group of methods perform optimization-based model selection, where the model performance is estimated by modeling the relation between meta-features and model performances.
Supervised Surrogates (Xu et al., 2012) learns a surrogate model that maps meta-features to model performance.
Recently, MetaOD (Zhao et al., 2021) outperformed all of these methods in selecting outlier detection algorithms.
As a method in this category, the proposed M ETA GL builds upon MetaOD and extends it for an effective and automatic GL model selection.
Most importantly, M ETA GL selects a graph learning model ( e.g. , link predictor) for the given graph , while MetaOD selects an outlier detection (OD) model for the given dataset ( n -dimensional input features).
To this end, M ETA GL designs meta features to capture the characteristics of graphs, while MetaOD designs meta features specialized for OD tasks.
Also, they adopt different meta-training objectives: M ETA GL adapts the top-1 probability for meta-training, whereas MetaOD uses an NDCG-based objective. Furthermore, M ETA GL learns the embeddings of models and graphs by applying a heterogeneous GNN over the G-M network, which allows a ﬂexible modeling of the relations between different models and graphs. By contrast, in MetaOD, the embeddings of models and datasets are optimized separately, where the relations between models and datasets are modeled rather indirectly via reconstructing the performance matrix.
Note that all of these earlier methods, except the ﬁrst simple approach, rely on meta-features, and they focus on non-graph datasets.
By using the proposed meta-graph features, they could be applied to the graph learning model selection task.
H. C OMPARISON W ITH M ODEL -A GNOSTIC M ETA -L EARNING (MAML) (F INN ET AL ., 2017)
MAML employs meta-learning to train a model’s initial parameters such that the model can perform well on a new task after the parameters have been updated via a few gradient steps using the data from the new task. In other words, given a model, MAML initializes one speciﬁc model’s parameters via meta-learning over multiple observed tasks, such that the meta-trained model can quickly adapt to a new task after learning from a small number of new data ( i.e. , few-shot learning). On the other hand,
M ETA GL employs meta-learning to carry over the prior knowledge of multiple different models’ performance on different graphs for evaluation-free selection of graph learning algorithms.
Since MAML meta-trains a speciﬁc model for fast adaptation to a new dataset, it is not for selecting a model from a model set consisting of a wide variety of learning algorithms.
Further, MAML ﬁne-tunes a meta-trained model in a few-shot learning setup, whereas in our problem setup, no training and evaluation is to be done given a new graph dataset.
Due to these reasons, MAML is not applicable to the proposed evaluation-free GL model selection problem (Problem 1).
I A DDITIONAL R ESULTS I. E FFECTIVENESS OF M ETA -G RAPH F EATURES
Figure 8 shows how accurately meta-learners can perform model selection when they use the proposed meta-graph features (Section 4.3) vs. six state-of-the-art graph-level embedding (GLE) techniques,
i.e. , GL2Vec
(Chen & Koga, 2019), Graph2Vec (Narayanan et al., 2017), GraphLoG (Xu et al., 2021),
WaveletCharacteristic (Wang et al., 2021), SF (de Lara & Pineau, 2018), and LDP (Cai & Wang, 2019).
As discussed in Section 5.3, all meta-learners achieve a higher model selection accuracy nearly
[Figure]
RR
[Figure]
L
[Figure]
V ec
[Figure]
MetaGL (Ours)
ISAC
AS
S
NCF
MetaOD
ALORS consistently by using M ETA GL’s meta-graph features than when they use these GLE techniques, and M ETA GL outperform all meta-learners, regardless of which features are used.
and [Table]
line 5.
10.15 node2vec 65.
722.66 deepwalk 7.
[Table]
84.35 HONE 203.
2082.06 node2bits 64.
284.42 deepGL 145.
2349.3 GraphSage 272.97 1451.
1171.54 GCN 26.
M ETA GL (meta feat.
gen.) 0.
2.05 (model pred.)
I. M ODEL S ELECTION T IME
[Figure]
mean, respectively. Table 7 shows results comparing the runtime (in seconds) for naive model selection with the runtime of M ETA GL.
Note that naive model selection requires training and evaluating each method in the model set, while in M ETA GL, the runtime involves only the time to generate meta-graph feature (penultimate row) and predict the best model via a forward pass (last row).
Results show that M ETA GL is fast, and incurs negligible computational overhead.
Figure 9 shows the distribution of the time taken for M ETA GL to predict the best model, divided by the time taken by M ETA GL for creating meta-graph features (in percentage).
On average, it takes ∼ 20% less time for M ETA GL to make an inference than to generate meta-graph features.
[Figure]
The proposed M ETA GL achieves the best performance, in comparison to the variants that adopt different modeling choices
I. A BLATION S TUDY
Figure 10 presents ablation studies on several components of M ETA GL for meta-learning, namely, meta-learning objective, the G-M network, the GNN encoder, and the input for embedding graphs.
To that end, we compare M ETA GL against several variants that differ from M ETA GL in just one aspect, which we explain below.
In Figure 10, M ETA GL refers to the proposed meta-learner as described in Section 4.
I.3. M ETA -L EARNING O BJECTIVE
To optimize the framework such that it can ﬁnd the best model, M ETA GL employs the topprobability as its meta-learning objective (see Section 4.1).
We evaluate how M ETA GL’s performance changes when it uses different loss functions for meta-training.
M ETA GL-MSE uses mean squared error (MSE), M ETA GL-MLE employs ListMLE (Xia et al., 2008), and M ETA GL-lambda uses
LambdaLoss (Wang et al., 2018).
Figure 10a shows that M ETA GL with the top-1 probability most effectively identiﬁes the top-performing model for the new graph, while variants that employ different learning objectives obtain suboptimal results in terms of both MRR and AUC.
I.3. G-M N ETWORK
To learn the embeddings of models and graphs, M ETA GL uses a G-M network (GM), which is a multirelational network containing multiple types of nodes and edges (see Section 4.4 and Appendix G.3).
M ETA GL-SingleType GM is a variant of M ETA GL that uses a single-relational G-M network, where all of the nodes and edges in the original G-M network have been converted to be of the same type.
M ETA GL-No GM is another variant where M ETA GL does not use the G-M network at all; instead, the input features of models and graphs, which are normally provided as input to the GNN in M ETA GL, were directly taken to be the ﬁnal model and graph embeddings.
Figure 10b shows that M ETA GL with the multi-relational G-M network is the most effective, and in utilizing the G-M network, it is helpful to be able to distinguish between the multiple types of nodes and edges.
Further, in comparison to M ETA GL-No GM, learning the embeddings of graphs and models via the G-M network greatly improves the model selection performance.
I.3. G RAPH N EURAL N ETWORKS
To effectively learn the embeddings of models and graphs over a multi-relational G-M network,
M ETA GL utilizes an attentive, heterogeneous graph neural network (GNN) as its graph encoder (see Section 4.4 and appendix G.4), which can adaptively determine the aggregation weight of the neighbors (thus attentive ), while being aware of and utilizing the multiple node and edge types in learning the embeddings (hence heterogeneous ).
To see how helpful attentive, heterogeneous GNNs are in learning effective model and graph embeddings, here we create the following variants of
M ETA GL, which use GNNs that are not attentive and/or not heterogeneous ( i.e. , no mechanism to handle different node and edge types).
M ETA GL-RGCN uses RGCN (Schlichtkrull et al., 2018), which is a heterogeneous GNN, but is not attentive.
M ETA GL-GAT uses GAT (Velickovic et al., 2018), which is an attentive GNN, but is not heterogeneous.
M ETA GL-GCN uses GCN Kipf &
Welling (2017), which is neither heterogeneous nor attentive.
In Figure 10c, the highest performance is achieved by M ETA GL that uses heterogeneous and attentive GNNs, while the lowest performance is obtained when it uses GCN, which is not attentive nor heterogeneous.
Both the ability to perform attentive neighborhood aggregation, and to be able to distinguish between and utilize different node and edge types are essential in effectively capturing the complex relations between graphs and models. I.3. I NPUT FOR E MBEDDING G RAPHS
We have two sources of information for graphs: one is the performance matrix P , and the other is graph data.
Meta-graph feature m captures the information from the graph data, while φ ( · ) captures the information from the performance matrix by estimating the latent graph factor obtained by factorizing P .
Therefore, the two terms m and φ ( m ) in Equation (2) aim to incorporate two different sources of information.
Here we create the following two variants of M ETA GL:
In Equation (2),
M ETA GL-m uses m alone, and similarly M ETA GL- φ ( m ) uses only φ ( m ) .
Figure 10d shows that they provide complementary information, and using both leads to the best result.
I. S ENSITIVITY TO THE V ARIANCE OF THE P ERFORMANCE M ATRIX
[Figure]
We evaluate how sensitive M ETA GL and baselines are to the variance of the performance matrix P , which can be introduced due to the non-deterministic nature of model training and evaluation.
For evaluation, we perturb all entries of the fully-observed performance matrix P to varying degrees, and measure the model selection performance using the perturbed performance matrix (cid:101) P .
Speciﬁcally, given a performance perturbation rate r ( r ≥ 0) , we replace each performance entry n in P with the perturbed performance (cid:101) n , which is uniformly randomly sampled from [ n (1 − r / 2 ) , n (1 + r / 2 )] , bounded by 0 and 1.
For instance, given a perturbation rate of 0 . 2 , a mean average precision of 0 .
5 is perturbed by sampling from [0 . 45 , 0 . 55] .
As a result, higher perturbation rate makes the perturbed matrix more random, and thus harder to predict.
Figure 11 shows that while an increased perturbation rate indeed makes model selection harder for all methods, M ETA GL consistently outperforms baseline model selection approaches across varying performance perturbation rates, obtaining up to 53% higher MRR than the best baseline.
I. C OMPARISON WITH THE A CTUAL B EST P ERFORMANCE
[Figure]
er f o r m a n ce
D i ff ere n ce
B e t w ee n
A c t u a l B e s t a nd
P re d i c t e d B
[Figure]
s t
[Figure]
Here we compare the actual best performance observed for each graph against the performance of the model chosen by different model selection approaches.
Figure 12 is a box plot showing how those performance differences are distributed.
Figure 12 shows that using M ETA GL leads to the smallest performance difference between the actual best and the predicted best models (27% and 20% less than the second smallest difference obtained by ALORS in terms of median and mean, respectively).
Further, the performance difference distribution obtained with M ETA GL is much tighter than that of baselines.