Does GNN Pretraining Help Molecular Representation?
Abstract
Extracting informative representations of molecules using Graph neural networks (GNNs) is crucial in AI-driven drug discovery.
Recently, the graph research community has been trying to replicate the success of self-supervised pretraining in natural language processing, with several successes claimed.
However, we ﬁnd the beneﬁt brought by self-supervised pretraining on small molecular data can be negligible in many cases.
We conduct thorough ablation studies on the key components of GNN pretraining, including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures, to see how they affect the accuracy of the downstream tasks.
Our ﬁrst important ﬁnding is, self-supervised graph pretraining do not always have statistically signiﬁcant advantages over non-pretraining methods in many settings.
Secondly, although noticeable improvement can be observed with additional supervised pretraining, the improvement may diminish with richer features or more balanced data splits.
Thirdly, hyper-parameters could have larger impacts on accuracy of downstream tasks than the choice of pretraining tasks, especially when the scales of downstream tasks are small.
Finally, we provide our conjectures where the complexity of some pretraining methods on small molecules might be insufﬁcient, followed by empirical evidences on different pretraining datasets.
Introduction
Graph neural networks (GNNs) , due to their effectiveness, have been adopted to model a wide range of structured data, such as social networks, road graphs, citation networks, etc.
Molecule modeling is one of these important applications, where it serves as the foundation of biomedicine and nurturing techniques like novel drug discovery.
However, labeling biomedical data are usually time-consuming and expensive and thus task-speciﬁc labels are extremely inadequate. This poses a big challenge to the ﬁeld.
Recently, inspired by the remarkable success of self-supervised pretraining from natural language processing [6, 2, 28] and computer vision domains [11, 5], researchers start trying to apply the pretrain-ﬁnetune paradigm to molecule modeling with GNN, hoping to boost the performance of various molecular tasks by pretraining the model on the enormous unlabeled data.
For instance, many methods have been proposed [32, 29], where signiﬁcant performance improvements are claimed by pretraining on large scale datasets [12, 22, 35–37].
Despite of the promising results, we ﬁnd that reproducing some of these outstanding gains via graph pretraining can be non-trivial, and sometimes the improvement largely relies on the experimental setup and the extensive hyper-parameter tuning of downstream tasks, rather than the design of pretraining objectives.
These observations motivate us to rethink the effectiveness of graph pretraining with unsupervised or self-supervised objectives, and investigate what factors would inﬂuence the effectiveness of self-supervised graph pretraining.
In this paper, we perform systematic studies to assess the performance of popular graph pretraining objectives on different types of datasets, and exploit various confounding components in experimental 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Configs
Learning Rate
Batch Size
Other params 1e-3 1e-4 ...
GNN Backbone 1-WL based GNN
GIN
GraphSage ......
Dataset
Data
Size
Zinc-subset 2M
SAVI 1B
Zinc-full 10M ......
Objective masked node predict
Sub-struct predict
Toxic?
Supervised
Contrastive
Graph Features
Molecule Graph
Basic ATOM_NUM
BOND_TYPE......
Rich
AROMATIC (all basics)
RINGBOND......
Data Split
Pretraining
Fine-tune Initialize
Scaffold TRAIN
TEST
Balanced TRAIN
TEST setup in deciding the performance of downstream tasks with or without pretraining.
Here, we restrict our studies to small molecular graphs, as opposed to other application domains, such as social networks or citation graphs.
The key insights and take-aways of this paper are:
• Among the pretraining tasks we evaluated, the self-supervised pretraining alone does not provide statistically signiﬁcant improvements over non-pretrained methods on downstream tasks.
• When additional supervised pretraining step is conducted after self-supervised pretraining, we observe statistically signiﬁcant improvements.
However, the gain becomes marginal on some speciﬁc data splits or diminishes if richer features are introduced.
• Beyond data splits and hand-crafted features, the usefulness of graph pretraining is also sensitive to the experimental hyperparameters, such as learning rates and number of study repeats.
Different setups can lead to opposite conclusions.
• In conclusion, different from the previous works, we do not observe clear and unconditional gains achieved by graph pretraining on molecular representation, indicating it is still too early to conclude graph pretraining is effective in molecular domain.
• We investigate the reason of above and hypothesize that the complexity of some pretraining methods on molecules is insufﬁcient, leading to less transferable knowledge for downstream tasks.
Despite the overall negative results we obtained, the main goal of this paper is not
to discourage the pretraining research for small molecules.
Instead, we hope to raise the attention on different aspects of experiments and the role of simple hand-crafted features, so as to provide useful information for designing better pretraining approaches.
Below we ﬁrst introduce the background of GNN and its pretraining in Section 2, and then our experimental design and results in Section 3 and Section 4, respectively.
Finally we conclude with our ﬁndings and the limitations in Section 5 and Section 6.
Preliminary
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Graph Neural Networks (GNNs).
Let G =
{ V, E } denote a molecule graph with V as the set of nodes and E as the set of edges.
Given the node features X i , most GNNs learn an embedding representation h
i for every node i ∈ V by aggregating representations from connected nodes and edges, denoted as graph convolution.
These procedure repeats for K times with the update equation as follows:
[Equation]
where N ( i ) is the set of neighbor nodes of i and h
i
=
X i .
The representation for entire graph G is then obtained by permutation-invariant transformation on node representation, h G =
READOUT ( h Ki | i ∈ V ) .
In this paper we mainly study the GNNs that belong to this family, namely the WL-1 GNNs.
Finetune.
After the pretraining, the pretrained model is used to ﬁnetune on the downstream tasks.
For molecule property prediction tasks, the graph-level representation obtained from the pretrained model is connected to linear classiﬁers to predict downstream task labels.
The ﬁne-tuning is performed in an end-to-end manner, where both the pretrained GNN and the linear classiﬁers are trainable.
Graph pretraining objectives.
The primary goal of pretraining is to learn representations with robust transferable knowledge of graphs from the abundant pretraining data and then generalize to downstream tasks with usually different supervision signals.
Generally the pretraining objectives can be categorized into self-supervised and supervised ones.
We present a brief overview of some representative objectives in the following sections.
2.1 Self-supervised (unsupervised) pretraining
In self-supervised pretraining, the pretraining objective is designed to learn self-generated targets from the structure of the molecules, such as the type of nodes and edges, prediction of local context, graph partition, node clustering, occurrence of some functional groups, and etc.
The predictive target can be node/edge level or entire graph level.
We present some representative ones below:
2.1.1 Node Prediction
Node prediction is a node-level classiﬁcation task given the masked context of entire graph.
Similar to Devlin et al.
[6], some portion of node attributes are masked and replaced with mask-speciﬁed indicators in the node input feature.
After graph convolution, the embedding output from GNN is used to predict the true attribute of the node, e.g. atom type in molecular graphs, through a linear classiﬁer on top of the node embedding.
2.1.2 Context Prediction
Context prediction task is a sub-graph level task aiming at learning embedding that can represent the local subgraph surrounding a node.
Generally it can be viewed as a masked task for substructure.
Since it is essentially a structured prediction which can be difﬁcult in general, Hu et al.
[12] leverages the adversarial learning to teach the model to distinguish the positive sub-graph embedding from the negative ones.
Rong et al.
[22] instead builds a dictionary of structures that captures the property of sub-graphs (e.g. type and quantity of neighbour nodes and bonds), and turns it into a multi-class classiﬁcation problem.
2.1.3 Motif Prediction
Motif prediction [22] is to predict the existence of functional groups, such as benzene ring or hydroxyl.
The motifs are extracted automatically from RDKit [15] .
The motif prediction task is formulated as a graph-level multi-label binary classiﬁcation task, where the graph embedding is used to jointly predict the occurrence of these semantic functional motifs.
2.1.4 Contrastive learning
Graph contrastive learning is to maximize the agreement of two augmented views of the same graph, and minimize the agreement of different graphs.
The optimization is conducted using contrastive loss in the latent embedding space [35, 10, 37, 29].
The augmentation function needs to transform graphs into realistic and novel augmentations without affecting semantic labels of the graphs.
For example, the transformation can be small perturbations or modiﬁcations on node/edge embedding, drop of a [Table] few nodes or edges, and so on.
These transformations enforce an underlying prior for contrastive learning, that is, local transformation does not change the semantic meaning of a graph [Table]
2.2 Supervised pretraining
Supervised pretraining aims to learn domain-speciﬁc graph-level knowledge from speciﬁcally designed pretraining tasks.
For molecular application, the supervised labels are generated from a diverse set of functional studies like biochemical assays.
The pretrainning task is to perform multiple binary classiﬁcation and jointly learn the supervised labels.
Although the pretraining mainly refers to unsupervised or self-supervised methods as they are not limited by the requirement of supervised labels, supervised pretraining is still a great source to investigate the graph pretraining in general.
[Table]
Experiment framework
To investigate pretraining on graphs for molecule representations [Table] we ﬁrst revisit the typical pretraining-ﬁnetuning pipeline used in the literature.
Figure 1 shows the overall procedure of deployment [Table] with several design choices presented at each stage of the pipeline.
Since different choices at each stage can lead to different performances on the downstream tasks, we investigate them one at a time while keeping others the unchanged.
The design principle of our experiment framework is to analyze the effect of every stage in the pipeline as comprehensive as possible, while also keeping it tractable to avoid exponentially many experiments.
3.1 Design choices
We consider the design choices for the four pretraining objectives.
Pretraining objective In Section 2 we have provided a brief literature review over the pretraining methods for molecule representation.
Here we categorize those pretraining by different principles, and present one well-recognized representative of each category.
The representatives are selected because they have more desired properties, such as better performance, compared with their counterparts.
• Masking.
We leverage the node prediction objective, which randomly masks 15% of the nodes’ feature and then ask GNN to make prediction on the node attributes of the masked ones.
This strategy resembles the BERT pretraining [6] in natural language processing.
• Structured.
Unlike text data where the topology is a sequence, the graph has rich structure information.
Following Hu et al.
[12], we use context prediction objective, which masks out the context from k 1 -hops to k 2 -hops and leverages adversarial training to predict the true context embeddings from the random context embeddings.
• Graph-level self-supervised.
Following [22], GNN is asked to predict whether a motif is contained in a molecule.
The motif can be extracted from the molecule with RDKit [15].
The motifs are 85 motifs 1 for multi-label classiﬁcation. [Table] [Table]
• Contrastive.
We generate two views of the same graph by corrupting the input node features with Gaussian noise. We leverage the contrastive learning loss proposed in [35]: we maximize the consistency between positive pairs (from same graphs) and minimize that between negative pairs (from different graphs). In this paper, we restrict ourselves to this speciﬁc contrastive training method, however, various contrastive learning methods can be further explored.
• Graph-level supervised.
Finally when applicable, we use the ChEMBL dataset with graph-level labels for graph-level supervised pretraining as Hu et al.
The above are the design choices for pretraining objectives.
Next, we consider other factors that inﬂuence graph-pretraining performance.
Graph Features Each molecule is represented by a graph with atoms as nodes and bonds as edges.
In this paper we mainly consider the graph representations without the 3D information.
For each molecule graph, chemical properties of nodes and edges are extracted to serve as node and edge features for the graph neural networks.
Depending on how rich the features are, we categorize the design choices into two categories:
• Basic features.
The basic set of features are the ones used in Hu et al.
Speciﬁcally, the node features contain the atom type and the derived features, such as formal charge list, chirality list, etc.
The edge features contain the bond types and the bond directions.
These features are categorical, and thus will be encoded in a one-hot vector individually and then concatenated together to form the feature vector for node/edge representation.
• Rich features.
The rich feature set is a superset of the basic features.
In addition to the basic ones mentioned above, it comes with the additional node features such as hydrogen acceptor match, acidic match and bond features such as ring information.
This set of features are used in Rong et al.
Additionally and importantly, we follow their setting to incorporate additional 2d normalized rdNormalizedDescriptors features 2 , which is used in the downstream tasks only and not in pretraining.
Please refer to the original papers for the full set of basic [12] and rich [22] features, respectively.
GNN Backbone The GNN architecture also plays a role in graph pretraining.
In Hu et al.
[12], the results show that pretraining on GNN variants like GIN [33] would improve the performance on downstream tasks, while the performance with architectures like GAT [27] would actually get worse performance with pretraining.
As the GNNs based on 1-Weisfeiler-Lehman (WL) test have similar representation power [33] bounded by the Weisfeiler-Lehman isomorphism check [23], we consider the two representative GNN architectures, namely the GIN [33] and GraphSage [9].
They have shown beneﬁts with graph pretraining in Hu et al.
Pretraining dataset In natural language pretraining, researchers observed a signiﬁcant performance boost due to self-supervised pretraining on large-scale data, that is, the larger the pretraining dataset [Table] [Table] is, the better the downstream performance it is [20].
Inspired by this success in natural language processing, we test the algorithms on two unlabeled pretraining datasets with different scales.
• ZINC15 [25] : ZINC15 contains 2 million molecules.
This dataset was preprocessed following Hu et al.
• SAVI [19] : The SAVI dataset contains about 1 billion molecules, which are signiﬁcantly larger than ZINC15.
To the best of our knowledge, it has never been used for pretraining tasks before.
This dataset contains drug-like molecules synthesized by computer simulated reactions.
Additionaly, we used ChEMBL [8] as the supervised datasets.
Different from the above ZINC15 and SAVI dataset which are only used for self-supervised pretraining, this dataset contains 500k drug-able molecules with 1,310 prediction target labels from bio-activity assays for drug discovery.
Thus like in Hu et al.
[12] we only leverage it for supervised pretraining.
Data split on downstream tasks The downstream tasks for molecular domain we used arebenchmark datasets from MoleculeNet [30] (See Appendix A.5 for more details).
The train/valid/test sets are split with ratio 8:1:1.
For molecule domain, the random split is not the most meaningful way to assess the performance, because the real-world scenarios often require generalization ability on out-of-distribution samples.
So we consider the following ways to split the data:
• Scaffold Split [12, 21]
This strategy ﬁrst sorts the molecules according to the scaffold (e.g. molecule structure), and then partition the sorted list into train/valid/test splits consecutively.
Therefore, the molecules in train and test sets are most different ones according to their molecule structure.
Note this strategy would yield deterministic data splits.
• Balanced Scaffold Split [1, 22]
This strategy introduces the randomness in the sorting and splitting stages above, thus one can run on splits with different random seeds and report the average performance to lower the evaluation variance.
We choose balanced scaffold as our major evaluation conﬁguration, because it allows us to evaluate the algorithm on multiple data splits while maintaining the ability to evaluate out of distribution samples (e.g. assess generalization ability).
Evaluating on one single split (such as scaffold split) can be subject to bias due to one speciﬁc split, leading to higher variance in evaluation.
3.2 Experiment protocol
As the total number of conﬁgurations for the entire pipeline can be combinatorially large which is not practical for us to exhaustively experiment with all of them, we design our protocol with a pairwise comparison principle.
Speciﬁcally, we ﬁrst anchor a vanilla conﬁguration with a certain design choice of combination for each stage.
To study the effect of each stage on the pretraining effectiveness, we vary the design choice one stage at a time compared to the vanilla conﬁguration .
For all these experiments, to assess the effectiveness of graph pretraining, we report the ROC-AUC on downstream tasks as well as the relative average gain over all downstream datasets with and without pretraining.
For each setting we will report the mean and standard deviation (in parenthesis) over three runs with different random seeds.
We tune the model on downstream tasks with the validation set, and report the evaluation metric on the test set using the model with best validation [Table] [Table] performance.
For each setup, we report the average performance obtained with three random seeds.
We tune the learning rate in { 1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 , 1 e − 2 , 5 e − 2 , 1 e − 1 } for each setup individually and select the one with best validation performance.
For GNNs we ﬁx the hidden dimension toand number of layers to 5.
Results
In this section, we present the results and discussions for a set of experiments designed with the protocols in Section 3.2.
Table 1 summarizes the experimental conﬁgurations for each following table.
We will elaborate on them in the following sections.
Due to space limit, we defer our investigation on different GNN architectures to appendix (Section A.1).
4.1 Vanilla conﬁguration
We choose the vanilla conﬁguration with the settings from existing works [12, 22].
Speciﬁcally, we use the rich feature with GIN backbone, pretrained on ZINC15 when pretraining is applied, and evaluate on the Balanced Scaffold Split for downstream tasks.
One important baseline is without pretraining.
For the ease of comparison, we include the results without pretraining in each table.
4.2 Self-supervised pretraining objectives
We compare the results pretrained with different self-supervised pretraining objectives.
As is presented in Section 3.1, we consider four representative types of pretraining objectives.
For the ease of comparing the performance, we only consider one objective at a time, instead of mixing different pretraining objectives to obtain a multi-task pretrained model.
Table 2 shows the performance on downstream molecule property prediction benchmarks with models initialized from different pretraining objectives.
The relative average gain compared to the one without pretraining is not statistically signiﬁcant, i.e., not larger than the standard deviations of multiple runs.
All the four different objectives obtain similar gains/loses regardless of very different designs.
To fully understand the effect of self-supervised pretraining on molecule representation, we further investigate the performance of different pretraining objectives in combination with other factors, such as input features or data splits, as described in the following sections.
4.3 Supervised pretraining objectives
In addition to the self-supervised objectives, we study the potential beneﬁts with supervised pretraining.
Unlike the self-supervised setting where the molecule graphs themselves are used for pretraining, the supervised pretraining requires extra cost of data labeling, and thus is not scalable for large scale pretraining.
In this paper, we present the results with supervised pretraining alone, as well as the joint pretraining.
e.g. pretrain with self-supervised objective and followed by supervised pretraining, in
Table 3.
We can see with the supervised pretraining, one can improve the downstream performance, which aligns with the observation from Hu et al.
Our hypothesis is that, supervised pretraining is helpful when the pretraining tasks are closely aligned with the downstream tasks.
In particular, [Table] the bio-activity labels provided by ChEMBL is highly related to the drug discovery purpose and drug discovery properties are the major topics evaluated in the downstream tasks.
Therefore, the positive correlation between the pretraining supervision and downstream tasks contribute the most to the performance improvement of downstream tasks.
4.4 Data split on downstream tasks
Molecular data is usually diverse and limited, so chemists are particularly interested in the generalization ability of GNNs on out of distribution data.
Also due to the same reason (i.e. limited and diverse data), the variance in performance of different splits is signiﬁcant, which poses challenges on robust evaluation.
In vanilla conﬁguration we use the balanced scaffold split, and here we show additional results with the scaffold split, which is a deterministic data split that makes the train/valid/test set differ from each other the most.
Table 4 and Table 5 respectively present the results using scaffold split with self-supervised without and with additional supervised pretraining.
Compared with Tableand Table 3, it is clear to see that Table 4 and Table 5 have signiﬁcantly lower ROC-AUC.
Speciﬁcally the AUC drops 6.2% on average for all benchmarks without pretraining.
On the other hand, we can see if we compare Table 4 with Table 2, or Table 5 with Table 3 respectively, the gain of pretraining is more signiﬁcant on the scaffold split.
We speculate the reason for the improvement of scaffold split is that the initialization of neural network parameters (e.g. from pretraining) are typically critical for the out-of-distribution generalization (e.g. scaffold split).
Similar observations have also been studied in the meta-learning literature [7].
Although the gain with supervised pretraining is signiﬁcant in
Table 5, the effect of self-supervised pretraining is mixed in Table 4.
This indicates the effectiveness of self-supervised pretraining on scaffold split is not signiﬁcant enough to claim “very helpful”.
4.5 Graph features
So far we have presented the results with rich features.
Now we want to see how those basic features used in Hu et al.
[12] affect the outcome.
Table 6 and Table 7 show the test ROC-AUC (%) performance with basic features on the balanced scaffold splits using self-supervised or supervised pretraining objectives, respectively.
Table 8 and Table 9 show the same results but on scaffold split.
In a nutshell, without pretraining, rich features lead to an average gain of 1 . 3% and 3 .
1% over basic features using balanced scaffold split and scaffold split, respectively.
Speciﬁcally, it achieves 76 .
7% vs 78 .
0% for balanced scaffold split, and 68 .
7% vs 71 . 8% on scaffold split.
The gain brought by the rich features are more signiﬁcant than the ones with different self-supervised pretraining objectives.
Table 6 to Table 9 show that pretraining has more positive impact when basic features are used.
In particular, the self-pretraining with context prediction shows signiﬁcant gains especially in the scaffold split setting.
However, the gain diminishes when careful feature engineering are applied to the downstream tasks (use rich feature in vanilla conﬁguration ).
The supervised pretraining continues the signiﬁcant gain under these settings, which shows the consistency and reliability of the situation with the labeled and downstream-task-aligned supervisions.
4.6 Pretraining datasets
As observed in natural language processing domain, more text pretraining data lead to better downstream performance.
Intuitively this can be true for molecule representation domain as well, so we run a new set of experiments with the model pretrained on SAVI dataset, which is about 500 times larger than the ZINC15 dataset we used in the above result sections.
We present the results pretrained on SAVI dataset using balanced scaffold split or scaffold split in Table 10 and Table 11, respectively.
Other conﬁgurations are the same as the vanilla conﬁguration .
[Table] Compared with the performance on ZINC15, the SAVI pretraining data does not lead to a signiﬁcant improvement either on balanced scaffold split (Table 2 vs Table 10) or scaffold split (Table 4 or
Table 11).
Similarly, the self-supervised pretraining objectives lead to negligible gain on downstream task performance, while the supervised one still achieves a clear gain.
As the result is counterintuitive, we further investigate the reason behind it by inspecting the pretraining performances with different training objectives on both ZINC15 and SAVI datasets.
We plot the curve of accuracy growth with the number of training steps iterated.
We can see from Figure 2 that in all settings the pretraining accuracy grows above 90% quickly after only 0.1 to 0.2M steps and also converges quickly.
Given that the model gets very high accuracy without even going throughepoch of the SAVI dataset, it is expected that the larger training data like SAVI may not provide more learning signals for the model, and partially explains why more molecules wouldn’t help signiﬁcantly in this case.
Furthermore, these ﬁgures might suggest several reasons of why the self-supervised pretraining may not be very effective in some situations:
• Tasks are easy.
Some of the self-pretraining tasks for molecules might be easy, so that model learns less useful information from pretraining.
For example in the masked node prediction case, the model is expected to predict the atomic number from a vocabulary with less than 100 candidate atoms.
Furthermore, due to the valence constraints, the graph topology may already exclude most of the wrong atoms.
As a comparison, the vocabulary size for text pretraining may be 100k or even higher.
Some structured prediction tasks like context prediction might be hard, but due to the difﬁculty of structured prediction itself and the proposal for high quality negative examples for contrastive learning, it can still be challenging for downstream task improvements.
Other strategies like motif prediction can be achieved by subgraph matching, which can be easy for GNN that intrinsically does the graph isomorphism test.
• Data lacks diversity.
Due to biophysical and functional requirements, molecules share many common sub-structures, e.g., functional motifs.
Hence, molecules may not be as diversiﬁed as text data.
This is why the model learns to generalize quickly within the training distribution.
• 2D Structure is not enough to infer functionality.
Some important biophysical properties (such as 3D structure, chirality) are barely reﬂected in the 2D-feature-based pretraining (e.g., using smiles or 2D graph features).
For example, the molecules with the same chemical formula and 2D feature, can have very different chirality, which leads to quite different toxicity [24] (e.g. ﬂipped toxicity labels).
This is not captured in the current GNN pretraining frameworks that we considered.
4.7 Hyper-parameters for downstream tasks
[Figure]
We also ﬁnd that hyper-parameters for downstream tasks are critical for the their performance that their choices may change the conclusion of the effectiveness of pretraining in some settings.
We can take the learning rate as an example.
As the models initialized from scratch and pretraining may have different scales, the most suitable learning rate required for downstream tasks may also be different.
Without tuning learning rate extensively, we may reach a misleading conclusion.
In particular, when we adopt the default learning rate for reproducing the existing success of pretraining in Tableof Appendix A.4, we indeed observe the advantage of pretraining.
However, if we follow our procedures (e.g. extensive search learning rate and averaging over three splits), the resulting Table 2 and Table 3 indicate no performance gain by pretraining.
So we suggest that the evaluation of pretraining should consider the hyper-parameter tuning and averaging over different splits.
Summary and takeaways
Based on our experiments in Section 4, we present our takeaways by empirically summarizing our conjectures on when the pretraining would/would not help the molecular representation learning.
When pretraining might help We ﬁnd it typically helps 1) if we can have the supervised pretraining with target labels that are aligned with the downstream tasks.
However, getting large amount of high-quality and relevant supervision is not always feasible; 2) if the high quality hand-crafted features are absent.
However, it seems that the gain obtained by self-supervised pretraining is not as signiﬁcant as these high quality hand-crafted features based on our current studies; 3) if the downstream train, valid and test dataset distributions are substantially different.
When the gain diminishes?
In some situations the gain of pretraining might diminish 1) if we already have the high quality hand-crafted features (e.g. rich features described in Section 3.1); 2) if we don’t have the highly relevant supervisions.
As shown in Section 4.6, many self-supervised pretraining tasks might be too easy for the model to learn meaningful embedding; 3) if the downstream data splits are balanced; 4) if the self-supervised learning dataset lacks diversity, despite its scale.
Why pretraining may not help in some cases?
In our paper we pretrained a GNN on a much larger dataset (SAVI) than before, hoping to replicate gain of pretraining like in NLP domain.
However, we do not obtain the expected gain.
The pretraining accuracy curve (Figure 2) provides some potential explanations of why pretraining may not work: some of the pretraining accuracy curve grows above 95% + quickly and converges fast, unlike pretraining in NLP which keeps growing to 70% and hardly plateaus. This suggests that
some of the pretraining methods like masked node label prediction might be easy (as the vocabulary size is much smaller compared to NLP) and
therefore
transfer less knowledge for downstream tasks.
Limitations of current study
Although we have tried our best to design a comprehensive study on the effectiveness of graph pretraining for molecular representation, there are still limitations we want to point out. Due to the limited time and resources we have, we are not able to fully cover the whole picture of the current pretraining paradigm in graph neural networks. Nevertheless, we list them here in hope of preventing the over-generalization of our conclusion. • Distribution of graphs .
Our study focuses on pretraining for small molecule graph inductive representation learning.
Recently there are works on pretraining transductive representation learning [36] on large graphs [13], where our conclusion may not be directly extended to these cases. • Graph architectures .
GNN is a popular research ﬁeld where many new architectures with probable expressiveness are/will be proposed. The results we have shown are on two representative 1-WL GNNs. It can be possible that the latest advances of deep GNN [17] and Transformer-based GNN [34, 3, 14] might yield different results. • Learning objectives .
Although we have presented results with different types of self-supervised losses, there are still many variants of each type that we did not explore, like different variants [26, 31] of contrastive learning.
Also, multi-task learning of different self-supervised objectives might be another direction for further exploration.
• Downstream datasets .
We obtained our conclusion mainly on the datasets from MoleculeNet [30]. Datasets like Alchemy [4] and drug-target Interaction [18] may show different results. A Appendix A.1 GNN architectures We show results with GraphSage [9] architecture as GNN backbone in Table 12 and Table 13.
We investigated the two different splits used in previous sections, as well as different self-supervised and supervised pretraining objectives. The overall performance using GraphSage architecture is comparable with results obtained using GIN architecture [Table] and the conclusion about pretraining objectives is the same with what we obtained on the GIN as well.
As generally these architectures have similar representation power [33], this outcome should be expected. Additionaly, we also explored graph pretraining with graph transformer proposed in [22], which is supposed to be more expressive. However, Table 18 in Appendix shows that the results are not competitive. For higherorder GNNs or the deeper GNNs the conclusion might be different, but in general we hold a conservative view towards whether the graph architecture can make a big difference in deciding whether graph pretraining is helpful
[Table]
[Table] [Table] A.2 Graph Parameters: GNN layers We also investigate the effect of the number of GNN layers in Graph pretraining.
We use the recommended graph parameters in [12] (number of GNN layer=5), which are selected for best performance. Table 14 below is using node mask pretraining for GNN layer = 7. Results of GNN layer = 5 is presented in Table 2 and 4. Compared with layer 5 vs 7, the conclusion that “pretraining does not help statistically signiﬁcantly" does not change.
[Table]
As many pretraining objectives to be evaluated are following [ [Table] [Table]
we used the intersection dataset of the two papers, except we eliminate CLINTOX.
Because CLINTOX has a signiﬁcant high variance in balanced scaffold split even without pretraining (see Table 15 and 16 below), so we remove it from the evaluation to avoid unstable and biased evaluation. Additionally, we provide CLINTOX results (Table 15 and 16). [Table] [Table]
We ﬁrst reprudce the results from Hu et al. [12] in Table 17, where we make sure the code base can obtain the similar results where the pretraining seems to be helpful. However as in our analysis, The three factors (feature engineering, data splits, tuning of hyper-parameters) all contribute positively to the results
[Table]
and the effect of these factors could be even larger than the pretraining itself [Table] Table 18 shows the results we reproduced for GROVER [22]. We compared the ﬁne-tuning results with the model initialized from the pretrained ones provided in their website.
However we didn’t see signiﬁcant gains over the model trained from scratch. We believe the hyper-parameters matters more in this case. A.5 Molecular benchmark description • BBBP : The Blood-brain barrier penetration (BBBP) dataset is extracted from a study on the modeling and prediction of the barrier permeability.
As a membrane separating circulating blood and brain extracellular ﬂuid, the blood-brain barrier blocks most drugs, hormones and neurotransmitters. Thus penetration of the barrier forms a long-standing issue in development of drugs targeting central nervous system.
This dataset includes binary labels for over 2000 compounds on their permeability properties. References: Martins, Ines Filipa, et al.
A Bayesian approach to in silico blood-brain barrier penetration modeling.
" Journal of chemical information and modeling 52.6 (2012): 1686-1697.
• BACE
The BACE dataset provides quantitative (IC50) and qualitative (binary label) binding results for a set of inhibitors of human β -secretase 1 (BACE-1). All data are experimental values reported in scientiﬁc literature over the past decade, some with detailed crystal structures available. A collection of 1522 compounds with their 2D structures and properties are provided. References: Subramanian, Govindan, et al. "
Computational modeling of β -secretase(BACE-1) inhibitors using ligand based approaches. " Journal of chemical information and modeling 56.10 (2016): 1936-1949. • TOX
The “Toxicology in the 21st Century” (Tox21) initiative created a public database measuring toxicity of compounds, which has
been used in the
2014 Tox21 Data Challenge
This dataset contains qualitative toxicity measurements for 8k compounds on
different targets, including
nuclear receptors and stress response pathways. The data ﬁle contains a csv table, in which columns below are used: "smiles" - SMILES representation of the molecular structure "NR-XXX" - Nuclear receptor signaling bioassays results "SR-XXX" - Stress response bioassays results please refer to the links at https: //tripod.nih.gov/tox21/challenge/data.jsp for details. References: Tox21 Challenge. https://tripod.nih.gov/tox21/challenge/
ToxCast is an extended data collection from the same initiative as Tox21, providing toxicology data for a large library of compounds based on in vitro high-throughput screening. The processed collection includes qualitative results of over 600 experiments on 8k compounds.
[Table]
[Table] [Table] The data ﬁle contains a csv table, in which columns below are used: "smiles" - SMILES representation of the molecular structure "ACEA_T47D_80hr_Negative", "Tanguay_ZF_120hpf_YSE_up" - Bioassays results please refer to the section "high-throughput assay information" at https://www.epa.gov/chemical-research/ toxicity-forecaster-toxcasttm-data for details.
References: Richard, Ann M., et al. "ToxCast chemical landscape: paving the road to 21st century toxicology." Chemical research in toxicology 29.8 (2016): 1225-1251. • SIDER
The Side Effect Resource (SIDER) is a database of marketed drugs and adverse drug reactions (ADR).
The version of the SIDER dataset in DeepChem has grouped drug side effects into 27 system organ classes following MedDRA classiﬁcations measured forapproved drugs.
The data ﬁle contains a csv table, in which columns below are used: "smiles" - SMILES representation of the molecular structure "Hepatobiliary disorders" "Injury, poisoning and procedural complications" - Recorded side effects for the drug Please refer to http: //sideeffects.embl.de/se/?page=98 for details on ADRs.
References: Kuhn, Michael, et
Nucleic acids research
2015): D
D1079.
Altae-Tran, al. "
drug http