Substructure-Atom Cross Attention for Molecular Representation Learning
Abstract
Designing a neural network architecture for molecular representation is crucial for
AI-driven drug discovery and molecule design.
In this work, we propose a new framework for molecular representation learning.
Our contribution is threefold: (a) demonstrating the usefulness of incorporating substructures to node-wise features from molecules, (b) designing two branch networks consisting of a transformer and a graph neural network so that the networks fused with asymmetric attention, and (c) not requiring heuristic features and computationally-expensive information from molecules.
Using 1.8 million molecules collected from ChEMBL and PubChem database, we pretrain our network to learn a general representation of molecules with minimal supervision.
The experimental results show that our pretrained network achieves competitive performance on 11 downstream tasks for molecular property prediction.
Introduction
Predicting properties of molecules is one of the fundamental concerns in various fields.
For instance, researchers apply deep neural networks (DNNs) to replace expensive real-world experiments to measure the molecular properties of a drug candidate, e.g., the capability of permeating the bloodbrain barrier, solubility, and affinity.
Such an attempt significantly reduces wet-lab experimentation that often takes more than ten years and costs $1 million [1, 2].
Among the DNN architectures, graph neural networks (GNNs) and Transformers are widely adopted to recognize graph structure of molecules.
GNNs are powerful in capturing local information of a node, but may lack the ability to encode information from far-away nodes due to over-smoothing and over-squashing issues [3, 4].
On the other hand, Transformer-based architectures can encode global information effectively as they consider attention between every pair of nodes from the first layer.
However, naive Transformers cannot incorporate structural information such as the edge and connectivity in graphs.
From the understanding of chemical structure, it is known that meaningful substructures can be found across different molecules, also known as motif or fragments [5].
For example, carbon rings and
NO2 groups are typical substructures contributed to mutagenicity [6] showing that proper usage of substructures can help a property prediction.
Molecular substructures are often represented as molecular fingerprints or molecular fragmentation.
Molecular fingerprints such as MACCS (Molecular ACCess System) keys [7] and Extended-Connectivity Fingerprints (ECFPs) [8] represent a molecule into a fixed binary vector where each bit indicates the presence of a certain motif in the
[Figure]
molecule.
With a predefined fragmentation dictionary, such as BRICS [9] or tree decomposition [10], a molecule can be decomposed into distinct partitions.
We propose a fusion architecture between a GNN and Transformer to incorporate molecular graph information and molecular substructures.
Molecular substructures and graph are encoded through
Transformer and GNN, respectively.
The Transformer is designed to recognize the molecular substructures.
With the Transformer only architecture, however, local information of molecules, such as atoms, bonds, and connectivity, can be lost from the structures.
For example, the two molecules shown in Figure 1b share the same representation with MACCS keys while having different structures.
To overcome, we use a separate GNN branch for preserving local information.
In our model, we inject the GNN feature into the intermediate Transformer layers through the fusing network.
In this way, substructures and local node information are interactively fused, producing a final representation for molecular graphs.
We name our network as Substructure-Atom Cross Attention (SACA) as it uses substructure as well as atom information in molecules and fuses them through cross-attention.
The architectural choices allows us to avoid heuristic features and our model reduces the complexity for attention calculation over the node-level Transformer models from O ( N 2 ) to O ( N ) , where N is the number of atoms.
To demonstrate the empirical effectiveness of the proposed network and see the ability to capture the general representation of molecules, we evaluate our model on 11 downstream tasks from
MoleculeNet [11].
Our approach achieves the competitive performance on 11 downstream tasks.
In what follows, we summarize the key contributions and benefits.
• We propose a novel network that combines the information from substructures and node features in a molecule.
Our model combines the advantages of both Transformer and GNN architectures to represent the information given to each architecture.
• We show the effectiveness of our model for molecular representation learning.
Our model achieves competitive performance upon strong baseline models on 11 molecular property tasks.
• Our model does not require computationally-expensive heuristic information of molecular graphs.
Related Work 2.1 Architectures for Molecular representation learning
Graph neural networks (GNNs)
Most common architecture for molecular representation learning is the GNNs since molecules can be naturally represented as a graph structure; a node as an atom, an edge as a connection.
Researchers have actively investigated variations of GNN architectures [12, 13, 14, 15], for molecules.
For example, MPNN [12] generalizes the message passing frameworks and explores some variants that predict molecular properties.
Directed MPNN (DMPNN) [13] proposed to replace the node-based message by edge-based messages to avoid unnecessary message loops.
Next, communicative MPNN (CMPNN) [15] improved DMPNN by additionally considering the node-edge interaction during the message passing phase.
AttentiveFP [14] extends the graph attention mechanism to allow for nonlocal effects at the intramolecular level.
Despite the advance of GNN architectures, there are known problems in GNN, such as over-smoothing and over-squashing problems [3, 4], which means the node representations become too similar, and the information from far nodes does not propagate well as the number of neighbors increases exponentially.
Furthermore, expressive powers of standard GNNs following neighborhood aggregation scheme are bounded to Weisfeiler-Lehman test (WL-test)
Therefore, GNNs with standard message passing cannot learn to discern a simple substructure such as cycles.
For example, the two molecules in Figure 1a cannot be distinguished by WL-test, hence standard GNNs cannot distinguish these two molecules [18].
A solution to this limited representation power of GNNs is to directly incorporate important substructures in the representation learning framework.
Transformers With recent advance of Transformer architectures and their promising performance in various domains, including NLP and computer vision [19, 20], Transformer-based architectures for molecular representation learning [21, 22, 23, 24] have been developed.
Transformer architecture calculates pair-wise attention between every node from the first layer.
Therefore, it can effectively capture the global information of a graph.
However, a vanilla Transformer architecture [25] is not directly applicable for molecular graph representation because it cannot incorporate structural information such as the edge and connectivity in graphs.
To bridge the gap, advanced Transformer architectures alter the self-attention layer [21, 22, 24] or incorporate message passing networks into Transformer architectures for input feature [23].
Specifically, MAT [21] uses adjacency and distance matrix of atoms to augment the self-attention layer.
GROVER [23] runs Dynamic Message
Passing Network (dyMPN) over the input node and edge features to extract queries, keys and values for self-attention layers.
CoMPT [26] uses the shortest path information between two nodes, and
Graphormer [22] encodes node’s degree, edges and the shortest path with edges between two nodes for molecular data.
Although these graph-specific features are found to be useful in graph representation, these features can be heuristic and impose excessive computational overhead, which limits the model’s applicability to large molecules.
Our proposed model does not require any computationally expensive heuristic features such as 3D information or shortest path for preprocessing or computation in attention layer.
2.2 Molecular Substructure
Substructure information extracted from molecules has been widely used in molecular generation, property prediction, and virtual screening [27, 28, 29, 10].
ECFPs [8] encode existing substructures within a circular distance from each atom in a molecule.
PMTNN [30] is a multi-task network that takes ECFPs as an input to predict molecular properties.
MACCS keys [7] extract substructures from molecules depending on the presence of pre-defined functional groups.
One example of the usage of MACCS keys is to encode known ligands of each protein, which leads to an improvement of prediction performance in protein-ligand interaction [31].
Molecular fragmentation method such as BRICS fragmentation [9, 32] or tree decomposition [10] is to decompose molecules in nonoverlapping partitions with pre-defined rules.
BRICS fragmentation divides molecules following chemical reaction based rules.
Tree decomposition proposed in Jin et al [10] also extracts junction tree by contracting certain edges.
One node in the junction tree represents a substructure of original molecules.
Our model receives molecular substructures as input to supplement GNN’s expressivity and it can flexibly encode any substructure vocabulary.
Our model combines Transformer and GNN, with molecular substructures and molecular graph as inputs to each network.
There are existing studies that have also considered combining GNN and
Transformer architecture [33, 34, 35] to learn graph representations.
Specifically, DMP [33] utilizes
GNN and Transformer to encode graphs and SMILES representation of molecules and train both branches using a consistency loss to match the two outputs of input molecules.
PoseGTAC [34] and
GraphFormers [35] combine GNN and Transformer layer alternatively to enlarge the receptive field or to mix the output of Transformer.
Our model is the first to encode substructures through Transformer and inject atom features through a separate GNN.
In this way, we preserve both substructures and local atom features of molecules.
P r op e r t y M L P
Q ×𝑀 ×𝑁
Self Attention
Fusing Network
Self Attention
Node Embedding
K
V
GNN
C O
N C
C O
N C
C C
C C MLP CLS
CLS
CLS CLS
SMILES
Substructures
In this section, we explain the architecture of our model.
Overall architecture Figure 2 shows the overall architecture of our model.
Our network consists of two branches: (i) a Transformer branch that uses the molecular substructures as input and (ii) a GNN branch that uses a molecular graph as input.
The two branches have different roles.
First, the Transformer branch is intended to capture global information of molecules.
It receives the molecular substructures that have important role in molecular properties, but cannot be easily captured by
GNNs, and learn the overall representation of molecules.
On the other hand, the GNN branch is intended to capture local node information of molecules.
The two different levels of information are mixed through the fusing network in the Transformer branch.
Transformer branch Our Transformer branch is to incorporate both molecular substructure information and local node features.
The input token for Transformer is the substructure embeddings of molecules.
Predefined substructures are first detected and then projected into separate embedding vectors.
For example, in Figure 2, substructures such as N-Heterocycle, carbonoxygen bond and methyl group are detected and embedded into learnable embedding vectors.
To identify substructure from the input molecules, we use MACCS keys [7], which indicates the presence of motifs in a molecule.
Note that our architecture is not limited to certain molecular substructures, but it can flexibly receive any substructure vocabulary.
The embeddings of substructures are mixed together and refined as they are passed through the self-attention module.
The substructure embeddings after self-attention layer are fused with node embeddings from a separate GNN branch.
The fusing network computes cross-attention between substructures and nodes where substructures are used as query and nodes are used as key and value.
The detailed computation of the fusing network is shown in Figure 3.
In the fusing network, the cross-attention between each pair of substructure embedding and node embedding is computed.
To be specific, for a given molecule having n atoms and m extracted substructures, we have substructure embeddings E s ∈ R m × d and node embeddings E n ∈ R n × d where d is the embedding dimension.
Then, the cross-attention is computed as follows:
[Equation]
where W Q , W K , W V ∈ R
d × d k are learnable parameters.
The cross-attention module outputs E ′ s ∈
R m × d k .
Through this fusing network, the substructure embeddings aggregate the local information from node embeddings.
Structurally important nodes are aggregated with more weights.
Instead of designing heuristic weights on the nodes, our model can learn to select structurally important nodes related to graph-level property by cross-attention.
Additionally, as the attention is computed between substructures and nodes, the space and time complexity of the self and cross attention map of our model is linear to the number of atoms, i.e., O ( N ) , whereas other transformer architectures for molecular graphs have quadratic complexity.
Structural Embedding
C C
C C
C C
N
C
O
Node Embedding
Q
K
V
C O
Attention Scores
C
N
C 𝑚×𝑑 𝑛×𝑑
Outputs 𝑚×𝑑 𝑚×𝑑
The self-attention between substructures and fusing network with node embeddings are repeated iteratively N times.
Before making the final prediction, we add M self-attention layers at the end of the network for making refinement on the substructures.
We add a residual connection from the input tokens to every output of the fusing network.
This ensures the input structural information last throughout the entire network.
Furthermore, we use a CLS token similar to special classification token in BERT [19] to aggregate the global representation of molecules.
The CLS token is shown in Figure 2.
It is a learnable latent vector and passed to the Transformer branch attached to the substructure tokens as input.
A CLS token has been used in many
Transformer-based architectures [21, 22] for molecular representation learning.
We attach an MLP head to the final CLS token to perform graph-level prediction tasks.
GNN branch The GNN branch is used to extract local node features from molecular graphs.
For
GNN architecture, we use GIN [17] with jumping knowledge [36].
The computed node features are injected into the Transformer through the fusing network with the same hierarchy.
For example, 0-hop node representations are injected into the first cross-attention module and 1-hop representations for the second cross-attention module.
This allows the model to encode local node features progressively from the shallow to deeper layers.
The computation of node representation through GNNs and injection to the Transformer allow us to take advantage of both GNNs and Transformer architectures.
Substructures that are hard to be captured by GNNs, but essential to molecular properties, are first detected and encoded through
Transformer.
Meanwhile, local node information that can be lost in using substructures alone is effectively captured by GNNs and fused with substructures.
Additionally, our architecture does not require computationally expensive high-order graph-level information.
As Transformer cannot naturally incorporate a graph’s edge connectivity information, existing work [23, 22, 26] mainly focuses on how to add structural bias such as the shortest path or 3D distance between two nodes into the Transformer self-attention computation.
However, these structural biases require computationally expensive preprocessing of the molecular datasets.
Our network that utilizes GNN as a separate branch can avoid these limitations.
Experiment 4.1 Experimental Setting
Pretraining We pretrain our network to obtain molecular representation transferable to various molecular datasets and tasks.
For pretraining, we extracted 200 real-valued descriptors of physicochemical properties from the pretraining datasets using RDKit [37] and train our network to predict these properties.
As the 200 molecular descriptors include a diverse set of molecular properties, the model can learn a representation of molecules that can be used for various downstream tasks.
Dataset We collected 1,858,081 number of unlabeled molecules from ChEMBL and PubChem databases [38, 39].
ChEMBL and PubChem are large-scale databases that include a variety of chemical and physical properties, and biological activities of molecules.
To obtain molecular substructures, we utilize MACCS key [7], a 166 dimensional vector that indicates a presence of certain substructure in molecules, and extract this for every molecule using RDKit [37].
We use OGB package [40] to convert SMILES [41], a text-representation for molecules, to molecular graphs.
Implementation details We set M = 4 and N = 3 , where M and N are defined in section 3.
We set 768-dimensional hidden units and 16 attention heads.
When pretraining, we used the AdamW optimizer [42] with a learning rate of 1e-4.
We divide the pretraining dataset into a 9:1 ratio and use them for training and validation sets.
The model is trained for 10 epochs, and the model with the best validation loss is used for downstream tasks.
Further details for pretraining setting is presented in
Appendix B.
4.2 Downstream tasks
Tasks We evaluate the performance on six classification tasks (BBBP, SIDER, ClinTox, BACE,
Tox21, and ToxCast) and five regression tasks (FreeSolv, ESOL, Lipo, QM7 and QM8) from
MoleculeNet [11].
Each task is related to molecular property from low-level, for example, water solubility in ESOL to high-level, possibility of blood-brain barrier penetraion in BBBP.
Further details about the dataset statistics and the downstream tasks and are available in Appendix C.
Experimental setting To evaluate each model, we use 3 different scaffold splits [11] following [23].
Scaffold split divides structurally different molecules into different subsets and provides more challenging and realistic test environment.
From the pretrained model, we replace the last MLP layer of the network with the task-specific MLP heads.
For each downstream dataset, we train our model for 100 epochs and report the test score corresponding to the best validation epoch.
We tune hyperparameters with Bayesian optimization search with a budget of 100 for learning rate, dropout, weight decay and the number of last prediction heads.
The hyperparameter search range is provided in Appendix B.
We compare the performance of our model on the downstream tasks with several GNN and Transformer based state-of-the-arts approaches for molecule representation learning.
TF-Robust [30] is a
DNN-based model that takes molecular fingerprints.
GNN-based models include GraphConv [43],
Weave [44] and SchNet [45] which are 3 graph convolutional networks, MPNN [12], DMPNN [13],
MGCN [46] and CMPNN [15] which are GNN models considering the edge features during message passing.
AttentiveFP [14] is an extension of graph attention network for molecule representation.
Transformer-based models include GROVER [23], MAT [21], Graphormer [22] and CoMPT [26].
Among the baselines, N-GRAM [47], [48], GraphLoG [49], MAT, GROVER, Graphormer, GEM [50] and MPG [51] are models that use pretraining strategies.
We report GROVER base model for a fair comparison in terms of the number of parameters.
To reproduce the results for models using pretraining strategies, we use the pretrained model made available by the authors.
We reproduce the results of MPG due to the different splits used in [51] 2 .
We also evaluate our model with the same data split used for MPG and the results can be found in Appendix G.
Results Table 1 shows the overall results of the baselines and our model on 11 MoleculeNet datasets.
Our model achieved the best performance on five downstream tasks: ClinTox, Tox21, ToxCast, ESOL and QM8, and the second best performance on five downstream tasks: SIDER, BACE, FreeSolv, Lipo and QM7.
We also computed the average rank for the classification and regression tasks, separately.
Our model achieved the best average rank among all compared models.
The result shows that our model generalizes well across different downstream tasks, which means local information aggregated from GNN can propagate globally as interacting with structural information in Transformer.
We find that some substructures weight high attention to nodes consisting of the substructures, and CLS token used for prediction also focuses on the substructures from attention scores in self-attention.
4.3 Ablation study
We report ablation study to justify each component and flexibility of our model architecture.
For the ablation study, we reduce the hyperparameter search space to only 6 learning rates {1e-3, 5e-4, 1e-4, 5e-5, 1e-5, 5e-6} to facilitate the comparison between different models.
Ablation on model components Figure 4 shows the performance comparison between four variations of our model on four downstream datasets.
We first verify the performance of our model without
GNN branch.
To do that, we replace all cross-attention layers with self-attention layers and exclude
GNN branch.
Begin-Concat and End-Concat examine different ways of combining substructure and local features.
Begin-Concat runs a vanilla-transformer encoder on top of a concatenated atom and
[Table]
7 MGCN [46]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
AttentiveFP
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
010) 2.3 Graphormer [22] ✓ .938 (.032) .625 (.009) .913 (.056) .848 (.023) .801 (.013) .718 (.007) 6.7 GraphLoG [49] ✓ .913 (.024) .595 (.039) - .845 (.012) .773 (.010) .677 (.008) 13.0 MPG* [51] ✓ .922 (.039) .628 (.014) - .864 (.028) .800 (.024) .712 (.009)
7.4 GEM [50] ✓ .921 (.026) .603 (.012) - .872 (.036) .815 (.016) .720 (.010) 6.
Ours ✓ .934 (.018) .646 (.009) .935 (.014) .877 (.032) .829 (.013) .730 (.005) 1.
Regression Tasks
Method
[Table]
TF_Robust [30] - 4.122 (.085) 1.722 (.038) .909 (.060) 120.6 (9.6) .024 (.001) 15.2 Weave [44] - 2.398 (.250) 1.158 (.055) .813 (.042) 94.7 (2.7) .022 (.001) 11.8 GraphConv [43] - 2.900 (.135) 1.068 (.050) .712 (.049) 118.9 (20.2) .021 (.001) 12.
SchNet [45] - 3.215 (.755) 1.045 (.064) .909 (.098) 74.2 (6.0) .020 (.002)
11.4 MPNN [12] - 2.185 (.952) 1.167 (.430) .672 (.051) 113.0 (17.2) .015 (.002) 10.0 DMPNN [13] - 2.177 (.914) .980 (.258) .653 (.046) 105.8 (13.2) .
(.002) 7.8 MGCN [46] - 3.349 (.097) 1.266 (.147) 1.113 (.041) 77.6 (4.7) .022 (.002)
13.6 AttentiveFP [14] - 2.030 (.420) .853 (.060) .650 (.030) 126.7 (4.0) .
[Table]
.001) 8.8 CMPNN [15] - 2.254 (.356) .841 (.090) .606 (.040) 70.6 (2.7) .
(.001) 4.8 CoMPT [26] - 2.125 (.590) .898 (.053) .
N-GRAM [47] ✓ 2.512 (
[Table] [Table]
[Table] [Table]
[Table]
003) 14.0 MAT [21] ✓ 2.116 (
[Table]
[Table] [Table]
[Table] [Table]
002) 6.6 GROVER [23] ✓ 1.
[Table]
[Figure] [Figure]
substructure embeddings.
End-Concat runs a vanilla-transformer encoder on top of atom embeddings and concatenates the substructure feature at the end to make the final prediction.
Random Embedding examines the effect of substructural embeddings by replacing them with random learnable embeddings.
Figure 4 shows our model outperforms other variations on all datasets, which justifies the necessity of each component.
Further details about ablation study are available in Appendix A
Different GNN branch Our model can flexibly utilize other GNN architectures as our GNN branch.
We test the changes in performance when our model design is applied with different
GNN architectures.
Figure 5 shows the comparison between commonly-used GNNs, i.e., Graph
Convolutional Network (GCN), Graph Attention Network (GAT) [52] and GIN, and our models with the GNN branch switched to each corresponding GNN on ToxCast dataset.
There is a significant performance improvement when our model is adopted to each GNN model.
Effectiveness of substructures Figure 6 shows the t-SNE embeddings of molecules with different substructures.
Our model discriminates molecules with similar substructures (aromatic rings with one or two different atoms) better than GIN.
[Figure] [Figure]
4.4 Analysis
[Figure]
Cl
Cl Cl Cl
Cl
[Figure]
C
[Figure]
C (a) Ring-Chain-Ring bond
[Figure]
⋯
Cl Cl
Cl Cl Cl
Cl
Cl Cl Cl Cl Cl
Cl
Cl Cl Cl
Cl
Atom tokens CLS s ub s t r u c t u r e t ok e n s
Cl Cl
Cl
[Figure]
Chlorine
[Figure]
7 shows the attention weights.
and node so we can interpret the attention score as the degree of focus between the substructures and nodes.
We check where the CLS token gives more attention since it is used for prediction.
We find out the CLS token gives strong attention to specific substructures.
More interestingly, often a substructure gives more attention on the nodes that consists the substructure itself.
Despite of not having any structural information between input substructures and nodes as input, our model can identify structurally related nodes in the cross attention layer, showcasing the ability of understanding molecular structure.
Conclusion
In this paper, we propose a novel framework that incorporates Transformer and GNN architecture for molecular representation learning.
Our model takes advantages of the two architectures to aggregate substructure and local information.
With the cross attention mechanism in fusing network, our model could achieve state-of-the-art performance on various molecular property prediction benchmarks.
Overall, our work highlights the effectiveness of SACA for molecular representation learning.
Acknowledgments and Disclosure of Funding
This work was partly supported by Institute for Information communications Technology Promotion (IITP) grant funded by the Korea government (MSIP) (No. 2019-0-01906: Artificial Intelligence
Graduate School Program(POSTECH), No.2021-0-02068:
AI Innovation Hub), the National Research
Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2022R1C1C1013366, 2022R1C1C1013366), and Technology Innovation Program (No.1415178073, Development of
BIT Convegent AI {+Architecture its Validation and Candidate Selection for COVID
 Antibopdy Repositioning and Novel Synthetic Chemical Therapeutics)+} funded by the Ministry of Trans, Industry Energy (MOTIE, Korea)
A Additional Ablation Study
[Figure]
QM7 (↓)
Change GNN branch to 3D-aware GNN To observe the effect of the choice of GNN architecture for GNN branch, we conduct additional experiments with 3D-aware GNN.
We changed our GNN branch from GIN to SGCN [53] that utilizes 3d coordinates of input molecules.
Please note that the two models are trained on the downstream dataset from scratch.
Figure 8 shows that the model with 3D-aware GNN shows better performance than our original model on QM7 dataset whose task is closely related to 3D information.
The result shows the flexibility of our framework that can be further tuned by using task-appropriate GNN architecture.
Other substructures as input tokens for Transformer Despite our model utilizes MACCS keys for the input of Transformer, our model can flexibly receive any substructure vocabulary.
We conduct experiments on ECFP fingerprints and Tree decomposition for other substructures.
Table 2 shows the results.
It shows using MACCS keys achieves better performance than ECFP fingerprint in most of the downstream tasks.
We speculate that MACCS keys include predefined functional groups whereas
ECFP encodes local substructure around an atom (i.e., a certain radius neighborhood of an atom).
In this aspect, ECFP fingerprint is similar to how GNN encodes node information and as we already utilize GNN to encode local information, ECFP would not bring new information about molecules.
[Table]
0.0152 ECFP0.
0.0151 Tree Decomposition 0.
Computation time for graph features Figure 9 shows the time required in millisecond to compute the MACCS keys, shortest path and 3D distance for each molecule with different number of atoms.
As shown in the figure, the time required for 3D distance and the shortest path increases dramatically as the number of atoms increase, which make these features impossible to be applied in large molecules.
However, identifying the MACCS key substructures does not depend on the input molecule’s number of atoms.
B Experimental Setting
In this section
[Table]
we explain further details of our experimental setting.
Pretraining.
The default hyperparameters for pretraining are listed in Table 3.
All attention layers have 16 attention heads and 768 hidden dimension.
Our model has 41M parameters.
[Table]
Hyperparameter Search Range.
For each downstream task, we search for the best hyperparameter combinations.
We perform the Bayesian optimization over the validation set and use the hyperparameters for the best validation score to report the test score.
The hyperparameter range that we searched over is shown in Table 4.
[Table]
Pretraining Task.
To pretrain our network
[Table]
we extract the 200 real-valued descriptors for each molecule using RDKit package [37].
This task is proposed by [54].
Through this task
[Table]
we can make the model to learn the physicochemical properties of the input molecules
[Table]
C Details of Downstream Datasets
We used 11 binary graph classification and regression datasets: BBBP, SIDER, ClinTox, BACE,
Tox21, ToxCast, FreeSolv, ESOL, Lipophilicity, QM7 and QM8 from Moleculenet [11].
The statistics of each dataset is shown in Table 5.
The details of each dataset are shown in Table 6.
Through the various datasets, we can test the generalization ability of our pretrained model
[Table]
[Table] [Table]
D Node and Edge Features
In this section, we present the node and edge features of molecules used for GNN branch.
We used
OGB package [40] to convert SMILES strings [41] to molecular graphs.
The molecular graphs are encoded through GIN [17] and injected into the Transformer branch by the cross-attention.
The molecular graphs have the following node and edge features.
Node features.
Each node has the following 9 dimensional features as shown in Table 7.
[Table]
Edge features.
Each edge has the following 3 dimensional features as shown in Table
[Table] [Table]
E Molecular Structural Keys
In this section, we explain the details of molecular substructures that we used for our model, i.e.,
Molecular ACCess System (MACCS) keys [7].
We extract MACCS keys for each molecule using
RDKit package [37].
MACCS keys have 166-dimensional features where each binary label indicates the presence of a particular substructure in the given molecule.
For example, the 139 th index of
MACCS keys indicates the presence of the Hydroxy group (-OH), and the 162 nd index indicates the presence of the aromatic ring in a molecule.
The full list of 166 MACCS keys can be found in the document 3 .
[Figure]
Figure 10 shows examples of how MACCS keys encode substructure information into a binary bit string.
The canonical SMILES representations of the left and right molecules are
C1=
CC=C(C
=
C1)O and Cc1cc(Cl)cc(Cl)c1CNC(=O)c1cccs1 , respectively.
Table 9 shows the MACCS keys statistics for pretraining and downstream datasets.
It includes the average number of MACCS keys that molecules in each dataset contain and the standard deviation.
Also, the minimum and the maximum number of MACCS keys for each dataset are reported.
[Table]
BBBP 46.
SIDER 46.
ClinTox 46.
BACE 61.
Tox 32.
ToxCast 33.
FreeSolv 15.
ESOL 23.
Lipophilicity 51.
QM 18.
QM 23.
https
F Parameters and space complexity of different models
We present the number of parameters of transformer-based molecular representation learning models in Table 10.
Except for CoMPT, our model has a comparable or lower number of parameters than other transformer-based models.
We also present the space complexity in Table 10.
The space complexity for our attention computation is linear to the number of atoms (i.e., O ( n ) where n is the number of atoms).
On the other hand, the space complexity of other transformer-based molecule representation learning models is quadratic to the number of atoms (i.e., O ( n 2 ) ).
[Table]
G Performance on MPG split
As MPG uses different splits to GROVER on the paper
[Table]
we reproduce the result of our model on the same split that MPG used for fair comparison.
As shown in Table
[Table]
our model outperforms MPG on most of the downstream datasets.
[Table]
H Others
URLs for pretrained model for baselines.
We use the following sources for pretrained models to reproduce the results.
Graphormer: https://github.com/microsoft/Graphormer
MAT: https://github.com/ardigen/MAT
GraphLoG: https://github.com/DeepGraphLearning/GraphLoG
GEM: https://github.com/PaddlePaddle/PaddleHelix/tree/dev/apps/pretrained_ compound/ChemRL/GEM