O NTO P ROTEIN : P ROTEIN P RETRAINING W ITH G ENE O NTOLOGY E MBEDDING
A BSTRACT
Self-supervised protein language models have proved their effectiveness in learning the proteins representations.
With the increasing computational power, current protein language models pre-trained with millions of diverse sequences can advance the parameter scale from million-level to billion-level and achieve remarkable improvement.
However, those prevailing approaches rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better protein representations.
We argue that informative biology knowledge in KGs can enhance protein representation with external knowledge.
In this work, we propose OntoProtein , the ﬁrst general framework that makes use of structure in GO (Gene Ontology) into protein pre-training models.
We construct a novel large-scale knowledge graph that consists of GO and its related proteins, and gene annotation texts or protein sequences describe all nodes in the graph.
We propose novel contrastive learning with knowledge-aware negative sampling to jointly optimize the knowledge graph and protein embedding during pre-training.
Experimental results show that OntoProtein can surpass state-of-the-art methods with pre-trained protein language models in TAPE benchmark and yield better performance compared with baselines in protein-protein interaction and protein function prediction 1 .
I NTRODUCTION
Protein science, the fundamental macromolecules governing biology and life itself, has led to remarkable advances in understanding the disease therapies and human health (Vig et al.
As a sequence of amino acids, protein can be viewed precisely as a language, indicating that they may be modeled using neural networks that have been developed for natural language processing (NLP).
Recent self-supervised pre-trained protein language models (PLMs) such as ESM (Rao et al.
(2021b)), ProteinBERT (Brandes et al.
, ProtTrans (Elnaggar et al.
(2020)) which can learn powerful protein representations, have achieved promising results in understanding the structure and functionality of the protein.
Yet existing PLMs for protein representation learning generally cannot sufﬁciently capture the biology factual knowledge, which is crucial for many protein tasks but is usually sparse and has diverse and complex forms in sequence.
By contrast, knowledge graphs (KGs) from gene ontology 2 contain extensive biology structural facts, and knowledge embedding (KE) approaches (Bordes et al.
, Zheng et al.
(2021)) can efﬁciently embed them into continuous vectors of entities and relations.
For example, as shown in Figure 1, without knowing PEX5 has speciﬁc biological processes and cellular components, it is challenging to recognize its interaction with other proteins.
Furthermore, since protein’s shape determines its function , it is more convenient for models to identify protein’s functions with the prior knowledge of protein functions having similar shapes.
Hence, considering rich knowledge can lead to better protein representation and beneﬁts various biology applications, e.g., protein contact prediction, protein function prediction, and protein-protein interaction prediction.
However, different from knowledge-enhanced approaches in NLP (Zhang et al.
(2019b), Wang et al.
(2021b)
, Wang et al.
(2021a))
, protein sequence and gene ontology are two different types of data.
Note that protein sequence is composed of amino acids while gene ontology is a knowledge graph with text description; thus, severe issues of structured knowledge encoding and heterogeneous information fusion remain.
In this paper, we take the ﬁrst to propose protein pre-training with gene ontology embedding ( OntoProtein ), which is the ﬁrst general framework to integrate external knowledge graphs into protein pre-training.
We propose a hybrid encoder to represent language text and protein sequence and introduce contrastive learning with knowledge-aware negative sampling to jointly optimize the knowledge graph and the protein sequence embedding during pre-training.
For the KE objective, we encode the node descriptions (go annotations) as their corresponding entity embeddings and then optimize them following vanilla KE approaches (Bordes et al.
We further leverage gene ontology of molecular function, cellular component, and biological process and introduce a knowledge-aware negative sampling method for the KE objective.
For the MLM (Mask Language
Modeling) objective, we follow the approach of existing protein pre-training approaches (Rao et al.
(2021b)).
OntoProtein has the following strengths: OntoProtein
inherits
the strong
ability of protein understanding from PLMs
with
the MLM
ob ject. OntoProtein can integrate biology knowledge into protein representation with the supervision from KG by the KE object.
(3) OntoProtein constitutes a model-agnostic method and is readily pluggable into a wide range of protein tasks without additional inference overhead since we do not modify model architecture but add new training objectives.
For pre-training and evaluating OntoProtein, we need a knowledge graph with large-scale biology knowledge facts aligned with protein sequences.
Therefore, we construct ProteinKG25 , which contains about 612,483 entities, 4,990,097 triples, and aligned node descriptions from GO annotations.
To the best of our knowledge, it is the ﬁrst large-scale KG dataset to facilitate protein pre-training.
We deliver data splits for both the inductive and the transductive settings to promote future research.
To summarize, our contribution is three-fold: (1) We propose OntoProtein, the ﬁrst knowledgeenhanced protein pre-training approach that brings promising improvements to a wide range of protein tasks.
(2) By contrastive learning with knowledge-aware sampling to jointly optimize knowledge and protein embedding, OntoProtein shows its effectiveness in widespread downstream tasks, including protein function prediction, protein-protein interaction prediction, contact prediction, and so on.
(3) We construct and release the ProteinKG25, a novel large-scale KG dataset, promoting the research on protein language pre-training.
(4) We conduct extensive experiments in widespread protein tasks, including TAPE benchmark, protein-protein interaction prediction, and protein function prediction, which demonstrate the effectiveness of our proposed approach.
M ETHODOLOGIES
We begin to introduce our approach of protein pre-training with ontology embedding (OntoProtein), as shown in Figure 2.
OntoProtein incorporates external knowledge from Gene Ontology (Go) into language representations by jointly optimizing two objectives.
We will ﬁrst introduce the hybrid encoder, masked protein modeling, and knowledge encoder, and then we will present the details of contrastive learning with knowledge-aware negative sampling.
Finally, we will illustrate the overall pre-training objects.
2. H YBRID E NCODER
We ﬁrst introduce the hybrid encoder to represent protein and GO knowledge.
For the protein encoder, we use the pre-trained ProtBert from Elnaggar et al.
ProtBert is pre-trained using the BERT architecture with UniRef100 datasets.
Compared to BERT Devlin et al.
(2019), ProtBert encodes amino acid sequences into token level or sentence level representations, which can be used for downstream protein tasks such as contacts prediction tasks.
The encoder takes a protein sequence of N tokens ( x 1 , ..., x N ) as inputs, and computes contextualized amnio acid representation H iProtein and sequence representation H Protein via mean pooling .
To bridge the gap between text and protein, we utilize afﬁne transformation (an extra linear layer) to project those representation to the same space.
We will discuss details of learning protein representation in Section Mask Protein Modeling.
For the Go encoder, we leverage BERT (Devlin et al.
(2019)), a Transformer (Vaswani et al.
based text encoder for biological descriptions in Gene Ontology entities.
The encoder takes a sequence of N tokens ( x 1 , ..., x N ) as inputs, and computes contextualized representations
H iGO ∈
R N × d through L layers of Transformer encoders.
Finally, we obtain contextualized representations H GO for each sequence by pooling output using [ CLS ] token representation.
Since the relations in Gene Ontology are important for representing the knowledge of biology features, thus, we utilize a relation encoder with the random initialization, and those embeddings of relations will be optimized and updated during pre-training.
2. K NOWLEDGE E MBEDDING
We leverage the knowledge embedding (KE) objective to obtain representations in the pre-training process since Gene Ontology is actually a factual knowledge graph.
Similar to Bordes et al.
(2013), we use distributed representations to encode entities and relations.
The knowledge graph here consists of lots of triples to describe relational facts.
We deﬁne a triplet as ( h, r, t ) , where h and t are head and tail entities, r is the relation whose type usually is pre-deﬁned in the schema 3 .
Note that there are two different types of nodes e GO and e protein in our knowledge graph.
e GO is denoted as nodes that exist in the gene ontology, such as molecular function or cellular component nodes, and e GO can be described by annotation texts.
e protein is the protein node that links to the gene ontology, and we also represent e protein with amnio acids sequences.
Concretely, the triplets in this knowledge graph can be divided into two groups, triple GO 2 GO and triple Protein 2 GO .
To integrate multi-modal descriptions into the same semantic space and address the heterogeneous information fusion issue, we utilize hybrid encoders introduced in the previous Section.
Note that protein encoder and GO encoder represent protein sequence and GO annotations separately.
2. M ASKED P ROTEIN M ODELING
We use masked protein modeling to optimize protein representations.
The masked protein modeling is similar to masked language modeling (MLM).
During model pre-training, we use a 15% probability to mask each token (amino acid) and leverage a cross-entropy loss (cid:96) MLM to estimate these masked tokens.
We initialize our model with the pre-trained model of ProtBert and regard (cid:96) MLM as one of the overall objectives of OntoProtein by jointly training KE (knowledge embedding) and
MLM.
Our approach is model-agnostic, and other pre-trained models can also be leveraged.
2. C ONTRASTIVE L EARNING WITH K NOWLEDGE - AWARE N EGATIVE S AMPLING
Knowledge embedding (KE) is to learn low-dimensional representations for entities and relations, and contrastive estimation represents a scalable and effective method for inferring connectivity patterns.
Note that a crucial aspect of contrastive learning approaches is the choice of corruption distribution that generates hard negative samples, which force the embedding model to learn discriminative representations and ﬁnd critical characteristics of observed data.
However, previous approaches either employ too simple corruption distributions, i.e., uniform, yielding easy uninformative negatives, or sophisticated adversarial distributions with challenging optimization schemes.
Thus, in this paper, we propose contrastive learning with knowledge-aware negative sampling, an inexpensive negative sampling strategy that utilizes the rich GO knowledge to sample negative samples.
Formally, the KE objective can be deﬁned as:
[Equation]
h
[Equation]
i ) is the negative sample, in which head or tail entities are random sampled to construct the corrupt triples.
n is the number of negative samples, σ is the sigmoid function, and γ means the margin.
d is the scoring function, and we use TransE (Bordes et al.
for simplicity, where
[Equation]
Speciﬁcally, we deﬁne triple sets and entity sets as T and E , all triplets are divided into two groups.
If the head entity is protein node and the tail entity is GO node, we denote the triple as T protein − GO .
Similarly, if head and tail entities are both GO nodes, we denote them as T GO − GO .
As Gene Ontology describes the knowledge of the biological domain concerning three aspects, all entities in Gene
Ontology belong to MFO (Molecular Function), CCO (Cellular Component), or BPO (Biological
Process).
To avoid plain negative samples, for those T GO − GO triples, we sample triples by replacing entities with the same aspect (MFO, CCO, BPO) 4 .
Finally we deﬁne the negative triple sets T (cid:48) and positive triple as ( h, r, t ) , the negative sampling process can be described as follows:
[Equation] [Equation]
where E (cid:48) ∈ { E MFO , E CCO , E BPO } , and we only replace the tail entities for T Protein − GO triples.
2. P RE - TRAINING O BJECTIVE
We adopt the mask protein modeling object and knowledge embedding objective to construct the overall object of the OntoProtein.
We jointly optimize the overall object as follows:
[Equation]
where α is the hyper-parameter.
Our approach can be embedded into existing ﬁne-tuning scenarios.
E XPERIMENT
Extensive experiments have been conducted to prove the effectiveness of our approach.
In the pretraining stage, we construct a new knowledge graph dataset that consists of Gene Ontology and public annotated proteins.
Our proposed model is pre-trained with this dataset and evaluated in several downstream tasks.
We evaluate OntoProtein in protein function prediction, protein-protein interaction and TAPE benchmark (Rao et al.
3. D ATASETS
Pre-training Dataset To incorporate Gene Ontology knowledge into language models, we build a new pre-training dataset called ProteinKG25 5 , which is a large-scale KG dataset with aligned descriptions and protein sequences respectively to GO terms 6 and proteins entities.
Gene Ontology consists of a set of GO terms (or concepts) with relations that operate between them, e.g., molecular function terms describe activities that occur at the molecular level.
A GO annotation is a statement about the function of a particular gene or gene product, e.g., the gene product “cytochrome c” can be [Table] described by the molecular function oxidoreductase activity.
Due to the connection between Gene
Ontology and Gene Annotations, we combine the two structures into a uniﬁed knowledge graph.
For each GO term in Gene Ontology, we align it to its corresponding name and description and concatenate them by a colon as an entire description.
For each protein in Gene annotation, we align it to the Swiss-Prot 7 , a protein knowledge database, and extract its corresponding sequence as its description.
In ProteinKG25, there exists 4,990,097 triples, including 4,879,951 T protein − GO and 110,146 T GO − GO triples.
Figure 3 illustrate the statistics of our ProteinKG25.
Detailed construction procedure and analysis of pre-train datasets can be found in Appendix A.1.
Downstream Task Dataset We use TAPE as the benchmark (Rao et al.
to evaluate protein representation learning.
There are three types of tasks in TAPE, including structure, evolutionary, and engineering for proteins.
Following Rao et al.
(2021a), we select 6 representative datasets including secondary structure (SS), contact prediction to evaluate OntoProtein.
Protein-protein interactions (PPI) are physical contacts of high speciﬁcity established between two or more protein molecules; we regard PPI as a sequence classiﬁcation task and use three datasets with different sizes for evaluation.
STRING is built by Lv et al.
(2021), which contains 15, proteins and 593,397 PPIs.
We also use SHS27k and SHS148k, which are generated by Chen et al.
Protein function prediction aims to assign biological or biochemical roles to proteins, and we also regard this task as a sequence classiﬁcation task.
We build a new evaluation dataset based on our
ProteinKG25 following the standard CAFA protocol
(Zhou et al.
Speciﬁcally, we design two evaluation settings, the transductive setting and the inductive setting, which simulate two scenarios of gene annotation in reality.
In the transductive setting, the model can generate embeddings of unseen protein entities with entity descriptions.
On the contrary, for the inductive setting, those entities have occurred in the pre-training stage.
The detailed construction of the dataset can be found in Appendix A.1.
As shown in Figure 3, proteins are, on average, annotated by 2 terms in CCO,in MFO, and 3 in BPO, indicating that protein function prediction can be viewed as a multi-label problem.
Notably, we notice that leaf GO terms tend to have more speciﬁc concepts than non-leaf
GO terms.
Meanwhile, there exists a challenging long-tail issue for the function prediction task.
3. R ESULTS TAPE B ENCHMARK
Baselines In TAPE, we evaluate our OntoProtein compared with ﬁve baselines.
The ﬁrst is the model with LSTM encoding of the input amino acid sequence, which provides a simple baseline.
The second is TAPE Transformer that provides a basic transformer baseline.
We further select
ResNet from He et al.
(2016) as a baseline.
The forth is the MSA Transformer (Rao et al.
(2021a)).
Note that MSA Transformer takes advantage of multiple sequence alignments (MSAs) and is the current state-of-the-art approach.
Finally, we use ProtBert (Elnaggar et al.
with 30 layers of
BERT encoder, which is the largest pre-trained model among baselines.
[Table] [Table] Results We detail the experimental result on TAPE in Table 1.
Concretely, we notice that OntoProtein yields better performance in all token level tests.
For the second structure (SS-Q3 and
SS-Q8) and contact prediction, OntoProtein outperforms TAPE Transformer and ProtBert, showing that it can beneﬁt from those informative biology knowledge graphs in pre-training.
Moreover,
OntoProtein can achieve comparable performance with MSA transformer.
Note that our proposed
OntoProtein does not leverage the information from MSAs.
However, with external gene ontology knowledge injection, OntoProtein can obtain promising performance.
In sequence level tasks, OntoProtein demonstrates its effectiveness in ﬂuorescence prediction.
However, we observe that OntoProtein does not perform well in protein engineering, homology, and stability prediction, which are all regression tasks.
We think this is due to the lack of sequence-level objectives in our pre-training object, and we leave this for future work.
P ROTEIN -P ROTEIN I NTERACTION
Baselines We choose four representative methods as baselines for protein-protein interaction.
PIPR (Chen et al.
, DNN-PPI (Li et al.
and DPPI (Hashemifar et al.
(2018)) are deep learning based methods.
GNN-PPI (Lv et al.
(2021)) is a graph neural network based method for better inter-novel-protein interaction prediction.
To evaluate our OntoProtein, we replace the initial protein embedding part of GNN-PPI with ProtBERT and OntoProtein as baselines.
Results From Table 2, we observe that the performance of OntoProtein is better than PIPR, which demonstrates that external structure knowledge can be beneﬁcial for protein-protein interaction prediction.
We also notice th at our method can achieve promising improvement in smaller dataset
SHS2K, even outperforming GNN-PPI and GNN-PPI (ProtBert).
With a larger size of datasets,
OntoProtein can still obtain comparable performance to GNN-PPI and GNN-PPI (ProtBert).
P ROTEIN F UNCTION P REDICTION
Baselines For simplicity, we leverage Seq2Vec (Littmann et al.
as the backbone for fair comparison and initialize embeddings with ProtBert and our OntoProtein.
Note that our approach is model-agnostic, and other backbones can also be leveraged.
Results We split the test sets into three subsets (BPO, MFO, and CCO) and evaluate the performance of models separately.
From Table 3, we notice that our OntoProtein can yield a 4% improvement with transductive setting and 2% advancement with inductive setting in BPO, further
[Table]
3. A NALYSIS
Table 4 illustrates a detailed experimental analysis on the contact prediction.
To further analyze the model’s performance, we conduct experiments to probe the performance of different sequences.
Speciﬁcally, protein sequence lengths from short-range ( 6 ≤ seq < 12 ) to long-range ( 24 ≤ seq ) are tested with three metrics (P@L, P@L/2, P@L/5).
We choose several basic algorithms such as LSTM and TAPE transformer as baselines.
For fairness, ProtBert is also leveraged for comparison.
It can be seen that the performance of OntoProtein exceeds all other methods in all test settings, which is reasonable because the knowledge injected from Gene Ontology is beneﬁcial.
Further, we random sample a protein instance from the test dataset and analyze its attention weight of OntoProtein.
We conduct visualization analysis as shown in Figure 4 to compare the contacts among amino acids with the contact label matrix.
3. D ISCUSSION
Applying techniques from NLP to proteins opens new opportunities to extract information from proteins in a self-supervised, data-driven way.
Here we show for the ﬁrst time that injecting external knowledge from gene ontology can help to learn protein representation better, thus, boosting the downstream protein tasks.
However, the gains in our proposed OntoProtein compared to previous pre-trained models using large-scale corpus is still relatively small.
Note that the knowledge graph
ProteinKG25 can only cover a small subset of all proteins, thus, limiting the advancement.
We will continue to maintain the knowledge graph by adding new facts from Gene Ontology.
Besides, previous studies (Liu et al.
; Zhang et al.
(2021a)) indicate that not all external knowledge are beneﬁcial for downstream tasks, and it is necessary to investigate when and how to inject external knowledge into pre-trained models effectively.
Finally, our proposed approach can be viewed as jointly pre-training human language and protein (the language of life).
Our motivation is to crack the language of life’s code with gene knowledge injected protein pre-training.
Our work is but a small step in this direction.
R ELATED W ORK 4. P RE - TRAINED L ANGUAGE M ODELS
Up to now, various efforts have been devoted to exploring large-scale PTMs, either for NLP (Peters et al.
; Devlin et al.
(2019)), or for CV (Tan & Bansal (2019)).
Fine-tuning large-scale PTMs such as ELMo (Peters et al.
, GPT3 (Brown et al.
, BERT (Devlin et al.
XLNet (Yang et al.
UniLM (Dong et al.
for speciﬁc AI tasks instead of learning models from scratch has also become a consensus (Han et al.
Apart from the of large scale language models for natural language processing, there has been considerable interest in developing similar models for proteins (Xiao et al.
; Rives et al.
Rao et al.
(2021a) is the ﬁrst to study protein Transformer language models, demonstrating that information about residueresidue contacts can be recovered from the learned representations by linear projections supervised with protein structures.
Vig et al.
(2021) performs an extensive analysis of Transformer attention, identifying correspondences to biologically relevant features, and also ﬁnds that different layers of the model are responsible for learning different features.
Elnaggar et al.
(2020) proposes ProtTrans, which explores the limits of up-scaling language models trained on proteins as well as protein sequence databases and compares the effects of auto-regressive and auto-encoding pre-training upon the success of the subsequent supervised training.
Human-curated or domain-speciﬁc knowledge is essential for downstream tasks, which is extensively studied such as Himmelstein & Baranzini (2015), Smaili et al.
(2018), Smaili et al.
(2019), Hao et al.
(2020), Ioannidis et al.
However these pre-training methods do not explicitly consider external knowledge like our proposed
OntoProtein.
4. K NOWLEDGE - ENHANCED L ANGUAGE M ODELS
Background knowledge has been considered as an indispensable part of language understanding ((Zhang et al., 2021a;
Deng et al., 2021; Li et al., 2021; Zhang et al., 2019a; Yu et al., 2020; Zhu et al., 2021; Zhang et al., 2021b;
Chen et al., 2021; Zhang et al., 2022b; Silvestri et al., 2021;
Zhang et al., 2021c; Yao et al., 2022; Zhang et al., 2022a)), which has inspired knowledge-enhanced models including ERNIE (Tsinghua)
(Zhang et al.
(2019b))
, ERNIE (Baidu) (Sun et al.
(2019)), KnowBERT (Peters et al.
, WKLM (Xiong et al.
, LUKE (Yamada et al.
, KEPLER (Wang et al.
(2021b))
, K-BERT (Liu et al.
K-Adaptor (Wang et al.
(2021a)), and CoLAKE (Sun et al.
ERNIE (Zhang et al.
(2019b)) injects relational knowledge into the pre-trained model
BERT, which aligns entities from Wikipedia to facts in WikiData.
KEPLER (Wang et al.
(2021b)) jointly optimizes knowledge embedding and pre-trained language representation (KEPLER), which can not only better integrate factual knowledge into PLMs but also effectively learn KE through the abundant information in the text.
Inspired by these works, we propose OntoProtein that integrates external knowledge graphs into protein pre-training.
To the best of our knowledge, we are the ﬁrst to inject gene ontology knowledge into protein language models.
C ONCLUSION AND F UTURE W ORK
In this paper, we take the ﬁrst step to integrating external factual knowledge from gene ontology into protein language models.
We present protein pretraining with gene ontology embedding (OntoProtein), which is the ﬁrst general framework to integrate external knowledge graphs into protein pre-training.
Experimental results on widespread protein tasks demonstrate that efﬁcient knowledge injection helps understand and uncover the grammar of life.
Besides, OntoProtein is compatible with the model parameters of lots of pre-trained protein language models, which means that users can directly adopt the available pre-trained parameters on OntoProtein without modifying the architecture.
These positive results point to future work in (1) improving OntoProtein by injecting more informative knowledge with gene ontology selection; (2) extending this approach to sequence generating tasks for protein design.
A CKNOWLEDGMENTS
We want to express gratitude to the anonymous reviewers for their hard work and kind comments.
This work is funded by NSFCU19B2027/NSFC91846204, National Key R&D Program of China (Funding No.SQ2018YFC000004), Zhejiang Provincial Natural Science Foundation of China (No.
LGG22F030011), Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduction Programme.
R EPRODUCIBILITY S TATEMENT
Our code and datasets are all available in the https://github.com/zjunlp/
OntoProtein for reproducibility.
Hyper-parameters are provided in the Appendix A.3.
A A PPENDIX A. C ONSTRUCTION OF P ROTEIN KG
To incorporate Gene Ontology knowledge into language models and train OntoProtein, we construct
ProteinKG25, a large-scale KG dataset with aligned descriptions and protein sequences respectively to GO terms and proteins entities.
We design two evaluation schemes, the transductive and the inductive settings, which simulate two scenarios of gene annotation in reality.
We use the latest Gene Ontology and Gene Annotations released in April 2020.
Gene Ontology depicts the relation between GO terms using GO-GO triplet, and Gene Annotation depicts the relations between protein and GO term using Protein-GO triplet.
Due to this connectivity, we combine the two structures into a uniﬁed knowledge graph.
For each GO term in Gene Ontology, we align it to its corresponding name and description and concatenate them by a colon as an entire description.
For each protein in Gene annotation, we align it to the Swiss-Prot and extract its corresponding sequence as its description.
The ﬁnal KG contains 4,990,097 triplets (4,879,951 Protein-GO triplets and 110,146 GO-GO triplets), 612,483 entities (565,254 proteins and 47,229 GO terms) andrelations.
Due to the tree-like hierarchical structure of Gene Ontology, we deﬁne the depth of GO terms using the shortest path from current GO terms to the root node of GO terms.
The distribution of ProteinGO triplets with respect to the depth of GO term is shown at the bottom of Figure 3.
Usually, the deeper a GO term is located, the more concrete deﬁnition the GO term has, e.g., the small molecule biosynthetic process is the child node of the biosynthetic process.
We notice that the number of gene annotations involved with leaf GO terms is a relatively small percentage in all three types of ontologies.
We think there are two possibilities: (1) The complexity of annotations of some concrete GO terms, e.g., the identiﬁcation of whether a protein is involved in the hexose biosynthetic process.
The relatively small number of proteins in nature involved with some speciﬁc GO terms is intrinsic to these GO terms.
i s _ a p a r t _ o f r e g u l a t e s h a s _ p a r t n e g a t
i v e l y _
r
e g
u l a t e s p
o s i
t i v e l y _
r e g
u l a t e s o cc u r
s
i n h a pp e n s _ d u r i n g e n d s _ d u r i n g e n a b
l
e
s i n v o l v e d _ i n a c t s _ u p s t r e a m _ o f _
o r _ w i
t h
i n c o l o c a li z e s _ w i t
h c o n t r i b u t e s _ t o
N O T | e n a b l e s
N O T | i n v o l v e d _ i n
N O T | p a r t _
o f a c t s _ u p s t r e a m _ o f
N O T | c o l o c a li z e s _ w i
t h
N O T | a c t s _ u p s t r e a m _ o f _ o r _ w i
t h i n a c t s _ u p s t r e a m _ o f _ p o s i t i v e _ e ff e c t a c t s _ u p s t r e a m _ o f _ o r _ w i
t
h i n _ p o s i t i v e _ e ff e c t
i s _ a c t i v e _
i n a c t s _ u p s t r e a m _ o f _ n e g a t i v e _ e ff
e c t a c t s _ u p s t
r e a m _ o f _
o r _ w i
t h i n _ n e g a t i v e _ e ff e c t
N O T | c
o
n t r i b u t e s _ t o
N O T | a c t s _ u p s t r e a m _ o f _ o r _ w i
t h i n _ n e g a t i v e _ e ff e c t l o c a t e d _ i n
N O T | l o c a t e d _ i n
N O T | i s _ a c t i v e _ i n
N O T | a c t s _ u p s t r e a m _ o f e n a b l e
s _ h y d
r
o
l
a s e _ a c t i v i
t y i n v o l v e d _ i n _ s i g n a l _ t r a n s d u c t
i
o
n
i n v o l v e d _ i n _ m e t a b o li c _ p r o c e ss e n a b l e s _ m e t
a l _ i o n _ b i n d i n g e n a b l e
s _ c a t a l y t i c
_ a c t i v i t
y e n a b l e s _ nu c l e o t i
d
e _ b i n d i n g p a r t _ o f _ m e m b r a n e p a r t _ o f _
i n t e g r a l _ c o m p o n e n t _ o f _ m e m b r a n e i n v o l v e d _ i n _ c e ll u l a r _ r e
s p o n s e _ t o _ D N
A _ d a m a g e _ s t i m u l u s p a r t _ o f _
m i t
o c h o n d
r
i o n p a r t _ o f _ e x t r a c e ll u l a r _ r e
g
i o n e n a b l e s _ R
N A _ b i n d i n g p a r t _ o f _
c y t
o p l a s m e n a b l e s _ D
N A _ b i n d i n g p a r t _ o f _
nu c l e u s e n a b l e s _ t r a n s f e r a s e _ a c t i v i
t y i n v o l v e d _ i n _ p r o t e o l y
s i s i n v o l v e d
_ i n _ m e t h y l a t i o n i n v o l v e d _ i n _ li p i
d _ m e t a b o li c
_ p r
o c e ss p a r t _ o f _ c y t
o s o l i n v o l v e d _ i n _ c a r b o h y d r a t e _ m e t a b o
li c
_ p r
o c e ss i n v o l v e d _ i n _ c e ll u l a r _ a m i n o _
a c i d _ b i
o s
y n t
h e t i c _ p r
o c e ss i n v o l v e d _ i n _ t r a n s m e m b r a n e _ t r a n s p o r t i n v o l v e d _ i n _ r e g u l a t i o n _
o f _ t r a n s c
r
i p t
i o n , _ D N A - t e m p l a t e d i n v o l v e d _ i n _ i o n _ t r a n s p o r t p a r t _ o f _
p l a s t i d i n v o l v e d _ i n _ p h
o s p
h
o r y l a t i o n i n v o l v e d _ i n _ c e ll _
c y c l e i n v o l v e d _ i n _ c e ll _ d
i v i s i o n i n v o l v e d _ i n _ p r o t e i n _ t r a n s p o r t i n v o l v e d _ i n _ t r a n s l a t i o n
N O T | i n v o l v e d _ i n _ t R N
A _ p r
o c e ss i n g e n a b
l e s _ s t r u c t
u
r a l _ c o n s t i t u e n t _
o f _ r i
b
o s o m
e p a r t _ o f _ r i
b
o s o m e 0.
2.00 1e i s _ a p a r t _ o f r e g u l a t e s h a s _ p a r t n e g a t
i v e l y _
r
e g
u l a t e s p
o s i
t i v e l y _
r e g
u l a t e s o cc u r
s
i n h a pp e n s _ d u r i n g e n d s _ d u r i n g e n a b
l
e
s i n v o l v e d _ i n a c t s _ u p s t r e a m _ o f _
o r _ w i
t h
i n c o l o c a li z e s _ w i t
h c o n t r i b u t e s _ t o
N O T | e n a b l e s
N O T | i n v o l v e d _ i n
N O T | p a r t _
o f a c t s _ u p s t r e a m _ o f
N O T | c o l o c a li z e s _ w i
t h
N O T | a c t s _ u p s t r e a m _ o f _ o r _ w i
t h i n a c t s _ u p s t r e a m _ o f _ p o s i t i v e _ e ff e c t a c t s _ u p s t r e a m _ o f _ o r _ w i
t
h i n _ p o s i t i v e _ e ff e c t
i s _ a c t i v e _
i n a c t s _ u p s t r e a m _ o f _ n e g a t i v e _ e ff
e c t a c t s _ u p s t
r e a m _ o f _
o r _ w i
t h i n _ n e g a t i v e _ e ff e c t
N O T | c
o
n t r i b u t e s _ t o
N O T | a c t s _ u p s t r e a m _ o f _ o r _ w i
t h i n _ n e g a t i v e _ e ff e c t l o c a t e d _ i n
N O T | l o c a t e d _ i n
N O T | i s _ a c t i v e _ i n
N O T | a c t s _ u p s t r e a m _ o f e n a b l e
s _ h y d
r
o
l
a s e _ a c t i v i
t y i n v o l v e d _ i n _ s i g n a l _ t r a n s d u c t
i
o
n
i n v o l v e d _ i n _ m e t a b o li c _ p r o c e ss e n a b l e s _ m e t
a l _ i o n _ b i n d i n g e n a b l e
s _ c a t a l y t i c
_ a c t i v i t
y e n a b l e s _ nu c l e o t i
d
e _ b i n d i n g p a r t _ o f _ m e m b r a n e p a r t _ o f _
i n t e g r a l _ c o m p o n e n t _ o f _ m e m b r a n e i n v o l v e d _ i n _ c e ll u l a r _ r e
s p o n s e _ t o _ D N
A _ d a m a g e _ s t i m u l u s p a r t _ o f _
m i t
o c h o n d
r
i o n p a r t _ o f _ e x t r a c e ll u l a r _ r e
g
i o n e n a b l e s _ R
N A _ b i n d i n g p a r t _ o f _
c y t
o p l a s m e n a b l e s _ D
N A _ b i n d i n g p a r t _ o f _
nu c l e u s e n a b l e s _ t r a n s f e r a s e _ a c t i v i
t y i n v o l v e d _ i n _ p r o t e o l y
s i s i n v o l v e d
_ i n _ m e t h y l a t i o n i n v o l v e d _ i n _ li p i
d _ m e t a b o li c
_ p r
o c e ss p a r t _ o f _ c y t
o s o l i n v o l v e d _ i n _ c a r b o h y d r a t e _ m e t a b o
li c
_ p r
o c e ss i n v o l v e d _ i n _ c e ll u l a r _ a m i n o _
a c i d _ b i
o s
y n t
h e t i c _ p r
o c e ss i n v o l v e d _ i n _ t r a n s m e m b r a n e _ t r a n s p o r t i n v o l v e d _ i n _ r e g u l a t i o n _
o f _ t r a n s c
r
i p t
i o n , _ D N A - t e m p l a t e d i n v o l v e d _ i n _ i o n _ t r a n s p o r t p a r t _ o f _
p l a s t i d i n v o l v e d _ i n _ p h
o s p
h
o r y l a t i o n i n v o l v e d _ i n _ c e ll _
c y c l e i n v o l v e d _ i n _ c e ll _ d
i v i s i o n i n v o l v e d _ i n _ p r o t e i n _ t r a n s p o r t i n v o l v e d _ i n _ t r a n s l a t i o n
N O T | i n v o l v e d _ i n _ t R N
A _ p r
o c e ss i n g e n a b
l e s _ s t r u c t
u
r a l _ c o n s t i t u e n t _
o f _ r i
b
o s o m
e p a r t _ o f _ r i
b
o s o m e
[Figure]
We further pre-process the ProteinKG25 dataset as follows.
We observe that the relations of ProteinGo have long-tailed distribution, and mostly focused on involved in , part of and enables .
Such data distribution will seriously affect the protein feature embedding during pre-training, so we preprocess our dataset to add more precise and ﬁne-grained relations.
Figure 5 illustrates the relation distribution of the original model and the distribution after being pre-poccessed.
We search for
GO terms whose frequency of occurrence in ProteinKG25 is the top 10 in MF and CC, top 20 in
BP, then we form a new type of Protein 2 GO relation by their corresponding relationship such as parts of cytoplasm .
A. D OWNSTREAM T ASK D EFINITION
We list the detailed deﬁnition of downstream tasks and its corresponding or similar tasks in nature language processing.
time training set validation set test set
Apr,
Aug,
Apr,
[Table]
• Secondary Structure Prediction is a token-level task and similar to NER (Name Entity Recognition).
Each token (amino acid) x
i is mapped to a label y i ∈ { Helix, Strand, Other } .
• Contact Prediction is a token-level matching task.
Each token (amino acid) pair
x
i , x j of sequence (protein) x is mapped to a label y ij ∈ { 0 , 1 } .
• Remote Homology Detection is a sequence-level classiﬁcation task.
Each input sequence (protein) x is mapped to a label y ∈ { 1 , ..., 1195 } which represents different possible protein folds.
• Fluorescence Landscape Prediction and Stability Landscape Prediction are regression tasks where each sequence (protein) x is mapped to a label y ∈ R .
• Protein Protein Interface is a sequence-level matching task.
Each sequence (protein) pair x i , x j is mapped to a label y ij ∈ { 0 , 1 } .
• Protein Function Prediction is a sequence-level classiﬁcation task or a knowledge graph completion task to prediction link of a protein to the Gene Ontology.
A. E XPERIMENTAL S ETTINGS
This section details the training procedures and hyperparameters for each of the datasets.
We utilize
Pytorch (Paszke et al.
to conduct experiments with Nvidia V100 GPUs.
In pre-training of
OntoProtein, similar to Elnaggar et al.
(2020), we use the same training protocol such as optimizer, learning rate schedule on BERT model.
We set γ to 12.0 and the number of negative sampling to 128 in Equation 1.
A. D ATAFLOW OF O NTO P ROTEIN
We use batches (protein-seq, protein-go, go-go) to jointly train the model, which is shown in Figure 7.
[Table]
TERM
DESCRIPTION
GO annotation
A statement about the function of a particular gene
GO term
A standard vocabulary term for biological function annotation
GO statement
A speciﬁc deﬁnition of GO term