Identiﬁcation of Enzymatic Active Sites with Unsupervised Language Modeling
Abstract
The ﬁrst decade of genome sequencing saw a surge in the characterization of proteins with unknown functionality.
Even still, more than 20% of proteins in well-studied model animals have yet to be identiﬁed, making the discovery of their active site one of biology’s greatest puzzle.
Herein, we apply a Transformer architecture to a language representation of bio-catalyzed chemical reactions to learn the signal at the base of the substrate-active site atomic interactions.
The language representation comprises a reaction simpliﬁed molecular-input line-entry system (SMILES) for substrate and products, complemented with amino acid (AA) sequence information for the enzyme.
We demonstrate that by creating a custom tokenizer and a score based on attention values, we can capture the substrateactive site interaction signal and utilize it to determine the active site position in unknown protein sequences, unraveling complicated 3D interactions using just 1D representations.
This approach exhibits remarkable results and can recover, with no supervision, 31.51% of the active site when considering co-crystallized substrate-enzyme structures as a ground-truth, vastly outperforming approaches based on sequence similarities only.
Our ﬁndings are further corroborated by docking simulations on the 3D structure of few enzymes.
This work conﬁrms the unprecedented impact of natural language processing and more speciﬁcally of the
Transformer architecture on domain-speciﬁc languages, paving the way to effective solutions for protein functional characterization and bio-catalysis engineering.
Introduction
The number of anticipated genes has exploded as high-throughput sequencing technologies have improved dramatically.
However, a considerable percentage of these newly found genes codiﬁes proteins with no known function, which is a fundamental hindrance in our understanding of molecular life.
The activity of a protein is directly related to the structure of the active site, a spatial region that, under external pressure, evolved its amino acids (AA) sequence to interact with speciﬁc molecules. The identiﬁcation of active site residues and the characterization of the corresponding protein function remains a difﬁcult task. While wet-lab experiments give the most accurate protein annotation, their limited throughput has increased the relevance of computational approaches.
In fact, automated functional annotation is essential for understanding genomic data, which is essential to generate hypotheses about how proteins ﬁt into processes and pathways. The last years have seen steady improvements in the ﬁeld.
Many top-performing algorithms frequently incorporate machine learning, and while a few are generally superior, no single method outperforms all the others [Jiang et al., 2016].
The most successful approaches for biological function annotation of proteins usually rely on 3D structural models [Yousaf et al., 2021, Sankararaman et al., 2010, Jiménez et al., 2017, Kozlovskii and Popov, 2021, Yang et al., 2013, Kozlovskii and Popov, 2020, Wass et al., 2010] to transfer functional insights, such as 3D features of protein-ligand binding sites, to the unknown proteins from structural homologies. More advanced schemes include sequence and protein-protein interaction network information to enhance the accuracy and coverage of the structure-based function predictions [Zhang et al., 2017]. There have also been a few efforts to identify active sites solely based on sequence similarity, such as PFAM [Mistry et al., 2020] and PSI-BLAST [Altschul et al., 1997].
In the last two years, BERT [Devlin et al., 2018], and more in general models pretrained via masked language modeling, have become the de-facto standard for different language understanding problems [Vig, 2019, Hoover et al., 2019, Rogers et al., 2020].
Their advent did not inﬂuence only the natural language domain but had an impact in multiple disciplines ranging from biology to chemistry, where string-based representations of proteins and molecules deﬁne a proteomic language [Rives et al., 2021, Elnaggar et al., 2020, Vig et al., 2020], respectively chemical language [Wang et al., 2019, Kim et al., 2021], that can be used to interpret and understand physical phenomena.
Unsupervised language models enabled the prediction of mutational effect and secondary structure, improving long-range contact prediction [Rives et al., 2021, Rao et al., 2019, Vig et al., 2020], the targeting of binding sites [Vig et al., 2020], the capture of important biophysical properties governing protein shape [Vig et al., 2020, Elnaggar et al., 2020].
For small molecules, Schwaller et al.
[2021] showed how language models trained via masked language modeling (MLM) on organic chemical reaction data, represented using SMILES [Weininger, 1988], can leverage attention to efﬁciently and accurately map atoms between precursors and products with no supervision.
Here, we approach the problem from a multi-modal angle to capture the signal between the active site amino acids and the atoms of the interacting molecules to identify a protein’s active site.
We extend the work of Schwaller et al.
[2021] by developing a new method that employs a publicly available collection of enzymatic reactions.
We use SMILES for modeling each interacting molecule combined with the AA sequence for the enzyme’s fundamental structure.
This approach can recover 31.51% of the active sites when considering co-crystallized substrate-enzyme structures as ground-truth with no supervision, largely outperforming current state-of-the-art methods relying only on sequence similarities
[Figure]
Methods
[Figure]
The trained model is used in inference to deﬁne a score, based on the attention values computed on the reaction SMILES provided as input, which allows the prediction of the active site (bold-face AAs) of the enzyme bio-catalyzing the reaction with no supervision or structural information (right).
We repurposed BERT [Devlin et al., 2018] by jointly training it with Masked Language Modeling (MLM) and a n-gram Masked Language Modeling [Xiao et al., 2020] (n-gram MLM, with n = 3 ), leveraging Multi-task Transfer Learning (MTL)
[Pesciullesi et al., 2020] on a combination of reaction
SMILES representing organic reactions [Lowe, 2012] (weight assigned 0.1) and bio-catalyzed reactions [Probst et al., 2021] (weight assigned 0.9).
For enzymatic reactions, each example consists of a reaction SMILES complemented with the AA sequence representation of the enzyme of interest (see Figure 1 for a depiction).
As we train the model via MLM and n-gram MLM, we sparsely mask the reactants and the products and densely mask the enzyme sequence.
To reduce the length of the resulting reactions [Filipavicius et al., 2020], we built a sub-word vocabulary for AA sequences using
Byte Pair Encoding (BPE)
[Gage, 1994] learned on UniProt [Consortium, 2020].
The prediction of the active regions of proteins is unsupervised and entirely based on the analysis of the attention values computed by the pretrained language model after encoding a reaction.
If we label S ∈ R l × d ( l = r + m + p ) as the embedding of a given reaction and r , m , and p refer to the length of the reactants, the enzyme, and the products, respectively, a forward pass of S through the model yields a sequence S (cid:48) with the same dimension as S .
Each encoder block computes the attention matrix A ∈ R l × l of the sequence S provided as input [Vaswani et al., 2017].
We construct a matrix P ∈ R r × m by summing two sub-matrices of A , describing the link between reactants and enzymes: P =
A [1 : r, 1 : m ]+ A
[ r +1 : r {++
+} + m, 1 : r ] T .
We use the matrix P as shown in the Algorithmto predict the active regions via a consensus scheme where each reactant’s atom has k votes to choose its best-bound enzyme’s token.
The selected enzyme’s tokens are uniquely gathered in a set and are considered the protein’s active region.
Hereinafter, the method will be referred to as RXNAAMapper.
1: procedure RXNAAMapper ( P ∈ R r × m , k ) 2: active _ site ← set () 3: for i in 1..r do 4: line ← P [ i ] 5: for j in argmax( line , k ) do 6: active _ site.
add ( j ) 7: return active _ site
Evaluation
We use a set of 5K co-crystallized ligand-protein pairs from the Protein–Ligand Interaction Proﬁler (PLIP)
[Salentin et al., 2015] as ground-truth, to perform a two-fold evaluation: (1) a sequence-based assessment benchmarking RXNAAMapper against a statistical baseline (Random Model), a pretrained
BERT model on natural language (BERT-base) and the alignments retrieved from Pfam [Mistry et al., 2020]; and (2) a structural validation with protein-ligand binding energies computed with docking. We used PFAM for a fair assessment with existing methods using sequence information only.
For the sequence-based evaluation, we use an overlap score between the prediction and the ground-truth, as well as the false positive rate.
The overlap score is deﬁned considering the active site as a set of non-overlapping segments in a sequence.
If S with | S | = n is a sequence of amino acid residues, the active region
A s of S is deﬁned as A s =
{ ( a i , b i ) } mi , where a i and b
i are the index boundaries of the segment i .
The overlap score ( OS ( A, A s ) ) between the predicted active region
A = { ( a pi , b pi ) } ni and the ground-truth A
s = { ( a si , b si ) } mi is deﬁned as:
[Equation]
Besides the overlap score, the false positive rate ( FPR ) of the predictions is deﬁned as:
[Equation]
For the structural assessment, on a set of protein-ligand active sites predictions, we evaluated the binding energy computed with Autodock Vina [Eberhardt et al., 2021, Trott and Olson, 2010] considering predicted active sites and the ground-truth from PLIP.
Experiments and Results
In our experiments, we train models using a dataset of organic reactions from USPTO [Lowe, 2012] ( ∼ 1M, splits as reported in Schwaller et al.
combined with a dataset of bio-catalyzed reactions from ECREACT [Probst et al., 2021].
ECREACT records were further processed by using
UniProt to replace and augment Enzyme Commission (EC) numbers with protein sequences from the corresponding EC classes.
The resulting dataset has been further split into a training set ( ∼ 7M reactions) and a validation set ( ∼ 4K reactions).
BERT has been trained with 4,094 as the batch size (leveraging gradient accumulation) and LAMB [You et al., 2020] as the optimization algorithm with 20,928 optimization steps.
Table 1 reports the performance of the different approaches on the PLIP ground-truth.
Notably, RXNAAMapper performs consistently better than other methods, even though the product information has been completely omitted, given the nature of the dataset.
[Table] Figure 2a shows an exemplar comparison of Pfam-based and RXNAAMapper predictions overlapped with the PLIP ground-truth for a ligand-protein pair.
Notably, RXNAAMapper controls the false positives better than Pfam alignments and matches the active site reported in PLIP. [Figure]
For a subset of PLIP’s protein-ligand pairs, we used Autodock Vina to compute the protein-ligand binding energy for the RXNAAMapper’s predicted active sites and for the corresponding experimental ones provided by PLIP.
We computed the binding energies by averaging the 3D coordinates of the active sites atoms and setting the box side length to 50 Å. In Figure 2b, we show the binding energies for the two active sites: RXNAAMapper and PLIP.
We can appreciate how the binding energy calculations performed on the active site predicted by RXNAAMapper accurately match the one using the PLIP experimental information.
Discussion
The unsupervised detection of active sites using only protein sequence information is an important step to identify the function of uncharacterized proteins.
We presented RXNAAMapper, a technology that uses pretrained language models on text-based molecule representations to identify active sites in long amino acid sequences.
Compared with protein-ligand interactions from PLIP, our approach outperforms other sequence-based baselines reconstructing >30% of proteins’ active regions while better controlling false positives.
Furthermore, we have been able to validate the RXNAAMapper active sites predictions using docking studies by comparing the binding energy conﬁgurations with the experimental references.
These results show how large language models can inherently capture 3D structural information and reaction mechanisms from 1D representations, thus, opening up novel avenues for their consistent application in enzyme engineering.