O N P RE - TRAINED L ANGUAGE M ODELS FOR A NTIBODY
Anonymous authors
Paper under double-blind review
A BSTRACT
Antibodies are vital proteins offering robust protection for the human body from pathogens.
The development of general protein and antibody-specific pre-trained language models both facilitate antibody prediction tasks.
However, few studies comprehensively explore the representation capability of distinct pre-trained language models on different antibody problems.
Here, to investigate the problem, we aim to answer the following key questions: (1) How do pre-trained language models perform in antibody tasks with different specificity?
(2) How many benefits will the model gain if we introduce the specific biological mechanism to the pre-training process?
(3) Do the learned antibody pre-trained representations make sense in real-world antibody problems, like drug discovery and immune process understanding?
Previously, no benchmark available largely hindered the study to answer these questions.
To facilitate the investigation, we provide an A n T ibody
U nderstanding E valuation ( ATUE ) benchmark.
We comprehensively evaluate the performance of protein pre-trained language models by empirical study along with conclusions and new insights.
I NTRODUCTION
Antibodies are special type of proteins extensively used as diagnostic and therapeutic tools against diverse diseases, such as SARS-CoV-2 (Zhu et al., 2022).
Deciphering the information stored in antibody sequences is highly important with regard to real-world therapeutic antibody development and immune understanding (Greiff et al., 2020;
Lu et al., 2018; Yermanos et al., 2018).
Fortunately, the recent progress of general P re-trained P rotein L anguage M odels (PPLM) and specific P re-trained
A ntibody L anguage M odels (PALM) provide new prospects for antibody-related tasks.
For example, many studies have shown that the representations learned by PPLMs are promising to transfer to antibody tasks (Kim et al., 2021; Zaslavsky et al., 2022).
For PALMs, they have been shown to improve model performance in antibody paratope predictions (Leem et al., 2022).
[Figure]
Despite the current success, few studies comprehensively investigate the representation capability of different pre-trained language models (e.g. general PPLM and specific PALM) on distinct antibody tasks, which limits our ability to design better architectures that can help antibody discovery and modification.
For example, in
Figure 1, we compare the performance of the pre-trained protein language model ESM (Rives et al., 2021), the pre-trained antibody language model AntiBERT (Leem et al., 2021), and the model trained from scratch on three antibody tasks.
These three tasks range from low to high in terms of antibody specificity.
Here, specificity refers to the antibody’s evolution, for example, the antibody evolves to obtain the ability to bind antigen (The definition is discussed in detail in
Section 3.1 and Appendix A.1).
It can be observed that although ESM performs well in task that has low relevance with antibodies, the performance dramatically degrades in tasks of higher relevance.
Besides, AntiBERT does not show clear advantages over the non- pretrain model in the high- specificity task
The results point out the weaknesses of current pre- training language models for antibody studies: Direct adaptation of general
PPLM representations may damage the performance; Pre-training strategies for PALMs cannot fit the specific antibody biological function well.
All of these call for a comprehensive guideline of model designing for different antibody tasks.
Mainly, we focus on answering the following questions:
(I) How do pre-trained language models perform in antibody tasks with different specificity?
Addressing of the question is mainly hindered by two challenges: the lack of a reliable antibodyspecific benchmark for performance evaluation and a comprehensive study of current PPLMs and
PALMs.
(II) How many additional benefits will the model gain if the biological mechanism is introduced to the pre-training process?
Antibody-specific evolution has been employed by many computational biology studies and shows promising results in antibody-related tasks, such as disease diagnosis and therapeutic antibody development (Yermanos et al., 2018;
Miho et al., 2019).
Then, it is interesting to know whether antibody representation learning can benefit from the incorporation of antibody-specific evolution information.
(III) Does the learned antibody pre-trained representations make sense in real-world antibody problems, like drug discovery and immune process understanding?
Since antibody is so important that is extensively used for drug development in the real world, it is interesting to know whether pre-trained representation can indeed help biologists to understand antibody functions or discover drugs.
To investigate these questions, we first propose antibody study benchmark A n T ibody U nderstanding
E valuation ( ATUE ).
This is the first antibody benchmark with four real-world supervised tasks covering therapeutic antibody engineering, B cell analysis, and antibody discovery.
To evaluate models on different aspects of antibody biological functions, these tasks are designed to have specificity ranging from low to high.
Based on ATUE , we perform comprehensive empirical studies to investigate the representation ability of distinct pre-trained language models.
We also explore the influence of involving specific biological mechanisms in antibody pre-training by introducing two simple objectives to tailor masked language modeling for evolution: (1) Ancestor germline prediction guides the model to discriminate the evolutionary relationship between antibody and ancestral sequences.
(2) Mutation position prediction
mimics hypermutation during the evolution.
The method is used as an analytic tool to investigate the representation ability of antibody evolutiontailored language model.
Finally, we take a close look at the SARS-CoV-2 antibody discovery to investigate the pre-trained representation under a real-world scenario.
Our contributions can be divided three-fold.
• For facilitating antibody application studies, we develop the first comprehensive antibody benchmark to benefit the community.
We also introduce two objectives tailored for antibody evolution.
• For providing guidelines for a better antibody representation, we have the following key observations: (i) PPLMs perform well on antibody tasks that have a high relationship with structure, such as the binding with antigen.
However, they perform worst in tasks with high antibody specificity, indicating representation benefiting protein structure prediction is harmful to antibody-specific tasks.
(ii)
In most cases, PALMs perform as well as or even better than PPLMs with less pretraining data.
(iii) PALMs can be improved by incorporating the evolution process.
However, the evolution information from MSAs does not always benefit the antibody tasks.
(iv) The introduction of two antibody biological mechanisms facilitates PALMs with more antibody-specific features and improves model performance in the task with high antibody specificity.
This is the first attempt showing how antibody specific evolutionary information can be incorporated in pre-training language model.
• For accelerating real-world antibody discovery, we identified 11 potential SARS-CoV-2 binders whose sequences are highly identical to existing therapeutic antibodies binding with the virus.
R ELATED W ORK
Our work focuses on researching the effectiveness of protein and antibody pre-trained language models for antibody-specific tasks.
Below we review the representative existing methods.
We list the details in Table 1.
Pretrained Protein Language Models (PPLMs)
There is an increasing interest in exploring largescale language models using protein sequences (Rao et al., 2019; Madani et al., 2020; Meier et al.,
These models have been shown to achieve state-of-art capacity in predicting protein structure and function.
ProtTrans (Elnaggar et al., 2021) and ESM-1b (Rives et al., 2021) take individual protein sequences as input and adopt Transformer language models for pre-training, demonstrating that self-supervision is a promising paradigm for protein secondary structure, contact, homology predictions, and function prediction.
To extract evolutionary information from protein sequences, Rao et al.
(2021) proposed the MSA-transformer/MSA-1b model utilizing multiple sequence alignment (MSA) instead of a single query sequence as input.
This model is superior to ESM-1b for structure prediction, demonstrating evolution information can benefit protein representation learning.
Despite the progress in the field, few studies reported protein PLM transfer learning results on antibody tasks.
Pretrained Antibody Language Models (PALMs) Encouraged by the success of PLMs in protein representation learning, series work seeks to learn antibody representations based on sequences of antibodies.
(Leem et al., 2021;
Ruffolo et al., 2021;
Olsen et al., 2022b; Prihoda et al., 2022;
Li et al., 2022).
AntiBERTy (Ruffolo et al., 2021) proposed the first antibody-specific language model, exploring a Transformer trained on 558M natural antibody sequences in the OAS database.
The case study in the work showed the representation obtained from the PLM is useful for antibody sequence clustering into trajectories resembling affinity maturation.
Olsen et al.
(2022b) train two language models for antibodies.
A heavy chain version Ablang-H is trained on 14M sequences and a light chain version Ablang-L on 0.19M sequences.
The study reported transfer learning results on restoring missing residues of antibody sequences, which is a task similar to pre-training objectives.
AntiBERTa (Leem et al., 2021) train the antibody language model on OAS.
Although this paper claims finetuning AntiBERTa for paratope position prediction can achieve state-of-the-art performance, the experimental results lack standard deviations, making it unclear how significant the results obtained are.
Recently, Li et al.
(2022) proposed a antibody-specific language model and explored its performance in SARS-CoV-2 antigen binding, showing context-dependent representations of antibody sequences benefit binding prediction.
[Table]
F RAMEWORK
In this section
[Table]
we first give a brief introduction to the antibody and its specific evolution.
Then we propose the first antibody-specific benchmark ( ATUE ) composed of four tasks with different specificities.
Finally [Table] we implement several PPLMs and PALMs baselines and design an evolutionaware PALM to incorporate the biological mechanism into the pre-training process.
3. B ACKGROUND
Antibody Antibodies are vital proteins generated by the immune system to remove harmful foreign pathogens in the human body.
they can specifically bind to antigens on the pathogen and recognize it.
Antibodies are composed of two identical heavy chains and two identical light chains and form a large Y-shaped structure.
Two tips on it contain highly variable loops, called Complementarity
Determining Regions (CDR), which function for antigen binding
[Table]
Antibody Specific Evolution
Notably, the antibody evolution process is significantly different from that of proteins, providing a good opportunity for us to investigate the impact of general PPLMs on specific subdomains.
To perform its protective function, the antibody sequence undergoes evolution selection to search for optimal patterns that can specifically recognize pathogens (Honjo & Habu,
[Figure]
Deciphering the information stored in antibody sequences may benefit our understanding of disease and accelerate therapeutic antibody development (Greiff et al., 2020; Lu et al., 2018; Yermanos et al., 2018).
During evolution, the random recombination of V/D/J-gene segments provides the initial diversity for the ancestor sequence (germline).
Upon exposure to a pathogen, this sequence undergoes frequent sequence mutations to search for progeny antibody sequences with optimal binding specificity.
In other words, gene recombination provides millions of germlines in the human body, and the germlines further mutate into a huge number of progeny antibodies.
Thus, the ancestor relationship between an antibody and its corresponding germline as well as the mutation it undergoes together determine the unique biological functions.
In brief, the evolutionary relationships between antibodies arise to gain new functions such as antigen binding.
It is significantly different from that of proteins, which are to certain functions across different organisms.
We further illustrate this process in Figure 7 in Appendix A.1.
Unsupervised Antibody Corpus To obtain the evolutionary information of antibody sequences, we utilize Observed Antibody Space (OAS), a database containing more than 1.5 billion natural antibody sequences (Kovaltsuk et al., 2018;
Olsen et al., 2022a)
The antibody sequences in the database have been precisely annotated with evolutionary and structural information, including the paired germline and CDR3 for each antibody.
To pair the antibody with its germline used in the pretraining task, we used the annotated sequences provided in the OAS database.
Further information on data processing can be found in Appendix A.2.
3. A N T IBODY U NDERSTANDING E VALUATION (ATUE)
We provide four biologically relevant downstream prediction tasks to serve as antibody benchmarks, covering four major application aspects for antibodies in the real world: therapeutic antibody engineering, disease diagnostics, antibody discovery, and B cell maturation analysis.
Notably, the antibody evolution relatedness of these tasks ranges from low to high, offering scaled tasks with subdomain specificity for pre-trained language model evaluation.
Detailed information is listed in
Figure 2.
All data are publicly open and used under the right license.
For each task, we focus on the following aspects and leave the details in Appendix:
[Definition] The formal definition of the task and the understanding ability required.
[Impact] The importance of the task in the biological area.
Details in Appendix A.3.
[Dataset] The data source and size.
Details of data processing in Appendix A. [Specificity] Antibody’s specific evolution characteristics that is different from general proteins.
The definition of antibody specific evolution is shown in Appendix A.1.
Quantitative analysis the scale of antibody specificity for every task is shown in Appendix A.4.
We use several classification metrics to evaluate the performance.
Accuracy ( ACC ) calculates the ratio of correct predictions.
Matthews Correlation Coefficient ( MCC ) is the coefficient between true and predicted values.
F1 is the average weighted score of precision and recall.
AUC is the area under the ROC curve, which shows the performance at all classification thresholds.
Antigen Binding Prediction is a binary sequence classification task to determine whether the CDR region of the antibody can bind to the specific antigen.
[Impact] A better understanding of the binding affinity between antibody and antigen can accelerate the affinity optimization of therapeutic antibodies.
[Dataset] We collect the antigen binding data from (Mason et al., 2021) and follow the training/validation/test split of 15,128/3,242/3,242.
The paratope data is collected from (Liberis et al., 2018) with 1,662 CDR segments on 277 antibodies.
[Specificity] Low.
All the antibodies sequence in the dataset are derived from a single germline sequence indicating the task is not antibody-specific evolution-related.
Paratope Prediction is to identify binding positions on the antibody sequence, which is a sequence labeling task to predict a 0/1 label for each residue of CDR fragments.
[Impact] The exploration of paratope (binding positions between antibody and antigen) can help to understand the binding mechanisms of therapeutic antibodies.
[Dataset] The paratope data is collected from (Liberis et al., 2018) with 1,662 CDR segments on 277 antibodies.
[Specificity] This task is medium specificity related because only partial antibodies from the database are derived from evolution.
B Cell Maturation Analysis It is a 6-category classification task to distinguish the maturation stage of B cell antibody sequences.
Each sequence belongs to one of { immature, transitional, mature, plasmacytes, memory IgD+, memory IgD- }.
It requires the model to learn a representation sensitive to different maturation states.
[Impact] It benefits the understanding of the mechanism during immune evolution, which is a critical biological process in the immune system affecting the function and antigen specificity of antibodies (Ghraichy et al., 2021; Meffre et al., 2000).
[Dataset] We collect 88,094 sequences from (Mroczek et al., 2014).
[Specificity] High.
Antibody evolution is highly coupled with B cell maturation.
(Meffre et al., 2000)
Antibody Discovery
The task is a binary sequence classification task to distinguish which antibody is directly responsible for SARS-CoV-2 binding.
The task is highly challenging from two aspects: (1)
Less than 1% of antibodies from SARS-CoV-2 patients are directly responsible for virus binding.
It is hard to get a reliable sequence-level classifier using unreliable and noisy individual-level labels.
[Impact] Antibody discovery from B cell repertoire has been widely recognized as a important approach to accelerate antibody discovery for diverse antigens ( Weiner, Pedrioli
& Oxenius , and achieved great success for
SARS-CoV -2 antibody discovery (Kovaltsuk et al., 2018; Cao et al., 2020; Shiakolas et al., 2022)
[Dataset] We collected antibody sequences from 133 SARS-CoV-2 patients and 87 health persons from OAS and followed the processing pipeline of (Kim et al., 2021).
Inspired Zaslavsky et al.
, we match the high-ranked sequences with the sequences in the CoVAbDab
(Raybould et al., 2021) database, which have been proved to bind SARS-CoV-2 using wet-lab experiments.
[Specificity] High.
It is widely reported antibodies derived from the same disease such as SARS-CoV-2 share strong convergent germline signals (Galson et al., 2020).
3. E XPERIMENT S ETUP
Based on the antibody benchmark ATUE , we evaluate the performance of current pertaining language models in different specificity tasks.
Furthermore, to investigate the benefit of introducing the biological mechanism, we incorporate evolution information as the extra pretraining objectives for
PALMs and propose EATLM .
The detailed description can be found in Appendix A.
Current Pre-trained language models Existing antibody and protein language models are summarized in Table 1.
Since the code and pre-training data of AntiBERTa are not released, we train a
BERT model named AntiBERT on the full OAS database following the same setting as the original study.
MSA-1b (Rao et al., 2021) takes protein-specific evolutionary sequences (Multiple Sequence
Alignment, MSA) as the input.
Because it is hard to align sequences between antibodies due to the diversity of CDR3, we take the germline and create pseudo-MSAs with depth 2.
We add a linear layer on top of the language models and finetune the whole model on the downstream tasks.
[Table]
Evolution-aware antibody pretraining method To incorporate the biological mechanism into the pre-training
, we propose a model with evolution information: A ntibody E volu T ion-aware pretraining
L anguage M odel.
The antibody can be represented as A and the germline of the individual antibody can be represented as G .
Typically, PALMs are trained with basic masked language modeling (MLM).
Based on it, we design another two pre-training objectives to simulate the biological mechanism of antibody evolution.
The evolutionary relationship between the antibody and its germline includes two folds: (i) Whether the antibody and the germline have an evolutionary relationship .
(ii) How to mutate residues from the germline to get the specific antibody .
Two evolution-related objectives are introduced to solve the above questions: ancestor germline prediction (AGP) and mutation position prediction (MPP) .
For ancestor germline prediction, we substitute the paired germline
G with random germline G ′ in the batch via a probability p .
The model is made to distinguish the ancestor germline of the antibody by capturing the shared features.
To predict mutation position, for each token in the germline G , the objective is to predict a 0/1 label for each token to indicate whether this token has been mutated.
For the antibody sequence S , we mask the mutation position and predict these tokens.
The technical details are described in Appendix A.5.1.
Hyper-parameters We use the base Transformer architecture (Vaswani et al., 2017) with 12 layers,heads, and 768 hidden states.
For each task in ATUE , we finetune the model with supervised data.
We follow the standard split of Antigen Binding Prediction.
For other tasks that do not provide a standard split, we use a 10-fold cross-validation.
Since our pre-training model learns the representation of the antibody sequence, we expand the CDR fragment to the full antibody by searching the biological database for therapeutic antibody engineering tasks.
We also use the same Transformer architecture to train from scratch for each downstream task.
This model is indicated as non-pretrain since it is not pre-trained on a protein/antibody database.
Reproduction We conduct 10-fold validation on paratope prediction, B cell maturation analysis, and antibody discovery.
For antigen binding prediction, we conduct three repetitive experiments with different random seeds.
We report the average results and the standard derivation.
The benchmark and code will be released.
R ESULTS AND A NALYSIS 4. S UPERVISED ANTIBODY PREDICTION
Antigen binding Here, we evaluate the performance PLMs models for antibody binding and paratope prediction, which are less antibody specific.
The results are shown in Table
It is observed that PPLMs and PALMs achieve comparable performance on this task, indicating PALMs can learn similar general protein representation as PPLMs.
Among different PALMs, Ablang-H outperforms
Ablang-L and AntiBERT.
It indicates that separate training for heavy and light chain sequences is helpful for this task.
The introduction of AGP and MPP provides a little improvement over AUC and
F1 metrics.
Paratope prediction As the results shown in Table 2, for paratope prediction, both PPLMs and
PALMs can significantly boost the prediction accuracy over the model with pre-training.
However,
PALM doesn’t show much significant advantage over PPLMs.
EATLM achieves the best performance, especially on F1 and MCC, while other pertaining models have a high recall and a low precision.
It indicates that those models tend to predict more residues as the binding site.
However, with the introduction of mutation residue prediction, EATLM can focus on the mutated positions that are adapted to bind with antigen.
Among the two PPLMs, MSA-1b outperforms ESM-1 on F1 and MCC, which benefits from the structure information learning from MSAs.
[Figure]
For each p ij in i − th row and j -th column, it means the frequency for the model to predict the antibody in i category to j category.
The number is normalized by row.
B Cell Analysis
In this task, we examine the representation capacity of different pre-trained language models for distinguishing different B cell mature states during evolution.
The results are shown in
Table 2.
In general, we find PPLMs can hardly distinguish the minor differences between B cell sequences by showing compromising results.
Both the performance of ESM-1 and MSA-1b is significantly worse than randomly initialized models.
MSA-1b performs the worst in all pre-trained language models, indicating representation that can perform well in protein structure prediction will be harmful to antibodyspecific tasks.
Conversely, all PALMs show promising results for the task.
The reason may lie in that the general protein has little relationship with the special antibody mature process and can hardly capture this feature during the protein pretraining process.
EATLM significantly outperforms the other PALMs.
This is because by explicitly modeling the biological mechanism, our model can effectively capture the evolution feature and better distinguish between B cells at different stages of maturation.
We conduct further analysis to figure out whether our EATLM successfully captures sequence characteristics during the evolutionary process.
We explore the probabilities of predicting antibodies in class i to class j .
The results shown in 3 reveal EATLM can easily classify the immature B cell with an accuracy of 0.9.
It is consistent with the biological study that CDR3 sequence length in immature
B cells is significantly shorter than that of the other mature B cells (Ghraichy et al., 2021).
From the diagonal, we can figure out that our model tends to mistake the B cell sequences with their previous or post-evolutionary stage, consistent with the biological process.
[Figure]
Antibody Discovery
Here, we examine PPLMs and PALMS for their capacity in benefiting realworld problems and enabling the discovery of antigenspecific antibodies.
Following the methods in this study Zaslavsky et al.
(2022), we considered two steps for the SARS-CoV-2 specific antibody discovery.
First, we generate a sequence classifier to distinguish SARS-CoV-2 antibodies using noisy individuallevel labels.
Then, the high-ranked sequences are matched with the sequences with true binding sequences in the CoV-AbDab (Raybould et al., 2021) database.
We use the 90% sequence identity as the threshold to determine whether the antibody is similar to existing SARS binders in the CoV-AbDab database.
A high similarity indicates a high probability of having the same biological functionality as the existing binders.
The details can be found in Appendix A.7.
In Figure 4, we plot the cumulative sum of matched sequences number in the order of the predicted probability of the classifiers.
We can find that the sequences predicted with high probability by
PALMs match with the existing binders better than PPLMs.
It means that PALMs can figure out the potential binders more efficiently.
Besides, EATLM significantly outperforms other PALMs as the red line shows.
In the early stage, this method is the quickest way to find potential binders.
Then it loses to Ablang-H but finally overtakes again and converges.
It means that EATLM is the first one to figure out all potential binders in this dataset.
Furthermore, we list several potential binders found by EATLM in Table 3.
Without supervised labels,
EATLM gives a high probability of 2 SARS-CoV-2 existing binding antibodies.
Besides, EATLM proposes 9 potential sequences with high CDR-H3 sequence identity.
Further analysis of the results reveals that EATLM enables diverse-epitope antibody discovery and priority selection, demonstrating that EATLM benefits therapeutic antibody discovery.
The detailed explanations can be found in
Appendix A.7.
To examine whether the antibody sequences with 90% sequence identity can indeed bind the same target, we investigate the 3D structure of the true binding antibody.
Table 4 shows only one signle atom difference between the predicted binder and the existing binder, suggesting the predicted binders are highly possible to interact with SARS-CoV-2.
[Table]
G2A highlights the single atom difference in No.3, indicating the predicted binder is highly likely to bind the virus.
[Table]
4. H OW DOES EVOLUTION PRETRAINING TASK INFLUENCE THE REPRESENTATION ?
To understand why EATLM shows better performance on antibody tasks, we analyze the pre-trained representations to evaluate the effectiveness of the evolution-aware pretraining strategies from two aspects: (1) Does the pre-trained antibody representation reflect its ancestor relationship?
(2) Is the specificity of antibodies captured by the evolution objective?
Ancestor Gerlime Visualization We perform UMAP visualization analyses in Figure 5.
First, we observe that antibodies evolved from the same germline are nicely clustered together (Figure 5a and 5b), indicating the learned embedding is encoded with germline information.
Besides, sequences with similar scales of evolutionary distance tend to cluster together, and a clear gradation of evolutionary distance can be observed in Figure 5c and 5d.
The visualization provides a sanity check for the ability of EATLM to extract the sequence information of antibody.
Accuracy of Mutation Position Based on the specific evolution process described in Section 3.1, we can find the mutation during the evolution process bring specificity to the antibody.
Thus, we explore the model’s ability to predict mutated residue from the masked token, which can reflect the specificity feature the model captures.
We find that although AntiBERT can predict with an accuracy of 0.8889 on all positions, it fails on mutation positions with a 0.0311 accuracy.
In contrast,
EATLM achieves an accuracy of 0.443 on mutation position, which indicates that the model captures the specificity information.
Note that during the MPP training, we mask the mutation position on antibody sequences, which are different from its germline.
Thus, the model cannot get the mutated residue from the germline directly.
The only way is to learn the underlying mutation rules.
The full results are shown in Table 12a in Appendix.
[Figure]
4. K EY O BSERVATIONS
The performance of pre-trained language models highly depends on the task specificity.
For tasks with low antibody-specificity relatedness, PPLMs show comparable performance with PALMs, indicating that transferring the general protein representations from PPLMs is an effective and time-saving way in these tasks.
PALMs show their advantage and outperform PPLMs on medium specificity relatedness tasks, like paratope prediction.
For tasks with high specificity, it is observed the performance of PPLMs dramatically decreased, suggesting general pre-trained protein models are not sufficient in antibody-specific representation learning.
Interestingly, the performance of MSA-1b is 20% less than the model without pre-training.
This observation is consistent with the biological study that the mechanism of antibody evolution is significantly different from that of proteins.
Therefore, the incorporation of protein evolution information is not always good for antibody tasks, especially the tasks that require antibody evolution information.
[Figure]
Incorporation of biological evolution mechanism into
PALM generally benefits antibody prediction tasks.
Evolution-related training objectives help detect mutation positions on antibodies, which is a distinguishing feature from germline.
Interestingly, the performance increase of
EATLM compared with other PALMs is correlated with task specificity relatedness.
The ablation study showed the removal of the evolution-related pretraining objectives results in performance decreasing, confirming that evolutionrelated objectives benefit the prediction.
Future studies to have more in-depth research in this direction could be promising.
Antibody pre-trained representations are helpful for real-world drug discovery.
Using the language model, we predict the probability for each antibody of being a binder with SARS-CoV-2.
Without accurate sequence-level labels, we successfully identify 11 potential antibody binders.
C ONCLUSIONS AND L IMITATIONS
In this paper, we have presented comprehensive empirical studies to characterize the impact of pre-trained protein and antibody language models for antibody prediction tasks.
Four important antibody tasks from four biological categories with evolution relatedness ranging from high to low are provided ATUE to facilitate the antibody and ML field.
However, there are some limitations to this work.
First, for the ATUE , the task diversity is highly limited because of data scarcity.
With the data increase in this field, we expect we can update our benchmark with more disease diversity and data size in the future.
Also, we do not test any 3D structure information during antibody pre-training.
As a special subgroup of proteins, antibody structures provide much more information such as geometry than sequences.
In the future, recruiting structure information for antibody pre-training may be able to improve the results.
A A PPENDIX A. A NTIBODY S PECIFIC E VOLUTION
Antibodies composed of two identical heavy chains and two identical light chains, form a large
Y-shaped structure, where the two tips are responsible for pathogens binding.
Antibody evolution , described by sequence-sequence relationships between ancestor and progeny antibodies, reflects antibodies’ key antigen-binding function (Honjo & Habu, 1985).
During antibody evolution (Figure 7), the initial diversity is encoded into the ancestor sequence through randomly recombination of V-,
D- and J-gene segments.
Upon exposure to a pathogen, the sequence undergoes frequent sequence mutations to search for progeny sequences with optimal binding specificity.
Sequence evolution analysis has been employed by many computational biology studies and shows promising results in antibody-related tasks, such as disease diagnosis and therapeutic antibody development (Yermanos et al., 2018;
Miho et al., 2019).
Importantly, antibody evolution is significantly different from that of proteins.
Antibodies only contain hundreds of thousands ancestor sequences so-called germline.
To bind dozens of millions of diverse antigens, antibodies need to mutate from the ancestor sequences to gain new functions (Figure 7).
Therefore, the non-conserved amino acids (mutated ones) plays important roles for structure and function.
On the contrary, the conserved amino acids (not mutated) in proteins determine structure and function.
During protein evolution, evolutionary pressure to maintain protein structure and functions leads to the conservation or co-evolution of residues located in structural folding core for binding interface.
Diverse methods have been developed to extract the co-evolution information from conserved amino acids sequences for structure and function prediction, such as AlphaFold (Jumper et al., 2021).
In brief (Figure 7), antibody evolution specificity distinct from that of proteins can be defined with two main features: (i) ancestor germlines; (ii) the mutated amino acids of germlines.
[Figure]
Protein evolution
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
IRAT RGID GSVSTPSGQ
R W
PG
WV
IYY GGMP AKSG KAVD GPLSGPSYA
K W
PG
WV
IYY ATMQ AQTP KSLD QSLGGPSGA
Q W
PG
WV
IYY ATNN VNQP KGLD PSITSGSGN
R W
PG
WV
IYY
Protein B
Protein A
GAMQWVKQAPGKGLEWVSSITSGSNNIYYR ...
FAE VPM YP
F KLVPM QL
F EKLVPWL
FAEK
LV
PM
FAEKLVPM
Antibody evolution
ancestorgermline mutation ( ) non mutated function mutation ) (mutated ancestor functions
A. D ATA P ROCESSING D ETAILS
Pairing Antibody with Germline For germline annotation in the pre-training task, we used the annotated germline sequences provided in the OAS database (Kovaltsuk et al., 2018).
For downstream benchmarks tasks like B-cell classification, therapeutic antibody engineering, and disease diagnosis, we completely followed the methods shown in the OAS database paper.
IgBLAST, an immunoinformatic benchmarking tool for the analysis of B-cell antibody repertoires was used for germline annotation (Ye et al., 2013).
The antibody nucleotide-containing FASTA file was aligned
[Table]
to germline and translated to acids
IgBLASTn aligned
IgBLASTp.
The germline databases for human patients used ImMunoGeneTics (IMGT) germline sequences derived from IMGT (Lefranc et al., 1999).
For each antibody, usually, multiple germline sequences can be obtained and only the single sequence showing the highest confidence score for the alignment was chosen.
Pre-training Data Processing We downloaded OAS Oct 2021 version from its website and removed duplicate sequences.
To avoid data leakage, we cluster sequences based on the CDR sequence and filter each cluster by 70% identity over the whole sequence using Linclust (Steinegger & Söding, 2018).
Then, we shuffle the dataset and split it into 100k-size chunks.
The last chunk is used as the validation set.
The dataset size is 20,245,249 and the split is 20,200,000/45,249.
Negative sampling ratio We have tried the ratio of 0.1/0.3/0.5/0.75 and found that this ratio has little influence on performance and convergent speech.
As we discuss in the appendix, the model can quickly converge for AGP and get an accuracy of 0.99.
Representation chosen for the pre-training tasks The two pre-training tasks are designed to simulate the evolution process in the immune system (as shown in Figure 1).
We detailed our motivation in Section 3.
The representations learned from different tasks are evaluated in the benchmark, and the performance can be found in the third part of Tables 2, 4, and 5.
More discussions are in the appendix.
A. ATUE D ETAILS
Antigen Binding Accurate antigen-binding prediction approaches could allow significantly more efficient antibody discovery with higher affinity.
Machine learning methods have already achieved some success in antibody binding capacity optimization.
We collect the antigen-binding data from
Mason et al.
and follow the training/validation/test split of 15,128/3,242/3,242.
The original dataset only has CDR3 fragments, and we extend them to the full antibody sequences.
For crossvalidation, we split the dataset by antibody sequences to ensure that no antibody sequences overlap between 90% training and 10% validation.
Paratope Prediction Paratope is the antibody residues involved in antigen binding.
The ability to accurately map the paratope can provide detailed knowledge about the binding mechanism and accelerate antibody discovery.
1D sequence-based deep learning methods have been employed for paratope prediction.
The paratope data is collected from Liberis et al.
(2018) with 1,662 CDR segments on 277 antibodies.
Each antibody contains three CDR fragments (CDR1, CDR2 and CDR3) in the heavy chain and three CDR fragments in the light chain.
We also search the full sequence for each antibody and use the whole sequence as input.
For cross-validation, we split the dataset by antibody sequences to ensure that no antibody sequences overlap between 90% training and 10% validation.
B Cell Analysis We formulate a 6-category classification task for B cell maturation analysis, which includes { immature, transitional, mature, memory IgD+, memory IgD-, plasmacytes, }.
The analysis of B cell maturation plays an important role in understanding the mechanisms underlying B cell responses in immune system Ghraichy et al.
; Meffre et al.
[Table]
The order of B cell type follows the evolutionary process in the immune system, from immature state to transitional state and finally becomes a memory B cell.
Both memory IgD- and IgD+ belong to memory B cells with different isotypes, and they have a high affinity to foreign antigens.
Among the other categories, the Plasmacytes PC sequences also have some affinity ability.
It is widely reported that changes in antibody sequence patterns correlate with B-cell maturation.
Therefore, we use this task to evaluate the representation learning capacity of the language model.
We collect 88,094 sequences from Mroczek et al.
They extracted from the peripheral blood of healthy adults and got six types of B cells with different maturity and antibody sequences.
The distribution of various types of B cells in the dataset is shown in Table
Antibody Discovery Antibody discovery from B cell repertoire has been widely recognized as a novel trend to improve the efficiency of antibody discovery for diverse antigens (Weiner, 2015; Pedrioli & Oxenius, 2021).
However, previous studies highly rely on expensive wet-lab experiments (Cao et al., 2020; Shiakolas et al., 2022).
Deep learning based method have shown potential capacity in help antibody discovery by reduce cost and increase efficiency (Widrich et al., 2020;
Wang et al., 2022).
Here, we ask whether EATLM can benefit real-world problems and enable fast-track neutralization SARS-CoV-2 antibody discovery.
In the first step, we develop a sequence classifier to distinguish which antibody sequence from the numerous sequences is responsible for the recognition of the SARS-CoV-2.
This task is highly challenging since we can hardly get the sequence-level disease label that indicates whether the antibody sequence is related to the disease.
Thus, we follow the practice of Roskin et al.
Zaslavsky et al.
(2022) to use the individual label as the rough sequence label and train a sequencelevel predictor.
Then, with the help of a sequence-level predictor, we can give each sequence a most likely label to help antibody discover, whose accuracy has been verified by the excellent results on individual prediction, which may accelerate the discovery of new antibody sequences.
We follow the condition of Kim et al.
(2021) to filter SARS-CoV-2 antibody data from the OAS database.
The basic condition is ‘ Chain = heavy; Isotype = IGHG; BSource = PBMC; Species = human; Vaccine = None ’.
We further add the condition of ‘ Unique Sequences >= 10000 ’.
For health/SARS we set the ‘ Disease ’ field to ‘ None ’, ‘ SARS-CoV-2 ’.
Then we obtain 87/133 patient profiles for each type.
To make a balanced dataset, we limit the size of the health profile and mix up the healthy ones and the ones with the SARS-CoV-2.
For cross-validation, we randomly split the dataset by profiles 10 times: 90% for training and 10% for validation.
We further select sequences with top100 redundancy to make the positive labels more accurate.
A. Q UANTITATIVE A NALYSIS OF ATUE T ASK S PECIFICITY
It is important to include statistical significance tests relative to the antibody-specific features in antibody functional tasks we proposed in the ATUE benchmark.
According to the evolution process shown in Figure 7, antibody evolution specificity distinct from that of proteins can be defined with two main features: (i) ancestor germlines; (ii) the mutated amino acids of germlines.
We implemented statistical significance tests of (i) ancestor germlines subtype usage; (ii) the number of mutated amino acids in antibodies against different labels of downstream tasks in ATUE to quantitatively assess the "Task specificity".
The analysis is now summarized in Table ?
In Appendix A.1.
Generally, it is clearly shown that ATUE benchmark comprises antibody tasks showing different scales of antibody
[Table]
specificity for later on modeling analysis.
Moreover, the which are used for statistical analysis of task specificity and pre-training model objectives in our study.
Antigen Binding In the Antigen binding dataset, both antibody binding and none antigen binding sequences share the same germline subtype sequence (IGHV3.1) (Figure 8A) as well as the same number of germline mutations 8B).
Therefore, None of two antibody specific features show significant distribution differences between data with different labels, demonstrating antigen binding is a task with low antibody specificity.
[Figure]
Paratope Prediction For the paratope prediction task, we first evaluate the germline subtype distribution difference between sequences with different numbers of binding sites (Figure 9A).
Kruskal Wallis test showed a p-value of 0.296 suggesting germline subtype usage is not statistically significant.
Also, we find the binding sites can be significantly mapped with more germline mutations than the non-binding sites, which is consistent with the knowledge of antibody specificity definition (Figure 9B).
One out of two antibody specific features show significant distribution differences between data with different labels.
Therefore, we define this task as a medium specificity task.
B Cell Analysis As shown in (Figure 10), the distribution of the germline usage as well as the number of germline mutations are significantly different between antibodies in B cells with different developmental stages.
This observation is highly consistent with previous studies Mroczek et al.
Ghraichy et al.
Since both of the antibody specific feature show significant distribution differences, this task is defined as a high-specificity task..
SARS antibody discovery Antibodies in SARS patients and healthy ones show a significant difference in their germline subtype usage and the number of germline mutations (Figure 11).
This observation is highly consistent with previous studies showing SARS antibody convergent among
[Figure] [Figure]
patients Galson et al.
Since both of the antibody specific feature are highly significant, this task is defined as a high-specificity task.
[Figure]
A. T RAINING D ETAILS
Antibody can be represented as A = { a 1 , a 2 , · · · , a m } and the germline of individual antibody can be represented as G = { g 1 , g 2 , · · · , g n } , where m and n are the lengths.
Each token a i or g j in the sequence is called a residue that belongs to the amino acid set A .
A includes 20 common amino acids with a residue ‘X’ that indicates the residue is unknown (mostly in the germline).
Typically, antibody
PLMs are trained with basic mask language modeling objective l MLM on the antibody sequences
S =
A = { a 1 , · · · , a m , } .
A.5. E VOLUTION - AWARE P RETRAINING
In order to incorporate the evolutionary information into the pre-training, we pair the antibody sequence A with its germline G and concatenate them into a long sequence with a special token ‘[SEP]’ as the delimiter: S = { s 1 , · · · , s m + n +1 } = { a 1 , · · · , a m , [SEP] , g 1 , · · · , g n } .
Thus, we optimize MLM objective on the long sequence S :
[Equation] [Equation]
where M is the index set of masked tokens.
It helps the model to learn the basic residue distribution for antibody sequences.
Besides, it can also capture the interaction between residues of the antibody and its germline.
Ancestor Germline Prediction
The ancestor relationship between the antibody and its germline determines the shared biological functions obtained in the evolution.
Antibody sequences with similar residues evolved from different germline sequences may have different biological functions.
When stimulated by a foreign antigen, the common ancestor germline evolve to various antibody sequences.
Similar antibody sequences may have different germline sequences, which will affect their biological functions.
Thus, the aim of this task is to determine whether the antibody have evolutionary relationship with the given germline.
During training, we substitute the paired germline G with random germline G ′ =
{ g 1 , · · · , g n } in the batch via a probability p = 0 .
The new sequence is denoted as S ′
=
{ a 1 , · · · , a m , [SEP] , g ′ 1 , · · · , g ′ n } and the training loss can be described as:
[Equation]
where y ∈ { 0 , 1 } indicate whether the noisy germline G ′ is the ancestor of the antibody S .
It can help the model to distinguish the ancestor germline of the antibody by capturing the shared features.
Mutation Position Prediction
The somatic hypermutations on the germline further give progeny antibodies the specificity of binding with the specific antigen.
In order to model this specificity, this task focus on predicting the mutation positions and the residues mutated to.
Specifically, for each
DTVMTQS……QYKHWPPYTFGRGTKLEIR
EIVMTQS……
QYNNWPXXXXXXXXXXXXX [SEP]
[Figure]
Encoder [CLS]
Antibody
Germline
[Figure]
Prediction (AGP) 1 1 0 0 0 … 1 1 D T K H P
[Figure]
MutationPositionPrediction(MPP) 𝒀𝝐{𝟎,𝟏} maskmutation substitutegermline
[Figure]
DTVMTQS……QYKHWPPYTFGRGTKLEIR
EIVMTQS……QYNNWPXXXXXXXXXXXXX
[Figure]
SEP]
Transformer
[Figure]
AntigenBinding
[Figure]
token g j in the germline G , the target is to predict a label y j ∈ { 0 , 1 } to indicate whether this token has been mutated.
For the antibody sequence S , we mask the mutation position and predict these tokens.
The objective can be formalized as:
[Equation]
Here, M ′ is the ground-truth mutation position and we mask these tokens on the antibody sequence.
This task is more difficult than MLM which equally masks tokens in the L , because the tokens on the mutation position of A get less information from the germline, compared with other shared residues between the antibody and the germline.
By optimize this objective, the model learn to capture the specificity obtained from the somatic hypermutation in the evolutionary process.
A.5. I MPLEMENTATION D ETAILS
We use the base Transformer architecture (Vaswani et al., 2017) with 12 layers, 12 heads, andhidden states.
The total parameters are 86M. We use Adam optimizer (Kingma & Ba, 2015) with the maximum learning rate of 2e-4 and the warm-up step of 24,000.
The maximum length is set to 400 since most antibody sequences are shorter than 180.
We first pre-train our model with the
MLM objective.
During the pre-training, 15% tokens are randomly selected with 80% masked, 10% replaced, and 10% kept.
Then we conduct further pre-training on two antibody-related tasks with a smaller learning rate of 1e-5.
For each task in ATUE , we finetune the model with supervised data.
We follow the standard split of
Antigen Binding Prediction.
For other tasks that do not provide a standard split, we conduct 10-cross validation and report the average results.
Since our pre-training model learns the representation of the antibody sequence, we expand the CDR fragment to the full antibody via searching the biological database for therapeutic antibody engineering tasks.
For finetuning, we limit the max epochs toand use the Adam optimizer with a max learning rate of 3e-5.
We use the mean representation oflayers as the sequence representation.
The model is trained with 108,000 steps and gets a 0.9606 token accuracy on the MLM task.
It takes further steps for AGP and MPP.
The model quickly converges for AGP and gets a 0.99 accuracy on the ancestor germline prediction because more than 80% residues are shared between the antibody and its germline.
For MPP, it can predict the mutation position with the accuracy of 1.00 and obtains a 0.4430 accuracy in the mutation position.
It means that the model can easily find the mutation positions by the self-attention between the antibody and germline, but it is still difficult to predict which residues this position will mutate to.
We assume it is because the ancestor germline can undergo different somatic hypermutations and get various progeny antibodies, resulting in different valid mutations at the same position.
We also compare this mutation accuracy with the model without MPP,
[Table]
which is only trained with MLM on the concatenation of the antibody and its germline.
With a high prediction accuracy of 0.8889 on all positions, it achieves only a 0.0311 accuracy on the mutations.
It implies that the masking among all positions on the sequence can do accurate predictions of the shared residues but hardly capture the mutation information
[Table]
We also conduct AGP and MPP to finetune the baseline model AntiBERT.
The pre-training results are shown in Table 8.
We can find that without the concatenation of the antibody and its germline, it is difficult to predict the ancestor relationship.
It also underperforms than EATLM in MPP.
Finetuned Protein Language Models and Larger Architecture
We pre-train our method with a larger architecture and compare it with ESM-1b, which also has 650M parameters.
We also further pre-trained the ESMs to transfer to the antibody field.
After that, we evaluate them on the antigen binding and paratope prediction task.
The results are shown in Table 9.
The result show that the larger architecture does show advantage in terms of the performance improvement.
For antigen binding,
ESM-1b has better performance than ESM-1.
However, for paratope prediction, it performs worse.
In addition, for ESM, fine-tuning of the antibody dataset may cause the overfitting problem, leading to the decrease in the performance of all three tasks
[Table] [Table]
A. D ISCUSSION ABOUT OF EATLM
First
[Table]
EATLM doesn’t use any 3D structure information during pre-training.
As a special subgroup of proteins, antibody structures provide much more information such as geometry than sequences.
In the future, recruiting structure information for antibody pre-training may be able to improve the results.
However, the data scale available for antibody structure is dramatically less than that of antibody sequences.
The largest dataset of antibody structures only contains thousands of 3D high-resolution structures, while the number of antibody sequences is in billions.
Using structure prediction methods like AlphaFold may help to bridge the gap between sequences and structures.
Second, EATLM requires germline as input during downstream tasks, this will slow down the prediction speed.
A. N EW SARS B INDER D ISCOVERY
Antibody discovery from B cell repertoire has been widely recognized as a novel trend to improve the efficiency of antibody discovery for diverse antigens (Weiner, 2015; Pedrioli & Oxenius, 2021).
However, previous studies highly rely on expensive wet-lab experiments (Cao et al., 2020; Shiakolas et al., 2022).
Deep learning based method have shown potential capacity in help antibody discovery by reduce cost and increase efficiency (Widrich et al., 2020;
Wang et al., 2022).
Here, we ask whether EATLM can benefit real-world problems and enable fast-track neutralization SARS-CoVantibody discovery.
Our sequence-to-individual pipeline gives an interpretable explanation for disease diagnosis.
As stated in Section A.3, the main challenge for disease diagnosis is to distinguish the disease-related antibodies from millions of antibody sequences in the individual profile.
Here, with the help of a sequence-level predictor, we can give each sequence a most likely label to help antibody discover, whose accuracy has been verified by the excellent results on individual prediction, which may accelerate the discovery of new antibody sequences.
Sequence-level Predictor for SARS As described in Section A.3, we first train a sequence-level predictor for SARS-CoV-2.
The results are shown in Table 10.
Compared with Compared with Figure 4 in the main text, we find that good results in the sequence-level predictor do not necessarily mean good results in the antibody discovery.
It can be mainly affected by the noisy label of the sequence level.
[Table]
As shown in Table
[Table]
we find 2 true SARS binders and 9 potential binders with the help of EATLM .
Specifically, we first use our sequence-level predictor to get a probability score for each sequence in the SARS dataset.
Then we select the sequence with high-ranked score (the probability > 0.5) and compare them with the public Cov-AbDab database Raybould et al.
(2021) 1 , which contains data on published/patented antibodies known to bind to SARS-CoV-2 (Raybould et al., 2021).
Since the
CDR3 fragment in the heavy chain is the most relevant to the binding of antibody and antigen, we calculate the edit distance between the CDR3 fragments in heavy chains (CDR-H3) with those of the known binder and use a threshold of 85% similarity as the sequence identity.
85% Hamming distance for B cell antibody sequence clustering (identify similar B cell antibody sequences responding to the same antigen/epitope) was previously suggested in this paper (Gupta et al., 2017).
This method then was widely used for B cell antibody repertoire analysis in different studies (Montague et al., 2021;
Wang et al., 2022).
To make it more stringent, we choose 90% Hamming distance for sequence clustering.
To provide a more intuitive analysis of the similarity between our predicted antibody and true SARS-CoV-2 binders, we investigate the 3D structure of the true binding antibodies and the mutation site of our predicted sequence on the corresponding structure.
High resolution structure of true binding antibody #3 with SARS-CoV-2 are shown in Figure 5 (PDB code: 7N62).
The interaction interface between the antibodies and SARS-CoV-2 spike/RBD are shown in Figure 3 in main body.
CDR-H3 were shown in orange.
Only one single atom highlighted in red is different between predicted binder and true binder.
Obviously, these different residues don’t localize to direct binding site and CDR-H3 founding core, suggesting the sequence difference likely will not affect antibody virus interaction.
[Table]
Furthermore, we found the epitopes of the 11 identified SARS-CoV-2 antibodies cover a wide range of different structures from traditional RBD domain to novel non-RBD epitopes like S2 and NTD as shown in Table 3.
This result shows our method enables diverse-epitope antibody discovery.
In Table 11, hit enrichment rate for different probability threshold predicted by the model were calculated.
As the prediction score increases, the hit enrichment rate corresponding increases from 0.528% to 0.562%, indicating that our model may enable priority selection of SARS-CoV-2 antibodies and reduce experimental costs.
During the training of our sequence-level predictor, we have no reliable ground-truth labels, which means that the model has never known which sequences can bind to SARS in the real-world scenario.
However, the model can learn from the noisy data and rank the real SARS binders with high probabilities.
Sequence identity of 1 means that the CDR-H3 fragment can be directly found in the
Cov-AbDab database, which implies that the sequences have been verified by wet laboratory testing.
The other sequences with an identity over 90% are thought to have a similar binding performance to existing binders, indicating that they are promising SARS binders that can help the discovery of therapeutic antibodies for SARS-CoV-2.
[Figure]
A. E XTENTED S TUDY FOR D ISEASE D IAGNOSIS
It would be interesting to see whether our sequence classifier can be used for other applications, such as disease diagnosis.
Each human is estimated to maintain about 10 8 − 10 10 distinct antibody sequences, constructing an informative encyclopedia recording the past and present health and disease.
Interpreting the pattern of the sequences has already proved useful in disease diagnosis and allows us to assess many infectious diseases without expensive laboratory testing.
However, it is difficult to distinguish which antibody sequence from the numerous sequences is responsible for the recognition of the specific antigen, which hinders the discovery of the antibody for diseases (Zaslavsky et al., 2022; Lu et al., 2018; Greiff et al., 2020).
Benefiting from the recent high-throughput sequencing, we can obtain millions of antibody sequences from the individual human.
At the same time, we can get a disease label that indicates whether the human is infected by the disease.
The main challenge is that we can hardly get the sequence-level disease label that indicates whether the antibody sequence is related to the disease.
Thus, we follow the practice of Roskin et al.
(2020) to use the individual label as the rough sequence label and train a sequence-level predictor.
Then we use this predictor to predict sequences of the individual profile and make the trimmed mean score as the individual score.
We follow the condition of Kim et al.
(2021) to filter disease data from the OAS database.
The basic condition is ‘ Chain = heavy; Isotype = IGHG; BSource = PBMC; Species = human; Vaccine =
None ’.
We further add the condition of ‘ Unique Sequences >= 10000 ’.
For health/SARS/HIV/Ebola/Allergy/SLE/MS, we set the ‘ Disease ’ field to ‘ None ’, ‘ Ebola ’, ‘ Allgery ’, ‘ SLE ’,‘ MS ’.
Then we obtain 87/133/51/14/12/8/8 patient profiles for each type.
To make a balanced dataset, we limit the size of the health profile and mix up the healthy ones and the ones with the disease.
For cross-validation, we randomly split the dataset by profiles 10 times: 90% for training and 10% for validation.
We further select sequences with top100 redundancy to make the positive labels more accurate.
[Table]
Sequence-level Predictor for Various Disease As described in Section A.3, we first train a sequence-level predictor for each disease.
The results are shown in Table 10.
Compared with Tablein the main text, we find that good results in the sequence-level predictor do not necessarily mean good results in the individual-level predictor.
It is mainly due to the trimmed mean we use to get individual-level results, which is a central estimate that is robust to noise labels.
Overall, our model has comparable results to other models in terms of sequence prediction with noisy labels, and has better results for individual diagnosis.
Individual-level Predictor for Various Disease
It is observed our evolution-aware EATLM performs the best in the individual-level classifier to determine whether the patient suffering from SARS.
Besides, PALMs significantly outperform PPLMs.
The results is shown in
Disease Classification We use all these diseases profile to build the Q7 classification task for disease diagnosis.
Previous biological studies mainly use this multi-classification task for disease diagnosis Zaslavsky et al.
; Wang et al.
(2022), highlighting the discriminatory power among different diseases are important for disease diagnosis.
The results are shown in Table 12.
We found both PPLM an PALM show comparable results as the random initialized model, suggesting the finetuning part play more important role and pretrained language model cannot help this task.
[Table] [Table]