---
license: cc-by-nc-nd-4.0
metrics:
- accuracy
tags:
- generated_from_trainer
- Text Generation
- Primary Sequence Prediction
model-index:
- name: protgpt2-finetuned-sarscov2-rbd
  results: []
---

# Model Card for `protgpt2-finetuned-sarscov2-rbd`

This model is a fine-tuned version of [nferruz/ProtGPT2](https://huggingface.co/nferruz/ProtGPT2) on sequences from the NCBI Virus Data Portal.

It achieves the following results on the evaluation set:
- Loss: 1.1674
- Accuracy: 0.8883

## Model description

This model is a fine-tuned checkpoint of
[ProtGPT2](https://huggingface.co/nferruz/ProtGPT2), which was originally
trained on the UniRef50 (version 2021_04) database. For a detailed overview
of the original model configuration and architecture, please see the linked
model card, or refer to the ProtGPT2 publication.

The model was finetuned on data from the SARS-CoV-2 Spike (surface glycoprotein)
receptor binding domain (RBD).

A repository with the training scripts, train and test data partitions, as well
as evaluation code is available on GitHub at
(https://github.com/rahuldhodapkar/PredictSARSVariants).

## Intended uses & limitations

This model is intended to generate synthetic SARS-CoV-2 surface glycoprotein
(a.k.a. spike protein) sequences for the purpose of identifying meaningful
variants for characterization either experimentally or through other
*in silico* tools.  These variants may be used to drive vaccine develop to
protect against never-before-seen point mutants that are probable in the future.

As this model is based on the original ProtGPT2 model, it is subject to many
of the same limitations as the base model.  Any biases present in the UniRef50
dataset will also be present in the model, which may include nonuniform skew
of peptides sampled across different taxonomic clades.  These limitations
should be considered when interpreting the output of this model.

## Training and evaluation data

SARS-CoV-2 spike protein sequences were obtained from the NIH Sars-CoV-2 Data Hub
accessible at 

    https://www.ncbi.nlm.nih.gov/labs/virus/vssi/

Note that the reference sequence for the surface glycoprotein can be found at:

    https://www.ncbi.nlm.nih.gov/protein/1791269090

As the loaded ProtGPT2 model was pretrained on the
UniRef50 (version 2021_04) dataset, it cannot have contained sequencing
data that was generated after that date.  Evaluations will be conducted using
SARS-CoV-2 sequences generated on or after May 2021.

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Framework versions

- Transformers 4.26.0.dev0
- Pytorch 1.11.0
- Datasets 2.8.0
- Tokenizers 0.13.2