README.md

GenerRNA

GenerRNA is a generative RNA language model based on a Transformer decoder-only architecture. It was pre-trained on 30M sequences, encompassing 17B nucleotides.

Here, you can find all the relevant scripts for running GenerRNA on your machine. GenerRNA enable you to generate RNA sequences in a zero-shot manner for exploring the RNA space, or to fine-tune the model using a specific dataset for generating RNAs belonging to a particular family or possessing specific characteristics.

Requirements

A CUDA environment, and a minimum VRAM of 8GB was required.

Dependencies

torch>=2.0
numpy
transformers==4.33.0.dev0
datasets==2.14.4
tqdm

Usage

Firstly, combine the split model using the command cat model.pt.part-* > model.pt.recombined

Directory tree

.
├── LICENSE
├── README.md
├── configs 
│   ├── example_finetuning.py
│   └── example_pretraining.py
├── experiments_data
├── model.pt.part-aa # splited bin data of *HISTORICAL* model (shorter context window, less VRAM comsuption)
├── model.pt.part-ab 
├── model.pt.part-ac
├── model.pt.part-ad
├── model_updated.pt # *NEWER* model, with longer context windows and being trained on a deduplicated dataset
├── model.py         # define the architecture
├── sampling.py      # script to generate sequences
├── tokenization.py  # preparete data
├── tokenizer_bpe_1024
│   ├── tokenizer.json
│   ├── ....
├── train.py # script for training/fine-tuning

De novo Generation in a zero-shot fashion

Usage example:

python sampling.py \
    --out_path {output_file_path} \
    --max_new_tokens 256 \
    --ckpt_path {model.pt} \
    --tokenizer_path {path_to_tokenizer_directory, e.g /tokenizer_bpe_1024}

Pre-training or Fine-tuning on your own sequences

First, tokenize your sequence data, ensuring each sequence is on a separate line and there is no header.

python tokenization.py \
    --data_dir {path_to_the_directory_containing_sequence_data} \
    --file_name {file_name_of_sequence_data} \
    --tokenizer_path {path_to_tokenizer_directory}  \
    --out_dir {directory_to_save_tokenized_data} \
    --block_size 256

Next, refer to ./configs/example_**.py to create a config file of GPT model.

Lastly, excute following command:

python train.py \
    --config {path_to_your_config_file}

Train your own tokenizer

Usage example:

python train_BPE.py \
    --txt_file_path {path_to_training_file(txt,each sequence is on a separate line)} \
    --vocab_size 50256 \
    --new_tokenizer_path {directory_to_save_trained_tokenizer} \
                

License

The source code is licensed MIT. See LICENSE