metadata

license: apache-2.0
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
datasets:
  - cerebras/SlimPajama-627B
  - bigcode/starcoderdata
  - monsoon-nlp/greenbeing-proteins
language:
  - en

tinyllama-proteinpretrain-quinoa

Full model finetuning of TinyLLaMA-1.1B on the "research" split (quinoa protein sequences) of GreenBeing-Proteins dataset.

Notes: pretraining only on sequences leads the model to only generate protein sequences, eventually repeating VVVV ot KKKK.

This model may be replaced with mixed training (bio/chem text and protein).
This model might need "biotokens" to represent the amino acids instead of using the existing tokenizer.

More details TBD