Hugging Face's logo

pkupie
/
Llama-2-7b-mon

PyTorch
English
Mongolian
llama
Model card Files Community

A continually pre-trained model based on Llama-2-7b-hf.

We use the Traditional Mongolian texts in MC^2 and English texts in RedPajama with a proportion of 4:1 for training.

Hyper-parameters:

  • lr: 3e-5
  • batch size: 1M (2K*512)
  • lr scheduler: cosine
  • min lr: 1e-6
  • lr decay iters: 10240
Downloads last month
6
Inference API
Unable to determine this model's library. Check the docs .

Model tree for pkupie/Llama-2-7b-mon

Base model

meta-llama/Llama-2-7b-hf
Finetuned
(597)
this model

Datasets used to train pkupie/Llama-2-7b-mon

Collection including pkupie/Llama-2-7b-mon