File size: 1,532 Bytes
d19f767 f3a6a81 9a220aa f3a6a81 9a220aa f3a6a81 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
---
language:
- ms
---
# Pretrain 1.1B 4096 context length Mistral on Malaysian text
README at https://github.com/mesolitica/malaya/tree/5.1/pretrained-model/mistral
- Dataset gathered at https://github.com/malaysia-ai/dedup-text-dataset/tree/main/pretrain-llm
- We use Ray cluster to train on 5 nodes of 4x A100 80GB, https://github.com/malaysia-ai/jupyter-gpu/tree/main/ray
WandB, https://wandb.ai/mesolitica/pretrain-mistral-1.1b?workspace=user-husein-mesolitica
WandB report, https://wandb.ai/mesolitica/pretrain-mistral-3b/reports/Pretrain-Larger-Malaysian-Mistral--Vmlldzo2MDkyOTgz
## how-to
```python
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch
TORCH_DTYPE = 'bfloat16'
nf4_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=getattr(torch, TORCH_DTYPE)
)
tokenizer = AutoTokenizer.from_pretrained('mesolitica/malaysian-mistral-1.1B-4096', model_input_names = ['input_ids'])
model = AutoModelForCausalLM.from_pretrained(
'mesolitica/malaysian-mistral-1.1B-4096',
use_flash_attention_2 = True,
quantization_config = nf4_config
)
prompt = '<s>nama saya'
inputs = tokenizer([prompt], return_tensors='pt', add_special_tokens=False).to('cuda')
generate_kwargs = dict(
inputs,
max_new_tokens=512,
top_p=0.95,
top_k=50,
temperature=0.9,
do_sample=True,
num_beams=1,
repetition_penalty=1.05,
)
r = model.generate(**generate_kwargs)
``` |