File size: 3,050 Bytes
ac444da 5feb78b 05c8ca0 5feb78b 154578c ba40d8c 5feb78b ac444da 5feb78b ac444da |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 |
---
license: apache-2.0
tags:
- merge
- mergekit
- vilm/vinallama-7b-chat
---
# VinaLLaMA - State-of-the-art Vietnamese LLMs
Read our [Paper](https://huggingface.co/papers/2312.11011)
Prompt Format (ChatML):
```
<|im_start|>system
Bạn là một trợ lí AI hữu ích. Hãy trả lời người dùng một cách chính xác.
<|im_end|>
<|im_start|>user
Hello world!<|im_end|>
<|im_start|>assistant
```
## Evaluation
This table is copied from VBD-Llama2 with updated results from VinaLLaMA-12.5B-chat-DUS
| Model | Model size | arc_vi (acc) | hellaswag_vi (acc) | mmlu_vi (acc) | truthfulqa_vi (acc) | Average |
| ------------------------ | ---------- | ------------ | ------------------ | ------------- | ------------------- | ------- |
| URA-LLaMA-13B | 13B | 0,3752 | 0,4830 | 0,3973 | 0,4574 | 0,4282 |
| BLOOMZ-7B | 7B | 0,3205 | 0,4930 | <b>0,3975</b> | 0,4523 | 0,4158 |
| PhoGPT-7B5-Instruct | 7B | 0,2470 | 0,2578 | 0,2413 | 0,4759 | 0,3055 |
| SeaLLM-7B-chat | 7B | 0,3607 | 0,5112 | 0,3339 | 0,4948 | 0,4252 |
| Vietcuna-7b-v3 | 7B | 0,3419 | 0,4939 | 0,3354 | 0,4807 | 0,4130 |
| VinaLLaMA-2.7B-chat | 7B | 0,3273 | 0,4814 | 0,3051 | 0,4972 | 0,4028 |
| VinaLLaMA-7B-chat | 7B | 0,4239 | 0,5407 | 0,3932 | 0,5251 | 0,4707 |
| VBD-LLaMA2-7B-50b | 7B | 0,3222 | 0,5195 | 0,2964 | 0,4614 | 0,3999 |
| VBD-LLaMA2-7B-50b-Chat | 7B | 0,3585 | 0,5207 | 0,3444 | 0,5179 | 0,4354 |
| <b>VinaLLaMA-12.5B-chat-DUS</b> | 12.5B | <b>0,4325</b>| <b>0,5816</b> | 0,3875 | <b>0,5850</b> | <b>0,4967</b> |
# Merging Methods
This model is a merge of the following models made with [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
* [vilm/vinallama-7b-chat](https://huggingface.co/vilm/vinallama-7b-chat)
## 🧩 Configuration
```yaml
slices:
- sources:
- model: vilm/vinallama-7b-chat
layer_range: [0, 16]
- sources:
- model: vilm/vinallama-7b-chat
layer_range: [8, 16]
- sources:
- model: vilm/vinallama-7b-chat
layer_range: [8, 16]
- sources:
- model: vilm/vinallama-7b-chat
layer_range: [16, 24]
- sources:
- model: vilm/vinallama-7b-chat
layer_range: [16, 24]
- sources:
- model: vilm/vinallama-7b-chat
layer_range: [24, 28]
- sources:
- model: vilm/vinallama-7b-chat
layer_range: [24, 28]
- sources:
- model: vilm/vinallama-7b-chat
layer_range: [28, 32]
merge_method: passthrough
dtype: bfloat16
``` |