Text Generation
Transformers
PyTorch
Safetensors
Arabic
English
llama
conversational
text-generation-inference
Inference Endpoints
File size: 8,252 Bytes
78b91a0
 
c90c2d3
 
 
 
 
 
 
78b91a0
c90c2d3
 
 
 
 
 
 
 
7f85874
c90c2d3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7994e6b
c90c2d3
7994e6b
c90c2d3
 
 
 
 
 
 
 
 
 
 
 
 
3d6aa5d
 
 
c90c2d3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3d6aa5d
 
 
 
 
 
 
c90c2d3
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
---
license: llama2
datasets:
- HuggingFaceH4/ultrachat_200k
- HuggingFaceH4/ultrafeedback_binarized
- HuggingFaceH4/cai-conversation-harmless
language:
- ar
- en
---



# SambaLingo-Arabic-Chat-70B

<img src="SambaLingo_Logo.png" width="340" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

<!-- Provide a quick summary of what the model is/does. -->
SambaLingo-Arabic-Chat-70B is a human aligned chat model trained in Arabic and English. It is trained using direct preference optimization on top the base model [SambaLingo-Arabic-Base-70B](https://huggingface.co/sambanovasystems/SambaLingo-Arabic-Base-70B). The base model adapts [Llama-2-70b](https://huggingface.co/meta-llama/Llama-2-70b-hf) to Arabic by training on 28 billion tokens from the Arabic split of the [Cultura-X](https://huggingface.co/datasets/uonlp/CulturaX) dataset. Try This Model at [SambaLingo-chat-space](https://huggingface.co/spaces/sambanovasystems/SambaLingo-chat-space).

## Model Description
<!-- Provide a longer summary of what this model is. -->

- **Developed by:** [SambaNova Systems](https://sambanova.ai/)
- **Model type:** Language Model
- **Language(s):** Arabic, English
- **Finetuned from model:** [Llama-2-70b](https://huggingface.co/meta-llama/Llama-2-70b-hf)
- **Paper:** [SambaLingo: Teaching Large Language Models New Languages](https://arxiv.org/abs/2404.05829)
- **Blog Post**: [sambalingo-open-source-language-experts](https://sambanova.ai/blog/sambalingo-open-source-language-experts)

## Getting Started

### Loading Model With Hugging Face
Please make sure to set use_fast=False when loading the tokenizer.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sambanovasystems/SambaLingo-Arabic-Chat-70B", use_fast=False)
model = AutoModelForCausalLM.from_pretrained("sambanovasystems/SambaLingo-Arabic-Chat-70B", device_map="auto", torch_dtype="auto")
```

### Interacting With Model Pipeline
Please make sure to set use_fast=False when loading the tokenizer.
```python
from transformers import pipeline
pipe = pipeline("text-generation", model="sambanovasystems/SambaLingo-Arabic-Chat-70B", device_map="auto", use_fast=False)
messages = [
                {"role": "user", "content": {YOUR_QUESTION}},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt)[0]
outputs = outputs["generated_text"]
```

### Suggested Inference Parameters
- Temperature: 0.8
- Repetition penalty: 1.0
- Top-p: 0.9

### Prompting Guidelines
To prompt this model, please use the following chat template:
```
<|user|>\n{question}</s>\n<|assistant|>\n
```

### Example Prompts and Generations
```
<|user|>
ูƒูŠู ุชุตู ุงู„ุงู„ูˆุงู† ู„ุดุฎุต ูˆู„ุฏ ุฃุนู…ู‰</s>
<|assistant|>
ู„ูŠุณ ู„ุฏูŠ ุงู„ู‚ุฏุฑุฉ ุนู„ู‰ ุงู„ุฑุคูŠุฉุŒ ูˆู„ูƒู† ูŠู…ูƒู†ู†ูŠ ุชู‚ุฏูŠู… ูˆุตู ู„ู„ุฃู„ูˆุงู† ุจู†ุงุกู‹ ุนู„ู‰ ู…ุง ุชุนู„ู…ุชู‡ ู…ู† ุงู„ุจูŠุงู†ุงุช.\n\nุจุงู„ู†ุณุจุฉ ู„ุดุฎุต ูˆู„ุฏ ุฃุนู…ู‰ุŒ ู‚ุฏ ูŠูƒูˆู† ู…ู† ุงู„ุตุนุจ ูˆุตู ุงู„ุฃู„ูˆุงู† ู„ุฃู†ู‡ุง ู„ู… ุชุฎุชุจุฑู‡ุง ู…ู† ู‚ุจู„. ูˆู…ุน ุฐู„ูƒุŒ ูŠู…ูƒู†ู†ุง ุงุณุชุฎุฏุงู… ุงู„ุงุณุชุนุงุฑุงุช ูˆุงู„ู…ู‚ุงุฑู†ุงุช ู„ู…ุณุงุนุฏุฉ ู‡ุฐุง ุงู„ุดุฎุต ุนู„ู‰ ูู‡ู… ุงู„ุฃู„ูˆุงู†. ุนู„ู‰ ุณุจูŠู„ ุงู„ู…ุซุงู„ุŒ ูŠู…ูƒู†ู†ุง ูˆุตู ุงู„ู„ูˆู† ุงู„ุฃุญู…ุฑ ุจุฃู†ู‡ ุฏุงูุฆ ูˆุญูŠูˆูŠุŒ ู…ุซู„ ู„ู‡ุจ ุงู„ู†ุงุฑ ุฃูˆ ุซู…ุฑุฉ ุงู„ูุฑุงูˆู„ุฉ ุงู„ู†ุงุถุฌุฉ. ูŠู…ูƒู†ู†ุง ูˆุตู ุงู„ู„ูˆู† ุงู„ุฃุฒุฑู‚ ุจุฃู†ู‡ ู‡ุงุฏุฆ ูˆู…ู‡ุฏุฆุŒ ู…ุซู„ ุงู„ุณู…ุงุก ุฃูˆ ุงู„ู…ุญูŠุท. ูŠู…ูƒู†ู†ุง ูˆุตู ุงู„ู„ูˆู† ุงู„ุฃุฎุถุฑ ุจุฃู†ู‡ ู…ู†ุนุด ูˆุทุจูŠุนูŠุŒ ู…ุซู„ ุฃูˆุฑุงู‚ ุงู„ุดุฌุฑ ุฃูˆ ุงู„ุนุดุจ. ูŠู…ูƒู†ู†ุง ูˆุตู ุงู„ู„ูˆู† ุงู„ุฃุตูุฑ ุจุฃู†ู‡ ู…ุดุฑู‚ ูˆู…ุจู‡ุฌุŒ ู…ุซู„ ุงู„ุดู…ุณ ุฃูˆ ุฒู‡ุฑุฉ ุนุจุงุฏ ุงู„ุดู…ุณ. ูŠู…ูƒู†ู†ุง ูˆุตู ุงู„ู„ูˆู† ุงู„ุฃุฑุฌูˆุงู†ูŠ ุจุฃู†ู‡ ู…ู„ูƒูŠ ูˆูุงุฎุฑุŒ ู…ุซู„ ู„ูˆู† ุงู„ุฌูˆุงู‡ุฑ ุฃูˆ ุบุฑูˆุจ ุงู„ุดู…ุณ. ููŠ ุงู„ู†ู‡ุงูŠุฉุŒ ู‚ุฏ ูŠูƒูˆู† ู…ู† ุงู„ุตุนุจ ูˆุตู ุงู„ุฃู„ูˆุงู† ู„ุดุฎุต ูˆู„ุฏ ุฃุนู…ู‰ุŒ ูˆู„ูƒู† ุจุงุณุชุฎุฏุงู… ุงู„ุงุณุชุนุงุฑุงุช ูˆุงู„ู…ู‚ุงุฑู†ุงุชุŒ ูŠู…ูƒู†ู†ุง ู…ุณุงุนุฏุชู‡ู… ุนู„ู‰ ูู‡ู… ุงู„ูุฑูˆู‚ ุงู„ุฏู‚ูŠู‚ุฉ ููŠ ุงู„ุฃู„ูˆุงู† ุจุดูƒู„ ุฃูุถู„.
```

## Training Details
The alignment phase follows the recipe for [Zephyr-7B](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), and comprises two stages: supervised fine-tuning (SFT) and Direct Performance Optimization (DPO).

The SFT phase was done on the [ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) dataset mixed with the Google translated version of the ultrachat_200k dataset. It was trained for one epoch with global batch size 512 and max sequence length 2048 tokens. We used a linear decay learning rate of 2e-5 and 10% warmup.

The DPO phase was done on the [ultrafeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) dataset and [cai-conversation-harmless](https://huggingface.co/datasets/HuggingFaceH4/cai-conversation-harmless) dataset, mixed with 10% of the data Google translated. It was trained with global batch size 32 and for three epochs. We used a linear decay learning rate of 5e-7, 10% warmup and ฮฒ=0.1 as the regularization factor for DPO. 


## Tokenizer Details
We extended the vocabulary of the base llama model from 32,000 tokens to 57,000 tokens by adding up to 25,000 non-overlapping tokens from the new language.

## Evaluation
For evaluation results see our paper: [SambaLingo: Teaching Large Language Models New Languages](https://arxiv.org/abs/2404.05829)

## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
Use of this model is governed by the Metaโ€™s [Llama 2 Community License Agreement](https://ai.meta.com/llama/license/). Please review and accept the license before downloading the model weights.


### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
SambaLingo should NOT be used for:

- Mission-critical applications
- Applications that involve the safety of others
- Making highly important decisions

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

Like all LLMs, SambaLingo has certain limitations:
- Hallucination: Model may sometimes generate responses that contain plausible-sounding but factually incorrect or irrelevant information.
- Code Switching: The model might unintentionally switch between languages or dialects within a single response, affecting the coherence and understandability of the output.
- Repetition: The Model may produce repetitive phrases or sentences, leading to less engaging and informative responses.
- Coding and Math: The model's performance in generating accurate code or solving complex mathematical problems may be limited.
- Toxicity: The model could inadvertently generate responses containing inappropriate or harmful content.

## Acknowledgments
We extend our heartfelt gratitude to the open-source AI community; this endeavor would not have been possible without open source. SambaNova embraces the open-source community and aspires to actively contribute to this initiative.

We would like to give a special thanks to the following groups:
- Meta for open sourcing LLama 2 and open sourcing FLORES-200 dataset
- Nguyen et al for open sourcing CulturaX dataset
- CohereAI for releasing AYA-101 and open sourcing a multilingual instruction tuning dataset
- EleutherAI for their open source evaluation framework
- Hugging Face-H4 team for open source the zephyr training recipe and alignment handbook repo


## Cite SambaLingo
```
@misc{csaki2024sambalingo,
      title={SambaLingo: Teaching Large Language Models New Languages}, 
      author={Zoltan Csaki and Bo Li and Jonathan Li and Qiantong Xu and Pian Pawakapan and Leon Zhang and Yun Du and Hengyu Zhao and Changran Hu and Urmish Thakker},
      year={2024},
      eprint={2404.05829},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```