zolicsaki commited on
Commit
1d61dc8
1 Parent(s): a67dc4b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +101 -0
README.md CHANGED
@@ -1,3 +1,104 @@
1
  ---
2
  license: llama2
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: llama2
3
+ datasets:
4
+ - uonlp/CulturaX
5
+ language:
6
+ - tr
7
+ - en
8
+ metrics:
9
+ - chrf
10
+ - accuracy
11
+ - bleu
12
  ---
13
+
14
+
15
+
16
+ # SambaLingo-Japanese-Base
17
+
18
+ <img src="SambaLingo_Logo.png" width="340" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
19
+
20
+ <!-- Provide a quick summary of what the model is/does. -->
21
+ SambaLingo-Japanese-Base is a pretrained Bi-lingual Japanese and English model that adapts [Llama 2](https://huggingface.co/meta-llama/Llama-2-7b-hf) to Japanese by training on 42 billion tokens from the Japanese split of the [Cultura-X](https://huggingface.co/datasets/uonlp/CulturaX) dataset. This model reports state of the art evaluation results in perplexity and FLORES-200 translation. For the chat version of this model, please see [sambanovasystems/SambaLingo-Japanese-Chat](https://huggingface.co/sambanovasystems/SambaLingo-Japanese-Chat).
22
+
23
+ ## Model Description
24
+ <!-- Provide a longer summary of what this model is. -->
25
+
26
+ - **Developed by:** [SambaNova Systems](https://sambanova.ai/)
27
+ - **Model type:** Language Model
28
+ - **Language(s):** Japanese, English
29
+ - **Finetuned from model:** [Llama 2](https://huggingface.co/meta-llama/Llama-2-7b-hf)
30
+ - **Blog Post**: Will be released soon!
31
+
32
+ ## Getting Started
33
+
34
+ ### Loading Model With Hugging Face
35
+ ```python
36
+ from transformers import AutoModelForCausalLM, AutoTokenizer
37
+
38
+ tokenizer = AutoTokenizer.from_pretrained("sambanovasystems/SambaLingo-Japanese-Base")
39
+ model = AutoModelForCausalLM.from_pretrained("sambanovasystems/SambaLingo-Japanese-Base", device_map="auto", torch_dtype="auto")
40
+ ```
41
+
42
+ ### Suggested Inference Parameters
43
+ We suggest setting do_sample=False as this is a pretrained checkpoint.
44
+
45
+ ### Prompting Guidelines
46
+ This model is a pretrained checkpoint, so to use it effectively please use few shot prompting with exemplars. The only other prompt templating required is the standard \<s\> (BOS) token from the Llama tokenizer. If you want to interact with this model with direct questions or queries, please use the chat version of the model that has been aligned with human preferences [sambanovasystems/SambaLingo-Japanese-Chat](https://huggingface.co/sambanovasystems/SambaLingo-Japanese-Chat).
47
+
48
+ ## Training Details
49
+ All pre-training is done on the [Cultura-X](https://huggingface.co/datasets/uonlp/CulturaX) dataset. We mix the data to be 75% data from the language we are adapting to, and 25% English as suggested by [Csaki et al.](https://arxiv.org/abs/2311.05741) We pack the data into sequences of length 4096, and ensure that when learning a token we only attend to previous tokens in the context of the corresponding text document. We train with a global batch size of 1024, sequence length of 4096, maximum learning rate of 1e-4 with cosine decay, warmup ratio of 0.01 and a weight decay of 0.1.
50
+
51
+ ## Tokenizer Details
52
+ We extended the vocabulary of the base llama model from 32,000 tokens to 57,000 tokens by adding up to 25,000 non-overlapping tokens from the new language.
53
+
54
+ ## Uses
55
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
56
+
57
+ ### Direct Use
58
+
59
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
60
+ This model is intended for commercial and research use.
61
+
62
+
63
+ ### Out-of-Scope Use
64
+
65
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
66
+ SambaLingo should NOT be used for:
67
+
68
+ - Mission-critical applications
69
+ - Applications that involve the safety of others
70
+ - Making highly important decisions
71
+
72
+ ## Bias, Risks, and Limitations
73
+
74
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
75
+
76
+ Like all LLMs, SambaLingo has certain limitations:
77
+ - Hallucination: Model may sometimes generate responses that contain plausible-sounding but factually incorrect or irrelevant information.
78
+ - Code Switching: The model might unintentionally switch between languages or dialects within a single response, affecting the coherence and understandability of the output.
79
+ - Repetition: The Model may produce repetitive phrases or sentences, leading to less engaging and informative responses.
80
+ - Coding and Math: The model's performance in generating accurate code or solving complex mathematical problems may be limited.
81
+ - Toxicity: The model could inadvertently generate responses containing inappropriate or harmful content.
82
+
83
+ ## Acknowledgments
84
+ We extend our heartfelt gratitude to the open-source AI community; this endeavor would not have been achievable without open source. SambaNova embraces the open-source community and aspires to actively contribute to this initiative.
85
+
86
+ We would like to give a special thanks to the following groups:
87
+ - Meta for open sourcing LLama 2 and open sourcing FLORES-200 dataset
88
+ - Nguyen et al for open sourcing CulturaX dataset
89
+ - CohereAI for releasing AYA-101 and open sourcing a multilingual instruction tuning dataset
90
+ - EleutherAI for their open source evaluation framework
91
+ - Hugging Face-H4 team for open source the zephyr training recipe and alignment handbook repo
92
+
93
+
94
+ ## Cite SambaLingo
95
+ ```
96
+ @software{sambalingo,
97
+ title = {{SambaLingo: Language Experts Adapted From Llama}},
98
+ author = {SambaNova Systems},
99
+ url = {https://huggingface.co/sambanovasystems/SambaLingo-Japanese-Base}
100
+ month = {2},
101
+ year = {2024},
102
+ version = {1.0},
103
+ }
104
+ ```