jon-tow commited on
Commit
f5eaca1
1 Parent(s): 7c750ac

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +123 -0
README.md ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - causal-lm
6
+ license: cc-by-nc-4.0
7
+ datasets:
8
+ - dmayhem93/ChatCombined
9
+ - tatsu-lab/alpaca
10
+ - nomic-ai/gpt4all_prompt_generations
11
+ - Dahoas/full-hh-rlhf
12
+ - jeffwan/sharegpt_vicuna
13
+ - HuggingFaceH4/databricks_dolly_15k
14
+ ---
15
+
16
+ # StableLM-Tuned-Alpha
17
+
18
+ ## Model Description
19
+
20
+ `StableLM-Tuned-Alpha` is a suite of 3B and 7B parameter decoder-only language models built on top of the `StableLM-Base-Alpha` models and further fine-tuned on various chat and instruction-following datasets.
21
+
22
+ ## Usage
23
+
24
+ Get started chatting with `StableLM-Tuned-Alpha` by using the following code snippet:
25
+
26
+ ```python
27
+ from transformers import AutoModelForCausalLM, AutoTokenizer
28
+
29
+ tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-tuned-alpha-7b")
30
+ model = AutoModelForCausalLM.from_pretrained("StabilityAI/stablelm-tuned-alpha-7b")
31
+ model.half().cuda()
32
+
33
+ inputs = tokenizer("What's your mood today?", return_tensors="pt")
34
+ tokens = model.generate(
35
+ **inputs,
36
+ max_new_tokens=64,
37
+ temperature=0.7,
38
+ do_sample=True,
39
+ )
40
+ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
41
+ ```
42
+
43
+ ## Model Details
44
+
45
+ * **Developed by**: [Stability AI](https://stability.ai/)
46
+ * **Model type**: StableLM-Tuned-Alpha models are auto-regressive language models based on the NeoX transformer architecture.
47
+ * **Language(s)**: English
48
+ * **Library**: [HuggingFace Transformers](https://github.com/huggingface/transformers)
49
+ * **License**: [CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
50
+ * **Contact**: For questions and comments about the model, please email {TODO: email address}
51
+
52
+ ## Training
53
+
54
+ | Parameters | Hidden Size | Layers | Heads |
55
+ |------------|-------------|--------|-------|
56
+ | 3B | 5120 | 40 | 40 |
57
+ | 7B | 5120 | 40 | 40 |
58
+
59
+ ### Training Dataset
60
+
61
+ `StableLM-Tuned-Alpha` models are fine-tuned on a combination of five datasets:
62
+ [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca), a dataset of 52,000 instructions and demonstrations generated by OpenAI's `text-davinci-003` engine.
63
+ [GPT4All Prompt Generations](https://huggingface.co/datasets/nomic-ai/gpt4all_prompt_generations), which consists of 400k prompts and responses generated by GPT-4;
64
+ [Anthropic HH](https://huggingface.co/datasets/Dahoas/full-hh-rlhf), made up of preferences about AI assistant helpfulness and harmlessness;
65
+ [DataBricks Dolly](https://github.com/databrickslabs/dolly), comprising 15k instruction/responses generated by Databricks employees in capability domains from the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA and summarization;
66
+ and [ShareGPT Vicuna (English subset)](https://huggingface.co/datasets/jeffwan/sharegpt_vicuna), a dataset of conversations retrieved from [ShareGPT](https://sharegpt.com/).
67
+
68
+ ### Training Procedure
69
+
70
+ Models are learned via supervised fine-tuning on the aforementioned datasets, trained in mixed-precision (FP16), and optimized with AdamW. We outline the following hyperparameters:
71
+
72
+ | Parameters | Batch Size | Learning Rate | Warm-up | Weight Decay | Betas |
73
+ |------------|------------|---------------|---------|--------------|-------------|
74
+ | 3B | 256 | 2e-5 | 50 | 0.01 | (0.9, 0.99) |
75
+ | 7B | 128 | 2e-5 | 100 | 0.01 | (0.9, 0.99) |
76
+
77
+ ## Use and Limitations
78
+
79
+ ### Intended Use
80
+
81
+ These models are intended to be used by the open-source community for chat-like applications in adherence with the [CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license.
82
+
83
+ ### Limitations and bias
84
+
85
+ Although the aforementioned datasets help to steer the base language models into "safer" distributions of text, not all biases and toxicity can be mitigated through fine-tuning. We ask that users be mindful of such potential issues that can arise in generated responses. Do not treat model outputs as substitutes for human judgment or as sources of truth. Please use responsibly.
86
+
87
+ ## Acknowledgements
88
+
89
+ This work would not have been possible without the helpful hand of Dakota Mahan ([@dmayhem93](https://huggingface.co/dmayhem93)).
90
+
91
+ ## Citations
92
+
93
+ ```bibtex
94
+ @misc{alpaca,
95
+ author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
96
+ title = {Stanford Alpaca: An Instruction-following LLaMA model},
97
+ year = {2023},
98
+ publisher = {GitHub},
99
+ journal = {GitHub repository},
100
+ howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
101
+ }
102
+ ```
103
+
104
+ ```bibtext
105
+ @misc{vicuna2023,
106
+ title = {Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality},
107
+ url = {https://vicuna.lmsys.org},
108
+ author = {Chiang, Wei-Lin and Li, Zhuohan and Lin, Zi and Sheng, Ying and Wu, Zhanghao and Zhang, Hao and Zheng, Lianmin and Zhuang, Siyuan and Zhuang, Yonghao and Gonzalez, Joseph E. and Stoica, Ion and Xing, Eric P.},
109
+ month = {March},
110
+ year = {2023}
111
+ }
112
+ ```
113
+
114
+ ```bibtex
115
+ @misc{gpt4all,
116
+ author = {Yuvanesh Anand and Zach Nussbaum and Brandon Duderstadt and Benjamin Schmidt and Andriy Mulyar},
117
+ title = {GPT4All: Training an Assistant-style Chatbot with Large Scale Data Distillation from GPT-3.5-Turbo},
118
+ year = {2023},
119
+ publisher = {GitHub},
120
+ journal = {GitHub repository},
121
+ howpublished = {\url{https://github.com/nomic-ai/gpt4all}},
122
+ }
123
+ ```