bol20162021 commited on
Commit
01666d6
1 Parent(s): 2f92d85
Files changed (1) hide show
  1. README.md +131 -1
README.md CHANGED
@@ -1,3 +1,133 @@
1
  ---
2
- license: bigcode-openrail-m
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: bsd-3-clause
3
  ---
4
+ # starcoder-toolbench
5
+
6
+ <!-- Provide a quick summary of what the model is/does. -->
7
+
8
+ starcoder-toolbench is a 15 billion parameter model used for api based action generation. It is instruction tuned from [starcoder](https://huggingface.co/bigcode/starcoder) on api based action generation datasets.
9
+
10
+ ## Model Details
11
+
12
+ ### Model Description
13
+
14
+ <!-- Provide a longer summary of what this model is. -->
15
+
16
+ - **Developed by:** [SambaNova Systems](https://sambanova.ai/)
17
+ - **Model type:** Language Model
18
+ - **Language(s):** English
19
+ - **License:**
20
+ - **Finetuned from model:** [starcoder](https://huggingface.co/bigcode/starcoder)
21
+
22
+ ### Basic Information
23
+
24
+ <!-- Provide the basic links for the model. -->
25
+ - **Paper**: [Link]
26
+ - **Github**: [Link]
27
+
28
+ ### Licensing
29
+
30
+ TBD
31
+
32
+ ## Uses
33
+ <details>
34
+ <summary>Click to expand</summary>
35
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
36
+
37
+ ### Direct Use
38
+
39
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
40
+ This model is intended for commercial and research use.
41
+
42
+
43
+ ### Out-of-Scope Use
44
+
45
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
46
+
47
+
48
+ starcoder-toolbench should NOT be used for purpose other than API based action generation.
49
+
50
+ ### Recommendations
51
+
52
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
53
+
54
+ Users should be made aware of the risks, biases, limitations, and restrictions of the model, which are listed down at the bottom of the page.
55
+
56
+ </details>
57
+
58
+
59
+ ---
60
+ ## How to Get Started with the Model
61
+
62
+ <details>
63
+ <summary>Click to expand</summary>
64
+
65
+ ### Loading in model with Huggingface
66
+
67
+ ```python
68
+ from transformers import AutoModelForCausalLM, AutoTokenizer
69
+
70
+ tokenizer = AutoTokenizer.from_pretrained("sambanovasystems/starcoder-toolbench")
71
+ model = AutoModelForCausalLM.from_pretrained("sambanovasystems/starcoder-toolbench", device_map="auto", torch_dtype="auto")
72
+ ```
73
+
74
+ </details>
75
+
76
+ ---
77
+
78
+ ## Training Details
79
+
80
+ <details>
81
+ <summary>Click to expand</summary>
82
+
83
+ ### Training Data
84
+
85
+ <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
86
+
87
+ - [Fenglu to add](https://huggingface.co/datasets/laion/OIG)
88
+
89
+
90
+ ### Training Procedure
91
+
92
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
93
+
94
+ We trained starcoder-toolbench on 4 80GB A100 gpu's. We started from [starcoder](https://huggingface.co/bigcode/starcoder). We finetuned it on XXX dataset.
95
+ All of the code used to prepare the datasets and the scripts to run training and inference are open-sourced and freely available at [githublink here](dummy link)
96
+
97
+
98
+ ### Prompting Style Used For Training
99
+ ```
100
+
101
+ ```
102
+
103
+ ### Hyperparameters
104
+
105
+ - Hardware: A100 GPU
106
+ - Optimizer: AdamW
107
+ - Grad accumulation: 1
108
+ - Epochs: 8
109
+ - Global Batch size: 16
110
+ - Batch tokens: 16 * 2048 = 32,768 tokens
111
+ - Learning Rate: 1e-5
112
+ - Learning Rate Scheduler: Fixed LR
113
+ - Weight decay: 0.1
114
+
115
+
116
+ </details>
117
+
118
+
119
+
120
+ ## Acknowledgment
121
+
122
+
123
+ ## Cite starcoder-toolbench
124
+ ```
125
+ @software{bloomchat,
126
+ title = {{BLOOMChat: a New Open Multilingual Chat LLM}},
127
+ author = {SambaNova Systems, Together Computer},
128
+ url = {https://huggingface.co/sambanovasystems/BLOOMChat-176B-v1}
129
+ month = {5},
130
+ year = {2023},
131
+ version = {1.0},
132
+ }
133
+ ```