michaelfeil commited on
Commit
deab3ca
1 Parent(s): d559e9c

Upload EleutherAI/pythia-160m ctranslate fp16 weights

Browse files
Files changed (3) hide show
  1. README.md +320 -0
  2. model.bin +2 -2
  3. special_tokens_map.json +5 -0
README.md ADDED
@@ -0,0 +1,320 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - ctranslate2
6
+ - int8
7
+ - float16
8
+ - pytorch
9
+ - causal-lm
10
+ - pythia
11
+ license: apache-2.0
12
+ datasets:
13
+ - the_pile
14
+ ---
15
+ # # Fast-Inference with Ctranslate2
16
+ Speedup inference by 2x-8x using int8 inference in C++
17
+
18
+ quantized version of [EleutherAI/pythia-160m](https://huggingface.co/EleutherAI/pythia-160m)
19
+ ```bash
20
+ pip install hf-hub-ctranslate2>=2.0.6 ctranslate2>=3.13.0
21
+ ```
22
+ Converted on 2023-05-19 using
23
+ ```
24
+ ct2-transformers-converter --model EleutherAI/pythia-160m --output_dir /home/michael/tmp-ct2fast-pythia-160m --force --copy_files tokenizer.json README.md tokenizer_config.json special_tokens_map.json .gitattributes --quantization float16
25
+ ```
26
+
27
+ Checkpoint compatible to [ctranslate2](https://github.com/OpenNMT/CTranslate2) and [hf-hub-ctranslate2](https://github.com/michaelfeil/hf-hub-ctranslate2)
28
+ - `compute_type=int8_float16` for `device="cuda"`
29
+ - `compute_type=int8` for `device="cpu"`
30
+
31
+ ```python
32
+ from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub
33
+ from transformers import AutoTokenizer
34
+
35
+ model_name = "michaelfeil/ct2fast-pythia-160m"
36
+ # use either TranslatorCT2fromHfHub or GeneratorCT2fromHfHub here, depending on model.
37
+ model = GeneratorCT2fromHfHub(
38
+ # load in int8 on CUDA
39
+ model_name_or_path=model_name,
40
+ device="cuda",
41
+ compute_type="int8_float16",
42
+ tokenizer=AutoTokenizer.from_pretrained("EleutherAI/pythia-160m")
43
+ )
44
+ outputs = model.generate(
45
+ text=["How do you call a fast Flan-ingo?", "User: How are you doing?"],
46
+ )
47
+ print(outputs)
48
+ ```
49
+
50
+ # Licence and other remarks:
51
+ This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.
52
+
53
+ # Original description
54
+
55
+
56
+ The *Pythia Scaling Suite* is a collection of models developed to facilitate
57
+ interpretability research. It contains two sets of eight models of sizes
58
+ 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two
59
+ models: one trained on the Pile, and one trained on the Pile after the dataset
60
+ has been globally deduplicated. All 8 model sizes are trained on the exact
61
+ same data, in the exact same order. We also provide 154 intermediate
62
+ checkpoints per model, hosted on Hugging Face as branches.
63
+
64
+ The Pythia model suite was deliberately designed to promote scientific
65
+ research on large language models, especially interpretability research.
66
+ Despite not centering downstream performance as a design goal, we find the
67
+ models <a href="#evaluations">match or exceed</a> the performance of
68
+ similar and same-sized models, such as those in the OPT and GPT-Neo suites.
69
+
70
+ <details>
71
+ <summary style="font-weight:600">Details on previous early release and naming convention.</summary>
72
+
73
+ Previously, we released an early version of the Pythia suite to the public.
74
+ However, we decided to retrain the model suite to address a few hyperparameter
75
+ discrepancies. This model card <a href="#changelog">lists the changes</a>;
76
+ see appendix B in the Pythia paper for further discussion. We found no
77
+ difference in benchmark performance between the two Pythia versions.
78
+ The old models are
79
+ [still available](https://huggingface.co/models?other=pythia_v0), but we
80
+ suggest the retrained suite if you are just starting to use Pythia.<br>
81
+ **This is the current release.**
82
+
83
+ Please note that all models in the *Pythia* suite were renamed in January
84
+ 2023. For clarity, a <a href="#naming-convention-and-parameter-count">table
85
+ comparing the old and new names</a> is provided in this model card, together
86
+ with exact parameter counts.
87
+ </details>
88
+ <br>
89
+
90
+ # Pythia-160M
91
+
92
+ ## Model Details
93
+
94
+ - Developed by: [EleutherAI](http://eleuther.ai)
95
+ - Model type: Transformer-based Language Model
96
+ - Language: English
97
+ - Learn more: [Pythia's GitHub repository](https://github.com/EleutherAI/pythia)
98
+ for training procedure, config files, and details on how to use.
99
+ - Library: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)
100
+ - License: Apache 2.0
101
+ - Contact: to ask questions about this model, join the [EleutherAI
102
+ Discord](https://discord.gg/zBGx3azzUn), and post them in `#release-discussion`.
103
+ Please read the existing *Pythia* documentation before asking about it in the
104
+ EleutherAI Discord. For general correspondence: [contact@eleuther.
105
+ ai](mailto:contact@eleuther.ai).
106
+
107
+ <figure>
108
+
109
+ | Pythia model | Non-Embedding Params | Layers | Model Dim | Heads | Batch Size | Learning Rate | Equivalent Models |
110
+ | -----------: | -------------------: | :----: | :-------: | :---: | :--------: | :-------------------: | :--------------------: |
111
+ | 70M | 18,915,328 | 6 | 512 | 8 | 2M | 1.0 x 10<sup>-3</sup> | — |
112
+ | 160M | 85,056,000 | 12 | 768 | 12 | 4M | 6.0 x 10<sup>-4</sup> | GPT-Neo 125M, OPT-125M |
113
+ | 410M | 302,311,424 | 24 | 1024 | 16 | 4M | 3.0 x 10<sup>-4</sup> | OPT-350M |
114
+ | 1.0B | 805,736,448 | 16 | 2048 | 8 | 2M | 3.0 x 10<sup>-4</sup> | — |
115
+ | 1.4B | 1,208,602,624 | 24 | 2048 | 16 | 4M | 2.0 x 10<sup>-4</sup> | GPT-Neo 1.3B, OPT-1.3B |
116
+ | 2.8B | 2,517,652,480 | 32 | 2560 | 32 | 2M | 1.6 x 10<sup>-4</sup> | GPT-Neo 2.7B, OPT-2.7B |
117
+ | 6.9B | 6,444,163,072 | 32 | 4096 | 32 | 2M | 1.2 x 10<sup>-4</sup> | OPT-6.7B |
118
+ | 12B | 11,327,027,200 | 36 | 5120 | 40 | 2M | 1.2 x 10<sup>-4</sup> | — |
119
+ <figcaption>Engineering details for the <i>Pythia Suite</i>. Deduped and
120
+ non-deduped models of a given size have the same hyperparameters. “Equivalent”
121
+ models have <b>exactly</b> the same architecture, and the same number of
122
+ non-embedding parameters.</figcaption>
123
+ </figure>
124
+
125
+ ## Uses and Limitations
126
+
127
+ ### Intended Use
128
+
129
+ The primary intended use of Pythia is research on the behavior, functionality,
130
+ and limitations of large language models. This suite is intended to provide
131
+ a controlled setting for performing scientific experiments. We also provide
132
+ 154 checkpoints per model: initial `step0`, 10 log-spaced checkpoints
133
+ `step{1,2,4...512}`, and 143 evenly-spaced checkpoints from `step1000` to
134
+ `step143000`. These checkpoints are hosted on Hugging Face as branches. Note
135
+ that branch `143000` corresponds exactly to the model checkpoint on the `main`
136
+ branch of each model.
137
+
138
+ You may also further fine-tune and adapt Pythia-160M for deployment,
139
+ as long as your use is in accordance with the Apache 2.0 license. Pythia
140
+ models work with the Hugging Face [Transformers
141
+ Library](https://huggingface.co/docs/transformers/index). If you decide to use
142
+ pre-trained Pythia-160M as a basis for your fine-tuned model, please
143
+ conduct your own risk and bias assessment.
144
+
145
+ ### Out-of-scope use
146
+
147
+ The Pythia Suite is **not** intended for deployment. It is not a in itself
148
+ a product and cannot be used for human-facing interactions. For example,
149
+ the model may generate harmful or offensive text. Please evaluate the risks
150
+ associated with your particular use case.
151
+
152
+ Pythia models are English-language only, and are not suitable for translation
153
+ or generating text in other languages.
154
+
155
+ Pythia-160M has not been fine-tuned for downstream contexts in which
156
+ language models are commonly deployed, such as writing genre prose,
157
+ or commercial chatbots. This means Pythia-160M will **not**
158
+ respond to a given prompt the way a product like ChatGPT does. This is because,
159
+ unlike this model, ChatGPT was fine-tuned using methods such as Reinforcement
160
+ Learning from Human Feedback (RLHF) to better “follow” human instructions.
161
+
162
+ ### Limitations and biases
163
+
164
+ The core functionality of a large language model is to take a string of text
165
+ and predict the next token. The token used by the model need not produce the
166
+ most “accurate” text. Never rely on Pythia-160M to produce factually accurate
167
+ output.
168
+
169
+ This model was trained on [the Pile](https://pile.eleuther.ai/), a dataset
170
+ known to contain profanity and texts that are lewd or otherwise offensive.
171
+ See [Section 6 of the Pile paper](https://arxiv.org/abs/2101.00027) for a
172
+ discussion of documented biases with regards to gender, religion, and race.
173
+ Pythia-160M may produce socially unacceptable or undesirable text, *even if*
174
+ the prompt itself does not include anything explicitly offensive.
175
+
176
+ If you plan on using text generated through, for example, the Hosted Inference
177
+ API, we recommend having a human curate the outputs of this language model
178
+ before presenting it to other people. Please inform your audience that the
179
+ text was generated by Pythia-160M.
180
+
181
+ ### Quickstart
182
+
183
+ Pythia models can be loaded and used via the following code, demonstrated here
184
+ for the third `pythia-70m-deduped` checkpoint:
185
+
186
+ ```python
187
+ from transformers import GPTNeoXForCausalLM, AutoTokenizer
188
+
189
+ model = GPTNeoXForCausalLM.from_pretrained(
190
+ "EleutherAI/pythia-70m-deduped",
191
+ revision="step3000",
192
+ cache_dir="./pythia-70m-deduped/step3000",
193
+ )
194
+
195
+ tokenizer = AutoTokenizer.from_pretrained(
196
+ "EleutherAI/pythia-70m-deduped",
197
+ revision="step3000",
198
+ cache_dir="./pythia-70m-deduped/step3000",
199
+ )
200
+
201
+ inputs = tokenizer("Hello, I am", return_tensors="pt")
202
+ tokens = model.generate(**inputs)
203
+ tokenizer.decode(tokens[0])
204
+ ```
205
+
206
+ Revision/branch `step143000` corresponds exactly to the model checkpoint on
207
+ the `main` branch of each model.<br>
208
+ For more information on how to use all Pythia models, see [documentation on
209
+ GitHub](https://github.com/EleutherAI/pythia).
210
+
211
+ ## Training
212
+
213
+ ### Training data
214
+
215
+ [The Pile](https://pile.eleuther.ai/) is a 825GiB general-purpose dataset in
216
+ English. It was created by EleutherAI specifically for training large language
217
+ models. It contains texts from 22 diverse sources, roughly broken down into
218
+ five categories: academic writing (e.g. arXiv), internet (e.g. CommonCrawl),
219
+ prose (e.g. Project Gutenberg), dialogue (e.g. YouTube subtitles), and
220
+ miscellaneous (e.g. GitHub, Enron Emails). See [the Pile
221
+ paper](https://arxiv.org/abs/2101.00027) for a breakdown of all data sources,
222
+ methodology, and a discussion of ethical implications. Consult [the
223
+ datasheet](https://arxiv.org/abs/2201.07311) for more detailed documentation
224
+ about the Pile and its component datasets. The Pile can be downloaded from
225
+ the [official website](https://pile.eleuther.ai/), or from a [community
226
+ mirror](https://the-eye.eu/public/AI/pile/).<br>
227
+ The Pile was **not** deduplicated before being used to train Pythia-160M.
228
+
229
+ ### Training procedure
230
+
231
+ All models were trained on the exact same data, in the exact same order. Each
232
+ model saw 299,892,736,000 tokens during training, and 143 checkpoints for each
233
+ model are saved every 2,097,152,000 tokens, spaced evenly throughout training,
234
+ from `step1000` to `step143000` (which is the same as `main`). In addition, we
235
+ also provide frequent early checkpoints: `step0` and `step{1,2,4...512}`.
236
+ This corresponds to training for just under 1 epoch on the Pile for
237
+ non-deduplicated models, and about 1.5 epochs on the deduplicated Pile.
238
+
239
+ All *Pythia* models trained for 143000 steps at a batch size
240
+ of 2M (2,097,152 tokens).<br>
241
+ See [GitHub](https://github.com/EleutherAI/pythia) for more details on training
242
+ procedure, including [how to reproduce
243
+ it](https://github.com/EleutherAI/pythia/blob/main/README.md#reproducing-training).<br>
244
+ Pythia uses the same tokenizer as [GPT-NeoX-
245
+ 20B](https://huggingface.co/EleutherAI/gpt-neox-20b).
246
+
247
+ ## Evaluations
248
+
249
+ All 16 *Pythia* models were evaluated using the [LM Evaluation
250
+ Harness](https://github.com/EleutherAI/lm-evaluation-harness). You can access
251
+ the results by model and step at `results/json/*` in the [GitHub
252
+ repository](https://github.com/EleutherAI/pythia/tree/main/results/json/).<br>
253
+ Expand the sections below to see plots of evaluation results for all
254
+ Pythia and Pythia-deduped models compared with OPT and BLOOM.
255
+
256
+ <details>
257
+ <summary>LAMBADA – OpenAI</summary>
258
+ <img src="/EleutherAI/pythia-12b/resolve/main/eval_plots/lambada_openai_v1.png" style="width:auto"/>
259
+ </details>
260
+
261
+ <details>
262
+ <summary>Physical Interaction: Question Answering (PIQA)</summary>
263
+ <img src="/EleutherAI/pythia-12b/resolve/main/eval_plots/piqa_v1.png" style="width:auto"/>
264
+ </details>
265
+
266
+ <details>
267
+ <summary>WinoGrande</summary>
268
+ <img src="/EleutherAI/pythia-12b/resolve/main/eval_plots/winogrande_v1.png" style="width:auto"/>
269
+ </details>
270
+
271
+ <details>
272
+ <summary>AI2 Reasoning Challenge—Easy Set</summary>
273
+ <img src="/EleutherAI/pythia-12b/resolve/main/eval_plots/arc_easy_v1.png" style="width:auto"/>
274
+ </details>
275
+
276
+ <details>
277
+ <summary>SciQ</summary>
278
+ <img src="/EleutherAI/pythia-12b/resolve/main/eval_plots/sciq_v1.png" style="width:auto"/>
279
+ </details>
280
+
281
+ ## Changelog
282
+
283
+ This section compares differences between previously released
284
+ [Pythia v0](https://huggingface.co/models?other=pythia_v0) and the current
285
+ models. See Appendix B of the Pythia paper for further discussion of these
286
+ changes and the motivation behind them. We found that retraining Pythia had no
287
+ impact on benchmark performance.
288
+
289
+ - All model sizes are now trained with uniform batch size of 2M tokens.
290
+ Previously, the models of size 160M, 410M, and 1.4B parameters were trained
291
+ with batch sizes of 4M tokens.
292
+ - We added checkpoints at initialization (step 0) and steps {1,2,4,8,16,32,64,
293
+ 128,256,512} in addition to every 1000 training steps.
294
+ - Flash Attention was used in the new retrained suite.
295
+ - We remedied a minor inconsistency that existed in the original suite: all
296
+ models of size 2.8B parameters or smaller had a learning rate (LR) schedule
297
+ which decayed to a minimum LR of 10% the starting LR rate, but the 6.9B and
298
+ 12B models all used an LR schedule which decayed to a minimum LR of 0. In
299
+ the redone training runs, we rectified this inconsistency: all models now were
300
+ trained with LR decaying to a minimum of 0.1× their maximum LR.
301
+
302
+ ### Naming convention and parameter count
303
+
304
+ *Pythia* models were renamed in January 2023. It is possible that the old
305
+ naming convention still persists in some documentation by accident. The
306
+ current naming convention (70M, 160M, etc.) is based on total parameter count.
307
+
308
+ <figure style="width:32em">
309
+
310
+ | current Pythia suffix | old suffix | total params | non-embedding params |
311
+ | --------------------: | ---------: | -------------: | -------------------: |
312
+ | 70M | 19M | 70,426,624 | 18,915,328 |
313
+ | 160M | 125M | 162,322,944 | 85,056,000 |
314
+ | 410M | 350M | 405,334,016 | 302,311,424 |
315
+ | 1B | 800M | 1,011,781,632 | 805,736,448 |
316
+ | 1.4B | 1.3B | 1,414,647,808 | 1,208,602,624 |
317
+ | 2.8B | 2.7B | 2,775,208,960 | 2,517,652,480 |
318
+ | 6.9B | 6.7B | 6,857,302,016 | 6,444,163,072 |
319
+ | 12B | 13B | 11,846,072,320 | 11,327,027,200 |
320
+ </figure>
model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:8f22f8ac57130a4d2cfaca6c98ebc9e703bdf1577e9e3b33ac804f4275867998
3
- size 163191140
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3aedd2ecb85a8df8ad10e182c1b2a9caf4a09aa14a3309769506eeb16f71bab2
3
+ size 324655580
special_tokens_map.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<|endoftext|>",
3
+ "eos_token": "<|endoftext|>",
4
+ "unk_token": "<|endoftext|>"
5
+ }