trojblue commited on
Commit
6e239fd
1 Parent(s): 26f7392

adding safetensor-only version of gpt2-medium

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ model.safetensors filter=lfs diff=lfs merge=lfs -text
37
+ *.ipynb filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,200 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: mit
4
+ ---
5
+
6
+ # GPT-2 Medium
7
+
8
+ ## Model Details
9
+
10
+ **Model Description:** GPT-2 Medium is the **355M parameter** version of GPT-2, a transformer-based language model created and released by OpenAI. The model is a pretrained model on English language using a causal language modeling (CLM) objective.
11
+
12
+ - **Developed by:** OpenAI, see [associated research paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) and [GitHub repo](https://github.com/openai/gpt-2) for model developers.
13
+ - **Model Type:** Transformer-based language model
14
+ - **Language(s):** English
15
+ - **License:** [Modified MIT License](https://github.com/openai/gpt-2/blob/master/LICENSE)
16
+ - **Related Models:** [GPT2](https://huggingface.co/gpt2), [GPT2-Large](https://huggingface.co/gpt2-large) and [GPT2-XL](https://huggingface.co/gpt2-xl)
17
+ - **Resources for more information:**
18
+ - [Research Paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
19
+ - [OpenAI Blog Post](https://openai.com/blog/better-language-models/)
20
+ - [GitHub Repo](https://github.com/openai/gpt-2)
21
+ - [OpenAI Model Card for GPT-2](https://github.com/openai/gpt-2/blob/master/model_card.md)
22
+ - Test the full generation capabilities here: https://transformer.huggingface.co/doc/gpt2-large
23
+
24
+ ## How to Get Started with the Model
25
+
26
+ Use the code below to get started with the model. You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
27
+ set a seed for reproducibility:
28
+
29
+ ```python
30
+ >>> from transformers import pipeline, set_seed
31
+ >>> generator = pipeline('text-generation', model='gpt2-medium')
32
+ >>> set_seed(42)
33
+ >>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)
34
+
35
+ [{'generated_text': "Hello, I'm a language model, I'm a language. I'm a compiler, I'm a parser, I'm a server process. I"},
36
+ {'generated_text': "Hello, I'm a language model, and I'd like to join an existing team. What can I do to get started?\n\nI'd"},
37
+ {'generated_text': "Hello, I'm a language model, why does my code get created? Can't I just copy it? But why did my code get created when"},
38
+ {'generated_text': "Hello, I'm a language model, a functional language...\n\nI'm a functional language. Is it hard? A little, yes. But"},
39
+ {'generated_text': "Hello, I'm a language model, not an object model.\n\nIn a nutshell, I need to give me objects from which I can get"}]
40
+ ```
41
+
42
+ Here is how to use this model to get the features of a given text in PyTorch:
43
+
44
+ ```python
45
+ from transformers import GPT2Tokenizer, GPT2Model
46
+ tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
47
+ model = GPT2Model.from_pretrained('gpt2-medium')
48
+ text = "Replace me by any text you'd like."
49
+ encoded_input = tokenizer(text, return_tensors='pt')
50
+ output = model(**encoded_input)
51
+ ```
52
+
53
+ and in TensorFlow:
54
+
55
+ ```python
56
+ from transformers import GPT2Tokenizer, TFGPT2Model
57
+ tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
58
+ model = TFGPT2Model.from_pretrained('gpt2-medium')
59
+ text = "Replace me by any text you'd like."
60
+ encoded_input = tokenizer(text, return_tensors='tf')
61
+ output = model(encoded_input)
62
+ ```
63
+
64
+ ## Uses
65
+
66
+ #### Direct Use
67
+
68
+ In their [model card about GPT-2](https://github.com/openai/gpt-2/blob/master/model_card.md), OpenAI wrote:
69
+
70
+ > The primary intended users of these models are AI researchers and practitioners.
71
+ >
72
+ > We primarily imagine these language models will be used by researchers to better understand the behaviors, capabilities, biases, and constraints of large-scale generative language models.
73
+
74
+ #### Downstream Use
75
+
76
+ In their [model card about GPT-2](https://github.com/openai/gpt-2/blob/master/model_card.md), OpenAI wrote:
77
+
78
+ > Here are some secondary use cases we believe are likely:
79
+ >
80
+ > - Writing assistance: Grammar assistance, autocompletion (for normal prose or code)
81
+ > - Creative writing and art: exploring the generation of creative, fictional texts; aiding creation of poetry and other literary art.
82
+ > - Entertainment: Creation of games, chat bots, and amusing generations.
83
+
84
+ #### Misuse and Out-of-scope Use
85
+
86
+ In their [model card about GPT-2](https://github.com/openai/gpt-2/blob/master/model_card.md), OpenAI wrote:
87
+
88
+ > Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true.
89
+ >
90
+ > Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do not recommend that they be deployed into systems that interact with humans unless the deployers first carry out a study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race, and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar levels of caution around use cases that are sensitive to biases around human attributes.
91
+
92
+ ## Risks, Limitations and Biases
93
+
94
+ **CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propogate historical and current stereotypes.**
95
+
96
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
97
+
98
+ The training data used for this model has not been released as a dataset one can browse. We know it contains a lot of unfiltered content from the internet, which is far from neutral. Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:
99
+
100
+ ```python
101
+ >>> from transformers import pipeline, set_seed
102
+ >>> generator = pipeline('text-generation', model='gpt2-medium')
103
+ >>> set_seed(42)
104
+ >>> generator("The man worked as a", max_length=10, num_return_sequences=5)
105
+
106
+ [{'generated_text': 'The man worked as a security guard in a military'},
107
+ {'generated_text': 'The man worked as a salesman in Mexico and eventually'},
108
+ {'generated_text': 'The man worked as a supervisor at the department for'},
109
+ {'generated_text': 'The man worked as a cleaner for the same corporation'},
110
+ {'generated_text': 'The man worked as a barman and was involved'}]
111
+
112
+ >>> set_seed(42)
113
+ >>> generator("The woman worked as a", max_length=10, num_return_sequences=5)
114
+
115
+ [{'generated_text': 'The woman worked as a social worker in a children'},
116
+ {'generated_text': 'The woman worked as a marketing manager, and her'},
117
+ {'generated_text': 'The woman worked as a customer service agent in a'},
118
+ {'generated_text': 'The woman worked as a cleaner for the same corporation'},
119
+ {'generated_text': 'The woman worked as a barista and was involved'}]
120
+ ```
121
+
122
+ This bias will also affect all fine-tuned versions of this model. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
123
+
124
+ ## Training
125
+
126
+ #### Training Data
127
+
128
+ The OpenAI team wanted to train this model on a corpus as large as possible. To build it, they scraped all the web
129
+ pages from outbound links on Reddit which received at least 3 karma. Note that all Wikipedia pages were removed from
130
+ this dataset, so the model was not trained on any part of Wikipedia. The resulting dataset (called WebText) weights
131
+ 40GB of texts but has not been publicly released. You can find a list of the top 1,000 domains present in WebText
132
+ [here](https://github.com/openai/gpt-2/blob/master/domains.txt).
133
+
134
+ #### Training Procedure
135
+
136
+ The model is pretrained on a very large corpus of English data in a self-supervised fashion. This
137
+ means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots
138
+ of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely,
139
+ it was trained to guess the next word in sentences.
140
+
141
+ More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence,
142
+ shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the
143
+ predictions for the token `i` only uses the inputs from `1` to `i` but not the future tokens.
144
+
145
+ This way, the model learns an inner representation of the English language that can then be used to extract features
146
+ useful for downstream tasks.
147
+
148
+ The texts are tokenized using a byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a
149
+ vocabulary size of 50,257. The inputs are sequences of 1024 consecutive tokens.
150
+
151
+ ## Evaluation
152
+
153
+ The following evaluation information is extracted from the [associated paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf).
154
+
155
+ #### Testing Data, Factors and Metrics
156
+
157
+ The model authors write in the [associated paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) that:
158
+
159
+ > Since our model operates on a byte level and does not require lossy pre-processing or tokenization, we can evaluate it on any language model benchmark. Results on language modeling datasets are commonly reported in a quantity which is a scaled or ex- ponentiated version of the average negative log probability per canonical prediction unit - usually a character, a byte, or a word. We evaluate the same quantity by computing the log-probability of a dataset according to a WebText LM and dividing by the number of canonical units. For many of these datasets, WebText LMs would be tested significantly out- of-distribution, having to predict aggressively standardized text, tokenization artifacts such as disconnected punctuation and contractions, shuffled sentences, and even the string <UNK> which is extremely rare in WebText - occurring only 26 times in 40 billion bytes. We report our main results...using invertible de-tokenizers which remove as many of these tokenization / pre-processing artifacts as possible. Since these de-tokenizers are invertible, we can still calculate the log probability of a dataset and they can be thought of as a simple form of domain adaptation.
160
+
161
+ #### Results
162
+
163
+ The model achieves the following results without any fine-tuning (zero-shot):
164
+
165
+ | Dataset | LAMBADA | LAMBADA | CBT-CN | CBT-NE | WikiText2 | PTB | enwiki8 | text8 | WikiText103 | 1BW |
166
+ |:--------:|:-------:|:-------:|:------:|:------:|:---------:|:------:|:-------:|:------:|:-----------:|:-----:|
167
+ | (metric) | (PPL) | (ACC) | (ACC) | (ACC) | (PPL) | (PPL) | (BPB) | (BPC) | (PPL) | (PPL) |
168
+ | | 15.60 | 55.48 | 92.35 | 87.1 | 22.76 | 47.33 | 1.01 | 1.06 | 26.37 | 55.72 |
169
+
170
+ ## Environmental Impact
171
+
172
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
173
+
174
+ - **Hardware Type:** Unknown
175
+ - **Hours used:** Unknown
176
+ - **Cloud Provider:** Unknown
177
+ - **Compute Region:** Unknown
178
+ - **Carbon Emitted:** Unknown
179
+
180
+ ## Technical Specifications
181
+
182
+ See the [associated paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) for details on the modeling architecture, objective, compute infrastructure, and training details.
183
+
184
+ ## Citation Information
185
+
186
+ ```bibtex
187
+ @article{radford2019language,
188
+ title={Language models are unsupervised multitask learners},
189
+ author={Radford, Alec and Wu, Jeffrey and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya and others},
190
+ journal={OpenAI blog},
191
+ volume={1},
192
+ number={8},
193
+ pages={9},
194
+ year={2019}
195
+ }
196
+ ```
197
+
198
+ ## Model Card Authors
199
+
200
+ This model card was written by the Hugging Face team.
config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "activation_function": "gelu_new",
3
+ "architectures": [
4
+ "GPT2LMHeadModel"
5
+ ],
6
+ "attn_pdrop": 0.1,
7
+ "bos_token_id": 50256,
8
+ "embd_pdrop": 0.1,
9
+ "eos_token_id": 50256,
10
+ "initializer_range": 0.02,
11
+ "layer_norm_epsilon": 1e-05,
12
+ "model_type": "gpt2",
13
+ "n_ctx": 1024,
14
+ "n_embd": 1024,
15
+ "n_head": 16,
16
+ "n_layer": 24,
17
+ "n_positions": 1024,
18
+ "n_special": 0,
19
+ "predict_special_tokens": true,
20
+ "resid_pdrop": 0.1,
21
+ "summary_activation": null,
22
+ "summary_first_dropout": 0.1,
23
+ "summary_proj_to_labels": true,
24
+ "summary_type": "cls_index",
25
+ "summary_use_proj": true,
26
+ "task_specific_params": {
27
+ "text-generation": {
28
+ "do_sample": true,
29
+ "max_length": 50
30
+ }
31
+ },
32
+ "vocab_size": 50257
33
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 50256,
4
+ "eos_token_id": 50256,
5
+ "transformers_version": "4.27.0.dev0"
6
+ }
generation_config_for_text_generation.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 50256,
4
+ "do_sample": true,
5
+ "eos_token_id": 50256,
6
+ "max_length": 50,
7
+ "transformers_version": "4.27.0.dev0"
8
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fc5a354a19255ad494f3d71549390baca1ccf61d1d822b9408971705c687c9cd
3
+ size 1519984962
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"model_max_length": 1024}
vocab.json ADDED
The diff for this file is too large to render. See raw diff