Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language: en
|
3 |
+
tags:
|
4 |
+
- exbert
|
5 |
+
|
6 |
+
license: mit
|
7 |
+
---
|
8 |
+
|
9 |
+
|
10 |
+
# GPT-2
|
11 |
+
|
12 |
+
|
13 |
+
Pretrained model on English language using a causal language modeling (CLM) objective. It was introduced in
|
14 |
+
[this paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
|
15 |
+
and first released at [this page](https://openai.com/blog/better-language-models/).
|
16 |
+
|
17 |
+
Disclaimer: The team releasing GPT-2 also wrote a
|
18 |
+
[model card](https://github.com/openai/gpt-2/blob/master/model_card.md) for their model. Content from this model card
|
19 |
+
has been written by the Hugging Face team to complete the information they provided and give specific examples of bias.
|
20 |
+
|
21 |
+
## Model description
|
22 |
+
|
23 |
+
GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This
|
24 |
+
means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots
|
25 |
+
of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely,
|
26 |
+
it was trained to guess the next word in sentences.
|
27 |
+
|
28 |
+
More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence,
|
29 |
+
shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the
|
30 |
+
predictions for the token `i` only uses the inputs from `1` to `i` but not the future tokens.
|
31 |
+
|
32 |
+
This way, the model learns an inner representation of the English language that can then be used to extract features
|
33 |
+
useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a
|
34 |
+
prompt.
|
35 |
+
|
36 |
+
## Intended uses & limitations
|
37 |
+
|
38 |
+
You can use the raw model for text generation or fine-tune it to a downstream task. See the
|
39 |
+
[model hub](https://huggingface.co/models?filter=gpt2) to look for fine-tuned versions on a task that interests you.
|
40 |
+
|
41 |
+
### How to use
|
42 |
+
|
43 |
+
You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
|
44 |
+
set a seed for reproducibility:
|
45 |
+
|
46 |
+
```python
|
47 |
+
from tf_transformers.models import GPT2Model
|
48 |
+
from transformers import GPT2Tokenizer
|
49 |
+
|
50 |
+
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
|
51 |
+
model = GPT2Model.from_pretrained("gpt2-medium")
|
52 |
+
|
53 |
+
text = "Replace me by any text you'd like."
|
54 |
+
inputs_tf = {}
|
55 |
+
inputs = tokenizer(text, return_tensors='tf')
|
56 |
+
|
57 |
+
|
58 |
+
inputs_tf["input_ids"] = inputs["input_ids"]
|
59 |
+
outputs_tf = model(inputs_tf)
|
60 |
+
```
|
61 |
+
|
62 |
+
### Limitations and bias
|
63 |
+
|
64 |
+
The training data used for this model has not been released as a dataset one can browse. We know it contains a lot of
|
65 |
+
unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their
|
66 |
+
[model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases):
|
67 |
+
|
68 |
+
> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases
|
69 |
+
> that require the generated text to be true.
|
70 |
+
>
|
71 |
+
> Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do
|
72 |
+
> not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a
|
73 |
+
> study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race,
|
74 |
+
> and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar
|
75 |
+
> levels of caution around use cases that are sensitive to biases around human attributes.
|
76 |
+
|
77 |
+
|
78 |
+
## Training data
|
79 |
+
|
80 |
+
The OpenAI team wanted to train this model on a corpus as large as possible. To build it, they scraped all the web
|
81 |
+
pages from outbound links on Reddit which received at least 3 karma. Note that all Wikipedia pages were removed from
|
82 |
+
this dataset, so the model was not trained on any part of Wikipedia. The resulting dataset (called WebText) weights
|
83 |
+
40GB of texts but has not been publicly released. You can find a list of the top 1,000 domains present in WebText
|
84 |
+
[here](https://github.com/openai/gpt-2/blob/master/domains.txt).
|
85 |
+
|
86 |
+
## Training procedure
|
87 |
+
|
88 |
+
### Preprocessing
|
89 |
+
|
90 |
+
The texts are tokenized using a byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a
|
91 |
+
vocabulary size of 50,257. The inputs are sequences of 1024 consecutive tokens.
|
92 |
+
|
93 |
+
The larger model was trained on 256 cloud TPU v3 cores. The training duration was not disclosed, nor were the exact
|
94 |
+
details of training.
|
95 |
+
|
96 |
+
## Evaluation results
|
97 |
+
|
98 |
+
The model achieves the following results without any fine-tuning (zero-shot):
|
99 |
+
|
100 |
+
| Dataset | LAMBADA | LAMBADA | CBT-CN | CBT-NE | WikiText2 | PTB | enwiki8 | text8 | WikiText103 | 1BW |
|
101 |
+
|:--------:|:-------:|:-------:|:------:|:------:|:---------:|:------:|:-------:|:------:|:-----------:|:-----:|
|
102 |
+
| (metric) | (PPL) | (ACC) | (ACC) | (ACC) | (PPL) | (PPL) | (BPB) | (BPC) | (PPL) | (PPL) |
|
103 |
+
| | 35.13 | 45.99 | 87.65 | 83.4 | 29.41 | 65.85 | 1.16 | 1,17 | 37.50 | 75.20 |
|
104 |
+
|
105 |
+
|
106 |
+
### BibTeX entry and citation info
|
107 |
+
|
108 |
+
```bibtex
|
109 |
+
@article{radford2019language,
|
110 |
+
title={Language Models are Unsupervised Multitask Learners},
|
111 |
+
author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
|
112 |
+
year={2019}
|
113 |
+
}
|
114 |
+
```
|
115 |
+
|
116 |
+
<a href="https://huggingface.co/exbert/?model=gpt2">
|
117 |
+
<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
|
118 |
+
</a>
|