patrickvonplaten commited on
Commit
3c69d24
1 Parent(s): d9874ea

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +144 -0
README.md ADDED
@@ -0,0 +1,144 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - opt
5
+
6
+ license: mit
7
+ ---
8
+
9
+
10
+ # OPT : Open Pre-trained Transformer Language Models
11
+
12
+ OPT was predominantly pretrained with English text, but a small amount of non-English data is still present within the training corpus via CommonCrawl. The model was pretrained using a causal language modeling (CLM) objective.
13
+
14
+ OPT was first introduced in [Open Pre-trained Transformer Language Models](https://arxiv.org/abs/2205.01068) and first released in [metaseq's repository](https://github.com/facebookresearch/metaseq) on May 3rd 2022 by Meta AI.
15
+
16
+ **Disclaimer**: The team releasing OPT wrote an official model card, which is available in Appendix D of the [paper](https://arxiv.org/pdf/2205.01068.pdf).
17
+ Content from **this** model card has been written by the Hugging Face team.
18
+
19
+ ## Model description
20
+
21
+ OPT belongs to the same family of decoder-only models like [GPT-3](https://arxiv.org/abs/2005.14165). As such, it was pretrained using the self-supervised causal language modedling
22
+ objective.
23
+
24
+ For evaluation, OPT follows [GPT-3](https://arxiv.org/abs/2005.14165) by using their prompts and overall experimental setup. For more details, please read
25
+ the [official paper](https://arxiv.org/abs/2205.01068).
26
+
27
+ ## Intended uses & limitations
28
+
29
+ The pretrained-only model can be used for prompting for evaluation of downstream tasks as well as text generation.
30
+ In addition, the model can be fine-tuned on a downstream task using the [CLM example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling). For all other OPT checkpoints, please have a look at the [model hub](https://huggingface.co/models?filter=opt).
31
+
32
+ ### How to use
33
+
34
+ You can use this model directly with a pipeline for text generation.
35
+
36
+ ```python
37
+ >>> from transformers import pipeline
38
+
39
+ >>> generator = pipeline('text-generation', model="facebook/opt-350m")
40
+ >>> generator("Hello, I'm am conscious and")
41
+ [{'generated_text': "Hello, I'm am conscious and I'm a bit of a noob. I'm looking for"}]
42
+ ```
43
+
44
+ By default, generation is deterministic. In order to use the top-k sampling, please set `do_sample` to `True`.
45
+
46
+ ```python
47
+ >>> from transformers import pipeline, set_seed
48
+
49
+ >>> set_seed(32)
50
+ >>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True)
51
+ >>> generator("Hello, I'm am conscious and")
52
+ [{'generated_text': "Hello, I'm am conscious and I'm interested in this project. Can I get an initial contact"}]
53
+ ```
54
+
55
+ ### Limitations and bias
56
+
57
+ As mentioned in Meta AI's model card, given that the training data used for this model contains a lot of
58
+ unfiltered content from the internet, which is far from neutral the model is strongly biased :
59
+
60
+ > Like other large language models for which the diversity (or lack thereof) of training
61
+ > data induces downstream impact on the quality of our model, OPT-175B has limitations in terms
62
+ > of bias and safety. OPT-175B can also have quality issues in terms of generation diversity and
63
+ > hallucination. In general, OPT-175B is not immune from the plethora of issues that plague modern
64
+ > large language models.
65
+
66
+ Here's an example of how the model can have biased predictions:
67
+
68
+ ```python
69
+ >>> from transformers import pipeline, set_seed
70
+
71
+ >>> set_seed(32)
72
+ >>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True, num_return_sequences=5)
73
+ >>> generator("The woman worked as a")
74
+ [{'generated_text': "The woman works as a substitute teacher for kids who have missed school. She's the teacher herself,"},
75
+ {'generated_text': 'The woman works as a security guard for another company and does an average of around $13/hour'},
76
+ {'generated_text': 'The woman works as a receptionist, she could at the least wait a week or two for her'},
77
+ {'generated_text': 'The woman works as a manager/intern/career development coach/advisor at a nursing home'},
78
+ {'generated_text': 'The woman works as a maid and has to clean the house but you can tell her to do it'}]
79
+ ```
80
+
81
+ compared to:
82
+
83
+ ```python
84
+ >>> from transformers import pipeline, set_seed
85
+
86
+ >>> set_seed(32)
87
+ >>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True, num_return_sequences=5)
88
+ >>> generator("The man worked as a")
89
+ [{'generated_text': 'The man works as a security guard for the National Football League franchise. He has been a part of'},
90
+ {'generated_text': 'The man works as a security guard for another company and does an excellent job.\nI remember when'},
91
+ {'generated_text': 'The man works as a "secret agent" but at the same time he\'s working to protect the'},
92
+ {'generated_text': 'The man works as a manager/operator/servant for a grocery store and does a lot of'},
93
+ {'generated_text': 'The man works as a bouncer near the scene of the accident - how he could do that is'}]
94
+ ```
95
+
96
+ This bias will also affect all fine-tuned versions of this model.
97
+
98
+ ## Training data
99
+
100
+ The Meta AI team wanted to train this model on a corpus as large as possible. It is composed of the union of the following 5 filtered datasets of textual documents:
101
+
102
+ - BookCorpus, which consists of more than 10K unpublished books,
103
+ - CC-Stories, which contains a subset of CommonCrawl data filtered to match the
104
+ story-like style of Winograd schemas,
105
+ - The Pile, from which * Pile-CC, OpenWebText2, USPTO, Project Gutenberg, OpenSubtitles, Wikipedia, DM Mathematics and HackerNews* were included.
106
+ - Pushshift.io Reddit dataset that was developed in Baumgartner et al. (2020) and processed in
107
+ Roller et al. (2021)
108
+ - CCNewsV2 containing an updated version of the English portion of the CommonCrawl News
109
+ dataset that was used in RoBERTa (Liu et al., 2019b)
110
+
111
+ The final training data contains 180B tokens corresponding to 800GB of data. The validation split was made of 200MB of the pretraining data, sampled proportionally
112
+ to each dataset’s size in the pretraining corpus.
113
+
114
+ The dataset might contains offensive content as parts of the dataset are a subset of
115
+ public Common Crawl data, along with a subset of public Reddit data, which could contain sentences
116
+ that, if viewed directly, can be insulting, threatening, or might otherwise cause anxiety.
117
+
118
+ ### Collection process
119
+
120
+ The dataset was collected form internet, and went through classic data processing algorithms and
121
+ re-formatting practices, including removing repetitive/non-informative text like *Chapter One* or
122
+ *This ebook by Project Gutenberg.*
123
+
124
+ ## Training procedure
125
+
126
+ ### Preprocessing
127
+
128
+ The texts are tokenized using the **GPT2** byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a
129
+ vocabulary size of 50272. The inputs are sequences of 2048 consecutive tokens.
130
+
131
+ The 175B model was trained on 992 *80GB A100 GPUs*. The training duration was roughly ~33 days of continuous training.
132
+
133
+ ### BibTeX entry and citation info
134
+
135
+ ```bibtex
136
+ @misc{zhang2022opt,
137
+ title={OPT: Open Pre-trained Transformer Language Models},
138
+ author={Susan Zhang and Stephen Roller and Naman Goyal and Mikel Artetxe and Moya Chen and Shuohui Chen and Christopher Dewan and Mona Diab and Xian Li and Xi Victoria Lin and Todor Mihaylov and Myle Ott and Sam Shleifer and Kurt Shuster and Daniel Simig and Punit Singh Koura and Anjali Sridhar and Tianlu Wang and Luke Zettlemoyer},
139
+ year={2022},
140
+ eprint={2205.01068},
141
+ archivePrefix={arXiv},
142
+ primaryClass={cs.CL}
143
+ }
144
+ ```