Upload README.md with huggingface_hub

README.md

@@ -1,199 +1,163 @@
 ---
-library_name: transformers
-tags: []
----
-
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+language: en
+inference: false
+tags:
+- text-generation
 
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
+license: other
+commercial: false
+---
 
-<!-- This should link to a Dataset Card if possible. -->
 
-[More Information Needed]
+# OPT : Open Pre-trained Transformer Language Models
 
-#### Factors
+OPT was first introduced in [Open Pre-trained Transformer Language Models](https://arxiv.org/abs/2205.01068) and first released in [metaseq's repository](https://github.com/facebookresearch/metaseq) on May 3rd 2022 by Meta AI.
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+**Disclaimer**: The team releasing OPT wrote an official model card, which is available in Appendix D of the [paper](https://arxiv.org/pdf/2205.01068.pdf). 
+Content from **this** model card has been written by the Hugging Face team.
 
-[More Information Needed]
+## Intro
 
-#### Metrics
+To quote the first two paragraphs of the [official paper](https://arxiv.org/abs/2205.01068)
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+> Large language models trained on massive text collections have shown surprising emergent
+> capabilities to generate text and perform zero- and few-shot learning. While in some cases the public
+> can interact with these models through paid APIs, full model access is currently limited to only a
+> few highly resourced labs. This restricted access has limited researchers’ ability to study how and
+> why these large language models work, hindering progress on improving known challenges in areas
+> such as robustness, bias, and toxicity.
 
-[More Information Needed]
+> We present Open Pretrained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M
+> to 175B parameters, which we aim to fully and responsibly share with interested researchers. We train the OPT models to roughly match 
+> the performance and sizes of the GPT-3 class of models, while also applying the latest best practices in data
+> collection and efficient training. Our aim in developing this suite of OPT models is to enable reproducible and responsible research at scale, and
+> to bring more voices to the table in studying the impact of these LLMs. Definitions of risk, harm, bias, and toxicity, etc., should be articulated by the
+> collective research community as a whole, which is only possible when models are available for study.
 
-### Results
+## Model description
 
-[More Information Needed]
+OPT was predominantly pretrained with English text, but a small amount of non-English data is still present within the training corpus via CommonCrawl. The model was pretrained using a causal language modeling (CLM) objective.
+OPT belongs to the same family of decoder-only models like [GPT-3](https://arxiv.org/abs/2005.14165). As such, it was pretrained using the self-supervised causal language modedling objective.
 
-#### Summary
+For evaluation, OPT follows [GPT-3](https://arxiv.org/abs/2005.14165) by using their prompts and overall experimental setup. For more details, please read 
+the [official paper](https://arxiv.org/abs/2205.01068).
 
+## Intended uses & limitations
 
+The pretrained-only model can be used for prompting for evaluation of downstream tasks as well as text generation.
+In addition, the model can be fine-tuned on a downstream task using the [CLM example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling). For all other OPT checkpoints, please have a look at the [model hub](https://huggingface.co/models?filter=opt).
 
-## Model Examination [optional]
+### How to use
 
-<!-- Relevant interpretability work for the model goes here -->
+You can use this model directly with a pipeline for text generation.
 
-[More Information Needed]
+```python
+>>> from transformers import pipeline
 
-## Environmental Impact
+>>> generator = pipeline('text-generation', model="facebook/opt-350m")
+>>> generator("What are we having for dinner?")
+[{'generated_text': "What are we having for dinner?\nI'm having a steak and a salad.\nI'm""}]
+```
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+By default, generation is deterministic. In order to use the top-k sampling, please set `do_sample` to `True`. 
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+```python
+>>> from transformers import pipeline, set_seed
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+>>> set_seed(32)
+>>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True)
+>>> generator("What are we having for dinner?")
+[{'generated_text': "What are we having for dinner?\n\nWith spring fast approaching, it’s only appropriate"}]
+```
 
-## Technical Specifications [optional]
+### Limitations and bias
 
-### Model Architecture and Objective
+As mentioned in Meta AI's model card, given that the training data used for this model contains a lot of
+unfiltered content from the internet, which is far from neutral the model is strongly biased : 
 
-[More Information Needed]
+> Like other large language models for which the diversity (or lack thereof) of training
+> data induces downstream impact on the quality of our model, OPT-175B has limitations in terms
+> of bias and safety. OPT-175B can also have quality issues in terms of generation diversity and
+> hallucination. In general, OPT-175B is not immune from the plethora of issues that plague modern
+> large language models. 
 
-### Compute Infrastructure
+Here's an example of how the model can have biased predictions:
 
-[More Information Needed]
+```python
+>>> from transformers import pipeline, set_seed
 
-#### Hardware
+>>> set_seed(32)
+>>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True, num_return_sequences=5)
+>>> generator("The woman worked as a")
+[{'generated_text': "The woman works as a substitute teacher for kids who have missed school. She's the teacher herself,"},
+ {'generated_text': 'The woman works as a security guard for another company and does an average of around $13/hour'},
+ {'generated_text': 'The woman works as a receptionist, she could at the least wait a week or two for her'},
+ {'generated_text': 'The woman works as a manager/intern/career development coach/advisor at a nursing home'},
+ {'generated_text': 'The woman works as a maid and has to clean the house but you can tell her to do it'}]
+```
 
-[More Information Needed]
+compared to:
 
-#### Software
+```python
+>>> from transformers import pipeline, set_seed
 
-[More Information Needed]
+>>> set_seed(32)
+>>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True, num_return_sequences=5)
+>>> generator("The man worked as a")
+[{'generated_text': 'The man works as a security guard for the National Football League franchise. He has been a part of'},
+ {'generated_text': 'The man works as a security guard for another company and does an excellent job.\nI remember when'},
+ {'generated_text': 'The man works as a "secret agent" but at the same time he\'s working to protect the'},
+ {'generated_text': 'The man works as a manager/operator/servant for a grocery store and does a lot of'},
+ {'generated_text': 'The man works as a bouncer near the scene of the accident - how he could do that is'}]
+ ```
 
-## Citation [optional]
+This bias will also affect all fine-tuned versions of this model.
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+## Training data
 
-**BibTeX:**
+The Meta AI team wanted to train this model on a corpus as large as possible. It is composed of the union of the following 5 filtered datasets of textual documents: 
 
-[More Information Needed]
+  - BookCorpus, which consists of more than 10K unpublished books,
+  - CC-Stories, which contains a subset of CommonCrawl data filtered to match the
+story-like style of Winograd schemas,
+  - The Pile, from which * Pile-CC, OpenWebText2, USPTO, Project Gutenberg, OpenSubtitles, Wikipedia, DM Mathematics and HackerNews* were included. 
+  - Pushshift.io Reddit dataset that was developed in Baumgartner et al. (2020) and processed in
+Roller et al. (2021)
+  - CCNewsV2 containing an updated version of the English portion of the CommonCrawl News
+dataset that was used in RoBERTa (Liu et al., 2019b)
 
-**APA:**
+The final training data contains 180B tokens corresponding to 800GB of data. The validation split was made of 200MB of the pretraining data, sampled proportionally
+to each dataset’s size in the pretraining corpus. 
 
-[More Information Needed]
+The dataset might contains offensive content as parts of the dataset are a subset of
+public Common Crawl data, along with a subset of public Reddit data, which could contain sentences
+that, if viewed directly, can be insulting, threatening, or might otherwise cause anxiety.
 
-## Glossary [optional]
+### Collection process
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+The dataset was collected form internet, and went through classic data processing algorithms  and
+re-formatting practices, including removing repetitive/non-informative text like *Chapter One* or
+*This ebook by Project Gutenberg.*
 
-[More Information Needed]
+## Training procedure
 
-## More Information [optional]
+### Preprocessing
 
-[More Information Needed]
+The texts are tokenized using the **GPT2** byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a
+vocabulary size of 50272. The inputs are sequences of 2048 consecutive tokens.
 
-## Model Card Authors [optional]
+The 175B model was trained on 992 *80GB A100 GPUs*. The training duration was roughly ~33 days of continuous training.
 
-[More Information Needed]
+### BibTeX entry and citation info
 
-## Model Card Contact
+```bibtex
+@misc{zhang2022opt,
+      title={OPT: Open Pre-trained Transformer Language Models}, 
+      author={Susan Zhang and Stephen Roller and Naman Goyal and Mikel Artetxe and Moya Chen and Shuohui Chen and Christopher Dewan and Mona Diab and Xian Li and Xi Victoria Lin and Todor Mihaylov and Myle Ott and Sam Shleifer and Kurt Shuster and Daniel Simig and Punit Singh Koura and Anjali Sridhar and Tianlu Wang and Luke Zettlemoyer},
+      year={2022},
+      eprint={2205.01068},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
 
-[More Information Needed]