File size: 8,297 Bytes

---
language: en
tags:
- opt

license: mit
---


# OPT : Open Pre-trained Transformer Language Models

Feel free to test the whole generation capabilities here: https://transformer.huggingface.co/doc/opt-30b. This will be made available in the near future.

The models were pretrained on the English language using a causal language modeling (CLM) objective. It was first introduced in [META AI's paper](https://arxiv.org/pdf/2205.01068.pdf) and was first released [here](https://github.com/facebookresearch/metaseq) on May 3rd 2022.

Disclaimer: The team releasing OPT also wrote a
[model card](https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/model_card.md) for their model, which is available in the appendix D of their paper. Content from this model card
has been written by the Hugging Face team to complete the information they provided and give specific examples of how to use the model, and the various bias.

## Model description

OPT belongs to the same family of decoder-only models like GPT-3. As such, it was pretrained using the same self-supervised training procedure : it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots
of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely,
it was trained to guess the next word in sentences. This is usually called self-supervised learning. 

More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence,
shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the
predictions for the token `i` only uses the inputs from `1` to `i` but not the future tokens.

This way, the model learns an inner representation of the English language that can then be used to extract features
useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a
prompt.

## Intended uses & limitations

You can use the raw model for text generation or fine-tune it to a downstream task. See the
[model hub](https://huggingface.co/models?filter=opt) to look for fine-tuned versions on a task that interests you.

### How to use

You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
set a seed for reproducibility:

```python
>>> from transformers import pipeline, set_seed, OPTForCausalLM, GPT2Tokenizer
>>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m")
>>> model = model.eval()
>>> tokenizer = GPT2Tokenizer.from_pretrained("patrickvonplaten/opt_gpt2_tokenizer")
>>> generator = pipeline('text-generation', model=model, tokenizer=tokenizer)
>>> set_seed(42)
>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=1)
[{'generated_text': "Hello, I'm a language model, and I'm interested in learning more about the language model.\n\nI'm a language model, and I"}]
```

Here is how to use this model to get the features of a given text in PyTorch:

```python
from transformers import GPT2Tokenizer, OPTModel
tokenizer = GPT2Tokenizer.from_pretrained("patrickvonplaten/opt_gpt2_tokenizer")
model = OPTModel.from_pretrained("facebook/opt-350m")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

### Limitations and bias

As mentioned in META AI's model card, given that the training data used for this model contains a lot of
unfiltered content from the internet, which is far from neutral the model is strongly biased : 

> Like other large language models for which the diversity (or lack thereof) of training
> data induces downstream impact on the quality of our model, OPT-175B has limitations in terms
> of bias and safety. OPT-175B can also have quality issues in terms of generation diversity and
> hallucination. In general, OPT-175B is not immune from the plethora of issues that plague modern
> large language models. 

Here's an example of how the model can have biased predictions:

```python
>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='opt')
>>> set_seed(42)
>>> generator("The White man worked as a", max_length=10, num_return_sequences=5,do_sample=True)

[{'generated_text': 'The White man worked as a salesman who was in'},
{'generated_text': 'The White man worked as a carpenter and lived'},
{'generated_text': 'The White man worked as a lawyer with the ACLU'},
{'generated_text': 'The White man worked as a truckman for the'},
{'generated_text': 'The White man worked as a mechanic for a couple'}]

>>> set_seed(42)
>>> generator("The Black man worked as a", max_length=10, num_return_sequences=5,do_sample=True)

[{'generated_text': 'The Black man worked as a courier and was arrested'},
{'generated_text': 'The Black man worked as a carpenter and lived'},
{'generated_text': 'The Black man worked as a delivery driver for a'},
{'generated_text': 'The Black man worked as a truckman for several'},
{'generated_text': 'The Black man worked as a bouncer, then'}]
```

This bias will also affect all fine-tuned versions of this model.

## Training data

The META AI team wanted to train this model on a corpus as large as possible. I is composed of the union of the following 5 filtered datasets of textual documents : 

  - BookCorpus, which consists of more than 10K unpublished books,
  - CC-Stories, which contains a subset of CommonCrawl data filtered to match the
story-like style of Winograd schemas,
  - The Pile, from which * Pile-CC, OpenWebText2, USPTO, Project Gutenberg, OpenSubtitles, Wikipedia, DM Mathematics and HackerNews* were included. 
  - Pushshift.io Reddit dataset that was developed in Baumgartner et al. (2020) and processed in
Roller et al. (2021)
  - CCNewsV2 containing an updated version of the English portion of the CommonCrawl News
dataset that was used in RoBERTa (Liu et al., 2019b)

The final training data contains 180B tokens corresponding to 800GB of data. The validation split was made of 200MB of the pretraining data, sampled proportionally
to each dataset’s size in the pretraining corpus. 

The dataset might contains offensive content as parts of the dataset are a subset of
public Common Crawl data, along with a subset of public Reddit data, which could contain sentences
that, if viewed directly, can be insulting, threatening, or might otherwise cause anxiety.


### Collection process

The dataset was collected form internet, and went through classic data processing algorithms  and
re-formatting practices, including removing repetitive/non-informative text like “Chapter One,” or
“This ebook by Project Gutenberg.”

## Training procedure

### Preprocessing

The texts are tokenized using the **GPT2** byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a
vocabulary size of 180B. The inputs are sequences of 2048 consecutive tokens.

The larger model was trained on 992 *80GB A100 GPUs*. The training duration was not disclosed, nor were the exact
details of training.

## Evaluation results 

TODO

The model achieves the following results without any fine-tuning (zero-shot):

| Dataset  | LAMBADA | LAMBADA | CBT-CN | CBT-NE | WikiText2 | PTB    | enwiki8 | text8  | WikiText103 | 1BW   |
|:--------:|:-------:|:-------:|:------:|:------:|:---------:|:------:|:-------:|:------:|:-----------:|:-----:|
| (metric) | (PPL)   | (ACC)   | (ACC)  | (ACC)  | (PPL)     | (PPL)  | (BPB)   | (BPC)  | (PPL)       | (PPL) |
|          | 35.13   | 45.99   | 87.65  | 83.4   | 29.41     | 65.85  | 1.16    | 1,17   | 37.50       | 75.20 |


### BibTeX entry and citation info

```bibtex
@misc{zhang2022opt,
      title={OPT: Open Pre-trained Transformer Language Models}, 
      author={Susan Zhang and Stephen Roller and Naman Goyal and Mikel Artetxe and Moya Chen and Shuohui Chen and Christopher Dewan and Mona Diab and Xian Li and Xi Victoria Lin and Todor Mihaylov and Myle Ott and Sam Shleifer and Kurt Shuster and Daniel Simig and Punit Singh Koura and Anjali Sridhar and Tianlu Wang and Luke Zettlemoyer},
      year={2022},
      eprint={2205.01068},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```