opt-125m_eval / README.md
MicPie's picture
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator (#17)
30eabc0
metadata
language: en
license: other
tags:
  - text-generation
  - opt
inference: false
commercial: false
model-index:
  - name: inverse-scaling/opt-125m_eval
    results:
      - task:
          type: zero-shot-classification
          name: Zero-Shot Text Classification
        dataset:
          name: inverse-scaling/NeQA
          type: inverse-scaling/NeQA
          config: inverse-scaling--NeQA
          split: train
        metrics:
          - type: accuracy
            value: 0.4666666666666667
            name: Accuracy
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMjBkYzg3OGQ2NGEwMzE3MmRlNDNjOTQ5YjI2ZmY5ZmExYmMwZGMzOGU4MDM5NmUxMmM0MzlmNmU3OGMxOWNlNyIsInZlcnNpb24iOjF9.6hSSu8iq_f8MCiI3vaVEE2x-Z_7SfVSXu2vEIGggKG1Z1oC1E3-Y7VbZM7cMJKzRvcskLBFaRHYoaU2uZi5gCA
          - type: loss
            value: 0.9069941281403104
            name: Loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTNhMDE3NGEyY2UwN2M4ZTNlYjA0YjM1OWZiNWI4MWRjYmRkOGFjMDA2YjZkZWM0YjczMjRhZDIxMmQxMmQ3MCIsInZlcnNpb24iOjF9.ngIQdf8pOt8WcuIo6_vR5nsLCuazdU2605JI-cvjuG6uyBfAE7xWV-ZLqqVZ85cfpGGso1e3FDcnjNgCuS19CQ
      - task:
          type: zero-shot-classification
          name: Zero-Shot Text Classification
        dataset:
          name: inverse-scaling/quote-repetition
          type: inverse-scaling/quote-repetition
          config: inverse-scaling--quote-repetition
          split: train
        metrics:
          - type: accuracy
            value: 0.96
            name: Accuracy
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzk1NTY4YmYzMzE3OGQ2OGM4NjljNmM0NTc0MWMxZTI3MGI3OTBkMzE3OTJkMjRiYzU2OGUwMjdhMTY1Y2M0MyIsInZlcnNpb24iOjF9.1uGnbKuVoPXeK2zF3nIqAPUeiWodBA78BhDgHk-8Kq9Vh6WtvcL0qwOvQVLjjPmL_7G56Y0d6cuXWycACwuhAQ
          - type: loss
            value: 0.04267331124324727
            name: Loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNGI3MTBiODBlNjNlZGExNzBhMjgxNjNhNDQ5OGQ5YTBjMjQzNTMwNWQ3MDY3NWY2NzJjOGYzNmFjZTE2ODYzNyIsInZlcnNpb24iOjF9.OoXOKgtCjrB3iku_GtinmPFeFdMJWExa2N-VbKKoymMX9pQJ3Wh9cVbKWI2nTHsoTQI_lu_3s9ZjVVk7_v9zAA
      - task:
          type: zero-shot-classification
          name: Zero-Shot Text Classification
        dataset:
          name: inverse-scaling/redefine-math
          type: inverse-scaling/redefine-math
          config: inverse-scaling--redefine-math
          split: train
        metrics:
          - type: accuracy
            value: 0.7566666666666667
            name: Accuracy
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTRkMzAyYzcwOGZmNDVhMTMwOGQxOWVhZDE2NzVkMGRkNDJjNzFlMjZkNDFlZDMyZTA0YjYwNTBjNTBlODg2NCIsInZlcnNpb24iOjF9.Mxc3griLDkTEYTJyF0EamDwHEtzN2IkiXKYY9HmIl6HbHvLoJn9Qz1Ot6EE_T0VJbL11Ih7XOgELgiZ35XU3Cw
          - type: loss
            value: 0.5209774699724383
            name: Loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZjZiZjIzZGUyOGFjODU2ZDk4N2ZmMjc5MmZkY2NmODAyNDhjODQ1MDZiMDc0NDdlM2VmZDc2ZWRhMmFjM2ZhMyIsInZlcnNpb24iOjF9.rWg9_9Z5YtqgO7H61K8w1cp_7GTGsyRpMhACpqioXSnQ6z0sL-rtkwb1QKjD0yQH3MEHr2Grwsh7iUmY0nWjDQ
      - task:
          type: zero-shot-classification
          name: Zero-Shot Text Classification
        dataset:
          name: inverse-scaling/hindsight-neglect-10shot
          type: inverse-scaling/hindsight-neglect-10shot
          config: inverse-scaling--hindsight-neglect-10shot
          split: train
        metrics:
          - type: accuracy
            value: 0.5047619047619047
            name: Accuracy
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTAxMTI4OWNkNzQ0NTZjOGZhNWJmYjBlZGMyMjg2YjJjZWJjNzU1MmIzNWM5MTg5MzhjYmQ0YzI5NzM5NTVjZiIsInZlcnNpb24iOjF9.dzv4FTu8IIWWu8V497AzCWSjytzv_PnxriQ9aWOUd6AkQCOZQeCLrLYLifoK_BJ2SBcuBum6TS-Ukx9MalklAA
          - type: loss
            value: 0.8965487285916295
            name: Loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2ExZjE2ZWIxODBjZTA0OTI1NzI0NTRlMTIxNDI1YjA4OTM5YzVkMzc4N2MzZTc4ZTA4OTFiYTlkMjcyYjY0MiIsInZlcnNpb24iOjF9.FjnpzThx7mRfh1U_R12KCUJ2wDxjaEKQC3iSSVAvzP1xXLESxA4c014Xzucw1Ugaq_P8s5ySzlPgGUp7qqTtBA
      - task:
          type: zero-shot-classification
          name: Zero-Shot Text Classification
        dataset:
          name: mathemakitten/winobias_antistereotype_test_cot_v3
          type: mathemakitten/winobias_antistereotype_test_cot_v3
          config: mathemakitten--winobias_antistereotype_test_cot_v3
          split: test
        metrics:
          - type: accuracy
            value: 0.47815533980582525
            name: Accuracy
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDdkNmEwOTQwZTI4MzE4YjlmNjIwZTIxMWM3YWM1YzYyMWM1OTY1YmZkNjhjYmUyZjJjZjZkMTljNjZlMzUwYyIsInZlcnNpb24iOjF9.PLaVz67JgdncUXDz3BXmZC41HKVl3_D1Iz5cgygbn2y4OsfVyvsyvU3GFqKgPb-gvXT4xGMxkV0FvA28gjTGDw
          - type: loss
            value: 0.8500587756725001
            name: Loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODFjNjUwMWI2Y2UwNzQ0NDE4NTU1NGI3YzQyMDNhOWU3YjU0MGRhMjEyZjNkMzczYWU2MDY0NGIyZmM5MWY5OCIsInZlcnNpb24iOjF9.9VQeAZ_lvyKC2RNQ2GmqSrxXCz2W8NZz14JhF3j4boBHXRm1V07wml6uNW_GfDt6Qwiu5IZCqMdvCavacDUoDw
      - task:
          type: zero-shot-classification
          name: Zero-Shot Text Classification
        dataset:
          name: mathemakitten/winobias_antistereotype_test_v5
          type: mathemakitten/winobias_antistereotype_test_v5
          config: mathemakitten--winobias_antistereotype_test_v5
          split: test
        metrics:
          - type: accuracy
            value: 0.5024271844660194
            name: Accuracy
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDRjYzkzZDI1MDRjY2JiNDUyNGJmNmVlZTMxYmJjODIzNDc2NGI3MzBjN2RkNGRjZjg5ZjJiYjM1ODQyMjQyMyIsInZlcnNpb24iOjF9.uLQjZb34N0QHPgeMnJkPk3xG3VI4Z_djPpCvah29a9D0fOHMuqdqynnySODmwfdbKecEV5za8wUf6_ny4qktDQ
          - type: loss
            value: 0.8860152396463484
            name: Loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOWIzODA0ZjExNzJiMDBjNzlkYzFjMzk2NGMxNzM0ODQyNmFhMDczM2EwMWU1N2VjMjcxNGEzMTdjN2IyNDJhNSIsInZlcnNpb24iOjF9.ipVZVlS7Rey-vsqEhAmOjcz4pkl85Brn8i1aTc4eSXQ2KgG5ScuAgeIVcxe3EbCSJsRkJowRqRqqWKBodiyAAQ

OPT : Open Pre-trained Transformer Language Models

OPT was first introduced in Open Pre-trained Transformer Language Models and first released in metaseq's repository on May 3rd 2022 by Meta AI.

Disclaimer: The team releasing OPT wrote an official model card, which is available in Appendix D of the paper. Content from this model card has been written by the Hugging Face team.

Intro

To quote the first two paragraphs of the official paper

Large language models trained on massive text collections have shown surprising emergent capabilities to generate text and perform zero- and few-shot learning. While in some cases the public can interact with these models through paid APIs, full model access is currently limited to only a few highly resourced labs. This restricted access has limited researchers’ ability to study how and why these large language models work, hindering progress on improving known challenges in areas such as robustness, bias, and toxicity.

We present Open Pretrained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We train the OPT models to roughly match the performance and sizes of the GPT-3 class of models, while also applying the latest best practices in data collection and efficient training. Our aim in developing this suite of OPT models is to enable reproducible and responsible research at scale, and to bring more voices to the table in studying the impact of these LLMs. Definitions of risk, harm, bias, and toxicity, etc., should be articulated by the collective research community as a whole, which is only possible when models are available for study.

Model description

OPT was predominantly pretrained with English text, but a small amount of non-English data is still present within the training corpus via CommonCrawl. The model was pretrained using a causal language modeling (CLM) objective. OPT belongs to the same family of decoder-only models like GPT-3. As such, it was pretrained using the self-supervised causal language modedling objective.

For evaluation, OPT follows GPT-3 by using their prompts and overall experimental setup. For more details, please read the official paper.

Intended uses & limitations

The pretrained-only model can be used for prompting for evaluation of downstream tasks as well as text generation. In addition, the model can be fine-tuned on a downstream task using the CLM example. For all other OPT checkpoints, please have a look at the model hub.

How to use

You can use this model directly with a pipeline for text generation.

>>> from transformers import pipeline

>>> generator = pipeline('text-generation', model="facebook/opt-125m")
>>> generator("Hello, I'm am conscious and")
[{'generated_text': 'Hello, I am conscious and aware of the fact that I am a woman. I am aware of'}]

By default, generation is deterministic. In order to use the top-k sampling, please set do_sample to True.

>>> from transformers import pipeline, set_seed

>>> set_seed(32)
>>> generator = pipeline('text-generation', model="facebook/opt-125m", do_sample=True)
>>> generator("Hello, I'm am conscious and")
[{'generated_text': 'Hello, I am conscious and active member of the Khaosan Group, a private, self'}]

Limitations and bias

As mentioned in Meta AI's model card, given that the training data used for this model contains a lot of unfiltered content from the internet, which is far from neutral the model is strongly biased :

Like other large language models for which the diversity (or lack thereof) of training data induces downstream impact on the quality of our model, OPT-175B has limitations in terms of bias and safety. OPT-175B can also have quality issues in terms of generation diversity and hallucination. In general, OPT-175B is not immune from the plethora of issues that plague modern large language models.

This bias will also affect all fine-tuned versions of this model.

Training data

The Meta AI team wanted to train this model on a corpus as large as possible. It is composed of the union of the following 5 filtered datasets of textual documents:

  • BookCorpus, which consists of more than 10K unpublished books,
  • CC-Stories, which contains a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas,
  • The Pile, from which * Pile-CC, OpenWebText2, USPTO, Project Gutenberg, OpenSubtitles, Wikipedia, DM Mathematics and HackerNews* were included.
  • Pushshift.io Reddit dataset that was developed in Baumgartner et al. (2020) and processed in Roller et al. (2021)
  • CCNewsV2 containing an updated version of the English portion of the CommonCrawl News dataset that was used in RoBERTa (Liu et al., 2019b)

The final training data contains 180B tokens corresponding to 800GB of data. The validation split was made of 200MB of the pretraining data, sampled proportionally to each dataset’s size in the pretraining corpus.

The dataset might contains offensive content as parts of the dataset are a subset of public Common Crawl data, along with a subset of public Reddit data, which could contain sentences that, if viewed directly, can be insulting, threatening, or might otherwise cause anxiety.

Collection process

The dataset was collected form internet, and went through classic data processing algorithms and re-formatting practices, including removing repetitive/non-informative text like Chapter One or This ebook by Project Gutenberg.

Training procedure

Preprocessing

The texts are tokenized using the GPT2 byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a vocabulary size of 50272. The inputs are sequences of 2048 consecutive tokens.

The 175B model was trained on 992 80GB A100 GPUs. The training duration was roughly ~33 days of continuous training.

BibTeX entry and citation info

@misc{zhang2022opt,
      title={OPT: Open Pre-trained Transformer Language Models}, 
      author={Susan Zhang and Stephen Roller and Naman Goyal and Mikel Artetxe and Moya Chen and Shuohui Chen and Christopher Dewan and Mona Diab and Xian Li and Xi Victoria Lin and Todor Mihaylov and Myle Ott and Sam Shleifer and Kurt Shuster and Daniel Simig and Punit Singh Koura and Anjali Sridhar and Tianlu Wang and Luke Zettlemoyer},
      year={2022},
      eprint={2205.01068},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}