README.md

---
language: en
tags:
- summarization
model-index:
- name: google/pegasus-xsum
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: samsum
      type: samsum
      config: samsum
      split: train
    metrics:
    - type: rouge
      value: 21.8096
      name: ROUGE-1
      verified: true
    - type: rouge
      value: 4.2525
      name: ROUGE-2
      verified: true
    - type: rouge
      value: 17.4469
      name: ROUGE-L
      verified: true
    - type: rouge
      value: 18.8907
      name: ROUGE-LSUM
      verified: true
    - type: loss
      value: 3.0317161083221436
      name: loss
      verified: true
    - type: gen_len
      value: 20.3122
      name: gen_len
      verified: true
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: xsum
      type: xsum
      config: default
      split: test
    metrics:
    - type: rouge
      value: 46.7782
      name: ROUGE-1
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzk4Njc5YTQyZDJhNWNmMWNiMDdmOGY3NGZkOTE5ODYxZWI1YzllYzVhZDBmZTdhMTUzYzBhYjg4NDExMDI0OCIsInZlcnNpb24iOjF9.FB6f5FsSE8JuwyPUC1usCF0GXFx4y7YnxNkkhu0xyuv1vG-8y2plnJqSfF30Jae1Bpb_6IGqtnCisuvC9_d_AA
    - type: rouge
      value: 24.3976
      name: ROUGE-2
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjg4ZTg0ZjRmNGFiMTY0MjVlNjBkOGI4NzhkYjE3M2YyMDhjOWY1MTVmMzBjMmQ4Y2ViNWQ3NGU0OGQzMmJhYiIsInZlcnNpb24iOjF9.DELSboK4-QhPB_JJvX9tBZDCMc73F-n7yqKUesEiAd7rMjPAc8RLJcO_1SBxLVc0w1Pxt84Z0V-Fz8Ee-LGwDg
    - type: rouge
      value: 38.9758
      name: ROUGE-L
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzQzNWY4Y2YxZTZjOGM3YzdmNTYxMTc0ZDJmZjNjNzEyZTdlMzYzZTMyYTcyZDgwZGZiZjNmZWQ4MzA3Y2UwMiIsInZlcnNpb24iOjF9.tMfwcvdN558uEuSa9aUXDR06q0jPKy-6s3f1h8LkO9lc7JV5oy9SSnsDXQNALIyzh3FhmyScegEcXr0LLIwUBA
    - type: rouge
      value: 39.0386
      name: ROUGE-LSUM
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTA3YjI3MWVmOWJjZDk1YzkyMDJlYzk0MjQyYzQ1MjZhMjI2YWQ3Y2Y2ZGZiNGJjOWFhOWU2NDNkMzQxMWQzZSIsInZlcnNpb24iOjF9._XvQukx6SpEEjOHf3ivplJ8YW5_Q7oj8mc1uu5YIJaXyK9yuf9HW1DhXFxYdUm_K_cAtSRa5PPCGeKkDJfTvDQ
    - type: loss
      value: 1.5713257789611816
      name: loss
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODhhNDFkMjdhNmI0MDc4NWFkYjkzZTc2OGM5MTY4NGMwZDE0NWZhMTBmZmY5ZGMyMWU5NTY3MjFjZWZkZTdmYiIsInZlcnNpb24iOjF9.PJcC1UpQpfSz44f8mQN5gp5ZFbEbDtRPLzK5RoPjTirRJ4cDPxX88yLI3rDiUMZRdXitEaWqQpLkFqu-5g75Bw
    - type: gen_len
      value: 23.089
      name: gen_len
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOGNlMDZmNjRjNWM1YTg0M2FmNDg4ZGE2OGMzYjc4MmE3MTk3YTQzNzM3ZmJmZmJhNDVlMGZlYWNiOGJmYmFlMSIsInZlcnNpb24iOjF9.w-ce3jWHW2dzLFaJe2R9hAiCvIdX-SIcrCe5ADTCDyBQwLrHOJf8-xFYLt9oE9EAlXJsbrhjlCMJbzFChNQTBg
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: cnn_dailymail
      type: cnn_dailymail
      config: 3.0.0
      split: test
    metrics:
    - type: rouge
      value: 22.2062
      name: ROUGE-1
      verified: true
    - type: rouge
      value: 7.6701
      name: ROUGE-2
      verified: true
    - type: rouge
      value: 15.4046
      name: ROUGE-L
      verified: true
    - type: rouge
      value: 19.2182
      name: ROUGE-LSUM
      verified: true
    - type: loss
      value: 2.681241273880005
      name: loss
      verified: true
    - type: gen_len
      value: 25.0234
      name: gen_len
      verified: true
---

### Pegasus Models
See Docs: [here](https://huggingface.co/transformers/master/model_doc/pegasus.html)

Original TF 1 code [here](https://github.com/google-research/pegasus)

Authors: Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019

Maintained by: [@sshleifer](https://twitter.com/sam_shleifer)

Task: Summarization

The following is copied from the authors' README.

# Mixed & Stochastic Checkpoints

We train a pegasus model with sampled gap sentence ratios on both C4 and HugeNews, and stochastically sample important sentences. The updated the results are reported in this table.

| dataset | C4 | HugeNews | Mixed & Stochastic|
| ---- | ---- | ---- | ----|
| xsum | 45.20/22.06/36.99 | 47.21/24.56/39.25 | 47.60/24.83/39.64|
| cnn_dailymail | 43.90/21.20/40.76 | 44.17/21.47/41.11 | 44.16/21.56/41.30|
| newsroom | 45.07/33.39/41.28 | 45.15/33.51/41.33 | 45.98/34.20/42.18|
| multi_news | 46.74/17.95/24.26 | 47.52/18.72/24.91 | 47.65/18.75/24.95|
| gigaword | 38.75/19.96/36.14 | 39.12/19.86/36.24 | 39.65/20.47/36.76|
| wikihow | 43.07/19.70/34.79 | 41.35/18.51/33.42 | 46.39/22.12/38.41 *|
| reddit_tifu | 26.54/8.94/21.64 | 26.63/9.01/21.60 | 27.99/9.81/22.94|
| big_patent | 53.63/33.16/42.25 | 53.41/32.89/42.07 | 52.29/33.08/41.66 *|
| arxiv | 44.70/17.27/25.80 | 44.67/17.18/25.73 | 44.21/16.95/25.67|
| pubmed | 45.49/19.90/27.69 | 45.09/19.56/27.42 | 45.97/20.15/28.25|
| aeslc | 37.69/21.85/36.84 | 37.40/21.22/36.45 | 37.68/21.25/36.51|
| billsum | 57.20/39.56/45.80 | 57.31/40.19/45.82 | 59.67/41.58/47.59|

The "Mixed & Stochastic" model has the following changes:
- trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples). 
- trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity).
- the model uniformly sample a gap sentence ratio between 15% and 45%.
- importance sentences are sampled using a 20% uniform noise to importance scores.
- the sentencepiece tokenizer is updated to be able to encode newline character.


(*) the numbers of wikihow and big_patent datasets are not comparable because of change in tokenization and data:
- wikihow dataset contains newline characters which is useful for paragraph segmentation, the C4 and HugeNews model's sentencepiece tokenizer doesn't encode newline and loose this information.
- we update the BigPatent dataset to preserve casing, some format cleanings are also changed, please refer to change in TFDS.


The "Mixed & Stochastic" model has the following changes (from pegasus-large in the paper):


trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples).
trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity).
the model uniformly sample a gap sentence ratio between 15% and 45%.
importance sentences are sampled using a 20% uniform noise to importance scores.
the sentencepiece tokenizer is updated to be able to encode newline character.


Citation
```


@misc{zhang2019pegasus,
    title={PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization},
    author={Jingqing Zhang and Yao Zhao and Mohammad Saleh and Peter J. Liu},
    year={2019},
    eprint={1912.08777},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```