---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- generated_from_trainer
- stacked summaries
- xsum
datasets:
- stacked-summaries/stacked-xsum-1024
pipeline_tag: summarization
model-index:
- name: flan-t5-large-stacked-XSUM-1024-WIP-2p8-850-stacked-xsum-1024-evaluated
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: xsum
      type: xsum
      config: default
      split: test
    metrics:
    - type: rouge
      value: 39.3614
      name: ROUGE-1
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMWZmZDNhNWM5YjcyMzVjNjUwMWE1NDg4YmRiNGMwY2EyZDYzMGZkY2NlNWE0MzQwNDYzN2JkNzYyOGUxNmI3ZiIsInZlcnNpb24iOjF9.1ucBm8VOqZgLXmUyDkPisiFfHJ8VYvOdvUsk6R_F0QGLIBXOCf2s_pbqHauTyEQM2mAn762DpR5L4AZg7hF_BA
    - type: rouge
      value: 17.5887
      name: ROUGE-2
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDU3MDQwNjYzMTE2MjU5NTE0ODU1ZmI2ZjhlY2QxODA3YTYyOWExZDdiM2Y4YzZhMTU3N2IwMGQ4M2MxMTNmZiIsInZlcnNpb24iOjF9.lb6R_xg5R1TABUCSRgvEGmdkxhSRavrfllxhsk_NxKA53EC4MXeE6o7nRWPoo2nrBOb5Lcajy_5y4oPOkv84Ag
    - type: rouge
      value: 32.6489
      name: ROUGE-L
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmFkOTc2MTIxMmYyNTY2MWE3Y2E4ZWYwODQ5MmU3NTIxZWM2Yzg2ZDNkYjE3NDgzM2VjYTMwOTkxNjQ1YmIyYiIsInZlcnNpb24iOjF9.AAAh5SnRDnTMCEXMfEp9N7pwHITv-crNloZTnbW7TMPXtMUe7vzATOxGVMZpMe-Nsf3Wkc3JbUdaZZ9bOb17Ag
    - type: rouge
      value: 32.6435
      name: ROUGE-LSUM
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjg1ZmNkODZlMzdkODA4MDUxMGQyNjFiMTkyYjIzMTE2NGMyOWQ1NmQ2YjY0OTRmZjVjZWNhODBiOWI1YzVlOCIsInZlcnNpb24iOjF9.GUVl2J3DCRQUqueSuCsFM8v7IDXH7EATFlQbFl730Bo8Y2aolA-V9uN7pkaU9IM1wWBz7hvILElBCE0sln6SAQ
    - type: loss
      value: 1.4964560270309448
      name: loss
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTViZTkwMzQ3MGNlZDJhNTk3NDE5NzBkMDZjMGEyNzNkZTI4ZmJhMWRlYTMwNmRmN2JhNzdkNTQ3N2FlODBmNyIsInZlcnNpb24iOjF9.lNWUw12R20SwZMZEuUnxYsWrkFBNoU9_5ZOiuFF5aT9QsHJC-FSmZ8DXTdVudv6J-BoeA-l5KYowr7GJfbzlDQ
    - type: gen_len
      value: 18.7302
      name: gen_len
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWM2MWQzN2YyY2U0NWZhZGFkMjk0MzFlMTA1YTgxYzAzYjhhZmVmZDI5Mzk4ODgzOGU1NjVhNTk3NmYyNzhkMSIsInZlcnNpb24iOjF9.bL56u1G46OIwdIqZJ-6og_T2yCKFTXrlPQeguZps3ixXokfKqlfCDxz3641yKA3AdMlLe5lDcN3UQReHtiWwBg
---


# flan-t5-large-stacked-XSUM-1024

<a href="https://colab.research.google.com/gist/pszemraj/561263b04b33d5aec04a18f572d68011/brief-demo-flan-t5-stacked-xsum.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

This model is a fine-tuned version of [google/flan-t5-large](https://huggingface.co/google/flan-t5-large) on the stacked-summaries/stacked-xsum-1024 dataset.

It achieves the following results on the evaluation set:
- eval_loss: 1.3314
- eval_rouge1: 46.5061
- eval_rouge2: 22.0588
- eval_rougeL: 37.5235
- eval_rougeLsum: 39.0234
- eval_gen_len: 46.1807
- eval_runtime: 9456.3608
- eval_samples_per_second: 1.896
- eval_steps_per_second: 0.119

> Note that the evaluation set is `stacked-summaries/stacked-xsum-1024` and not `xsum` itself
## Model description

This model card presents a model trained on a stacked dataset that aims to improve summarization by testing the benefits of "task-oriented pretraining". The model is designed to learn how to effectively condense and distill information from text by stacking summaries and separating them into independent concepts. In this way, the model can learn to identify essential information without simply mimicking the style of the dataset summaries.

The token used to identify a new concept in the summary is `[NEXT_CONCEPT]`. You can split an output summary based on this token to see how it split the input text information: `summary_text.split("[NEXT_CONCEPT]")` etc.


## Intended uses & limitations

- max input length (in tokens): 1024

## Training and evaluation data

Refer to `stacked-summaries/stacked-xsum-1024`

Trained for approx 3 epochs before ROUGE scores stabilized on most recent run:


### scores

![stable-scores](https://i.imgur.com/4tvhHVy.png)


### gradients

![gradients](https://i.imgur.com/V6zcmAb.png)