Hugging Face's logo

stacked-summaries
/
flan-t5-large-stacked-xsum-1024

Summarization
Transformers
PyTorch
Safetensors
English
t5
text2text-generation
Generated from Trainer
stacked summaries
xsum
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
5
flan-t5-large-stacked-xsum-1024 / README.md
pszemraj's picture
pszemraj
Add evaluation results on the default config and test split of xsum (#5)
9a7a861
preview code
|
raw
history blame
4.37 kB
metadata
language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - generated_from_trainer
  - stacked summaries
  - xsum
datasets:
  - stacked-summaries/stacked-xsum-1024
pipeline_tag: summarization
model-index:
  - name: flan-t5-large-stacked-XSUM-1024-WIP-2p8-850-stacked-xsum-1024-evaluated
    results:
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: xsum
          type: xsum
          config: default
          split: test
        metrics:
          - type: rouge
            value: 39.3614
            name: ROUGE-1
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMWZmZDNhNWM5YjcyMzVjNjUwMWE1NDg4YmRiNGMwY2EyZDYzMGZkY2NlNWE0MzQwNDYzN2JkNzYyOGUxNmI3ZiIsInZlcnNpb24iOjF9.1ucBm8VOqZgLXmUyDkPisiFfHJ8VYvOdvUsk6R_F0QGLIBXOCf2s_pbqHauTyEQM2mAn762DpR5L4AZg7hF_BA
          - type: rouge
            value: 17.5887
            name: ROUGE-2
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDU3MDQwNjYzMTE2MjU5NTE0ODU1ZmI2ZjhlY2QxODA3YTYyOWExZDdiM2Y4YzZhMTU3N2IwMGQ4M2MxMTNmZiIsInZlcnNpb24iOjF9.lb6R_xg5R1TABUCSRgvEGmdkxhSRavrfllxhsk_NxKA53EC4MXeE6o7nRWPoo2nrBOb5Lcajy_5y4oPOkv84Ag
          - type: rouge
            value: 32.6489
            name: ROUGE-L
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmFkOTc2MTIxMmYyNTY2MWE3Y2E4ZWYwODQ5MmU3NTIxZWM2Yzg2ZDNkYjE3NDgzM2VjYTMwOTkxNjQ1YmIyYiIsInZlcnNpb24iOjF9.AAAh5SnRDnTMCEXMfEp9N7pwHITv-crNloZTnbW7TMPXtMUe7vzATOxGVMZpMe-Nsf3Wkc3JbUdaZZ9bOb17Ag
          - type: rouge
            value: 32.6435
            name: ROUGE-LSUM
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjg1ZmNkODZlMzdkODA4MDUxMGQyNjFiMTkyYjIzMTE2NGMyOWQ1NmQ2YjY0OTRmZjVjZWNhODBiOWI1YzVlOCIsInZlcnNpb24iOjF9.GUVl2J3DCRQUqueSuCsFM8v7IDXH7EATFlQbFl730Bo8Y2aolA-V9uN7pkaU9IM1wWBz7hvILElBCE0sln6SAQ
          - type: loss
            value: 1.4964560270309448
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTViZTkwMzQ3MGNlZDJhNTk3NDE5NzBkMDZjMGEyNzNkZTI4ZmJhMWRlYTMwNmRmN2JhNzdkNTQ3N2FlODBmNyIsInZlcnNpb24iOjF9.lNWUw12R20SwZMZEuUnxYsWrkFBNoU9_5ZOiuFF5aT9QsHJC-FSmZ8DXTdVudv6J-BoeA-l5KYowr7GJfbzlDQ
          - type: gen_len
            value: 18.7302
            name: gen_len
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWM2MWQzN2YyY2U0NWZhZGFkMjk0MzFlMTA1YTgxYzAzYjhhZmVmZDI5Mzk4ODgzOGU1NjVhNTk3NmYyNzhkMSIsInZlcnNpb24iOjF9.bL56u1G46OIwdIqZJ-6og_T2yCKFTXrlPQeguZps3ixXokfKqlfCDxz3641yKA3AdMlLe5lDcN3UQReHtiWwBg

flan-t5-large-stacked-XSUM-1024

Open In Colab

This model is a fine-tuned version of google/flan-t5-large on the stacked-summaries/stacked-xsum-1024 dataset.

It achieves the following results on the evaluation set:

  • eval_loss: 1.3314
  • eval_rouge1: 46.5061
  • eval_rouge2: 22.0588
  • eval_rougeL: 37.5235
  • eval_rougeLsum: 39.0234
  • eval_gen_len: 46.1807
  • eval_runtime: 9456.3608
  • eval_samples_per_second: 1.896
  • eval_steps_per_second: 0.119

Note that the evaluation set is stacked-summaries/stacked-xsum-1024 and not xsum itself

Model description

This model card presents a model trained on a stacked dataset that aims to improve summarization by testing the benefits of "task-oriented pretraining". The model is designed to learn how to effectively condense and distill information from text by stacking summaries and separating them into independent concepts. In this way, the model can learn to identify essential information without simply mimicking the style of the dataset summaries.

The token used to identify a new concept in the summary is [NEXT_CONCEPT]. You can split an output summary based on this token to see how it split the input text information: summary_text.split("[NEXT_CONCEPT]") etc.

Intended uses & limitations

  • max input length (in tokens): 1024

Training and evaluation data

Refer to stacked-summaries/stacked-xsum-1024

Trained for approx 3 epochs before ROUGE scores stabilized on most recent run:

scores

stable-scores

gradients

gradients