Hugging Face's logo

philschmid
/
bart-base-samsum

Summarization Transformers PyTorch
English bart text2text-generation sagemaker Eval Results Inference Endpoints
Model card Files Community
3
bart-base-samsum / README.md
philschmid's picture
philschmid HF staff
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator (#2)
ed8aad5
preview code
|
raw history blame contribute delete
No virus
5.33 kB
metadata
language: en
license: apache-2.0
tags:
  - sagemaker
  - bart
  - summarization
datasets:
  - samsum
widget:
  - text: >-
      Jeff: Can I train a 🤗 Transformers model on Amazon SageMaker? 

      Philipp: Sure you can use the new Hugging Face Deep Learning Container. 

      Jeff: ok.

      Jeff: and how can I get started? 

      Jeff: where can I find documentation? 

      Philipp: ok, ok you can find everything here.
      https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face  
model-index:
  - name: philschmid/bart-base-samsum
    results:
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: samsum
          type: samsum
          config: samsum
          split: test
        metrics:
          - type: rouge
            value: 45.3438
            name: ROUGE-1
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2JhY2U3M2ViYTVhNTAzM2M3NjhjMzBjYTk0N2I2MzlmN2Q0N2M1YzFlNGU1ZWVlMGI1YjYzMzZhYjNmMDk1MCIsInZlcnNpb24iOjF9.tLr7VUXSYDd9LaMtVIV8dheZRxX7pf1kyn9Kd4MQY8L_pj13_CeWenqOauVsHzRAZ5Jt5RuHjYFBWbV2TNjvDQ
          - type: rouge
            value: 21.6953
            name: ROUGE-2
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmExODAyMTcwNjU5MjM0MzkzNjZlMGY5YzMyMjNiZjM5OWQ5NzFhODIyMWJiYjUwZGY4ZGM0MzE5OTJiYzEyMSIsInZlcnNpb24iOjF9.qR_Cge1A4NfJL_do4W7Y1kHxU0L98Ds6tbZy-4e-FVNW4aG5zRBxgOX8ieB93N2E19gtzqGE6BdpQfVcZAgXBQ
          - type: rouge
            value: 38.1365
            name: ROUGE-L
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTA5ZTgyNDYxNzgzN2FhNTBlN2NjNzE0MDgyMzZkMTNjMGUyMDk3N2EzOThhMGFhZTQyYzZhZjQ5NjlkOTVlYyIsInZlcnNpb24iOjF9.dKns4BLmyWGUWweYSLYFttHIoWw57z1GKnvatMjkyVvcgwd_iF9imZ7QnJjjLAkc-AUMwwoxoOjEVF8FNf8JBA
          - type: rouge
            value: 41.5913
            name: ROUGE-LSUM
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmJiMzY3ODEwY2Q0YzNjM2QwMjI2MGRmOTEyYjQ3ZmNhZThmYWUxNDJkZDY1NTg3NGQzOGI0YmZlYjI2MDNlZSIsInZlcnNpb24iOjF9.pBrKwWa1mjacdhXSXMUQ0nv1wbcwscW_9uVFkicF2PbJ-JQjzUbL10Jy-b_yBOiJeY5I9ApJySgUH5JMq3_pBg
          - type: loss
            value: 1.5832244157791138
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMWZhNGZjNjJiODIyNDU0NjZjMGExOWE1NWJhMmFiOGY5MDNiZWY0MjExYzA3Njg1OTJhNjEyZjI2MTg0N2I5YiIsInZlcnNpb24iOjF9.T6xwQM5yZ8eD8upqo5zjcUxcX0mqY9wx7f8j0zN9txAe39hURHY-8ibLYJvWckepTvpdUA6is4AC9RUWia24AA
          - type: gen_len
            value: 17.9927
            name: gen_len
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzU4ZGI1ZjJlMjg0NTBkYzlkOWQzMWUzZDZkODZkZjVhNTAyMTI4YTA2MWExM2U2YTQwM2YxMDQ2ODE0Yjc0NSIsInZlcnNpb24iOjF9.mDGhriDLXIJq_yb3Yqj6MBJSCxXXrRN1LfHsGkV8i1oOpkLiSLic7D8fSFMdTZTkl2XmzQfkVU2Wv298YyQEBg

bart-base-samsum

This model was trained using Amazon SageMaker and the new Hugging Face Deep Learning container.

You can find the notebook here and the referring blog post here.

For more information look at:

Hyperparameters 

{
    "dataset_name": "samsum",
    "do_eval": true,
    "do_train": true,
    "fp16": true,
    "learning_rate": 5e-05,
    "model_name_or_path": "facebook/bart-base",
    "num_train_epochs": 3,
    "output_dir": "/opt/ml/model",
    "per_device_eval_batch_size": 8,
    "per_device_train_batch_size": 8,
    "seed": 7
}

Train results

key value
epoch 3
init_mem_cpu_alloc_delta 180190
init_mem_cpu_peaked_delta 18282
init_mem_gpu_alloc_delta 558658048
init_mem_gpu_peaked_delta 0
train_mem_cpu_alloc_delta 6658519
train_mem_cpu_peaked_delta 642937
train_mem_gpu_alloc_delta 2267624448
train_mem_gpu_peaked_delta 10355728896
train_runtime 98.4931
train_samples 14732
train_samples_per_second 3.533

Eval results

key value
epoch 3
eval_loss 1.5356481075286865
eval_mem_cpu_alloc_delta 659047
eval_mem_cpu_peaked_delta 18254
eval_mem_gpu_alloc_delta 0
eval_mem_gpu_peaked_delta 300285440
eval_runtime 0.3116
eval_samples 818
eval_samples_per_second 2625.337

Usage 

from transformers import pipeline
summarizer = pipeline("summarization", model="philschmid/bart-base-samsum")

conversation = '''Jeff: Can I train a 🤗 Transformers model on Amazon SageMaker? 
Philipp: Sure you can use the new Hugging Face Deep Learning Container. 
Jeff: ok.
Jeff: and how can I get started? 
Jeff: where can I find documentation? 
Philipp: ok, ok you can find everything here. https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face                                           
'''
nlp(conversation)