philschmid
HF staff
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator (#2)
ed8aad5
| language: en | |
| license: apache-2.0 | |
| tags: | |
| - sagemaker | |
| - bart | |
| - summarization | |
| datasets: | |
| - samsum | |
| widget: | |
| - text: "Jeff: Can I train a \U0001F917 Transformers model on Amazon SageMaker? \n\ | |
| Philipp: Sure you can use the new Hugging Face Deep Learning Container. \nJeff:\ | |
| \ ok.\nJeff: and how can I get started? \nJeff: where can I find documentation?\ | |
| \ \nPhilipp: ok, ok you can find everything here. https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face\ | |
| \ " | |
| model-index: | |
| - name: philschmid/bart-base-samsum | |
| results: | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: samsum | |
| type: samsum | |
| config: samsum | |
| split: test | |
| metrics: | |
| - type: rouge | |
| value: 45.3438 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2JhY2U3M2ViYTVhNTAzM2M3NjhjMzBjYTk0N2I2MzlmN2Q0N2M1YzFlNGU1ZWVlMGI1YjYzMzZhYjNmMDk1MCIsInZlcnNpb24iOjF9.tLr7VUXSYDd9LaMtVIV8dheZRxX7pf1kyn9Kd4MQY8L_pj13_CeWenqOauVsHzRAZ5Jt5RuHjYFBWbV2TNjvDQ | |
| - type: rouge | |
| value: 21.6953 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmExODAyMTcwNjU5MjM0MzkzNjZlMGY5YzMyMjNiZjM5OWQ5NzFhODIyMWJiYjUwZGY4ZGM0MzE5OTJiYzEyMSIsInZlcnNpb24iOjF9.qR_Cge1A4NfJL_do4W7Y1kHxU0L98Ds6tbZy-4e-FVNW4aG5zRBxgOX8ieB93N2E19gtzqGE6BdpQfVcZAgXBQ | |
| - type: rouge | |
| value: 38.1365 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTA5ZTgyNDYxNzgzN2FhNTBlN2NjNzE0MDgyMzZkMTNjMGUyMDk3N2EzOThhMGFhZTQyYzZhZjQ5NjlkOTVlYyIsInZlcnNpb24iOjF9.dKns4BLmyWGUWweYSLYFttHIoWw57z1GKnvatMjkyVvcgwd_iF9imZ7QnJjjLAkc-AUMwwoxoOjEVF8FNf8JBA | |
| - type: rouge | |
| value: 41.5913 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmJiMzY3ODEwY2Q0YzNjM2QwMjI2MGRmOTEyYjQ3ZmNhZThmYWUxNDJkZDY1NTg3NGQzOGI0YmZlYjI2MDNlZSIsInZlcnNpb24iOjF9.pBrKwWa1mjacdhXSXMUQ0nv1wbcwscW_9uVFkicF2PbJ-JQjzUbL10Jy-b_yBOiJeY5I9ApJySgUH5JMq3_pBg | |
| - type: loss | |
| value: 1.5832244157791138 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMWZhNGZjNjJiODIyNDU0NjZjMGExOWE1NWJhMmFiOGY5MDNiZWY0MjExYzA3Njg1OTJhNjEyZjI2MTg0N2I5YiIsInZlcnNpb24iOjF9.T6xwQM5yZ8eD8upqo5zjcUxcX0mqY9wx7f8j0zN9txAe39hURHY-8ibLYJvWckepTvpdUA6is4AC9RUWia24AA | |
| - type: gen_len | |
| value: 17.9927 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzU4ZGI1ZjJlMjg0NTBkYzlkOWQzMWUzZDZkODZkZjVhNTAyMTI4YTA2MWExM2U2YTQwM2YxMDQ2ODE0Yjc0NSIsInZlcnNpb24iOjF9.mDGhriDLXIJq_yb3Yqj6MBJSCxXXrRN1LfHsGkV8i1oOpkLiSLic7D8fSFMdTZTkl2XmzQfkVU2Wv298YyQEBg | |
| ## `bart-base-samsum` | |
| This model was trained using Amazon SageMaker and the new Hugging Face Deep Learning container. | |
| You can find the notebook [here]() and the referring blog post [here](). | |
| For more information look at: | |
| - [🤗 Transformers Documentation: Amazon SageMaker](https://huggingface.co/transformers/sagemaker.html) | |
| - [Example Notebooks](https://github.com/huggingface/notebooks/tree/master/sagemaker) | |
| - [Amazon SageMaker documentation for Hugging Face](https://docs.aws.amazon.com/sagemaker/latest/dg/hugging-face.html) | |
| - [Python SDK SageMaker documentation for Hugging Face](https://sagemaker.readthedocs.io/en/stable/frameworks/huggingface/index.html) | |
| - [Deep Learning Container](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#huggingface-training-containers) | |
| ## Hyperparameters | |
| ```json | |
| { | |
| "dataset_name": "samsum", | |
| "do_eval": true, | |
| "do_train": true, | |
| "fp16": true, | |
| "learning_rate": 5e-05, | |
| "model_name_or_path": "facebook/bart-base", | |
| "num_train_epochs": 3, | |
| "output_dir": "/opt/ml/model", | |
| "per_device_eval_batch_size": 8, | |
| "per_device_train_batch_size": 8, | |
| "seed": 7 | |
| } | |
| ``` | |
| ## Train results | |
| | key | value | | |
| | --- | ----- | | |
| | epoch | 3 | | |
| | init_mem_cpu_alloc_delta | 180190 | | |
| | init_mem_cpu_peaked_delta | 18282 | | |
| | init_mem_gpu_alloc_delta | 558658048 | | |
| | init_mem_gpu_peaked_delta | 0 | | |
| | train_mem_cpu_alloc_delta | 6658519 | | |
| | train_mem_cpu_peaked_delta | 642937 | | |
| | train_mem_gpu_alloc_delta | 2267624448 | | |
| | train_mem_gpu_peaked_delta | 10355728896 | | |
| | train_runtime | 98.4931 | | |
| | train_samples | 14732 | | |
| | train_samples_per_second | 3.533 | | |
| ## Eval results | |
| | key | value | | |
| | --- | ----- | | |
| | epoch | 3 | | |
| | eval_loss | 1.5356481075286865 | | |
| | eval_mem_cpu_alloc_delta | 659047 | | |
| | eval_mem_cpu_peaked_delta | 18254 | | |
| | eval_mem_gpu_alloc_delta | 0 | | |
| | eval_mem_gpu_peaked_delta | 300285440 | | |
| | eval_runtime | 0.3116 | | |
| | eval_samples | 818 | | |
| | eval_samples_per_second | 2625.337 | | |
| ## Usage | |
| ```python | |
| from transformers import pipeline | |
| summarizer = pipeline("summarization", model="philschmid/bart-base-samsum") | |
| conversation = '''Jeff: Can I train a 🤗 Transformers model on Amazon SageMaker? | |
| Philipp: Sure you can use the new Hugging Face Deep Learning Container. | |
| Jeff: ok. | |
| Jeff: and how can I get started? | |
| Jeff: where can I find documentation? | |
| Philipp: ok, ok you can find everything here. https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face | |
| ''' | |
| nlp(conversation) | |
| ``` | |