|
--- |
|
language: |
|
- de |
|
|
|
license: mit |
|
|
|
tags: |
|
- summarization |
|
|
|
datasets: |
|
- swiss_text_2019 |
|
|
|
--- |
|
|
|
# mT5-small-sum-de-mit-v1 |
|
|
|
This is a German summarization model. It is based on the multilingual T5 model [google/mt5-small](https://huggingface.co/google/mt5-small). The special characteristic of this model is that, unlike many other models, it is licensed under a permissive open source license (MIT). Among other things, this license allows commercial use. |
|
|
|
[](https://www.welove.ai/) |
|
This model is provided by the [One Conversation](https://www.welove.ai/) |
|
team of [Deutsche Telekom AG](https://www.telekom.com/). |
|
|
|
## Training |
|
|
|
The training was conducted with the following hyperparameters: |
|
|
|
- base model: [google/mt5-small](https://huggingface.co/google/mt5-small) |
|
- source_prefix: `"summarize: "` |
|
- batch size: 3 (6) |
|
- max_source_length: 800 |
|
- max_target_length: 96 |
|
- warmup_ratio: 0.3 |
|
- number of train epochs: 10 |
|
- gradient accumulation steps: 2 |
|
- learning rate: 5e-5 |
|
|
|
## Datasets and Preprocessing |
|
|
|
The datasets were preprocessed as follows: |
|
|
|
The summary was tokenized with the [google/mt5-small](https://huggingface.co/google/mt5-small) tokenizer. Then only the records with no more than 94 summary tokens were selected. |
|
|
|
This model is trained on the following dataset: |
|
|
|
| Name | Language | Size | License |
|
|------|----------|------|-------- |
|
| [SwissText 2019 - Train](https://www.swisstext.org/2019/shared-task/german-text-summarization-challenge.html) | de | 84,564 | Concrete license is unclear. The data was published in the [German Text Summarization Challenge](https://www.swisstext.org/2019/shared-task/german-text-summarization-challenge.html). |
|
|
|
We have permission to use the Swisstext dataset and release the resulting summarization model under MIT license (see [permission-declaration-swisstext.pdf](https://huggingface.co/deutsche-telekom/mt5-small-sum-de-mit-v1/resolve/main/permission-declaration-swisstext.pdf)). |
|
|
|
## Evaluation on MLSUM German Test Set (no beams) |
|
|
|
| Model | rouge1 | rouge2 | rougeL | rougeLsum |
|
|-------|--------|--------|--------|---------- |
|
| deutsche-telekom/mt5-small-sum-de-mit-v1 (this) | 16.8023 | 3.5531 | 12.6884 | 14.7624 |
|
| [ml6team/mt5-small-german-finetune-mlsum](https://huggingface.co/ml6team/mt5-small-german-finetune-mlsum) | 18.3607 | 5.3604 | 14.5456 | 16.1946 |
|
| **[deutsche-telekom/mt5-small-sum-de-en-01](https://huggingface.co/deutsche-telekom/mt5-small-sum-de-en-v1)** | **21.7336** | **7.2614** | **17.1323** | **19.3977** |
|
|
|
## License |
|
|
|
Copyright (c) 2021 Philip May, Deutsche Telekom AG |
|
|
|
Licensed under the MIT License (the "License"); you may not use this work except in compliance with the License. You may obtain a copy of the License by reviewing the file [LICENSE](https://huggingface.co/deutsche-telekom/mt5-small-sum-de-mit-v1/blob/main/LICENSE) in the repository. |
|
|
