File size: 5,444 Bytes
034fd08 1b38365 922bffd 034fd08 dbbf244 4410bf5 4a38914 4410bf5 4a38914 4410bf5 034fd08 1b38365 034fd08 06364a4 f13bee7 034fd08 f13bee7 7ab48a9 f13bee7 7ab48a9 f13bee7 034fd08 f13bee7 034fd08 f13bee7 034fd08 06364a4 034fd08 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 |
---
license: apache-2.0
tags:
- text2text-generation
- generated_from_trainer
metrics:
- rouge
- bleu
datasets:
- domenicrosati/QA2D
model-index:
- name: QA2D-t5-small
results:
- task:
name: Question to Declarative Sentence
type: text2text-generation
dataset:
name: domenicrosati/QA2D
type: domenicrosati/QA2D
args: plain_text
metrics:
- name: Rouge1
type: rouge
value: 89.8753
- name: Rouge2
type: rouge
value: 81.8104
- name: Rougel
type: rouge
value: 85.4253
- name: Rougelsum
type: rouge
value: 85.4236
- name: Bleu
type: bleu
value: 72.1080
widget:
- text: "where in the world is carmen sandiego. she is in abruzzo"
example_title: "Where is Carmen Sandiego?"
- text: "which province is halifax in. nova scotia"
example_title: "A Halifact"
---
# QA2D-t5-small
This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on [QA2D](https://huggingface.co/datasets/domenicrosati/QA2D).
It achieves the following results on the evaluation set:
- Loss: 0.3236
- Rouge1: 89.8753
- Rouge2: 81.8104
- Rougel: 85.4253
- Rougelsum: 85.4236
- Bleu: 72.1080
See: [https://wandb.ai/domenicrosati/huggingface/runs/n1yallpe](https://wandb.ai/domenicrosati/huggingface/runs/n1yallpe) for training and eval stats and [https://github.com/domenicrosati/qa2d-models](https://github.com/domenicrosati/qa2d-models) for the code!
## Model description
A t5-model model to convert questions, answer pairs into statements.
Due to the way it's been trained the input should be all lower case and punctuation removed.
Use with `. ` as the seperator between question and answer.
> "where in the world is carmen. abruzzo"
> Output: "carmen is in abruzzo"
Thought punctation and upper case works.
```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained('domenicrosati/QA2D-t5-small')
model = AutoModelForSeq2SeqLM.from_pretrained('domenicrosati/QA2D-t5-small')
question = "where in the world is carmen sandiego"
answer = "she is in abruzzo"
SEP = ". "
prompt = f'{question}{SEP}{answer}'
input_ids = tokenizer(prompt, return_tensors='pt').input_ids
output_ids = model.generate(input_ids)
responses = tokenizer.batch_decode(output_ids, skip_special_tokens=True)
# ['carmen sandiego is in abruzzo']
```
## Intended uses & limitations
To convert questions, answer pairs into statements.
## Training and evaluation data
Uses [QA2D](https://huggingface.co/datasets/domenicrosati/QA2D).
See [https://github.com/domenicrosati/qa2d-models](https://github.com/domenicrosati/qa2d-models)
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5.6e-05
- train_batch_size: 12
- eval_batch_size: 12
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20
- mixed_precision_training: Native AMP
### Training results
| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Bleu |
|:-------------:|:-----:|:------:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| 0.3177 | 1.0 | 5060 | 0.3144 | 89.6379 | 81.3168 | 85.2036 | 85.1904 | 71.4255 |
| 0.2479 | 2.0 | 10120 | 0.3035 | 89.7816 | 81.6556 | 85.3541 | 85.3406 | 71.7248 |
| 0.2268 | 3.0 | 15180 | 0.3015 | 89.8287 | 81.698 | 85.3434 | 85.3387 | 71.8344 |
| 0.2111 | 4.0 | 20240 | 0.3014 | 89.8082 | 81.7192 | 85.4094 | 85.406 | 71.9172 |
| 0.1991 | 5.0 | 25300 | 0.3023 | 89.8776 | 81.7607 | 85.3912 | 85.3842 | 71.9417 |
| 0.1886 | 6.0 | 30360 | 0.3012 | 89.901 | 81.7614 | 85.3345 | 85.3315 | 72.0218 |
| 0.1803 | 7.0 | 35420 | 0.3010 | 89.8776 | 81.8189 | 85.4154 | 85.4097 | 72.0533 |
| 0.1724 | 8.0 | 40480 | 0.3041 | 89.9168 | 81.8663 | 85.4457 | 85.4447 | 72.1470 |
| 0.1654 | 9.0 | 45540 | 0.3076 | 89.8901 | 81.8536 | 85.4857 | 85.4863 | 72.0830 |
| 0.1601 | 10.0 | 50600 | 0.3083 | 89.9186 | 81.881 | 85.4653 | 85.4594 | 72.1048 |
| 0.1546 | 11.0 | 55660 | 0.3136 | 89.8958 | 81.8533 | 85.4217 | 85.4238 | 72.0752 |
| 0.1502 | 12.0 | 60720 | 0.3138 | 89.903 | 81.8604 | 85.4301 | 85.4267 | 72.1373 |
| 0.1461 | 13.0 | 65780 | 0.3140 | 89.8867 | 81.7945 | 85.3698 | 85.3662 | 72.0718 |
| 0.1423 | 14.0 | 70840 | 0.3171 | 89.8985 | 81.8221 | 85.4348 | 85.4331 | 72.1168 |
| 0.1392 | 15.0 | 75900 | 0.3186 | 89.8938 | 81.8246 | 85.402 | 85.3991 | 72.0858 |
| 0.1366 | 16.0 | 80960 | 0.3208 | 89.859 | 81.8133 | 85.4194 | 85.4182 | 72.1014 |
| 0.1344 | 17.0 | 86020 | 0.3222 | 89.8909 | 81.828 | 85.4392 | 85.435 | 72.1380 |
| 0.1324 | 18.0 | 91080 | 0.3226 | 89.8906 | 81.8351 | 85.4506 | 85.4441 | 72.1622 |
| 0.1309 | 19.0 | 96140 | 0.3231 | 89.8925 | 81.8369 | 85.4375 | 85.4366 | 72.1552 |
| 0.1305 | 20.0 | 101200 | 0.3236 | 89.8753 | 81.8104 | 85.4253 | 85.4236 | 72.1080 |
### Framework versions
- Transformers 4.18.0
- Pytorch 1.11.0+cu113
- Datasets 2.1.0
- Tokenizers 0.12.1
|