Hugging Face's logo

vocabtrimmer
/
mt5-small-trimmed-it-60000-itquad-qa

Text2Text Generation
Transformers
PyTorch
Italian
mt5
question answering
Eval Results
Inference Endpoints
text-generation-inference
Model card Files Community
asahi417 commited on
Commit
66257ae
•
1 Parent(s): ae7d616

commit files to HF hub

Browse files
Files changed (4) hide show
  1. README.md +21 -21
  2. eval/metric.first.answer.paragraph_question.answer.lmqg_qg_itquad.default.json +1 -1
  3. eval/samples.test.hyp.paragraph_question.answer.lmqg_qg_itquad.default.txt +0 -0
  4. eval/samples.validation.hyp.paragraph_question.answer.lmqg_qg_itquad.default.txt +0 -0
README.md CHANGED
@@ -29,33 +29,33 @@ model-index:
29
  metrics:
30
  - name: BLEU4 (Question Answering)
31
  type: bleu4_question_answering
32
- value: 0.0
33
  - name: ROUGE-L (Question Answering)
34
  type: rouge_l_question_answering
35
- value: 0.03
36
  - name: METEOR (Question Answering)
37
  type: meteor_question_answering
38
- value: 0.1
39
  - name: BERTScore (Question Answering)
40
  type: bertscore_question_answering
41
- value: 67.52
42
  - name: MoverScore (Question Answering)
43
  type: moverscore_question_answering
44
- value: 55.39
45
  - name: AnswerF1Score (Question Answering)
46
  type: answer_f1_score__question_answering
47
- value: 0.03
48
  - name: AnswerExactMatch (Question Answering)
49
  type: answer_exact_match_question_answering
50
- value: 0.0
51
  ---
52
 
53
  # Model Card of `vocabtrimmer/mt5-small-trimmed-it-60000-itquad-qa`
54
- This model is fine-tuned version of [vocabtrimmer/mt5-small-trimmed-it-60000](https://huggingface.co/vocabtrimmer/mt5-small-trimmed-it-60000) for question answering task on the [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) (dataset_name: default) via [`lmqg`](https://github.com/asahi417/lm-question-generation).
55
 
56
 
57
  ### Overview
58
- - **Language model:** [vocabtrimmer/mt5-small-trimmed-it-60000](https://huggingface.co/vocabtrimmer/mt5-small-trimmed-it-60000)
59
  - **Language:** it
60
  - **Training data:** [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) (default)
61
  - **Online Demo:** [https://autoqg.net/](https://autoqg.net/)
@@ -91,16 +91,16 @@ output = pipe("question: Quale batterio ha il nome del paese che colpisce di pi
91
 
92
  | | Score | Type | Dataset |
93
  |:-----------------|--------:|:--------|:-----------------------------------------------------------------|
94
- | AnswerExactMatch | 0 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
95
- | AnswerF1Score | 0.03 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
96
- | BERTScore | 67.52 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
97
- | Bleu_1 | 0.03 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
98
- | Bleu_2 | 0 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
99
- | Bleu_3 | 0 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
100
- | Bleu_4 | 0 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
101
- | METEOR | 0.1 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
102
- | MoverScore | 55.39 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
103
- | ROUGE_L | 0.03 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
104
 
105
 
106
 
@@ -112,10 +112,10 @@ The following hyperparameters were used during fine-tuning:
112
  - input_types: ['paragraph_question']
113
  - output_types: ['answer']
114
  - prefix_types: None
115
- - model: vocabtrimmer/mt5-small-trimmed-it-60000
116
  - max_length: 512
117
  - max_length_output: 32
118
- - epoch: 20
119
  - batch: 32
120
  - lr: 0.0005
121
  - fp16: False
 
29
  metrics:
30
  - name: BLEU4 (Question Answering)
31
  type: bleu4_question_answering
32
+ value: 9.49
33
  - name: ROUGE-L (Question Answering)
34
  type: rouge_l_question_answering
35
+ value: 34.13
36
  - name: METEOR (Question Answering)
37
  type: meteor_question_answering
38
+ value: 29.49
39
  - name: BERTScore (Question Answering)
40
  type: bertscore_question_answering
41
+ value: 90.7
42
  - name: MoverScore (Question Answering)
43
  type: moverscore_question_answering
44
+ value: 76.17
45
  - name: AnswerF1Score (Question Answering)
46
  type: answer_f1_score__question_answering
47
+ value: 56.49
48
  - name: AnswerExactMatch (Question Answering)
49
  type: answer_exact_match_question_answering
50
+ value: 40.52
51
  ---
52
 
53
  # Model Card of `vocabtrimmer/mt5-small-trimmed-it-60000-itquad-qa`
54
+ This model is fine-tuned version of [ckpts/mt5-small-trimmed-it-60000](https://huggingface.co/ckpts/mt5-small-trimmed-it-60000) for question answering task on the [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) (dataset_name: default) via [`lmqg`](https://github.com/asahi417/lm-question-generation).
55
 
56
 
57
  ### Overview
58
+ - **Language model:** [ckpts/mt5-small-trimmed-it-60000](https://huggingface.co/ckpts/mt5-small-trimmed-it-60000)
59
  - **Language:** it
60
  - **Training data:** [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) (default)
61
  - **Online Demo:** [https://autoqg.net/](https://autoqg.net/)
 
91
 
92
  | | Score | Type | Dataset |
93
  |:-----------------|--------:|:--------|:-----------------------------------------------------------------|
94
+ | AnswerExactMatch | 40.52 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
95
+ | AnswerF1Score | 56.49 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
96
+ | BERTScore | 90.7 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
97
+ | Bleu_1 | 20.1 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
98
+ | Bleu_2 | 15.3 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
99
+ | Bleu_3 | 12.04 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
100
+ | Bleu_4 | 9.49 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
101
+ | METEOR | 29.49 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
102
+ | MoverScore | 76.17 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
103
+ | ROUGE_L | 34.13 | default | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) |
104
 
105
 
106
 
 
112
  - input_types: ['paragraph_question']
113
  - output_types: ['answer']
114
  - prefix_types: None
115
+ - model: ckpts/mt5-small-trimmed-it-60000
116
  - max_length: 512
117
  - max_length_output: 32
118
+ - epoch: 15
119
  - batch: 32
120
  - lr: 0.0005
121
  - fp16: False
eval/metric.first.answer.paragraph_question.answer.lmqg_qg_itquad.default.json CHANGED
@@ -1 +1 @@
1
- {"validation": {"Bleu_1": 0.0001960476787954754, "Bleu_2": 3.1996057100058593e-12, "Bleu_3": 9.000678990283672e-15, "Bleu_4": 5.118884440960065e-16, "METEOR": 0.0009749709661537914, "ROUGE_L": 0.00011002983433641913, "BERTScore": 0.6597895539532291, "MoverScore": 0.5600741075175579, "AnswerF1Score": 0.01228893330352129, "AnswerExactMatch": 0.0}, "test": {"Bleu_1": 0.00034127363319908747, "Bleu_2": 3.870566274273233e-12, "Bleu_3": 9.51608049643211e-15, "Bleu_4": 5.024575858905814e-16, "METEOR": 0.0010271453728117376, "ROUGE_L": 0.00026125355465024037, "BERTScore": 0.6751906266270744, "MoverScore": 0.5539017346678483, "AnswerF1Score": 0.02851633281135252, "AnswerExactMatch": 0.0}}
 
1
+ {"validation": {"Bleu_1": 0.2168200141551461, "Bleu_2": 0.16457544807858454, "Bleu_3": 0.13015003475283593, "Bleu_4": 0.10269429902258806, "METEOR": 0.3176781214512819, "ROUGE_L": 0.3446343748642225, "BERTScore": 0.9211360962824245, "MoverScore": 0.7948918389120013, "AnswerF1Score": 62.031363955784265, "AnswerExactMatch": 48.258641082928115}, "test": {"Bleu_1": 0.20095433209719146, "Bleu_2": 0.15300907487721307, "Bleu_3": 0.12044830411721509, "Bleu_4": 0.09492166933146612, "METEOR": 0.2948779214599961, "ROUGE_L": 0.34130328206695704, "BERTScore": 0.9069560774915211, "MoverScore": 0.7617222323179461, "AnswerF1Score": 56.491622198303375, "AnswerExactMatch": 40.51780785911421}}
eval/samples.test.hyp.paragraph_question.answer.lmqg_qg_itquad.default.txt CHANGED
The diff for this file is too large to render. See raw diff
 
eval/samples.validation.hyp.paragraph_question.answer.lmqg_qg_itquad.default.txt CHANGED
The diff for this file is too large to render. See raw diff