Hugging Face's logo

lmqg
/
mt5-small-squad-qg

Text2Text Generation
Transformers
PyTorch
English
mt5
question generation
Eval Results
Inference Endpoints
Model card Files Community
1
asahi417 commited on
Commit
29d3896
•
1 Parent(s): 6a4a582

model update

Browse files
Files changed (1) hide show
  1. README.md +77 -92
README.md CHANGED
@@ -33,205 +33,186 @@ model-index:
33
  metrics:
34
  - name: BLEU4
35
  type: bleu4
36
- value: 0.21650505934418166
37
  - name: ROUGE-L
38
  type: rouge-l
39
- value: 0.489464328982525
40
  - name: METEOR
41
  type: meteor
42
- value: 0.23833897056449205
43
  - name: BERTScore
44
  type: bertscore
45
- value: 0.9000723844397448
46
  - name: MoverScore
47
  type: moverscore
48
- value: 0.62747065964027
49
  - task:
50
  name: Text2text Generation
51
  type: text2text-generation
52
  dataset:
53
- name: lmqg/qg_itquad
54
  type: default
55
  args: default
56
  metrics:
57
  - name: BLEU4
58
  type: bleu4
59
- value: 0.005438910607183992
60
  - name: ROUGE-L
61
  type: rouge-l
62
- value: 0.05010570221421983
63
  - name: METEOR
64
  type: meteor
65
- value: 0.05890828426558759
66
  - name: BERTScore
67
  type: bertscore
68
- value: 0.7260160158030385
69
  - name: MoverScore
70
  type: moverscore
71
- value: 0.5023119088393686
72
  - task:
73
  name: Text2text Generation
74
  type: text2text-generation
75
  dataset:
76
- name: lmqg/qg_jaquad
77
  type: default
78
  args: default
79
  metrics:
80
  - name: BLEU4
81
  type: bleu4
82
- value: 4.4114578660129224e-08
83
  - name: ROUGE-L
84
  type: rouge-l
85
- value: 0.06084267343290677
86
  - name: METEOR
87
  type: meteor
88
- value: 0.005149267426183168
89
  - name: BERTScore
90
  type: bertscore
91
- value: 0.6608093198082075
92
  - name: MoverScore
93
  type: moverscore
94
- value: 0.46526108687696893
95
  - task:
96
  name: Text2text Generation
97
  type: text2text-generation
98
  dataset:
99
- name: lmqg/qg_ruquad
100
  type: default
101
  args: default
102
  metrics:
103
  - name: BLEU4
104
  type: bleu4
105
- value: 4.229109829516021e-12
106
  - name: ROUGE-L
107
  type: rouge-l
108
- value: 0.009881091250723615
109
  - name: METEOR
110
  type: meteor
111
- value: 0.017796529053904556
112
  - name: BERTScore
113
  type: bertscore
114
- value: 0.7089446693028568
115
  - name: MoverScore
116
  type: moverscore
117
- value: 0.49098728551715626
118
  - task:
119
  name: Text2text Generation
120
  type: text2text-generation
121
  dataset:
122
- name: lmqg/qg_dequad
123
  type: default
124
  args: default
125
  metrics:
126
  - name: BLEU4
127
  type: bleu4
128
- value: 9.242783121165897e-12
129
  - name: ROUGE-L
130
  type: rouge-l
131
- value: 0.01556150764938016
132
  - name: METEOR
133
  type: meteor
134
- value: 0.04809700451843158
135
  - name: BERTScore
136
  type: bertscore
137
- value: 0.7353078946893743
138
  - name: MoverScore
139
  type: moverscore
140
- value: 0.5036973829954939
141
  - task:
142
  name: Text2text Generation
143
  type: text2text-generation
144
  dataset:
145
- name: lmqg/qg_esquad
146
  type: default
147
  args: default
148
  metrics:
149
  - name: BLEU4
150
  type: bleu4
151
- value: 0.0059191752064594125
152
  - name: ROUGE-L
153
  type: rouge-l
154
- value: 0.05208940592236566
155
  - name: METEOR
156
  type: meteor
157
- value: 0.06021086135293597
158
  - name: BERTScore
159
  type: bertscore
160
- value: 0.7494422899749911
161
  - name: MoverScore
162
  type: moverscore
163
- value: 0.5062373132800192
164
  - task:
165
  name: Text2text Generation
166
  type: text2text-generation
167
  dataset:
168
- name: lmqg/qg_frquad
169
  type: default
170
  args: default
171
  metrics:
172
  - name: BLEU4
173
  type: bleu4
174
- value: 0.0171464639522496
175
  - name: ROUGE-L
176
  type: rouge-l
177
- value: 0.1583673053928925
178
  - name: METEOR
179
  type: meteor
180
- value: 0.08244973027319356
181
  - name: BERTScore
182
  type: bertscore
183
- value: 0.7291012183458674
184
  - name: MoverScore
185
  type: moverscore
186
- value: 0.509610854598101
187
  - task:
188
  name: Text2text Generation
189
  type: text2text-generation
190
  dataset:
191
- name: lmqg/qg_koquad
192
  type: default
193
  args: default
194
  metrics:
195
  - name: BLEU4
196
  type: bleu4
197
- value: 1.4750917137316939e-12
198
  - name: ROUGE-L
199
  type: rouge-l
200
- value: 0.0006466767450454226
201
  - name: METEOR
202
  type: meteor
203
- value: 0.007310046912436679
204
  - name: BERTScore
205
  type: bertscore
206
- value: 0.6634288882769679
207
  - name: MoverScore
208
  type: moverscore
209
- value: 0.4586124640357038
210
  ---
211
 
212
  # Model Card of `lmqg/mt5-small-squad`
213
- This model is fine-tuned version of [google/mt5-small](https://huggingface.co/google/mt5-small) for question generation task on the
214
- [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) (dataset_name: default) via [`lmqg`](https://github.com/asahi417/lm-question-generation).
215
 
216
 
217
- Please cite our paper if you use the model ([https://arxiv.org/abs/2210.03992](https://arxiv.org/abs/2210.03992)).
218
-
219
- ```
220
-
221
- @inproceedings{ushio-etal-2022-generative,
222
- title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
223
- author = "Ushio, Asahi and
224
- Alva-Manchego, Fernando and
225
- Camacho-Collados, Jose",
226
- booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
227
- month = dec,
228
- year = "2022",
229
- address = "Abu Dhabi, U.A.E.",
230
- publisher = "Association for Computational Linguistics",
231
- }
232
-
233
- ```
234
-
235
  ### Overview
236
  - **Language model:** [google/mt5-small](https://huggingface.co/google/mt5-small)
237
  - **Language:** en
@@ -243,48 +224,53 @@ Please cite our paper if you use the model ([https://arxiv.org/abs/2210.03992](h
243
  ### Usage
244
  - With [`lmqg`](https://github.com/asahi417/lm-question-generation#lmqg-language-model-for-question-generation-)
245
  ```python
246
-
247
  from lmqg import TransformersQG
 
248
  # initialize model
249
- model = TransformersQG(language='en', model='lmqg/mt5-small-squad')
 
250
  # model prediction
251
- question = model.generate_q(list_context=["William Turner was an English painter who specialised in watercolour landscapes"], list_answer=["William Turner"])
252
 
253
  ```
254
 
255
  - With `transformers`
256
  ```python
257
-
258
  from transformers import pipeline
259
- # initialize model
260
- pipe = pipeline("text2text-generation", 'lmqg/mt5-small-squad')
261
- # question generation
262
- question = pipe('<hl> Beyonce <hl> further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records.')
263
 
264
- ```
 
265
 
266
- ## Evaluation Metrics
267
 
 
268
 
269
- ### Metrics
270
 
271
- | Dataset | Type | BLEU4 | ROUGE-L | METEOR | BERTScore | MoverScore | Link |
272
- |:--------|:-----|------:|--------:|-------:|----------:|-----------:|-----:|
273
- | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) | default | 0.217 | 0.489 | 0.238 | 0.9 | 0.627 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval/metric.first.sentence.paragraph_answer.question.lmqg_qg_squad.default.json) |
274
 
 
 
 
 
 
 
 
 
 
 
275
 
276
 
277
- ### Out-of-domain Metrics
278
 
279
- | Dataset | Type | BLEU4 | ROUGE-L | METEOR | BERTScore | MoverScore | Link |
280
- |:--------|:-----|------:|--------:|-------:|----------:|-----------:|-----:|
281
- | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) | default | 0.005 | 0.05 | 0.059 | 0.726 | 0.502 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_itquad.default.json) |
282
- | [lmqg/qg_jaquad](https://huggingface.co/datasets/lmqg/qg_jaquad) | default | 0.0 | 0.061 | 0.005 | 0.661 | 0.465 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_jaquad.default.json) |
283
- | [lmqg/qg_ruquad](https://huggingface.co/datasets/lmqg/qg_ruquad) | default | 0.0 | 0.01 | 0.018 | 0.709 | 0.491 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_ruquad.default.json) |
284
- | [lmqg/qg_dequad](https://huggingface.co/datasets/lmqg/qg_dequad) | default | 0.0 | 0.016 | 0.048 | 0.735 | 0.504 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_dequad.default.json) |
285
- | [lmqg/qg_esquad](https://huggingface.co/datasets/lmqg/qg_esquad) | default | 0.006 | 0.052 | 0.06 | 0.749 | 0.506 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_esquad.default.json) |
286
- | [lmqg/qg_frquad](https://huggingface.co/datasets/lmqg/qg_frquad) | default | 0.017 | 0.158 | 0.082 | 0.729 | 0.51 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_frquad.default.json) |
287
- | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) | default | 0.0 | 0.001 | 0.007 | 0.663 | 0.459 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_koquad.default.json) |
288
 
289
 
290
  ## Training hyperparameters
@@ -310,7 +296,6 @@ The full configuration can be found at [fine-tuning config file](https://hugging
310
 
311
  ## Citation
312
  ```
313
-
314
  @inproceedings{ushio-etal-2022-generative,
315
  title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
316
  author = "Ushio, Asahi and
 
33
  metrics:
34
  - name: BLEU4
35
  type: bleu4
36
+ value: 21.65
37
  - name: ROUGE-L
38
  type: rouge-l
39
+ value: 48.95
40
  - name: METEOR
41
  type: meteor
42
+ value: 23.83
43
  - name: BERTScore
44
  type: bertscore
45
+ value: 90.01
46
  - name: MoverScore
47
  type: moverscore
48
+ value: 62.75
49
  - task:
50
  name: Text2text Generation
51
  type: text2text-generation
52
  dataset:
53
+ name: lmqg/qg_dequad
54
  type: default
55
  args: default
56
  metrics:
57
  - name: BLEU4
58
  type: bleu4
59
+ value: 9.242783121165897e-12
60
  - name: ROUGE-L
61
  type: rouge-l
62
+ value: 0.01556150764938016
63
  - name: METEOR
64
  type: meteor
65
+ value: 0.04809700451843158
66
  - name: BERTScore
67
  type: bertscore
68
+ value: 0.7353078946893743
69
  - name: MoverScore
70
  type: moverscore
71
+ value: 0.5036973829954939
72
  - task:
73
  name: Text2text Generation
74
  type: text2text-generation
75
  dataset:
76
+ name: lmqg/qg_esquad
77
  type: default
78
  args: default
79
  metrics:
80
  - name: BLEU4
81
  type: bleu4
82
+ value: 0.0059191752064594125
83
  - name: ROUGE-L
84
  type: rouge-l
85
+ value: 0.05208940592236566
86
  - name: METEOR
87
  type: meteor
88
+ value: 0.06021086135293597
89
  - name: BERTScore
90
  type: bertscore
91
+ value: 0.7494422899749911
92
  - name: MoverScore
93
  type: moverscore
94
+ value: 0.5062373132800192
95
  - task:
96
  name: Text2text Generation
97
  type: text2text-generation
98
  dataset:
99
+ name: lmqg/qg_frquad
100
  type: default
101
  args: default
102
  metrics:
103
  - name: BLEU4
104
  type: bleu4
105
+ value: 0.0171464639522496
106
  - name: ROUGE-L
107
  type: rouge-l
108
+ value: 0.1583673053928925
109
  - name: METEOR
110
  type: meteor
111
+ value: 0.08244973027319356
112
  - name: BERTScore
113
  type: bertscore
114
+ value: 0.7291012183458674
115
  - name: MoverScore
116
  type: moverscore
117
+ value: 0.509610854598101
118
  - task:
119
  name: Text2text Generation
120
  type: text2text-generation
121
  dataset:
122
+ name: lmqg/qg_itquad
123
  type: default
124
  args: default
125
  metrics:
126
  - name: BLEU4
127
  type: bleu4
128
+ value: 0.005438910607183992
129
  - name: ROUGE-L
130
  type: rouge-l
131
+ value: 0.05010570221421983
132
  - name: METEOR
133
  type: meteor
134
+ value: 0.05890828426558759
135
  - name: BERTScore
136
  type: bertscore
137
+ value: 0.7260160158030385
138
  - name: MoverScore
139
  type: moverscore
140
+ value: 0.5023119088393686
141
  - task:
142
  name: Text2text Generation
143
  type: text2text-generation
144
  dataset:
145
+ name: lmqg/qg_jaquad
146
  type: default
147
  args: default
148
  metrics:
149
  - name: BLEU4
150
  type: bleu4
151
+ value: 4.4114578660129224e-08
152
  - name: ROUGE-L
153
  type: rouge-l
154
+ value: 0.06084267343290677
155
  - name: METEOR
156
  type: meteor
157
+ value: 0.005149267426183168
158
  - name: BERTScore
159
  type: bertscore
160
+ value: 0.6608093198082075
161
  - name: MoverScore
162
  type: moverscore
163
+ value: 0.46526108687696893
164
  - task:
165
  name: Text2text Generation
166
  type: text2text-generation
167
  dataset:
168
+ name: lmqg/qg_koquad
169
  type: default
170
  args: default
171
  metrics:
172
  - name: BLEU4
173
  type: bleu4
174
+ value: 1.4750917137316939e-12
175
  - name: ROUGE-L
176
  type: rouge-l
177
+ value: 0.0006466767450454226
178
  - name: METEOR
179
  type: meteor
180
+ value: 0.007310046912436679
181
  - name: BERTScore
182
  type: bertscore
183
+ value: 0.6634288882769679
184
  - name: MoverScore
185
  type: moverscore
186
+ value: 0.4586124640357038
187
  - task:
188
  name: Text2text Generation
189
  type: text2text-generation
190
  dataset:
191
+ name: lmqg/qg_ruquad
192
  type: default
193
  args: default
194
  metrics:
195
  - name: BLEU4
196
  type: bleu4
197
+ value: 4.229109829516021e-12
198
  - name: ROUGE-L
199
  type: rouge-l
200
+ value: 0.009881091250723615
201
  - name: METEOR
202
  type: meteor
203
+ value: 0.017796529053904556
204
  - name: BERTScore
205
  type: bertscore
206
+ value: 0.7089446693028568
207
  - name: MoverScore
208
  type: moverscore
209
+ value: 0.49098728551715626
210
  ---
211
 
212
  # Model Card of `lmqg/mt5-small-squad`
213
+ This model is fine-tuned version of [google/mt5-small](https://huggingface.co/google/mt5-small) for question generation task on the [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) (dataset_name: default) via [`lmqg`](https://github.com/asahi417/lm-question-generation).
 
214
 
215
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
216
  ### Overview
217
  - **Language model:** [google/mt5-small](https://huggingface.co/google/mt5-small)
218
  - **Language:** en
 
224
  ### Usage
225
  - With [`lmqg`](https://github.com/asahi417/lm-question-generation#lmqg-language-model-for-question-generation-)
226
  ```python
 
227
  from lmqg import TransformersQG
228
+
229
  # initialize model
230
+ model = TransformersQG(language="en", model="lmqg/mt5-small-squad")
231
+
232
  # model prediction
233
+ questions = model.generate_q(list_context="William Turner was an English painter who specialised in watercolour landscapes", list_answer="William Turner")
234
 
235
  ```
236
 
237
  - With `transformers`
238
  ```python
 
239
  from transformers import pipeline
 
 
 
 
240
 
241
+ pipe = pipeline("text2text-generation", "lmqg/mt5-small-squad")
242
+ output = pipe("<hl> Beyonce <hl> further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records.")
243
 
244
+ ```
245
 
246
+ ## Evaluation
247
 
 
248
 
249
+ - ***Metric (Question Generation)***: [raw metric file](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval/metric.first.sentence.paragraph_answer.question.lmqg_qg_squad.default.json)
 
 
250
 
251
+ | | Score | Type | Dataset |
252
+ |:-----------|--------:|:--------|:---------------------------------------------------------------|
253
+ | BERTScore | 90.01 | default | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) |
254
+ | Bleu_1 | 54.07 | default | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) |
255
+ | Bleu_2 | 37.62 | default | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) |
256
+ | Bleu_3 | 28.18 | default | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) |
257
+ | Bleu_4 | 21.65 | default | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) |
258
+ | METEOR | 23.83 | default | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) |
259
+ | MoverScore | 62.75 | default | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) |
260
+ | ROUGE_L | 48.95 | default | [lmqg/qg_squad](https://huggingface.co/datasets/lmqg/qg_squad) |
261
 
262
 
263
+ - ***Metrics (Question Generation, Out-of-Domain)***
264
 
265
+ | Dataset | Type | BERTScore| Bleu_4 | METEOR | MoverScore | ROUGE_L | Link |
266
+ |:--------|:-----|---------:|-------:|-------:|-----------:|--------:|-----:|
267
+ | [lmqg/qg_dequad](https://huggingface.co/datasets/lmqg/qg_dequad) | default | 73.53 | 0.0 | 4.81 | 50.37 | 1.56 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_dequad.default.json) |
268
+ | [lmqg/qg_esquad](https://huggingface.co/datasets/lmqg/qg_esquad) | default | 74.94 | 0.59 | 6.02 | 50.62 | 5.21 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_esquad.default.json) |
269
+ | [lmqg/qg_frquad](https://huggingface.co/datasets/lmqg/qg_frquad) | default | 72.91 | 1.71 | 8.24 | 50.96 | 15.84 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_frquad.default.json) |
270
+ | [lmqg/qg_itquad](https://huggingface.co/datasets/lmqg/qg_itquad) | default | 72.6 | 0.54 | 5.89 | 50.23 | 5.01 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_itquad.default.json) |
271
+ | [lmqg/qg_jaquad](https://huggingface.co/datasets/lmqg/qg_jaquad) | default | 66.08 | 0.0 | 0.51 | 46.53 | 6.08 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_jaquad.default.json) |
272
+ | [lmqg/qg_koquad](https://huggingface.co/datasets/lmqg/qg_koquad) | default | 66.34 | 0.0 | 0.73 | 45.86 | 0.06 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_koquad.default.json) |
273
+ | [lmqg/qg_ruquad](https://huggingface.co/datasets/lmqg/qg_ruquad) | default | 70.89 | 0.0 | 1.78 | 49.1 | 0.99 | [link](https://huggingface.co/lmqg/mt5-small-squad/raw/main/eval_ood/metric.first.sentence.paragraph_answer.question.lmqg_qg_ruquad.default.json) |
274
 
275
 
276
  ## Training hyperparameters
 
296
 
297
  ## Citation
298
  ```
 
299
  @inproceedings{ushio-etal-2022-generative,
300
  title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
301
  author = "Ushio, Asahi and