Hugging Face's logo

lmqg
/
mt5-small-squad-qg

Text2Text Generation
Transformers
PyTorch
English
mt5
question generation
Eval Results
Inference Endpoints
Model card Files Community
1
mt5-small-squad-qg / README.md
asahi417's picture
asahi417
model update
4616cbd
preview code
|
raw history blame
9.29 kB
metadata
license: cc-by-4.0
metrics:
  - bleu4
  - meteor
  - rouge-l
  - bertscore
  - moverscore
language: en
datasets:
  - lmqg/qg_squad
pipeline_tag: text2text-generation
tags:
  - question generation
widget:
  - text: >-
      <hl> Beyonce <hl> further expanded her acting career, starring as blues
      singer Etta James in the 2008 musical biopic, Cadillac Records.
    example_title: Question Generation Example 1
  - text: >-
      Beyonce further expanded her acting career, starring as blues singer <hl>
      Etta James <hl> in the 2008 musical biopic, Cadillac Records.
    example_title: Question Generation Example 2
  - text: >-
      Beyonce further expanded her acting career, starring as blues singer Etta
      James in the 2008 musical biopic,  <hl> Cadillac Records <hl> .
    example_title: Question Generation Example 3
model-index:
  - name: lmqg/mt5-small-squad
    results:
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_squad
          type: default
          args: default
        metrics:
          - name: BLEU4
            type: bleu4
            value: 0.21650505934418166
          - name: ROUGE-L
            type: rouge-l
            value: 0.489464328982525
          - name: METEOR
            type: meteor
            value: 0.23833897056449205
          - name: BERTScore
            type: bertscore
            value: 0.9000723844397448
          - name: MoverScore
            type: moverscore
            value: 0.62747065964027
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_itquad
          type: default
          args: default
        metrics:
          - name: BLEU4
            type: bleu4
            value: 0.005438910607183992
          - name: ROUGE-L
            type: rouge-l
            value: 0.05010570221421983
          - name: METEOR
            type: meteor
            value: 0.05890828426558759
          - name: BERTScore
            type: bertscore
            value: 0.7260160158030385
          - name: MoverScore
            type: moverscore
            value: 0.5023119088393686
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_jaquad
          type: default
          args: default
        metrics:
          - name: BLEU4
            type: bleu4
            value: 4.4114578660129224e-8
          - name: ROUGE-L
            type: rouge-l
            value: 0.06084267343290677
          - name: METEOR
            type: meteor
            value: 0.005149267426183168
          - name: BERTScore
            type: bertscore
            value: 0.6608093198082075
          - name: MoverScore
            type: moverscore
            value: 0.46526108687696893
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_ruquad
          type: default
          args: default
        metrics:
          - name: BLEU4
            type: bleu4
            value: 4.229109829516021e-12
          - name: ROUGE-L
            type: rouge-l
            value: 0.009881091250723615
          - name: METEOR
            type: meteor
            value: 0.017796529053904556
          - name: BERTScore
            type: bertscore
            value: 0.7089446693028568
          - name: MoverScore
            type: moverscore
            value: 0.49098728551715626
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_dequad
          type: default
          args: default
        metrics:
          - name: BLEU4
            type: bleu4
            value: 9.242783121165897e-12
          - name: ROUGE-L
            type: rouge-l
            value: 0.01556150764938016
          - name: METEOR
            type: meteor
            value: 0.04809700451843158
          - name: BERTScore
            type: bertscore
            value: 0.7353078946893743
          - name: MoverScore
            type: moverscore
            value: 0.5036973829954939
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_esquad
          type: default
          args: default
        metrics:
          - name: BLEU4
            type: bleu4
            value: 0.0059191752064594125
          - name: ROUGE-L
            type: rouge-l
            value: 0.05208940592236566
          - name: METEOR
            type: meteor
            value: 0.06021086135293597
          - name: BERTScore
            type: bertscore
            value: 0.7494422899749911
          - name: MoverScore
            type: moverscore
            value: 0.5062373132800192
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_frquad
          type: default
          args: default
        metrics:
          - name: BLEU4
            type: bleu4
            value: 0.0171464639522496
          - name: ROUGE-L
            type: rouge-l
            value: 0.1583673053928925
          - name: METEOR
            type: meteor
            value: 0.08244973027319356
          - name: BERTScore
            type: bertscore
            value: 0.7291012183458674
          - name: MoverScore
            type: moverscore
            value: 0.509610854598101
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_koquad
          type: default
          args: default
        metrics:
          - name: BLEU4
            type: bleu4
            value: 1.4750917137316939e-12
          - name: ROUGE-L
            type: rouge-l
            value: 0.0006466767450454226
          - name: METEOR
            type: meteor
            value: 0.007310046912436679
          - name: BERTScore
            type: bertscore
            value: 0.6634288882769679
          - name: MoverScore
            type: moverscore
            value: 0.4586124640357038

Language Models Fine-tuning on Question Generation: lmqg/mt5-small-squad

This model is fine-tuned version of google/mt5-small for question generation task on the lmqg/qg_squad (dataset_name: default).

Overview

Usage 


from transformers import pipeline

model_path = 'lmqg/mt5-small-squad'
pipe = pipeline("text2text-generation", model_path)

# Question Generation
question = pipe('<hl> Beyonce <hl> further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records.')

Evaluation Metrics

Metrics

Dataset Type BLEU4 ROUGE-L METEOR BERTScore MoverScore Link
lmqg/qg_squad default 0.217 0.489 0.238 0.9 0.627 link

Out-of-domain Metrics

Dataset Type BLEU4 ROUGE-L METEOR BERTScore MoverScore Link
lmqg/qg_itquad default 0.005 0.05 0.059 0.726 0.502 link
lmqg/qg_jaquad default 0.0 0.061 0.005 0.661 0.465 link
lmqg/qg_ruquad default 0.0 0.01 0.018 0.709 0.491 link
lmqg/qg_dequad default 0.0 0.016 0.048 0.735 0.504 link
lmqg/qg_esquad default 0.006 0.052 0.06 0.749 0.506 link
lmqg/qg_frquad default 0.017 0.158 0.082 0.729 0.51 link
lmqg/qg_koquad default 0.0 0.001 0.007 0.663 0.459 link

Training hyperparameters

The following hyperparameters were used during fine-tuning:

  • dataset_path: lmqg/qg_squad
  • dataset_name: default
  • input_types: ['paragraph_answer']
  • output_types: ['question']
  • prefix_types: None
  • model: google/mt5-small
  • max_length: 512
  • max_length_output: 32
  • epoch: 15
  • batch: 64
  • lr: 0.0005
  • fp16: False
  • random_seed: 1
  • gradient_accumulation_steps: 1
  • label_smoothing: 0.15

The full configuration can be found at fine-tuning config file.

Citation

TBA