metadata

license: cc-by-4.0
metrics:
  - bleu4
  - meteor
  - rouge-l
  - bertscore
  - moverscore
language: es
datasets:
  - lmqg/qg_esquad
pipeline_tag: text2text-generation
tags:
  - question generation
  - answer extraction
widget:
  - text: >-
      generate question: del <hl> Ministerio de Desarrollo Urbano <hl> ,
      Gobierno de la India.
    example_title: Question Generation Example 1
  - text: >-
      generate question: a <hl> noviembre <hl> , que es también la estación
      lluviosa.
    example_title: Question Generation Example 2
  - text: >-
      generate question: como <hl> el gobierno de Abbott <hl> que asumió el
      cargo el 18 de septiembre de 2013.
    example_title: Question Generation Example 3
  - text: >-
      extract answers: <hl> En la diáspora somalí, múltiples eventos islámicos
      de recaudación de fondos se llevan a cabo cada año en ciudades como
      Birmingham, Londres, Toronto y Minneapolis, donde los académicos y
      profesionales somalíes dan conferencias y responden preguntas de la
      audiencia. <hl> El propósito de estos eventos es recaudar dinero para
      nuevas escuelas o universidades en Somalia, para ayudar a los somalíes que
      han sufrido como consecuencia de inundaciones y / o sequías, o para reunir
      fondos para la creación de nuevas mezquitas como.
    example_title: Answer Extraction Example 1
  - text: >-
      extract answers: <hl> Los estudiosos y los histori a dores están divididos
      en cuanto a qué evento señala el final de la era helenística. <hl> El
      período helenístico se puede ver que termina con la conquista final del
      corazón griego por Roma en 146 a. C. tras la guerra aquea, con la derrota
      final del reino ptolemaico en la batalla de Actium en 31 a. Helenístico se
      distingue de helénico en que el primero abarca toda la esfera de
      influencia griega antigua directa, mientras que el segundo se refiere a la
      propia Grecia.
    example_title: Answer Extraction Example 2
model-index:
  - name: lmqg/mt5-small-esquad-qg-ae
    results:
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_esquad
          type: default
          args: default
        metrics:
          - name: BLEU4 (Question Generation)
            type: bleu4_question_generation
            value: 8.79
          - name: ROUGE-L (Question Generation)
            type: rouge_l_question_generation
            value: 23.13
          - name: METEOR (Question Generation)
            type: meteor_question_generation
            value: 21.66
          - name: BERTScore (Question Generation)
            type: bertscore_question_generation
            value: 83.39
          - name: MoverScore (Question Generation)
            type: moverscore_question_generation
            value: 58.34
          - name: >-
              QAAlignedF1Score-BERTScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_f1_score_bertscore_question_answer_generation_with_gold_answer
            value: 79.06
          - name: >-
              QAAlignedRecall-BERTScore (Question & Answer Generation (with Gold
              Answer))
            type: >-
              qa_aligned_recall_bertscore_question_answer_generation_with_gold_answer
            value: 81.94
          - name: >-
              QAAlignedPrecision-BERTScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_precision_bertscore_question_answer_generation_with_gold_answer
            value: 76.46
          - name: >-
              QAAlignedF1Score-MoverScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_f1_score_moverscore_question_answer_generation_with_gold_answer
            value: 54.49
          - name: >-
              QAAlignedRecall-MoverScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_recall_moverscore_question_answer_generation_with_gold_answer
            value: 56.21
          - name: >-
              QAAlignedPrecision-MoverScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_precision_moverscore_question_answer_generation_with_gold_answer
            value: 52.96
          - name: BLEU4 (Answer Extraction)
            type: bleu4_answer_extraction
            value: 23.89
          - name: ROUGE-L (Answer Extraction)
            type: rouge_l_answer_extraction
            value: 48.58
          - name: METEOR (Answer Extraction)
            type: meteor_answer_extraction
            value: 43.11
          - name: BERTScore (Answer Extraction)
            type: bertscore_answer_extraction
            value: 89.77
          - name: MoverScore (Answer Extraction)
            type: moverscore_answer_extraction
            value: 80.64
          - name: AnswerF1Score (Answer Extraction)
            type: answer_f1_score__answer_extraction
            value: 75.31
          - name: AnswerExactMatch (Answer Extraction)
            type: answer_exact_match_answer_extraction
            value: 57.63

Model Card of `lmqg/mt5-small-esquad-qg-ae`

This model is fine-tuned version of google/mt5-small for question generation and answer extraction jointly on the lmqg/qg_esquad (dataset_name: default) via lmqg.

Overview

Language model: google/mt5-small
Language: es
Training data: lmqg/qg_esquad (default)
Online Demo: https://autoqg.net/
Repository: https://github.com/asahi417/lm-question-generation
Paper: https://arxiv.org/abs/2210.03992

Usage

With lmqg

from lmqg import TransformersQG

# initialize model
model = TransformersQG(language="es", model="lmqg/mt5-small-esquad-qg-ae")

# model prediction
question_answer_pairs = model.generate_qa("a noviembre , que es también la estación lluviosa.")

With transformers

from transformers import pipeline

pipe = pipeline("text2text-generation", "lmqg/mt5-small-esquad-qg-ae")

# answer extraction
answer = pipe("generate question: del <hl> Ministerio de Desarrollo Urbano <hl> , Gobierno de la India.")

# question generation
question = pipe("extract answers: <hl> En la diáspora somalí, múltiples eventos islámicos de recaudación de fondos se llevan a cabo cada año en ciudades como Birmingham, Londres, Toronto y Minneapolis, donde los académicos y profesionales somalíes dan conferencias y responden preguntas de la audiencia. <hl> El propósito de estos eventos es recaudar dinero para nuevas escuelas o universidades en Somalia, para ayudar a los somalíes que han sufrido como consecuencia de inundaciones y / o sequías, o para reunir fondos para la creación de nuevas mezquitas como.")

Evaluation

Metric (Question Generation): raw metric file

	Score	Type	Dataset
BERTScore	83.39	default	lmqg/qg_esquad
Bleu_1	24.5	default	lmqg/qg_esquad
Bleu_2	16.48	default	lmqg/qg_esquad
Bleu_3	11.83	default	lmqg/qg_esquad
Bleu_4	8.79	default	lmqg/qg_esquad
METEOR	21.66	default	lmqg/qg_esquad
MoverScore	58.34	default	lmqg/qg_esquad
ROUGE_L	23.13	default	lmqg/qg_esquad

Metric (Question & Answer Generation): raw metric file

	Score	Type	Dataset
QAAlignedF1Score (BERTScore)	79.06	default	lmqg/qg_esquad
QAAlignedF1Score (MoverScore)	54.49	default	lmqg/qg_esquad
QAAlignedPrecision (BERTScore)	76.46	default	lmqg/qg_esquad
QAAlignedPrecision (MoverScore)	52.96	default	lmqg/qg_esquad
QAAlignedRecall (BERTScore)	81.94	default	lmqg/qg_esquad
QAAlignedRecall (MoverScore)	56.21	default	lmqg/qg_esquad

Metric (Answer Extraction): raw metric file

	Score	Type	Dataset
AnswerExactMatch	57.63	default	lmqg/qg_esquad
AnswerF1Score	75.31	default	lmqg/qg_esquad
BERTScore	89.77	default	lmqg/qg_esquad
Bleu_1	35.18	default	lmqg/qg_esquad
Bleu_2	30.48	default	lmqg/qg_esquad
Bleu_3	26.92	default	lmqg/qg_esquad
Bleu_4	23.89	default	lmqg/qg_esquad
METEOR	43.11	default	lmqg/qg_esquad
MoverScore	80.64	default	lmqg/qg_esquad
ROUGE_L	48.58	default	lmqg/qg_esquad

Training hyperparameters

The following hyperparameters were used during fine-tuning:

dataset_path: lmqg/qg_esquad
dataset_name: default
input_types: ['paragraph_answer', 'paragraph_sentence']
output_types: ['question', 'answer']
prefix_types: ['qg', 'ae']
model: google/mt5-small
max_length: 512
max_length_output: 32
epoch: 5
batch: 16
lr: 0.001
fp16: False
random_seed: 1
gradient_accumulation_steps: 4
label_smoothing: 0.15

The full configuration can be found at fine-tuning config file.

Citation

@inproceedings{ushio-etal-2022-generative,
    title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
    author = "Ushio, Asahi  and
        Alva-Manchego, Fernando  and
        Camacho-Collados, Jose",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, U.A.E.",
    publisher = "Association for Computational Linguistics",
}