asahi417's picture
model update
1be914d
metadata
license: cc-by-4.0
metrics:
  - bleu4
  - meteor
  - rouge-l
  - bertscore
  - moverscore
language: es
datasets:
  - lmqg/qg_esquad
pipeline_tag: text2text-generation
tags:
  - question generation
  - answer extraction
widget:
  - text: >-
      generate question: del <hl> Ministerio de Desarrollo Urbano <hl> ,
      Gobierno de la India.
    example_title: Question Generation Example 1
  - text: >-
      generate question: a <hl> noviembre <hl> , que es también la estación
      lluviosa.
    example_title: Question Generation Example 2
  - text: >-
      generate question: como <hl> el gobierno de Abbott <hl> que asumió el
      cargo el 18 de septiembre de 2013.
    example_title: Question Generation Example 3
  - text: >-
      extract answers: <hl> En la diáspora somalí, múltiples eventos islámicos
      de recaudación de fondos se llevan a cabo cada año en ciudades como
      Birmingham, Londres, Toronto y Minneapolis, donde los académicos y
      profesionales somalíes dan conferencias y responden preguntas de la
      audiencia. <hl> El propósito de estos eventos es recaudar dinero para
      nuevas escuelas o universidades en Somalia, para ayudar a los somalíes que
      han sufrido como consecuencia de inundaciones y / o sequías, o para reunir
      fondos para la creación de nuevas mezquitas como.
    example_title: Answer Extraction Example 1
  - text: >-
      extract answers: <hl> Los estudiosos y los histori a dores están divididos
      en cuanto a qué evento señala el final de la era helenística. <hl> El
      período helenístico se puede ver que termina con la conquista final del
      corazón griego por Roma en 146 a. C. tras la guerra aquea, con la derrota
      final del reino ptolemaico en la batalla de Actium en 31 a. Helenístico se
      distingue de helénico en que el primero abarca toda la esfera de
      influencia griega antigua directa, mientras que el segundo se refiere a la
      propia Grecia.
    example_title: Answer Extraction Example 2
model-index:
  - name: lmqg/mbart-large-cc25-esquad-qg-ae
    results:
      - task:
          name: Text2text Generation
          type: text2text-generation
        dataset:
          name: lmqg/qg_esquad
          type: default
          args: default
        metrics:
          - name: BLEU4 (Question Generation)
            type: bleu4_question_generation
            value: 7.61
          - name: ROUGE-L (Question Generation)
            type: rouge_l_question_generation
            value: 20.95
          - name: METEOR (Question Generation)
            type: meteor_question_generation
            value: 19.58
          - name: BERTScore (Question Generation)
            type: bertscore_question_generation
            value: 79.36
          - name: MoverScore (Question Generation)
            type: moverscore_question_generation
            value: 56.05
          - name: >-
              QAAlignedF1Score-BERTScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_f1_score_bertscore_question_answer_generation_with_gold_answer
            value: 81.13
          - name: >-
              QAAlignedRecall-BERTScore (Question & Answer Generation (with Gold
              Answer))
            type: >-
              qa_aligned_recall_bertscore_question_answer_generation_with_gold_answer
            value: 84.91
          - name: >-
              QAAlignedPrecision-BERTScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_precision_bertscore_question_answer_generation_with_gold_answer
            value: 77.75
          - name: >-
              QAAlignedF1Score-MoverScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_f1_score_moverscore_question_answer_generation_with_gold_answer
            value: 54.86
          - name: >-
              QAAlignedRecall-MoverScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_recall_moverscore_question_answer_generation_with_gold_answer
            value: 57.16
          - name: >-
              QAAlignedPrecision-MoverScore (Question & Answer Generation (with
              Gold Answer))
            type: >-
              qa_aligned_precision_moverscore_question_answer_generation_with_gold_answer
            value: 52.82
          - name: BLEU4 (Answer Extraction)
            type: bleu4_answer_extraction
            value: 21.5
          - name: ROUGE-L (Answer Extraction)
            type: rouge_l_answer_extraction
            value: 46.66
          - name: METEOR (Answer Extraction)
            type: meteor_answer_extraction
            value: 40.42
          - name: BERTScore (Answer Extraction)
            type: bertscore_answer_extraction
            value: 86.7
          - name: MoverScore (Answer Extraction)
            type: moverscore_answer_extraction
            value: 77.96
          - name: AnswerF1Score (Answer Extraction)
            type: answer_f1_score__answer_extraction
            value: 70.95
          - name: AnswerExactMatch (Answer Extraction)
            type: answer_exact_match_answer_extraction
            value: 52.81

Model Card of lmqg/mbart-large-cc25-esquad-qg-ae

This model is fine-tuned version of facebook/mbart-large-cc25 for question generation and answer extraction jointly on the lmqg/qg_esquad (dataset_name: default) via lmqg.

Overview

Usage

from lmqg import TransformersQG

# initialize model
model = TransformersQG(language="es", model="lmqg/mbart-large-cc25-esquad-qg-ae")

# model prediction
question_answer_pairs = model.generate_qa("a noviembre , que es también la estación lluviosa.")
  • With transformers
from transformers import pipeline

pipe = pipeline("text2text-generation", "lmqg/mbart-large-cc25-esquad-qg-ae")

# answer extraction
answer = pipe("generate question: del <hl> Ministerio de Desarrollo Urbano <hl> , Gobierno de la India.")

# question generation
question = pipe("extract answers: <hl> En la diáspora somalí, múltiples eventos islámicos de recaudación de fondos se llevan a cabo cada año en ciudades como Birmingham, Londres, Toronto y Minneapolis, donde los académicos y profesionales somalíes dan conferencias y responden preguntas de la audiencia. <hl> El propósito de estos eventos es recaudar dinero para nuevas escuelas o universidades en Somalia, para ayudar a los somalíes que han sufrido como consecuencia de inundaciones y / o sequías, o para reunir fondos para la creación de nuevas mezquitas como.")

Evaluation

Score Type Dataset
BERTScore 79.36 default lmqg/qg_esquad
Bleu_1 22.05 default lmqg/qg_esquad
Bleu_2 14.55 default lmqg/qg_esquad
Bleu_3 10.34 default lmqg/qg_esquad
Bleu_4 7.61 default lmqg/qg_esquad
METEOR 19.58 default lmqg/qg_esquad
MoverScore 56.05 default lmqg/qg_esquad
ROUGE_L 20.95 default lmqg/qg_esquad
Score Type Dataset
QAAlignedF1Score (BERTScore) 81.13 default lmqg/qg_esquad
QAAlignedF1Score (MoverScore) 54.86 default lmqg/qg_esquad
QAAlignedPrecision (BERTScore) 77.75 default lmqg/qg_esquad
QAAlignedPrecision (MoverScore) 52.82 default lmqg/qg_esquad
QAAlignedRecall (BERTScore) 84.91 default lmqg/qg_esquad
QAAlignedRecall (MoverScore) 57.16 default lmqg/qg_esquad
Score Type Dataset
AnswerExactMatch 52.81 default lmqg/qg_esquad
AnswerF1Score 70.95 default lmqg/qg_esquad
BERTScore 86.7 default lmqg/qg_esquad
Bleu_1 32.77 default lmqg/qg_esquad
Bleu_2 28.12 default lmqg/qg_esquad
Bleu_3 24.52 default lmqg/qg_esquad
Bleu_4 21.5 default lmqg/qg_esquad
METEOR 40.42 default lmqg/qg_esquad
MoverScore 77.96 default lmqg/qg_esquad
ROUGE_L 46.66 default lmqg/qg_esquad

Training hyperparameters

The following hyperparameters were used during fine-tuning:

  • dataset_path: lmqg/qg_esquad
  • dataset_name: default
  • input_types: ['paragraph_answer', 'paragraph_sentence']
  • output_types: ['question', 'answer']
  • prefix_types: ['qg', 'ae']
  • model: facebook/mbart-large-cc25
  • max_length: 512
  • max_length_output: 32
  • epoch: 5
  • batch: 2
  • lr: 0.0001
  • fp16: False
  • random_seed: 1
  • gradient_accumulation_steps: 32
  • label_smoothing: 0.15

The full configuration can be found at fine-tuning config file.

Citation

@inproceedings{ushio-etal-2022-generative,
    title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
    author = "Ushio, Asahi  and
        Alva-Manchego, Fernando  and
        Camacho-Collados, Jose",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, U.A.E.",
    publisher = "Association for Computational Linguistics",
}