MixQG: Neural Question Generation with Mixed Answer Types
Abstract
Asking good questions is an essential ability for both human and machine intelligence.
However, existing neural question generation approaches mainly focus on short factoid type of answers.
In this paper, we introduce a neural question generator, MixQG, to bridge this gap.
We combine nine question answering datasets with diverse answer types, including yes/no, multiple-choice, extractive, and abstractive answers, to train a single generative model.
We show with empirical results that our model outperforms existing work in both seen and unseen domains, and can generate questions with different cognitive levels when conditioned on different answer types.
We run a human evaluation study to assess the quality of generated questions and ﬁnd that MixQG outperforms the next best model by 10%.
Our code and model checkpoints will be released and integrated with the HuggingFace library to facilitate various downstream applications.
Introduction
Question generation (QG) aims to automatically create questions from a given text passage or document with or without answers.
It has a wide range of applications such as improving question answering (QA) systems (Duan et al., 2017) and search engines (Han et al., 2019) through data augmentation, making chatbots more engaging (Wang et al., 2018; Laban et al., 2020), enabling automatic evaluation (Rebuffel et al., 2021) and fact veriﬁcation (Pan et al., 2021) and facilitating educational applications (Chen et al., 2018).
Earlier QG approaches relied on syntactic rules that incorporated linguistic features into the QG process (Heilman and Smith, 2010; Khullar et al., 2018).
Du et al.
(2017) pointed out some of the limitations of such rule-based systems and formulated the task of question generation as a sequence-to-sequence learning problem.
Based on this formulation, recent works rely on pre-trained
Context :
In the late 17th century, Robert Boyle proved that air is necessary for combustion.
English chemist John Mayow (1641–1679) reﬁned this work by showing that ﬁre requires only a part of air that he called spiritus nitroaereus or just nitroaereus.
In one experiment he found that placing either a mouse or a lit candle in a closed container over water caused the water to rise and replace one-fourteenth of the air’s volume before extinguishing the subjects.
From this he surmised that nitroaereus is consumed in both respiration and combustion.
Question :
Who proved that air is necessary for combustion?
Ext.
Short Answer : Robert Boyle
Question : How did John Mayow ﬁnd that spiritus nitroaereus is consumed in
[Table]
and
[Table]
[Table]
[Table]
discover
[Table]
Transformer-based models to generate answeraware questions (Dong et al., 2019a; Yan et al., 2020a; Lelkes et al., 2021).
However, the majority of QG research so far has been performed on the
SQuAD dataset (Rajpurkar et al., 2016), and as a result, it mainly focuses on factoid short answer questions (Zhang and Bansal, 2019;
Zhou et al., 2019; Su et al., 2020).
In reality, answers can come in a variety of types and forms, e.g., short/long, multiple-choice, yes-no, and extractive/abstractive answers.
We hypothesize that answer types are as important as question types , and that different answer types have their unique QG challenges and result in questions with different cognitive levels.
MixQG combines nine
QA datasets with varied answer types to build a more robust and versatile QG model.
We use pretrained generative language models like T5 (Raffel et al., 2020) and BART (Lewis et al., 2019) without question-speciﬁc or domain-speciﬁc preﬁxes to generate the questions.
Figure 1 illustrates the [Table] above, showing MixQG-generated questions of different cognitive levels for different answer types.
The contribution of this paper is summarized as follows: 1) We train a uniﬁed QG model that achieves state-of-the-art performance in both seen and unseen domains.
We release training code and model checkpoints (base, large, 3B) to facilitate various downstream QG applications 1 . 2)
We show that MixQG is able to produce different cognitive level questions by controlling the answer types.
We conduct a human evaluation study which conﬁrms that MixQG leads to improvements in question quality in a practical quiz design setting.
Methodology 2. Datasets
We leverage nine commonly used QA datasets (Table 1) to train our MixQG model, including six MRQA 2019 Shared Task (Fisch et al., 2019) datasets, NarrativeQA (Koˇciský et al., 2018), MCTest (Richardson et al., 2013), and
BoolQ (Clark et al., 2019).
These represent the majority of large-scale publicly available QA datasets.
We obtain in total 560,193 training examples with different answer types and source domains.
We reserve their validation set for in-domain evaluation.
In most general sense, a QA dataset comprises of < C, Q, A > tuples, where C is a context document, Q is a human-written question, and A is its corresponding answer.
Following a common classiﬁcation of answer types, we bucket each dataset into one of the below categories: 1) Extractive [EX] : the answer to the question is a substring of the context passage.
2) Abstractive [AB] : the answer to the question is written in free-form and is not necessarily contained within the context passage.
3) Multiple-Choice [MC] : question comes with multiple answers to select from, including a single correct option and several distractors.
Yes-No [YN] : the answer is a boolean response.
Datasets that do not comply with the above format, such as ELI5 (Fan et al., 2019) and GooAQ (Khashabi et al., 2021), were excluded from training.
We leave their exploration to future work.
We also leverage a set of datasets unseen during training to evaluate our model’s generalization ability.
Similar to the train datasets, these cover several text sources, domains, and answer types.
Quoref (Dasigi et al., 2019) questions can have disjoint spans as answers and often require coreference resolution.
DROP (Dua et al., 2019) questions require discrete reasoning over the context paragraphs.
QAConv (Wu et al., 2021) uses informative conversations such as emails, channels, and panels as a knowledge source, and it includes extractive answers from multiple text spans.
TweetQA (Xiong et al., 2019) uses social media as an information source and contains abstractive answers.
Note that to generate ﬂuent questions, we need to place some restrictions on the training data we use.
For example, we disregard "ﬁll-in-the-blank" (a.k.a Cloze-style) reading comprehension datasets as their questions are implicit and thus do not aid the QG model.
Similarly, we ensure that our training data does not contain unanswerable questions or multiple-choice questions that are too general (e.g., “which of the following is TRUE according to the passage?”).
[Table]
2. Language Modeling
We rely on a text-to-text framework as a basis for
MixQG (Training details are in Section A).
When combining our training datasets, we encode all inputs and outputs into a uniﬁed plain-text format.
For answer-aware question generation, the input is usually formatted in one of the two ways: (1) prepending ( -pre )
the answer before the context and separating it from the rest of the text by a special separator token or (2) highlighting ( -hl )
the answer span within the context with special highlight tokens (Chan and Fan, 2019).
To maintain ﬂexibility, we rely on prepending the answer since highlighting is only applicable to the extractive answer types.
In particular, we format the inputs to our model such that the answer always precedes the context paragraph and use a “\n” separator in between, as shown in Table 2.
For MC type of data, we only take the correct answer and disregard the distractor options.
For
YN data, we extract entities from the question using spaCy’s NER model 2 and append them to the answer.
The reason for adding additional entities is to restrict the domain of questions, as given a context paragraph, there are many boolean questions whose answer would be yes or no, without further restriction.
Note that no type-speciﬁc preﬁxes are added to the input representation, and the corresponding questions are used as output.
Experimental Results 3. Automatic Metrics
We report the commonly-used metrics applied in the QG research:
BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) scores.
We also report
BERTScore (Zhang et al., 2020), which relies on contextual embeddings to produce the ﬁnal score.
3. In-Domain Analysis
In Table 3, we compare baselines trained solely on the target in-domain dataset against MixQG and
MixQG finetuned .
MixQG indicates our model that is joint trained on nine QA datasets with random sampling, and MixQG finetuned is the one further ﬁne-tuned on the target dataset.
We show results on two datasets: SQuAD and NQ.
Since SQuAD is the most common benchmark for QG, we additionally compare MixQG against existing question generation models such as ProphetNet (Qi et al., 2020) and other T5 variants.
The results show that
MixQG outperforms an equally sized model trained directly on the target dataset.
Given that question styles and dataset domains may vary across
MixQG’s seed datasets, additional ﬁne-tuning on the target dataset further improves the scores.
This shows that MixQG is a strong pretrained model which can be further adapted to speciﬁc use cases.
3. Out-of-Domain Analysis
Table 4 summarizes the evaluations on out-ofdomain datasets of extractive and abstractive answer types.
We observe that a dedicated model trained on the target dataset outperforms MixQG in a zero-shot setting.
One potential reason is that answer and question style in different QA datasets may differ signiﬁcantly.
For example, answers are ambiguous pronouns in the Quoref dataset, and questions in DROP dataset are intentionally created for discrete reasoning.
However, MixQG finetuned obtains the best overall scores after further ﬁnetuning on the target training set, suggesting that MixQG is a strong starting point for further ﬁnetuning question generation models.
3. Human Evaluation
Recent studies have shown that n-gram based metrics may not correlate well with human judgements
Nema and Khapra (2018).
The objective of human evaluation is to evaluate QG models by measuring how useful they are as a tool to aid teachers in quiz creation.
We compare seven QG models and collect 3,164 human-annotated samples fromrecruited teachers.
More details are in Section B.
Quiz Design Task
Given an article on the quiz topic selected from Wikipedia, teachers are asked to specify a quiz concept (a subset of the article) they want to test their students on.
This is used as the target answer input for QG models.
Teachers can then approve a generated question to be included on the quiz or reject it and provide a reason for rejection.
The success of a QG model depends on its question approval rate.
Besides MixQG, three GPT2 baselines (Radford
[Table] [Table] [Table]
ProphetNet
[Table]
[Figure]
et al., 2019), two BART baselines (Lewis et al., 2019), and ProphetNet-Large ﬁnetuned on SQuAD are evaluated.
In Figure 2, we see that MixQG attains a 68.4% acceptance rate, outperforming the next best model by 10%.
MixQG also generates the smallest number of disﬂuent and off target (answer mismatch) questions - with majority of errors coming from wrong context (too general or too speciﬁc) questions.
Generating questions with the right level of speciﬁcity remains a challenge and is a promising direction for future work.
3. Qualitative Analysis
First, we compare MixQG generated questions to the gold questions annotated in ﬁve public QA datasets (Table 5).
We ﬁnd that the generated questions are ﬂuent, relevant, and reasonable to the provided answer and context, even if they differ from the gold label.
This further motivates the need of human evaluation for QG research.
Second, we use the HuggingFace summarization pipeline to obtain the summary of the context, and we feed each sentence of the summary as the target answer to MixQG to obtain questions.
In this way, we can test MixQG’s generalization ability to abstractive answers.
As shown in Figure 5, we observe that feeding in long and abstractive answers can still generate ﬂuent and reasonable questions, suggesting that it is possible to control the question’s cognitive level by its answer.
We leave as future work further research into summary-based unsupervised QA-pair generation.
Lastly, in the Quiz Design study, we ﬁnd there are 106 cases in which the teachers only accepted a single candidate question into the quiz.
MixQG produced the accepted candidate 47 times, more than any of the other models.
We provide three examples of such MixQG-only success cases as well as three instances in which the MixQG’s question was not accepted in Table 6.
Related Work
Question generation’s practical importance has lead to an increasing interest in the ﬁeld.
The early work in QG relied on linguistic templates and rules to produce questions from declarative sentences (Heilman and Smith, 2010; Labutov et al., 2015).
With the success of neural techniques in text generation tasks, applying neural sequence-to-sequence generation models became more common (Du et al., 2017; Sun et al., 2018).
More recent works leverage pre-trained transformer based networks, such as T5 (Raffel et al., 2020), BART (Lewis et al., 2019), PEGASUS (Zhang et al., 2019) and ProphetNet (Yan et al., 2020b), for question generation which have been successful in many applications (Dong et al., 2019b; Lelkes et al., 2021; Rebuffel et al., 2021; Pan et al., 2021).
However, most of the earlier work focuses on using a single QA dataset, such as SQuAD (Rajpurkar et al., 2016).
While working on generation of openended (Cao and Wang, 2021), controllable (Cao and Wang, 2021), multi-hop
(Cho et al., 2021) or cause-effect (Stasaski et al., 2021) questions has gained attention, each direction is studied in isolation as it usually requires a separate QA dataset.
Most directly related to our work is UniﬁedQA (Khashabi et al., 2020), which successfully crosses format boundaries of different QA datasets to train a robust QA system.
It advocates for more general and broader system designs not limited to speciﬁc dataset formats.
Similar to their approach,
MixQG combines multiple QA datasets and trains a single QG system in a text-to-text paradigm.
Conclusion
In this paper, we present MixQG, a question generation model pre-trained on a collection of QA datasets with a mix of answer types.
We show through experiments that the resulting model is a strong starting point for further ﬁne-tuning which achieves state-of-the-art results on target datasets in commonly-used similarity metrics as well as our designed human evaluation.
We release our code and the model checkpoints to facilitate QG research and downstream applications.
Ethical Considerations
MixQG is subject to biases found in the training data of both the underlying text-to-text models and all QA datasets that we have used for pre-training.
We do not collect a new dataset for question generation and instead reuse data from previously published works.
As such, we rely on the published works to follow the responsible data collection practices.
The model is currently English language only which limits its practical applications in the real world.
We hope to make MixQG multilingual as more diverse QA datasets become available in the future.
We validate the proposed model by conducting a human evaluation.
We recruited 10 teachers for a study that lasted a maximum of two hours and gifted each participant a $50 gift card.
A Training Details
Training datasets are listed in Table 1.
For training MixQG, we use several pre-trained text-totext model checkpoints from the HuggingFace library (Wolf et al., 2020).
We ﬁnetune them for question generation using our combined dataset described in Section 2.1.
For most experiments done in this paper, we ﬁnetune on a T5-base model (Raffel et al., 2020).
We also scale up the model and report results for T5-large, T5-3B, and BART-large settings (Appendix D).
We train for 100,000 steps (or 22 epochs) with a learning rate of 3 × 10 −using the AdamW (Loshchilov and Hutter, 2017) optimizer and a batch size of 32.
All training was done on eight A100 NVIDIA GPUs and took approximately 35 hours.
B Quiz Design Task Details
We recruit teachers or ex-teachers from an online group forum.
In total, 20 participants ﬁlled out the interest form, 14 were selected, and 10 completed the study.
The participants had been teachers for at least a year and 3.6 years on average, and had taught diverse subjects such as sciences, history, literature, and IT topics, at various levels from primary school to college-level.
The study was meant to last a maximum of two hours, and participants were gifted a $50 gift card upon completion.
Participants were tasked with creating between 57 quizzes, each with a minimum of 8 concepts, and could pick from a set list of 7 quiz topics, which we pre-selected from the list of featured Wikipedia articles 3 .
We purposefully selected articles within different domains to benchmark the QGen models in diverse topical settings: two in physics (Sustainable Energy, Californium Atom), two in biology (DNA, Enzymes), two in history (Statue of Liberty,
Palazzo Pitti), and one in geology (the K-T extinction).
Participants were given the ﬁrst 500 words 3 https://en.wikipedia.org/wiki/ Wikipedia:Featured_articles of the Wikipedia page of each topic as reading material to select Quiz concepts from.
User interface is shown in Figure 4.
Hierarchical categorization of errors for question generation is shown in Figure 3.
C Qualitative Study Details
To understand MixQG’s performance beyond automated metrics, we analyze its generated questions in Table 5.
It shows several examples of questions generated by MixQG-3B on the validation sets of different datasates along with the ground-truth questions.
We also generate question-answer pairs on Wikipedia articles using a pipeline approach as shown in Figure 5.
First, we use a summarization model 4 to obtain the summary of the context.
Then we feed each sentence of the summary as the target answer to MixQG and obtain the questions.
We observe that the generated questions are grammatically ﬂuent, relevant to the input, and answerable by the target answer paragraph.
We ﬁnd that feeding in longer answers to the model generates more general, higher-level questions about the source article, while short answers prompt more factoidstyle questions.
As a result, we are able to generate questions of varied cognitive levels from the same source document by restricting the answer part of the input.
D Scaling
Table 7 shows the performance of differently sized
MixQG models on SQuAD dataset.
We additionally train MixQG model based on BART-large checkpoint, referred to as MixQG BARTlarge .
As expected, the largest MixQG model (3 billion parameters) performs best among the different model size variants.
[Figure] , Off
Target, Wrong Context) each with several subtypes.
Californium
Californium is a radioactive chemical element with the symbol Cf and atomic number 98.
The element was rst synthesized in 1950 at the
Lawrence Berkeley National Laboratory (then the University of
California Radiation Laboratory), ✖ by bombarding curium with alpha particles (helium-4 ions) . It is an actinide element, the sixth transuranium element to be synthesized, and has the second-highest atomic mass of all the elements that have been produced in amounts large enough to see with the unaided eye (after einsteinium). The element was named after the university and the U.S. state of California.
Two crystalline forms exist for californium under normal pressure: one above and one below 900 °C (1,650 °F).
A third form exists at high Quiz Design [Figure] Quiz Questions
[Figure]
✖
[Table]
on the Wii [Table] for the Wii
[Table]
represents the country’s diversity much better
[Table]
stimulation experiment
[Table]
experiment
[Table]
Context
Model
Question
Annotation
For example, californium can be used [...] when studying materials using neutron diffraction and neutron spectroscopy .
MixQG-L
What methods are used to study materials using californium as a source of neutrons?
No Error
BART-L
What is Californium used to study materials?
Disﬂuent
With the exception of some ectothermic species [...] no tetrapods weighing more than 25 kilograms (55 pounds) survived.
MixQG-L
What size tetrapods did not survive the extinction?
No Error
ProphetNet
How much did tetrapods weigh at the time of the CretaceousPaleogene extinction?
Off Tgt.
The two DNA strands are known as polynucleotides as they are composed of simpler monomeric units called nucleotides .
MixQG-L
What are polynucleotides composed of?
No Error
BART-L
What are polynucleotides?
Off Tgt.
The Statue of Liberty ( Liberty
Enlightening the World ) is a colossal neoclassical sculpture on [...]
ProphetNet
What is another name for the Statue of Liberty?
No Error
MixQG-L
What is the English translation of the
Statue of Liberty?
Off Tgt.
Californium.
The element was named after the university and the U.S. state of California .
ProphetNet
What is Californium named after?
No Error
MixQG-L
Where did Californium get its name?
Wrong
Ctxt
Fossil fuels provide 85% of the world’s energy consumption and the energy system [...]
BART-L
How much of the world’s energy consumption does fossil fuels provide?
No Error
MixQG-L
What percentage of the world’s energy consumption is fossil fuels?
Disﬂuent
[Table]