Original: L EARNING TO G ENERATE Q UESTIONS BY R ECOVERING A NSWER - CONTAINING S ENTENCES
Revised: L EARNING TO G ENERATE Q UESTIONS BY R ECOVERING A NSWER - CONTAINING S ENTENCES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: To train a question answering model based on machine reading comprehension (MRC), signiﬁcant effort is required to prepare annotated training data composed of questions and their answers from contexts.
Revised: To train a question answering model based on machine reading comprehension (MRC), signiﬁcant effort is required to prepare annotated training data composed of questions and their answers from contexts.

Original: To mitigate this issue, recent research has focused on synthetically generating a question from a given context and an annotated (or generated) answer by training an additional generative model, which can be utilized to augment the training data.
Revised: To mitigate this issue, recent research has focused on synthetically generating a question from a given context and an annotated (or generated) answer by training an additional generative model, which can be utilized to augment the training data.

Original: In light of this research direction, we propose a novel pre-training approach that learns to generate contextually rich questions, by recovering answer-containing sentences.
Revised: In light of this research direction, we propose a novel pre-training approach that learns to generate contextually rich questions, by recovering answer-containing sentences.

Original: Our approach is composed of two novel components, (1) dynamically determining K answers from a given document and (2) pre-training the question generator on the task of generating the answer-containing sentence.
Revised: Our approach is composed of two novel components, (1) dynamically determining K answers from a given document and (2) pre-training the question generator on the task of generating the answer-containing sentence.

Original: We evaluate our method against existing ones in terms of the quality of generated questions as well as the ﬁne-tuned MRC model accuracy after training on the data synthetically generated by our method.
Revised: We evaluate our method against existing ones in terms of the quality of generated questions as well as the ﬁne-tuned MRC model accuracy after training on the data synthetically generated by our method.

Original: Experimental results demonstrate that our approach consistently improves the question generation capability of existing models such as UniLM, and shows state-of-theart results on MS MARCO and NewsQA, and comparable results to the state-ofthe-art on SQuAD.
Revised: Experimental results demonstrate that our approach consistently improves the question generation capability of existing models such as UniLM, and shows state-of-theart results on MS MARCO and NewsQA, and comparable results to the state-ofthe-art on SQuAD.

Original: Additionally, we demonstrate that the data synthetically generated by our approach is beneﬁcial for boosting up the downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modiﬁcation to the existing MRC models.
Revised: Additionally, we demonstrate that the data synthetically generated by our approach is beneﬁcial for boosting up the downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modiﬁcation to the existing MRC models.

Original: Furthermore, our experiments highlight that our method shines especially when a limited amount of training data is given, in terms of both pre-training and downstream MRC data.
Revised: Furthermore, our experiments highlight that our method shines especially when a limited amount of training data is given, in terms of both pre-training and downstream MRC data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine reading comprehension (MRC), which ﬁnds the answer to a given question from its accompanying paragraphs (called context), is an essential task in natural language processing.
Revised: Machine reading comprehension (MRC), which ﬁnds the answer to a given question from its accompanying paragraphs (called context), is an essential task in natural language processing.

Original: With the release of high-quality human-annotated datasets for this task, such as SQuAD-v1.1 (Rajpurkar et al., 2016), SQuAD-v2.0 (Rajpurkar et al., 2018), and KorQuAD (Lim et al., 2019), researchers have proposed MRC models even surpassing human performance.
Revised: With the release of high-quality human-annotated datasets for this task, such as SQuAD-v1.1 (Rajpurkar et al., 2016), SQuAD-v2.0 (Rajpurkar et al., 2018), and KorQuAD (Lim et al., 2019), researchers have proposed MRC models even surpassing human performance.

Original: These datasets commonly involve ﬁnding a snippet within a context as an answer to a given question.
Revised: These datasets commonly involve ﬁnding a snippet within a context as an answer to a given question.

Original: However, these datasets require signiﬁcant amount of human effort to create questions and their relevant answers from given contexts.
Revised: However, these datasets require signiﬁcant amount of human effort to create questions and their relevant answers from given contexts.

Original: Often the size of the annotated data is relatively small compared to that of data used in other self-supervised tasks such as language modeling, limiting the accuracy.
Revised: Often the size of the annotated data is relatively small compared to that of data used in other self-supervised tasks such as language modeling, limiting the accuracy.

Original: To overcome this issue, researchers have studied models for generating synthetic questions from a given context along with annotated (or generated) answers on large corpora such as Wikipedia.
Revised: To overcome this issue, researchers have studied models for generating synthetic questions from a given context along with annotated (or generated) answers on large corpora such as Wikipedia.

Original: Golub et al.
Revised: Golub et al.

Original: (2017) suggest a two-stage network of generating question-answer pairs which ﬁrst chooses answers conditioned on the paragraph and then generates a question conditioned on the chosen answer.
Revised: (2017) suggest a two-stage network of generating question-answer pairs which ﬁrst chooses answers conditioned on the paragraph and then generates a question conditioned on the chosen answer.

Original: Dong et al.
Revised: Dong et al.

Original: (2019) showed that pre-training on uniﬁed language modeling from large corpora including Wikipedia improves the question generation capability.
Revised: (2019) showed that pre-training on uniﬁed language modeling from large corpora including Wikipedia improves the question generation capability.

Original: Similarly, Alberti et al.
Revised: Similarly, Alberti et al.

Original: (2019) introduced a self-supervised pre-training technique for question generation via the nextsentence generation task.
Revised: (2019) introduced a self-supervised pre-training technique for question generation via the nextsentence generation task.

Original: However, self-supervised pre-training techniques such as language modeling or next sentence generation are not speciﬁcally conditioned on the candidate answer and instead treat it like any other
Revised: However, self-supervised pre-training techniques such as language modeling or next sentence generation are not speciﬁcally conditioned on the candidate answer and instead treat it like any other

Original: [Figure]
Revised: [Figure]

Original: phrase, despite the candidate answer being a strong conditional restriction for the question generation task.
Revised: phrase, despite the candidate answer being a strong conditional restriction for the question generation task.

Original: Also, not all sentences from a paragraph may be relevant to the questions or answers, so  their generation may not be an ideal candidate  question generation tasks.
Revised: Also, not all sentences from a paragraph may be relevant to the questions or answers, so task of their generation may not be an ideal candidate as a pre-training method for question generation tasks.

Original: Moreover, choosing the number of candidate answers when they are not annotated will help to avoid generating too few or lower-quality answers.
Revised: Moreover, in question generation it is important to determine which part of a given context can be a suitable answer for generating questions.

Original: To address these issues, we propose a novel training method called Answer-containing Sentence
Revised: To address these issues, we propose a novel training method called Answer-containing Sentence

Original: Generation (ASGen) for a question generator.
Revised: Generation (ASGen) for a question generator.

Original: ASGen is composed of two steps: (1) dynamically predicting K answers to generate diverse questions and (2) pre-training the question generator on the answer-containing sentence generation task.
Revised: ASGen is composed of two steps: (1) dynamically predicting K answers to generate diverse questions and (2) pre-training the question generator on the answer-containing sentence generation task.

Original: We evaluate our method against existing ones in terms of the generated question quality as well as the ﬁne-tuned MRC model accuracy after training on the data synthetically generated by our method.
Revised: We evaluate our method against existing ones in terms of the generated question quality as well as the ﬁne-tuned MRC model accuracy after training on the data synthetically generated by our method.

Original: Experimental results demonstrate that our approach consistently improves the question generation quality of existing models such as UniLM (Dong et al., 2019) and shows state-of-the-art results on
Revised: Experimental results demonstrate that our approach consistently improves the question generation quality of existing models such as UniLM (Dong et al., 2019) and shows state-of-the-art results on

Original: MS MARCO (Nguyen et al., 2016), NewsQA (Trischler et al., 2017) as well as comparable results to the state-of-the-art on SQuAD.
Revised: MS MARCO (Nguyen et al., 2016), NewsQA (Trischler et al., 2017) as well as comparable results to the state-of-the-art on SQuAD.

Original: Additionally, we demonstrate that the synthetically generated data by our approach can boost up downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modiﬁcation to the existing MRC models.
Revised: Additionally, we demonstrate that the synthetically generated data by our approach can boost up downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modiﬁcation to the existing MRC models.

Original: Furthermore, our experiments highlight that our method shines especially when a limited amount of training data is given, in terms of both pre-training and downstream MRC data.
Revised: Furthermore, our experiments highlight that our method shines especially when a limited amount of training data is given, in terms of both pre-training and downstream MRC data.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: This section discusses our proposed training method called Answer-containing Sentence Generation (ASGen).
Revised: This section discusses our proposed training method called Answer-containing Sentence Generation (ASGen).

Original: While ASGen can be applied to any generative model, we use a simple Transformer (Vaswani et al., 2017) based generative model as our baseline, which we call BertGen.
Revised: While ASGen can be applied to any generative model, we use a simple Transformer (Vaswani et al., 2017) based generative model as our baseline, which we call BertGen.

Original: First, we will describe how the BertGen model generates synthetic questions and answers from a context.
Revised: First, we will describe how the BertGen model generates synthetic questions and answers from a context.

Original: Next, we will explain the novel components of our methods and how we pre-trained the question generator in BertGen based on them.
Revised: Next, we will explain the novel components of our methods and how we pre-trained the question generator in BertGen based on them.

Original: BertGen encodes given paragraphs with two networks, the answer generator and the question generator.
Revised: BertGen encodes given paragraphs with two networks, the answer generator and the question generator.

Original: Answer Generator.
Revised: Answer Generator.

Original: To make the contextual embeddings and to predict answer spans for a given context without the question, we utilize a BERT (Devlin et al., 2019) encoder (Fig. 1-(1), BERT
Revised: To make the contextual embeddings and to predict answer spans for a given context without the question, we utilize a BERT (Devlin et al., 2019) encoder (Fig. 1-(1), BERT

Original: Encoder-A).
Revised: Encoder-A).

Original: We estimate the number of answer candidates K by applying a fully connected layer on the contextual embedding of BERT’s classiﬁcation token “[CLS]”.
Revised: We estimate the number of answer candidates K by applying a fully connected layer on the contextual embedding of BERT’s classiﬁcation token “[CLS]”.

Original: Depending on the estimated number K , we select the K top candidate answer spans from the context.
Revised: Depending on the estimated number K , we select the K top candidate answer spans from the context.

Original: We use the K selected answer spans as input to the question generator.
Revised: We use the K selected answer spans as input to the question generator.

Original: Question Generator.
Revised: Question Generator.

Original: Next, we generate a question conditioned on each answer predicted from the answer generator.
Revised: Next, we generate a question conditioned on each answer predicted from the answer generator.

Original: Speciﬁcally, we give as input to a BERT encoder the context and an indicator for the answer span location in the context (Fig. 1-(2), BERT Encoder-Q).
Revised: Speciﬁcally, we give as input to a BERT encoder the context and an indicator for the answer span location in the context (Fig. 1-(2), BERT Encoder-Q).

Original: Next, a Transformer decoder generates the question word-by-word based on the encoded representation of the context and the answer span.
Revised: Next, a Transformer decoder generates the question word-by-word based on the encoded representation of the context and the answer span.

Original: When pre-training the question generator on an answer-containing sentence generation task, we exclude the answer-containing sentence from the original context and train the model to generate the excluded sentence given the modiﬁed context and the answer span as input.
Revised: When pre-training the question generator on an answer-containing sentence generation task, we exclude the answer-containing sentence from the original context and train the model to generate the excluded sentence given the modiﬁed context and the answer span as input.

Original: Finally, we generate synthetic questions and answers from a large corpus, e.g., all the paragraphs in Wikipedia.
Revised: Finally, we generate synthetic questions and answers from a large corpus, e.g., all the paragraphs in Wikipedia.

Original: After generating this data, we train the MRC model on the generated data in the ﬁrst phase and then ﬁne-tune on the downstream MRC dataset (e.g., SQuAD) in the second phase.
Revised: After generating this data, we train the MRC model on the generated data in the ﬁrst phase and then ﬁne-tune on the downstream MRC dataset (e.g., SQuAD) in the second phase.

Original: In this paper, we use BERT as the default MRC model, since BERT or its variants achieve state-of-the-art performance across numerous MRC tasks.
Revised: In this paper, we use BERT as the default MRC model, since BERT or its variants achieve state-of-the-art performance across numerous MRC tasks.

Original: 2. D YNAMIC A NSWER P REDICTION
Revised: 2. D YNAMIC A NSWER P REDICTION

Revised: In question generation, it is important to determine which part of a given context can be a suitable answer for generating questions.

Original: We predict the number of answers K in a given context, W = { w t } Tt =0 , using regression as
Revised: To this end, we predict the number of answer K in a given context W = { w t } Tt =0 to obtain a more appropriate set of “answer-like” phrases, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where T is the number of word tokens in the context, and position 0 reserved for classiﬁcation token ‘ [ CLS ] ’.
Revised: where T is the number of word tokens in the context, and position 0 reserved for classiﬁcation token ‘ [ CLS ] ’.

Original: f k represents a fully connected unit with two hidden layers that have hidden dimensions equal to H and 1 , respectively, where H is the hidden dimension of BERT Encoder-A. For training, we use the mean squared error loss between K and ground-truth number of answers K gt .
Revised: f k represents a fully connected unit with two hidden layers that have hidden dimensions equal to H and 1 , respectively, where H is the hidden dimension of BERT Encoder-A. For training, we use the mean squared error loss between the output value of f k and ground-truth number of answers K gt .

Original: To calculate the score s i for start index i of a predicted answer span, we compute the dot product of the encoder output with a trainable vector v s .
Revised: To calculate the score s i for start index i of a predicted answer span, we compute the dot product of the encoder output with a trainable vector v s .

Original: For each start index i , we calculate the span end index score e i,j for index j in a similar manner with a trainable vector v e , i.e.,
Revised: For each start index i , we calculate the span end index score e i,j for index j in a similar manner with a trainable vector v e , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where f s represents a fully connected layer with hidden dimension H and ⊕ indicates the concatenation operation.
Revised: where f s represents a fully connected layer with hidden dimension H and ⊕ indicates the concatenation operation.

Original: For training, we use cross-entropy loss on the s i , e i,j and ground truth start, end of the answer span for each token.
Revised: For training, we use cross-entropy loss on the s i , e i,j and ground truth start, end of the answer span for each token.

Original: Predicting the number of answers and the answer span are jointly trained to minimize the sum of their respective losses.
Revised: Predicting the number of answers and the answer span are jointly trained to minimize the sum of their respective losses.

Original: During inference, we choose the K top answer spans with the highest score summation of start index score and end index score, i.e.,
Revised: During inference, we choose the K top answer spans with the highest score summation of start index score and end index score, i.e.,

Original: [Equation]
Revised: [Equation]

Original: The K selected answer spans A spank are then given to the question generator as input in the form of an indication of the answer span location in the given context.
Revised: The K selected answer spans A spank are then given to the question generator as input in the form of an indication of the answer span location in the given context.

Original: 2. P RE - TRAINING Q UESTION G ENERATOR
Revised: 2. P RE - TRAINING Q UESTION G ENERATOR

Original: In order to generate questions conditioned on different answers that may arise in a context, we generate a question for each of the K answers.
Revised: In order to generate questions conditioned on different answers that may arise in a context, we generate a question for each of the K answers.

Original: Alberti et al.
Revised: Alberti et al.

Original: (2019) proposed to pre-train this generation model using  self-supervised tasks such as language modeling or next-sentence generation.
Revised: (2019) proposed a pre-training method for this generative model using the self-supervised task of generating the next-sentence.

Original: We identify several issues with this approach.
Revised: We identify several issues with this approach.

Original: This technique is not speciﬁcally conditioned on the answer, despite the answer being a strong condition for the question generation task.
Revised: This technique is not speciﬁcally conditioned on the answer, despite the answer being a strong condition for the question generation task.

Original: Also, not all sentences from a paragraph may be relevant to the questions or answers from within that paragraph, so their generation is not an ideal candidate for pre-training question generation model.
Revised: Also, not all sentences from a paragraph may be relevant to the questions or answers from within that paragraph, so their generation is not an ideal candidate for pre-training question generation model.

Original: To address these issues, we modify the context to exclude the sentence containing the previously generated answer and pre-train the question generation model on the task of generating this excluded answer-containing sentence, conditioned on the answer and the modiﬁed context.
Revised: To address these issues, we modify the context to exclude the sentence containing the previously generated answer and pre-train the question generation model on the task of generating this excluded answer-containing sentence, conditioned on the answer and the modiﬁed context.

Original: Speciﬁcally, we exclude answer-containing sentence S ans while retaining the answer, modifying the original context D to D ans as
Revised: Speciﬁcally, we exclude answer-containing sentence S ans while retaining the answer, modifying the original context D to D ans as

Original: [Equation]
Revised: [Equation]

Original: ] ; D [ i : j ] ; D [ p e :] ] , ( p s , p e , i, j ) ∈ S ans , where ( i, j ) ∈
Revised: where ( i, j ) ∈

Original: A spank .
Revised: A spank .

Original: Note that we change S ans to not exclude the answer-containing sentence for ﬁne-tuning the question generator, i.e.,
Revised: Note that we change S ans to not exclude the answer-containing sentence for ﬁne-tuning the question generator, i.e.,

Original: [Equation]
Revised: [Equation]

Original: In BertGen, we pass the previously generated answer to the generation model in the form of an additional position encoding M ans that indicates the answer location within the context, i.e.,
Revised: In BertGen, we pass the previously generated answer to the generation model in the form of an additional position encoding M ans that indicates the answer location within the context, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where m 0 and m 1 indicate trainable vectors corresponding to encoding id 0 and 1 , respectively.
Revised: where m 0 and m 1 indicate trainable vectors corresponding to encoding id 0 and 1 , respectively.

Original: That is, we assign the encoding id for each word in the context as 0 and each word in the answer as 1 .
Revised: That is, we assign the encoding id for each word in the context as 0 and each word in the answer as 1 .

Original: A ∗ B indicates the operation of stacking vector A for B many times.
Revised: A ∗ B indicates the operation of stacking vector A for B many times.

Original: Next, we generate answer-containing sentence output embedding W o = { w ot } T 0using as
Revised: Next, we generate answer-containing sentence output words probability W o = { w ot } T 0 as

Original: [Equation]
Revised: [Equation]

Original: where C enc is encoded representation of the context and E ∈ R d × D represents a word embedding matrix with vocabulary size D shared between the BERT Encoder-Q and the Transformer decoder.
Revised: where C enc is encoded representation of the context and E ∈ R d × D represents a word embedding matrix with vocabulary size D shared between the BERT Encoder-Q and the Transformer decoder.

Original: Finally, we calculate the loss of the generated words using the cross-entropy loss as
Revised: Finally, we calculate the loss of the generated words using the cross-entropy loss as

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: where y indicates a ground-truth one-hot vector of the answer-containing sentence word.
Revised: where y indicates a ground-truth one-hot vector of the answer-containing sentence word.

Original: Note that y is the question word in the case of ﬁne-tuning.
Revised: Note that y is the question word in the case of ﬁne-tuning.

Original: In this manner, we pre-train the question generation model using a task similar to the ﬁnal task of conditionally generating the question from a given answer and a context.
Revised: In this manner, we pre-train the question generation model using a task similar to the ﬁnal task of conditionally generating the question from a given answer and a context.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Pre-training Dataset.
Revised: Pre-training Dataset.

Original: To build the dataset for answer-containing sentence generation tasks (ASGen) and the synthetic MRC data for pre-training the downstream MRC models, we collect all paragraphs from the entire English Wikipedia dump and synthetically generate questions and answers on these paragraphs.
Revised: To build the dataset for answer-containing sentence generation tasks (ASGen) and the synthetic MRC data for pre-training the downstream MRC models, we collect all paragraphs from the entire English Wikipedia dump and synthetically generate questions and answers on these paragraphs.

Original: We apply ﬁltering and clean-up steps that are detailed in the appendix.
Revised: We apply ﬁltering and clean-up steps that are detailed in the appendix.

Original: Using BertGen, we  generate  43M triples of question-answer-paragraph for the synthetic data.
Revised: Using BertGen, we extract answers from each given paragraph, and then generate questions for each answer-paragraph pairs. Finally, we obtain 43M triples of question-answer-paragraph for the synthetic data.

Original: For pre-training on answer-containing sentence generation, we sample 25M answer-paragraph pairs (Full-Wiki) from the ﬁnal Wikipedia dataset to avoid extremely short contexts less than 500 characters.
Revised: For pre-training on answer-containing sentence generation, we sample 25M answerparagraph pairs (Full-Wiki) from the ﬁnal Wikipedia dataset to avoid extremely short contexts less than 500 characters.

Original: For ablation studies on pre-training approaches, we sample 2.5M pairs (Small-Wiki) 1 from Full-Wiki and split 25K pairs (Test-Wiki) to evaluate the pre-training method.
Revised: For ablation studies on pre-training approaches, we sample 2.5M pairs (SmallWiki) 1 from Full-Wiki and split 25K pairs (Test-Wiki) to evaluate the pre-training method.

Original: Benchmark Datasets.
Revised: Benchmark Datasets.

Original: In most MRC datasets, a question and a context are represented as a sequence of words, and the answer span (indices of start and end words) is annotated from the context words based on the question.
Revised: In most MRC datasets, a question and a context are represented as a sequence of words, and the answer span (indices of start and end words) is annotated from the context words based on the question.

Original: Among these datasets, we choose SQuAD as the primary benchmark dataset for question generation, since it is the most popular human-annotated MRC dataset.
Revised: Among these datasets, we choose SQuAD as the primary benchmark dataset for question generation, since it is the most popular human-annotated MRC dataset.

Original: For fair comparison with existing question generation methods, we use the same splits of SQuAD-v1.1, as previously done in Du et al.
Revised: For fair comparison with existing question generation methods, we use the same splits of SQuAD-v1.1, as [Table] [Table] [Table]

Revised: [Table]

Original:  (2017), Kim et al.
Revised: [Table] [Table] previously done in Du et al. (2017), Kim et al.

Original: (2019), and Dong et al.
Revised: (2019), and Dong et al.

Original: We refer to this dataset as Split1.
Revised: We refer to this dataset as Split1.

Original: This split has 77K/10K/10K samples for train/dev/test sets.
Revised: This split has 77K/10K/10K samples for train/dev/test sets.

Original: We also evaluate on the reversed dev-test split, referred to as Split2.
Revised: We also evaluate on the reversed dev-test split, referred to as Split2.

Original: Additionally, we evaluate our question generationmethod on MS MARCO (Nguyen et al., 2016) and NewsQA (Trischler et al., 2017).
Revised: Additionally, we test our question generation on MS MARCO (Nguyen et al., 2016) and NewsQA (Trischler et al., 2017) for evaluating generalization of our method to other datasets.

Revised: In the case of MS MARCO, questions are collected from real user query logs in Bing.

Original: For these datasets, we follow pre-processing of Tuan et al.
Revised: For these datasets, we follow pre-processing of Tuan et al.

Original: (2020), sampling a subset of original data where the answers are sub-spans of their corresponding paragraphs to obtain train/dev/test sets with 51K/6K/7K samples for MS MARCO and 76K/4K/4K samples forNewsQA. To calculate the scores
Revised: (2020), sampling a subset of original data where the answers are sub-spans of their corresponding paragraphs to obtain train/dev/test sets with 51K/6K/7K samples for MS MARCO and 76K/4K/4K samples for

Original: [Table]
Revised: NewsQA.

Original: [Table] [Table] BLEU-4 (Papineni et al., 2002a), METEOR (Banerjee & Lavie, 2005b), and ROUGE-L (Lin, 2004), we use the scripts from Du et al.
Revised: To calculate the scores BLEU-4 (Papineni et al., 2002a), METEOR (Banerjee & Lavie, 2005b), and ROUGE-L (Lin, 2004), we use the scripts from Du et al.

Original: To evaluate the effectiveness of generated synthetic MRC data, we test the ﬁne-tuned MRC model on the downstream MRC dataset after training on the generated synthetic data.
Revised: To evaluate the effectiveness of generated synthetic MRC data, we test the ﬁne-tuned MRC model on the downstream MRC dataset after training on the generated synthetic data.

Original: We calculate the
Revised: We calculate the

Original: EM/F1 score of the MRC model on SQuAD-v1.1 and v2.0 development set.
Revised: EM/F1 score of the MRC model on SQuAD-v1.1 and v2.0 development set.

Original: We also evaluate on the test set of KorQuAD, a Korean dataset created with the same procedure as SQuAD-v1.1.
Revised: We also evaluate on the test set of KorQuAD, a Korean dataset created with the same procedure as SQuAD-v1.1.

Revised: To further demonstrate the effectiveness of our approach, we additionally conduct experiments on question generation with Natural Questions (Kwiatkowski et al., 2019) and on the downstream MRC task with QUASAR-T (Dhingra et al., 2017) and BioASQ (Tsatsaronis et al., 2015) in the appendix.

Original: Implementation Details.
Revised: Implementation Details.

Original: For all experiments and models, we use all ofﬁcial original hyperparameters unless otherwise stated below.
Revised: For all experiments and models, we use all ofﬁcial original hyperparameters unless otherwise stated below.

Original: For BertGen model, we use pre-trained BERT (Base and Large) as encoder and 12 stacked layers of Transformer as decoder.
Revised: For BertGen model, we use pre-trained BERT (Base and Large) as encoder and 12 stacked layers of Transformer as decoder.

Original: For large version of the model, we use 24 layers of the encoder and the decoder with 737M parameters.
Revised: For large version of the model, we use 24 layers of the encoder and the decoder with 737M parameters.

Revised: For dynamic answer prediction, we use the annotated answers in SQuAD for learning the number of answer candidates

Original:  For the generation of unanswerable questions in SQuAD-v2.0, we separate unanswerable and answerable cases and then train separate generation models.
Revised: K and the answer spans. For the generation of unanswerable questions in SQuAD-v2.0, we separate unanswerable and answerable cases and then train separate generation models.

Original: For all BertGen models, we pre-train the question generator for 5 epochs on Wikipedia and ﬁne-tune it for 30 epochs on MRC dataset with batch size of 32.
Revised: For all BertGen models, we pre-train the question generator for 5 epochs on Wikipedia and ﬁne-tune it for 30 epochs on MRC dataset with batch size of 32.

Original: For other question generation models, we pre-train for 1 epoch  on Wikipedia.
Revised: For other question generation models, we pre-train for 1 epoch [Table] [Table] on Wikipedia.

Revised: For UniLM, the input is formulated as sequence-to-sequence, the ﬁrst input segment is the concatenation of context and answer, while the second output segment is a missing answercontaining sentence or a question to be generated, using all ofﬁcial UniLM settings and pre-trained

Revised: UniLM weights.

Original: The training time depends on the data size and the model complexity.
Revised: The training time depends on the data size and the model complexity.

Original: For Zhao et al.
Revised: For Zhao et al.

Original: (2018), pre-training on Full-Wiki takes only 48 hours.
Revised: (2018), pre-training on Full-Wiki takes only 48 hours.

Original: Pre-training BertGen on Small-Wiki in Table 4 takes 48 hours with 8 Tesla V GPU, resulting in 5.1, 4.3 BLEU-4 improvement on Split1, Split2 respectively.
Revised: Pre-training BertGen on Small-Wiki in Table 4 takes 48 hours with 8 Tesla V100 GPU, resulting in 5.1, 4.3 BLEU-4 improvement on Split1, Split2 respectively.

Original: The pre-training for BertGen (Large) with Full-Wiki takes 1,224 hours and ﬁne-tuning takes 72 hours.
Revised: The pre-training for BertGen (Large) with Full-Wiki takes 1,224 hours and ﬁne-tuning takes 72 hours.

Original: For MRC models, we use BERT (Large and WWM).
Revised: For MRC models, we use BERT (Large and WWM).

Original: Mecab (Papineni et al., 2002b) is used for Korean tokenizer.
Revised: Mecab (Kudo, 2006) is used for Korean tokenizer.

Original: Comparison of the Pre-training Method.
Revised: Comparison of the Pre-training Method.

Original: We compare ASGen with a method from Alberti et al.
Revised: We compare ASGen with a method from Alberti et al.

Original: (2019), which is pre-training on next-sentence generation task (NS), and with a method from Golub et al.
Revised: (2019), which is pre-training on next-sentence generation task (NS), and with a method from Golub et al.

Original: (2017), which only trains the generative model on the ﬁnal MRC dataset.
Revised: (2017), which only trains the generative model on the ﬁnal MRC dataset.

Original: We reproduced these methods on BertGen as described in their original work and evaluate question generation scores on the SQuAD splits as well as corresponding sentence generation scores on Test-Wiki.
Revised: We reproduced these methods on BertGen as described in their original work and evaluate question generation scores on the SQuAD splits as well as corresponding sentence generation scores on Test-Wiki.

Original: Comparison of Downstream Results.
Revised: Comparison of Downstream Results.

Original: To check the effectiveness of our method on downstream
Revised: To check the effectiveness of our method on downstream

Original: MRC tasks, we evaluate our generated synthetic data on SQuAD-v1.1, v2.0, and KorQuAD by training MRC models (BERT and BERT+CLKT) on generated data followed by ﬁne-tuning on the[Table] [Table] train set for each dataset.
Revised: MRC tasks, we evaluate our generated synthetic data on SQuAD-v1.1, v2.0, and KorQuAD by training MRC models (BERT and BERT+CLKT) on generated data followed by ﬁne-tuning on the train set for each dataset.

Original: The structure of BERT+CLKT model is the same as that of original
Revised: The structure of BERT+CLKT model is the same as that of original

Original: BERT except that the model is pre-trained for the Korean language.
Revised: BERT except that the model is pre-trained for the Korean language.

Original: Due to the absence of common pre-trained BERT for Korean, we used this model as a baseline.
Revised: Due to the absence of common pre-trained BERT for Korean, we used this model as a baseline.

Original:  E XPERIMENTAL R ESULTS 4. Q UESTION AND A NSWER G ENERATION
Revised:  E XPERIMENTAL R ESULTS 4. Q UESTION AND A NSWER G ENERATION

Original: Comparison to Existing Methods.
Revised: Comparison to Existing Methods.

Original: To evaluate ASGen, we ﬁne-tune the question generation models on both SQuAD splits, after pre-training on answer-containing sentence generation task.
Revised: To evaluate ASGen, we ﬁne-tune the question generation models on both SQuAD splits, after pre-training on answer-containing sentence generation task.

Original: As shown in Table 1, ‘BertGen (Large) + ASGen’ and ‘UniLM + ASGen’ outperforms UniLM on both splits.
Revised: As shown in Table 1, ‘BertGen (Large) + ASGen’ and ‘UniLM + ASGen’ outperforms UniLM on both splits.

Original: As shown in Table 3, ‘BertGen (Large) + ASGen’ outperforms all existing models on all scores on both MS MARCO and NewsQA, except for comparable METEOR scores in NewsQA.
Revised: As shown in Table 3, ‘BertGen (Large) + ASGen’ outperforms all existing models on all scores on both MS MARCO and NewsQA, except for comparable METEOR scores in NewsQA.

Original: Application to Existing Methods.
Revised: Application to Existing Methods.

Original: As shown in Table 2, ASGen consistently improves the performance when applied to other question generation models such as Zhao et al.
Revised: As shown in Table 2, ASGen consistently improves the performance when applied to other question generation models such as Zhao et al.

Original: (2018) and UniLM across all metrics for both splits.
Revised: (2018) and UniLM across all metrics for both splits.

Original: In particular, applying ASGen on UniLM further improves its question generation capability, achieving BLEU-4, METEOR, and ROUGE-L as 23.7, 25.9, 52.2, and 25.3, 26.7, 53.3 on both splits, respectively.
Revised: In particular, applying ASGen on UniLM further improves its question generation capability, achieving BLEU-4, METEOR, and ROUGE-L as 23.7, 25.9, 52.2, and 25.3, 26.7, 53.3 on both splits, respectively.

Original: We reproduce Zhao et al.
Revised: We reproduce Zhao et al.

Original: and use the ofﬁcial code of UniLM with no architecture changes or additional hyper-parameter tuning.
Revised: and use the ofﬁcial code of UniLM with no architecture changes or additional hyper-parameter tuning.

Original: Ablation Study of Pre-training Task.
Revised: Ablation Study of Pre-training Task.

Original: We also compare the BLEU-4 scores between various pretraining tasks to show the effectiveness of ASGen.
Revised: We also compare the BLEU-4 scores between various pretraining tasks to show the effectiveness of ASGen.

Original: As shown in Table 4, ASGen outperforms NS in the recreation score of sentence on Test-Wiki, e.g. 5.2 vs. 1.4 in Small-Wiki and 8.2 vs. 3.4 in
Revised: As shown in Table 4, ASGen outperforms NS in the recreation score of sentence on Test-Wiki, e.g. 5.2 vs. 1.4 in Small-Wiki and 8.2 vs. 3.4 in

Original: Full-Wiki.
Revised: Full-Wiki.

Original: Also, ASGen outperforms NS in question generation, e.g. 22.2 vs. 20.6 and 24.2 vs.
Revised: Also, ASGen outperforms NS in question generation, e.g. 22.2 vs. 20.6 and 24.2 vs.

Original: 22.6 in the two splits, respectively.
Revised: 22.6 in the two splits, respectively.

Original: We also observe that conditioning on a given answer improves
Revised: We also observe that conditioning on a given answer improves

Original: ASGen, e.g. 20.1 vs. 19.9 in Split1 and 21.4 vs. 21.0 in Split2.
Revised: ASGen, e.g. 20.1 vs. 19.9 in Split1 and 21.4 vs. 21.0 in Split2.

Original: Human Evaluation.
Revised: Human Evaluation.

Original: Additionally, we also judge the quality of questions by human evaluation involving 10 evaluators over metrics such as syntax, validation of semantics, question to context relevance and question to answer relevance on 50 randomly chosen samples on SQuAD-v1.1 dev set.
Revised: Additionally, we also judge the quality of questions by human evaluation involving 10 evaluators over metrics such as syntax, validation of semantics, question to context relevance and question to answer relevance on 50 randomly chosen samples on SQuAD-v1.1 dev set.

Original: As shown in Table 5, applying ASGen consistently improves the human evaluation scores.
Revised: As shown in Table 5, applying ASGen consistently improves the human evaluation scores.

Original: Answer Prediction.
Revised: Answer Prediction.

Original: Table 6 shows the effectiveness of our method in generating the number of answers in a given context.
Revised: Table 6 shows the effectiveness of our method in generating the number of answers in a given context.

Original: In the case of ﬁxed K , the MAE from the ground-truth is smallest at K pred = 5 at 1.92 and 0.
Revised: In the case of ﬁxed K , the MAE from the ground-truth is smallest at K pred = 5 at 1.92 and 0.

Original: for test set of Split1 and Split2, respectively.
Revised: for test set of Split1 and Split2, respectively.

Original: Thresholding on the sum of the start and end logits shows an error of 2.31 and 1.12 on the two splits, respectively.
Revised: Thresholding on the sum of the start and end logits shows an error of 2.31 and 1.12 on the two splits, respectively.

Original: In contrast, our method generates an appropriate number of answers, by reducing MAE to 1.24 and 0.76.[Table] [Table] [Table]
Revised: In contrast, our method generates an appropriate number of answers, by reducing MAE to 1.24 and 0.76.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: 4. D OWNSTREAM MRC T ASK P ERFORMANCE
Revised: 4. D OWNSTREAM MRC T ASK P ERFORMANCE

Original: To show the effectiveness of the generated synthetic data, we train MRC models on generated data, before ﬁne-tuning on the downstream data.
Revised: To show the effectiveness of the generated synthetic data, we train MRC models on generated data, before ﬁne-tuning on the downstream data.

Original: As shown in Table [Figure] the synthetic data generated by ‘BertGen (Large) + ASGen’ consistently improves the performance of BERT (Large, WWM) by a signiﬁcant margin.
Revised: As shown in Table 7, the synthetic data generated by ‘BertGen (Large) + ASGen’ consistently improves the performance of BERT (Large, WWM) by a signiﬁcant margin.

Original: Pre-training BERT on synthetic data improves F1 scores by 1.8 on SQuAD-v and 5.6 on SQuAD-v2.0 for BERT (Large), and 0.7 on SQuAD-v1.1 and 2.5 on SQuAD-v2.0 for
Revised: Pre-training BERT on synthetic data improves F1 scores by 1.8 on SQuADv1.1 and 5.6 on SQuAD-v2.0 for BERT (Large), and 0.7 on SQuAD-v1.1 and 2.5 on SQuAD-v2.

Original:  BERT (WWM).
Revised: for BERT (WWM).

Original: Synthetic data also improves BERT+CLKT performance on KorQuAD.
Revised: Synthetic data also improves BERT+CLKT performance on KorQuAD.

Revised: Also, to show improvement due to our pre-training method in the downstream MRC task, we compare between the EM/F1 scores of BERT (Large) models trained on synthetic data generated by different question generation models, ‘BertGen’, ‘BertGen + NS’ and ‘BertGen + ASGen’.

Revised: As shown in

Revised: Table 8, our method outperforms other methods both on SQuAD-v1.1 and SQuAD-v2.0.

Original: 4. E FFECTS OF D OWNSTREAM AND S YNTHETIC D ATA S IZE
Revised: 4. E FFECTS OF D OWNSTREAM AND S YNTHETIC D ATA S IZE

Original: Fig.
Revised: Fig.

Original: 2 shows the effects of varying amounts of downstream MRC data and synthetic data on F scores of BERT (Large).
Revised: 2 shows the effects of varying amounts of downstream MRC data and synthetic data on F scores of BERT (Large).

Original: In Fig. 2-(a), where we ﬁx the size of synthetic data as 43M, pre-training with ‘BertGen + ASGen’ consistently outperforms ‘BertGen + NS’ for all sizes of downstream data.
Revised: In Fig. 2-(a), where we ﬁx the size of synthetic data as 43M, pre-training with ‘BertGen + ASGen’ consistently outperforms ‘BertGen + NS’ for all sizes of downstream data.

Original: While the performance difference is particularly apparent for smaller sizes of downstream data, it still persists even on using the entire MRC data (SQuAD-v1.1).
Revised: While the performance difference is particularly apparent for smaller sizes of downstream data, it still persists even on using the entire MRC data (SQuAD-v1.1).

Original: In Fig. 2-(b), we also conduct experiments by training BERT (Large) using different amounts of generated synthetic data, while using the full size of downstream MRC data.
Revised: In Fig. 2-(b), we also conduct experiments by training BERT (Large) using different amounts of generated synthetic data, while using the full size of downstream MRC data.

Original: The total number of pre-training steps for all data sizes is kept the same as that of 10M synthetic data.
Revised: The total number of pre-training steps for all data sizes is kept the same as that of 10M synthetic data.

Original: Increasing the amount of synthetic data used consistently improves the accuracy of the MRC model.
Revised: Increasing the amount of synthetic data used consistently improves the accuracy of the MRC model.

Original: 4. Q UALITATIVE C OMPARISON OF Q UESTIONS G ENERATION
Revised: 4. Q UALITATIVE A NALYSIS OF Q UESTIONS G ENERATION

Original:  We qualitatively compare the generated questions after pre-training BertGen with NS and ASGen to demonstrate the effectiveness of our method.
Revised: Comparison of Sample Questions. We qualitatively compare the generated questions after pretraining BertGen with NS and ASGen to demonstrate the effectiveness of our method.

Original: For the correct answer “49.6%” as shown in the ﬁrst sample in Table 8, the word “Fresno”, which is critical to make the question speciﬁc, is omitted by NS, while ASGen’s question does not suffer from this issue.
Revised: For the correct answer “49.6%” as shown in the ﬁrst sample in Table 9, the word “Fresno”, which is critical to make the question speciﬁc, is omitted by NS, while ASGen’s question does not suffer from this issue.

Original: Note that the word “Fresno” occurs in the answer-containing sentence.
Revised: Note that the word “Fresno” occurs in the answer-containing sentence.

Original: This issue also occurs in the second sample, where NS uses the word “available” rather than relevant words from the answer-containing sentence, but ASGen uses many of these words such as “most” and “popular” to generate contextually rich questions.
Revised: This issue also occurs in the second sample, where NS uses the word “available” rather than relevant words from the answercontaining sentence, but ASGen uses many of these words such as “most” and “popular” to generate contextually rich questions.

Original: Also, the question from NS is about “two” libraries, while the answer is about “three” libraries, showing the lack of sufﬁcient conditioning on the answer.
Revised: Also, the question from NS is about “two” libraries, while the answer is about “three” libraries, showing the lack of sufﬁcient conditioning on the answer.

Original: Similarly, the third example also shows that ASGen generates more contextual questions than NS by including the exact subject “TARDIS” based on the corresponding answer. 
Revised: Similarly, the third example also shows that ASGen generates more contextual questions than NS by including the exact subject “TARDIS” based on the corresponding answer. Based on these observations and from [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Original: Based on these observations and from the score improvements in Table 4, we conjecture that ASGen leads the question generation model to better condition on the answer and to generate more contextualized questions than NS.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: the score improvements in Table 4, we conjecture that ASGen leads the question generation model to better condition on the answer and to generate more contextualized questions than NS.

Revised: Categorization of Reasoning Type.

Revised: We manually categorized the reasoning type of 150 randomly sampled generated questions on Wikipedia for both answerable and unanswerable questions.

Revised: The results Table 10 and Table 11 show that generated questions using ASGen often require multi-hop or other non-trivial reasoning.

Revised: We follow the same categorization as done by SQuAD-v1.1 (Rajpurkar et al., 2016) and SQuAD-v2.0 (Rajpurkar et al., 2018).

Revised: Note that each example can be assigned to multiple reasoning types for the answerable questions.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Question Generation.
Revised: Question Generation.

Original: Research on question generation has a long history, such as Kalady et al.
Revised: Research on question generation has a long history, such as Kalady et al.

Original: (2010) and Skalban et al.
Revised: (2010) and Skalban et al.

Original: Researchers have actively studied question generation for various purposes, including for data augmentation in question answering.
Revised: Researchers have actively studied question generation for various purposes, including for data augmentation in question answering.

Original: Du et al.
Revised: Du et al.

Original: (2017) proposed an attention-based model for question generation by encoding sentence-level as well as paragraphlevel information.
Revised: (2017) proposed an attention-based model for question generation by encoding sentence-level as well as paragraphlevel information.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2018) utilized a gated self-attention encoder with a max-out unit to handle long paragraphs.
Revised: (2018) utilized a gated self-attention encoder with a max-out unit to handle long paragraphs.

Original: Song et al.
Revised: Song et al.

Original: (2018) introduced a query-based generative model to jointly solve question generation and answering tasks.
Revised: (2018) introduced a query-based generative model to jointly solve question generation and answering tasks.

Original: Kim et al.
Revised: Kim et al.

Original: (2019) separately encoded the answer and the rest of the paragraph for question generation.
Revised: (2019) separately encoded the answer and the rest of the paragraph for question generation.

Original: Ma et al.
Revised: Ma et al.

Original: (2020) suggested sentence-level semantic matching and answer-position-aware question generation.
Revised: (2020) suggested sentence-level semantic matching and answer-position-aware question generation.

Original: Tuan et al.
Revised: Tuan et al.

Original: (2020) show that incorporating interactions across multiple sentences enhances question generation performance.
Revised: (2020) show that incorporating interactions across multiple sentences enhances question generation performance.

Original: Our approach can further improve the question generation quality of these methods by pre-training them with the answer-containing sentence generation task.
Revised: Our approach can further improve the question generation quality of these methods by pre-training them with the answer-containing sentence generation task.

Original: Transfer Learning.
Revised: Transfer Learning.

Original: Pre-training methods are popular in natural language processing for learning contextualized word representations.
Revised: Pre-training methods are popular in natural language processing for learning contextualized word representations.

Original: Open-GPT (Radford et al., 2018), BERT (Devlin et al., 2019),
Revised: Open-GPT (Radford et al., 2018), BERT (Devlin et al., 2019),

Original: XLNet (Yang et al., 2019), PEGASUS (Zhang et al., 2019), ERNIE-GEN (Xiao et al., 2020),
Revised: XLNet (Yang et al., 2019), PEGASUS (Zhang et al., 2019), ERNIE-GEN (Xiao et al., 2020),

Original: UniLM (Dong et al., 2019) and UniLMv2 (Bao et al.,  2020) utilize Transformer (Vaswani et al., 2017) to learn different types of language models on a large dataset followed by ﬁne-tuning on a downstream task.
Revised: UniLM (Dong et al., 2019), UniLMv2 (Bao et al., 2020), T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) utilize Transformer (Vaswani et al., 2017) to learn different types of language models on a [Table] [Table] [Table]

Revised: large dataset followed by ﬁne-tuning on a downstream task.

Original: While our approach is similar to these, our pre-training task for the question generator  focuses on generating missing answer-containing sentence in the context to learn better representations more suitable to the question generation task.
Revised: These pre-training approaches tend to be very generic, while our approach is a more appropriate pre-training method focused on the speciﬁc task of question generation. Lee et al. (2019b) suggested a pre-training method for information retrieval called Inverse Cloze Task which treats a sentence as a pseudo-query and its surrounding context as a pseudo-target. Unlike this method, our pre-training task for the question generator is strongly conditioned on the answer and focuses on generating missing answer-containing sentence in the context to learn better representations more suitable to the question generation task.

Original: Synthetic Data Generation.
Revised: Synthetic Data Generation.

Original: Subramanian et al.
Revised: Subramanian et al.

Original: (2018) show that neural models generate better candidate answers from a given paragraph than using off-the-shelf tools for selecting named entities and noun phrases. 
Revised: (2018) show that neural models generate better candidate answers from a given paragraph than using off-the-shelf tools for selecting named entities and noun phrases. Yang et al.

Revised: (2017) introduced a training method for the MRC model by combining synthetic data and human-annotated data.

Original: Similar to our method, Golub et al.
Revised: Similar to our method, Golub et al.

Original: (2017) proposed to generate questions conditioned on generated answers by separating the answer generation and the question generation.
Revised: (2017) proposed to generate questions conditioned on generated answers by separating the answer generation and the question generation.

Original: Unlike this paper, they do not estimate the number of answers, and they do not pre-train their question generator.
Revised: Unlike this paper, they do not estimate the number of answers, and they do not pre-train their question generator.

Original: Dong et al.
Revised: Dong et al.

Original: (2019) also show that utilizing synthetic data boosts the performance of MRC models.
Revised: (2019) also show that utilizing synthetic data boosts the performance of MRC models.

Original: Inspired by these previous studies, we propose a newly designed pre-training technique that improves capability of question generation models.
Revised: Inspired by these previous studies, we propose a newly designed pre-training technique that improves capability of question generation models.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose a novel pre-training method called ASGen to learn generating contextually rich questions better conditioned on the answers.
Revised: We propose a novel pre-training method called ASGen to learn generating contextually rich questions better conditioned on the answers.

Original: Our approach improves question generation ability of existing methods, achieves new state-of-the-art results on MS MARCO and NewsQA, and the synthetic data increases downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modiﬁcation to the existing MRC models.
Revised: Our approach improves question generation ability of existing methods, achieves new state-of-the-art results on MS MARCO and NewsQA, and the synthetic data increases downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modiﬁcation to the existing MRC models.

Original: A PPENDIX A Q UESTION G ENERATION ON MORE MRC D ATASETS
Revised: A PPENDIX A Q UESTION G ENERATION ON MORE MRC D ATASETS

Original: We also evaluate the question generation model on another data split (Split3) from Zhao et al.
Revised: We also evaluate the question generation model on another data split (Split3) from Zhao et al.

Original: Split3 is obtained by dividing the original development set in SQuAD-v1.1 into two equal halves randomly and choosing one of them as the development set and the other as test set while retaining the train set in SQuAD-v1.1.
Revised: Split3 is obtained by dividing the original development set in SQuAD-v1.1 into two equal halves randomly and choosing one of them as the development set and the other as test set while retaining the train set in SQuAD-v1.1.

Original: As shown in Table 9, applying ASGen to the reproduced question generation model from Zhao et al.
Revised: As shown in Table 12, applying ASGen to the reproduced question generation model from Zhao et al.

Original: (2018) improves BLEU-4, METEOR and ROUGE-L score on
Revised: (2018) improves BLEU-4, METEOR, and ROUGE-L score on

Original: Split3 by 1.3, 0.9, and 1.3, respectively.
Revised: Split3 by 1.3, 0. and 1.3, respectively.

Original: As shown in Table applying ASGen to BertGen shows improvement in the question generation score in Natural Questions (Kwiatkowski et al., 2019)short answer dataset, where questions are  from  user query logs used in Google, similar to MS Marco. Small-Wiki is used to pre-train the models.
Revised: We also test generalization capability of our method by evaluating on Natural Questions (Kwiatkowski et al., 2019) dataset, where questions are collected from real user query logs on Google and may have less biased questions than other datasets.

Original: Models with ‘*’ indicate those results
Revised: As shown in Table 13, applying

Original: we
Revised: ASGen to BertGen shows improvement in the question generation score on Natural Questions short answer dataset by 3.8 BLEU-4, 2.5 METEOR and 1.1 ROUGE-L. is used to

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B T RAINING E LECTRA MRC M ODEL WITH G ENERATED S YNTHETIC D ATA
Revised: B T RAINING E LECTRA MRC M ODEL WITH G ENERATED S YNTHETIC D ATA

Original: We also apply our synthetic data generated from Small-Wiki to another MRC model, ELECTRA (Clark et al., 2020), which shows the state-of-the-art results.
Revised: We also apply our synthetic data generated from Small-Wiki to another MRC model, ELECTRA (Clark et al., 2020), which shows the state-of-the-art results.

Original: As shown in Table 11, pre
Revised: In Table 14, we report the mean

Original: training
Revised: EM/F1 score on SQuAD 2.0 development set of four runs by using ofﬁcial Electra source code 3 and the pre-trained checkpoint.

Original:  ELECTRA on the generated synthetic data using ASGen improves 0.5 EM and 0.6 F1 score on the -v2.0, even when using only Small-Wiki
Revised: Pre-training ELECTRA on the generated synthetic data using ASGen improves 0.5 EM and 0.6 F1 score on the downstream MRC dataset, SQuAD-v2.0, even when using only Small-Wiki.

Original: [Table]
Revised: [Table]

Original: C T RANSFER L EARNING TO O THER  D ATASETS
Revised: C T RANSFER L EARNING TO O THER MRC D ATASET (QUASAR-T)

Original: To show that our generated data is useful for other MRC datasets, we ﬁne-tune and test the MRC model on QUASAR-T (Dhingra et al., 2017), which is another large-scale MRC dataset, after training on the synthetic data generated from SQuAD-v1.1.
Revised: To show that our generated data is useful for other MRC datasets [Table] we ﬁne-tune and test the MRC model on QUASAR-T (Dhingra et al., 2017), which is another large-scale MRC dataset, after training on the synthetic data generated from SQuAD-v1.1.

Original: In this experiment, we ﬁrst ﬁne-tune ‘BertGen + ASGen’ using SQuAD-v1.1, and using synthetic data generated by this model, we train the
Revised: In this experiment, we ﬁrst ﬁne-tune ‘BertGen + ASGen’ using SQuAD-v1. [Table] and using synthetic data generated by this model, we train the

Original: BERT (Large) MRC model.
Revised: BERT (Large) MRC model.

Original: Afterwards, we ﬁne-tune BERT (Large) for the downstream MRC task using QUASAR-T data.
Revised: Afterwards, we ﬁne-tune BERT (Large) for the downstream MRC task using QUASAR-T data.

Original: QUASAR-T has two separate datasets, one with short snippets as context, and the other with long paragraphs as context.
Revised: QUASAR-T has two separate datasets, one with short snippets as context, and the other with long paragraphs as context.

Original: As shown in Table 12, training with our synthetic data improves the F1 score on the test set by 2.2 and 1.7 for the two cases, respectively.
Revised: As shown in Table 15, training with our synthetic data improves the F1 score on the test set by 2.2 and 1.7 for the two cases, respectively.

Original: [Table]
Revised: [Table]

Original: D D OWNSTREAM MRC A BLATION OF A NSWER G ENERATION A PPROACHES 
Revised: D C OMPARISON OF A NSWER G ENERATION A PPROACHES ON MRC TASK

Original: We also evaluate the effectiveness of dynamic- K answer prediction by pre-training the BERT (Large) (Devlin et al., 2019)
Revised: We also evaluate the effectiveness of dynamic- K answer prediction by pre-training the BERT (Large) (Devlin et al., 2019)

Original: MRC model on our synthetic data from Small-Wiki followed by ﬁnetuning on the downstream MRC dataset, SQuAD-v2.0.
Revised: MRC model on our synthetic data from Small-Wiki followed by ﬁnetuning on the downstream MRC dataset, SQuAD-v2.0.

Original: As shown in Table 13, dynamic-
Revised: As shown in Table 16, dynamic-

Original: K answer prediction shows 0.3 EM and 0.2 F1 score improvements from the baseline approach, ﬁxed- K 
Revised: K answer prediction shows 0.3 EM and 0.2 F1 score improvements from the baseline approach, ﬁxed- K .

Original: [Table]

Original: [Table]
Revised: [Table]

Original: E D ETAILS OF W IKIPEDIA P REPROCESSING
Revised: E D ETAILS OF W IKIPEDIA P REPROCESSING

Original: To build the answer-containing sentence generation data and the synthetic MRC data for
Revised: To build the answer-containing sentence generation data and the synthetic MRC data for

Original: SQuAD (Rajpurkar et al., 2016)
Revised: SQuAD (Rajpurkar et al., 2016)

Original: , we collect all paragraphs from all articles of the entire English
Revised: , we collect all paragraphs from all articles of the entire English

Original: Wikipedia dump and generate questions and answers on these paragraphs.
Revised: Wikipedia dump and generate questions and answers on these paragraphs.

Original: We apply extensive ﬁltering and clean-up to only retain the highest-quality paragraphs from Wikipedia, as follows.
Revised: We apply extensive ﬁltering and clean-up to only retain the highest-quality paragraphs from Wikipedia, as follows.

Original: To ﬁlter out low-quality articles, we remove those with less than 200 cumulative page-views including all re-directions in a two-month period.
Revised: To ﬁlter out low-quality articles, we remove those with less than 200 cumulative page-views including all re-directions in a two-month period.

Original: In order to calculate the number of page-views, ofﬁcial
Revised: In order to calculate the number of page-views, ofﬁcial

Original: Wikipedia page-view dumps were used.
Revised: Wikipedia page-view dumps were used.

Original: Of the 5.4M original Wikipedia articles, ﬁltering by pageviews leaves 2.8M articles.
Revised: Of the 5.4M original Wikipedia articles, ﬁltering by pageviews leaves 2.8M articles.

Original: We also remove those articles with less than 500 characters, as they are often low-quality stub articles, which further removes additional 16% of the articles.
Revised: We also remove those articles with less than 500 characters, as they are often low-quality stub articles, which further removes additional 16% of the articles.

Original: We remove all “meta” namespace pages such as talk, disambiguation, user pages, portals, etc.
Revised: We remove all “meta” namespace pages such as talk, disambiguation, user pages, portals, etc.

Original: as they often contain irrelevant text or casual conversations between editors.
Revised: as they often contain irrelevant text or casual conversations between editors.

Original: In order to extract clean text from the wiki-markup format of the Wikipedia articles, we remove extraneous entities from the markup including table of contents, headers, footers, links/URLs, image captions, IPA double parentheticals, category tables, math equations, unit conversions, HTML escape codes, section headings, double brace templates such as info-boxes, image galleries, HTML tags, HTML comments, and allother tables.
Revised: In order to extract clean text from the wiki-markup format of the Wikipedia articles, we remove extraneous entities from the markup including table of contents, headers, footers, links/URLs, image captions, IPA double parentheticals, category tables, math equations, unit conversions, HTML escape codes, section headings, double brace templates such as info-boxes, image galleries, HTML tags, HTML comments, and all tables.

Original: We then split the cleaned text into paragraphs and remove all paragraphs with less than 150 characters or more than 3,500 characters.
Revised: We then split the cleaned text into paragraphs and remove all paragraphs with less than 150 characters or more than 3,500 characters.

Original: Paragraphs with the number of characters between 150 towere sub-sampled such that these paragraphs make up 16.5% of the ﬁnal dataset, as originally done for the SQuAD dataset
Revised: Paragraphs with the number of characters between 150 towere sub-sampled such that these paragraphs make up 16.5% of the ﬁnal dataset, as originally done for the SQuAD dataset

Original: Since the majority of the paragraphs in Wikipedia are rather short, out of the 60M paragraphs from the ﬁnal 2.4M articles, our ﬁnal Wikipedia dataset contains 8.3M paragraphs.
Revised: Since the majority of the paragraphs in Wikipedia are rather short, out of the 60M paragraphs from the ﬁnal 2.4M articles, our ﬁnal Wikipedia dataset contains 8.3M paragraphs.

Original: Finally, we generate 43M answer-paragraph pairs from the ﬁnal Wikipedia dataset with the answer generator of BertGen in this paper.
Revised: Finally, we generate 43M answer-paragraph pairs from the ﬁnal Wikipedia dataset with the answer generator of BertGen in this paper.

Original: FC ENTRAL T ENDENCY AND V ARIATION FOR H UMAN E VALUATION
Revised: F T RANSFER L EARNING TO O THER L IMITED D OMAIN D ATA (B IO ASQ)

Revised: We conduct experiments on BioASQ (Tsatsaronis et al., 2015) dataset to show the effectiveness of our model in limited-data domains having less annotated data. As shown in Table 17, ASGen improves the question generation scores by 6.0 BLEU-4, 7.8 METEOR and 6.9 ROUGE-L on BioASQ factoid-type 6b. Moreover, using ‘Full-Wiki’ data enhances the performance of BERT(Large) by a large margin and outperforms BioBERT (Lee et al., 2019a), by 0.95 Macro F1 (Yes/No) and 1.

Revised: F1 (List).

Revised: Note that BioBERT is speciﬁcally pre-trained on a medical corpus (PubMed) whereas we use a generic corpus Wikipedia, ‘Full-Wiki’, with our generation models ﬁne-tuned on SQuAD.

Revised: [Table]

Original: Human evaluation involves 10 evaluators over metrics such as syntax (ST), validation of semantics (SM), question to context relevance (CR) and question to answer relevance (AR) on 50 randomly chosen samples on SQuAD-v1.1 development set.
Revised: Human evaluation involves 10 evaluators over metrics such as syntax (ST), validation of semantics (SM), question to context relevance (CR) and question to answer relevance (AR) on 50 randomly chosen samples on SQuAD-v1.1 development set.

Original: Each score is in the range 1 to 5.
Revised: Each score is in the range 1 to 5.

Original: Central tendency and variation can be found in Table 14. [Table]
Revised: Central tendency and variation can be found in Table 18.

Original: G C ENTRAL T ENDENCY AND V ARIATION FOR  D OWNSTREAM SQ U AD- V 1. AND V 2.
Revised: H C ENTRAL T ENDENCY AND V ARIATION FOR THE D OWNSTREAM T ASKS

Original: For the EM and F1 scores on downstream SQuAD-v1.1 and v2.0 development set in Table 7 of our main paper, we selected 5 model checkpoints from the same pre-training on the synthetic data in different numbers of training steps.
Revised: For the EM and F1 scores on downstream SQuAD-v1.1 and v2.0 development set in Table 7 of our main paper, we selected 5 model checkpoints from the same pre-training on the synthetic data in different numbers of training steps.

Original: We then ﬁne-tuned each of these models on the ﬁnal downstream data three times each, chose the best performing model on the development set
Revised: We then ﬁne-tuned each of these models on the ﬁnal downstream data three times each, chose the best performing model on the development set, and reported its score.

Revised: Central tendency and variation can be found in Table 19.

Original: [Table]
Revised: [Table]

Original: and reported its score.
Revised: [Table]

Original: Central tendency and variation can be found in Table
Revised: The mechanism of generating questions may differ in generating answerable and unanswerable questions. For example, the model could exploit a mismatching phrase to make a question plausible but unanswerable.

Original: [Table]
Revised: In order to reﬂect these characteristics, we train answerable and unanswerable models separately. We ﬁrst take the BertGen model pre-trained on the ASGen task and then ﬁne-tune this model on the no-answer question generation on SQuAD-v2.0. We infer with this model on the entire Wikipedia to make negative examples for un-answerble synthetic data for pre-training MRC models on SQuAD-v2.0. J D ISCUSSION ON W EAK S UPERVISION FOR D YNAMIC - K P REDICTION

Revised: In question generation, it is important to ﬁnd which elements of a given context are suitable answer.

Revised: To do this, we predict the number of answers to obtain a more appropriate set of “answer-like” phrases that humans tend to choose when they are preparing a question, rather than all possible entity phrases.

Revised: This tendency can also be found in the SQuAD dataset, which has a varying number of annotated answers per context, even though the annotators were recommended to create up to ﬁve answers, as shown in Table 20.

Revised: While we do not have the ground-truth number of answers for all contexts, this characteristic of SQuAD annotation can still be a useful weak supervision for learning the number of answer candidates.

Original: [Table]
Revised: [Table]

Original: H BLEU-4, METEOR, AND ROGUE-L
Revised: K BLEU-4, METEOR, AND ROGUE-L

Original: BLEU (Papineni et al., 2002c), METEOR (Banerjee & Lavie, 2005a) and ROUGE (Lin, 2004) are widely-used metrics for evaluating the quality of generated text, where the quality indicates the degree of correspondence between generated text and reference texts.
Revised: BLEU (Papineni et al., 2002b), METEOR (Banerjee & Lavie, 2005a) and ROUGE (Lin, 2004) are widely-used metrics for evaluating the quality of generated text, where the quality indicates the degree of correspondence between generated text and reference texts.

Original: BLEU uses modiﬁed precision to compare a generated text against the reference texts.
Revised: BLEU uses modiﬁed precision to compare a generated text against the reference texts.

Original: BLEU-4 calculates a weighted score of unigram, bigram, trigram, and 4-gram based matching.
Revised: BLEU-4 calculates a weighted score of unigram, bigram, trigram, and 4-gram based matching.

Original: METEOR uses harmonic mean between precision and recall of unigrams, but with for recall given more importance than precision.
Revised: METEOR uses harmonic mean between precision and recall of unigrams, but with for recall given more importance than precision.

Original: Unlike
Revised: Unlike

Original: BLEU, METEOR also tries to match synonyms and performs stemming instead of just relying on exact word matching.
Revised: BLEU, METEOR also tries to match synonyms and performs stemming instead of just relying on exact word matching.

Original: ROUGE-L is the longest common sub-sequence based word matching.
Revised: ROUGE-L is the longest common sub-sequence based word matching.

Original: The longest co-occurrence in sequences of n-grams between generated text and reference texts are considered for calculating the score.
Revised: The longest co-occurrence in sequences of n-grams between generated text and reference texts are considered for calculating the score.

Original: To calculate these evaluation scores, we follow the script from
Revised: To calculate these evaluation scores, we follow the script from

Original: Du et al.
Revised: Du et al.

Original: (2017), except for the corresponding scripts from other question generation models
Revised: (2017), except for the corresponding scripts from other question generation models

Original: when
Revised: when

Original: ASGen is applied to them.
Revised: ASGen is applied to them.

Original: I L INKS TO D OWNLOADABLE C OMPONENTS
Revised: L L INKS TO D OWNLOADABLE C OMPONENTS

Original: For Wikipedia data, we downloaded English Wikipedia dump in Feb 2019 from ( https: //dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.
Revised: For Wikipedia data, we downloaded English Wikipedia dump in Feb 2019 from ( https: //dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.

Original: xml.bz2 ).
Revised: xml.bz2 ).

Original: Page views were obtained from ( https://dumps.wikimedia.org/ other/pageviews/2019/2019-01/ ) and ( https://dumps.wikimedia.org/ other/pageviews/2019/2019-02/ ).
Revised: Page views were obtained from ( https://dumps.wikimedia.org/ other/pageviews/2019/2019-01/ ) and ( https://dumps.wikimedia.org/ other/pageviews/2019/2019-02/ ).

Original: For applying our method to other existing question generation models, we reproduce Zhao et al.
Revised: For applying our method to other existing question generation models, we reproduce Zhao et al.

Original: (2018) using publicly available code ( https://github.com/seanie12/neural-question-generation ) and use the ofﬁcial code of Dong et al.
Revised: (2018) using publicly available code ( https://github.com/seanie12/neural-question-generation ) and use the ofﬁcial code of Dong et al.

Original: (2019) ( https://github.com/microsoft/unilm ).
Revised: (2019) ( https://github.com/microsoft/unilm ).