Post-Training with Interrogative Sentences for Enhancing BART -based K orean Question Generator (2022)

Abstract: The pre-trained language models such as KoBART often fail in generating perfect interrogative sentences when they are applied to Korean question generation. This is mainly due to the fact that the language models are much experienced with declarative sentences, but not with interrogative sentences. Therefore, this paper proposes a novel post-training of KoBART to enhance it for Korean question generation. The enhancement of KoBART is accomplished in three ways: (i) introduction of question infilling objective to KoBART to enforce it to focus more on the structure of interrogative sentences, (ii) augmentation of training data for question generation with another data set to cope with the lack of training instances for post-training, (iii) introduction of Korean spacing objective to make KoBART understand the linguistic features of Korean. Since there is no standard data set for Korean question generation, this paper also proposes KorQuAD-QG, a new data set for this task, to verify the performance of the proposed post-training. Our code are publicly available at https://github.com/gminipark/post_training_qg

## introduction question generation is a task that aims to generate a question automatically from a given context text. since it is a kind of text generation task, it has wide applications. for instance, it has been used for constructing robust question answering systems @xcite , augmenting data for machine reading comprehension (mrc) @xcite , and making goal-oriented dialogue systems @xcite . the main approach of question generation is to adopt a pre-trained language model trained with a large-scale corpus and then fine-tune the model with a data set for question generation @xcite @xcite ). in answer-aware question generation, it is important to figure out which part of a content is most relevant and understand the structure of interrogative sentences. however, most current pre-trained language models are not much experienced with the domain of question generation and interrogative sentences. as a result, even the fine-tuned model does not reflect the characteristics of question generation fully. one solution to this problem is to enforce a language model to contain proper knowledge for question generation. @xcite proposed a language model trained with a knowledge graph and plain texts to make the language model knowledgeenhanced. however, this approach requires a lot of resources to train such a language model since the language model usually has more parameters than ordinary language models. on the other hand, @xcite added an adapter to a pre-trained language model, and only the adapter is trained to capture some knowledge for question generation. however, this approach requires external knowledge for question generation which is difficult to obtain. another solution is to adopt the idea of posttraining @xcite which adapts a language model to a new task by making the language model learn the objective of the new task or augmenting its training data with those of the task. for instance, @xcite and @xcite showed that bert could be improved in dialogue response selection by learning, as posttraining, dialogue data which bert did not experience in the pre-training step. many previous studies proved that post-training enhances a pre-trained language model in several classification and text generation tasks @xcite @xcite , but there is no study that a pre-trained language model improves question generation through post-training with well-designed objectives. this paper proposes a novel post-training of ko-bart, a korean bart, for korean question generation. the proposed post-training tackles four issues about post-training a bart-based korean question generator. first, a new data set, korquad-qg, is developed following the work of lim et al. (2019), since there is no public data set for korean question generation. note that kobart reveals a weakness in generating interrogative sentences since it never experienced the question generation task in its pre-training step. thus, the proposed post-training adapts kobart to question generation by enforcing it to focus more on questions with a new objective question infilling. the performance of pre-trained language models is affected by the number of training instances. thus, kobart is post-trained with external mrc data as well as korquad-qg. the last issue is related with korean language. kobart is missing some linguistic characteristics of korean interrogative sentences. therefore, the proposed posttraining injects the characteristics explicitly to ko-bart by introducing a new objective korean spacing. ## related work recent previous studies have shown that large-scale pre-trained language models show prominent performance in many nlp tasks including question generation @xcite @xcite . for instance, @xcite proposed a unified language model for solving various nlp tasks. for this, they contrived three language modeling objectives of unidirectional objective, bidirectional objective, and seq-to-seq objective, and then applied all the objectives to language modeling. on the other hand, erine-gen achieved the sota performance by applying an infilling generation mechanism and a noise-aware generation method to the multi-flow attention archi-tecture @xcite . however, these language models share a problem that plenty of resources are needed to train them. in addition, they suffer from a lack of domain knowledge of question generation task since they did not experience the sentences for question generation in their pre-training. one solution to these problems is to post-train a language model before fine-tuning. post-training of a language model has shown a great performance in many nlp tasks @xcite @xcite . @xcite proposed a post-training for response selection which optimizes bert with the next sentence prediction (nsp) and masked language model (mlm) using the corpus of response selection and then finetines it with the objective of response selection. on the other hand, @xcite replaced nsp of bert with utterance relevance classification (urc) that is more relevant to response selection. they reported that the use of urc instead of nsp led to performance improvement. ## korean question generation question generation is a task of generating a question q from a context c and an answer span a within the context. thus, a question generator produces an interrogative sentence that maximizes this paper adopts kobart foot_0 , a korean bart, for p (•). bart is a denoising autoencoder which reconstructs an original text from a corrupted text. it is optimized by minimizing the negative log likelihood the fine-tuned kobart shows a reasonable performance for question generation, but yet has three problems. one is that kobart is not pre-trained with the sentences for question generation, another is that the learning objectives of kobart is not directly related with question generation, and the other is that it often fails in grasping the structure of korean interrogative sentences. therefore, this paper solves these problems by post-training ko-bart between the pre-training step and the finetuning step as shown in figure 1 . after kobart is pre-trained with equation (1), it is post-trained with d qg augmented by another data set d aug using new objectives, question infilling and korean spacing, for question generation. then, the post-trained kobart is fine-tuned again with equation (2). the new objectives for posttraining will be explained in the following section. ## post-training question generator the proposed post-training for question generation enhances the pre-trained kobart in three ways. first, kobart is allowed to experience the domain of question generation through post-training. note that kobart is not pre-trained with the sentences from question generation. thus, kobart is updated with d qg , a data set for question generation. in order to make kobart learn the domain of question generation effectively, a new objective, question infilling (qi), is proposed. question infilling is equivalent to text infilling except that the mask token can replace a word only at the question q, not in the context c. as a result, kobart focuses more on a question than a context. this is achieved by a loss even if kobart is trained with korean sentences, it often generates a grammatically wrong question. this is because kobart does not capture the structure of questions perfectly. to solve this problem, kobart is forced to learn how to space a word-concatenated sequence, since word spacing of questions helps kobart understand the questions syntactically and semantically. in addition, word spacing is helpful for kobart to find out which part of a context is related to a given question. this is achieved by introducing a new objective of korean spacing formulated as ## conclusions this paper has proposed a novel post-training of the pre-trained kobart for korean question generation. the proposed post-training enhances the pretrained kobart in three ways. first, by question infilling, the post-trained kobart could not only be adapted to question generation, but also focus on the context area which is related to a question. second, by learning korean spacing, the post-trained kobart understands the korean interrogative sentences semantically and semantically better than the pre-trained kobart. lastly, since transformerbased language models are sensitive to the number of training instances, the data set for question generation is augmented with additional mrc data. this data augmentation is empirically proven to be most effective in enhancing kobart for question generation. in addition, since there is no standard data set for korean question generation, this paper proposed a new data set of korquad-qg for the task.