Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation (2022)

Abstract: Today the pre-trained language models achieve great success for question generation (QG) task and significantly outperform traditional sequence-to-sequence approaches. However, the pre-trained models treat the input passage as a flat sequence and are thus not aware of the text structure of input passage. For QG task, we model text structure as answer position and syntactic dependency, and propose answer localness modeling and syntactic mask attention to address these limitations. Specially, we present localness modeling with a Gaussian bias to enable the model to focus on answer-surrounded context, and propose a mask attention mechanism to make the syntactic structure of input passage accessible in question generation process. Experiments on SQuAD dataset show that our proposed two modules improve performance over the strong pre-trained model ProphetNet, and combing them together achieves very competitive results with the state-of-the-art pre-trained model.

## introduction question generation (qg) aims to generate questions for a given passage, which is a challenging task and shows great value in many practical applications. for instance, qg helps in building reading comprehension tests for course assessments @xcite ; qg can be an important skill for chatbots to start a conversation with human users @xcite ; qg can help reduce the human labor for collecting large-scale questionanswer datasets to improve question answering systems @xcite . existing qg methods mostly employ neural network models by treating it as a sequence-tosequence generative task, where researchers have tried to incorporate text structure to improve the performance. for answer-aware qg, the answer position is widely explored. @xcite propose three strategies to match the answer with passage; @xcite model the relative distance between context words and answer to help generate answer-related questions; @xcite enhance model's ability in capturing semantic and positional information via multi-task learning. some other neural network models dig out deep text structure for generating better questions. for instance, @xcite construct a passage graph to utilize text structure; @xcite construct a semanticlevel graph to learn the global structure of a document for deep qg. however, most of these methods adopt standard neural network, and few of them are based on the strong pre-trained models. nowadays, pre-trained models largely boost the performance of qg systems @xcite @xcite , which utilize huge amounts of unlabeled text corpora in pre-training stage to learn general language representations, and then be fine-tuned in supervised qg tasks. however, in most cases, the input passage and answer are formatted as "answer [sep] passage" to feed into the pre-trained model during the fine-tuning stage, as a result that the neural network cannot explicitly capture the syntactic structure of sentence and is not aware of the answer position @xcite , which is vital to generate high-quality answer-aware questions. in this end, how to incorporate text structure into pre-trained models for qg remains an open issue. fortunately, several previous works have tried to utilize external knowledge and position information for nlp tasks. for instance, @xcite introduce token localness in attention mechanism for machine translation tasks, and @xcite propose a knowledge-enabled language model (k-bert) to incorporate domain knowledge for question answering systems. motivated by these works, we propose a novel strategy to explicitly incorporate syntactic knowledge and answer information to pre-trained models for qg task. the overall architecture is illustrated in figure 1 . we present a localness modeling by designing a gaussian bias to regulate the attention calculation, and propose a new attention mechanism to incorporate syntactic structure. specially, in localness modeling, we adopt answer position as the center of gaussian distribution and predict window size automatically, in order that the content around answer will be more focused and the range can be adjusted dynamically according to different token length. for syntactic knowledge, we first compute a syntactic vector through syntactic mask attention, which we design to blind the tokens outside of dependency triples, and then constitute a syntactic-aware representation by adding syntactic vector to the original context vector. we base our method on the strong pre-trained model prophetnet @xcite , which achieves state-of-the-art results on many generation tasks including qg. we conduct experiments on squad dataset @xcite . the automatic evaluation shows that our model boosts the performance of pre-trained models and achieves comparable stateof-the-art performance. further human judgement demonstrates that our model produces high-quality questions by capturing syntactic knowledge and answer-surrounded context. since our method is only a modification to the self-attention module, and there is little increase in parameter number, one can easily apply our method to other transformer-based pre-trained models while almost keeping the original speed without any much computation resources in the training stage. our codes and data will be publicly available at the github repository. contributions: to summarize, we (1) propose a modified attention mechanism to incorporate syntactic structure and position information, which can be easily applied to any transformer-based pre-trained language models for qg task; and (2) with selection of syntactic knowledge, our model is able to reach comparable state-of-the-art performance for question generation. ## conclusion in this work, we enhance the ability of qg systems by incorporating text structure, including syntactic dependency relations and answer position. we strengthen the localness modeling via a learnable gaussian bias with answer span as center, and present a syntactic mask attention mechanism to fuse structure information. specifically, we obtain the knowledge-aware context vector by adapting a visible matrix, where each token is only visible to its related token in the knowledge triple. experimental results on the widely-explored squad dateset demonstrate the effectiveness of our method. this work is based on the pre-trained prophetnet, but our methods can be easily applied to other transformer-based pre-trained models. in future work, we will validate and expand our method to other nlp tasks, such as summarization, dialog generation, machine reading comprehension, etc.