C opy BERT : A Unified Approach to Question Generation with Self-Attention (2020)

Abstract: Contextualized word embeddings provide better initialization for neural networks that deal with various natural language understanding (NLU) tasks including Question Answering (QA) and more recently, Question Generation(QG). Apart from providing meaningful word representations, pre-trained transformer models (Vaswani et al., 2017), such as BERT (Devlin et al., 2019) also provide self-attentions which encode syntactic information that can be probed for dependency parsing (Hewitt and Manning, 2019) and POStagging (Coenen et al., 2019). In this paper, we show that the information from selfattentions of BERT are useful for language modeling of questions conditioned on paragraph and answer phrases. To control the attention span, we use semi-diagonal mask and utilize a shared model for encoding and decoding, unlike sequence-to-sequence. We further employ copy-mechanism over self-attentions to acheive state-of-the-art results for Question Generation on SQuAD v1.1 (Rajpurkar et al., 2016).

## introduction automatic question generation (qg) is the task of generating meaningful questions from text. with more question answering (qa) datasets like squad @xcite that have been released recently @xcite @xcite , there has been an increased interest in qg, as these datasets can not only be used for creating qa models but also for qg models. qg, similar to qa, gives an indication of machine's ability to comprehend natural language text. both qa and qg are used by conversational agents. a qg system can be used in the creation of artificial question answering datasets which in-turn helps qa @xcite . it specifically can be used in conversational agents for starting a conversation or draw attention to specific information with answer phrase {a i } a @xmath0.2), the model explicitly uses h multi-headed self-attention matrices from l layers of transformers to create @xmath1. this matrix along with @xmath2, obtained from the bert sequence output @xmath3, is used to learn copy probability p c (q i |.) ( §3.3.2). finally, a weighted combination p(q i |.) is obtained with simple generation probability p g (q i |.) ( §3.4). @xcite . @xcite and @xcite use qg to create and augment conversational characters. in a similar approach, @xcite creates a virtual instructor to explain clinical documents. in this paper, we propose a qg model with following contributions: ## related work most of the qg models that use neural networks rely on a sequence-to-sequence architecture where a paragraph and an answer is encoded appropriately before decoding the question. sun et al. ( 2018 ) uses an answer-position aware attention to enrich the encoded input representation. recently, @xcite showed that learning to predict clue words based on answer words helps in creating a better qg system. with similar motivation, gated self-networks were used by @xcite to fuse appropriate information from paragraph before generating question. more recently, self-attentions of a transformer has been shown to perform answer agnostic question generation @xcite . the pre-training task of masked language modeling for bert @xcite and other such models @xcite make them suitable for natural language generation tasks. @xcite argues that bert can be used as a generative model. however, only few attempts have been made so far to make use of these pre-trained models for conditional language modeling. @xcite and @xcite use a single bert model for both encoding and decoding and achieve state-of-the-art results in qg. however, both of them use the [mask] token as the input for predicting the word in place, which makes the training slower as it warranties recurrent generation @xcite or generation with random masking @xcite . both models only consider the output representations of bert to do language modeling. however, @xcite and @xcite show that bert learns different linguistic features at different layers. also, @xcite successfully probed for dependency trees from self-attention matrices of bert. with this, we hypothesize that bert can implicitly encode the different aspects of input for qg @xcite within the selfattentions across layers. as self-attention can learn soft-alignments, it can be used explicitly for copy mechanism ( §3.3.2), and can yield better results ( §4.3) than a model that only implicitly use selfattentions for qg ( §3.3.1). similar to @xcite , we also employ a shared architecture for unified encoding-decoding but make an explicit use of self-attentions across layers, leading to similar or better results at a fraction of their training cost. ## model in sequence-to-sequence learning framework, a separate encoder and a decoder model is used. such an application to bert will lead to high computational complexity. to alleviate this, we use a shared model for encoding and decoding @xcite . this not only leads to a reduced number of parameters but also allows for cross attentions between source and target words in each layer of the transformer model. while such an architecture can be used in any conditional natural language generation task, here we apply it for qg. ## experiments we apply the different variations of copybert model as mentioned in the previous section on squad v1.1 @xcite . for our experiments foot_1 , we follow the training, validation and test split as used in @xcite . ## conclusion we showed that having a unified encoder-decoder transformer model initialized with contextualized word embeddings and further extended with copy mechanism can already give state-of-the-art, without additional pre-training on generation tasks @xcite . we also sped up the training of qg models that use bert by choosing predictions on output embeddings that are offset by one position ( §3.3). this work shows the significance of explicitly using self-attentions of bert like models. these models can further be used in other tasks such as abstractive summarization and machine translation to see qualitative improvements.