A SSUMPTION Q UESTIONING : L ATENT C OPYING AND R EWARD E XPLOITATION IN Q UESTION G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Question generation is an important task for improving our ability to process natural language data, with additional challenges over other sequence transformation tasks.
Recent approaches use modiﬁcations to a Seq2Seq architecture inspired by advances in machine translation, but unlike translation the input and output vocabularies overlap signiﬁcantly, and there are many different valid questions for each input.
Approaches using copy mechanisms and reinforcement learning have shown promising results, but there are ambiguities in the exact implementation that have not yet been investigated.
We show that by removing inductive bias from the model and allowing the choice of generation path to become latent, we achieve substantial improvements over implementations biased with both naive and smart heuristics.
We perform a human evaluation to conﬁrm these ﬁndings.
We show that although policy gradient methods may be used to decouple training from the ground truth and optimise directly for quality metrics that have previously been assumed to be good choices, these objectives are poorly aligned with human judgement and the model simply learns to exploit the weaknesses of the reward source.
Finally, we show that an adversarial objective learned directly from the ground truth data is not able to generate a useful training signal.
I NTRODUCTION
Posing questions about a document in natural language is a crucial aspect of the effort to automatically process natural language data, enabling machines to ask clariﬁcation questions (Saeidi et al., 2018), become more robust to queries (Yu et al., 2018), and to act as automatic tutors (Heilman &
Smith, 2010).
While questions often have a unique answer, the inverse is rarely true; there are multiple ways of phrasing a question, and there can be semantically different questions with the same answer.
The ability to generate questions therefore provides a mechanism for augmenting existing question answering datasets or automatically annotating new ones, improving the resilience of question answering models.
Recent approaches to question generation or answer questioning (AQ) have used Seq2Seq (Sutskever et al., 2014) models with attention (Bahdanau et al., 2014) and a form of copy mechanism (Vinyals et al., 2015; Gulcehre et al., 2016).
Such models are trained to generate a plausible question, conditioned on an input document and answer span within that document (Zhou et al., 2018;
Du et al., 2017; Du & Cardie, 2018; Yuan et al., 2017).
Many innovations in sequence generation have been led by a motivation to improve neural machine translation (NMT) models, where the input and output vocabularies are largely orthogonal and the correct lengths of the input and output sequences are often similar.
Shi et al.
(2016) showed that speciﬁc units within NMT models learn to count the sequence length during decoding, to help achieve the correct output length.
In this way, NMT can be viewed as separate alignment and per-token transformation tasks.
There is no particular intuition that the lengths of a context document and a question about that document should be related; there is a correlation of only 0.009 between the lengths of contexts and related questions in the popular SQuAD question answering dataset (Rajpurkar et al., 2016).
The document and the question are also written in the same language, and so use overlapping vocabu- also appear in at document.
AQ therefore presents signiﬁcantly different challenges to NMT.
Recent works on AQ have used various formulations of copy mechanisms, but have not investigated which approach should be preferred.
We show that by removing inductive bias from the model and allowing the choice of generation path to become latent, we achieve substantial improvements over both a naive and more principled biased implementation.
There are currently no dedicated question generation datasets, and recent work has used the contextquestion-answer triples available in SQuAD.
Only a single question is available for each contextanswer pair, and models are trained using teacher forcing (Williams & Zipser, 1989).
The combination of these factors exacerbates the problem of exposure bias (Ranzato et al., 2015), whereby the model does not learn how to distribute probability mass over sequences that are valid but different to the ground truth.
Recent work has investigated training the models directly on a performance based objective, either by optimising for BLEU score (Kumar et al., 2018a) or other quality metrics (Yuan et al., 2017).
There is an implicit assumption that these metrics are in fact good proxies for question quality.
We perform ﬁne tuning using a range of rewards, including an adversarial objective.
We show that although this leads to increases in those scores, the resulting models perform worse when evaluated by human workers, and the generated questions exploit weaknesses in the reward models.
R ELATED W ORK
Neural machine translation (NMT) has led to a number of advances in sequence modelling (Sutskever et al., 2014;
Bahdanau et al., 2014; Gulcehre et al., 2016;
Gehring et al., 2017), but for NMT the input and output sequences often have comparable lengths (Shi et al., 2016).
This means that the attention weights correspond closely to per-token alignments, and the task becomes closer to a context-aware per-token transformation.
Question generation requires identifying the important sections of the context before reordering some phrases and constructing some new ones, and so poses somewhat different challenges.
Summarisation involves taking a context document as input and generating a summary that should be considerably shorter, and is therefore more similar to the task of question generation.
Cheng &
Lapata (2016) propose an extractive neural summarisation framework that makes use of a Seq2Seq architecture with an attention mechanism, and Gu et al.
(2016a) extend this by adding a pointer network.
Nallapati b present model, extend these approaches by adding a pointer network.
Paulus et al.
(2017) propose a framework for ﬁne tuning a summarisation model using reinforcement learning.
Early systems for generating questions were generally based around the use of some sort of templating or rule-based reordering of the context tokens (Heilman & Smith, 2010; Heilman, 2011; Agarwal et al., 2011; Ali et al., 2010; Danon & Last, 2017; Popowich & Winne, 2013; Chali & Golestanirad, 2016; Labutov et al., 2015; Mazidi & Nielsen, 2014).
Similar to these template based approaches, neural techniques have been used to generate questions from entities and relations in a knowledge base (Serban et al., 2016; Indurthi et al., 2017), but these require knowledge of the relations in advance and do not work from the raw textual input.
AQ systems can be used to augment datasets for training QA models, and Wang et al.
(2017) and
Tang et al.
(2017) approach the task with this in mind.
They generate questions using a Seq2Seq model, but primarily focus on the resulting improvement to the QA model.
Yang et al.
(2017) take a similar approach, using an AQ model to facilitate semi-supervised training of a QA model on a range of different domains.
The AQ task has also been approached by using only the document to generate questions, without conditioning on a speciﬁc answer.
Subramanian et al.
(2017) used named entity recognition to identify key phrases in the context before feeding this reduced input to a Seq2Seq model.
They report an improved rating by human evaluators, but do not give any automated evaluation metrics for the generated questions.
Kumar et al.
(2018b) use a similar two-stage approach, adding attention and a pointer network to the decoder.
Kumar et al.
(2018a) further update this model by performing secondary policy gradient training, using BLEU and other automatic metrics as the rewards.
Du et al.
(2017) use a Seq2Seq based model to generate questions conditioned on context-answer pairs, and build on this work by preprocessing the context to resolve coreferences and adding a pointer network (Du & Cardie, 2018).
Similarly, Zhou et al.
(2018) use a part-of-speech tagger to augment the embedding vectors.
Both works perform a human evaluation of their models, and show signiﬁcant improvement over their baseline.
Song et al.
(2018) use a modiﬁed context encoder based on multi-perspective context matching (Wang et al., 2016), similar to cross attention.
Bahuleyan et al.
(2017) used a variational encoder to generate multiple questions from a single context sentence.
Gao et al.
(2018) propose splitting the training data by the difﬁculty of question, and including this difﬁculty as part of the conditioning on the decoder.
Yuan et al.
(2017) describe a Seq2Seq model with attention and a pointer network, with an additional encoding layer for the answer.
They also describe a method for further tuning their model on a language model and question answering reward objective using policy gradient, but unfortunately do not perform any human evaluation.
M ODEL D ESCRIPTION 3. T ASK DEFINITION
The training data consists of context-question-answer triples ( D , Q , A ) , that have been tokenised such that D
= { d 1 , d 2 , . . .
, d | D | } where | D | is the number of tokens in the document, and similarly for the question and answer.
The task is to generate a natural language question ˆ Y , conditioned on a document D and answer
A , by sampling from the parameterised conditional distribution at each time step given by p (ˆ y t ) =
p θ (ˆ y t | ˆ y <t , D , A ) .
For example, given the input document D = ”this paper investigates assumptions in question generation” and A = ”question generation”, the model should produce a question such as ˆ Y = ”what is investigated in the paper ?”.
3. E NCODER
The context tokens are transformed into an embedding representation d t , by looking up the relevant entry in the word embedding matrix.
We initialise with vectors from GloVe (Pennington et al., 2014) if the word exists in the GloVe vocabulary; otherwise we initialise with a random vector (Glorot &
Bengio, 2010).
We limit these embeddings to the top 2000 words in the SQuAD contexts and questions combined, ranked by frequency.
Context words not in this vocabulary are mapped to the out-of-vocabulary token for the purpose of embedding.
The embedded tokens d t are augmented with an additional binary feature, indicating whether that token comprises part of the answer or not, so that ˜ d t =
[ d t ; I ( d t ∈ A )] .
The sequence of augmented embeddings is passed through a bidirectional LSTM layer (Hochreiter & Schmidhuber, 1997) to generate the context encodings h dt .
For the model proposed by Yuan et al.
(2017) with additional condition encoding, the encodings at the time steps corresponding to the answer span are concatenated with the word embeddings of these tokens, and this (shorter) sequence is passed through a second bidirectional LSTM layer.
The condition encoding vector h a is given by the RNN output at the last time step.
For the standard Seq2Seq model, the condition encoding is calculated as the mean context encoding of the answer span, h a = 1 | A | (cid:80) t ∈
A h dt .
3. D ECODER
At each time step, a weighted context vector v t is computed using an attention mechanism to calculate soft alignments with the context document, and taking a weighted sum of the context encodings.
This vector v t =
| D | (cid:80) t (cid:48)
α t,t (cid:48)
h dt (cid:48) is used as the input to a unidirectional LSTM layer, giving the outputs o t .
The initial state for the decoder RNN is calculated according to s 0 =
tanh( W 0 r + b 0 )
where r
= Lh a + 1 n | D | (cid:80) t h dt , and L , W 0 , and b 0 are learned parameters. Alignment scores α t are calculated using an attention mechanism (Bahdanau et al., 2014).
Brieﬂy, this takes the form of a fully connected network with a single hidden layer and a softmax output layer, that takes the current context encoding and previous hidden state as inputs, and produces a distribution over context time steps α t,i = exp( e t,i ) (cid:80) j exp( e t,j ) as output, where e t = f
( v t , s t − 1 ) is a fully connected neural network with tanh activation.
3. C OPY M ECHANISM
In order to handle unknown words, the model is able to generate tokens from two vocabularies: a shortlist vocabulary V s of common context-independent words, and the location or copy vocabulary
V c formed of the words in the context document, indexed by their location in the context.
To calculate the distribution over shortlist tokens p st , the output of the LSTM cell at each step is projected into the dimensionality of the shortlist vocabulary, and normalised with a softmax activation, so that p st = softmax ( W s o t + b s ) where W s , b s are learned parameters and o t is the output of the LSTM at each time step.
The distribution over context locations for the pointer network p ct is calculated by reusing the alignments from the attention mechanism, giving p ct = α t .
The combined distribution is then calculated by concatenating the shortlist and location distributions, weighted by a switch variable that controls the degree of mixing of the two distributions.
This switch variable z t is calculated at each step by passing v t and y t − 1 as inputs to a feedforward network with two hidden layers and tanh activation, with a single output variable passed through a sigmoid activation, z t = σ ( f ( v t , y t − 1 ) .
The ﬁnal output distribution over shortlist and location vocabularies is therefore given by p t =
[ z t p st ; (1 − z t ) p ct ] , where [] is used to denote concatenation.
3. T RAINING
The ground truth data was encoded as a sequence of one-hot vectors ˜q ( t ) over the combined shortlist and location vocabularies, with ˜ q
i ( t ) =
I ( w i = q t ) for w i ∈ V s .
Tokens that did not occur either in the shortlist or context were encoded as an out-of-vocabulary token.
We trained the model on a maximum likelihood objective with teacher forcing Williams & Zipser (1989), using the Adam (Kingma & Ba, 2014) optimisation algorithm, and perform early stopping based on the negative log-likelihood of the development set.
Dropout (Srivastava et al., 2014) and variational dropout (Gal & Ghahramani, 2015) were used where appropriate.
The ofﬁcial SQuAD test set is not public, and we use the split published by Du et al.
(2017) for our experiments.
For inference, we used beam search (Graves, 2012) with a width of 32.
We also zeroed out the probability mass for the out-of-vocabulary token, to force the decoder to generate valid words.
E XPERIMENTS 4. C OPY M ECHANISM F ORMULATION
Using a copy mechanism allows the model to generate language from a mixture of two vocabularies: a pre-deﬁned shortlist of common words, and a context speciﬁc location vocabulary of words that appear in the source document.
The probability of generating a token from one of these vocabularies compared to the other is controlled by the switch variable z t .
For training samples where the shortlist and location vocabularies do not overlap, the correct value of the switch variable can easily be inferred, since there is only one way to generate each word.
In practice this is rarely the case: the vocabularies often overlap signiﬁcantly, and there may also be repetition within the context.
The original use of pointer networks for NLP (Gulcehre et al., 2016) was in the context of NMT, where the source and target language are different and the vocabularies can be assumed to be orthogonal, except for words which are named entities and must therefore be copied. Gulcehre et al. (2016) therefore assumed that words are generated from the shortlist by default, except when they must be copied.
For the case where there is a choice of copy location, they simply selected the earliest, and we consider this approach as our default model.
CopyNet, concurrently proposed by Gu et al.
(2016b), outputs a mixture of logits for the shortlist and location vocabularies, takes a softmax over these logits, and subsequently sums the probability mass for the overlapping tokens.
This effectively makes the switch variable and choice of location into latent variables.
See et al.
and Du & Cardie (2018) also combine the probabilities for tokens with the same value.
Yuan et al.
(2017) use a copy mechanism as part of their model, but do not explicitly discuss the overlapping vocabulary problem and we assume that they treat all shortlist and location tokens as orthogonal.
In each of these cases, the design choices are somewhat arbitrary, and were not tested.
There are two ways in which the vocabularies may overlap: within the location vocabulary, and between the location and shortlist vocabulary.
We can remove a source of bias from the model by making the choice of copy location latent, by summing the probabilities of generating a token from each possible location, so that p (cid:48) ct ( w )
= (cid:80) i p ct ( w i ) I (
w i = w ) where i is an index on the locations in the context.
We can also treat the switch variable as latent by summing over vocabularies, giving p (cid:48) t ( w ) =
z t p st ( w ) + (1 − z t )
p ct ( w ) for w ∈ V s ∩ V c and p (cid:48) t ( w ) =
p t ( w ) otherwise.
It is not always clear that removing bias from a model will improve performance, and it may be better to use of our understanding of the problem to guide the model.
We expect that the question should be similar to the language found near the answer span, and so design a heuristic detailed in
Appendix A to incorporate this prior belief.
The heuristic seeks to bias the model during training to copy contiguous runs of tokens from close to the answer span.
4. F INE T UNING
Generated questions should be formed of language that is both ﬂuent and relevant to the context and answer.
We experiment with ﬁne tuning a trained model using rewards given by the negative perplexity under a LSTM language model and the F1 score attained by a question answering system, as well as a weighted combination of both.
The language model is a standard recurrent neural network formed of a single LSTM layer.
For the QA system, we use QANet (Yu et al., 2018) as implemented by Kim (2018).
Additionally, we propose a novel approach by learning the reward directly from the training data, using a discriminator detailed in Appendix B. We pre-trained the discriminator to predict whether an input question and associated context-answer pair was generated by our model, or originated from the training data.
We also interleaved updates to the discriminator within the ﬁne tuning phase, allowing the discriminator to become adversarial and adapt alongside the generator.
These rewards R ( ˆ Y ) were used to update the model parameters via the REINFORCE policy gradient algorithm (Williams, 1992), according to ∇L = ∇ 1 l (cid:80) t
( R (ˆ Y ) − µ R σ R ) log p (ˆ y t | ˆ y <t , D , A ) .
We teacher forced the decoder with the generated sequence to reproduce the activations calculated during beam search to enable backpropagation.
All rewards were normalised with a simple form of PopArt (Hasselt et al., 2016), with the running mean µ R and standard deviation σ R updated online during training.
We continued to apply a maximum likelihood training objective during this ﬁne tuning.
4. E VALUATION M ETRICS
We report the negative log-likelihood (NLL) of the test set under the model, as well as the corpus level BLEU-4 score (Papineni et al., 2002) of the generated questions compared to the ground truth.
We report the macro-averaged F1 score attained by a QA system, which can be viewed as a form of reconstruction score, since it should be possible to recover the answer used to generate a good question.
We also report the perplexity of generated questions under a LSTM language model (LM) trained on the questions from SQuAD.
R ESULTS 5. A UTOMATIC M ETRICS
[Table]
Table 1 shows the values of the automatic metrics for various conﬁgurations of the copy mechanism, for both a standard Seq2Seq condition encoding and for the additional encoding used by Yuan et al.
The LM perplexity of generated questions is lower than the ground truth for all conﬁgurations; this is to be expected, since the question generator is itself effectively a conditional language model.
For the standard Seq2Seq condition encoding, using a smarter copy location heuristic to bias the model during training leads to a small improvement of +0.5 BLEU.
Allowing the model to instead learn this heuristic by making both the copy location and the switch variable latent leads to signiﬁcant improvements of +1.5 BLEU and +2.0 QA score.
When the additional condition encoding is included, we no longer observe signiﬁcant improvement for the latent formulations of the copy mechanism, and instead ﬁnd that performance stays the same or decreases by up to 0.7 BLEU.
The additional encoding layer increases the number of parameters in the model; when this is coupled with the additional freedom in the copy mechanism, the model is unable to learn as effectively.
Table 2 shows the changes in automatic metrics after ﬁne tuning using various external rewards.
Optimising for QA, LM and discriminator rewards improves those scores, although a larger improvement in LM score is achieved with a combined QA and LM reward.
The biggest improvement in discriminator score is achieved using an adversarial objective, and using a weighted sum of all three objectives leads to improvements in all three rewards.
The BLEU score decreases in all cases, as the rewards are not coupled to the training data.
5. H UMAN E VALUATION
We follow the standard approach in evaluating machine translation systems (Koehn & Monz, 2006), as used for AQ by Du & Cardie (2018).
We asked three workers to rate 300 generated questions between 1 (poor) and 5 (good) on two separate criteria: the ﬂuency of the language used, and the relevance of the question to the context document and answer.
[Table]
[Table]
The mean of the pairwise inter-rater Fleiss’ Kappa (L. Fleiss, 1971) agreement metrics was 0.45 for ﬂuency, and 0.44 for relevance, corresponding to moderate agreement.
While this seems low, the metric considers the rating classes to be unordered and equally different from each other, and so is pessimistic.
As shown in Table 3, removing bias by making the copy mechanism handle overlapping words in a latent manner leads to improved ﬂuency and relevance.
The models that were ﬁne tuned on external rewards achieve worse human scores.
We note that although BLEU has been shown to be an imperfect metric
(Paulus et al., 2017; Chaganty et al., 2018), in this instance it is sufﬁcient to predict the human ranking of the different models.
[Figure]
Figure 1 shows the automatic scores against human ratings for all rated questions.
The correlation coefﬁcient between human relevance and automatic QA scores was 0.439, and between ﬂuency and
LM score was only 0.355.
While the automatic scores are good indicators of whether a question will achieve the lowest human rating or not, they do not differentiate clearly between the higher ratings: training a model on these objectives will not necessarily learn to generate better questions.
A good question will likely attain a high QA and LM score, but the inverse is not true; a sequence may exploit the weaknesses of the metrics and achieve a high score despite being unintelligible to a human.
Using a weighted combination of rewards is not sufﬁcient to provide a useful training objective.
Table 4 shows examples of generated questions for the ﬁne tuned models.
Training on a QA reward has caused the model to learn to exploit this reward, by simply using a few keywords to point at the answer.
This suggests an alternative application of AQ models for generating adversarial data for
QA systems and exposing their failure cases, similar to the work by Jia & Liang (2017).
Training on an adversarial objective should prevent the generator from being able to exploit the weaknesses of the reward model.
We ﬁnd that although the generated sequences appear reasonable, the model ﬁne tuned on an adversarial reward was not more highly rated by the human workers.
[Table]
although united methodist practices and interpretation of beliefs have evolved over time , these practices and beliefs can be traced to the writings of the church ’s founders , especially john wesley and charles wesley ( anglicans ) , but also philip william otterbein and martin boehm ( united brethren ) , and jacob albright ( evangelical association ) .
[Table]
are the writings of the church ’s founders
[Table]
who founded the church ’s founders ?
Discriminator reward, adversarial discriminator who were two western methodist practices ?
LM, QA and discriminator reward, adversarial discriminator who are the anglicans of the church ?
learned to simply point at the answer and exploit the QA model, while the model trained on a language model objective has learned to repeat common phrase templates.
C ONCLUSION
In this paper we clarify two fundamental assumptions in recent work on question generation.
We show that, for standard Seq2Seq models, removing inductive bias by making the source of nonunique words latent improves the quality of generated questions.
We perform a human evaluation to conﬁrm these ﬁndings.
We also ﬁnd that although policy gradient methods can be used to optimise for external rewards, these rewards do not correlate well with question quality despite being intuitively good choices.
The generator may simply learn to exploit the weaknesses of the reward model, suggesting a possible use of AQ systems to generate adversarial training data for those reward models.
Fine tuning on an adversarial objective did not lead to improved question quality.
A S MART C OPY H
EURISTIC if the token exists only once in the context then use that location elseif there are multiple instances and the previously encoded token was from the location vocabulary and the token at the next position in the context is the correct token then use that position else ﬁnd the occurrence nearest the answer span
B D ISCRIMINATOR A RCHITECTURE
[Figure]
We used an architecture based on a modiﬁed QANet as shown in Figure 2, replacing the output layers of the model to produce a single probability.
Since the discriminator is also able to consider a full context-question-answer triple as input (as opposed to a context-question pair for the QA task), we fused this information in the output layers.
Speciﬁcally, we applied max pooling over time to the output of the ﬁrst two encoders, and we took the mean of the outputs of the third encoder that formed part of the answer span.
These three reduced encodings were concatenated, a 64 unit hidden layer with ReLU activation applied, and the output passed through a single unit sigmoid output layer to give the estimated probability that an input context-question-answer triple originated from the ground truth dataset or was generated.
C H YPERPARAMETER VALUES
[Table]
[Table] [Table]