Exploring Input Representation Granularity for Generating Questions Satisfying Question-Answer Congruence (2021)

Abstract: In question generation, the question produced has to be well-formed and meaningfully related to the answer serving as input. Neural generation methods have predominantly leveraged the distributional semantics of words as representations of meaning and generated questions one word at a time. In this paper, we explore the viability of form-based and more fine-grained encodings, such as character or subword representations for question generation. We start from the typical seq2seq architecture using word embeddings presented by De Kuthy et al. (2020), who generate questions from text so that the answer given in the input text matches not just in meaning but also in form, satisfying question-answer congruence. We show that models trained on character and subword representations substantially outperform the published results based on word embeddings, and they do so with fewer parameters. Our approach eliminates two important problems of the word-based approach: the encoding of rare or out-of-vocabulary words and the incorrect replacement of words with semantically-related ones. The character-based model substantially improves on the published results, both in terms of BLEU scores and regarding the quality of the generated question. Going beyond the specific task, this result adds to the evidence weighing different form- and meaning-based representations for natural language processing tasks.

## introduction question generation (qg) is a challenging nlp task, where both language form and meaning play a vital role in the production of questions that have to be well-formed and meaningfully related to the envisaged answer. neural models have been shown to be very promising for qg, with most recent approaches formulating the task as a sequence learning problem with the goal of mapping a sentence onto a question (e.g., @xcite @xcite . the research typically targets qg in the context of question answering, where the task is to generate a question that is related to the information in a given paragraph. the qa task ensures a general functional link between the question and the meaning of the passage that answers it. the datasets designed for such question answering/generation provide paragraph-level contexts for each question that span multiple sentences or even multiple passages. note that the question here is related to the information expressed in the text passage, not to the way in which this information is structured and expressed in the text. consider the example from the squad dataset shown in figure 1 . the first question pertains to the first sentence of the passage. while the concept gravity mentioned in that sentence is needed to answer the question, the question cannot be answered using the first sentence as such. for the second question, the information needed to answer the question is expressed in a sentence that is more in line with the question, but still falls short of the so-called question-answer congruence @xcite required for the sentence to serve as a direct answer to the question. context: in meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity. the main forms of precipitation include drizzle, rain, sleet, snow, graupel and hail. what causes precipitation to fall? gravity q 2 : what is another main form of precipitation besides drizzle, rain, snow, sleet and hail? graupel figure 1 : example question-answer pairs from the squad dataset @xcite complementing qg in the prominent qa context, there are other strands of qg research that aim at generating questions that can be answered by a sentence as given in the text, putting a premium on question-answer congruence. this includes qg work in the educational application domain, where the perspective of the question is supposed to reflect the perspective of the author of a given text passage that the student is supposed to learn about @xcite @xcite . recent work under this perspective includes @xcite , who propose a neural question generation architecture for the generation of cause-and-effect questions. they extract cause and effect relations from text, which are then used as answers for the neural question generation, aiming at direct question-answer congruence. a second strand of work for which the relation between the question and the answer sentence as expressed in the text plays a crucial role is the research interested in discourse. an early example of research investigating the role of discourse structure for question generation is @xcite . they identify discourse relations in a text as cues motivating the generation of a question and then formulate questions that can be answered by the sentences with those discourse relations, while ensuring direct question answer congruence. in a related vein, approaches making use of socalled questions under discussion (quds) to identify the information structure of a sentence in a given discourse also rely on such a direct relationship between question and answer. in a recent paper pursuing this perspective, de @xcite show that a seq2seq based neural approach can successfully generate meaningful, well-formed questions that can function as questions under discussions in a formal theory of discourse. similarly, @xcite showed that using questionanswer pairs obtained through crowdsourcing can be used reliably to annotate discourse. based on their crowdsourced data, they train a pipeline of neural models to directly generate such questionanswer pairs from text. the overall goal of question generation supporting discourse analysis is to generate a question for every sentence in a text to explicitly characterize the evolving discourse. viewed from the perspective of question generation for tasks requiring question-answer congruence, the qg task in essence consists of two steps: (i) replace the answer phrase in the source sentence with a matching question word and (ii) transform the rest of the sentence into a well-formed question. all the words that the generated question consists of are already given, so only the question word that matches the answer phrase needs to be generated anew. the sentence-question pair in example ( 1 ) taken from de kuthy et al. (2020) illustrates this. (1) a: auch also otto otto graf graf lambsdorf lambsdorf ist is gegen against zweierlei double wahlrecht. voting rights otto graf lambsdorf is also against double voting rights. q: wogegen what against ist is auch also otto otto graf graf lambsdorf? lambsdorf what is otto graf lambsdorf against, too? except for the answer phrase gegen zweierlei wahlrecht ('against double voting rights'), all words from the source sentence reappear in the generated question, including the named entity otto graf lambsdorf. the only new material in the question is the question word wogegen ('what against'). while the text thus includes all the language needed to successfully generate the question, for seq2seq-based approaches based on word embeddings, the challenge arises that words present in the source sentence which do not appear in the material the embeddings were trained on are not adequately represented. as admitted in de kuthy et al. (2020), unknown and rare words are therefore a problem and cannot be correctly generated in the question. rare words are often replaced by semantically related words that are inappropriate in the given context. in this paper, we explore an alternative: characters and subwords as form-based and more finegrained representations of both the input and output of the question generation task. we will show that this avoids the unknown/rare word problem and results in a substantial improvement both in a quantitative bleu evaluation and in terms of a qualitative analysis of the questions. going beyond the particular qg task, our results contribute to the general endeavour of exploring the best choices of form or meaning-based input and output representations for neural approaches for a range of nlp tasks depending on their characteristics. ## related work traditional question generation approaches that leveraged syntactic structures and linguistic features @xcite @xcite to define transformation rules on parse trees are inherently limited in their scope and ability to deal with authentic language data. deep learning has, in recent years, supplanted such methods given its ability to learn the syntactic and semantic properties and characteristics of language when provided with large amounts of natural language text. neural question generation is generally realised as a sequence learning problem, so a sequence-tosequence (seq2seq) architecture @xcite is a logical fit for this type of task. here, the encoder network learns the latent representation of the source sentence and the decoder network generates the target question one word at a time. the work done by @xcite introduces two such models, which are provided with the source sentence and paragraph-level information that encodes the context of the generated question. borrowing from reinforcement learning, the work by @xcite introduces policy gradients along with pos tags and named entity mentions to assign task-specific rewards to the training objective. pointer-generator networks @xcite with gated self-attention have been deployed to address the problem of rare and out-ofvocabulary words and larger contexts @xcite . the neural question generation models mentioned above, and many more in this vein, primarily focus on generating questions in english and consider words to be the atomic unit of meaning. they consequently approach the representation learning and text generation tasks at the word level. this assumption does not necessarily hold for languages such as chinese, where the individual characters contain rich internal information. neural language models that are trained on characterlevel inputs have been shown to capture more salient information about morphology than their word-level counterparts @xcite . character-aware question answering systems @xcite have similarly been shown to be resilient to the unknown word problem. to capture and combine information about language form and meaning, @xcite proposed treating words as bags of character n-grams to enrich word embeddings with subword information. byte-pair encoding @xcite has seen a recent resurgence in the context of generative language models where it is employed to perform subword segmentation without the necessity of tokenization or mor-phological analysis. subword-level embeddings learned with the help of this method have been competitive in many downstream nlp tasks @xcite @xcite . ## data in terms of datasets for neural question generation models, contemporary approaches are generally trained on datasets created in the question answering context. these datasets, such as squad @xcite , quac @xcite , and coqa @xcite , are not well-suited for training models for tasks requiring high questionanswer congruence, and they focus on english. multilingual datasets like xquad @xcite , mlqa @xcite , xnli @xcite , and tydi qa @xcite are similarly unsuitable as they contain only little data, intended as benchmark for the evaluation of question answering systems. given these limitations of the established english datasets for the research goals we are pursuing, we instead obtained the german qa answer corpus created by de kuthy et al. (2020) and base our explorations on that dataset. the corpus contains 5.24 million sentence-question-answer triples which were generated by a transformation-based question generation system @xcite on articles from the german newspaper die tageszeitung (taz, http://taz.de ). the corpus exhibits over 30 different types of questions, the most common of which are wh-questions asking for subject and object phrases (such as who or what questions in english) as well as various types of questions asking for adverbial modifiers (such as, for example, when or where questions). some typical questionanswer pairs will be discussed later in section 5. ## our character and subword-based neural qg approach as the starting point and baseline of our approach, we take the same basic architecture as de kuthy et al. (2020), a word-embedding based sequence-tosequence model @xcite with multiplicative attention @xcite . this was done in order to ensure comparability of our results with theirs. furthermore, any fundamental changes to the neural architecture -such as using a transformer @xcite or a pointer-generator (zhao et al., 2018) network -would make it more difficult to distinguish between any improvements offered exclusively by the change in input representation and those by the change in architecture. to introduce character-and subword-level tokens, we defined an input pipeline consisting of the following steps: 1) utf-8 text normalization was performed on the input sentence, 2) the normalized input sentence was parsed using spacy's (honnibal et al., 2020) de core news sm model to perform word-level tokenization and part-of-speech (pos) tagging, 3) a second tokenization pass was performed on each word token to generate character and subword tokens, and 4) each character and subword token pertaining to a given word token was assigned the latter's pos tag and the answer phrase indicator. for character-level tokenization, each word was decomposed into a list of its component unicode codepoints. subword tokenization was performed with the huggingface tokenizer library @xcite . the library provides byte-pair encoding @xcite and unigram @xcite tokenization algorithms. bpe first constructs a baseline vocabulary with all unique symbols in a corpus. then, merge rules that combine two symbols in the base vocabulary into a new symbol are learned iteratively until a desired final vocabulary size is reached. conversely, unigram tokenization starts with a large initial vocabulary from which it repeatedly removes symbols that have the least effect on a loss function defined over the training data of a unigram language model. to reduce the size of the base vocabulary in both models, base symbols are directly derived from bytes rather than (all) unicode codepoints. the library also includes the sentencepiece @xcite algorithm, which processes the input as raw string sequences obviating the need for pre-tokenization. finally, bidirectional lstm was used as the re-current unit in the encoder as we expect the contextual information provided by the backward pass to not only enrich the sentential representation learned in the encoder but also lower the effective reduction in learnable parameters caused by the smaller vocabulary sizes of the character-and subword-level models. the per-timestep input to the encoder is the concatenation of the token embedding, pos embedding, and the answer phrase indicator. the final outputs of the encoder (hidden state, sequences, cell state) is the concatenation of the respective backward and forward layers of each output. for the character-level models, a fixed-size vocabulary consisting of all the unique codepoints in the qa corpus was generated. similarly, the subword tokenizers were trained on the entire corpus to generate vocabularies with 10k symbols each. foot_0 ## evaluation for a comprehensive comparison, we trained five models: a word-level model to replicate de kuthy et al. (2020), three subword models with different tokenization algorithms (byte-level bpe, senten-cepiece bpe, and sentencepiece unigram), and a character model. all models were trained on the same 400k training samples from the qa corpus for 20 epochs, and validation was performed on 40k samples. for each type of input representation, the model with the lowest validation loss was was evaluated on a held-out test set of 15k samples. for their original model, de kuthy et al. (2020) implemented a post-processing copy module to replace oov marker tokens in the generated question with the original tokens from the source sentence; this behaviour was replicated for our word-level model. as model hyperparameters, we used: batch size: 128, encoder: bi-lstm, decoder: lstm, encoder/decoder hidden size: 256/512, encoder/decoder dropout: 0.5, word/subword/character embedding dim: 300, decoder beam search width: 5. ## conclusion we explored the prospect of neural question generation at the character-and subword-level using finergrained input representations than word tokens by adopting de kuthy et al. (2020)'s task of generating questions under discussion for german. the models that were trained on character and subword tokens showed significant leaps in bleu scores in comparison to the baseline word-level model, even in the absence of extra linguistic information. in addition to eliminating the problem of out-ofvocabulary and rare words, our manual analysis of the generated questions revealed that those models were able to learn and exploit both semantic and orthographic information with fewer parameters, producing questions with fewer errors relating to word order and word replacement. the character model, in particular, is able to fully eliminate the latter error category. considering the relevance of the research beyond the specific question generation task, the results reported in this paper provide further evidence and motivation to consider the advantages of formfocused neural representations and character-level natural language generation for tasks such as machine translation and extractive text summarization.