Towards automatically generating Questions under Discussion to link information and discourse structure (2020)

Abstract: Questions under Discussion (QUD; Roberts, 2012) are emerging as a conceptually fruitful approach to spelling out the connection between the information structure of a sentence and the nature of the discourse in which the sentence can function. To make this approach useful for analyzing authentic data, Riester, Brunetti & De Kuthy (2018) presented a discourse annotation framework based on explicit pragmatic principles for determining a QUD for every assertion in a text. De Kuthy et al. (2018) demonstrate that this supports more reliable discourse structure annotation, and Ziai and Meurers (2018) show that based on explicit questions, automatic focus annotation becomes feasible. But both approaches are based on manually specified questions. In this paper, we present an automatic question generation approach to partially automate QUD annotation by generating all potentially relevant questions for a given sentence. While transformation rules can concisely capture the typical question formation process, a rule-based approach is not sufficiently robust for authentic data. We therefore employ the transformation rules to generate a large set of sentence-question-answer triples and train a neural question generation model on them to obtain both systematic question type coverage and robustness.

## introduction as the attention in linguistics is shifting from the analysis of isolated sentences to how information is encoded in discourse, the information structure of sentences and how it is expressed in a given language is receiving increasing interest. in order to connect the information structure of sentences to the overall structure of the discourse, so-called questions under discussion (qud) @xcite @xcite have emerged as a way to spell out the hinge between the properties of the sentence and the nature of the discourse in which the sentence can function. quds make explicit, how the meaning expressed by a sentence fits into the functional structure of the evolving discourse. example (1) illustrates this with the implicit qud q 2 connecting assertion a 2 to the previous discourse a 1 . concretely, the sentence a 2 answers the question q 2 given discourse a 1 . while explicitly annotating corpora with quds appears to be a key for reliable manual or automatic annotation of information and discourse structure, in all of the above approaches it is a complex manual step. in this paper, we propose to partially automate the qud annotation process for authentic german data. proceeding sentence by sentence through the text, we propose to automatically generate all potentially relevant questions for a given sentence. in principle, transformation rules can transparently express the potential types of question-answer pairs, e.g., a who question asking for the subject of a sentence, or a when question asking for a temporal adverbial. but while the relationship between the question phrase and the answer phrase can concisely be expressed by such transformation rules, the selection of the proper question phrase, the identification and removal of the answer phrase, and the reformulation of the sentence into question form and word order depends on a complex interplay of factors. we therefore proceed in two steps. we first employ a rule-based approach for german to produce questions for a large german newspaper corpus. we then pair each question with its answer phrase and the sentence it was generated from and use it to train a neural question generation model. the structure of the training data ensures high-quality question-answer congruence for a range of question types. at the same time, the variability of the authentic newspaper data that the rule-based approach was applied to provides a rich empirical basis for training a robust neural qg model that is capable of generalizing to data for which the rule-based approach fails to generate questions. the paper is structured as follows: section 2 provides some background on qud annotation to establish the nature of the questions we aim to generate. section 3 introduces the different methods we will combine in our approach, rule-based and neural qg. in section 4, we spell out how the rule-based approach is used to generate the data needed to train the neural qg, with the training being discussed in section 5. in the evaluation section 6, we illustrate the success of the methods in quantitative terms using bleu scores by comparing the questions generated by the neural approach in relation to the rulebased method. we then show that the neural approach generates high-quality, well-formed questions for more sentences than the rule-based approach. finally, we provide a qualitative analysis of the generated questions, also leading to ideas for future work discussed in section 7. ## discourse annotation with quds while quds were originally discussed in theoretical linguistics using constructed examples, the annotation framework introduced in @xcite presents a method for the reconstruction of quds in authentic data. the characterization of a qud for each assertion in a text is guided by explicit principles adapted from the formal semantic literature @xcite @xcite . the three central principles defined by @xcite are: q-a-congruence: quds must be answerable by the assertion(s) they immediately dominate. ## obtaining data for a neural question generation approach the productivity and rich compositionality of human language makes handcrafting rule-based systems for robust question generation very difficult. deep neural qg methods, on the other hand, can learn latent representations of syntactic and semantic language characteristics from large sets of data. for this one needs pairs of questions and the text they are asking about, i.e., a lot of instances of the generalizations encoded by the transformation-rules in a large, variable set of authentic language data. an nqg model trained on such a rich set of data then can potentially be more robust since it learns the systematic transformation patterns mixed in with the variable language patterns characterizing authentic use. obtaining appropriate training corpora thus is an essential ingredient of nqg. in current research approaching qg in the context of question answering (qa), qa corpora such as squad @xcite , coqa @xcite , or quac @xcite are typically used to train and evaluate the nqg models. however, such corpora are not well-suited given our goal of generating quds. these corpora provide a paragraph-level context for each question, where the question is related to the information encoded in the paragraph, not to the way this information is structured and presented in a sentence. so q-a-congruence between the question and a sentence that answers it is not ensured. for research like ours that investigates the relationship between the information structure of a sentence and the question context, corpus data that does not ensure q-a-congruence is insufficient. in addition, the existing corpora overwhelmingly provide only english data. the few multilingual, parallel datasets such as xquad @xcite and mlqa @xcite are evaluation datasets of very limited size. automatically translating a corpus or designing a neural model architecture to jointly translate, align and generate questions @xcite , while potentially promising, substantially increases complexity and potentially reduces performance due to translation error propagation. as empirical basis for developing a german nqg approach with the vision of generating quds, we created a suitable corpus by leveraging the rule-based question generation system of @xcite , which he kindly made available to us. creating such a corpus required a large, authentic german text source. we initially considered using the german version of wikipedia. yet the encyclopedic nature of these texts collecting facts from a wide of perspectives makes them suboptimal for our approach designed to identify quds in a text with a coherent discourse structure. we therefore settled on the german newspaper die tageszeitung (taz, https://taz.de ) which in the science edition is available in xml format and has also been used for the german tüba-d/z treebank @xcite . we extracted the text of 450k individual taz articles from years 1995 to 2001 using beautiful soup 4 ( https://crummy.com/software/beautifulsoup ) and performed tokenization and sentence segmentation using spacy's ( https://spacy.io ) de core news sm model. to eliminate potential segmentation problems and focus on declarative sentences, we filtered out sentences with fewer than four tokens, not starting with an uppercase letter, or not ending with a period or exclamation mark. the resulting 5.46 million sentences were fed into an updated version of the rule-based qg of @xcite , producing a corpus of 5.24 million triples of the @xmath0, question, and the answer phrase in the sentence given the @xmath1. this includes over 30 different types of question phrases, also discussed in section 6. the most common types of answer phrases are np subjects and objects as well as various types of adverbial modifiers. (3) and ( 4 in (3), the complex subject np shown in bold is replaced by the matching question word wer ('who'), and the question is generated with adjusted agreement morphology on the finite verb (@xmath2. in (4), the pp object in bold is replaced by the question word wovon ('what of'), and the question appropriately integrates the originally sentence-initial phrase (nostalgiker). ## training a neural question generation model question generation is a sequence learning problem where the model accepts an input sequence x 1 , ..., x n and learns the conditional probability p(y|x, z) of generating the target question y 1 , ..., y m while conditioned on the answer z: we implemented a seq2seq model with multiplicative attention @xcite using tensorflow 2.0 @xcite , with our code available upon request. the (surface-form) tokens of the source sentence, their part-of-speech tags, and the span of the answer phrase were used as inputs to the model. spacy ( https://spacy.io ) with the de core news sm pretrained model was used for tokenization, tagging, and parsing. the answer span was encoded in iob format. all input sequences were padded with special leading and trailing tokens to indicate their beginning and end. in the encoder stage of the model, the input at each timestep was the concatenation of the embeddings of the token and the pos tag, and the answer span indicator. pretrained fasttext embeddings @xcite were used to initialize the token embedding matrix, which was then frozen during training. the embedding matrix for the pos tags was randomly initialized. a fixed vocabulary was used for both input and target sequences, which is generated from 100k most frequent words in the corpus. out-of-vocabulary (oov) tokens were replaced with a special marker token. the model hyperparameters we used are given in we observed that many generated questions could be considered well-formed were it not for the appearance of the special token in place of oov words. most such occurrences appear in noun phrases, often involving proper nouns. to remedy this, we developed a simple post-processing module to heuristically improve the model's prediction. whenever the oov marker token appears in the generated question, the post-processing locally aligns the dependency parse tree of the noun phrases at the level of triplets @xcite between the source sentence and the generated question. the oov marker tokens in the question then are replaced with the aligned original tokens in the input sentence. given the heuristic nature of this align-and-copy post-processing step, its performance is limited by its local scope, the accuracy of the dependency parse, and the quality of the generated question. the corpus introduced in section 4 was iteratively undersampled to create multiple sets of training, validation and test data for different sample sizes with the same distribution of question types. we trained two versions of the model, one on 150k training samples and the other on 400k. validation datasets of 15k samples were used for both models. teacher forcing was enabled to ensure training stability. ## conclusion and outlook based on the insight that texts annotated with quds are an important source for reliable discourse analysis of larger corpora, we set out to explore how the labor-intensive manual annotation of quds can be partially automated by generating questions for every sentence in a given text. we employed a rule-based question generation model to generate a large corpus of sentence-question-answer triples. the corpus was used to train and test a neural question generation model that, given a sentence and a possible answer phrase, generates the matching question. the quantitative analysis using bleu scores and a manual evaluation of a sample set of generated questions showed that the neural model successfully generated meaningful, well-formed questions. it learned to predict correct question words for a given answer phrase and generated questions reflecting the appropriate word order characteristics. as a next step, we envision a question generation model that is able generate all possible questions that can be answered by a single sentence. for fully-automated qud annotation, the remaining step then is the selection of the specific question that is the appropriate qud in a given discourse. as an alternative, we also plan to investigate whether the neural question generation approach can be extended with discourse context to support qud generation in a single step. in terms of future work on the neural network approach, we also plan to leverage the insight that the vocabulary of the generated questions almost entirely derives from that of the source sentence by explicitly encoding it in the neural network's architecture using pointer networks @xcite or copy mechanisms @xcite . using pretraining methods @xcite to impart the model with a deeper understanding of the problem domain is another potential avenue for this future research. developing an approach supporting the generation of a specific qud for each sentence in a given discourse can also be seen as a step towards a normal form realization of quds. such normal forms could facilitate automatic comparison of discourse annotations, which currently is hampered by the fact that the pragmatic principles formulated by de @xcite characterize the meaning to be encoded by the question, so often there are multiple options for the concrete formulation of the question.