Unsupervised Generation of Long-form Technical Questions from Textbook Metadata using Structured Templates (2022)

Abstract: We explore the task of generating long-form technical questions from textbooks. Semi-structured metadata of a textbook — the table of contents and the index — provide rich cues for technical question generation. Existing literature for long-form question generation focuses mostly on reading comprehension assessment, and does not use semi-structured metadata for question generation. We design unsupervised template based algorithms for generating questions based on structural and contextual patterns in the index and ToC. We evaluate our approach on textbooks on diverse subjects and show that our approach generates high quality questions of diverse types. We show that, in comparison, zero-shot question generation using pre-trained LLMs on the same meta-data has much poorer quality.

## introduction we address automated generation of long-form technical questions from textbooks. such questions can then be used for technical assessments, such as in interviews and examinations. existing work on long-form question generation mostly focuses on questions for reading comprehension assessment @xcite @xcite @xcite @xcite . we observe that textbook metadata -specifically, the index and the table of contents (toc)provide rich cues for question generation. fig. 1 and fig. 2 show fragments from the index and the toc of a textbook on python. the index structure is hierarchical and often parsed into an subject and a context (e.g. functions and classes, managing). the toc is similarly hierarchical. the main challenge in question generation from unrestricted natural language content is identifying the relevant entity (or entities) and their context. the mentions of some of the relevant entities is often far away from the context, and the context may also use complex linguistic constructs. in contrast, the grammar for the metadata is significantly restricted, thereby simplifying the detection of the entities and contexts. additionally, their hierarchical structure compactly and completely captures relevant context. in this paper, we focus on automatically generating technical long-form questions using the index and toc of textbooks. while structured or semi-structured data has been used in nlp tasks such as knowledge graph construction @xcite , table -to-text generation @xcite ) and factoid question generation has been explored from knowledge-graphs @xcite , to the best of our knowledge there is no existing work that uses semi-structured content to generate questions either for technical assessment or for reading comprehension. we use structural patterns in the index and toc to design question templates of varying types and complexity, and then use unsupervised regular expression-based algorithms to generate questions by instantiating these templates using index and toc entries. template based approaches have been used in question generation from free text, typically with higher precision than deep learning based counterparts, while generating fewer questions @xcite @xcite . however, we are not aware of use of templates over semi-structured content for question generation. we apply our approach to generate questions from multiple textbooks on diverse subjects such as machine learning, java and pl/sql to demonstrate its generality. using automated and manual evaluation, we show that the generated questions are complex and diverse while having very high quality. we compare our approach with zero-shot question generation using llms, gpt-3 (brown et al., 2020) and bart @xcite . we show that these perform poorly in terms of both validity and diversity of the generated questions. in this section, we describe our approach for question generation from text-book index and toc using structural templates. we focus on generating questions that are answerable from book context. these have reference answers in the book, which can be used reliably for assessment. consider the index fragment from fig. 1 ## experimental evaluation in this section, we present experimental evaluation of our question generation approach. data: we use text books on diverse subjects -python @xcite , plsql (feuerstein and pribyl, 2014), java @xcite , machine @xcite and deep learning @xcite . the first two have the richest metadata structure, with 3-level indexes and tocs. in contrast, java has a 2-level index, while ml and dl have 1-level indexes and dl has 2-level tocs. we process the textbook pdfs to extract their index and toc automatically. details are in the appendix. ## conclusions we have motivated the task of automated technical question generation from semi-structured text-book index and toc. we have proposed an unsupervised approach based on structured templates that make use of their restricted grammar and hierarchical structure. we have shows using extensive evaluation that this approach performs better according to many different aspects of quality on multiple textbooks on a variety of subjects compared to zeroshot approaches using large language models. ming zhu, aman ahuja, da-cheng juan, wei wei, and chandan k reddy. 2020. question answering with long multiple-span answers. in findings of the association for computational linguistics: emnlp.