A Practical Toolkit for Multilingual Question and Answer Generation (2023)

Abstract: Generating questions along with associated answers from a text has applications in several domains, such as creating reading comprehension tests for students, or improving document search by providing auxiliary questions and answers based on the query. Training models for question and answer generation (QAG) is not straightforward due to the expected structured output (i.e. a list of question and answer pairs), as it requires more than generating a single sentence. This results in a small number of publicly accessible QAG models. In this paper, we introduce AutoQG, an online service for multilingual QAG along with lmqg, an all-in-one python package for model fine-tuning, generation, and evaluation. We also release QAG models in eight languages fine-tuned on a few variants of pre-trained encoder-decoder language models, which can be used online via AutoQG or locally via lmqg. With these resources, practitioners of any level can benefit from a toolkit that includes a web interface for end users, and easy-to-use code for developers who require custom models or fine-grained controls for generation.

## introduction question and answer generation @xcite is a text generation task seeking to output a list of questionanswer pairs based on a given paragraph or sentence (i.e. the context). it has been used in many nlp applications, including unsupervised question answering modeling @xcite @xcite , fact-checking @xcite , semantic role labeling @xcite , and as an educational tool @xcite . the most analysed setting in the literature, however, has been question generation (qg) with predefined answers, as this simplifies the task and makes the evaluation more straightforward. despite its versatility, qag remains a challenging task due to the difficulty of generating compositional outputs containing a list of question and answer pairs as shown in figure 1 , with recent works mainly relying on extended pipelines that include several ad-hoc models @xcite . these works integrate qag into their in-house software, preventing models to be publicly released, and their complex pipelines make them hard to reproduce and use by practitioners. in this paper, we introduce an open set of software tools and resources to assist on the development and employment of qag models for different types of users. we publicly release the following resources: foot_0 ## resources: models and datasets our qag toolkit makes use of pre-existing models and datasets, fully compatible with the hug-gingface hub. this makes our library easily extendable in the future as newer datasets and better models emerge. in this section, we describe the datasets ( § 2.1) and models ( § 2.2) currently available through lmqg and autoqg. ## lmqg: an all-in-one qag toolkit in this section, we introduce lmqg (language model for question generation), a python library for fine-tuning lms on qag ( § 3.1), generating question-answer pairs ( § 3.2), and evaluating qag models ( § 3.3). additionally, with lmqg, we build a rest api to host qag models to generate question and answer interactively ( § 5). lmqg is interoperable with the huggingface ecosystem, as it can directly make use of the datasets and models already shared on the huggingface hub. ## evaluation we rely on the qag models and datasets included in the library (see § 2). the individual qg components of each model (i.e. the generation of a question given an answer in a paragraph) were extensively evaluated in @xcite . for this evaluation, therefore, we focus on the quality of the predicted questions and answers given a paragraph (i.e. the specific answer is not pre-defined). for each model, we fine-tune, make predictions and compute their qaaligned scores via lmqg. ## autoqg finally, we present autoqg ( https://autoqg.net ), an online qag demo where users can generate question-answer pairs for texts in eight languages (english, german, spanish, french, italian, japanese, korean, russian) by simply providing a context document. we deploy the qag models described in § 2. in addition to the features described above, the online demo shows perplexity computed via lmppl, 7 a python library to compute perplexity given any lm architecture. this feature helps us provide a ranked list of generation to the user. although we can compute perplexity for non-english generations based on the qag models in each language, it entails large memory requirements on the the hosting server. as such, we compute a lexical overlap between the question and the document as a computationally-light alternative to the perplexity, which is defined as: figure 2 and figure 3 show examples of the interface with english and japanese qag, where there is a tab to select qag models, language, and parameters at generation including the beam size and the value for nucleus sampling @xcite . optionally, users can specify an answer and generate a single question on it with the qg model, as shown in figure 4 . a short introduction video to autoqg is available at https://youtu.be/ t6g-d9jtyyc . ## conclusion in this paper, we introduced lmqg, a python package to fine-tune, evaluate and deploy qag models with a few lines of code. the library implements the qag task as an efficient integration of answer extraction and question generation, and includes automatic reference-based metrics for model evaluation. finally, we showcase autoqg, an online demo where end users can benefit from qag models without any programming knowledge. autoqg enables the selection of features going from different models and languages to controlling the diversity of the generation.