Generating Quizzes to Support Training on Quality Management and Assurance in Space Science and Engineering (2022)

Abstract: Quality management and assurance is key for space agencies to guarantee the success of space missions, which are high-risk and extremely costly. In this paper, we present a system to generate quizzes, a common resource to evaluate the effectiveness of training sessions, from documents about quality assurance procedures in the Space domain. Our system leverages state of the art auto-regressive models like T5 and BART to generate questions, and a RoBERTa model to extract answers for such questions, thus verifying their suitability.

## introduction the complexity, cost, and risk of space missions involving public or private investment and even human lives make quality management a critical requirement to guarantee their success. the european space agency (esa) makes a continuous effort to train their staff in quality procedures and standards. trainees are evaluated to determine the effectiveness of the training sessions, with quizzes as one of the main tools used in such evaluations. we present spaceqquiz (space quality quiz), a system designed to help trainers to generate quizzes from documents describing quality procedures. such documents cover topics like anomaly and problem identification, reporting and resolution or configuration management, and include stakeholder responsibilities, activities, performance indicators and outputs, among others. to design spaceqquiz we use state-of-the-art models based on transformers for question generation (qg) and question answering (qa). since we could not find specialized models for the space or quality management domains, we reuse models already pre-trained on general-purpose document corpora and fine-tuned on squad 1 . 1 the stanford question answering dataset https:// rajpurkar.github.io/squad-explorer/ ## quiz generation system figure 1 shows the high-level architecture of space-qquiz foot_0 . a question generation model is run on each passage extracted from the document. the generated questions and the corresponding passages are fed to a question answering model that extracts the answer from the passage. only questions with answers are included in the candidate list that then is refined by the trainer to generate the quiz. the process starts when the trainer uploads a quality procedure document. the system extracts the text from the pdf document using apache pdfbox foot_1 and uses regular expressions to identify sections, subsections and paragraphs while removing non relevant text such as headers and footers. the trainer is presented with a list of candidate sections so that she can choose the most interesting ones for the quiz. ## evaluation to evaluate the quiz generation system, we generate a quiz with 50 questions and answers pairs out of a quality procedure titled ops procedure for configuration management. then a quality management expert evaluates the generated questions using relevance and correctness as evaluation criteria. the result of this manual evaluation is reported in for some correct and relevant questions (see for example question 4 in finally the evaluator as domain expert reported that in some cases the problem is the underlying text used to generate the question that is not clear enough to formulate appropriate questions. thus, wrong questions might indicate text excerpts that need to be reviewed by the author to convey the messages clearly. ## conclusions we describe spaceqquiz, a system to help quality management trainers in the space domain to speed up the generation of evaluation material. space-qquiz uses autoregresive models such as t5 and bart to generate the questions, and roberta autoencoder to extract answers that are used as indicators of the validity of the questions.