Generation and Evaluation of Multiple-choice Reading Comprehension Questions for S wedish (2024)

Abstract: Multiple-choice questions (MCQs) provide a widely used means of assessing reading comprehension. The automatic generation of such MCQs is a challenging language-technological problem that also has interesting educational applications. This article presents several methods for automatically producing reading comprehension questions MCQs from Swedish text. Unlike previous approaches, we construct models to generate the whole MCQ in one go, rather than using a pipeline architecture. Furthermore, we propose a two-stage method for evaluating the quality of the generated MCQs, first evaluating on carefully designed single-sentence texts, and then on texts from the SFI national exams. An extensive evaluation of the MCQ-generating capabilities of 12 different models, using this two-stage scheme, reveals that GPT-based models surpass smaller models that have been fine-tuned using small-scale datasets on this specific problem.

## introduction in several educational stages, multiple-choice questions (mcqs) provide a widely used means of assessing reading comprehension @xcite . tests that consist of mcqs have the very appealing property of allowing swift, automatic, and objective grading. however, creating such tests is far from being swift or automatic, but rather is time-consuming and requires a great deal of expertise @xcite . in this work, we analyze to what extent the creation of mcq tests for reading comprehension in swedish could be automated using publicly available language models (both closed-source models via public apis, and open-source ones). the focus of this article is on mcqs aimed at assessing reading comprehension of second-language learners of swedish, specifically aimed at the swedish for immigrants courses (sfi). our contributions 1 are: an mcq consists of a question proper (the stem), the correct answer (the key), and a number of wrong but plausible options (the distractors). we will refer to the key and distractors together as alternatives (alt). contrary to prior work @xcite @xcite @xcite , we do not split the problem of generating stem and key from the problem of generating distractors. instead, we aim to generate the whole "package" at once and be able to offer more than one mcq per text. we assume that the texts are already given, and that they are on the appropriate level for testing reading comprehension, e.g., they are of the appropriate length, split into paragraphs, use the vocabulary of the appropriate complexity for the second-language learners, etc. in this article we do not conduct any assessment on how appropriate the given texts are for testing reading comprehension. the interested reader is referred to @xcite for more discussion on this matter. in the aforementioned prior work, researchers have tried a two-stage approach to mcq generation, first generating a stem-key pair using one method and then generating the distractors using another method. such approaches have a number of advantages, e.g., the key can be extracted directly from the text and thus guar-anteed to be correct, or the stem/key formulation can be edited to (hopefully) get higher quality distractors. nevertheless, generating the whole "package" at once, as we attempt to do in this article, has also its advantages. our motivation is twofold. the first reason is that a stem-key pair produced at stage 1 might not necessarily allow for good distractors to be produced at stage 2. the hope is that when generating the whole mcq in one go, the model will learn to only generate stems that have reasonable alternatives. the second reason is speed of execution, meaning that it is more resource-efficient to run one model and directly obtain the entire mcq, instead of running several models that generate each part of the mcq separately. ## related work automatic question generation from text has been studied before, mainly for the english language. results obtained up to 2020 are summarized in the survey article by @xcite . however, very little work has been done on generating whole mcqs (rather than just the questions), although some researchers have focused on generating distractors separately using large language models @xcite @xcite @xcite . to the best of our knowledge, there have been no prior attempts on generating the whole reading comprehension mcqs for swedish using fully open-source and free-to-use models. the only prior attempt in this direction was by @xcite , where they generated mcqs using openai's gpt-3 @xcite , which is neither open-source nor free to use. however, there have been attempts at generating parts of mcqs for swedish. @xcite generated stems and keys separately using a data-driven rule inductor based on universal dependencies for creating templates for question-answer pairs and then attempting to apply them to single sentences during the generation process. @xcite experimented with a method based on the swedish bert @xcite for generating distractors given the text, the stem, and the key. ## method in this work, we have experimented with fine-tuning two publicly available large language models: swedish bert @xcite , and swectrl-mini @xcite . we compared the finetuned models to two baselines described in section 4.4, as well as gpt-3 @xcite , specifically textdavinci-003, and chatgpt foot_3 , specifically gpt-3.5-turbo-0301. we did not use gpt-4 foot_4 in this work, because at the time of writing, access to its api is limited for the general public. the same goes for gpt-sw3 foot_5 . ## experimental setup in this work we fine-tuned models for the 4 proposed methods (kb/bert lrv, kb/bert aov-a, kb/bert aov-b, and swectrl-mini). each of these models was trained on two datasets (swequad-mc, and quasi), resulting in 4 • 2 = 8 models. each fine-tuned model was trained for 10 epochs using the adamw optimizer @xcite with the default huggingface training parameters: initial learning rate 5 × 10 -5 , 𝛽 1 = 0.9, 𝛽 2 = 0.999, 𝜖 = 1×10 -8 , without learning rate scheduling or weight decay. the gradients were clipped to the norm of 1. the training was conducted on a single nvidia 3090 gpu with 24gb of vram using a batch size of 8 for models based on swectrl-mini, and a batch size of 16 for those based on kb/bert. the checkpoints for each model were saved for each epoch, resulting in 10 checkpoints per fine-tuned model. the major challenge with both kb/bert and swectrl-mini is their limited and rather small context window size (512 tokens for kb/bert, and 256 tokens for swectrl-mini). at all times the context window should accommodate both the text and the mcq. to ensure that, we limited the number of text-related tokens to at most 𝐿 𝑇 tokens. the exact value of this limit was model-specific, namely 𝐿 𝑇 = 441 for kb/bert lrv, 𝐿 𝑇 = 384 for kb/bert aov-a and aov-b, and 𝐿 𝑇 = 192 for swectrl-mini. however, if the mcqs in the training data could not be fit in the remainder of the context window, we automatically decreased these limits 9 . at all times we ensure that the basis for the correct answer from the text (the information that is provided by both datasets) is included in the context window. for the ud-based baseline, we relied on the training set of swequad-mc for generating both qa-pairs 10 , and distractors 11 . for this work, we opted out of inducing such templates on quasi, because most of the texts in quasi are (partially) non-continuous texts. since quinductor was designed to work on single sentences from continuous texts, it is therefore unlikely that templates induced on quasi will end up being generalizable (or will be induced at all). the zero-shot swectrl-mini baseline did not require any specific further training (by the nature of being zero-shot). this brings the total number of models to ten. 9 for more information on this heuristic we refer to the source code accompanying the article. 10 using quinductor templates provided by @xcite in the associated github repository 11 using only raw texts from the training set of swequad-mc when evaluating, similarly to kalpakchi and boye (2023a), the longer the text 𝑇 , the more mcqs we attempted to generate. more specifically, we requested 𝑁 𝑇 𝑞 mcqs for each 𝑇 using the following formula: ## model selection the goal for this section is to select the best checkpoint for each of the eight fine-tuned models. since there are ten checkpoints per model, this step requires evaluating the quality of 80 checkpoints, which is prohibitively expensive to do using human evaluation. instead we resort to using metrics that could be calculated automatically. furthermore, since there is no reliable way to estimate how good the mcqs are using the automatic metrics, we aim at estimating how many definitely bad mcqs were generated by each checkpoint. to define definitely bad mcqs we employed the following badness metrics (listed from the most to the least severe). in the list below, "mcq%" means "percentage of mcqs", and "alts" means "alternatives" (the key and distractors together), whereas "alt" means "an alternative" (either the key or any distractor). for all of the metrics below, the lower, the better. we evaluated all checkpoints on the texts from the development set of swequad-mc. the texts that are larger than 𝐿 𝑇 tokens are split into chunks of max 𝐿 𝑇 tokens. for each model, we have generated 𝑁 𝑇 𝑞 mcqs, as calculated by equation 1 . because generating mcqs using models based on kb/bert is computationally heavy (and we need to generate mcqs for 80 checkpoints), we calculated the badness metrics only for the mcqs generated on the first 100 text chunks from the development set (when all the texts are sorted alphabetically). based on the badness metrics reported in figure 1 , we selected the checkpoint with the fewest and least severe errors for each model (recall that the introduced badness metrics are listed from the most to the least severe). additionally, everything else being the same, we preferred earlier checkpoints to reduce the risk of overfitting (given that the training sets were quite small, especially for the swectrl-based models). the selected checkpoints per model (denoted by the number of training epochs) are reported in ## human evaluation for human evaluation, we compare eight best checkpoints selected in section 6, two baseline models, and two gpt-based models, namely gpt-3 (text-davinci-003), and chatgpt (gpt-3.5-turbo-0301). in this section the evaluation begins with the following basic question for the set of produced mcqs per model per text: q0. did the model produce the requested number 𝑁 𝑇 𝑞 of mcqs? then each generated mcq is evaluated separately on the following aspects: q1. are the question (stem) and all alternatives grammatically correct? q2. is the stem answerable by the text? q3. are all alternatives relevant for the given stem? q4. is the alternative (a) the only correct answer? if the answer to q2 is no, we investigate the reasons behind the stems being unanswerable. following @xcite , we categorize such stems into contradictive (including presuppositions disagreeing with the text), undiscussed (inquiring about information not present in the text), or ambiguous (not providing enough information to choose one definite alternative). if the answer to q3 is no, we also investigate the reasons behind it. following kalpakchi and boye (2023a), we categorize the problematic alternatives into misfocused or heterogeneous. the former category means that at least one of the alternatives does not provide the type of information requested in the stem. for instance, the stem "when was alfred nobel born?" with one of the alternatives being "stockholm" is enough to classify such mcq as having misfocused alternatives. the latter category means that one or more of the provided alternatives "stick out" and could provide a metaclue to the students. for instance, the stem "when was alfred nobel born?" with the alternatives "21 october 1833", "1848", "1792", "1835" would be classified to have heterogeneous alternatives, since the first alternative (which happens to be the key) is more detailed than the others and thus "sticks out". additionally, we introduce two new categories: empty alternative(s) (when at least one of the generated alternatives is an empty string), or duplicate alternatives (when there are at least two identical alternatives, lowering the effective number of alternatives). if the answer to q4 is no, we look into three subquestions to understand why. if any sub-question gets a negative answer, we do not investigate the latter ones. the first sub-question is whether any alternative is the key (the correct answer), to begin with. the second subquestion is whether there is more than one alternative that could be considered to be the key, the case which is referred to as overlapping alternatives. the final sub-question is whether the only present key is actually the alternative (a). answering q1 -q4 required manual annotations, which we did ourselves using an iterative annotation process (annotating -discussing issues -reannotating). we used an instance of textinator @xcite as the annotation tool. the annotation process was blind, meaning that the generated mcqs and their texts were presented in a random order without any indication as to which model they were sampled from (all model-specific tokens and all question numbers were removed). after the evaluation was done, the separately generated key file (previously unseen by the annotator) was used to match the text annotations with their corresponding models. ## discussion we have observed a substantial performance difference between chatgpt (gpt-3.5-turbo-0301) and all other models for the mcq generation task. we conducted only very limited manual hyper-parameter tuning for the non-gpt models when fine-tuning, which means that there is a possibility that the performance of these models could be boosted further. however, as the datasets used for fine-tuning are quite small, it is unlikely that such a boost will be enough to reach the performance of chatgpt, which produces acceptable mcqs at a rate at least three times higher. additionally, one should bear in mind that the focus of this evaluation was mainly on uncovering foundational problems with the generated mcqs. this means that even mcqs judged to be acceptable might not necessarily be of sufficient quality to be useful in a high-stakes testing scenario (such as national exams). for instance, the following mcq produced by chatgpt on text 8 was marked as acceptable. vilken funktion hade tjejens rullstol? (which function did the girl's wheelchair have?) a) den var elektrisk (it was electrical) b) den var manuell (it was manual) c) den var gjord av trä (it was made of wood) d) den hade ingen funktion (it didn't have any function) while, formally, this mcq did not have any of the problems brought up in this work, it asks about a very minor detail. the type of the wheelchair is mentioned twice in the text but really has nothing to do with the plot of the story. while this mcq could be useful to test how attentive the reader was, it is not aimed at assessing reading comprehension skills but rather memory, which might also be desirable depending on the target audience for the test. furthermore, alternative (c) is arguably not plausible, since being made of wood is not a function. nevertheless, such an alternative might be deemed relevant depending on the target audience of the test (for instance, at the entrance-level classes). similarly, the use of "none of the above" alternatives, such as in the mcq below generated by chatgpt on text 7, might not necessarily be desirable. vad hoppas fredrika på angående sitt jobb? (what does fredrika hope for regarding her job?) a) att hon får ett jobb som känns meningsfullt, även om det inte är så bra betalt (that she will have a job that feels meaningful, even if it is not very well paid) b) att hon får ett jobb som ger henne hög lön (that she will have a high-paying job) c) att hon får ett jobb som hon inte behöver engagera sig i (that she will have a job that requires minimal effort) d) ingenting nämns om vad hon hoppas på angående sitt jobb (nothing is mentioned about what she hopes for regarding her job) in fact, @xcite report that opinions are split about using alternatives such as (d), which is why such mcqs might not necessarily be judged as acceptable in the larger-scale evaluation. ## conclusion in this article, we have compared the mcq-generating capabilities of 12 different models, eight models finetuned by ourselves, two baselines, as well as gpt-3 (text-davinci-003), and chatgpt (gpt-3.5-turbo-0301). the gpt-based models perform better than the rest of the models. chatgpt performs substantially better than all other models in all evaluation settings. gpt-3 performs substantially better than non-gpt models when tested on single-sentence texts, whereas the performance gap on the texts from sfi national tests (the plugga corpus) is less pronounced in comparison to other models (excluding chatgpt). additionally, we noticed that gpt-based models have an inductive bias for producing mcqs with four alternatives, even when the number of alternatives is unspecified in the prompt. in our limited experiments with the introduced grammatical errors, we noticed that gpt-based models avoid using anglicisms even if they were introduced in the original text, while other models tend to stick to the text much more frequently. the behavior of the models was less conclusive for the introduced ocr error, except for gpt-3 (which produced substantially fewer mcqs than requested though) and kb/bert lrv trained on quasi that did not use the formulation with the ocr error. at the same time, when sticking to the text was necessary, such as in the example with the alternative spellings of the capital of ukraine, most models used the spelling from the text with chatgpt, and some models based on kb/bert taking the lead. all models based on swectrl-mini were lagging behind and used the formulation that was more frequent in their training data (148 thousand occurrences of the word kiev vs 2052 occurrences of the word kyiv) 13 . in summary, all conducted evaluations point to the fact that the best model for generating mcqs in swedish is chatgpt, followed by gpt-3, followed by swectrl-mini fine-tuned on the swequad-mc corpus. the results based on the toy domain of singlesentence texts (ssts) closely resemble those on the larger-scale texts from the sfi national exam. this indicates that sst-based evaluation might be a viable lower-cost alternative to the full-scale human evaluation, and warrants more extensive studies on the strength of the correlation, and the extent of the time savings.