Quasi: a synthetic Question-Answering dataset in S wedish using GPT -3 and zero-shot learning (2023)

Abstract: This paper describes the creation and evaluation of a synthetic dataset of Swedish multiple-choice questions (MCQs) for reading comprehension using GPT-3. Although GPT-3 is trained mostly on English data, with only 0.11% of Swedish texts in its training material, the model still managed to generate MCQs in Swedish. About 44% of the generated MCQs turned out to be of sufficient quality, i.e. they were grammatically correct and relevant, with exactly one answer alternative being correct and the others being plausible but wrong. We provide a detailed analysis of the errors and shortcomings of the rejected MCQs, as well an analysis of the level of difficulty of the accepted MCQs. In addition to giving insights into GPT-3, the synthetic dataset could be used for training and evaluation of special-purpose MCQ-generating models.

## introduction openai's gpt-3 @xcite is the current state-of-the-art model for text generation. one of the more impressive properties of this model is the way it can perform natural-language tasks without any labeled examples, with so-called zero-shot learning. in the case of gpt-3, zero-shot learning entails that the model receives a prompt describing the task verbally (for example, "translate from english to spanish"), some input data for the task (a text in english in this example), and then produces the output (the spanish translation in the example). most research using gpt-3 has focused on english, because the bulk of gpt-3's training data (92.6% of words) is english text. only 0.11% 1 1 hyperlink to a csv file with the training data statistics of the training data is swedish text. this might sound insignificant at first, but it actually amounts to 220.9 million words, which is quite a sizeable corpus! in addition, swedish and english are both germanic languages, so it is possible that some cross-lingual learning has taken place during training. taking all this into account, we want to test whether gpt-3 would be able to handle tasks in swedish in a zero-shot fashion. specifically, the article has the following two goals. 1. provide a pilot evaluation of gpt-3's ability to generate multiple-choice questions (mcq) in a zero-shot manner. an mcq consists of a text, a question (called the stem) on the text, and a set of answer alternatives, of which exactly one is correct (called the key) and all the others are wrong, but plausible (called distractors). as we will show, gpt3 is good but far from perfect in generating swedish mcqs from a swedish text: more than half of the generated mcqs were incorrect, sometimes in subtle ways. this means that gpt-3 does not provide an ultimate solution to the mcq-generation task, and that special-purpose models are still required. the synthetic dataset presented here could potentially be used as extra training material for such specialpurpose models. in this paper, we provide a detailed analysis of the errors and shortcomings of the rejected mcqs, as well as an analysis of the level of difficulty of the accepted mcqs, giving insights into the strengths and weaknesses of gpt-3. ## related work the idea of creating synthetic datasets is not new both in nlp in general @xcite , and for question answering (qa) specifically @xcite . to the best of our knowledge, no synthetic datasets of multiple choice questions (mcqs) for testing reading comprehension have been created either for english or swedish. however, for english, the use of synthetic mcqs has been explored for other domains, such as natural sciences @xcite or factual qa @xcite . qa for swedish is an under-researched area with very few existing datasets. there has been an attempt to translate squad @xcite , which does not contain mcqs, into swedish foot_1 with no information on whether translations were manually checked. to the best of our knowledge, the only existing mcq dataset in swedish is swequad-mc @xcite , which has been manually constructed. ## evaluation methodology we are interested to know how well gpt-3 followed the instructions given in our prompt. for each given text, we investigated the following properties: q1. were n q mcqs generated? q2. did every mcq include a stem and 4 alternatives? q3. did the formatting conform to the requested one (mcqs are numbered, alternatives are labeled with letters a, b, c, d, etc)? q4. were all mcqs distinct? the next group of questions is interesting only for distinct mcqs with a stem and 4 alternatives. we will refer to these as d-questions, with "d" for "distinct". d1. were all stems grammatically correct and answerable after reading the text? d2. for mcqs having stems compliant with the requirements in d1, were all alternatives grammatically correct and relevant? the final 3 questions are interesting only for those cases where the answer was yes for both d1 and d2. we will refer to these as r-questions, with "r" for "relevant". r1. was only one alternative always correct, while the others were always wrong, but plausible? r2. was the correct alternative always a? r3. were the mcqs always ordered from the easiest to the hardest? although requiring some manual annotation, the questions above are all trivial to check, with the exception of r3, which is non-trivial since the concept of mcq difficulty is not well-defined. in fact, mcq difficulty depends on many things that are hard to keep constant, e.g., the reader's skills and background knowledge, whether the test is taken under time pressure, etc. for the purpose of this case study, we have relied on a definition of difficulty outlined in the section below and further detailed in appendices c.1, c.2, and c.3. ## results recall that we collected 96 texts and asked gpt-3 to generate n q mcqs for each of them, where n q is calculated based on the length of each text (the longer the text, the more mcqs we asked for). in total, gpt-3 made 718 generation attempts. to answer all questions posed in the previous section, we have made all required manual annotations ourselves using an iterative annotation process (annotating -discussing issues -reannotating). all annotations for this section have been performed using the textinator foot_3 annotation tool @xcite . q1. were n q mcqs always generated? answer: no, but very often (for 89.6% of the texts) for 86 out of 96 texts, gpt-3 generated exactly the requested n q mcqs. the mismatch between as can be seen, most of the mismatch happens for longer texts and there are mostly fewer mcqs generated than requested. one possible explanation could have been that gpt-3 simply did not have enough tokens in its context window. however, figure 1 illustrates that in the vast majority of cases, gpt-3 stopped generating mcqs after reaching the stop token. in fact, only in one case was the generation interrupted because the context window was too short (gpt-3 failed even to produce a stem for this example). this means that 717 out of 718 generation attempts resulted in an mcq. ## discussion and conclusions gpt-3 has been able to generate around 30% of mcqs that conformed to all criteria (excluding ordering by difficulty), and 44% of mcqs which were of sufficient quality (also excluding the requirement that a is the correct answer). together with our additional annotations, detailed with examples in appendix e, these 44% of mcqs constitute quasi, the first synthetic dataset of mcqs for testing reading comprehension of adult language learners of swedish (available at https: //github.com/dkalpakchi/quasi/ blob/main/annotated/quasi.json). the fact that 44% of mcqs turned out to be of sufficient quality is impressive, given that (a) it is zero-shot, and (b) only 0.11% of gpt-3's training data was in swedish. although gpt-3 did not manage to order mcqs from easiest to hardest for most of the texts, the model could still generate mcqs of varying difficulty levels. the easiest mcq scored theoretically minimal 3 points, whereas the hardest scored 14 (just 1 off from the theoretical maximum!). that said, as 56% of mcqs turned out to be of insufficient quality (available at https:// github.com/dkalpakchi/quasi/blob/ main/annotated/poor_quality.json), sometimes for subtle reasons, manual curation is not only desired, but is in fact required (not least to identify the correct alternative). why not ask gpt-3 to choose the correct alternative? one counter-argument is that it would consume more tokens, which leaves less tokens for mcqs, and leads to higher costs. another reason is that there is no convincing argument why gpt-3 would be able to always provide the correct answer. if it could, then it should have been able to put it as alternative a all the time, which it did not. furthermore, it should have been able to always generate only one correct answer, which it did not do for 16.6% of mcqs. in fact, this finding is in line with the previously published evaluation of a bert-based model for generating distractors in swedish @xcite , where the most frequent reason for rejecting distractors was that they were not wrong (leading to more than 1 correct answer). could gpt-3 handle ocr errors, if there were any? yes, it could! to give an example, one of the e-mail addresses in one of the texts was incorrectly recognized by the ocr system as "ifhs.info qimh.se", which we unfortunately didn't notice. gpt-3 was still able to generate "ifhs.info@imh.se" as one of the alternatives. this is most probably, because there was "epost:" (eng. "e-mail:") before this string, which the gpt-3's attention mechanism was able to capture. that said we didn't do any rigorous evaluation to quantify how well gpt-3 can mitigate ocr errors, so the caution is advised when trying to generalize from this insight. choosing α is a bit tricky. in reality one can ask way more than 1 mcq per sentence, but then not all sentences are worth asking even 1 mcq. in hopes that these two groups cancel each other out, we have assumed α = 1, meaning 1 mcq per sentence, for the purpose of this article.