QUA - RC : the semi-synthetic dataset of multiple choice questions for assessing reading comprehension in U krainian (2023)

Abstract: In this article we present the first dataset of multiple choice questions for assessing reading comprehension in Ukrainian. The dataset is based on the texts from the Ukrainian national tests for reading comprehension, and the MCQs themselves are created semi-automatically in three stages. The first stage was to use GPT-3 to generate the MCQs zero-shot, the second stage was to select MCQs of sufficient quality and revise the ones with minor errors, whereas the final stage was to expand the dataset with the MCQs written manually. The dataset is created by the Ukrainian language native speakers, one of whom is also a language teacher. The resulting corpus has slightly more than 900 MCQs, of which only 43 MCQs could be kept as they were generated by GPT-3.

## introduction assessing reading comprehension is of interest both for the native speakers of any language (for instance, through pisa (oecd, 2019) assessments), and for the foreigners learning the language (e.g., through ielts foot_0 for english, dele foot_1 for spanish, or delf foot_2 for french). in both cases the skills are frequently assessed on the same scale, namely the one proposed by the common european frame of @xcite ). one of the assessment formats recommended on any cefr-level is multiple choice questions (mcqs), which consist of the following components: the key and the distractors together are called alternatives. note that reading comprehension mcqs require carefully selected texts, which are absolutely crucial, since reading comprehension mcqs are not designed to stand on their own. in practice, the assessment with mcqs is rather popular because it enables fast, automatic, and thus objective grading. on the other hand, creating mcqs is comparatively slow and requires a lot of manual efforts, which motivated the research on nlp methods for generating mcqs automatically. as @xcite report, researchers have tried different techniques for mcq generation, ranging from the manually created pipelines to more recent methods based on learning from data. indeed, the introduction of large language models (llms), such as bert @xcite , or gpt-3 @xcite , resulted in new approaches being tested for many nlp tasks, not least for mcq generation, especially for english @xcite @xcite . by comparison, mcq generation problem (particularly for reading comprehension) received much less attention in other languages, and especially in ukrainian. in this work we aim to bridge the gap for ukrainian by making the following contributions: both the dataset, and the accompanying source code are available on github: https://github.com/ dkalpakchi/qua-rc . ## related work to the best of our knowledge there has been no prior work on creating datasets of mcqs specifically for ukrainian first, let alone semi-automatically. in parallel with this work @xcite have developed belebele benchmark where they have created a parallel reading comprehension dataset in 122 languages, with ukrainian being among them. the texts and mcqs in the dataset have been manually translated from english with reportedly rigorous curation process. the texts for this dataset were taken from three sources: wikinews, wikivoyage and wikibooks. by their nature, such texts contain mostly facts, lacking, for instance, literary devices or dialogues, and often appear additionally structured (compared to narrative texts) for ease of reading. moreover, the translations for the dataset were produced to maximize the alignment between 122 languages, which could lead to the increased use of translationese @xcite , as the authors themselves note. by contrast, the texts used in our dataset are taken directly from the ukrainian national tests for reading comprehension, meaning they are guaranteed to not contain translationese, and are considered to be of suitable quality by the experts. the translated datasets in ukrainian are scarce even when looking at the broader field of question answering. the only work that we are aware of is an attempt at translating the squad dataset @xcite to ukrainian foot_3 . however, it is unclear to what extent the translations have been curated, and the dataset contains no distractors (similar to the original squad). in general, the idea of creating synthetic qa datasets is not new, and has been rejuvenated by the advent of large language models (llms). for instance, @xcite produced synthetic questionanswer pairs by using three different bert @xcite ) models fine-tuned on squad2 @xcite to perform three different tasks: (1) extract the potential answer, (2) generate the question for that answer, and (3) answer this new question to check for the roundtrip consistency and filter-out the inconsistent questions. the idea of creating synthetic mcq datasets is not new either. for instance, @xcite generated mcqs using openai's gpt-3 @xcite in a zero-shot manner. after curating the output, 44% of mcqs turned out to be of acceptable quality. in this work we build on the work of @xcite and expand it in the following ways: additionally, in contrast to @xcite , we also attempt to revise the generated mcqs that did not meet the quality standards. furthermore, we expand the dataset with manually written mcqs, instead of relying entirely on the synthetically generated mcqs, thus taking a semi-automatic approach. we also conduct a pilot investigation and check to what extent the synthesised mcqs could inspire the creation of the new ones. ## data any mcq dataset for reading comprehension consists of the texts and mcqs based on these texts. the choice of texts is crucial in this endeavour as it partly defines what kinds of mcqs would appear in the dataset (e.g., those testing simple text scanning skills, or more advanced, asking the reader to compare or contrast). in this paper we took the texts from the ukrainian national tests in the ukrainian language and literature, which are part of the university admission exams in ukraine, called external independent evaluation, eie (ukr. "зовнiшнє незалежне оцiнювання, зно"). specifically, we took the texts from the "reading" section of the tests administered between 2007 and 2021 (the last year before the radical change of format). we have cleaned the texts by removing titles and/or subtitles of the original texts, numeration of the text parts, and other notes (e.g., names of the authors, number of the words included to the text). additionally, we have filtered out texts that included non-continuous elements (following the definition of oecd (2019), e.g., lists) or relied on images for the narration. furthermore, we were forced to split the vast majority of the texts into parts, which resulted in 62 excerpts from the 32 original texts. the reason behind the aforementioned splitting is illustrated by figure 3 , which shows that one word in ukrainian corresponded to between slightly less than 7 and 8.5 gpt-3 tokens, (in stark contrast to roughly 1.33 5 tokens per word for english). while the aforementioned problem is often solved using the sliding window approach, we would like to argue that it is not sufficient for this particular problem. the reason behind this is that the generated mcqs need to go beyond "local" factual questions about the information that is presented in a couple of sentences. indeed, we are also interested in the mcqs that test higher-order reading skills (e.g. making high-level inferences or drawing conclusions from a text), for instance, mcqs with such stems as "what is the main idea of the text?", "why did x do y and not z?", or 5 based on the information here: https://help.openai.com/en/articles/ 4936856-what-are-tokens-and-how-to-count-them "what is the relationship between x and y, according to the text?", which are prevalent in real-world reading comprehension tests. if we require a model to generate such stems in a reliable way, the whole text must be provided to a generation model. bearing in mind that we asked gpt-3 to generate 𝑁 𝑞 mcqs per text, we have empirically identified that the excerpts should be at most 250 words long to allow enough space for the mcqs themselves. additionally, we took only those excerpts which discuss a particular topic and/or convey a certain idea, so that each of them can be perceived as a standalone text. the extracted 62 excerpts are divided into the following three types: later in the article we will refer to these 62 excerpts as simply texts. ## method in this work we have investigated the three-stage semiautomatic approach to creating the mcq dataset. at the first stage, we have seeded gpt-3 with the following prompt in ukrainian in an attempt to synthesise to aid the reader, we supply the english translation of the prompt, although we stress again that the prompt was fed to gpt-3 in ukrainian. write 𝑁 𝑇 𝑞 different reading comprehension tasks for this text. in each task there should be one question, enumerated with arabic numbers (1, 2, 3 …). from these 𝑁 𝑇 𝑞 tasks, 𝑆 𝑇 2 contain two answer alternatives, 𝑆 𝑇 the number 𝑁 𝑇 𝑞 was calculated as follows: in equation 1 𝑊 𝑇 denotes the number of spaceseparated tokens in the text 𝑇 , and 𝑊 denotes the average number of space-separated tokens per text in the corpus. in this article we have empirically calculated 𝑊 = 14 based on the collected 62 texts. each 𝑆 𝑇 𝑥 is a string of the form "𝑁 𝑇 𝑥 <@xmath0, where 𝑁 𝑇 𝑥 is the requested number of mcqs with 𝑥 alternatives, @xmath1. should in this context) grammatically aligned with the number 𝑁 𝑇 𝑥 , which is calculated as follows: in equation 2, 1 𝑁 𝑇 𝑞 %3>4-𝑥 is an indicator function taking the value of 1 if 𝑁 𝑇 𝑞 % 3 > 4 -𝑥 holds, and 0 otherwise. since 𝑁 𝑇 𝑞 %3 ≤ 2, the aforementioned condition enables distributing the remainder 𝑁 𝑇 𝑞 % 3 roughly equally between 𝑁 𝑇 𝑥 , by first incrementing 𝑁 𝑇 4 , and then 𝑁 𝑇 3 . at the second stage we went through all synthesised mcqs and divided them into the following three types: for the sake of simplicity, we will refer to the discarded mcqs and the original mcqs behind the revised ones count type new inspired expanded as mcqs of insufficient quality. for these mcqs we have identified and categorised the problems causing their poor quality, as described in section 4.1. to re-iterate, the introduced revisions are meant to keep the original meaning of the stem and alternatives if it can be derived. if such revisions are impossible, we proceed to the next stage and create a new mcq. at the third stage we attempted to complete the dataset with manually written mcqs so that there are at least 𝑁 𝑇 𝑞 mcqs of sufficient quality for each text. to be more specific, this means that we needed to write at least max(0, 𝑁 𝑇 𝑞 -𝑁 𝑇 𝑘 -𝑁 𝑇 𝑟 ) for each text 𝑇 . here we differentiate between three types of mcqs: formally, the goal of this stage is that for every text 𝑇 the following inequality holds: ## evaluation to categorise the mcqs as outlined in section 4, we have manually annotated all mcqs generated by gpt-3. the annotations of the generated mcqs were performed by the first author of this paper who has background in teaching. however, we followed an iterative annotation process (annotating -discussing issues -re-annotating) with both authors (native speakers of ukrainian) contributing to the discussion and reannotation. the manually added mcqs, created by the first author, were mostly annotated by the second author (although even here we followed the very same iterative annotation process). both kinds of annotations were performed using the textinator annotation tool @xcite . the results of the annotations for the second stage described in section 4 are presented in figure 4 . as can be seen from the figure, gpt-3 has produced the required 𝑁 𝑇 𝑞 mcqs only for slightly less than half of the texts (30 out of 62 texts). interestingly, although in line with findings of @xcite , for slightly more than half of cases where gpt-3 did not reach 𝑁 𝑇 𝑞 (18 out of 32) the generation was stopped because of reaching the stop token (hatched bars in figure 4 ), and not the maximum number of tokens, meaning more mcqs could potentially be generated for 18 texts. the number of mcqs that could be kept as they are (green in figure 4 ) is very low, only 43 of 525 mcqs, and is distributed unequally among the texts. the number of mcqs that could be revised (orange in figure 4 ) also differs substantially between the texts. in total for 36 texts (58% of texts) the number of discarded mcqs is larger or equal to the number of kept and revised ones together. this observation reveals a substantial problem with using gpt-3 for generating mcqs in ukrainian, since discarded mcqs are those that could not be revised without re-writing the major parts of the mcq. recall that we have also requested different number of mcqs with two, three, and four alternatives, attempting to keep each number roughly equal to one third of 𝑁 𝑇 𝑞 . figure 6 shows the distribution of the number of alternatives for the generated mcqs per text. as can be seen, gpt-3 failed to meet the aforementioned request for all texts. for some texts gpt-3 has also generated mcqs with only one alternative, or even no alternatives at all. most frequently, gpt-3 generated mcqs with either two or four alternatives, with three alternatives being very rare. this suggests that gpt-3 might have an inductive bias towards generating two or four alternatives (as such cases might have been much more frequent in its training data). additionally, we note that most of the kept mcqs (green in figure 6 ) had only two alternatives (which often were of yes/no type), of which only one was a distractor. in an attempt to reach 𝑁 𝑇 𝑞 mcqs per text we have proceeded to the third stage described in section 4, which is summarised in figure 5 . note that for all texts where gpt-3 stopped generating mcqs of its own accord we could manually add the required number of mcqs (and beyond that). for 9 texts most of the newly added mcqs were in fact inspired by the deficient ones produced by the gpt-3. this indicates that mcqs produced by the gpt-3 could potentially be used as an inspiration for the mcqs rather than blindly relied upon. at the same time, we note that for 10 texts none of the mcqs produced by gpt-3 provided the inspiration for the new mcqs (entirely blue bars in figure 5 ). in total, our efforts on correcting the generated mcqs and adding the new ones resulted in expanding the dataset from 43 automatically generated mcqs that could be kept as they are, to 926 mcqs. observe that mcqs with the same stem but with the alternatives of different types are counted as different mcqs. to exemplify, consider the stem "who wrote the stories about hercule poirot?", and the following three sets of alternatives: (1) agatha christie, arthur conan doyle; for this example the exact translations do not matter but note the word "вiдповiдь" (eng. answer) that is present in both mcqs. in the first mcq it comes after the four alternatives and provides the correct answer (not following the request in the prompt of simply making the correct answer the second one). in the second mcq, this word comes before the alternatives and is absolutely redundant. while such formatting inaccuracies might seem minor, they impede fully automatic processing of mcqs, i.e. getting the stem, the key and each distractor as separate strings. the second category of problems by severity is language errors (light orange bars in figure 7 ). observe that fixable grammatical errors in the stem and the alternatives belong to the top three most frequent errors, accompanied by the lexical errors in the stem. one interesting kind of grammatical errors made by gpt-3 is introduced by the use of anglicisms, where words or phrases are translated word-by-word from english, as in the stem below: у якiй мовi вона робить записи українських пiсень? (in what language does she record ukrainian songs?) here the beginning of the stem "у якiй мовi" is a word-by-word mapping of the english in what language, whereas the correct phrase in ukrainian contains only two words, namely "якою мовою". another example which is likely an anglicism concerns capitalisation of nationalities, as in the stem below: що пропагував французький лiтературознавець жуль ренар? (what did the french literary critic jules renard promote?) here the capitalisation of the nationality french is transferred to the stem in ukrainian as "французький", although nationalities must not be capitalised in ukrainian. these two small examples suggest that gpt-3 might be prone to translationese @xcite and use the direct translations of phrases from english. this hypothesis seems plausible given that texts in english constituted 92.64% of the training data of gpt-3, whereas texts in ukrainian constituted only 0.00763% foot_4 . however, further investigations on the matter are required. an interesting example of lexical errors are russianisms, for instance, as in the stem below: як ван гог вiдносився до своєї працi? (how did van gogh relate to his work?) here the ukrainian word "вiдносився" ('vidnosyvsja', eng. related) is likely taken from the russian "относился" ('otnosilsja', eng. treated), whereas the correct verb in ukrainian is "ставився" ('stavyvsja', eng. treated). this suggests that the ukrainian texts that were included in the training data of gpt-3 have not necessarily been lexically correct to the fullest extent, something that should be investigated further. the third category of problems by severity is semantic errors (dark orange bars in figure 7 ). here the three most frequent errors are all stem-related, namely ambiguous formulation, misleading grammatical errors and too literal text interpretation. the last one is especially interesting, since one of the motivations behind the use of large language models is exactly to avoid such cases. to exemplify, consider the following mcq generated by gpt-3: text: галина бабiй, радiожурналiст: буваючи на вiдпочинку чи у вiдрядженнях за кордоном, зауважила, що вiльний обмiн книжками там дуже поширений. [...] (halyna babiy, radio journalist: while on vacation or on a business trip abroad, i noticed that the free exchange of books is very common there. […]) в якому мiсцi галина бабiй зауважила поширення обмiну книжками? (in which place did halyna babiy notice the spread of book exchange?) a) у вiдпочинку (in vacation) b) за кордоном (abroad) observe that the mcq is based on the single provided sentence which itself does not point to a specific place but rather to a situation (on vacation or on a business trip). hence asking "in what place" is inappropriate in these circumstances, let alone the fact that this detail is very minor and is unlikely to be asked in a real-world reading comprehension test. ## discussion the presented dataset of mcqs is created semiautomatically and has its own limitations regarding both the automatic and the manual parts. one limitation that concerns both parts is that the dataset follows no particular principles for ordering either the mcqs for each text, or the alternatives within one mcq. while any or both of these could play a role in a real-life testing scenario, we are unaware of any systematic investigation on this matter. furthermore, any such investigation would be constrained to the particular groups of students, something that is beyond our control in this work. hence, all alternatives in our dataset are presented in a random order. regarding the automatic part, as we have previously discussed, gpt-3 seems to have a number of problems related to translationese, i.e. applying the phrases or grammar rules of other languages (most notably, english and russian) to ukrainian. bearing in mind that ukrainian texts constituted only a tiny part of the training data of gpt-3 (0.00763%), such finding is to be expected. avoiding such problems is one of the strongest arguments either for language models trained specifically for ukrainian language, or on multilingual language models, where texts in all languages are represented equally. that said, we do not believe that fine-tuning gpt-3 on ukrainian texts is a feasible way forward due to multiple reasons. first and foremost, to the best of our knowledge, it is currently impossible to estimate what quantity of texts would be enough to reach increase in the model's performance. secondly, we believe that fine-tuning for this task would require high quality texts in ukrainian, which are not readily available copyright-free. lastly, it is very likely that the amount of texts in english in the training data of gpt-3 is higher than all (copyright-free) texts in ukrainian we will be able to find. all of these arguments together with the cost of fine-tuning gpt-3, and potentially maintaining access to the fine-tuned version for the general public, make such approach practically infeasible for this research. another discovery worth further investigation concerns the cases where gpt-3 failed to identify the boundary between the prompt and the supplied text. in our investigation, this manifested itself as mcqs asking about the details of the prompt rather than about the content of the supplied text. our suggestion is to conduct a systematic investigation on whether such problem occurs across languages and language models (and, ideally, also across nlp tasks). we have also noticed that gpt-3 did not succeed in "decoding" literary devices (e.g., metaphors, rhetorical figures) and phraseological units, as in the mcq below: text: але ретельнi рентгенiвськi дослiдження засвiдчили, що всi роботи митця написанi зi «швидкiстю виконання й без вагань», «на одному подиховi». [...] (but careful x-ray studies proved that all the works of the artist were written with "speed of execution and without hesitation", "in one breath". […]) на якому подиховi були написанi всi роботи в. ван гога? (in what breath were all the works of v. van gogh written?) here in one breath is a phraseological unit with a stable meaning of "very quickly, without difficulties", and its component parts cannot be separated from each other. below there is an example of the mcq which contains a metaphor: text: квiтка вступила до нью-йоркської консерваторiї. оперне майбутнє не склалося, а її американською «дiйснiстю» стає... рекламний конвеєр, i ось вонацей янгол -спiває дивним тембром сто мiльйонiв разiв якiсь «трелi»-заставки для кока-коли. було в її кар'єрi й залучення до «великого» кiно. але це так -мимохiдь -так i не розквiтла для «оскара». але родичi чiтко усвiдомили: призначення цього херувима не кока-кола, а щось неземне. [...] (kvitka entered the new york conservatory. the future in the opera did not materialise, and her american "reality" becomes… an advertising conveyor belt, and here she -this angel -is for a hundred million times singing some "trills"screensavers for coca-cola -in a strange timbre. her career also involved "big" movies. it was, though, very circumstantial, and she never blossomed for "oscar". however, her relatives clearly understood: the destination of this cherub is not coca-cola ads, but something otherworldly. […]) що було призначенням херувима квiтки? (what was the destination of the kvitka's cherub? or what was the destination of kvitka, the cherub?) a) кока-кола (coca-cola) б) щось неземне (something otherwordly) here the cherub is a metaphor to describe kvitka herself, and not her property; neither was kvitka a real cherub -something that gpt-3 did not manage to catch. we note that the aforementioned performance problems were documented when we tested gpt-3 in a zero-shot way (e.g., just a prompt with a task specification, without any examples). it is theoretically possible that giving some examples of texts and mcqs for these texts (i.e. formulating the problem as few-shot) could bring more mcqs of sufficient quality. however, in practice, given that one word in ukrainian amounts together with that, if the text itself focuses on describing what is not happening (e.g., factors which do not cause a certain disease) and the stem requires to name the opposite (e.g., what can cause the named disease), such mcq is also considered obverse. however, if a stem retains the negation which is already stated in the text (e.g., still asking which factors cannot cause the named disease, while the factors are mentioned in the text as those not leading to the disease), such mcq is not considered obverse. our goal was to investigate whether it was possible to manually create at least one mcq for each of the aforementioned missing mcq types. the results of the aforementioned endeavour are presented in figure 8 . as can be seen, we could create an mcq of at least one missing type for each text. at the same time, only 13 out of 62 texts received at least one mcq of all missing types. this shows that not every kind of text could provide grounds for every missing mcq type. for instance, the obverse mcqs could be created for the vast majority of the texts, since it is usually enough to have a single fact (which are usually abundant in the texts of various genres) for such mcq. on the contrary, applicative mcqs require the text to include some kind of knowledge that can be applied, which, for instance, immediately excludes vast majority of the biographies and descriptive texts. similarly, relational mcqs can not be written for each and every text, since they require at least two objects or concepts that could be compared/contrasted. additionally, shorter texts (especially those consisting of only a couple of sentences) tend to give less opportunities for these kinds of mcqs. in addition to the mcq types mentioned above, there are also so-called tabular questions. these mcqs are associated with the texts that contain information that could have been arranged in a table. for instance, one of the texts in our dataset describes the history of comic books and includes information about the names of the comic books in different countries. such information could be represented as a table with the name of the country in one column and the corresponding name of the comic book in another. a tabular mcq from our dataset for this text is: у якому рядку правильно визначено вiдповiднiсть мiж країною походження та термiном, що використовується? (in which line the correspondence between the country and the used term is correctly specified?) сша -«комiкс», францiя -«мальованi iсторiї», японiя -«манга», україна -«стрiчка малюнкiв» (the us -"comics", france -"drawn stories", japan -"manga", ukraine -"picture tape") сша -«комiкс», францiя -«стрiчка малюнкiв», японiя -«манга», україна -«мальованi iсторiї» (the us -"comics", france -"picture tape", japan -"manga", ukraine -"drawn stories") сша -«стрiчка малюнкiв», францiя -«комiкс», японiя -«манга», україна -«мальованi iсторiї» (the us -"picture tape", france -"comics", japan -"manga", ukraine -"drawn stories") сша -«мальованi iсторiї», францiя -«комiкс», японiя -«манга», україна -«стрiчка малюнкiв» (the us -"drawn stories", france -"comics", japan -"manga", ukraine -"picture tape") the notable feature of tabular mcqs is that one could create many mcqs by simply varying the number of items in each alternative (the number of countries in this example), and matching the values of different columns in various ways (grouping the country name with its name of the comic books in this exmaple). in our dataset, we tend to keep only a few examples of tabular mcqs, without providing its all possible variations. similarly, mcqs which include names and digits are mainly represented in one variation only, while the names can often be altered from the full to the shortened ones (or vice versa), including those with only first letters of the name left, and the numbers can be represented in digits or in words. an opposite kind of mcqs with no variation in alternatives, are the yes/no mcqs, which most frequently contain only two alternatives ("yes" and "no"), sometimes more (e.g., including "maybe"). among mcqs generated by gpt-3, only 21 were of such type, of which only one was kept as it was and nine were revised. for such mcqs, the set of alternatives is always fixed. the same concerns the stems like "what is the theme of the text?" where the stem is fixed, while the alternatives change with each new text. writing such mcqs equates to constructing only a stem or only a set of alternatives which makes the process faster but of the lower priority for automation. taking that into account, we did not add such mcqs manually. another transformation that could expand the dataset is the use of synonymous reformulations or paraphrases of the given stems, which might potentially allow to manipulate difficulty of the mcqs but where an extensive coverage is hardly reachable. mainly focused on covering content-related aspects, we leave vocabulary alterations and difficulty evaluation for the future work. however, already from the mcqs included into this dataset (from those created both automatically and manually), we have noticed that their difficulty might vary depending on a personality-based factor (oecd, 2019) of previous knowledge -the knowledge a reader already had before beginning to read the given text. since such factor can hardly be controlled, it can be tricky to judge whether an mcq is suitable for testing the reader's reading skills rather than their previous knowledge. for instance, mcqs which are to some extent based on the so-called "common knowledge" may require making complex inferences with respect to the text but become absolutely trivial for people with the relevant previous knowledge. we noticed that the particular kinds of previous knowledge for which it is true are stable facts (those that could be verified from multiple credible sources and are not based on opinions), meanings of idioms, and definitions of terms. to exemplify further, consider the following text and an mcq: text: микола леонтович збирав народнi пiснi й адаптував їх для хорового спiву. [...] композитор, як рiзьбяр, зробив навколо основної поспiвочки витончену оправу. поєднавши прийоми народного багатоголосся з досягненням класичної полiфонiї, вiн домiгся того, що кожен голос почав вiдiгравати самостiйну роль, вiдтворюючи найтоншi змiни настрою. леонтович кiлька разiв переробляв твiр, аж поки 1916 року не створив досконалий хорал. (mykola leontovych collected folk songs and adapted them for choral singing. […] the composer, like a carver, made an elegant frame around the main song. by combining the techniques of folk polyphony with the achievements of classical polyphony, he made each voice play an independent role, reproducing the most subtle mood changes. leontovych revised the work several times until the year 1916, when he managed to create a perfect chorale.) here m.leonotych's profession might be a completely unknown fact for some people while they are able to infer the information from the text. however, students of a music school or students with broad knowledge in arts and/or ukrainian culture are likely to answer this mcq without even reading the text. another example of an mcq which is potentially answerable without reading the text is presented below: text: практика надання допомоги безпритульним тваринам сягає xvii ст. саме 1695 р. в японiї, у мiстi едо (нинi токiо), з'явився перший (з вiдомих нам) притулок для собак. [...] (the practice of helping homeless animals dates back to the 17th century. it was in 1695 in japan, in the city of edo (now tokyo), that the first (we know about) shelter for dogs appeared. […]) яке мiсто мало назву едо? (which city used to be named edo?) a) токiо (tokyo) б) львiв (lviv) в) київ (kyiv) г) кiото (kyoto) here students might be completely unaware of the first name of the city of tokyo; however, it is a stable reallife fact which a student can know from school subjects (e.g., geography, arts) or other sources not related to the reading material used for a reading comprehension test. in practice it means that no pre-generated set of mcqs can be blindly taken as it is for real-life learning and is still to be verified by a person (likely, a teacher) who knows their target audience, peculiarities of the learning process of this audience, exact objectives of a test, and so on. ## conclusions despite the format of mcqs being widely used in the ukrainian educational system, specifically for reading comprehension tests as part of the university admission exams, automatic generation of these questions in ukrainian has not been introduced yet. inspired by what has been achieved in the nlp field for other languages, we created a semi-synthetic mcq dataset for reading comprehension in ukrainian which can be used as training or evaluation data for models specialising in mcq generation and answering. as expected, to achieve the sufficient quality of the dataset, manual editing was necessary to fix the errors made by gpt-3 and diversify the whole dataset by creating additional mcqs. however, the extent to which human assistance appeared necessary is surprisingly high -more than 90 per cent of the generated mcqs. the faults by the model are named and additionally categorised according to their impact on further revision. we also note that prompted to generate mcqs with a different number of alternatives, gpt-3 failed to meet the request, which means that this aspect of generating tasks in the multiple-choice format appears to be hardly controllable in the zero-shot scenario, which is a likely way real-world teachers would interact with such models. additionally, we found that the effective context window size for ukrainian is much smaller than 4096 1.33 words, since one word in ukrainian is roughly equal to 8 tokens of gpt-3. such limitation prevented us from generating mcqs on real-length reading comprehension texts, and calls for development of models with tokenisers keeping the token-to-word ratio closer to 1. that given, this particular model, gpt-3 (as of july 2023), does not seem to be appropriate for reading comprehension mcqs generation in ukrainian. however, more tests with other models and languages are needed to determine the extent to which llms can be used as a helpful generative tool for the stated task.