Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers (2022)

Abstract: This paper presents the first multi-objective transformer model for generating open cloze tests that exploits generation and discrimination capabilities to improve performance. Our model is further enhanced by tweaking its loss function and applying a post-processing re-ranking algorithm that improves overall test structure. Experiments using automatic and human evaluation show that our approach can achieve up to 82% accuracy according to experts, outperforming previous work and baselines. We also release a collection of high-quality open cloze tests along with sample system output and human annotations that can serve as a future benchmark.

## introduction open cloze @xcite tests are a common type of exercise where words are removed from a piece of text and must then be filled in by the students without any options to choose from. they are often used in language learning environments as a quick and effective way to test vocabulary, grammar and reading comprehension @xcite . however, designing high-quality cloze tests for language learning is a laborious process that involves finding an optimal distribution of gaps based on aspects such as function, distance, number of answers, etc. @xcite . in this paper, we propose a strategy to construct open cloze exercises using transformer models @xcite . our transformer-based architecture employs two objectives to predict the words that should be gapped in a text passage. our main objective is standard token classification, where we aim to minimise the error of classifying a token as a gap or not. the second and auxiliary objective is a language-model-based objective whereby we attempt to minimise the language model error when predicting the right answer for each gap. our solution is based on a pre-trained electra @xcite model that is finetuned on the two described objectives in a multitask scenario. our output aims to mimic the style of open cloze tests in the first certificate in english (fce) exam foot_0 , which is targeted at learners of english at the b2 proficiency level of the common european framework of reference (cefr) for languages @xcite . unlike other tests, the fce open cloze task aims to simultaneously test many aspects of grammar and vocabulary that students are expected to know at this level. since the tests are created from a text passage, they must be skilfully designed in order to ensure an optimal distribution of gaps that adheres to guidelines. a shortened example is shown in figure 1 . our system is evaluated under two settings: 1) automatic evaluation, where the generated gaps are compared to gold-standard gaps proposed by test experts, and 2) human evaluation, where the quality of the generated gaps is judged by test experts. the main contributions of our work are as follows: 1) we are the first to employ transformer models for open cloze test generation, 2) unlike previous studies, we work at the paragraph level, which is a much more challenging task, 3) we propose a multi-task learning approach with two objectives: one is to classify tokens into gaps/non-gaps and the other to minimise the error of re-generating the gapped word, 4) we report state-of-the-art results, outperforming previous work and a strong baseline, 5) we propose additional components to control the structure of the final cloze tests as human experts do, 6) we perform both automatic and human evaluation and 7) we make our test data, system output and human annotations available to the research community foot_1 . ## related work while research into automatic cloze test generation is vast @xcite @xcite , work on open cloze tests for language learning is scarce. @xcite generate open cloze questions using sample sentences from a learners' dictionary based on four linguistic criteria: (grammatical) complexity, welldefined context (collocations), grammaticality and length. a later version of their system adds hints for gapped words @xcite . exercise maker @xcite is a rule-based open source system that attempts to emulate exercises in cambridge english examinations based on the most frequently tested words. most of the gaps it proposes were found to be useful and the automated exercises were hard to differentiate from authentic tests. @xcite generate open cloze exercises for phrasal verbs by extracting sentences from news articles and generating a pair of questions and answers where the identified particle verbs are gapped. similarly, @xcite gap words in sentences according to their part of speech in order to practise articles, prepositions, etc. finally, @xcite use lstms to build sequence labelling and classification models that decide where to insert a single gap in a single sentence. automatic evaluation against goldstandard gaps showed the method was effective. other work has focused on creating automated cloze tests by controlling aspects of the proposed gaps so that they correlate with a target proficiency level. @xcite , for example, manipulate the difficulty of c-tests (open cloze tests with hints, @xcite ) by varying the position and word length of the gaps. a similar concept is presented by @xcite and @xcite , although difficulty is predicted using a machine-learning model that correlates with cefr levels. in these cases, tests are dynamically adapted to the examinee's proficiency level during the test session. from a different perspective, @xcite show that controlling gap entropy can be useful for designing open cloze tests at different cefr levels. the work we present in this paper, however, aims to model the more complex task of predicting a full set of gaps at the paragraph level that comply with design and testing principles and is, to the best of our knowledge, the first to employ and adapt transformer-based models for this task. system evaluation is also challenging, since there is usually more than one potential word in the text that could constitute a good gap. while previous work often made a choice between automatic @xcite or human evaluation @xcite for their experiments, we perform both: automatic evaluation to identify the best models during development and human evaluation to measure test quality in the final output. ## model we define open cloze generation as the task of predicting a set of tokens that should be gapped in the text. unlike previous approaches that work at the sentence level, our models work at the paragraph level (i.e. take the full text as input), since we believe the interactions between gaps can only be optimally captured when the text is processed as a whole rather than sentence by sentence. given a text passage, we aim to predict the words that should be gapped in order to create a cloze test that would reliably assess student ability. the task is modelled as a supervised sequence tagging problem where each token is classified as being a good potential gap or not. we employ electra @xcite , one of the state-of-the-art pre-trained transformer-based language representation models @xcite . electra is an extension of bert @xcite with a different pretraining task which is a discriminator (rather than a generator) and aims to detect replaced tokens (rather than generating words for the masks). we believe that this discrimination objective makes it more suitable for our token classification task. moreover, we also exploit electra's generation capabilities as a language model for estimating the answers to the proposed gaps as an auxiliary task. hence, to make the most of this pre-trained model, we fine-tune it using two training objectives, as depicted in figure 2 : 1 a token classification objective which aims to minimise the error of classifying each token as a potential gap or not. 2 a language modelling objective that aims to minimise the negative log-likelihood of regenerating the words that have been gapped. the first objective is typical of any standard token classification model and constitutes our key task. in particular, we use electra's discriminator head with softmax to tag each word in the input sequence as a 'good' gap or not. all the gaps in our training data are replaced with the first intended target answer and labelled positive, while the remaining tokens are labelled negative ( a ). the second and auxiliary objective attempts to model our preference for gaps with a restricted number of answers while also ensuring that the original word can be guessed from the context. this is to avoid generating gaps that are too 'open' and therefore ineffective, such as a gap that accepts any noun or adjective. specifically, we mask the words in the positions that are predicted as gaps by the discriminator and use electra's generative head to generate the expected words in the blanks ( b ). while the input layers are shared between the discriminator and the generator model, the two branches of the system leading to the two objectives are fine-tuned in parallel in a multi-task setting. ## extensions our neural transformer-based sequence tagging model can be very effective at proposing potentially good gaps, but the task becomes more challenging when we expect the output to meet additional requirements such as no repetitions, no gap interdependence, a minimum distance between gaps and a varied selection of lexico-grammatical items. we address these issues using two complementary strategies: a manipulation of the loss function and a post-processing module. ## data to the best of our knowledge, there are no public datasets of full-text open cloze tests that could be used for our task. the cloth dataset @xcite , for example, contains gapped passages designed for language learners, but it is primarily focused on reasoning and reading comprehension and uses multiple choice questions where distractors play a major role, making it substantially different to the task we aim to model. for this reason, we use a collection of expertly created open cloze tests at the b2 cefr level that was kindly provided by cambridge university press & assessment (cup&a) for research purposes. each task consists of a text passage of no more than 300 tokens, a variable number of gaps (between 8 and 16) and a list of valid answers for each gap (between 1 and 7). during the design process, the tasks undergo extensive quality control and pretesting, so their gaps are guaranteed to be very effective at assessing student ability. for training, we reconstruct the texts by replacing each gap with its first answer and we split the whole collection into train, dev and test. details of our dataset are shown in given the lack of publicly available data, we make our test set available with this paper so as to provide a common benchmark for the task and to encourage further research in this area. all the texts were tokenised and parsed using spacy v2.3 4 . along with all the tokens in the sequences, we also input dependency parsing information to the system. more specifically, we concatenate the electra representation of each token with the representation of its head in the dependency graph. 6 on top of the encoding layers, we have two branches that are being learned simultaneously (figure 2 ). the first branch is a simple linear layer that aims to classify each token as a gap or non-gap. for the second branch, we add electra's generation layer plus a linear layer which aims to predict the best word from the whole vocabulary as an auxiliary task. we are only interested in predicting the answer words for the gaps. therefore, we change the input to the second branch by masking the words that are predicted as gaps by the first branch at each step of training. we employ crossentropy loss on each branch and ignore the loss values for the tokens that are not masked in the second branch. the whole architecture is updated based on the sum of the two losses. fine-tuning parameters are specified in appendix b. ## conclusion and future work we described the first transformer-based approach to open cloze test generation. our electrabased model is trained on two objectives: token classification (gap/non-gap) and language modelling (for predicting the expected answer). the model is further improved by manipulating the loss function and post-processing the results. system accuracy using automatic evaluation is 53.89% while human evaluation ranges between 75% -82%, showing that at least 7 out of 10 gaps predicted are considered useful by experts. a detailed analysis of results reveals a few structural problems such as gaps in close proximity and inappropriate difficulty, which we plan to address in future work. our test data and human annotations are released with this paper.