Paragraph Similarity Matches for Generating Multiple-choice Test Items (2021)

Abstract: Multiple-choice questions (MCQs) are widely used in knowledge assessment in educational institutions, during work interviews, in entertainment quizzes and games. Although the research on the automatic or semi-automatic generation of multiple-choice test items has been conducted since the beginning of this millennium, most approaches focus on generating questions from a single sentence. In this research, a state-of-the-art method of creating questions based on multiple sentences is introduced. It was inspired by semantic similarity matches used in the translation memory component of translation management systems. The performance of two deep learning algorithms, doc2vec and SBERT, is compared for the paragraph similarity task. The experiments are performed on the ad-hoc corpus within the EU domain. For the automatic evaluation, a smaller corpus of manually selected matching paragraphs has been compiled. The results prove the good performance of Sentence Embeddings for the given task.

## introduction the multiple-choice test items (mcqs) are frequently used for knowledge assessment, e.g., ongoing assessment or during an examination. they are also used for career-significant assessment in medical training and certification, e.g., the medical college admission test (mcat) or the united states medical licensing examination (usmle) as well as in law schools and as a part of the multistate bar examination (mbe). some employers also use mcqs to evaluate the knowledge of candidates applying for a job since it is an easy and fast way to do so. tests of this kind are also extremely popular in game shows like who wants to be a millionaire? and alike, in myriads of mobile apps or facebook quizzes. moreover, the covid-19 pandemic has already affected significantly the way assessment is conducted in education institutions switching to extensive use of technology. mcqs are recommended to be used for online education, and many virtual learning environments offer in-built tools for the composition of such tests @xcite . for our research, we decided to experiment with the eu law domain. mcqs are used to assess knowledge of job seekers applying for positions in in the european union institutions. furthermore, mcqs are frequently used during exams in law schools and universities. therefore, the primary reason for selecting this domain is our motivation to assist education professionals and offer a helpful tool for knowledge assessment. we conduct the experiments on eu textbooks used in classrooms. despite the popularity and extensive use of multiple-choice tests, their manual creation is a laborious and time-consuming task @xcite . ambiguously worded questions, too easy or too difficult distractors can lead to the poor performance of testtakers and misleading results. there have been several attempts to automatically generate multiple-choice questions starting from various nlp techniques, including the use of wordnet, shallow parsing, corpora, ontologies @xcite . for the last couple of years, researchers have been experimenting with machine learning @xcite . in recent papers on this topic, deep neural networks are successfully applied for this task @xcite . furthermore, various strategies have been used for choosing distractors or testing on various types of instructive material (such as linguistics @xcite or medical texts @xcite , sports domain @xcite , as well as wikipedia articles @xcite . most research papers are based on the sentencebased approach, where questions are generated from one sentence that contains a key term @xcite . in recent research, the questions are based on the whole paragraph with the use of deep neural models @xcite . we aim to evaluate the performance of two existing deep neural models, doc2vec and sbert, for the generation of multiple-choice test items based on multiple sentences. our contributions are as follows: the mcq corpus is needed for the experiments. it should contain the question-answer (qa) pair and the paragraph on which the question is based. in addition, we aim to create the corpus that can be used for knowledge assessment of eu law. therefore, the qas should follow the criteria of mcqs used in classroom, i.e. they should not focus on too narrow topics or specific cases. in addition, we want to expand this research in the future by generating distractors automatically. that is why the answer should be concise. the requirements of mcq corpus design are listed in section 3.1. more detailed description of the corpus and experiments can be found in the master's dissertation @xcite . ## related work early research was focused on the use of traditional natural language processing methods for generating questions and distractors automatically. for example, @xcite , who were the pioneers in generating mcqs automatically, used such nlp techniques as term extraction, parsing, a corpus, and wordnet. their system-generated test questions and distractors based on a linguistics textbook. a few years later, @xcite generated multiple-choice test items from medical text using rapid item generation (rig) and the umls thesaurus. a medical textbook served as the source texts, while a much more extensive collection of medline texts was used as the reference corpus. in another research, @xcite employed various nlp techniques such as automatic term extraction, shallow parsing, sentence transformation and computing of semantic distance as well as corpora and ontologies. in contrast to the methodologies that highly depend on the used domain, @xcite approach is domainindependent as they employed specific ontologybased strategies and owl. singh @xcite proposed selecting sentences using existing test items in the web as well as presented a technique for creating named entity distractors from wikipedia. @xcite used owl ontologies to generate multiplechoice test items and proposed a psychologicallybased theory to control the question difficulty. @xcite suggested an unsupervised dependency-based approach to identify the most important named entities and terms and define semantic relations between them. this approach did not use any prior knowledge about the semantic types of the relations but was based on a dependency tree model. the results were evaluated in respect of their readability, usefulness of semantic relations, relevance, acceptability of questions and distractors and general usability of multiple-choice test items. a couple of years ago, the focus of the research community shifted towards the use of neural networks for natural language processing tasks. for example, @xcite investigated how machine-learning models, in particular featurebased and neural net (nn) based ranking models, can be used for distractor selection. @xcite proposed a hierarchical encoder-decoder framework to generate question items for reading comprehension questions from real examinations. @xcite investigated methods for automatically generating distractors for multiplechoice questions on english vocabulary and used semantic similarity and collocation information. @xcite used a topic modeling procedure, machine learning, and natural language processing to generate distractors based on students' misconceptions. in the aforementioned words, various approaches were used to generate questions or distractors automatically. mostly, the questions were generated on the basis of a single sentence. in our research, we do not create questions automatically. instead, we employ the most recent methods of semantic textual similarity matching to find similar paragraphs in the reference corpus. the question-answer pair attached to the incoming paragraph that is fed to the model is transferred to the newly found paragraph and then post-edited to create a new question-answer pair. this approach aims at reducing the teachers' effort to create new test items from scratch. ## methodology this research was inspired by translation memory (tm) matching used in computer-assisted translation (cat) tools. tm consists of aligned on the sentence level source-and-target pairs @xcite . these are the previous translations done by a human translator on the same or similar topic. basically, after the source text is segmented according to segmentation rules, usually according to sentence-closing punctuation marks, the source sentence is searched in the translation memory. when a similar sentence in the source language is found, its translation is suggested for the target segment. the translation memory matches are retrieved according to a certain threshold chosen by a translator based on how similar the segments are. this score is normally presented in a percentage value. totally exact segments constitute a 100% match, while exact segments within the same context, i.e., the same sentences or phrases before and after this segment, are called 101% matches or context-matches. the other matches below the 100% score are referred to as fuzzy matches. in this work, an analogous method is used for finding similar paragraphs. however, there are several differences. firstly, only one language, english, is used in the experiments. moreover, the segmentation rules are different. instead of sentences, paragraphs are used. a paragraph is defined as a sequence of characters ending with closing punctuation marks, i.e., a full stop, exclamation mark, question mark, followed by a new line symbol. moreover, in the case of lists, such as the text presented in the form of bullet points, additional rules apply. if colons and semicolons are followed by a new line symbol, the new line is removed so that such phrases comprise a part of a larger paragraph. otherwise, if left unchanged, each new bullet point phrase would make a separate paragraph. the paragraph consisting of a few words is not useful for this research. to imitate the translation memory matches, two corpora are needed. the paragraph cell in the above-mentioned mcq corpus serves as a source text segmented into individual characters. a paragraph is searched for in the huge reference corpus that represents the translation memory. when a similar paragraph is found, it is suggested to a user. the question, anchor, and distractors attached to the source paragraph are used now for the target paragraph. they can be post-edited to reflect the contents of the new paragraph better. although the reference corpus used in this research does not include any translations, for the simplicity of explaining the mechanism behind the research idea, it is called the translation memory, or tm corpus. ## experiments experimenting with semantic textual similarity has been within the scope of recent research in natural language processing. for example, @xcite evaluated the implications contextual word embeddings have on unsupervised semantic textual similarity methods. they carried out their experiments with several dataset including the sick dataset and bio-medical dataset for english as well as the dataset in spanish. such methods as cosine similarity using average vectors, word mover's distance and cosine similarity using smooth inverse frequency with contextualised word embeddings were evaluated to calculate semantic similarity between pairs of texts. they came to a conclusion that contextual word embeddings can be employed for various languages and domains for unsupervised machine learning tasks. @xcite experiment with siamese neural networks @xcite for semantic textual similarity. they mention that "siamese networks contain two or more identical sub-networks. the networks are identical in the sense that they have the same configuration with the same parameters and weights. in addition, parameter updating is mirrored across these subnetworks". the authors state that this type of neural networks performs well for finding similarity or a relationship between two comparable things, e.g., signature verification, face verification, image similarity as well as sentence similarity. in this research, we will use sentence transformers for the paragraph similarity tasks that also use siamese and triplet network structures. in @xcite , sentence encoders are used to improve the matching and retrieving process in translation memories systems. in our research, we use a similar approach while experimenting with paragraph encoders. ## evaluation in this study, a simple method of automatic evaluation is used. specifically, the paragraphs from the mcq corpus are fed to the algorithms. with the help of either doc2vec or sbert, semantically similar paragraphs are found in the reference corpus. the results are written to a file together with the paragraphs similarity score and the paragraph manually found as a gold standard. paragraph similarity scores resemble fuzzy matches in translation memory systems. after that, the number of identical matches, i.e., paragraphs produced by the algorithms and the paragraphs from the evaluation corpus, is calculated. the performance of the algorithms is presented in the percentage of correct matches out of 50 paragraphs. for the purposes of evaluation, a small corpus of fifty paragraphs was made. it consists of a paragraph column with the paragraphs from the mcq corpus; a question column with the questions attached to those paragraphs; and a match column with the paragraphs similar to the ones in column 1. all the matches are selected manually. in order to do so, the reference corpus is searched for key words manually identified in the paragraph from the mcq corpus. it is done with the find option. after that, the found results are checked in terms of their content and size. the paragraph from the mcq corpus and its match from the reference corpus should talk about the same thing and include the same key words. moreover, it is important that the question from the mcq corpus attached to one paragraph can be answered if shown only the paragraph from the reference corpus. in addition, the correct answer (anchor) for this question should also be included in the new match. it was required that the length of both paragraphs was roughly similar where possible. ## conclusion and future work in this research, the ways to automatically generate multiple-choice test items from a whole paragraph are investigated. in particular, it was inspired by translation memory matches in modern translation management systems. the experiments are conducted within the european union law domain. a small corpus of 200 multiple-choice test items together with the corresponding paragraphs is manually created from scratch. the larger reference corpus (tm corpus) that consists of 1,000+ pages of processed text from a textbook on the same topic was also created. the performance and efficiency of two deep learning models, doc2vec and sbert, was tested. both algorithms provide paragraph vectors used to find semantically similar paragraphs. although doc2vec was almost six times faster than sentence transformers when run on a regular computer with a cpu, it failed to produce the desired results. in fact, the performance of doc2vec was 0% when evaluated automatically. in contrast, sbert achieved 44% performance in the same automatic evaluation task. moreover, in some cases, it outperformed human judgement and found the paragraphs with higher similarity scores than those selected manually. nevertheless, the fact that sbert is not computationally efficient when used on a regular laptop remains its main drawback. in the future, this research can be broadened to include automatic generation of distractors and their evaluation. moreover, the corpus of multiplechoice test items can be extended. it would also be beneficial to have this corpus revised by an expert on the european union law. moreover, inclassroom experiments with the students of law and teachers are needed to fully evaluate the quality of produced multiple-choice test items and to test whether the suggested method truly reduces the teachers' effort in performing knowledge assessment.