Selecting Context Clozes for Lightweight Reading Compliance (2022)

Abstract: We explore a novel approach to reading compliance, leveraging large language models to select inline challenges that discourage skipping during reading. This lightweight ‘testing’ is accomplished through automatically identified context clozes where the reader must supply a missing word that would be hard to guess if earlier material was skipped. Clozes are selected by scoring each word by the contrast between its likelihood with and without prior sentences as context, preferring to leave gaps where this contrast is high. We report results of an initial human-participant test that indicates this method can find clozes that have this property.

## introduction ideally, college students would complete assigned readings before class, allowing professors to lean on that shared knowledge, extending and deepening understanding rather than reteaching the textbook context during the class. however, there have been a number of studies showing that when student work is not directly checked in some way, reading compliance is low (burchfield and sappington, 2000; clump et al., 2004; connor-greene, 2000). an obvious approach to encouraging reading compliance is for the professor or publisher to create quizzes that confirm whether students have completed the associated reading. while such questions can aid learning, thoughtfully drawing connections between various parts of the text or encouraging deeper thinking, they also require time to create, complete and score and perhaps become less useful over time as the answers begin to circulate online. perusall is a social learning platform designed specifically to improve reading compliance (johnson, 2019). the tool segments students into small groups, who can then annotate and discuss the readings online. the authors report impressive results, increasing reading compliance to as much as 90% in some cases. however, this method uses group learning and written responses, which may not be possible in all situations or desirable to all students. we propose a new approach, demonstrated in a prototype application named gloze (from gloss + cloze). a traditional cloze exercise requires a student to fill in words removed randomly or at fixed intervals from a passage. such cloze exercises can be used to assess language proficiency, and have a long history in that literature (alderson, 1979). in gloze, we hope to leverage the cloze concept to increase reading compliance of long texts without time spent creating or grading external assessments. shown in figure 1 , the method periodically requires the reader to choose the correct next word, using multiple choice with confusers to reduce the disruption of typing during reading. a key requirement of this approach is selecting challenges such that answering is easy if prior context has been read but difficult if not. as an example of how context impacts a cloze exercise, consider the human-crafted sentence pair: he caught the pass and scored another touchdown. there was nothing he enjoyed more than a good game of . (federmeier and kutas, 1999). note that the answer (football) is clear with the context of the first sentence (assuming familiarity with the sport), but that with only the partial second sentence, the answer is ambiguous. we define this particular cloze formulation as a context cloze, where presence of prior context has an outsize (perhaps even opposite) impact relative to the immediate context. lambada (paperno et al., 2016) leverages a similar framing, though with humancomputer roles reversed, to evaluate language understanding in large language models (llms). a test set is selected by having humans perform cloze exercises with and without broader context, selecting clozes that are easy with context and hard without. @xcite in this work, we focus on the issues of selecting context clozes with one contrasting confuser and validating that llms generally model the performance of human participants in this domain. note that there are many more issues required for gloze to be useful that are not addressed here, some of which are enumerated in the future work section. all our examples and tests in this work use the text of the 685-page, freely available, anonymous college-level introduction to psychology ((removed), 2015). however, the method could in theory be applied in any domain where confirmation that a long document has been read is important (e.g., legal agreements, safety manuals or human resources training documents). ## context cloze selection a correlation between llms and human word predictions has already been demonstrated. goldstein et al. (2020) conducted an experiment with human participants, asking them to predict each next word in a long narrative. they denoted the predictability of a word as the percentage of respondents that correctly generated it. comparing human predictability scores to those from gpt-2 (radford, 2020) on the same task, they found a strong correlation (@xmath0.79 with a 100-word prior context). therefore, it seems reasonable to leverage llms to approximate human predictability based on various contexts. in what follows, we use gpt-2 for our predictions. @xcite to choose the best context clozes with the llm, we evaluate all words in the text, scoring each based on how the predictability changes with and without context. the reading application can use this complete weighted ordering of words to select the highest scoring cloze within some region of text. note that this approach is not explicitly leveraging part of speech, text markup for key terms or measures of importance such as term frequency-inverse document frequency (though our method may be implicitly finding similar "important" items, it isn't required). after eliminating stop words, for each word in the text (a target), we compute this score by selecting the entire prior sentence and the partial sentence consisting of the words of the target's sentence up to the target. @xcite if we define: t 0 = p (target|partial) t 1 = p (target|prior+partial) then we prefer targets that maximize t 1t 0 (i.e., targets with high likelihood with context and low likelihood without). note that a high-scoring target does not necessarily need to be related to the content of the chapter but simply one with the right shift in predictability. as we aim to present these targets as cloze exercises during reading as multiple choice selections, we also consider whether there is a candidate confuser that actually has the opposite predictability movement. as above, we define: c 0 = p (confuser|partial) c 1 = p (confuser|prior+partial) to select a confuser to contrast with the target from the same context, we examine the probabilities of the top 25 words in both contexts, selecting the confuser that maximizes c 0c 1 (i.e., the confuser that has the largest decrease in probability when context is included). with these four next-word probabilities from gpt-2, we can define a target's score. for targets where t 1 > t 0 (target more likely with context), c 0 > t 0 (confuser more likely than target without context) and t 1 > c 1 (target more likely than confuser with context), we define a score @xmath1. for the purposes of this work, all other words have @xmath2. however, we noted after initial examination of high-scoring targets that the score did not accurately capture the predictability of a student reading a textbook for a class. in particular, the student knows the subject area she is reading about, which shapes the predictability even in a partial context. to account for this observation, we added the first paragraph from wikipedia's entry describing the field of psychology as context in front of all prompts (wikipedia contributors, 2021). with this prior partial target there are also individual differences in need for sleep. some people do quite well with fewer than 6 hours if a sound occurs on your left side, the left ear will receive the sound slightly sooner than the right ear, and the sound it receives will be more intense, allowing you to quickly determine the location of the sound. although the distance between our two ears when we are awake, our brain activity is characterized by the presence of very fast beta waves. the bart is a computer task in which the participant pumps up a series of simulated balloons by pressing on a computer key. ## human-participant comparison the premise of the context cloze is that we can identify places in the text where the correct next word is unlikely given only local context (the sentence so far) but likely given the prior context (the past sentence). while the probability-based scoring used to rank context clozes guarantees this condition is true for the llm, and we know that in general llms approximate human language models, how can we know that this scoring method provides a reasonable model of human responses to these cloze exercises? in this section, we describe an experiment we conducted to explore this question. foot_4 ## future work as noted earlier, the initial test presented here only addresses one piece of what an actual system would require to be successful. in future work, we want to explore larger prior context windows as well as examine fine-tuning as a replacement for the fielddescription paragraph. it is also critical to characterize the frequency of adequate context clozes in textbooks relative to the frequency that would be required to ensure a particular level of reading attention. in addition, many aspects of confusers need to be explored: how they impact the choice of context clozes, how various selection strategies impact compliance metrics and how to ensure llmgenerated confusers, while not the right answer by design, aren't creating confusion or demonstrating bias by being presented in a particular context. we removed some high-scoring clozes from this human-participant test, for reasons that we believe would be alleviated when the system is run in context (isolated content concerns) or through additional improvements to text processing (tokenization and filtering)-however we would need to demonstrate this is true at scale. finally, we hope to evaluate this approach in a few classes through a mobile reading application that uses our scoring method. ## conclusion we described a method for selecting context clozes to encourage reading compliance, and ran this algorithm on an introductory college textbook. we took some of the best scoring clozes and conducted a human-participant test, which confirmed our hypothesis that llms are a reasonable proxy for human predictability for context cloze scoring and that these clozes on average demonstrate the desired shift in predictability with and without context.