A dynamic model of lexical experience for tracking of oral reading fluency (2023)

Abstract: We present research aimed at solving a problem in assessment of oral reading fluency using children’s oral reading data from our online book reading app. It is known that properties of the passage being read aloud impact fluency estimates; therefore, passage-based measures are used to remove passage-related variance when estimating growth in oral reading fluency. However, passage-based measures reported in the literature tend to treat passages as independent events, without explicitly modeling accumulation of lexical experience as one reads through a book. We propose such a model and show that it helps explain additional variance in the measurements of children’s fluency as they read through a book, improving over a strong baseline. These results have implications for measuring growth in oral reading fluency.

## introduction teaching young children the skill of reading is one of the major tasks of an education system. in the u.s., a common solution to monitoring the development of reading skill is the periodic administration of oral reading fluency (orf) tests, as fluency scores can serve as indicators of early literacy skills @xcite @xcite @xcite . for example, the popular dibels test is administered three times a year. a specific passage is given in a particular grade at a particular time; e.g., the passage titled trees is administered in the spring of 3rd grade @xcite . orf is typically measured as words read correctly per minute of oral reading (wcpm), which accounts for both accuracy and speed @xcite . each passage is normed so that a student's performance can be mapped to a percentile score relative to peers. one of the weaknesses of this system of monitoring is the need to administer specific, pre-set assessment passages. first, time is taken away from reading for learning and pleasure to read for a test. second, in striving for socio-culturally responsive assessment, one would want to give agency to teachers and students in choosing what to read, as choice and interest could enhance engagement and performance. our reading app, relay reader, foot_0 addresses this weakness by letting students read different books aloud as a learning-and-pleasure activity and measuring orf in the background. this solution also allows for continuous monitoring, which means that students receive frequent opportunities to demonstrate their skill. a fundamental challenge in this endeavor is that since students read from a variety of stories throughout the year, it is not feasible to collect sufficient readings of every passage of every story to create norms. one might wonder why passagespecific norms are needed to begin with -won't students whose reading rate is 90 words per minute read any text at this rate? alas, readers exhibit a distribution of wcpm across passages @xcite @xcite , as the reader's fluency is not the only factor accounting for some of the variance in the wcpm measurements. in particular, passage effects are a known source of variance. a variety of measures proposed in the literature control for passage effects, including aspects of text complexity, genre, local discourse structure, and prosody @xcite . all these measures assume passages are independent -as they typically are in a testing context. however, passages in a book of fiction are not independent; there is continuity of characters and settings in a well-crafted narrative that create an immersive reading experience. this continuity could impact oral readingwhile a reader might stumble on hogwarts for the first time due to the word's unfamiliarity, the 50th encounter is likely to be less challenging. it is not only the rarest words that would become less challenging when mentioned many times; repeated encounters in general are known to produce faster readings @xcite . had it been the case that the first chapter introduced all the word types to be used in the book and subsequent chapters repeated those in various combinations, one would expect a steady increase in the reading pace as the reader moves through the story. such an increase would be only partially related to the improvement in the general orf skill of the reader, since the increase relies heavily on repetition of the same limited vocabulary and will likely disappear, at least partially, foot_1 when unrelated text is read. to the best of our knowledge, little is known about the relationship between repetition and story location. we hereby pose to the community a novel challenge of modeling the dynamic of a reader's lexical experience. we offer an initial exploration and show empirical results that suggest practical usefulness of further research in this area. ## surprisal the reader does not start reading harry potter with a blank lexical slate, so-to-speak; the within-thebook experience is a continuation of an ongoing lexical experience that accumulates across prior reading materials (and other language experiences, with more or less direct connection to reading). we therefore model a reader's prior knowledge using a large corpus, with the book experience viewed as an addition to the corpus -dynamically, one word at a time. 3 for every word token in the book, we use a measure of surprisal at seeing this word at this location in the book -namely, surprisal given the starting background knowledge and the withinbook experience up until the current location. in prior research, surprisal is typically defined as log of inverse of probability @xcite , that is, for a random variable y , the surprisal of the value @xmath0. in our case, the estimate of the probability p (y) for a word y is updated continuously as the student progresses through the book, token by token. thus, words that are rare in general but frequent in the book will become less surprising as the reader moves through the book, as their estimated probability will increase. surprisal will be highest for completely new words appearing near the end of the bookthis is the first occurrence in all the experience so far (background + book). in contrast, words that are generally more frequent than in the current book would become gradually more surprising, but the increase will be small, since a frequent word has accumulated a lot of prior occurrences and the impact of any new ones is relatively small. thus, if a book generally has a lower frequency of the word the than the background corpus, the will become more surprising as one adds the book to their lexical experience, but since even a long single book is orders of magnitude shorter than a large corpus that models the background knowledge, the book will only have a small impact on the surprisal values of generally frequent words. ## related work there exists a substantial body of work investigating the relationship between stand-alone properties of passages and the speed of reading. beigman @xcite showed evidence that text complexity and prosody explain variance in children's wcpm. @xcite reviewed a variety of indices used to characterize the language in passages and found that text complexity, narrativity (the extent to which a passage is story-like rather than informational), and referential cohesion were predictive of wcpm, with complexity entering with a negative coefficient, while narrativity and cohesion enter the model with positive coefficients. referential cohesion quantifies "the extent to which words overlap across sentences in the text" and is thus capturing an aspect of local, sentenceto-sentence predictability. a related but even more localized notion of predictability -within sentences rather than across sentences -was found to predict speedup in silent reading in adults; both syntactic and lexical immediate contexts were significant predictors @xcite . given the findings, it is possible to manipulate the difficulty of a story by, for example, susbtituting shorter words instead of longer words or by repeating words across sentences. 8 these would, however, alter the language of the story and could reduce its literary quality and authenticity. in contrast, surprisal can be manipulated without changing the language by sequencing stories -having the first harry potter book in your prior reading experience would make a lot of the vocabulary in the second book less surprising. another related body of literature is the work on modeling word frequency distributions @xcite @xcite . in particular, the finding that various types of corpora, including single books, tend to exhibit certain consistent largescale patterns of keyword burstiness is promising for generalization of findings such as ours across books @xcite @xcite . the extensive work on language modeling in nlp, including the advances achieved with transformer models, can be brought to bear on modeling surprisal at various granularities (word, sentence, passage) and given various types of prior experience (model pre-training, fine-tuning). furthermore, the assumption that an encounter results in reduction in surprisal for that word only is an over-simplification, as the literature on associative and semantic priming suggests that related words are also somewhat activated @xcite @xcite . transformer models were recently shown to exhibit certain priming effects themselves @xcite , making them a promising basis for modeling surprisal while accounting for priming effects. our work with a word-level dynamic surprisal is just a first step. ## conclusion in this paper, we presented a new nlp challenge coming out of the need to estimate the latent skill of oral reading fluency based on measurements of words read correctly per minute as readers move through a book using our electronic book reading app. since the measurements are known to systematically depend on the properties of the passage, it is important to control for the passage-based variance in order to produce more precise skill estimates. in particular, work presented here suggests that it is not only stand-alone properties of reading passages that are implicated in explaining slow-downs or speed-ups in oral reading, but also properties of a particular passage that have to do with its specific position in the reader's overall reading experience. as the reader reads through a book, they become more familiar with the special (invented or rare) vocabulary used in the book; this, in turn, could result in a speed-up in the reading. while the reader might be having an experience of increasing flu-ency, some of the gain might be book-specific and therefore not generalize to the next book the developing reader tackles. accurate tracking of oral reading fluency -a foundational reading skill that is a robust predictor of other skills such as comprehension -is a practical issue that will be helped by further research into dynamic models of a reader's lexical experience.