From Miscue to Evidence of Difficulty: Analysis of Automatically Detected Miscues in Oral Reading for Feedback Potential (2024)

Abstract: This research is situated in the space between an existing NLP capability and its use(s) in an educational context. We analyze oral reading data collected with a deployed automated speech analysis software and consider how the results of automated speech analysis can be interpreted and used to inform the ideation and design of a new feature – feedback to learners and teachers. Our analysis shows how the details of the system’s performance and the details of the context of use both significantly impact the ideation process.

## introduction reading a text fluently -accurately and with a good speed -is evidence of development of foundational reading skills, such as decoding and word recognition @xcite . research suggests that the development of oral reading fluency is an essential bridge from decoding to comprehension of text @xcite . instructional approaches to foster fluency include modeling fluent reading to the developing reader; repeated reading (re-reading passages multiple times); and engaging students in wide independent reading @xcite @xcite . extensive reading was also shown to support fluency development in students of english as a foreign language @xcite . given the importance of fluency for reading development, we built relay reader™, an app 1 where readers can practice by taking turns reading out loud from full-length stories with skilled audiobook narrators @xcite . the narrator reads a passage while the user follows in the text; then the user reads the next passage aloud, and so on. users can set the word counts of their own and narrator turns between 70 and 200 1 the app is available freely on https:// relayreader.org/ . during the development of the app, we conducted a needs assessment with teachers which showed that obtaining estimates of students' oral reading fluency and accuracy was the top priority, followed by being able to see students' specific difficulties in reading through miscue analysis @xcite . accordingly, a speech analysis system was developed and is currently used to provide fluency information to teachers. fluency is measured as words read correctly per minute; hence the system transcribes the audio and compares to the passage text in order to provide fluency estimates. as a byproduct, the system produces an alignment between the transcript and the passage from which the miscues can be easily recovered. the goal of this study is to explore the potential of using these miscues for feedback, guided by the following research questions: (1) what is the extent of miscues in the data? (2) how are miscues distributed in reading passages? (3) how does the extended reading context come into play? (4) how reliable is miscue detection? the main contribution of our work is the exploration of the space between a deployed nlp capability and its use case. we show how the analysis of the data collected through the system can support ideation of using the system in a new wayfor feedback, specifically regarding frequency and content of such feedback. more generally, in the context of using nlp for building educational applications, we zoom in on the process of ideating a new feature in an existing ecosystem, and show how the analysis of existing data can inform the ideation process. ## related work research on automated speech recognition (asr) for young readers suggests that misreadings and slow reading constitute significant challenges @xcite ; focus on subword units @xcite and data augmentation with synthetically generated mistakes @xcite are some of the approaches proposed to improve identification of misreadings. the technical challenges notwithstanding, asr has long been used for feedback in automated reading tutors. the reading tutor from project listen, an influential early system that entered classrooms in the 1990s, displayed the text one sentence at a time. as the student read, the system interrupted if a word was read incorrectly and not self-corrected by underlining the incorrect word and occasionally "coughing" to get the student's attention @xcite . lalilo is another reading tutor for early elementary students. students record themselves reading a word, phrase, or sentence; their recording is played back, followed by a fluent model of the sentence. the reader gets feedback when the system is confident that it was correct ('perfect') or incorrect ('try again'); if uncertain, the student is asked whether their recording matched the fluent one and is encouraged with 'good job!' @xcite . bookbuddy is a chat bot that converses with young readers about the story they are reading by answering their questions, quizzing them, and automatically evaluating their spoken answers @xcite . the charlesbridge reading fluency program 'listens' as a student is reading, and when a child misreads or struggles with a word, the machine models it and asks the child to repeat it and continue reading; problem words are marked in a separate report for review and practice @xcite . the virtual reading tutor marni tracks the student while reading aloud by moving the cursor to each word as it is spoken @xcite . a reading tutor for dutch supports reading individual words, word lists, and short stories; for the latter, the student is asked to reread the sentences where they read incorrectly one or more words, as detected by asr software @xcite . in general, prior work on automated fluency sup-port tends to focus on very young learners (k-2) and on an early stage of fluency development, using words, sentences, or, at most, very short stories, and on helping the student get every item right. in contrast, relay reader is targeting a more advanced stage of fluency, with a focus on immersive extended reading. in this context, it may not be very important to get every word right, especially if it comes at the cost of breaking the flow of reading. still, detailed speech analysis data similar to that available in reading tutors can be obtained and can therefore be used for stakeholder feedback. this work is a preliminary investigation towards designing miscue-based feedback appropriate to the extended reading application. ## data the data for this study come from users-in-thewild and from study participants in school and summer programs. users-in-the-wild may choose to respond to a few demographic questions during app sign-up -who the target reader is (self, child, student, other) and whether the reader is a native speaker of english. non-native speakers using the app themselves is the largest group, followed by native-speaking children. study participants in schools and summer camps were predominantly upper elementary students (grades 3-5) in the north-east of the usa at schools and camps catering to majority african american and hispanic students. different books were added to the library at different times and received more or fewer readings, depending on study designs and reader interest. we start with a subset of the data with reasonably complete readings, that is, recordings where at least 70% of the words of the passage were found in the automated transcription (reading @xmath0. the 70% cutoff helps filter out data that is unlikely to be useful for studying reading errors, for two reasons: (a) low accuracies often correspond to cases where large stretches of the passage are left unread (skipped) or to very noisy recordings; feedback in such cases, if any, might have to focus on improving engagement in the reading activity or on improving the quality of recordings, rather than on mispronunciation of specific words. (b) the automated speech analysis is less reliable on low-accuracy recordings @xcite . more information about the system cam be found in @xcite . the system produces fluency estimates that correlate with those obtained using human transcribed data at @xmath1.94 for recordings above the 70% cutoff @xcite . the resulting dataset (bypassage) consists of 9,432 recordings by 293 readers of 2,009 unique passages from 24 books. these recordings cover 7,511 word types (unique words) and 136,450 word tokens (all occurrences of the words). to detect miscues, we use the automated alignment of the recording to the text of the passage generated as part of the accuracy computation @xcite ; we consider as miscues all deletions of words in the passage and all substitutions of words in the passage with other words; insertions of words that were not in the passage were ignored. ## patterns of miscue occurrence reading accuracy, namely, the proportion of words read correctly out of all words in a passage, averaged 91.1%. that is, readings of about 9 in 100 words are miscues; this answers rq1. to answer rq2, we investigate whether miscues tend to cluster together. to determine the proximity of errors to one another, we cluster errors occurring within five tokens of each other with the condition that tokens in a cluster must be part of the same paragraph. thus, the sequence eceeccccecce, where e stands for error and c for correct, will be considered as one cluster, since there is no stretch of more than four cs in the sequence. we find that while 32.4% of errors occur singly, most errors are proximal to other errors (see we observe that 5.2% of the errors occur in clusters of 11 errors or more (see ## extended reading the app contains a mix of short and long stories, including novels, such as the adventures of pinocchio and the wonderful wizard of oz, each with about 40k word tokens. a novel is different from a sequence of short stories that amount to a similar overall word count in that there tends to be continuity of characters, relationships, and settings throughout the story, with the corresponding repetition of key vocabulary. for example, the word marionette, a generally infrequent word, repeats 185 times in pinocchio. such frequency of occurrence, sometimes in narrator turns and sometimes in reader turns, would provide a lot of opportunities for readers to hear the model performance of the word as well as to practice reading it themselves. the interleaved reading activity itself thus constitutes a kind of feedback to the reader, albeit not immediate and indirect: frequently occurring words may be self-corrected in subsequent encounters, perhaps making immediate corrective feedback to the reader unnecessarily intrusive. for our next analysis, we use readings from readers who completed pinocchio, the most read book in the app. for every word type in the book, we collect all its readings from those readers who misread it at least once; these are readers who have correction potential since they made a mistake on the word. words with fewer than five such readers are discarded. the dataset pinocchio has 19,763 readings of 631 word types read by 47 readers. each point in the plot in figure 1 corresponds to a word type; the size of the dot corresponds to the total number of readers with correction potential. on the x-axis is the log 2 total number of occurrences of the word in the book. on the y-axis is the proportion of readers with correction potential who had at least one correct reading of that word. we start with @xmath4, since for @xmath5. the figure suggests that generally the more occurrences in the book, the higher the chances of readers figuring out the correct reading even without explicit corrective feedback. we observe that the area to the right of @xmath7.32 (20 occurrences or more) and under @xmath8.9 (<90% of readers with correction potential with at least one correct reading) is empty, with the exception of the word would. as a rough estimate, it seems that about 20 occurrences suffice for the word to largely stop being a problem. we checked this threshold on the wizard of oz data extracted similarly to the pinocchio data (11,224 readings of 480 word types read by 36 readers) and found it violated by only two words. these observations suggest that we may want to concentrate the explicit corrective feedback on words that do not occur frequently enough in the book to make self-correction through repeated exposure a near certainty. this would mean that the actual proportion of miscues that are candidates for explicit feedback to the reader may be lower than the 8.9% overall estimate. removing words with at least 20 occurrences in a story from the list of candidates for explicit feedback for that story, we observe that the proportion of feedback-eligible miscues goes down from 8.9% of all word tokens to 3.3%, for the bypassage dataset. for an average passage of 109 words, this would correspond to about 3.5 miscues eligible for correction per passage, on average, instead of 9.8. this reduction in the number of miscues eligible for correction is an affordance of the extended reading context; this finding, therefore, answers rq3. ## reliability of miscue detection before designing feedback to readers or teachers based on automatically detected miscues, we estimate how reliably the system points out miscues (rq4). in particular, our focal measure is precision of miscue detection -if a system declares an error, which would presumably trigger feedback, how often is there indeed an error? we considered words with 50% or lower %correct, reasoning that these would be likely loci for error flagging. there were 87 such words that were read by at least 10 readers each. we excluded 12 non-dictionary words that may not have a standard pronunciation. 3 for every one of the 75 word types, we randomly sampled 3 readings where the machine classified the reading as 'correct' and 3 readings classified as 'incorrect'. in cases with fewer than 3 predicted 'correct's, we used all the instances the machine deemed 'correct' (2 or 1). there were 446 cases in total for the 75 words, of which 221 had the machine's prediction of 'correct' and 225 'incorrect'. while performing the annotation, we observed that even when the final execution of a word was correct, there were often indicators that the reader was having some difficulty, such as pausing right before or right after the word, making one or more mistakes leading to the word, or repeating part of the word (e.g., a fresh convul-convulsion seized her). the reader's difficulties may manifest in the acoustic signal and, in turn, make it more difficult for the machine to tell whether the reading was correct or incorrect. we therefore considered a different construct for analysis -that of 'evidence of difficulty' vs. 'no evidence of difficulty' -for the human classification. all instances marked by the human as 'incorrect' in the previous round were labeled as 'evidence of difficulty' by default, whereas the 'correct' cases were further classified into cases with or without evidence of difficulty. comparing human classification of 'evidence of difficulty' / 'no evidence of difficulty' to the machine's 'incorrect' / 'correct' classification (see to confirm the reliability of these findings, a second annotator unrelated to the project with a master's degree in applied linguistics and prior experience annotating speech and oral reading data annotated a reliability sample of 90 randomly selected recordings out of the 446 (about 20%) for (1) correctness of the reading of the target word, and (2) for those items marked as correct, whether there is evidence of difficulty (appendix a shows the annotation protocol). cohen's κ between raters for the 3-way classification (incorrect, correct with evidence of difficulty, correct without evidence of difficulty) was 0.604; it was nearly the same (0.601) for a binary classification where 'incorrect' and 'correct with evidence of difficulty' were combined into a single 'evidence of difficulty' class and contrasted with the 'correct with no evidence of difficulty' class. using the 90 instances annotated by the second annotator, we also confirmed that the machine's precision was higher in detecting the second rater's 'evidence of difficulty' annotations than the second rater's 'incorrect' annotations (precision of 90% for 'evidence of difficulty' and 84.2% for 'incorrect'). the precision for the first annotator's data for the same subset of 90 instances was 84.2% vs 76.3% for the two constructs, respectively. to summarize: our analyses suggest that the machine's detection of a miscue corresponds more precisely to what a human listener would consider as a reading showing evidence of difficulty (80.4% precision) than to what a human listener would designate as a miscue (65.8% precision). this is because readers sometimes ended up reading the word correctly, perhaps after an initial stumble or a partial reading, or recovering from a misreading of a few words just before the current one; the machine often did not recognize these as correct readings. ## discussion: implication for feedback ideation our analysis of the automatically transcribed read aloud data from an interleaved book reading app shows a substantial extent of reading difficulty in the readers: about 9% of all word readings in the eligible transcripts show evidence of difficulty based on an automated analysis. the actual extent of difficulty, as detected by a human listener, is likely to be higher, since, while the system shows fairly high precision (0.80) in detecting what a human listener would consider evidence of difficulty, it misses many such cases, since the recall stands at 0.61. inspecting the patterns in about 9.5k recordings by 293 readers of 2k unique passages (excerpts from novels and short stories), we observed that reading difficulties tend to cluster in 3-4 consecutive words, suggesting that corrective feedback to the reader may need to contain a model performance of whole phrases rather than individual words. further, we examined the interleaved extended reading and listening itself as a kind of delayed (not immediate) and indirect feedback to the reader that does not require to break the flow of reading. we estimated that a word that occurs 20 times or more in the book is likely to have sufficient exposure in narrator and reader turns for 90% of the readers who misread it at least once to also produce at least one correct reading. assuming that there is no urgency that the reader learn a particular word now instead of a few chapters later, we may want to forgo giving the reader direct feedback on misreadings of words that will almost certainly get fixed by the time the reader finishes the story, focusing instead on misreadings of words that do not get repeated very often in the story. finally, when designing the feedback based on automatically detected misreadings, it is impor-tant to keep in mind that, at least with the speech recognition technology currently implemented in the app and the type of data typical of this use case (no acoustic control of the environment, consumerlevel devices and headsets), the detection of miscues is only 66% precise. 4 however, in 80% of the cases where the machine flags a miscue, there is evidence that the reader is having some difficultywhether or not they produced a correct reading in the end. the ideation and design of feedback will need to reflect this shift in the construct. this finding also suggests that, in terms of learner modeling, automatically detected reading errors provide evidence not only on the knowledge dimension, but also on a behavioral dimension -miscues flagged by the system may provide a first-cut detection of loci where evidence of multiple attempts, selfcorrections, pausing to consider the difficult word, and other behaviors related to the trait of perseverance may be found, upon further analysis. as a first step in exploring feedback to the teacher based on evidence of difficulty, we created class-level heatmaps per paragraph for an ongoing reading of pinocchio in a 4th grade classroom and sent the teacher the heatmaps for the paragraphs that were most difficult for the class, one per chapter. in an interview, the teacher described her use of the heatmap shown below. she told the students she was showing them a challenging passage and explained the darker red as standing for more readers having a difficulty. she told students that some of it was a bit of a tongue-twister for any reader (she said she would have had a hard time herself); she then praised the class for reading much of the passage well and for giving the more challenging part a go. the class also had a brief discussion of what a gold-piece indigestion meant. the teacher thus used the feedback not only for providing a correct reading of the miscue cluster "gold-piece indigestion" that occurred in many of her students' readings, but also for a brief but rich motivational, affective, and comprehension-related activity. pinocchio ate least of all. he asked for a bite of bread and a few nuts and then hardly touched them. the poor fellow, ## conclusion in this paper, we analyzed miscues detected by an automated speech analysis system deployed through a publicly available reading app where readers take turns reading books out loud with a pre-recorded skilled narrator. the impetus for considering miscues came from teachers' request to provide such data; however, it is not a-priori clear our analysis shows that miscuing, or, rather, readers experiencing some difficulty reading the word, even if they do pronounce it correctly in the end, is extensive -about nine in a hundred words and possibly more, since our automated system does not detect all such cases. second, problems tend to cluster together, suggesting that corrective feedback may better be presented by modeling the reading of a phrase rather than of individual words. third, in order to minimize the interference with the flow of reading, one may want to prioritize modeling misread words that do not occur very frequently in the book. we found that words with 20 or more repetition are very likely to be learned through the interleaved reading activity itself, without additional explicit feedback. finally, examining the reading instances flagged by the system as miscues, we found that these are not necessarily incorrectly read words but is closer to what a human listener would consider as evidence of some difficulty on the reader's part, whether or not the word came out correct in the end. this opens up the possibility of considering the automatically detected miscues as a first-cut detection of instances of reader struggles -not only those that manifest as an error but also those that show gearing-up or preparation (pause before), persistence (multiple attempts), or self-correction (successful final readings) -all providing evidence not only on the skill dimension, but also on important learner traits such as perseverance. from the point of view of feedback development, our analyses suggest that when designing feedback to the reader, it may be incorrect to start from the vision of "give feedback on every miscue." first, there may be too many of them. second, it would make sense to fuse feedback on multiple miscues since they tend to cluster together. third, some of the miscued words have a verifiably high chance of getting fixed during the activity without explicit feedback. finally, the content of the feedback would not actually be a correction of a miscue, because there may not have been an actual miscue -or misreading -to begin with; it may have been a successful reading following some struggle. this shift in the construct suggests feedback not only, or not necessarily, along the dimension of reading skill, but also learner traits such as perseverance. our first trial with a teacher shows promise in that the teacher was able to use the evidence of difficulty feedback not only for a corrective purpose, but also for a motivational and affective one. more broadly, our case study shows how a detailed examination of existing data from a new angle may provide new insights into the performance of the system and support ideation of a new use of the system to the benefit of the stakesholders.