Assessing Reading Literacy of B ulgarian Pupils with Finger–tracking (2024)

Abstract: The paper reports on the first steps in developing a time-stamped multimodal dataset of reading data by Bulgarian children. Data are being collected, structured and analysed by means of ReadLet, an innovative infrastructure for multimodal language data collection that uses a tablet as a reader’s front-end. The overall goal of the project is to quantitatively analyse the reading skills of a sample of early Bulgarian readers collected over a two-year period, and compare them with the reading data of early readers of Italian, collected using the same protocol. We illustrate design issues of the experimental protocol, as well as the data acquisition process and the post-processing phase of data annotation/augmentation. To evaluate the potential and usefulness of the Bulgarian dataset for reading research, we present some preliminary statistical analyses of our recently collected data. They show robust convergence trends between Bulgarian and Italian early reading development stages.

## introduction good reading and text comprehension skills are key competences and essential prerequisites for high-quality education @xcite . reading ability can predict performance in all subjects where reading and writing play a role, with reading speed being the most important indicator. in the long term, students with early reading difficulties have serious challenges with general learning, academic performance and social integration @xcite . hence, it is not surprising that educational systems across europe have put considerable effort into teaching children core reading competencies. literacy research has been at the forefront of this effort, supporting evidence-based practices for reading and language classes of schools everywhere. ideally, education should be supported by continual observation of actual reading behaviour. however, major international organisations such as unicef @xcite and oecd (oecd, 2023) have lamented a regrettable shortage of large scale reading data. so far, two major sources of reading complexity have slowed down progress in collecting longitudinal reading data at scale. first, most recent and influential (eye-tracking) research on reading has typically focused on reading single words or sentences @xcite . however, the need to monitor real reading data in real-life settings raises increasing concerns with the ecological validity of behavioural language data @xcite @xcite , and requires shifting the research focus away from specific, highly controlled phenomena, to real-time processing issues @xcite @xcite . secondly, the advent of eye-tracking technology at the services of eye movement research started a prolonged period of little interest in the vocal component of reading (with only few exceptions such as de @xcite ) and a general neglect of the inherently multi-sensory nature of reading. in fact, a key cognitive insight in developing this ability occurs as learners are able to integrate three emerging sources of information about print and speech: i) the auditorily anchored understanding of syllables, ii) the linguistic-conceptual knowledge of words, and iii) the unfolding visuospatial understanding of printed words built upon the visual and (possibly) tactile exploration of the words' spatial dimension, as it occurs in finger-point reading @xcite . in attaining an efficient synchronisation between word pointing and the onset of word articulation, the learner must resolve the competing information between the multiple syllables that (s)he hears and feels and individual words that (s)he sees on a printed page (mesmer and lake, 2010; @xcite . while some of these reading aspects have been explored and investigated independently, much less work has been conducted so far to study their interaction, also because of the technical difficulty with concurrently recording asynchronous time-series of multimodal signals. the readlet infrastructure @xcite @xcite was designed and developed to meet most of these methodological and technical desiderata, with a view to collecting finely annotated, time-stamped, naturalistic reading sessions of connected texts, in both silent and oral reading modalities, by either child or adult readers. here, we report on an ongoing project that uses readlet to collect sessions of reading data of bulgarian primary school pupils, audio-recorded and "finger-tracked" in sofia. the project aims to: a) design and implement assessment strategies for monitoring and evaluating the reading and word comprehension skills of bulgarian and italian early graders; b) collect reading and comprehension data from the two populations of children using the same protocols and comparable, rigorously controlled reading texts; c) compare the data collected from children of different ages and languages (bulgarian and italian) and modelling the results. activities (b) and (c) are still underway and only some preliminary results are reported here. in what follows, we provide a broad outline of the readlet architecture (section 2), a technical description of the materials and the experimental protocol adopted for data collection (section 3), an overview of the data collected so far (section 4), and a battery of preliminary analyses (section 5). some conclusions and prospective directions of the current project are offered in the final section. ## readlet the readlet infrastructure supports a battery of specialised web services based on a cloud computing architecture (figure 1 ). the user endpoint is a native app running on an ordinary tablet with an internet connection. the app manages the administration of the reading protocol and the concurrent recording of finger touch events and the reader's voice in oral reading sessions. upon a reading session being completed, data are encrypted, pseudonymised and transmitted to the central repository through a secure internet protocol. no session data are locally saved in the tablet's internal storage after data transmission is successfully carried out. the core of the architecture is a cloud server that exposes a set of functionalities interfacing proceedings of clib 2024 the central repository with the app user. as new data are stored in the repository, cloud services are run to process text, audio and haptic data offline. since all multi-modal data are eventually time-aligned, processes can cooperate to make the analysis more robust. results are stored back to the central repository, where they remain available for post-processing. a simple interface is provided for clinicians and operators to manage customised protocols and user profiles, configure a screening session, personalise the text files (e.g reading texts, questionnaires etc.) required by a specific screening protocol. recorded data, as well as the output of offline data processing, are eventually made available through a set of web services provided by the cloud server. web services are accessed through personal authentication by means of username and password, to allow for the rapid prototyping of third-party applications for data visualisation, analysis and modelling, while complying with requirements for personal data protection. during a reading session, participants were sitting at a school desk, in front of a tablet in portrait position. for the present collection campaign, we used a 10.5 inch samsung galaxy tab a8 (2.0 ghz dual+1.8 ghz hexa-core, 3 gb ram, 32 gb emmc, android 11), with a 246.8 x 161.9 x 6.9 mm screen and a 1920x1200 pixels resolution and a 1.4 inch samsung galaxy tab s6 lite (2.3 ghz, 1.8 ghz, 4 gb ram, 64 gb emmc, android 12), with a 244.5 x 154.3 x 7 mm screen and a 2000x1200 pixels resolution. the reading text was displayed in arial font (21.25pt). each child participated in two consecutive reading sessions, one silent and one oral. in each session, the child was asked to read one of four short children's stories originally created for this project, consisting of 5 self-contained episodes organised according to levels of increasing reading complexity (see section 3.2). the length of each episode varied between 120 and 155 words (see before starting a reading session, participants were instructed to use the tip of the index finger of their dominant hand for finger-point reading. a short excerpt from a bulgarian translation of the pinocchio novel was used as practice session. the session was repeated if the child finger-tracked less than 60% of the practice text. after reading each text episode, the child was asked two reading comprehension questions consisting of a question stem (i.e. the actual question) and four randomly shuffled answers, only one of which was correct. due to the different number of episodes read by children in different grades, the number of questions ranged from a minimum of 4 (2 nd graders) to a maximum of 10 questions (5 th graders). ## data post-processing the data collected with readlet include: speech recording, the tracking record of finger movements on the tablet touchscreen, the time taken to answer comprehension questions, and the answers provided by the reader. data are then post-processed for them to be converted into time-series and then aligned with the text through the following steps. the position of the text on the tablet touchscreen is encoded with the pixel coordinates (x min, y min, x max y max) of the bounding box of each text character on the screen (including blank spaces and punctuation marks). 3 in turn, continuous finger movements are discretized into touchmove events on the tablet touchscreen, with each touchmove event being associated with its time onset and its pixel coordinates on the screen. textcoordinates and finger-coordinates are then aligned using a custom convolutional algorithm that finds the largest match between text and finger coordinates @xcite . finally, after fingercoordinates are matched with letter-coordinates, we compute the finger-tracking time of each letter in the text as the difference between the last time tick and the first time tick in the time series of touch events falling within a letter's bounding box. the finger-tracking time for a text unit containing more letters is computed as a summation of the tracking times of the letters the unit spans over. at the moment of writing the paper, the audiorecordings of oral reading sessions are in the pro-cess of being automatically converted into text using whisperx @xcite , a free opensource toolkit built on top of whisper @xcite . for each spoken word, the toolkit outputs an alphabetic transcription and the associated confidence level, together with onset and offset time-points of the word's articulation. after this first processing step, a procedure aligning word transcriptions with the original text is executed using an alignment model (downloadable here), based on a version of wav2vec2 xls-r @xcite fine-tuned on bulgarian speech data (downloadable here). this second step is taken to provide more reliable timestamps associated with the actual word in the original text. at the time of writing the paper, bulgarian children's reading data are being post-processed for speech-to-text conversion. thus, the present preliminary analyses are exclusively focused on finger-tracking data. original audio-recordings of reading sessions will not be made openly available. nonetheless, we provide open-access information about the onset and offset time-points of a word's articulation, as computed by the speech-to-text conversion tool. ## data analysis the original dataset was trimmed by excluding individual data points (word tokens) whose fingertracking time was lower than 0.01 seconds or higher than 3.5 seconds. this procedure resulted in 4.4% of the original data being removed, corresponding to 2 subjects of the original set of participants. the resulting dataset was analysed with r using generalised additive models (gams), using the package gamm4, version 0.2-6 (wood, 2017). to understand the factors affecting the pace of finger-point reading, we entered token tracking time (i.e. the time taken by the finger to underline an individual word token) as the dependent variable of two gam models with the independent variables grade level (from 2 nd to 5 th ) and reading type (aloud vs. silent) as categorical factors, and word length or word frequency as numeric predictors. finally, to take into account the inter-individual variability in our sample and control for effects of lexical variability in our texts, subjects and word tokens were entered as random effects. results are plotted in figure 3 . here, the box plot in the top panel shows that finger-tracking times are significantly shorter in silent reading than in the aloud reading condition for grades 3, 4 and 5 (p-@xmath0.001). in addition, one can observe a decreasing trend of finger-tracking times for increasing grade levels (p-@xmath1, with a plateau between grade 4 and 5 for both reading conditions. we believe this levelling effect to be due to the greater complexity of the additional episodes read by 5 th graders. effects of word length (left bottom panel) and word frequency (right bottom panel) on tracking times exhibit a significant interaction with the grade level, with early graders (2 nd and 3 rd ) being more affected by both word length and word frequency than late graders (4 th and 5 th ) (p-@xmath2. the effect, also observed in italian child's reading data collected with the same finger-tracking technology @xcite , shows that the orthographic lexicon of bulgarian 4 and 5 graders contains not only more word tokens than the orthographic lexicon of 2 and 3 graders, but also distinctively longer and less frequent ones. as a result, the finger-tracking pace of bulgarian 4 and 5 graders is less prominently slowed down by longer words (left panel) and less frequent words (right panel), than the pace of bulgarian 2 and 3 graders is. the evidence confirms that even very young readers of a script such as the bulgarian cyrillic alphabet foot_4 tend to opt for a lexical reading strategy as early as possible for the sake of reading efficiency. finally, the two plots also show a significant interaction between word length or frequency and reading conditions, with finger-point reading being significantly more affected by both word length and word frequency in the aloud reading condition than in the silent one (p-@xmath3. not only does this evidence suggest that longer and rarer words are more difficult to process and access. also their articulation take longer to be planned and executed. incidentally, this provides further, indirect evidence of the strong correlation between finger-tracking times and articulation times in finger-point reading. we expect this evidence to be confirmed by text-aligned and timealigned speech-recognition data. ## discussion and outlook in this study, we capitalised on the huge potential of mobile information technology, cloud computing and nlp for behavioural data collection and analysis, to investigate developmental trends in the reading data of bulgarian early graders. preliminary results significantly replicate benchmark effects attested in the reading literature on transparent scripts, and are in line with the finger-tracking data collected from italian children with the same protocol @xcite . overall, the data confirm that children confronted with a transparent script tend to resort to a lexical reading strategy as early as possible, since direct access to orthographic lexical information allows for a more fluent and efficient reading performance than a sublexical reading strategy. there are several reasons to recommend an extensive usage of finger-tracking and nlp technologies for literacy research and education. first, the use of a simple tablet supports unobtrusive collection of multimodal reading data in ecological contexts. following the italian experience with the readlet infrastructure, we were able to replicate, on a smaller scale, a relatively straightforward and efficient protocol for bulgarian child's reading data collection at school. pupils were extremely responsive to the task, and showed a great familiarity with using the tablet for reading. this made the process of data collection surprisingly quick and most effective, suggesting that finger-tracking can aptly be used for extensive reading assessment in primary schools. in particular, a tablet can be used as a reading book for large-scale studies, paving the way to generalisable results. in addition, the possibility to take repeated single-subject measurements makes finger-tracking evidence suitable not only for group assessment, but also for individual developmental studies. on a more technical note, the use of a tablet as a reading book allowed us to collect finger-tracking and audio-recording streams concurrently, and take advantage of their being both text-aligned and timealigned. in spite of recent progress in the accuracy of nlp technologies, they can occasionally be brittle and error-prone, particularly when confronted with real language data, which are collected in inherently noisy, ecological communication contexts. the bonus of having multiply time-aligned multimodal data streams is that, in processing raw data, noise in one channel can be filtered out by integrat-ing synchronous information coming from a less noisy channel. for example, the drift of a fingertracking signal in a particular time window, can be corrected using the voice signal sampled and text-aligned in the same time window @xcite . in fact, the latter can provide reliable information about which text line the reader is currently processing. this is expected to offer better fingertracking data but also better transcribed spoken data, which, in turn, can be aligned more reliably both individually with the text being read, and with each other. in the near future, we consider exploring several lines of research. first, we intend to investigate the complementary role of n-gram frequencies in affecting bulgarian finger-point reading, which is likely to attest to a dynamic interaction of sublexical and lexical reading strategies through literacy development @xcite . in addition, based on speech-recognition data, we intend to compute the correlation between finger-tracking times and articulation timing in aloud reading, and assess the implicit prosody hypothesis @xcite , i.e. the idea that, in silent reading, readers activate prosodic representations that are similar to those they would produce when reading the text aloud. another aspect of our present study worth exploring in the near future is the interactive dynamic between voice recording and finger-tracking data. a recent analysis of the correlation between voice articulation, finger-tracking patterns and eyetracking patterns in adult reading @xcite showed that, in finger-point reading, the finger is most often located few characters ahead of the voice, and that the finger's pace is a rhythmic proxy for a reader's articulation rate. in particular, the distance of the finger from the voice correlates with the distance of the eye from the voice (or eyevoice span: inhoff et al., 2011), a measure of the capacity of a reader's phonological buffer and reading fluency @xcite . thus, one can reasonably expect that monitoring the development of the finger-voice span in early readers can provide evidence of more and less typical developmental patterns of reading skills in the first years of primary school, when reading difficulties are more critical but manifest less clearly. this will offer a suitable benchmark for continual assessment of reading proficiency.