Predicting Text Readability from Scrolling Interactions (2021)

Abstract: Judging the readability of text has many important applications, for instance when performing text simplification or when sourcing reading material for language learners. In this paper, we present a 518 participant study which investigates how scrolling behaviour relates to the readability of English texts. We make our dataset publicly available and show that (1) there are statistically significant differences in the way readers interact with text depending on the text level, (2) such measures can be used to predict the readability of text, and (3) the background of a reader impacts their reading interactions and the factors contributing to text difficulty.

## introduction there are multiple attributes of a written text that impact how difficult it is to read. this concept is formally known as the readability of text, where readability is defined as the sum total of elements within a text that impact a reader's understanding, reading speed and level of interest @xcite . many factors can influence the readability of text, such as the lexical and syntactic complexity, level of conceptual difficulty and style of writing @xcite . for instance, figure 1 presents an example sentence (b) that has been rewritten to a more readable format shown in (a). 2 automatically measuring the readability of text has many useful applications. for example, it is used in text simplification @xcite , when sourcing reading material for language learners @xcite , in ranking search engine content for dyslexic users @xcite and to ensure critical consumer information is delivered at an appropriate level @xcite . however, current approaches for measuring readability 1 our dataset is available at https://github.com/ siangooding/readability_scroll 2 example from the onestopenglish dataset by @xcite . furthermore, traditional readability approaches do not work well for online content @xcite . this is due to systems being highly sensitive to noise; requiring well formed sentences and performing poorly on short passages @xcite . linguistically driven techniques are also language specific and require sophisticated models to extract features. for lowresource languages, such tools and models may be unavailable @xcite . achieving a more inclusive assessment of text readability requires collecting subjective measures on multiple levels (e.g., reading interactions and comprehension-based questions) across different genres and populations. therefore, alternative approaches must be examined to further the understanding of text difficulty and the multiple factors that influence it. in this paper, we introduce such an alternative approach, by using crowdsourcing to collect scroll-based interactions whilst participants read texts at differing levels. we present a 518 participant study, recording the reading interactions of individuals and obtain statistically significant results showing that reading interactions differ dependent on the textual complexity. in the following sections, we discuss relevant previous work, outline our data collection and report results from our study. as a preliminary evaluation, we report the statistical significance of reading interactions based on text difficulty, and integrate scroll features into a readability classifier. finally, we investigate group differences in readability based on the first language of participants. to conclude, in this paper we make the following contributions: ## data collection we design a controlled reading task and record user reading interactions. we collect responses from participants in the united states and india. this decision was made based on our pilot studies which showed participants from india were much more likely to know english as a second language. the experiment was designed in line with common practises in scroll interaction research @xcite . this study represents best efforts in the collection of a novel dataset of reading interactions for text at different levels of readability. ## conclusions to conclude, the use of scroll features for judging readability has numerous benefits. such measures are language agnostic, unobtrusive and are robust to noisy text. furthermore, implicit user feedback allows an insight into readability at an individual level, thereby allowing for a more inclusive and personalisable assessment. we present a 518 participant study to investigate the impact of text readability on reading interactions. in this paper, we confirm that there are statistically significant differences in the way that readers interact with advanced and elementary texts, and that the comprehension scores of individuals correlate with specific measures of scrolling interaction. we demonstrate that, even with a simple model, aggregate scroll interactions can be used to predict readability. moreover, we show that individual scroll behaviour can provide an insight into the subjective readability for an individual. finally, we improve a state-ofthe-art readability classifier with the integration of scroll-interaction features, demonstrating that interaction features are highly complementary to traditional linguistic approaches. in future work, we will focus on investigating which aspects of readability scroll-based measures index.