You Don’t Have Time to Read This: An Exploration of Document Reading Time Prediction (2020)

Abstract: Predicting reading time has been a subject of much previous work, focusing on how different words affect human processing, measured by reading time. However, previous work has dealt with a limited number of participants as well as word level only predictions (i.e. predicting the time to read a single word). We seek to extend these works by examining whether or not document level predictions are effective, given additional information such as subject matter, font characteristics, and readability metrics. We perform a novel experiment to examine how different features of text contribute to the time it takes to read, distributing and collecting data from over a thousand participants. We then employ a large number of machine learning methods to predict a user’s reading time. We find that despite extensive research showing that word level reading time can be most effectively predicted by neural networks, larger scale text can be easily and most accurately predicted by one factor, the number of words.

## introduction understanding how we read and process text has proven a large area of both cognitive science and natural language processing (nlp) research @xcite @xcite @xcite . online content providers and consumers are also interested in this research; in the increasingly busy world of today, consumers lack the time to read long articles, prompting content creators to aim for specific reading lengths. many providers 1 have even examined traffic patterns in order to determine the ideal content length, with the general consensus finding 3-7 minutes of work done as part of a capstone course with adobe 1 medium's study can be found here. content optimal. thus, having established the optimal content length, article writers now face the next hurdle: when has their post reached the ideal length? a news article about last night's football game may be easier to read than a technical post about nlp. perhaps the font type or size influences the consumer's comprehension, slowing down the reading process. there are many factors, both textual and stylistic, that quickly come to mind when considering the potential reading time of an article. although there has been an extensive body of work on reading time prediction applied to single words @xcite @xcite , to the best of our knowledge there has been no research into understanding these effects on document sized text. in this paper, we seek to address this area by building models to predict, understand, and interpret factors that could affect an article's reading time. our contributions to this area include a methodically designed statistical study, consisting of 1130 experimental trials and 32 different articles, experimental results for a broad collection of machine learning algorithms on this novel task, and discussion of potential reasons why more complex models fail. to the best of our knowledge, this is the largest experimental study for reading time research, in terms of participants and breadth of factors. all code and datasets are publicly available. foot_0 ## related work researchers have made significant progress in predicting the reading time of single words, illustrating the effect of different words on the human brain @xcite @xcite for many different texts @xcite . although this effort is focused more on the cognitive effects of words, these results show that scientists can accurately predict the reading time of individual words in context. with the rise in popularity of machine learning techniques, many scientists have found the most success through these methods, with the most recent research showing significant improvements from combining neural networks as language models with linear mixed models (lmms) @xcite @xcite . however, all previous research has been confined to the effect of a specific word in context, which naturally leads to the question of how this research generalizes. a separate but similar line of research, readability, measures the reading difficulty of a body of text. this research area has investigated effects of readability in a plethora of areas: online vs paper @xcite , color and contrast @xcite , and writing style @xcite . the most famous readability metric for english, the @xcite , uses the number of syllables and words to determine readability. other scientists have attempted to improve upon this simple metric, showing success in reading level classification with unigram language models @xcite or svm models built on top of these basic textual characteristics @xcite . as previous metrics seem to be sufficient, recent research has focused on evaluating and comparing the diverse metrics on different domains @xcite . we use these readability works to influence our choice of features, as readability seems inherently interwoven with reading time. we employ the py-readability-metrics package to include 7 state-of-the-art metrics that we add to our data for the modeling task (section 4, appendix b). ## experimental design we collected our reading time data from a statistical survey performed on amazon's mechanical turk. since we were not physically present to observe the respondents we took a number of precautions and controls to ensure data quality. we note however, that the inclinations of mechanical turk users align with our target audience: we would expect most readers of online content to be of a younger demographic, tech-savy, and prone to read as fast as possible. in this section we will discuss our survey design, validation, and results. ## modeling with the data gathered and readability metrics calculated (see section 2), we explore the results from a variety of different models. we employ three categories of models: models that only use extracted features, models that only use the text, and models that stack textual-only models with model features. basic extracted feature models include a vanilla linear regression (lr) with only the number of words variable ("word"), a linear regression model with all variables ("all"), random forests, k-nearest neighbors (knn), and a multi-layered perceptron (mlp). as using the entire article as input for the text only models is not computationally feasible, we use modern neural networks to embed the text as a document embedding, using a linear output layer for regression. we tried various state-of-the-art embedding models including roberta @xcite , xlnet @xcite , and elmo @xcite . the stacked models combine the document embedding with the extracted features, feeding them both into an mlp. embeddings use the flair @xcite and huggingface @xcite libraries. we use two baselines: a commonly used rule-ofthumb for online reading estimates, 240 words per minute (wpm), and the sum of the word-level predictions (surprisal-sum) from a surprisal model in order to compare with recent works @xcite . for the surprsial-sum baseline predictions, we employ the model used in @xcite , where predictions are made by training a linear mixed model over surprisal data. ## results the results from our experiments are found in given the amount of empirical evidence from word level reading time prediction, we were surprised to see a dearth of similar results for document level prediction. models that provide strong results in word level prediction, such as varieties of neural networks, fail to be as effective as the simpler models. perhaps this is due to the length of ## conclusion given previous work in single word reading time prediction, we conducted a large novel study to test whether document level reading time could be predicted. we carefully designed an experiment containing a myriad of potential factors to measure reading time, distributed the survey to more than a thousand people, and collected the results into the first dataset of its kind. we then employed machine learning techniques to predict the time to read, finding that simpler models were the most competitive, with the number of words as the sole critical factor in predicting reading time. we hope this resource can benefit future research into developing techniques to model and understand human responses to document sized text.