Neural Speed Reading Audited (2020)

Abstract: Several approaches to neural speed reading have been presented at major NLP and machine learning conferences in 2017–20; i.e., “human-inspired” recurrent network architectures that learn to “read” text faster by skipping irrelevant words, typically optimizing the joint objective of minimizing classification error rate and FLOPs used at inference time. This paper reflects on the meaningfulness of the speed reading task, showing that (a) better and faster approaches to, say, document classification, already exist, which also learn to ignore part of the input (I give an example with 7% error reduction and a 136x speed-up over the state of the art in neural speed reading); and that (b) any claims that neural speed reading is “human-inspired”, are ill-founded.

## introduction a new natural language processing (nlp) task, called neural speed reading, or simply speed reading, has attracted a lot of attention within the last four years @xcite @xcite @xcite @xcite @xcite @xcite . the basic idea is to model "human speed reading techniques" @xcite for more efficient nlp, including document classification, named entity recognition, and machine comprehension. neural speed reading architectures are typically recurrent neural networks -long short term memory (lstm) networks @xcite -that jointly learn to process documents and ignore parts of them in making their decisions. the term "speed reading" comes from psycholinguistics, where it refers to fast-paced human reading, associated with fewer eye fixations, short fixation times, and longer saccades. however, while figure 1 : our argument, schematically some of the above authors claim to model human speed reading, e.g., @xcite , they do not evaluate their ability to do so, say by evaluating against eye-tracking data from readers. 1 surveying the psycholinguistics literature, however, it turns out that the notion of "human speed reading" is surrounded by controversy; there is in fact little evidence that humans can read significantly faster without also incurring a significant information loss @xcite . neural speed reading is therefore not -and can never be -a cognitive modeling effort of modeling human speed reading strategies. neural speed reading is therefore not a new task, but reduces to the well-known task of computationally efficient nlp, e.g., document classification with a time budget @xcite @xcite . moreover, as i show be-1 such data is readily available for normal-paced reading in the form of corpora such as the dundee corpus and the geco corpus: https://www2.ling.ohio-state. edu/golddundee/ and http://expsy.ugent.be/ downloads/geco/ , respectively. these datasets have been used in machine learning experiments aimed at predicting fixations during reading @xcite , as well as as auxiliary data for various nlp tasks @xcite . @xcite , for example, show that jointly predicting fixations during reading is beneficial for a sentence compression model, trying to shorten and simplify input sentences. low, neural speed reading architectures perform poorly compared to simple baseline approaches to fast document classification. contributions in sum, this paper makes the following contributions: (a) i argue speed reading reduces to computationally efficient nlp, e.g., fast document classification. (b) i therefore present a heads-to-heads comparison of a state-of-the-art speed reading architecture to a simple n-grambased classifier. (c) our simple n-gram-based classifier is shown to be significantly better and faster than the speed reading architecture. ## speed reading speed reading, as a machine learning task, was only introduced about three years ago, but has attracted a lot of attention @xcite @xcite @xcite @xcite : all proposed models so far are extensions of recurrent neural networks for text classification or sequence labeling -mostly long short-term memory networks (lstms) @xcite -that learn to either skip, skim or re-read words, jump elsewhere in the text or to make early predictions. as mentioned, none of the papers on neural speed reading, some of which are reviewed below, evaluate the extent to which they simulate human speed reading strategies. while the idea of human speed reading has intrigued modern society for decades -at least since evelyn wood introduced her reading dynamics training program in 1959 -the psycholinguistic literature argues very convincingly that human speed reading is in fact implausible: 2 the reason is physical: in order to read, people need to move their eyes so as to place the fovea foot_1 over the region that they want to process @xcite . fixation times (150-200ms) and saccade times (20-35ms) are relatively fixed, and this puts a lower bound on reading time. 4 in other words, while speed reading courses claim readers can learn to obtain information from a large area of text in a single fixation, it seems there is lit-tle scientific support for such claims: humans can not read significantly faster without a significant loss in comprehension. speed reading architectures have therefore also not been evaluated against, say, eye-tracking data from human speed reading experiments, and we therefore argue neural speed reading simply reduces to fast nlp. we review prominent architectures below. @xcite present a model that reads a fixed number of words, and then may decide to jump up to n words ahead or stop reading. the number of jumps permitted is also bounded to m, the objective is to learn how best to spend the m jumps. the authors propose to use simply policy gradient training @xcite (because jumps lead to non-decomposable loss), using classification accuracy as a reward function. note that it is not part of the objective to minimize the number of flops. they report their modified lstm with jumping is up to 6 times faster than their baseline lstm, while maintaining the same or even better accuracy. extending the work of @xcite , yu et al. (2018b) use actorcritic training rather than policy gradient training and a reward function combining task performance and flop reduction. the approach taken in @xcite is also very similar to that of @xcite , except their model allows backwards jumps, enabling re-reading of text snippets. @xcite propose a simple speed reading architecture that simply learns when to stop reading. @xcite combine a large and a small recurrent neural network and learns, at each time step, to choose which to use. the small network is thought of as only skimming the text. since this discrete choice leads to non-decomposable loss, they train the network using gumbel softmax. @xcite presents an architecture that can learn to skip (rather than skim) individual words. @xcite introduce a speed reading model for sentiment classification, in which a simple submodel determines whether or not to use an lstm or an n-gram-based classifier. their proposal, however, relies on the assumption that an lstm, in general, outperforms (all) n-gram-based classifiers on these document classification problems. we show that this assumption is false, and that (some) n-gram-based classifiers consistently outperform state-of-the-art speed reading architectures. @xcite will be our baseline in the experiments below. we therefore describe this model in some detail: structural-jump-lstm combines a standard lstm network with two simple agents: the skip agent and the jump agent. each of these agents predicts a transition distributions, from which actions can be sampled from: skipping amounts to ignoring the next word in the sequence, i.e., not updating the lstm, whereas jumping ignores all information up to some point, which can either be the next clausal separator symbol (, or ;), or the next sentence segmentator (., ! or ?), or the end of the document. 5 the motivation for adding the jump agent, which is what differentiates structural-jump-lstm from previous models, is the computational advantage (flop reduction) of being able to ignore n words without having to query the skip agent n times. the input in each time step is the previous actions of the skip agent, of the jump agent, and of the current input. the output from the previous lstm state representation is used by the agents in combination with the input to make a skip/jump decision -if the word is skipped or jumped over, the lstm state will not be updated. both agents consist of a fully connected layer, but which is significantly smaller than the lstm cell size. using these agents to skip part of the input reduces the number of flops used when processing input sequences. @xcite use a combination of maximum likelihood and actor-critic training to train their structural-jump-lstm architecture. they do so in order to jointly minimize classification error and the number of reads. since the number of reads does not decompose over the input, they cannot rely solely on maximum likelihood training and instead use a3c training @xcite with a baseline offset. 6 3 experiments datasets in our experiments, we use the three document classification datasets most commonly used in the speed reading literature: imdb and rottentomatoes are both datasets of positive and negative movie reviews collected from the imdb moview review database. imdb is larger than rottentomatoes and also contains significantly longer documents. ag news, on the other hand, is a document classification dataset, where news are classified by their topic. the ag news corpus consists of news articles from a corpus of news articles on the web, focusing only on the four largest classes. the dataset contains 30,000 training examples for each class, and 1,900 examples for each class for testing. all three datasets are balanced classification tasks, and we thus simply report accuracies on held-out evaluation samples. ## discussion and conclusion we have argued that traditional n-gram classifiers are fully adequate baselines for neural speed reading architectures, in the context of document classification. in some of the neural speed reading papers cited above, including @xcite , the authors also report results on sequence labeling problems such as entity recognition or machine comprehension. are those experiments more meaningful than the document classification exper-{1, 2, 3}, and {1, 2, 3, 4}.