Human Sentence Processing: Recurrence or Attention? (2021)

Abstract: Recurrent neural networks (RNNs) have long been an architecture of interest for computational models of human sentence processing. The recently introduced Transformer architecture outperforms RNNs on many natural language processing tasks but little is known about its ability to model human language processing. We compare Transformer- and RNN-based language models’ ability to account for measures of human reading effort. Our analysis shows Transformers to outperform RNNs in explaining self-paced reading times and neural activity during reading English sentences, challenging the widely held idea that human sentence processing involves recurrent and immediate processing and provides evidence for cue-based retrieval.

## introduction recurrent neural networks (rnns) are widely used in psycholinguistics and natural language processing (nlp). psycholinguists have looked to rnns as an architecture for modelling human sentence processing (for a recent review, see @xcite . rnns have been used to account for the time it takes humans to read the words of a text @xcite and the size of the n400 event-related brain potential as measured by electroencephalography (eeg) during reading @xcite @xcite . simple recurrent networks (srns; @xcite have difficulties capturing long-term patterns. alternative rnn architectures have been proposed that address this issue by adding gating mechanisms that control the flow of information over time; allowing the networks to weigh old and new inputs and memorise or forget information when appropriate. the best known of these are the long short-term memory (lstm; @xcite and gated recurrent unit (gru; @xcite models. in essence, all rnn types process sequential information by recurrence: each new input is processed and combined with the current hidden state. while gated rnns achieved state-of-the-art results on nlp tasks such as translation, caption generation and speech recognition @xcite @xcite , a recent study comparing srn, gru and lstm models' ability to predict human reading times and n400 amplitudes found no significant differences @xcite . unlike the lstm and gru, the recently introduced transformer architecture is not simply an improved type of rnn because it does not use recurrence at all. a transformer cell as originally proposed @xcite consists of selfattention layers @xcite followed by a linear feed forward layer. in contrast to recurrent processing, self-attention layers are allowed to 'attend' to parts of previous input directly. although the transformer has achieved state-ofthe art results on several nlp tasks @xcite @xcite , not much is known about how it fares as a model of human sentence processing. the success of rnns in explaining behavioural and neurophysiological data suggests that something akin to recurrent processing is involved in human sentence processing. in contrast, the attention operations' direct access to past input, regardless of temporal distance, seems cognitively implausible. we compare how accurately the word surprisal estimates by transformer-and gru-based language models (lms) predict human processing effort as measured by self-paced reading, eye tracking and eeg. the same human reading data was used by @xcite to compare rnn types. we believe the introduction of the transformer merits a simi-lar comparison because the differences between transformers and rnns are more fundamental than among rnn types. all code used for the training of the neural networks and the analysis is available at https://github.com/ dannymerkx/next_word_prediction 2 background ## methods we train lms with transformer and gru architectures and compare how well their surprisal explains human behavioural and neural data. although a state-of-the-art pre-trained model can achieve higher lm quality, we opt to train our own models for several reasons. firstly, the predictive power of surprisal increases with language model quality @xcite , so to separate the effects of lm quality from those of the architectural differences, the architectures must be compared at equal lm capability. we also need to make sure both models have seen the same sentences. training our own models gives us control over training material, hyper-parameters and lm quality to make a fair comparison. perhaps most importantly, we test our models on previously collected human sentence processing data. most popular large-scale pre-trained models use efficient byte pair encodings as input rather than raw word tokens. this is a useful technique for creating the best possible lm, but also a crucial mismatch with how our test material was presented to the human subjects. it is not possible to directly compare the surprisal generated on bpes to wholeword measures such as gaze durations and reading times. ## discussion we trained several language models based on transformer and gru architectures to investigate how well these neural networks account for human reading data. at equal lm quality, the transformers generally outperform the grus. it seems that their attention-based computation allows them to better fit the self-paced reading and eeg data. this is an unexpected result, as we considered the transformer's unlimited memory and access to past inputs implausible given current theories on human language processing. notably, the transformer outperformed the gru on the two datasets where sentences were presented to participants word by word (spr and eeg). neurophysiological evidence suggests that natural (whole sentence) reading places different demands on the reader than word-by-word reading, leading to different encoding and reading strategies @xcite . metzner et al. speculate that word-by-word reading places greater demand on working memory, leading to faster retrieval of previously processed material. this seems to be supported by our results; the transformer has direct access to previous inputs and hidden states and is better at explaining the rt and n400 data from the word-by-word reading experiments. however, when we split the spr data by sentence length, the results suggest that the transformers' advantage is mainly due to performing better on the longer spr sentences. on the other hand, the transformer did outperform the gru on the eeg dataset which contains only the shorter subset of sentences. the question remains whether the transformer's unlimited memory is an advantage on longer sentences only, or if it could also explain why it performs better on data presented word-by-word. this question could be resolved with new data gathered in experiments where the same set of stimuli is used in spr and eeg. furthermore, future research could do a more specific error analysis to identify on which sentences the transformer performs better, and perhaps even on which sentences the gru performs better. such an analysis may reveal the models are sensitive to certain linguistic properties allowing us to form testable hypotheses. surprisingly, adding a gru layer did not improve performance, and even hurt it on reading time data, despite previous research showing that increasing layer depth in rnns allows them to capture more complex patterns in linguistic data @xcite . the transformers did show improvement when adding a second layer but did not improve much with four layers. as explained in section 2, a single-layer transformer cannot make use of implicit order information in the sequence. when adding a single layer to our transformer, the second layer operates no longer on raw input embeddings but on contextualised hidden states allowing the model to utilise implicit input order information. further layers increase the complexity of the model but do not make such a fundamental difference in how input is processed. in future work we could investigate how powerful this implicit order information is, and whether multi-layer transformer lms no longer require the additional explicit order information. our results raise the question how good recurrent models are as models of human sentence processing if they are outperformed by a cognitively implausible model. however, one could also interpret the results in favour of transformers (and the attention mechanism) being plausible as a cognitive model. while unlimited working memory is certainly implausible, some argue that the capacity of working memory is even smaller than often thought (only 2 or 3 items) and that language processing depends on rapid direct-access retrieval of items from storage @xcite . cue based retrieval theory posits that items are rapidly retrieved based on how well they match the cue @xcite . this is compatible with the attention mechanism used in transformers which, simply put, weighs previous inputs based on their similarity to the current input. cue-based retrieval models due have a recency bias due to decaying activation of memory representations but it is possible to implement a similar mechanism in transformers @xcite . interestingly, @xcite claims that serial order information is retrieved too slowly to support sentence comprehension. however, our two-layer transformer outperforms the single layer transformer, presumably due to order information implicitly arising as a natural result from the attention operation being performed. the use of serial order information could be compatible with cue-figure 3 : top row: the results of the linear mixed effects regression analysis on the spr data, where the data is split by whether the sentences were present in the et/eeg experiment or not. these scatter-plots show the resulting goodness-of-fit values plotted against the average surprisal over the included test data. middle row: the smooths resulting from the gams fitted on the goodness-of-fit data, with their 95% confidence intervals. bottom row: the estimated differences in goodness-of-fit score with intervals where 0 is not withing the 95% confidence interval marked by vertical lines and markers on the x-axis. each curve represents a comparison between two models, with an estimated difference above zero meaning the first model performed better and vice versa for differences below zero. in conclusion, we investigated how the transformer architecture holds up as a model of human sentence processing compared to the gru. our transformer lms are better at explaining the eeg and spr data which contradicts the widely held idea that human sentence processing involves recurrent and immediate processing with lossy retrieval of previous input and provides evidence for cuebased retrieval in sentence processing.