Predict but Also Integrate: an Analysis of Sentence Processing Models for E nglish and H indi (2024)

Abstract: Fluent speakers make implicit predictions about forthcoming linguistic items while processing sentences, possibly to increase efficiency in real-time comprehension. However, the extent to which prediction is the primary mode of processing human language is widely debated. The human language processor may also gain efficiency by integrating new linguistic information with prior knowledge and the preceding context, without actively predicting. At present, the role of probabilistic integration, as well as its computational foundation, remains relatively understudied. Here, we explored whether a Delayed Recurrent Neural Network (d-RNN, Turek et al., 2020), as an implementation of both prediction and integration, can explain patterns of human language processing over and above the contribution of a purely predictive RNN model. We found that incorporating integration contributes to explaining variability in eye-tracking data for English and Hindi.

## introduction languages are acquired and processed in real time. the transient quality of spoken language is evident, as it vanishes the moment it is spoken. and while written words appear fixed on a page, skilled readers assimilate them in sequence rapidly, seldom needing to double-back and review past words, and even skipping words entirely. this transitory aspect of language, coupled with the remarkable efficiency and speed at which humans use it, suggests that our brains harness specialized processes for managing information that fluidly unfolds in a sequence. one proposed cognitive mechanism is prediction, the process by which a listener or reader anticipates upcoming linguistic information during language comprehension. this anticipation is based on internalized knowledge of language, previous local context information, and accumulated worldknowledge from semantic and episodic long-term memory. psycholinguistic research suggests that individuals often implicitly predict elements such as the next word or grammatical structure while engaging with language, which allows for more efficient processing and understanding @xcite @xcite . prediction can occur at multiple levels, from anticipating the completion of a familiar phrase to forecasting the thematic content of a conversation or narrative. probabilistic word prediction has been explicitly implemented in a class of cognitive recurrent models since the inception of the recurrent neural network @xcite . however, an unresolved debate in psycholinguistics centers around the extent to which the human language processor anticipates upcoming information (prediction) and how it assimilates incoming linguistic information with existing knowledge (integration, @xcite @xcite . a mechanism of probabilistic integration would not necessarily try to predict upcoming material, but instead increase efficiency by evaluating the probability of the preceding context given each heard/read linguistic item (e.g., the current word). in recent work @xcite this mechanism has been modeled successfully using n-gram language models as the backward transitional probability, p(prior context | word) as a proxy for integration, as opposed to prediction in the form of forward transitional probability, p(word | current word). here, we conducted an exploratory analysis to determine whether a recurrent language model implementing integration can explain patterns of human language processing (online reading times revealed by eye movements from existing psycholinguistic datasets) over and above the contribution of a purely predictive language model. we did so by comparing two types of recurrent neural networks (rnns), namely the classic rnn, and the delayed rnn (d-rnn) as proposed by @xcite . the classic rnn can be considered an implementation of prediction, while the d-rnn implements both prediction and integration (see details below). to test the robustness of our method, we applied it to reading data from two languages, english and hindi, that differ typologically in several ways, reflecting their distinct linguistic origins, families, and structures. our work has value in attempting to model probabilistic integration explicitly, as an additional important cognitive mechanism underlying language processing that is currently underappreciated in psycholinguistic modeling. ## model architectures we evaluate two recurrent neural network architectures: a vanilla recurrent neural network (rnn, @xcite , and a variant that introduces a processing delay (d-rnn, @xcite . the former has a long tradition in cognitive modeling @xcite @xcite , as it is naturally suited to implement forward prediction over sequential inputs. the latter was first proposed in the context of nlp, to incorporate sensitivity to backward dependencies (i.e., approximating bidirectional rnns); however, it has also been used to model language acquisition @xcite . rnns are forward models by design because they are trained to predict the upcoming word in a sentence (x t+1 ) based on two sources of information: the current word x t and the hidden state of the network, computed in the previous step (h t-1 ). the d-rnn implements next-word prediction in the same way, but its key feature is the addition of a processing delay d such that, for an input word x t , its output is produced at time t + d (i.e. the predicted word is ŷt+d ). thus, the weights of the d-rnn are only updated after d extra words, delaying learning. @xcite showed that a large enough delay approximates bidirectional processing, suggesting that the delay allows the network to capture backward dependencies. importantly, while bidirectional models do exploit context to the left and right of a target word to be predicted, they appear unsuited as cognitive models of real-time incremental language processing, as they require entire sentences or paragraphs to compute their predictions. instead, and crucially, the d-rnn combines classic forward prediction for incremental input with sensitivity to backward dependencies, making it a suitable cognitive model of prediction and integration. ## data sources. the choice of english and hindi is based on three criteria. first, we required languages with different word order, to ensure enough variability in forward and backward dependencies. while english is strictly svo, hindi favours sov order. second, we chose languages that differ in terms of morphological typology. our statistical analysis is done at the word level, so we use this criterion to ensure the languages are comprised of words that cannot easily be separated into multiple morphemes, as in agglutinative languages. english is an analytic language that uses specific words rather than inflection to express syntactic relations. generally, this entails having one morpheme per word. hindi is a fusional language: it 'fuses' morphemes together in a word where it is not easy to distinguish the individual morphemes @xcite . thirdly, for model training and validation against human reading patterns (specifically, eye fixations on words), we sourced publicly accessible datasets for each language, containing texts of a uniform style. we utilized the potsdam-allahabad hindi eye-tracking corpus (pac, @xcite comprising word-level eyetracking data from 30 individuals reading 83 sentences sourced from newspapers. for english, our source was the multilingual eye-tracking corpus (meco, @xcite , which includes eye-tracking data captured from 46 participants reading 112 encyclopedic sentences. both human reading datasets align with expository writing, prompting us to select wikipedia articles for training our language models. these articles provide a congruent encyclopedic text style and are widely available across languages from wikimedia foundation dumps. pre-processing. we pre-process the wikipedia articles and the pac and meco sentences using the stanza library @xcite for tokenization, part-of-speech annotation and lemmatization. we use lowercased text and remove all punctuation (but we process sentences separately). in the case of wikipedia texts, we remove article titles using regular expressions, and randomly sample 200,000 sentences for each language. the chosen corpora appear comparable in their mean sentence lengths: 18.73 and 21.07 words-per-sentence (wps) for the english training set and meco sentences, respectively, and 18.94 and 16.2 wps for the hindi training set and pac sentences, respectively. we introduce an unknown token to handle outof-vocabulary (oov) and rare words. when setting low frequency words in the corpus to the unknown token, we are reducing the vocabulary size the rnns need to train on. this simplifies the task, reducing training time @xcite , but it also reduces the amount of text in meco and pac sentences. we find that a cut-off word frequency of 24 leaves us with almost 90% of the data within meco and pac, while the vocabulary of the training set in english and hindi is reduced to some 11 thousand words. the type-token ratio (t t r) calculated after the vocabulary size reduction shows meco and pac have a higher degree of lexical variety than the rnn training sets foot_0 . this is as expected since these datasets contain fewer sentences compared to the amount sampled from wikipedia. moreover, the similarity of the training sets' t t rs indicates that the rnns' task difficulty is similar across languages. finally, meco and pac require minor preprocessing. in both human reading datasets, we remove skipped words, as they do not help us quantify the predictability of a word and the processing effort required to read it. the code used for preprocessing and the rest of our research is available at https://github.com/ninadelcaro/ predict-integrate-cmcl. ## experimental setup language models. we use the code for the rnn and d-rnn by @xcite , with a delay of 1 for the d-rnn. both networks have three layers: an embedding layer, a recurrent one, and a fully connected layer with softmax activation. we feed the networks with the tokenized sentences described above, and we use cross-entropy loss on next-word prediction objective. we update the weights with stochastic gradient descent. we train until the loss becomes stable (around 45 epochs) in the classic rnn and use the same number of epochs to train the d-rnn (41 epochs for english and 45 for hindi). hyperparameter optimization is done using ran-dom search @xcite . we train on 80% of the wikipedia articles, using 10% as the validation set and the other 10% as the testing set. the hyperparameters we optimize are the word embedding and hidden state dimensions as well as the learning rate. we select the rnn model with the lowest loss on the validation set and make sure there is no overfitting by comparing the validation loss to the training loss. our final model has a hidden state size of 682, an embedding size of 426, and a learning rate of 0.001. we use these same hyperparameters across all models (i.e., for both rnn variants and languages). ## discussion the ephemeral nature of language is evident, as it rapidly vanishes from our sensory experience upon its completion -being spoken or read. while current psycholinguistics research primarily emphasizes probabilistic prediction as a mechanism that facilitates efficient language learning and realtime processing, the computational modeling of integration and its interplay with prediction in human sentence processing remain less understood. addressing this, we used an rnn to model pure prediction and a d-rnn for the combined processes of prediction and integration, and assessed the relationship between language model-derived entropy and surprisal measures and eye-tracking data. the d-rnn's entropy contribution across languages suggests that language models incorporat-ing integration explain variability in eye-tracking data beyond prediction alone, although surprisal did not yield similar results. a tentative interpretation is that the time course of integration is better reflected in a metric like entropy, which measures uncertainty based on the current state of knowledge of the model, rather than in an a-posteriori and word-specific metric like surprisal. this may be a consequence of the specific operationalization of integration provided by the d-rnn, which delays learning until subsequent words have been processed. such operationalization is in fact reminiscent of the lookahead mechanism used in the parsing literature, which peeks at a number of upcoming tokens in a sentence in order to decide between alternative syntactic analyses @xcite @xcite . the different outcomes in english and hindi data could suggest that integration and prediction may be employed differently in various languages, possibly influenced by the distinct word orders of the languages we examined-english being svo and hindi sov-and how they interact with rnn model metrics and eye-tracking measures. these observations call for additional investigations into a broader spectrum of languages to discern how language structure might tip sentence processing toward either integration or prediction. note that in modeling reading processes, we strived for cognitive plausibility. while more recent and powerful architectures such as bidirectional recurrent networks and encoder-decoder transformers could potentially implement integration, they also do it using text from the future, i.e. they require entire sentences or passages to predict a masked word and train its algorithm. since relying on future words is not cognitively plausible when processing language word-by-word incrementally, we opted for classic rnn implementations. other models like long-short term memory networks @xcite and decoder-only transformers trained unidirectionally @xcite meet our requirements, and we leave the investigation of their suitability to future work.