The neural dynamics of word recognition and integration (2023)

Abstract: Listeners recognize and integrate words in rapid and noisy everyday speech by combining expectations about upcoming content with incremental sensory evidence. We present a computational model of word recognition which formalizes this perceptual process in Bayesian decision theory. We fit this model to explain scalp EEG signals recorded as subjects passively listened to a fictional story, revealing both the dynamics of the online auditory word recognition process and the neural correlates of the recognition and integration of words. The model reveals distinct neural processing of words depending on whether or not they can be quickly recognized. While all words trigger a neural response characteristic of probabilistic integration — voltage modulations predicted by a word’s surprisal in context — these modulations are amplified for words which require more than roughly 150 ms of input to be recognized. We observe no difference in the latency of these neural responses according to words’ recognition times. Our results support a two-part model of speech comprehension, combining an eager and rapid process of word recognition with a temporally independent process of word integration. However, we also developed alternative models of the scalp EEG signal not incorporating word recognition dynamics which showed similar performance improvements. We discuss potential future modeling steps which may help to separate these hypotheses.

## model our model consists of two interdependent parts: a cognitive model of the dynamics of word recognition, and a neural model that estimates how these dynamics drive the eeg response to words. ## methods and dataset we jointly infer 2 across-subject parameters of the cognitive model ( dataset we analyze eeg data recorded as 19 subjects listened to hemingway's the old man and the sea, published in @xcite . the 19 subjects each listened to the first hour of the recorded story while maintaining fixation. we analyze 5 sensors distributed across the centroparietal scalp: one midline sensor and two lateral sensors per hemisphere at central and posterior positions. the eeg data were acquired using a 128channel activetwo system at a rate of 512 hz, and down-sampled offline to 128 hz and re-referenced to the mastoid channels. we follow the authors' preprocessing method, which includes band-pass filtering the eeg signal between 0.5 and 8 hz, visual annotation of bad channels, and removal of eyeblink components via independent component analysis. 3 the dataset also includes force-aligned annotations for the onsets and durations of both words and phonemes in these time series. we generate a predictor time series x t aligned with this eeg time series (appendix b), ranging from stimulus features (features of the speech envelope and spectrogram) to sublexical cognitive features (surprisal and entropy over phonemes). by including these control features in our models, we can better understand whether or not there is a cognitive and neural response to words distinct from responses to their constituent properties (see section 4.2 for further discussion). we generate in addition a set of word-level feature vectors x @xmath0, consisting of an onset feature and 1. word surprisal in context, computed with gpt neo 2.7b @xcite ), foot_3 and 2. word unigram log-frequency, from sub-tlexus 2 @xcite . likelihood estimation our cognitive model requires an estimate of the confusability between english phonemes (equation 2 ). we draw on the experimental data of @xcite , who estimated patterns of confusion in phoneme recognition within english consonants and vowels by asking subjects to transcribe spoken syllables. their raw data consists of count matrices ψ c , ψ v for consonants and vowels, respectively, where each cell ψ[ij] denotes the number of times an experimental subject transcribed phoneme j as phoneme i, summing over different phonological contexts (syllable-initial or -final) and different levels of acoustic noise in the stimulus presentation. we concatenate this confusion data into a single matrix, imputing a count of 1 for unobserved confusion pairs, and normalize each column to yield the required conditional probability distributions. ## results we first evaluate the baseline model relative to a trf model which incorporates no word-level features x v except for a word onset feature, and find that this model significantly improves in held-out prediction performance (@xmath1.91, @xmath2.000113). the model recovers a negative response to word surprisal centered around 400 ms post word onset (figure 6 ), which aligns with recent eeg studies of naturalistic language comprehension in both listening @xcite @xcite and reading @xcite . we next separately infer optimal model parameters for the shift and variable models, and evaluate their error on held-out test data. we find that the variable model significantly exceeds the baseline model (@xmath3.15, @xmath4.70×10 these inferred recognition times maximize the likelihood of the neural data under the linking variable model parameters θ. figure 3 shows the variable model's parameters describing a neural response to word surprisal for each of three recognition time quantiles, time locked to word onset. we see two notable trends in the n400 response 5 a direct comparison of the variable model and shift model performance also favors the variable model (@xmath5.49, @xmath6.24 × 10 -5 ). these model comparisons and analyses of optimal parameters yield answers to our original questions about the dynamics of word recognition and integration: response properties: neural modulations due to surprisal are exaggerated for words recognized late after their acoustic onset. onset: the variable model, which asserted integration processes are initiated relative to words' onsets rather than their recognition times, demonstrated a better fit to the data. the optimal parameters under the variable model further showed that while word recognition times seem to affect the amplitude of neural modulations due to surprisal, they do not affect their latency. ## discussion this paper presented a cognitive model of word recognition which yielded predictions about the recognition time of words in context τ i . a second neural linking model, the variable model, estimated the neural response to words recognized at early, intermediate, and late times according to the cognitive model's predictions. this latter model significantly improved in held-out generalization performance over a baseline model which did not allow for differences in the neural signal as a function of a word's recognition time. we also found, however, that a neural model which estimated distinct shapes of the neural response to words based on their surprisal -not their recognition timesalso improved beyond our baseline, and was indistinguishable from the variable model. more elaborate neural linking theories describing how words' features drive the neural response will be necessary to distinguish these models (see e.g. the encoding model of @xcite . our positive findings are consistent with a twopart model of auditory word recognition and integration, along the lines suggested by van den @xcite and @xcite . in this model, listeners continuously combine their expectations with evidence from sensory input in order to load possible lexical interpretations of the current acoustic input into a memory buffer. our model's prediction of a word's recognition time τ i measures the time at which this buffer resolves in a clear lexical inference. a second integration process reads out the contents of this buffer and merges them with representations of the linguistic context. our latency results show that the timing of this process is independent of a listener's current confidence in their lexical interpretations, instead time-locked to word onset. this integration process thus exhibits two distinct modes depending on the listener's buffer contents: one standard, in which the buffer is clearly resolved, and one exceptional, in which the buffer contents are still ambiguous, and additional inferential or recovery processes must be deployed in order to proceed with integration. future work could spell out this distinction mech- early mid late recognition time quantile low mid high surprisal quantile 2676 768 100 702 1885 1029 180 970 2438 500 1000 1500 2000 2500 (a) confusion matrix comparing partitions of words by the prior-variable model (based on word surprisal; vertical axis) and the optimal word recognition model (based on recognition time; horizontal axis). context prior-only prediction rec. time prediction . . . he looked at it in disgust mid late . . . the old man was now definitely and finally mid late . . . drew his knife across one of the strips mid late . . . on his cheeks. the blotches high mid ¶he knelt high mid ¶"i mid early (b) examples of disagreements in word labeling between the prioronly model and the recognition model. anistically in order to explain how buffers in the "exceptional" state elicit these distinct neural responses. ## conclusion this paper presented a model of the cognitive and neural dynamics of word recognition and integration. the model recovered the classic n400 integration response, while also detecting a distinct treatment of words based on how and when they are recognized: words not recognized until more than 150 ms after their acoustic onset exhibit significantly amplified neural modulations by surprisal. despite this processing difference, we found no distinction in the latency of integration depending on a word's recognition time. however, we developed an alternative model of the neural signal not incorporating word recog-nition dynamics which also exceeded baseline models describing the n400 integration response. more substantial linking hypotheses bridging between the cognitive state of the word recognition model and the neural signal will be necessary to separate these distinct models.