CDRNN : Discovering Complex Dynamics in Human Language Processing (2021)

Abstract: The human mind is a dynamical system, yet many analysis techniques used to study it are limited in their ability to capture the complex dynamics that may characterize mental processes. This study proposes the continuous-time deconvolutional regressive neural network (CDRNN), a deep neural extension of continuous-time deconvolutional regression (Shain & Schuler, 2021) that jointly captures time-varying, non-linear, and delayed influences of predictors (e.g. word surprisal) on the response (e.g. reading time). Despite this flexibility, CDRNN is interpretable and able to illuminate patterns in human cognition that are otherwise difficult to study. Behavioral and fMRI experiments reveal detailed and plausible estimates of human language processing dynamics that generalize better than CDR and other baselines, supporting a potential role for CDRNN in studying human language processing.

## introduction central questions in psycholinguistics concern the mental processes involved in incremental human language understanding: which representations are computed when, by what mental algorithms @xcite @xcite @xcite @xcite @xcite ? such questions are often studied by caching out a theory of language processing in an experimental stimulus, collecting human responses, and fitting a regression model to test whether measures show the expected effects (e.g. grodner and @xcite . regression techniques have grown in sophistication, from anova (e.g. @xcite to newer linear mixed-effects approaches @xcite that enable direct word-by-word analysis of effects in naturalistic human language processing (e.g. @xcite . however, these methods struggle to account for delayed effects. because the human mind operates in real time and experiences computational bottlenecks of various kinds @xcite @xcite , delayed effects may be pervasive, and, if left uncontrolled, can yield misleading results @xcite . continuous-time deconvolutional regression (cdr) is a recently proposed technique to address delayed effects in measures of human cognition @xcite . cdr fits parametric continuous-time impulse response functions (irfs) that mediate between word features and response measures. an irf maps the time elapsed between a stimulus and a response to a weight describing the expected influence of the stimulus on the response. cdr models the response as an irf-weighted sum of preceding stimuli, thus directly accounting for effect latencies. empirically, cdr reveals fine-grained processing dynamics and generalizes better to human reading and fmri responses than established alternatives. however, cdr retains a number of simplifying assumptions (e.g. that the irf is fixed over time) that may not hold of the human language processing system. deep neural networks (dnns), widely used in natural language processing (nlp), can relax these strict assumptions. indeed, psycholinguistic regression analyses and nlp systems share a common structure: both fit a function from word features to some quantity of interest. however, psycholinguistic regression models face an additional constraint: they must be interpretable enough to allow researchers to study relationships between variables in the model. this requirement may be one reason why black box dnns are not generally used to analyze psycholinguistic data, despite the tremendous gains dnns have enabled in natural language tasks @xcite @xcite , in part by better approximating the complex dynamics of human cognition as encoded in natural language @xcite @xcite @xcite . this study proposes an attempt to leverage the flexibility of dnns for psycholinguistic data analysis. the continuous-time deconvolutional regressive neural network (cdrnn) is an extension of cdr that reimplements the impulse response function as a dnn describing the expected influence of preceding events (e.g. words) on future responses (e.g. reading times) as a function of their properties and timing. cdrnn retains the deconvolutional design of cdr while relaxing many of its simplifying assumptions (linearity, additivity, homosketasticity, stationarity, and context-independence, see section 2), resulting in a highly flexible model. nevertheless, cdrnn is interpretable and can shed light on the underlying data generating process. results on reading and fmri measures show substantial generalization improvements from cdrnn over baselines, along with detailed insights about the underlying dynamics that cannot easily be obtained from existing methods. foot_0 ## background psycholinguists have been aware for decades that processing effects may lag behind the words that trigger them @xcite @xcite @xcite @xcite , possibly because cognitive "buffers" may exist to allow higher-level information processing to catch up with the input @xcite @xcite @xcite . they have also recognized the potential for non-linear, interactive, and/or time-varying relationships between word features and language processing (smith and @xcite @xcite . no prior regression method can jointly address these concerns in non-uniform time series (e.g. words with variable duration) like naturalistic psycholinguistic experiments. discrete-time methods (e.g. lagged/spillover regression, @xcite @xcite ignore potentially meaningful variation in event duration, even if some (e.g. generalized additive models, or gams, @xcite permit non-linear and non-stationary (time-varying) feature interactions @xcite . cdr @xcite addresses this limitation by fitting continuous-time irfs, but assumes that the irf is stationary (time invariant), that features scale linearly and combine additively, and that the response variance is constant (homoskedastic). by implementing the irf as a time-varying neural network, cdrnn relaxes all of these assumptions, incorporating the featural flexibility of gams while retaining the temporal flexibility of cdr. previous studies have investigated latency and non-linearity in human sentence processing. for example, smith and @xcite attach theoretical significance to the functional form of the relationship between word surprisal and processing cost, using gams to show that this relationship is linear and arguing on this basis that language processing is highly incremental. this claim is under active debate @xcite , underlining the importance of methods that can investigate questions of functional form. smith and @xcite also investigate the timecourse of surprisal effects using spillover and find a more delayed surprisal response in self-paced reading (spr) than in eye-tracking. @xcite support the latter finding using cdr, and in addition show evidence of strong inertia effects in spr, such that participants who have been reading quickly in the recent past also read more quickly now. however, this outcome may be an artifact of the stationarity assumption: cdr may be exploiting its estimates of rate effects in order to capture broad non-linear negative trends (e.g. task adaptation, @xcite in a stationary model. similarly, the generally null word frequency estimates reported in @xcite may be due in part to the assumption of additive effects: word frequency and surprisal are related, and they may coordinate interactively to determine processing costs @xcite . thus, in general, prior findings on the timecourse and functional form of effects in human sentence processing may be influenced by method- subscripts omitted to reduce clutter. the irf g(τ ) at an event computes the expected contribution of each feature of the event vector h (0) to each element of the parameter vector s of the predictive distribution for a particular response value. the first layer of the irf depends non-linearly on the properties of the event via h in and (optionally) on context via h rnn , which requires the recurrent connections in gray. elements with random effects have dotted outlines. for variable definitions, see appendix a. ological limitations: the gam models of smith and levy (2013) ignore variable event duration, the cdr models of shain and schuler (2021) ignore non-linearity, and both approaches assume stationarity, context-independence, constant variance, and additive effects. by jointly relaxing these potentially problematic assumptions, cdrnn stands to support more reliable conclusions about human language comprehension, while also possibly enabling new insights into cognitive dynamics. ## results since cdrnn is designed for scientific modeling, the principal output of interest is the irf itself and the light it might shed on questions of cognitive dynamics, rather than on performance in some task (predicting reading latencies or fmri measures are not widely targeted engineering goals). however, predictive performance can help establish the trustworthiness of the irf estimates. to this end, as a sanity check, this section first evaluates predictive performance on human data relative to existing regression techniques. while results may resemble "bake-off" comparisons familiar from machine learning (and indeed cdrnn does outperform all baselines), their primary purpose is to establish that the cdrnn estimates are trustworthy, since they describe the phenomenon of interest in a way that generalizes accurately to an unseen sample. baseline models, including cdr, are as reported in shain and schuler (2021). foot_3 ## conclusion this study proposed and evaluated cdrnn, a deep neural extension of continuous-time deconvolutional regression that relaxes implausible simplifying assumptions made by widely used regression techniques in psycholinguistics. in so doing, cdrnn provides detailed estimates of human language processing dynamics that are difficult to obtain using other measures. results showed plausible estimates from human data that generalize better than alternatives and can illuminate hitherto understudied properties of the human sentence processing response. this outcome suggests that cdrnn may play a valuable role in analyzing human experimental data.