Do we read what we hear? Modeling orthographic influences on spoken word recognition (2021)

Abstract: Theories and models of spoken word recognition aim to explain the process of accessing lexical knowledge given an acoustic realization of a word form. There is consensus that phonological and semantic information is crucial for this process. However, there is accumulating evidence that orthographic information could also have an impact on auditory word recognition. This paper presents two models of spoken word recognition that instantiate different hypotheses regarding the influence of orthography on this process. We show that these models reproduce human-like behavior in different ways and provide testable hypotheses for future research on the source of orthographic effects in spoken word recognition.

## introduction the abstract theory of spoken word recognition (swr) assumes that the process of speech recognition comprises two phases: a prelexical and a lexical level @xcite . the prelexical level contains prelexical representations, like phonological units, which are the result of having processed the raw acoustic signal. these units are assumed to be activated before accessing meaning representations of words in the lexical level. by instantiating the process of swr in a computational model the underlying theory can then be validated or further refined based on insights into the model's architecture and its behavior. influential models of swr are for example the cohort model @xcite @xcite , the trace model @xcite or the shortlist model @xcite . these models typically have a connectionist architecture with localist or feature-based representations as their inputs and outputs @xcite , usually mapping phonological onto semantic representations. there is evidence, however, that orthographic information could be coactivated during phonological processing. for example, words with frequent and consistent soundspelling relations have been proven to be beneficial for auditory word recognition (orthographic consistency effect, initially discovered by @xcite . consistent words, i.e., words with phonological rhymes that can be spelled in only one way (e.g. /2k/ -uck, as in duck) produce shorter reaction times in a lexical decision task, thus are easier to process, compared to inconsistent words whose rhymes can be spelled in multiple ways (e.g. /aip/ can be spelled ipe like in pipe or ype like in type). this effect is replicated in a variety of studies, using different experimental paradigms and languages (see @xcite 1 , for an overview, but also @xcite @xcite . furthermore, @xcite demonstrate that not only the phonological but also the orthographic neighborhood size of a word has an impact on swr. they report two opposing effects, the inhibitory phonological effect, and the facilitatory orthographic effect. depending on a large phonological or orthographic neighborhood of a word, the swr process is either impeded or facilitated. there is still a debate on how orthography exactly influences the process of swr. however, there are two prominent hypotheses about the source of orthographic effects in swr @xcite . according to the online hypothesis, orthographic representations are co-activated during phonological processing, whereas the offline hypothesis claims that phonological representations change through the acquisition of reading and writing such that they also incorporate orthographic information. in what follows, we present two models of swr using a long short-term memory (lstm) architecture @xcite and distributed representations, while focusing on german as a language. our major outcomes are: (1) we design two models of swr that instantiate the offline and the online hypothesis on the source of orthographic effects, respectively. (2) we replicate the inhibitory phonological and facilitatory orthographic effect, showing that these models are able to reproduce human-like behavior. (3) we provide testable hypotheses for future research based on the models' behavior, which allows us to further validate the online or offline hypothesis. ## conclusion in this work, we propose two models of swr that instantiate either the online or the offline hypothesis on the source of orthographic effects. we show that both models perform well in word meaning retrieval and in simulating the inhibitory phonological and facilitatory orthographic effect. the online model achieves the best training and testing performance, and shows the same pattern of results independent of the language of the data. it is not influenced by consistency, which indicates that the size of the orthographic neighborhood is at the origin of the facilitatory orthographic effect under the online hypothesis. this contrasts with the offline model that produces an orthographic consistency effect. when words don't differ in their consistency, the facilitatory orthographic effect is not present, which suggests that consistency is the underlying mechanism for this effect under the offline hypothesis. the models predict mutually exclusive outcomes in a lexical decision task in a language like finnish that has a high phonology-orthography consistency. by testing these predictions, further evidence for either the offline or the online hypothesis can be provided.