Are Frequent Phrases Directly Retrieved like Idioms? An Investigation with Self-Paced Reading and Language Models (2023)

Abstract: An open question in language comprehension studies is whether non-compositional multiword expressions like idioms and compositional-but-frequent word sequences are processed differently. Are the latter constructed online, or are instead directly retrieved from the lexicon, with a degree of entrenchment depending on their frequency? In this paper, we address this question with two different methodologies. First, we set up a self-paced reading experiment comparing human reading times for idioms and both highfrequency and low-frequency compositional word sequences. Then, we ran the same experiment using the Surprisal metrics computed with Neural Language Models (NLMs). Our results provide evidence that idiomatic and high-frequency compositional expressions are processed similarly by both humans and NLMs. Additional experiments were run to test the possible factors that could affect the NLMs’ performance.

## introduction it is a fact that some linguistic forms are stored in the mental lexicon, while some others have to be computed 'on the fly' by composition from smaller parts. however, the debate in linguistics and cognitive science concerns where to put the divide between 'on the fly' construction and direct retrieval @xcite . theories arguing for a primary role for composition @xcite @xcite assume that rules would be responsible for the 'on the fly' computation of regular forms, while the irregular ones have to be stored in the lexicon and retrieved as a whole. on the other hand, usage-based constructionist approaches consider frequency as a crucial factor and claim that frequent forms are stored in the lexicon, while the composition mechanism is reserved to infrequent ones @xcite . accordingly, the more often a linguistic expression is encountered, the more its representation is entrenched and the easier its retrieval from the mental lexicon is @xcite . the usage-based view found some strong supporting evidence in self-paced reading, eeg, and sentence recall experiments @xcite @xcite , where the speed at which highly frequent word sequences were processed suggested that they are stored and processed unitarily in the mental lexicon at least to some degree. in this research, considerable attention has been devoted to a class of recurring and conventional phrases denominated multiword units, phraseological units or formulaic units across different theoretical frameworks @xcite @xcite @xcite . among multiword expressions, the mechanisms underlying idiom comprehension and production have been at the core of extensive research; indeed, idioms (e.g., break the ice, cut the mustard) convey a figurative interpretation not determined by a compositional syntactic and semantic analysis of their component words @xcite @xcite . these expressions have been associated with facilitation effects in reading @xcite and a more positive electric signal in brain activity @xcite . to our knowledge, not many studies have directly compared the processing times of idiomatic multiword expressions and frequent compositional combinations, with the exception of the study by @xcite on three-word phrases (see section 2.1). in this paper, we set up a self-paced reading experiment in which we compare human reading times of english verb-determiner-noun constructions in three different conditions: idiomatic (steal my thunder), high-frequency compositional (steal my wallet) and low-frequency compositional (steal my trolley). additionally, given the success of modern neural language models (nlms) and the increasing interest in using their probabilistic predictions to account for sentence processing phenomena @xcite @xcite @xcite @xcite , we repeated the experiment by extracting the surprisal values @xcite of the words in the stimuli with several rnn-and transformer-based models, to compare them with the human results. we chose this measure because surprisal is considered an indicator of the processing load associated with a word; experiments have found a strong correlation between biometric and computational values @xcite . our results show that humans process idiomatic and high-frequency compositional expressions significantly faster than low-frequency compositional ones and, in parallel, nlms assign to them significantly lower surprisal values. among the models we tested, we found out that the smaller version of gpt2 and a 2-layer lstm obtained the exact same score patterns as human subjects; we observed no significant difference between the surprisal scores in the idiomatic and the high-frequency conditions, but the values for the infrequent condition were significantly higher. foot_0 2 related work ## discussion this study is part of a broad research about how people access meaning during language processing and to what extent nlms replicate human behavior. in our view, comparing idioms to frequent literal expressions may provide novel insights into the influence of phrase frequency on language processing and the integration of compositional and noncompositional mechanisms. in the spr experiment, we found that people read idioms and frequent compositional units at comparable speeds. the results of this study require further investigation. for instance, we could analyze the influence of context on comprehension by collecting reading times of the stimuli presented without the contextual sentence; as well, we could present the same stimuli in an eye-tracking paradigm to record more fine-grained measures than mere reading time. secondly, instead of relying only on corpus frequencies, we could explore the relationship between reading times and other ratings, such as cloze probability, plausibility, or meaningfulness @xcite . moreover, we restricted this study to n-det-v pattern, but we are planning to apply the experiment to other types of multiword expressions. finally, we are planning to extend this investigation to other languages to assess the cross-linguistic validity of our findings. the experimental evidence provided by the computational experiment confirms our behavioral findings: both idiomatic and frequent expressions are highly expected by gpt2 models. interestingly, the models that mirrored more closely human reading patterns are the smallest ones, in agreement with the findings recently reported by the literature on inverse scaling in nlms. future research includes replicating this study with other architectures, including the successor of gpt2, namely gpt3. a compelling behavior of nlms regards the role of context: it seems to affect little or not at all the surprisal scores. this evidence suggests that the surprisal of a word depends more on the ease of access to a word in the vocabulary than on the semantic integration with previous words. in other words, frequent expressions might be 'memorized' and easily retrieved, and context words do not show relevant priming effects. we plan to investigate this outcome in future experiments and verify how humans react without the contextual sentence. besides, we can conclude that the converging evidence from humans and lms suggests that multiword expressions, both idiomatic and compositional ones, are processed more holistically than compositionally. our experiment opens up to many possibilities for further analyses and refinements. for example, considering the behavioral experiment, a peculiarity of our design is that the point at which an idiom becomes recognizable is located at the end of the target phrase. even if reading times on this specific word gives us insight into the facilitation access to construction meaning, the cognitive effort in processing that word is not limited to the word itself but could emerge in the subsequent text (spillover effect; @xcite ; @xcite ). considering the computational experiment, we just analyzed the probability output of a target word through the surprisal scores, but in the future, it would be useful to adopt interpretability techniques to get more insights on the hidden representations of the nlms @xcite . we hope that our findings can contribute to the existing research in multiword expression processing, paving the way for forthcoming studies on how the compositional and noncompositional mechanisms alternate during interpretation.