When Language Models Fall in Love: A nimacy Processing in Transformer Language Models (2023)

Abstract: Animacy—whether an entity is alive and sentient—is fundamental to cognitive processing, impacting areas such as memory, vision, and language. However, animacy is not always expressed directly in language: in English it often manifests indirectly, in the form of selectional constraints on verbs and adjectives. This poses a potential issue for transformer language models (LMs): they often train only on text, and thus lack access to extralinguistic information from which humans learn about animacy. We ask: how does this impact LMs’ animacy processing—do they still behave as humans do? We answer this question using open-source LMs. Like previous studies, we find that LMs behave much like humans when presented with entities whose animacy is typical. However, we also show that even when presented with stories about atypically animate entities, such as a peanut in love, LMs adapt: they treat these entities as animate, though they do not adapt as well as humans. Even when the context indicating atypical animacy is very short, LMs pick up on subtle clues and change their behavior. We conclude that despite the limited signal through which LMs can learn about animacy, they are indeed sensitive to the relevant lexical semantic nuances available in English.

## introduction animacy plays a significant role in cognitive processing, as evidenced by the fact that animate entities are easier to remember and prioritized in visual processing @xcite @xcite . it is so important that even young children can distinguish animate and inanimate entities @xcite , and these are processed in distinct domain-specific brain regions @xcite . animacy distinctions also manifest in language; however these distinctions may appear indirectly. while some languages explicitly mark animacy, an-imacy distinctions in english often take the form of selectional constraints that limit the use of certain verbs or adjectives with in/animate entities. for example, only animate entities can walk or think. so, while animacy is a rich distinction at the cognitive level, at the linguistic level, its signal can be muted. today's pre-trained transformer language models (lms), however, are trained only on linguistic input. if they are to learn to process animacy, they must thus do so only from its downstream effects in text, unlike humans, who use visual and physical stimuli. we therefore ask: do such lms respond to animacy in language as humans do? we answer this by treating lms as psycholinguistic test subjects, probing how they react to violations of animacy-related selectional constraints. like prior work @xcite , we first study lms' responses in scenarios involving typical animacy. in such situations, animacy is a simple mapping between an object (e.g. a peanut) and its usual animacy (inanimate). we find that like humans, lms generally prefer sentences that respect animacy-related selectional constraints, assigning higher probabilities to such sentences. unlike prior work, we also study atypical animacy @xcite , where a typically inanimate object becomes animate. we draw on @xcite , which measured human n400 responses in scenarios with atypically animate entities like a peanut in love. we compare lm surprisal to human n400 brain responses and find that like humans, lms are initially surprised to encounter entities like a peanut in love, but quickly adapt, becoming less surprised. stronger lms are more able to replicate the large magnitude of human n400 reduction. given lms' success at adapting to atypical animacy with a long context, we test them on short sentences about atypically animate entities, and measure the extent to which their outputs reflect this atypical animacy. we find that even with lim-ited context, lms adapt their output distribution, treating the entity as animate. we conclude that, despite training without the modalities that humans use to learn about animacy, lms respond to shifting animacy in a surprisingly human-like way. code for our experiments is available at https://github.com/hannamw/lms-in-love . ## models we experiment with these models: gpt-2 small, medium, large, and xl @xcite ; opt 2.7b, 6.7b, and 13b @xcite ; and llama 7b, 13b and 30b @xcite . 1 we use autoregressive lms, as we need to compute probabilities for whole sentences. moreover, we choose open-source models, to make our work replicable. we provide implementation details in appendix a. ## typical animacy we test models' responses to animacy in situations where the animacy of a given token, or instance of entity, aligns with the animacy of its type more generally (e.g. cats are animate; rocks are not). experiment we test the models in section 3 on the animate-transitive and animate-passive datasets of the blimp benchmark @xcite . each dataset contains 1,000 minimal pairs of synthetic english sentences that differ only by one or two words ( results figure 1 displays results for each model. it also includes human baselines, reported directly from @xcite , which indicate the proportion of examples where annotators preferred acc? sentence t yes naomi had cleaned a fork. t no that book had cleaned a fork. p yes lisa was kissed by the boys. p no lisa was kissed by the blouses. the acceptable sentence of the given minimal pair. random performance for both datasets is 50%. models attain high performance in both scenarios. on transitive examples, they reach over 80% accuracy; some models prefer the sentence that respects animacy constraints more often than humans do (>87%). in the passive scenario, the gap between models (80%) and humans (86%) is wider. this difference between the transitive and passive cases may be due more to setup differences than distinct animacy processing in the two scenarios. in the passive case, the target word is always in the last position, so model performance is determined only by the target's probability. in contrast, the target word is not the final token in the transitive case, so model success is determined by the probability of a longer string. discussion our results indicate that models respect animacy constraints in typical scenarios: they match human performance on the transitive dataset, and are close behind on the passive. however, this test cannot distinguish between a model that truly understands animacy, and one that just associates words (types) with other words that reflect that word's typical animacy. for example, the model might simply associate a word like "shoe" with verbs that take inanimate objects, without understanding that the inanimacy of an individual shoe is what prohibits its use with animate-selecting verbs. to solve this problem, our analysis must move beyond type-level animacy, and test models' processing of animacy at the token level. we thus test models' responses to entities whose token-level animacy is atypical, distinct from their usual type-level animacy. if models process these entities according to their type-level animacy, their understanding of animacy is rather shallow. in contrast, models that process entities according to their token-level animacy may better understand animacy in full. ## atypical animacy in this section, we attempt to determine if lms can capture animacy not only at the type-level, but also at the token-level. we do so by comparing model and human responses in cases of atypical animacy, where entities' canonical type-level animacy and their actual token-level animacy differ. for human data, we turn to two similar studies- @xcite and @xcite -that relied on the n400, a brain response measured via eeg that is elevated when processing semantically anomalous input. both studies measured participants' n400 responses while they read stories where a typically inanimate entity acted as animate (figure 2 ), similar in tone and content to a cartoon, or fairy-tale. both found that while participants were initially surprised by the atypically animate entity, they quickly adapted, yielding low n400 responses to the entity. we ask if the same is true of pre-trained lms: can they adapt to entities that are animate at the token-level, despite being typically inanimate? or is their processing of animacy limited to a simple type-level understanding? to answer this question, we replicate these studies with pre-trained lms, using their surprisal to model n400 responses. we replicate three experiments: nieuwland and van berkum's repetition experiment; their context experiment; and boudewyn et al.'s adaptation experiment. 2 for each, we first explain the original study. then, we explain how we adapt the experiment for lms. finally, we report our results and compare them the original study's results. a nurse was talking to the sailor/oar [1] who had been in a violent boating accident. the sailor/oar cried for a long time over the storm that had raged over the lake for hours. the nurse consoled the sailor/oar [3], saying that he would soon be well again. the sailor/oar complained of a bad headache that would not go away. the nurse gave the sailor/oar [5] a large dose of aspirin. the sailor/oar thanked her and fell asleep. ## low-context atypical animacy the previous experiments have shown that lms can adapt in scenarios with atypically animate entities; however, lms could have exploited shallow context features to do so, without any specific understanding or representation of animacy. we now investigate the extent to which lms can leverage cues in the context by testing their behavior on very short sentences exhibiting atypical animacy. ## conclusions although animacy manifests only indirectly in english-it is not morphologically marked-pretrained lms demonstrate relatively good animacy processing abilities. they both respect typical animacy and adapt to atypical animacy at close-tohuman levels, although differences remain. they also demonstrate some ability to adapt to atypical animacy even when indicated by a very short context. lm adaptation still lags behind that of humans, but large models increasingly shrink the gap. we conclude that in the scenarios we test, lms respond to animacy like humans do; however, our behavioral methodology can yield no conclusions about how models achieve this. related work on world models @xcite suggests that by using causal techniques to search for internal structure in models, future work could not only demonstrate that lms respond well to animacy, but also explain how they do so.