Who did what to Whom? Language models and humans respond diversely to features affecting argument hierarchy construction (2022)

Abstract: Pre-trained transformer-based language models have achieved state-of-the-art performance in many areas of NLP. It is still an open question whether the models are capable of integrating syntax and semantics in language processing like humans. This paper investigates if models and humans construct argument hierarchy similarly with the effects from telicity, agency, and individuation, using the Chinese structure “NP1+BA/BEI+NP2+VP”. We present both humans and six transformer-based models with prepared sentences and analyze their preference between BA (view NP1 as an agent) and BEI (NP2 as an agent). It is found that the models and humans respond to (non-)agentive features in telic context and atelic feature very similarly. However, the models show insufficient sensitivity to both pragmatic function in expressing undesirable events and different individuation degrees represented by human common nouns vs. proper names. By contrast, humans rely heavily on these cues to establish the thematic relation between two arguments NP1 and NP2. Furthermore, the models tend to interpret the subject as an agent, which is not the case for humans who align agents independently of subject position in Mandarin Chinese.

## introduction pre-trained transformer-based language models (lms) keep achieving state-of-the-art performance in nlp tasks. many studies have indicated that pre-trained lms can learn syntactic knowledge (e.g., @xcite for subject-verb agreement, @xcite for filler-gap dependencies, @xcite for garden-path effects) and semantic knowledge (e.g., @xcite for telicity , @xcite for causality bias, @xcite for semantic priming, @xcite for typicality, ettinger 2020 for role reversal and same-category distinctions). however, to what extent lms can acquire knowledge in the syntax-semantics interface is still an open question. to answer this question, we explore arguments hierarchy construction which identifies the thematic roles of arguments in the semantic domain and aligns arguments and subject/object in the syntactic domain. in this hierarchy, the active, controlling agent (prototypical actor) outranks the affected patient (prototypical undergoer), i.e., who did what to whom? @xcite @xcite . the mapping between thematic roles (agent/patient) and syntactic structure (subject/object) varies depending on various features. in this paper, we investigate whether pre-trained transformer-based lms and humans behave similarly in the argument hierarchy construction using the chinese structure "np1+ba/bei+np2+vp". this structure provides a unique opportunity to examine the alignment through the occurrence of ba/bei @xcite , without interference from morphology or word order. for example, human name zhang-san (np1) in the subject position of sentence (1a) with ba is interpreted as an agent, and human name li-si (np2) in the object position is viewed as a patient. by contrast, if bei occurs as in (1b), subject zhang-san is viewed as a patient, and object li-si is considered an agent. this inverse interpretation depending on ba/bei allows us to use word prediction to study lms without taskspecific fine-tuning. it also avoids tokenization issues since both ba and bei are single characters. the construction of argument hierarchy can be affected by different cues related to telicity, agency, and individuation via notion transitivity @xcite @xcite . for example, a cue emphasizing the agentive property of np1 (e.g., by adding the adverbial volitionally) increases the probability of np1 being viewed as an agent @xcite , making ba more natural than bei. by contrast, a cue denoting the non-agentive property of np1 (e.g. by adding the clause what happend to np1 was that...) decreases the probability of np2 being viewed as an agent, making bei more natural than ba. to examine the effects of these cues, we carry out a human acceptability judgment experiment using sentences with ba/bei and compare the result with the probability of masked token ba/bei predicted by the six pre-trained transformer-based lms: bert-base, electra-large, roberta-base, ernie 1.0, and macbert-base/large. the results show that the models and humans construct similar argument hierarchy with atelic feature, and both agentive and non-agentive feature in telic context. however, (a) lms show insufficient sensitivity to the pragmatic function of bei in forming adversative passives with disposal verbs, but humans depend on it in establishing thematic relation between the arguments. ## materials we prepare a dataset including the sentences highlighting telicity-, agency-, and individuationrelated features. to avoid gender effect, we choose frequently used male surnames and first names to form np1 and np2 in the structure "np1+ba/bei+np2+vp". for each condition, we make a hypothesis about human judgment in ba/bei-preference based on previous studies about features in the structure. ## results average b lm and b hum are visualized in figure 1 . b lm is averaged for every condition within each lm. b hum is averaged over all the participants for every condition. we further examine average accep and average surp for ba and bei from roberta-base for each condition in figure 2 (other lms present similar results, see figure 5 in appendix). the human accep for all items in each condition show a lower averaged coefficient of variation over all the conditions than surp of 7 this non-exclusivity is also verified in our study by the result of higher human acceptability for both ba and bei in telic-condition than in atelic-condition, see figure 2 . all lms (0.42 vs. 0.64, detailed results see figure 4 in appendix). statistically, the temporally telic and spatially telic context in all the conditions except for telicand np2 com -condition show quite consistent pattern regarding the ba/bei-preference in both human accep and surp of lms, suggesting that the difference between temporally telic and spatially telic context play a limited role in the ba/bei-preference for these conditions. thus we compare the results between temporally telic and spatially telic context only for telicand np2 comcondition. the human accep and surp of each lm for each condition are fitted with a linear mixed-effects model using the lme4 package in r @xcite . the model treated variable ba/bei as a fixed effect with a random intercept for each verb (detailed results see ## discussion this study compares lms and human behavior in argument hierarchy construction. the results show that lms and humans perform more similarly with atelic feature than with telic feature. in telic context, lms and humans show similar behaviour with (non-)agentive features, but differently with individuation-related features. we discuss these (dis)similarities from the following four perspectives. lms rely on non-durative property to construct argument hierarchy in a telic context. in telic-condition, spatially telic sentences with adverb dao 'arrive' (like 3a) signal non-durative events and show a consistent preference for all lms, while temporally telic sentences (like 3b) describe durative events and display an inconsistent preference among the lms. a previous study has suggested that non-duration plays a crucial role for lms to make telic interpretation @xcite . our results further develop the importance of nondurative property: lms rely more strongly on the non-durative property (compared to durative property) to construct a consistent argument hierarchy in a telic context. lms lack sufficient sensitivity in pragmatic function to make the human-like prediction. bei has a specific pragmatic function in forming adversative passives which express undesirable, unfortunate events @xcite @xcite and often comes with disposal verbs denoting unfavorable meaning like piping 'criticize' and da 'hit' @xcite @xcite . the majority of the temporally telic sentences (7 out of 9) contain disposal verbs whose close connection with bei may directly contribute to the human bei-preference in the temporally telic-condition. the pragmatic function of bei may also increase human bei-preference for happen-condition. the verb fa-sheng 'happen' has a negative prosody (i.e., is likely to occur in a negative context) @xcite @xcite , making bei natural to occur in happen-condition in our results. however, lms fail to show sensitivity in this pragmatic function of bei, as no human-like preference is found for both temporally telicand happen-condition. our results are in line with previous study that pre-trained transformer-based lms have shortage in acquiring pragmatic knowledge @xcite . lms are inclined to interpret the subject as an agent in a spatially telic context. as both np1 and np2 are proper nouns, humans show high acceptability of both ba and bei in a spatially telic context. it indicates that participants do not interpret argument hierarchy based on the linear position of arguments, at least in mandarin chinese @xcite , that is, the sequence subject-verb-object does not determine the argument assignment. however, lms show a clear preference for ba in a spatially telic context where both nps are common nouns (c cl /c cl -condition) or proper names (telic-condition), indicating that lms intend to interpret the subject in the telic context as an agent. this ba-preference in lms may be explained by 1) unbalanced occurrences between active and passive voice, as more active sentences increase the probability of subjects interpreted as agents, and 2) a higher occurrence frequency of ba over bei during training. the occurrence frequencies of active/passive and ba/bei in the lms' training corpus worth further investigation. individuation degree plays a different role between lms and humans in spatially telic context. proper names have a higher degree of individuation than common nouns. a proper name is more likely to function as an agent than a common np @xcite , which agrees with the results in spatially telic context for humans: 1) ba-preference in np2 com -condition and 2) high acceptability of bei in np1 com -condition foot_5 . however, lms show an opposite tendency in viewing a common np in spatially telic context as an agent through ba-preference for np1 comcondition for all lms. the follow-up study in spatially telic context further confirms the agentive interpretation of common nouns in lms. lms fall short to interpret proper names as agents, which may be attributed to their low occurrence frequency during training. moreover, almost each character in proper names has separate semantic meanings. we use zhang-wei as an example. zhang is usually used as a classifier for flat objects like table and paper and wei forms a number of adjectives meaning great and grand. therefore, lms may have difficulty in interpreting the combination of these characters as human names (lake and murphy, 2021; yu and ettinger, 2020). ## future work note that telic predicates in the agency-and individuation-related conditions are necessary to build items in the chinese structure "np1+ba/bei+np2+vp" @xcite , which is also verified by the high acceptability of ba and bei in telic-condition (low judgment scores in figure 2(a) ) in our experiment. future work could continue to explore lms' sensitivity to agency-and individuation-related features isolated from telic context in syntax-semantics-interface. moreover, as we treat lms as a whole and pay attention to their final predictions of ba/bei to compare with human judgment in our study, more probing measures, such as attention probing, could be taken to deepen our understanding about internal performance of lms. ## conclusion this study uses ba/bei-preference in the chinese structure "np1+ba/bei+np2+vp" to examine if pre-trained transformer-based language models construct similar argument hierarchy like humans, i.e., the interpretation of who did what to whom, with the effect of telicity-, agency-and individuation-related features. the results show that lms and humans behave similarly for atelic and non-agentive/agentive features, but differently to telic and individuation-related features in the tested context. specifically, their discrepancy in the temporally telic context suggests that unlike humans, lms lack sufficient sensitivity to pragmatic function of bei describing undesirable events with disposal verbs. the different ba/bei-preference in the sentences with human common vs. proper nouns between lms and humans indicates that unlike humans who perceive proper nouns as agents, lms tend to interpret common nouns as agents. yiyun zhao, jian gang ngui, lucy hall hartley, and steven bethard. 2021. do pretrained transformers infer telicity like humans? in proceedings of the 25th conference on computational natural language learning, pages 72-81.