Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge (2021)

Abstract: Prior research has explored the ability of computational models to predict a word semantic fit with a given predicate. While much work has been devoted to modeling the typicality relation between verbs and arguments in isolation, in this paper we take a broader perspective by assessing whether and to what extent computational approaches have access to the information about the typicality of entire events and situations described in language (Generalized Event Knowledge). Given the recent success of Transformers Language Models (TLMs), we decided to test them on a benchmark for the dynamic estimation of thematic fit. The evaluation of these models was performed in comparison with SDM, a framework specifically designed to integrate events in sentence meaning representations, and we conducted a detailed error analysis to investigate which factors affect their behavior. Our results show that TLMs can reach performances that are comparable to those achieved by SDM. However, additional analysis consistently suggests that TLMs do not capture important aspects of event knowledge, and their predictions often depend on surface linguistic features, such as frequent words, collocations and syntactic patterns, thereby showing sub-optimal generalization abilities.

## introduction people can discriminate between typical (e.g., a cop arrested a thief ) and atypical events (e.g., a thief arrested a cop) and exploit this ability in online sentence processing to anticipate the upcoming linguistic input. brains have been claimed to be "prediction machines" @xcite and psycholinguistic research has shown that a crucial ingredient of such predictive ability is the knowledge about events and their typical participants stored in human semantic memory, also referred to as generalized event knowledge (gek) by @xcite . to make an example, if we were asked to think about things that are played with a guitar, we would quickly and more or less unanimously think of words such as song, piece or riff. computational models of predicate-argument typicality, generally referred to as thematic fit in the psycholinguistic literature @xcite , extract typical arguments from parsed corpora. however, gek is not just storing relations between words: the fact that this knowledge is generalized -that is, it is based on an abstract representation of what is typical -allows us to easily classify new argument combinations as typical or atypical. furthermore, psycholinguistic studies @xcite have shown that humans are able to combine and dynamically update their expectations during sentence processing: for example, their expectations given the sequence the barber cut the differ from the ones given the lumberjack cut the , since the integration of knowledge "cued" by the agent argument with the verb will lead to the activation of different event scenarios. in distributional semantics, sophisticated models of the gek have been proposed to make predictions on upcoming arguments by integrating the cues coming from the verb and the previously-realized arguments in the sentence @xcite . since such knowledge is acquired from both first-hand and linguistic experience @xcite , an important assumption of this literature is that, at least for its "linguistic subset", the gek can be modeled with distributional information extracted from corpora @xcite . language models are trained to make predictions given a context, and thus, they can also be viewed as models of gek. this approach is promising if one considers the success of recent transformer-based language models (henceforth tlms), which are trained on huge corpora and contain a massive number of parameters. even if these models receive extensive training and have been shown to capture linguistic properties @xcite , it is not obvious whether they acquire the aspects of gek that have been modeled explicitly in previous approaches. to the best of our knowledge, transformers have never been tested on dynamic thematic fit modeling, nor their performance has been compared with traditional distributional models. our current work is addressing this issue. ## related work in its classical form, the thematic fit estimation task consists in comparing a candidate argument or filler (e.g., wine) with the typical fillers of a given verb role (e.g., agent, patient, etc.), either in the form of exemplars previously attested in a corpus @xcite @xcite or in the form of a vector-based prototype @xcite @xcite @xcite @xcite . additionally, recent studies explored the use of masked language modeling with bert for scoring the candidate arguments @xcite . performance in the thematic fit task is typically measured with the correlation between the output scores of the model and human-elicited typicality judgments for verb-argument pairs @xcite @xcite @xcite . in the simplest and most common version of this task, the typicality of verb argument-pairs is evaluated in isolation. thematic fit is instead a dynamic concept: the expectations for an argument in a given verb role do not depend just on the verb, but also on the compositional combination with the other arguments in the sentence @xcite . to check the ability of computational models to account for the compositional update of argument expectations, @xcite framed the problem as a binary classification task: a system is presented a sentence pair, with one sentence expressing a typical real-world situation (the journalist is checking the report) and the other sentence expressing a plausible but less typical one (the mechanic is checking the report), and the task is to assign a higher thematic fit/typicality score to the former. notice that the two sentences differ only for one argument, and that the "atypical" one might, however, be a common filler with respect to the verb target role (e.g., report is a typical patient for check, it is just less plausible in combination with mechanic as an agent). several models have tried to tackle the "dynamic" version of the thematic fit task, either based on classical distributional spaces @xcite or on more sophisticated neural network architectures @xcite . on the evaluation side, those works made use of the experimental materials of the study by @xcite , which are, however, limited to agentverb-patient triples. the recently-introduced dt-fit dataset @xcite is, in comparison, larger in size and provides more variety of fillers and roles (including instruments, locations and time). other studies introduced larger datasets, but focused on more specific notions of event plausibility (e.g. the plausibility depending on the physical properties of the participants) @xcite @xcite . ## results and analysis in this section, we provide the results of the experiments on the dtfit datasets. since the models cover different portions of the original tuples, we performed the evaluation over the common pairs. at a glance, we observe that both sdm and tlms obtain quite strong correlations, going from 0.46 to a maximum of 0.74 across datasets and models. specifically, we notice that tlms tend to reach higher performances compared to the distributional approach. however, a marginally significant improvement of the correlations over sdm is obtained only for location dtfit (@xmath0.05 for locations, @xmath1.1 for the other roles). 5 this result is interesting, considering that sdm is trained on a really small corpus compared to tlms (for instance, roberta is trained on 160 gb of text). another remark is that even if tlms differ for architecture, training objective and data, bert-large, roberta and gpt-2 tend to achieve very similar performances, while correlation scores are lower for bert-base. as there is no significant difference between sdm and tlms results, we plotted an example of the relationship between the human ratings and the model-derived scores to provide a better picture of the models' predictions. for visualization purposes, we applied a logarithmic transformation to the scores. for agent dtfit , we observe that sdm and bert-large have a different trend. in the former (see figure 1a ), the majority of the points follow a roughly linear relationship, and there is a small variation around the regression line (with few outliers). on the contrary, bert-large scores show more variance (figure 1b ). this trend is confirmed (even if it is less evident) for patient dtfit , where both sdm (figure 1c ) and bert-large (figure 1d ) have a large amount of variance, and quite a few outliers. to verify these observations, we compared the sum of the bert-large residuals with that of sdm (we first normalized the models' scores with min-max scaling in order to make them comparable). for both subsets, the sum of residuals is higher for bert-large than sdm, which is especially the case for agent dtfit (31.43 versus 17.85; 67.04 versus 63.47 for patient dtfit ). finally, we also performed a binary classification task for agent dtfit and patient dtfit . in this case, we evaluated models on their ability to assign a higher score to the filler in the typical condition. as shown in the above results prima facie suggest that tlms are able to model the dynamic interaction between the sentence elements to compute the typicality value of a candidate filler. however, analyzing the errors of the tlms can be revealing of how they make their predictions. furthermore, tlms' natural preference for what is more frequent could help them in the typicality task, as a typical event is often a frequent one. their good performance could be due to the fact that they memorize frequent sequences during training. therefore we tested tlms on a different dataset, in which atypical but physically plausible events (e.g., the cloth erased the cream) are distinguished from atypical and implausible ones (e.g., the cloth erased the house). frequency effects on performance should be alleviated in this setting, as both types of events in the dataset are atypical and, hence, rare. this task requires fine-grained knowledge of the properties of arguments, which is still an important component of gek. additionally, different frequency variations in the training data could influence tlms performance. since the models' knowledge of the world is mediated by language, it is likely that an argument filler may or may not be predicted depending on the frequency of the word chosen to refer to it. we investigated this issue by testing the models on another diagnostic dataset obtained by replacing typical fillers with low-frequency synonyms. the last question we explored is whether tlms can be influenced by the way statements of event typicality are syntactically expressed. so, we evaluated tlms by feeding them with sentences encoding typical events with a transformed and more complex syntactic form than the one used in the dtfit experiments. tuple expected preferred mason mix cement (@xmath2.65, @xmath3.41) soup (@xmath4.95, @xmath5.54) climber climb rock (@xmath6.8, @xmath7.29) staircase (@xmath8.6, @xmath9.05) blacksmith pour metal (@xmath10.5, @xmath11.03) wine (@xmath12.6, @xmath13.6) chemist pour compound (@xmath14.25, @xmath15.4) juice (@xmath16.75, @xmath17.18) cat drink milk (@xmath18.6, @xmath19.89) coffee (@xmath20.45, @xmath21.65) i. tlms seem to prefer frequent collocations, but only when they are plausible. errors reported in to approximate the word distributions encountered in the training data, we extracted lmi values from a 2018 dump of english wikipedia and we evaluated only the bert model (base and large) on the new dataset, as wikipedia is a considerable part of the training only for this model. examples of the new test pairs are the following: the terrorist released the hostage/ album, the truck hit the car/ ball, the soldier heard the command/ case. to evaluate bert performance, we computed the accuracy scores on the diagnostics dataset in the same way as in the main experiment (binary classification task). results show that the models generally assign low probabilities to atypical fillers. they choose the atypical event in some cases (9 in bert-base, 6 in large), but mainly when the contrast between the atypical event and our expectations is less evident (the smuggler sold the property is preferred to weapon, the soldier throw the ball is preferred to bomb). as already observed in the main experiment, bert seems to be able to look beyond salient local associations and build representations of global events flexibly. however, this issue should be further explored for the other roles as well. for instance, given the sentence the engineer completed the project in the , the models must consider more contextual elements to make the correct prediction. on the other hand, even if sdm design aims at capturing this aspect of gek, the manipulations we made in this dataset cause a drop in the model performance (14 pairs out of 31 are classified wrongly). this drop is probably due to aspects of the implementation such as data sparsity. specifically, if there are no events in which the subject occurs with a direct object, the prototype of the patient is built only from the verb's most associated patients, disregarding the fact they are implausible given the whole context. ## conclusions in this paper, we tested transformer-based language models on tasks related to generalized event knowledge. in the main experiment, we evaluated their ability to model event typicality, that is, discern typical from atypical events, on a dataset designed for this task, dtfit. results show that tlms scores positively correlate with human judgments. however, they do not significantly outperform the distributional prototype-based model (sdm) that we selected for comparison. this confirms the ability of sdm to dynamically update the semantic representation of a sentence, which was recently shown for the challenging task of logical metonymy interpretation @xcite . however, we decided to go beyond the simple evaluation against human judgments. we carried out several additional small-scale experiments with the specific aim to understand which factors could affect the predictions of tlms. the results suggest that models are often too dependent on what they observe during training and lack some key aspects of human event knowledge. in particular, we observed that, in some cases, they are unable to compose all elements of the input to make predictions, and they tend to rely more on salient local associations between words. however, further analysis is needed. secondly, their performance drop on the physical plausibility task, which requires the ability to infer physical properties necessary for an object to participate in a given event. lastly, their probabilities are dependent on the specific words that have to be predicted rather than on their meaning, and on the canonical word order in which these words tend to occur. noticeably, even a distributional model of event knowledge (sdm) showed similar limitations, generally likely to be due to data sparsity and inherent limitations of distributional representations obtained from text data. to conclude, we believe that the experiments we reported are the first step towards a deep investigation of "how general" is the generalized event knowledge in computational models. future work might include the creation of a larger version of our diagnostic datasets, in order to make available to nlp researchers a more robust benchmark for tasks related to generalized event knowledge.