Modeling Event Plausibility with Consistent Conceptual Abstraction (2021)

Abstract: Understanding natural language requires common sense, one aspect of which is the ability to discern the plausibility of events. While distributional models—most recently pre-trained, Transformer language models—have demonstrated improvements in modeling event plausibility, their performance still falls short of humans’. In this work, we show that Transformer-based plausibility models are markedly inconsistent across the conceptual classes of a lexical hierarchy, inferring that “a person breathing” is plausible while “a dentist breathing” is not, for example. We find this inconsistency persists even when models are softly injected with lexical knowledge, and we present a simple post-hoc method of forcing model consistency that improves correlation with human plausibility judgements.

## introduction of the following events, a human reader can easily discern that (1) and (2) are semantically plausible, while (3) is nonsensical. ## related work while plausibility is difficult to define precisely, we adopt the following useful distinctions from the literature: the state of the art is to take the conditional probability of co-occurrence as estimated by a distributional model as an approximation of event plausibility @xcite . our fine-tuned roberta baseline follows this approach. similar in spirit to our work, he et al. (2020) extend this baseline method by creating additional training data using the probase taxonomy @xcite in order to improve conceptual generalization; specifically, for each training example they swap the event's arguments with its hypernym or hyponym, and they take this new, perturbed example to be an implausible event. there is also recent work focusing on monotonic inferences in semantic entailment @xcite @xcite . plausibility contrasts with entailment in that plausibility is not strictly monotonic with respect to hypernymy/hyponymy relations: the plausibility of an entity is not sufficient to infer the plausibility of its hyponyms (i.e., not downward entailing: it is plausible that a person gives birth but not that a man gives birth) nor hypernyms (i.e., not upward entailing: it is plausible that a baby fits inside a shoebox but not that a person does). non-monotonic inferences have recently been explored in the context of defeasible reasoning @xcite : inferences that may be strengthened or weakened given additional evidence. the change in plausibility between an event and its abstraction can be formulated as a type of defeasible inference, and our findings may contribute to future work in this area. ## problem formulation given a vocabulary of subjects s, verbs v, and objects o, let an event be represented by the s-v-o triple @xmath0. we take g to be a ground-truth, total ordering of events expressed by the ordering function g(e) > g(e ) iff e is more plausible than e . our objective is to learn a model f : s × v × @xmath1, i.e., g(e) > g(e ) =⇒ f (e) > f (e ). this simplification follows from previous work @xcite , and the plausibility score for a given triple can be considered the relative plausibility of the respective event across all contexts and realizations. while meaning is sensitive to small linguistic perturbations, we are interested in cases where one event is more plausible than another marginalized over context. consider that person-breathe-air is more plausible than thought-breathe-car regardless of the choice of determiners or tense of the verb. in practice, we would like to learn f without supervised training data, as collecting a sufficiently large dataset of human judgements is prohibitively expensive @xcite , and supervised models often learn dataset-specific correlations @xcite @xcite . therefore, we train model f with distant supervision and evaluate by correlation with human ratings of plausibility which represent the ground-truth ordering g. ## models the models that we consider are all of the same general form. they take as input an event and output a relative plausibility score. ## training models are all trained with the same objective to discriminate plausible events from less plausible ones. given a training set d of event pairs (e, e ) where e is more plausible than e , we minimize the binary cross-entropy loss in practice, d is created without supervised labels. for each (e, e ) ∈ d, e is an event attested in a corpus with subject s, verb v, and object o. e is a random perturbation of e uniformly of the form (s , v, o), (s, v, o ), or (s , v, o ) where s and o are arguments randomly sampled from the training corpus by occurrence frequency. this is a standard pseudo-disambiguation objective. our training procedure follows recent works that learn plausibility models with self-supervised fine-tuning @xcite @xcite . for the models that use wordnet, we use a filtered set of synsets: we remove synsets with a depth less than 4, as these are too broad to provide useful generalizations @xcite . we also filter out synsets whose corresponding lemma did not appear in the training corpus. the wordnet models also require sense disambiguation. we use the raw triple as input to bert-wsd (yap et al., 2020) which outputs a probability distribution over senses. we take the argmax to be the correct sense. we train all models with gradient descent using an adam optimizer, a learning rate of 2e-5, and a batch size of 128. we train for two epochs over the entire training set of examples with a linear warm-up of the learning rate over the first 10,000 iterations. fine-tuning roberta takes five hours on a single nvidia v100 32gb gpu. fine-tuning conceptinject takes 12 hours and concept-max 24 hours. ## predicting human plausibility judgements we evaluate models by their correlation with human plausibility judgements. each dataset consists of events that have been manually labelled to be plausible or implausible ( ## consistency evaluation for every event e in the evaluation sets of human plausibility judgments ( §6), we disambiguate e using bert-wsd and then calculate models' estimates for the plausibility of every possible abstraction of e (figure 4 ). based on these estimates, we can analyze the consistency of each model across abstractions. ## conclusion while the state of the art in modeling plausibility has improved in recent years, models still fall short of human ability. we show that model estimates are inconsistent with respect to a lexical hierarchy: they correlate less with human judgments as compared to model estimates that are forced to be consistent, and they do not satisfy our intuitively defined quantitative measures of consistency. in addition, we show that simply injecting lexical knowledge into a model is not sufficient to correct this limitation. conceptual consistency appears to require a more discrete, hierarchical bias. interesting questions for future work are: 1) can we design a non-monotonic, consistent model of plausibility that better correlates with human judgements? 2) can we induce a hierarchy of abstractions rather than using a manually created lexical hierarchy?