A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement (2023)

Abstract: We present a novel dataset for physical and abstract plausibility of events in English. Based on naturally occurring sentences extracted from Wikipedia, we infiltrate degrees of abstractness, and automatically generate perturbed pseudo-implausible events. We annotate a filtered and balanced subset for plausibility using crowd-sourcing, and perform extensive cleansing to ensure annotation quality. In-depth quantitative analyses indicate that annotators favor plausibility over implausibility and disagree more on implausible events. Furthermore, our plausibility dataset is the first to capture abstractness in events to the same extent as concreteness, and we find that event abstractness has an impact on plausibility ratings: more concrete event participants trigger a perception of implausibility.

## introduction the ability to discern plausible from implausible events is a crucial building block for natural language processing (nlp). most previous work on modelling plausibility however focuses on the kinds of semantic knowledge necessary for distinguishing a physically plausible event from an implausible one @xcite . as illustrated in fig. 1 , the current study extends the traditional focus to discern physically plausible events such as cat-eat-sardine from physically implausible ones such as rain-breakbelly. furthermore, while recent datasets include some events with conceptually abstract participants @xcite , as to our knowledge no dataset nor model up to date has paid attention to the interaction of event plausibility and abstractness of the involved concepts. we propose to systematically examine plausibility across levels of abstractness, and distinguish between abstractly plausible events such as lawprohibit-discrimination and abstractly implausible ones such as humour-require-merger. we hypothesize that (i) plausible vs. implausible events can be captured through physical vs. abstract levels, and that (ii) integrating degrees of abstractness into events fosters the understanding and modelling of plausibility (cf. fig. 1 ). we start out with a set of attested, i.e., plausible, natural language events in form of s-v-o triples from the english wikipedia, assign abstractness ratings to event participants, and partition the triples into bins with varying degrees of abstractness. we then automatically generate pseudo-implausible event triples and assign degrees of abstractness in a similar way. to obtain human plausibility ratings for each event triple, we conduct a crowd-sourcing annotation study. we collect and validate a total of 15,571 judgements amounting to an average of 8.9 ratings for 1,733 event triples. human intuition regarding the assessment of plausibility is, however, incredibly multi-faceted, highly individual, and not easily reproducible automatically @xcite . in particular, boundaries between categories to be annotated or predicted might not necessarily be strictly true or false, i.e., either plausible or implausible, thus reflecting the true underlying distribution of nondeterministic human judgements with inherent disagreement about labels @xcite . over the past decade, a growing body of work has emphasized the need to incorporate such disagreement in nlp datasets to reflect a more realistic and holistic picture across nlp tasks @xcite @xcite @xcite . accordingly, we argue for the necessity to preserve and examine disagreement when annotating and modelling plausibility, and represent inherent disagreement in annotation in order to devise a range of silver standards for analysis and modelling. more specifically, we disentangle subjectivity from annotation error, limitations of the annotation scheme, and interface @xcite , and examine disagreements in physical and abstract plausibility annotation. overall, we find that our annotators tend to favor plausibility over implausibility, and we observe stronger disagreements for implausible in comparison to plausible events. second, we explore the impact of abstractness on plausibility ratings. here, our results reveal a positive relation between plausibility and events consisting of more abstract words, while implausibility is mostly found in predominantly concrete events. ## construction of event targets our first goal is to create a dataset foot_3 that systematically (a) covers both plausible event triples that are selectionally preferred or unusual, (b) captures events attested in the real world, i.e., extracted from triples produced in natural language, (c) measures plausibility on a degree scale from plausible to implausible, and (d) puts equal emphasis on both abstractly and physically plausible events. we visualize the dataset construction process in fig. 2 . ## human annotation our second goal targets the annotation of the collected event triples with respect to subjective assessments of plausibility on a degree scale (1-5) ranging from implausible to plausible. for this, we perform a human annotation study. ## final dataset: aggregations to foster learning with and from disagreement, we release not only (i) the raw annotator ratings, but also (ii) provide the following standard aggregations to enable various perspectives for interpretation and modelling; for further aggregation options see e.g., @xcite . we account for both multi-class (label p t1, 2, 4, 5u) and binary (label either plausible p t4, 5u or implausible p t1, 2u) categorizations. the dataset is available at https://github.com/anneroseeichel/pap . ## discussion we formulated the task of automatically distinguishing abstract plausible events from implausible ones as an extension of @xcite who focused specifically on physical plausible events. based on the presented findings, we affirm our hypothesis as to (i) whether plausible and implausible events can be systematically captured on physical and abstract levels by (ii) integrating degrees of abstractness for combinations of words. we further note differences in collected annotations with assignment likelihood of plausible ratings increasing with abstractness of events' constituents, while concreteness seems to facilitate the detection of more implausible events. we hypothesize that more concrete words evoke a more stable mental image grounded in the real world. events like our introductory example rain-breaksbelly that represent a violation of quite fixed mental images are thus more often recognized as implausible. in contrast, more abstract words that lack a tangible reference object seem to open up a greater space of potentially plausible interpretations. this possibly invites annotators to cooperate and use their imagination resulting in more plausible ratings for more abstract triples. our findings further suggest that it is the recipient who comes up with an interpretation, thus making sense of the seemingly implausible. moreover, generating fully implausible events is not trivial, which should be taken into account when using automatically generated implausible triples. lastly, while events based on s-v-o triples or comparably simple constructions have been successfully leveraged for exploring selection preference and thematic fit @xcite @xcite , the addition of context exceeding sentences constructed from s-v-o triples could potentially resolve present ambiguity and possibly reduce disagreement. we thus encourage future work extending this work by collecting and analyzing plausibility ratings for more complex constructions within broader contexts. ## conclusion we presented a novel dataset for physical and abstract plausibility for events in english. based on naturally occurring sentences extracted from wikipedia, we infiltrated degrees of abstractness, and automatically generated perturbed pseudoimplausible events. we annotated a filtered and balanced dataset for plausibility using crowd-sourcing and performed extensive cleaning steps to ensure annotation quality. we provided in-depth analyses to explore the relationship between abstractness and plausibility and examined annotator disagreement. we hope that the presented dataset is used for both analyzing and modelling the notion of plausibility as well as the exploration of closely related tasks such as selectional preference and thematic fit and relevant downstream tasks including commonsense reasoning, nli, and coreference resolution. moreover, we make both raw annotations and a range of aggregations publicly available to foster research on disagreement and enable interpretation from various perspectives.