ADEPT : An Adjective-Dependent Plausibility Task (2021)

Abstract: A false contract is more likely to be rejected than a contract is, yet a false key is less likely than a key to open doors. While correctly interpreting and assessing the effects of such adjective-noun pairs (e.g., false key) on the plausibility of given events (e.g., opening doors) underpins many natural language understanding tasks, doing so often requires a significant degree of world knowledge and common-sense reasoning. We introduce ADEPT – a large-scale semantic plausibility task consisting of over 16 thousand sentences that are paired with slightly modified versions obtained by adding an adjective to a noun. Overall, we find that while the task appears easier for human judges (85% accuracy), it proves more difficult for transformer-based models like RoBERTa (71% accuracy). Our experiments also show that neither the adjective itself nor its taxonomic class suffice in determining the correct plausibility judgement, emphasizing the importance of endowing automatic natural language understanding systems with more context sensitivity and common-sense reasoning.

## introduction discerning the varying effects of adjectival modifiers on the reading of a sentence is critical in a variety of tasks involving natural language understanding. consider the following examples: the reading of these sentences with and without the modifier dead is notably different. the plausibility judgement of the event where a monkey turns on a light switch decreases when the adjectival modifier dead is added, while in the 1b or 1c examples, adding the same modifier leads to no change or an increase in event plausibility, respectively. this observation has important ramifications for many nlp applications like information extraction (ie) and recognizing textual entailment (rte), where solutions have often relied on normative rules that group the effects of adjectives according to either the adjective or its taxonomic class @xcite @xcite . these taxonomies distinguish adjectives like false, dead, alleged (non-subsective) from others like red, large, or valid (subsective). specifically, while the 1a example may influence systems to adopt the rule that adding a nonsubsective adjective like dead to a noun leads to a decrease in plausibility, the other examples suggest a conflicting rule. distinguishing the effects of different adjectives (beyond just their denotation) may thus require common-sense and world knowledge. powerful, massively pre-trained language models (lms) have pushed the performance on various natural language understanding benchmarks to impressive figures; transformer architectures including bert and roberta are believed to perform at near human-level performance on a number of natural language inference (nli) tasks @xcite , while the recently proposed deberta, which builds upon the former two architectures, performs at state-of-the-art on mnli, rte, qnli and wnli @xcite . it is however unclear whether the complex effects of the classes of modifiers exampled above are captured by the competing models given their sparsity in both the corpora and existing nli benchmarks. to examine the ability of lms to capture and distinguish the effects of adjectives on events plausibility, we present a challenge task formulated as a plausibility classification problem consisting of sentence pairs with and without inserting possible adjectives. we do so to understand the strengths and weaknesses of lms that have led to state-ofthe-art performance in downstream nli-tasks. we introduce a novel plausibility task: using automated mechanisms to extract, filter and construct natural sentences, we create adept-a large human-labeled semantic plausibility task consisting of 16 thousand pairs of sentences that differ only by one adjective added to a noun, and designed to resist the statistical correlations that might underpin modern distributional lexical semantics. 1 we show that transformer-based models are not yet adept at adept: our findings suggest performance gaps between humans and large language representation models on adept, which appears to be in large part due to the models' insensitivity to context, indicating an important area for their improvement. we show that the effect of adjectival modifiers on event plausibility is context dependent: we quantify the degree to which plausibility judgements vary for the same adjective and taxonomic class, finding that rules based only on the adjective or its denotation are insufficient when assessing the plausibility readings of events. for example, in our task, the non-subsective adjective like dead led to a decrease in events plausibility as frequently as it led to no change at all. building on prior work showing that normative rules are often broken for subsective adjectives @xcite , we inves-tigate possible effects across all types of adjectives, beyond just the taxonomical categories. the scope of our analysis also goes beyond entailment effects, examining the effects on plausibility, which can be seen as both complimentary and even an extension to entailment tasks. this classification scheme has been adopted for many nlp applications including ie and rte @xcite @xcite . for rte, inference rules were developed according to whether the adjective was non-subsective or not. for ie, non-subsective adjectives were treated as special cases for extracting open ie relations @xcite . we show that there is also a relation between an adjective and the plausibility of the rest of the clause, even for subsective adjectives. this has direct implications for the extraction of generalizable abstract knowledge that can be extracted from a corpus. aspects of this classification scheme have since been challenged, resulting in efforts to either expand on its definitions or abandon the taxonomy altogether. del pinal (2015) suggests that the meaning of certain nouns are only partially modified by non-subsective adjectives (e.g., only the functional features are modified), while @xcite tackle the categorization problem with a statistical approach focused on the proportion of properties shared by the noun and the adjective noun pair. even more recently, inference rules relying on the original taxonomy were observed not to be without exceptions; @xcite used human annotators to highlight cases where the deletion of non-subsective adjectives from a sentence does not necessarily result in non-entailment. these on-going examinations and revisions underpin a profound linguistic phenomenon of mutual dependence: while adjectives play a crucial role in the correct interpretation of a sentence context, the context words are just as instrumental in determining the effect of an adjective; resulting in a number of exceptions to taxonomically-based rules. inspired by this, our work explores the broader question of how dependent the effect of any adjective (beyond their taxonomical class) is on the interpretation of a sentence. for this, we frame our exploration in terms of changes in the plausibility of events, which we believe it can be seen as an extension to entailment. ## the task: adept we develop adept, a semantic plausibility task that features over 16 thousand instances consisting of two sentences, where the second sentence differs from the first only by the inclusion of an adjectival modifier. examples of these instances are in formally, given the original sentence s and the modified sentence s , s is identical to s except for the addition of an adjective a before the root noun of the original sentence. the task is to assess the plausibility difference in the reading of s versus that of s. the possible plausibility ratings are: noun-amod extraction: clean up raw text, and use syntactic parsing to extract noun-adjectival modifier pairs. noun filtering: create entries corresponding to unique nouns and adjectival modifiers, keeping only items that correspond to valid and appropriate english words. predicate extraction + filtering: using conceptnet, we extract predicates for nouns under predefined relations, creating {noun, adjectivelist, predicate-list} triples as entries and filtering out entries with explicit or ungrammatical predicates. ## conclusions we present a new large-scale corpus and task, adept, for assessing semantic plausibility. our corpus contains over 16 thousand difficult task instances, specifically constructed to test a system's ability to correctly interpret and reason about adjective-noun pairs within a given context. our experiments suggest a persistent performance gap between human annotators and large language representation models, with the later exhibiting a lower sensitivity to context. finally, our task provides deeper insight into the effects of various classes of adjectives on event plausibility, and suggests that rules based solely on the adjective or its denotation do not suffice in determining the correct plausibility readings of events. in the future, we wish to investigate how adept could be used to improve performance on related natural language inference tasks (e.g. mnli, snli & scitail @xcite ). we also plan to develop new models on adept and transfer them to other semantic plausibility tasks.