P enguins Don’t Fly: Reasoning about Generics through Instantiations and Exceptions (2023)

Abstract: Generics express generalizations about the world (e.g., birds can fly) that are not universally true (e.g., newborn birds and penguins cannot fly). Commonsense knowledge bases, used extensively in NLP, encode some generic knowledge but rarely enumerate such exceptions and knowing when a generic statement holds or does not hold true is crucial for developing a comprehensive understanding of generics. We present a novel framework informed by linguistic theory to generate exemplars—specific cases when a generic holds true or false. We generate ~19k exemplars for ~650 generics and show that our framework outperforms a strong GPT-3 baseline by 12.8 precision points. Our analysis highlights the importance of linguistic theory-based controllability for generating exemplars, the insufficiency of knowledge bases as a source of exemplars, and the challenges exemplars pose for the task of natural language inference.

## introduction generics express generalizations (e.g., birds can fly) that allow humans to reason and act with incomplete world knowledge @xcite . generics allow us to draw plausible inferences about individuals (e.g., polly is a bird, so polly can fly) even when we know of counterexamples (e.g., penguins cannot fly). despite the utility of generalizations, knowledge of counterexamples is necessary for modeling generics and effectively reasoning with them in computational systems. recent studies of generics (e.g., @xcite and commonsense kbs (e.g., @xcite provide repositories of generic knowledge. however, these resources rarely mention exceptions (i.e., counterexamples) or instantiations (i.e., cases where the generic holds); collectively exemplars. for systems using these resources as a source of world knowledge, such incomplete information can xp (1) : penguins can't xp (2) : sparrows can't xp (i) : sparrows can xp (j) : birds can # gpt2-xl neurologic a*esque @xcite & constraints is a is a generator figure 1 : we present exemplars generator: given a generic like "birds can fly" it generates truthful statements where the generic does (instantiations) and does not (exceptions) hold. we extract commonsense knowledge (e.g., from conceptnet @xcite ) in linguistically-informed prompts and constraints for constrained generation @xcite . we use trained discriminators to filter for quality. in our work, we unify two distinct linguistic theories and use linguistic theory-guided decoding to generate exemplars. although large-scale neural language models such as @xcite have been increasingly successful in fewshot text generation tasks, such generation is both expensive and not easily controllable. therefore, we instead use the constrained generation algorithm neurologic a ⋆ esque @xcite , which can be applied to any auto-regressive language model; we choose to use gpt-2 @xcite . in this manner, we generate 12562 instantia-tions and 6297 exceptions for ∼650 generic statements from @xcite . we conduct human evaluation of the generations and show that our system outperforms few-shot generation by gpt-3 by 12.8 precision points. our analyses demonstrate not only the importance of linguistic modeling for generating exemplars and the insufficiency of kbs as a source of exemplars, but also the challenges exemplars pose for natural language reasoning. our contributions are as follows: commonsense knowledge while large-scale ckbs capture a range of commonsense knowledge @xcite @xcite , they contain necessarily incomplete (i.e., the open-world assumption @xcite ) general knowledge. furthermore, although recent works have created kbs specifically of generics @xcite and proposed methods to identify generics in text @xcite , they do not identify or model exem-plars. in our work, we focus directly on automatically generating exemplars, providing richer commonsense knowledge. the application of generics to specific individuals is influenced by prototypicality @xcite , with small sets of prototypical norms collected in cognitive science for a range of kinds @xcite @xcite . however, recent work has shown that neural models have only moderate success at mimicking human prototypicality @xcite or producing commonsense facts without guidance @xcite . hence, we combine neural models with a kb of concepts, using linguistic-theory-guided decoding, to generate generics exemplars. reasoning reasoning with generics is closely related to non-monotonic reasoning @xcite ; specifically default inheritance reasoning @xcite @xcite @xcite @xcite . contrary to the proposed solutions for linguistic tests on default inheritance reasoning @xcite , e.g.,can a conclusion about inheritance be inferred based on provided evidence?), later works showed that the presence of generics exemplars in the evidence impacts what humans perceive as the correct answer @xcite @xcite . these results highlight the importance of identifying generics and accurately modeling their relationships in machine reasoning. while natural language inference (nli), a form of deductive reasoning well-studied in nlp (i.a., dagan et al. (2013); bowman et al. (2015) ), captures notions of inference, studies on nonmonotonic reasoning and nli are limited @xcite @xcite and do not include default inheritance reasoning. therefore, in this work we analyze the interactions between generics exemplars and nli and highlight the importance of modeling this relationship in machine reasoning. ## framework for exemplars a generic statement describes a relation between a concept and a property. usually, a concept k is a type or kind (e.g., bird) while a property p in our framework, we first categorize ( §3.1) generics and derive their logical forms ( §3.2). these logical forms for generics serve as our basis for formulating exemplars ( §3.3) and designing templates suitable for generation ( §3.4). ## methodology we propose a pipeline system to automatically generate generics exemplars. our system takes as input a generic g and the templates derived from its category foot_4 ( §3.4), and outputs a set of generated exemplars (fig. 2 ). first, the system assembles and populates the templates according to the input generic ( §4.1, fig. 2 ). then, the filled templates are converted into prompts and constraints that control the generation decoding process ( §4.2). finally, the output is filtered to remove non-viable ( §4.3, fig. 2 ) or irrelevant ( §4.4, fig. 2 ) exemplars. ## experiment details we discuss our experimental setup and specify full hyperparameters in appendix c. ## evaluation using our computational framework, we generate 18859 exemplars for 653 generics ( ## conclusion in this work, we draw on insights from linguistics to propose a novel computational framework to automatically generate valid exemplars for generics, as a step towards capturing the nuances of human reasoning for generics. our system generates ∼19k exemplars for 653 generics and outperforms gpt-3 at generating viable examples, while remaining more controllable. we also demonstrate the limitations of ckbs and the importance of explicit linguistic modeling in generating ex-emplars. that is, the importance of linguistictheory-based decoding and semantics-based filtering with nli. finally, we highlight the inability of current nli models to reason about and represent the default-inheritance-reasoning relationship between generics and exemplars. to all possible generics in all languages. in particular, our system does not handle generics where valid instantiations include negating ( §3.2) the concept. this is due to the restriction that most english generation is left-to-right and it is not possible to define a closed set of possible concept negations for the prompt. in this work, we do not generate exemplars for generics involving human referents (e.g., professions, nationalities). we exclude generics involving human referents to mitigate the risk of generating socially biased exemplars or harmful stereotypes (e.g., "black folks go to jail for crimes" for the generic "people go to jail for crimes"). additionally, handling of human stereotypes require methods that are beyond the scope of this paper. for example, a socially-aware exceptions to a generic like "girls wear dresses" would be "boys wear dresses, too". this would require the understanding of the possible subtext of such a statement (e.g. "only girls wear dresses"), which is beyond the current capabilities of this study and worthy of future exploration. finally, we note that while it is not the intended purpose of our system, a malicious user could still use our system to generate exemplars for a generic involving a person and propagate potentially harmful social biases.