Commonsense Reasoning for Natural Language Processing (2020)

Abstract: Commonsense knowledge, such as knowing that “bumping into people annoys them” or “rain makes the road slippery”, helps humans navigate everyday situations seamlessly. Yet, endowing machines with such human-like commonsense reasoning capabilities has remained an elusive goal of artificial intelligence research for decades. In recent years, commonsense knowledge and reasoning have received renewed attention from the natural language processing (NLP) community, yielding exploratory studies in automated commonsense understanding. We organize this tutorial to provide researchers with the critical foundations and recent advances in commonsense representation and reasoning, in the hopes of casting a brighter light on this promising area of future research. In our tutorial, we will (1) outline the various types of commonsense (e.g., physical, social), and (2) discuss techniques to gather and represent commonsense knowledge, while highlighting the challenges specific to this type of knowledge (e.g., reporting bias). We will then (3) discuss the types of commonsense knowledge captured by modern NLP systems (e.g., large pretrained language models), and (4) present ways to measure systems’ commonsense reasoning abilities. We will finish with (5) a discussion of various ways in which commonsense reasoning can be used to improve performance on NLP tasks, exemplified by an (6) interactive session on integrating commonsense into a downstream task.

## description what is commonsense? the tutorial will start with a brief overview of what commonsense is, how it is defined in the literature, and how hu-mans acquire it @xcite . we will discuss notions of social commonsense @xcite and physical commonsense @xcite . we will cover the differences between taxonomic and inferential knowledge @xcite , and differentiate commonsense knowledge from related concepts (e.g., script learning; @xcite . how to represent commonsense? we will review existing methods for representing commonsense, most of which focus solely on english. at first, symbolic logic approaches were the main representation type @xcite . while still in use today @xcite , computational advances have allowed for more data-driven knowledge collection and representation (e.g., automatic extraction; @xcite @xcite . we will cover recent approaches that use natural language to represent commonsense @xcite , and while noting the challenges that come with using datadriven methods @xcite . what do machines know? pretrained language models (lms) have recently been described as "rediscovering the nlp pipeline" @xcite , i.e. replacing previous dedicated components of the traditional nlp pipeline, starting from low-and mid-level syntactic and semantic tasks @xcite @xcite , to high-level semantic tasks such as named entity recognition, coreference resolution and semantic role labeling @xcite ). we will discuss recent investigations into pretrained lms' ability to capture world knowledge @xcite and learn or reason about commonsense @xcite . how to incorporate commonsense knowledge into downstream models? given that large number of nlp applications are designed to require commonsense reasoning, we will review efforts to integrate such knowledge into nlp tasks. various works have looked at directly encoding commonsense knowledge from structured kbs as additional inputs to a neural network in generation @xcite , dialogue @xcite , qa @xcite @xcite @xcite , and classification @xcite @xcite tasks. for applications without available structured knowledge bases, researchers have relied on commonsense aggregated from corpus statistics pulled from unstructured text @xcite @xcite . more recently, rather than providing relevant commonsense as an additional input to neural networks, researchers have looked into indirectly encoding commonsense knowledge into the parameters of neural networks through pretraining on commonsense knowledge bases @xcite or explanations @xcite , or by using multi-task objectives with commonsense relation prediction @xcite . how to measure machines' ability of commonsense reasoning? we will explain that, despite their design, many natural language understanding (nlu) tasks hardly require machines to reason about commonsense @xcite . this prompted efforts in creating benchmarks carefully designed to be impossible to solve without commonsense knowledge @xcite . in response, recent work has focused on using crowdsourcing and automatic filtering to design large-scale benchmarks while maintaining negative examples that are adversarial to machines @xcite . we will review recent benchmarks that have emerged to assess whether machines have acquired physical (e.g., @xcite , social (e.g., @xcite , or temporal commonsense reasoning capabilities (e.g., @xcite , as well as benchmarks that combine commonsense abilities with other tasks (e.g., reading comprehension; @xcite @xcite . ## prerequisites we will not expect attendees to be familiar with previous research on commonsense knowl-edge representation and reasoning, but participants should be familiar with: