Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding (2021)

Abstract: Large-scale, pre-trained language models (LMs) have achieved human-level performance on a breadth of language understanding tasks. However, evaluations only based on end task performance shed little light on machines’ true ability in language understanding and reasoning. In this paper, we highlight the importance of evaluating the underlying reasoning process in addition to end performance. Toward this goal, we introduce Tiered Reasoning for Intuitive Physics (TRIP), a novel commonsense reasoning dataset with dense annotations that enable multi-tiered evaluation of machines’ reasoning process. Our empirical results show that while large LMs can achieve high end performance, they struggle to support their predictions with valid supporting evidence. The TRIP dataset and our baseline results will motivate verifiable evaluation of commonsense reasoning and facilitate future research toward developing better language understanding and reasoning models.

## introduction recent years have seen a surge of research activities toward commonsense reasoning in natural language understanding. dozens of relevant, large-scale benchmark datasets have been developed, and online leaderboards encourage broad participation in solving them. in the last few years, extraordinary performance gains on these benchmarks have come from large-scale language models (lms) pre-trained on massive amounts of online text @xcite @xcite ). today's best models can achieve impressive performance and have surpassed human performance in challenging language understanding tasks, including benchmarks for commonsense inference @xcite @xcite . this rapid period of growth and progress has been an undoubtedly exciting time for nlp. despite these exciting results, it is a subject of scrutiny whether these models have a deep understanding of the tasks they are applied to @xcite . a key concern is widespread bias in language benchmarks leading to superficial correlations between context and class labels @xcite @xcite , allowing systems to bypass reasoning and achieve artificially high performance @xcite . consequently, it remains unclear whether the problems are truly solved, and whether machines can perform verifiable reasoning as humans do. in this work, we first introduce tiered reasoning for intuitive physics (trip), a benchmark targeting physical commonsense reasoning. trip poses a high-level end task for story plausibility classification, a common proxy task for commonsense reasoning problems @xcite @xcite . notably, however, it includes dense annotations for each story capturing multiple tiers of reasoning beyond the end task. from these annotations, we propose a tiered evaluation, where given a pair of highly similar stories (differing only by one sentence which makes one of the stories implausible), systems must jointly identify (1) the plausible story, (2) a pair of conflicting sentences in the implausible story, and (3) the underlying physical states in those sentences causing the conflict. the goal of trip is to enable a systematic evaluation of machine coherence toward the end task prediction of plausibility. in particular, we evaluate whether a high-level plausibility prediction can be verified based on lower-level understanding, for example, physical state changes that would support the prediction. we further present several baseline systems powered by large lms. our empirical results show that while large lms can achieve high end task performance (up to 78% accuracy), they struggle to jointly support their predictions with the proper evidence (only up to 11% of examples supported with correct physical states and conflicting sentences). consequently, the predictions from these powerful systems are overwhelmingly not accountable to their understanding of how the world works. the contributions of this work are the first-ofits-kind dataset trip and new metrics that facilitate quantitative evaluation of coherent reasoning in commonsense language understanding. our detailed analysis by applying large lms on this dataset demonstrates key disconnections between low-level and high-level predictions in the reasoning process. this dataset and our baseline results motivate future work to develop systems that are capable of verifiable language understanding and reasoning. ## a tiered baseline for trip figure 2 displays a high-level view of our proposed baseline system to solve trip. it individually embeds each sentence-entity pair in each story, classifies physical precondition and effect states, then identifies conflicting sentences from these. given a pair of stories, it aggregates conflict predictions for each story to decide which is more plausible. ## experiments using trip, we evaluate several variations of the proposed reasoning system powered by selected pre-trained language models: bert @xcite , roberta @xcite , and de-berta @xcite ). 6 these models offer a range in design choices such as model complexity and size of pre-training data. we begin with an evaluation from the perspective of the end task, then take a detailed look at the lower-level tasks. ## related work physical commonsense. there exist a few nlp datasets around physical commonsense reason-ing which offer various classification tasks. propara @xcite tracks existence and location of entities in each sentence, similar to trip's physical state classification, but in a more restricted state space. physical interaction question answering (piqa) from @xcite provides a similar high-level end task of multiplechoice text plausibility classification targeting physical commonsense. other benchmarks focus on specific domains of physical reasoning, such as temporal reasoning @xcite and spatial reasoning @xcite . visual @xcite and multimodal @xcite @xcite benchmarks also investigate systems' commonsense understanding of the physical world through perception and interaction. different from these existing benchmarks, trip is the first dataset of its kind with dense annotation to support evaluation of verifiable reasoning toward the end task prediction. robust language inference. in the face of statistical bias enabling artificially high performance in nlp models, several works have explored ways to evaluate and enable robust language inference. several probing studies have examined how well surface-level syntactic and semantic phenomena are captured in contextual language embeddings @xcite @xcite @xcite . for stronger evaluation of potentially biased systems, others have explored specialized natural language inference tasks @xcite and logic rules @xcite to support and evaluate consistency of models across instances of the end task. some approaches have been proposed to instead remove biases from language by filtering out data too easily discriminated by state-of-the-art text classifiers @xcite , and to improve robustness of systems against exploiting various types of biases @xcite @xcite . recent work has attempted to compile large amounts of semi-structured commonsense knowledge @xcite and inject this knowledge into pre-trained language models @xcite in order to enable knowledge-supported language understanding and on-the-fly explanation. different from these efforts, this paper enables direct training and evaluation of consistent and verifiable language inference by providing a dataset that makes explicit the underlying evidence chains behind a high-level text classification task. ## conclusion and discussion in this work, we proposed trip, a tiered benchmark dataset for physical commonsense reasoning posing a new challenge of jointly solving low-level to high-level tasks to form a coherent reasoning process. we experimented with several variations of tiered systems to solve the tasks. our results show that in many cases, supervising large lms based on high-level classification tasks in order to learn commonsense language understanding leads to inconsistent and unverifiable reasoning, and inability to capture intermediate evidence toward the end task. instead, we should train systems to jointly incorporate multiple types of lower-level evidence to solve reasoning tasks coherently. our detailed analysis of results offers strong intuition for future progress toward this goal. as such, trip and our baselines provide an important first step toward verifiable, human-aligned commonsense language understanding, and a direction for development of ai systems in this area. 8 broader impact. we use physical commonsense reasoning as an example in this work, but expect that a similar approach can apply to many aspects of language understanding. our results have shown that a new challenge for the future will be to build machines that can reason logically and coherently, similar to what we expect from human reasoning. as these machines ultimately will work with humans, such alignment in reasoning is critical, as it will improve accountability and transparency in human-machine enterprise.