TR at S em E val-2020 Task 4: Exploring the Limits of Language-model-based Common Sense Validation (2020)

Abstract: In this paper, we present our submission for subtask A of the Common Sense Validation and Explanation (ComVE) shared task. We examine the ability of large-scale pre-trained language models to distinguish commonsense from non-commonsense statements. We also explore the utility of external resources that aim to supplement the world knowledge inherent in such language models, including commonsense knowledge graph embedding models, word concreteness ratings, and text-to-image generation models. We find that such resources provide insignificant gains to the performance of fine-tuned language models. We also provide a qualitative analysis of the limitations of the language model fine-tuned to this task.

## introduction the task of assimilating general world knowledge from textual data for the purpose of commonsense reasoning and inference has been a long-standing challenge in natural language understanding @xcite . a general approach to this problem that has gained recent popularity is the use of neural-network-based language models (lm) @xcite . such models, when trained on massive amounts of diverse text corpora, have been found to capture implicitly a remarkable amount of commonsense knowledge @xcite . moreover, recent advances in training deep contextualized word representations using language-model-type objectives over large text corpora have substantially improved the state-of-the-art performance on a wide variety of natural language understanding tasks @xcite @xcite @xcite , including an even greater ability to capture commonsense knowledge @xcite . despite this success, a significant gap remains between the performance of such pre-trained lms and human-level performance on commonsense reasoning tasks. in fact, there is evidence suggesting that current models fail to learn factual knowledge effectively @xcite and have difficulty with a variety of basic reasoning abilities necessary for commonsense inference @xcite . this motivates us to explore supplementary resources aimed at augmenting the world knowledge inherent in pre-trained lms. in this paper, we explore the ability of a state-of-the-art pre-trained lm in tackling the comve subtask a @xcite . in particular, given a pair of sentences, we evaluate the model's ability in identifying which sentence least agrees with common sense. we consider the model from both a language modeling and a supervised learning perspective. moreover, we explore the utility of additional resources meant to augment the perceptual world knowledge of language models in solving this task. finally, we provide a categorization of the types of sentences that our best performing model struggles against. ## related work a variety of datasets have been created to examine a system's general commonsense knowledge and inference capability through such tasks as anaphora resolution @xcite and question answering @xcite @xcite . certain tasks have attempted to measure commonsense knowledge specifically pertaining to the physical @xcite , temporal @xcite , and causal @xcite aspects of reasoning. recent approaches to tackling these challenges have focused on methods to inject external knowledge into pre-trained lms. the methods vary by the choice of knowledge sources and the training objective @xcite @xcite @xcite @xcite . complementary to such approaches, methods have been developed to automatically expand the coverage of existing commonsense knowledge bases (ckb) @xcite @xcite . due to the immense number of relations that would be needed to capture the wide variety of commonsense knowledge, @xcite proposed a method to "conceptualize" explicit relations into broader, more abstract concepts. in our work, we examine the benefit of using external resources such as ckb-enhanced word embeddings as simple, additional features to a model, rather than attempting to infuse the knowledge within the pre-trained lm. we also consider more perception-based resources, such as word concreteness ratings and text-to-image generation models. ## results and discussion ## conclusion and future work in this paper, we evaluated the performance of a state-of-the-art pre-trained lm on the task of common sense validation. we also explored the usefulness of external resources meant to supplement the implicit commonsense knowledge of the lm. we found that a subset of these resources provide value to a system relying only lm probabilities, but give negligible improvement to an lm fine-tuned to the task. further experiments are needed to evaluate the performance of pre-trained lms on this task when they are explicitly adapted with external commonsense knowledge bases. additionally, we leave as future work the study of tti models with improved object coverage and generation quality that may eventually add value to the subspace of common sense validation that depend significantly on visual world knowledge.