Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models (2021)

Abstract: Can we get existing language models and refine them for zero-shot commonsense reasoning? This paper presents an initial study exploring the feasibility of zero-shot commonsense reasoning for the Winograd Schema Challenge by formulating the task as self-supervised refinement of a pre-trained language model. In contrast to previous studies that rely on fine-tuning annotated datasets, we seek to boost conceptualization via loss landscape refinement. To this end, we propose a novel self-supervised learning approach that refines the language model utilizing a set of linguistic perturbations of similar concept relationships. Empirical analysis of our conceptually simple framework demonstrates the viability of zero-shot commonsense reasoning on multiple benchmarks.

## introduction natural language processing has recently experienced unprecedented progress, boosting the performance of many applications to new levels. however, this gain in performance does not equally transfer to applications requiring commonsense reasoning capabilities, which has largely remained an unsolved problem @xcite . in order to assess the commonsense reasoning capabilities of automatic systems, several tasks have been devised. among them is the popular winograd schema challenge (wsc), which frames commonsense reasoning as a pronoun resolution task @xcite . although appearing evident and natural to the human mind, modern machine learning methods still struggle to solve this challenge. lately, the research community has experienced an abundance of methods proposing utilization of 1 the source code can be found at: https://github.com/sap-samples/ emnlp2021-contrastive-refinement/ "the trophy does not fit into the suitcase, because it is too big." "the trophy does not fit into the suitcase, because it is too small." "the medal does not fit into the box, because it is too big." "the medal does not fit into the box, because it is too small." language models (lm) to tackle commonsense reasoning in a two-stage learning pipeline. starting from an initial self-supervised learned model, commonsense enhanced lms are obtained in a subsequent fine-tuning (ft) phase. fine-tuning enforces the lm to solve the downstream wsc task as a plain co-reference resolution task. however, such supervised approaches are prone to leverage statistical data artifacts for reasoning, giving rise to the "clever hans" effect @xcite . as such, instead of truly featuring reasoning capabilities, approaches become very good in faking. on the other hand, the lack of commonsense reasoning capabilities of lms can be partially attributed to the training corpora itself, as the commonsense knowledge is often not incorporated into the training text due to the assumed triviality @xcite @xcite @xcite @xcite . we hypothesize that the current self-supervised tasks used in the pre-training phase are insufficient to enforce the model to generalize commonsense concepts @xcite . this shortcoming is easily unveiled by the susceptibility of lm to semantic variations. in this regard, it has been shown that lms are sensitive to linguistic perturbations @xcite . a case in point is the wsc example in fig. 1 . it shows a pair of sentences sub-ject to semantic variations establishing the same relationship between entities. this can be defined as the joint concept triplet consisting of two nouns and a verb that determines the relationship between the nouns, e.g., (container, item, fit). inappropriate semantic sensitivity to semantic variants leads to inadequate "conceptualization" and misconstruction of such triplets. to address this, we propose self-supervised refinement. it seeks to achieve generalization through a task agnostic objective. to this end, we tackle the problem of commonsense reasoning from a zero-shot learning perspective. leveraging zero-shot models to gauge the intrinsic incorporation of commonsense knowledge suggests being the more valid approach than finetuned models. that can be attributed to the exploitation of implicit biases less likely to occur in this setup. hence, the associated benchmarks constitute a more realistic and reliable benchmark @xcite . other zero-shot methods for commonsense reasoning either use large supervised datasets winogrande @xcite ) or very large lms such as @xcite . in contrast, the proposed method takes a pre-trained lm as input, which undergoes a refinement step. during refinement, the lm is exposed to semantic variations, aiming at improved concept generalization by making the model more robust w.r.t. perturbations. motivated by the recent advancements in contrastive representation learning @xcite @xcite , we propose refining the lm in a self-supervised contrastive fashion. this entails refinement without the use of any labels and hence with no gradient update on the downstream datasets. consequently, the supervision level is identical to the test time of the winograd schema challenge. our contributions are two-fold: (i) we introduce the task of zero-shot commonsense reasoning for wsc by reformulating the task as performing selfsupervised refinement on a pre-trained language model (ii) we propose a self-supervised refinement framework which leverages semantic perturbations to facilitate zero-shot commonsense reasoning. ## discussion and conclusion we introduced a method for self-supervised refinement of lms. its conceptual simplicity facilitates generic integration into frameworks tackling commonsense reasoning. a first empirical analysis on multiple benchmarks indicates that the proposed approach consistently outperforming the baselines in terms of standard pre-trained lms, confirming the fundamental viability. we believe that the performance gain will be more pronounced when leveraging larger perturbation datasets for lm refinement. hence, future work will focus on the generation of perturbations. this could specifically entail the consideration of sample-specific perturbations.