Contrastive Self-Supervised Learning for Commonsense Reasoning (2020)

Abstract: We propose a self-supervised method to solve Pronoun Disambiguation and Winograd Schema Challenge problems. Our approach exploits the characteristic structure of training corpora related to so-called “trigger” words, which are responsible for flipping the answer in pronoun disambiguation. We achieve such commonsense reasoning by constructing pair-wise contrastive auxiliary predictions. To this end, we leverage a mutual exclusive loss regularized by a contrastive margin. Our architecture is based on the recently introduced transformer networks, BERT, that exhibits strong performance on many NLP benchmarks. Empirical results show that our method alleviates the limitation of current supervised approaches for commonsense reasoning. This study opens up avenues for exploiting inexpensive self-supervision to achieve performance gain in commonsense reasoning tasks.

## introduction natural language representation learning (e.g., bert @xcite , etc.) can capture rich semantics from text and consistently improve the performance of downstream natural language processing (nlp) tasks. however, despite the recent progress, the task of commonsense reasoning is still far from being solved. among many factors, this can be attributed to the strong correlation between attainable accuracy and training corpora size and quality. a particular case in point is the winograd schema challenge (wsc) @xcite . despite its seeming simplicity for humans, it is still not solved by current algorithms. below is a popular example of a question-answer pair from the binary-choice pronoun coreference problem @xcite of wsc: sentence-1: the trophy doesn't fit in the suitcase because it is too small. answers: a) the trophy b) the suitcase sentence-2: the trophy doesn't fit in the suitcase because it is too big. answers: a) the trophy b) the suitcase for humans resolving the pronoun "it" to "the suitcase" is straightforward. however, a system without the capacity of commonsense reasoning is unable to conceptualize the inherent relationship and, therefore, unable to distinguish "the suitcase" from the alternative "the trophy". recently, the research community has experienced an abundance in methods proposing to utilize latest word embedding and language model (lm) technologies for commonsense reasoning @xcite @xcite @xcite . the underlying assumption of these methods is that, since such models are learned on large text corpora (such as wikipedia), they implicitly capture to a certain degree commonsense knowledge. as a result, models permit reasoning about complex relationships between entities at inference time. most of the methods proposed a two-stage learning pipeline. they are starting from an initial self-supervised model, commonsense-aware word embeddings are then obtained in a subsequent finetuning (ft) phase. fine-tuning enforces the learned embedding to solve the downstream wsc task only as a plain co-reference resolution task. however, solving this task requires more than just employing a language model learned from large text corpora. we hypothesize that the current self-supervised pre-training tasks (such as next sentence prediction, masked language model, etc.) used in the word embedding phase are too "easy" to enforce the model to capture commonsense. consequently, the supervised fine-tuning stage is not sufficient nor adequate for learning to reason commonsense. this is particularly more severe when pretraining on commonsense-underrepresented corpora such as wikipedia, where the authors often skip incorporating such information in the text, due to the assumed triviality. in this case, the supervised fine-tuning does not seem to be enough to solve the task, and can only learn to "artificially" resolve the pronoun based on superficial cues such as dataset and language biases @xcite @xcite @xcite . in this work, we propose to use minimal existing supervision for learning a commonsense-aware representation. specifically, we provide the model with a supervision level identical to the test time of the winograd challenge. for that, we introduce a self-supervised pre-training task, which only requires pair of sentences that differ in as few as one word (namely, "trigger" words). it should be noted that the notion of trigger words is inherent to the concept of winograd schema questions. trigger words are responsible for switching the correct answer choice between the questions. in the above example, the adjectives big and small act as such trigger words. given the context established by the trigger word, candidate answer a is either right in the first sentence and wrong in the second, or vice-versa. as is evident from the example, trigger words give rise to the mutual-exclusive relationship of the training pairs. the proposed approach targets to incorporate this pairwise relationship as the only supervisory signal during the training phase. training in such a contrastive self-supervised manner is inducing a commonsense-aware inductive bias. this can be attributed to several factors. optimization enforces the classifier to be more rigorous in its decision as well as consistent across pairs while being discriminative. specifically, in the absence of strong individual sentence signals, the model seeks to combine weak signals across pairs. this unsupervised task is much harder to learn compared to the supervised task, and resolving the respective associations requires a notion of commonsense knowledge. consequently, we postulate that training with contrastive self-supervised fashion allows for learning more in-depth word relationships that provide better generalization properties for commonsense reasoning. for that, we propose to incorporate a mutual exclusive (mex) loss @xcite during the representation learning phase by maximizing the mutual exclusive probability of the two plausible candidates. specifically, given a pair of training sentence, the pronoun to be resolved is masked out from the sentence, and the language model is used to predict such only one of the candidates can fill in the place of masked pronoun while fulfilling the mutual-exclusivity condition. in this self-supervised task, the labels (i.e., correct candidates) do not have to be known a priori. thus it allows learning in an unsupervised manner by exploiting the fact that the data is provided in a pairwise fashion. our contributions are two-fold: (i) we propose a novel self-supervised learning task for training commonsense-aware representation in a minimally supervised fashion. (ii) we introduce a pair level mutual-exclusive loss to enforce commonsense knowledge during representation learning. ## previous works there is a wealth of literature on commonsense reasoning, but we only discuss here the ones most related to our work and refer the reader to the recent analysis paper by @xcite . traditional attempts on commonsense reasoning usually involve heavy utilization of annotated knowledge bases (kb), rule-based reasoning, or hand-crafted features @xcite @xcite . only very recently and after the success of natural language representation learning, several works proposed to use supervised learning to discover commonsense relationships, achieving state-of-the-art in multiple benchmarks (see, e.g., @xcite @xcite ). as an example, @xcite has proposed to exploit the labels for commonsense reasoning directly and showed that the performance of multiple language models on winograd consistently and robustly improves when fine-tuned on a similar pronoun disambiguation problem dataset. despite the success of these methods, we posit that unsupervised learning is still more attractive for commonsense reasoning tasks, because curating a labeled dataset entailing all existing commonsense is likely to be an unattainable objective. very recently, unsupervised learning has also been applied successfully to improve commonsense reasoning in a few works (trinh and the man lifted the boy onto his shoulders. the man lifted the boy onto his bunk bed. the man lifted the boy onto his shoulders. the man lifted the boy onto his bunk bed. lm loss mex loss s1: figure 1: contrastive self-supervised learning for a particular sentence. colors show the likelihood of different words. weak commonsense signal manifests in the likelihood of both candidates to be around 0.5 for the lm-only loss (shown in dash lines); incorporating the mex loss (shown in solid lines) leverages mutual exclusivity of the candidates, enforcing the classifier to be more rigorous and consistent across pairs (best shown in color). le, 2018; @xcite . the most pioneering work in this space is probably by (trinh and le, 2018), where the authors proposed to use bert as a (pseudo) language model to compute the likelihood of candidates replacing the pronoun, and the corresponding ratio giving rise to answer. in another recent work, @xcite proposed a metric based on the maximum attention score for commonsense reasoning. while these papers show that bert can implicitly learn to establish complex relationships between entities, our results suggest that solving commonsense reasoning tasks require more than unsupervised models learned from massive text corpora. we note that our model is different from all of the methods above. a key difference is that they require fine-tuning, or explicit substitution or heuristic-based rules, whereas our method learns a commonsense-aware representation in self-supervised fashion. ## contrastive self-supervised reasoning the goal of the proposed approach is to exploit the mutual-exclusive nature of the training samples of commonsense reasoning corpora. given two sentences where the only difference between them is the trigger word(s), we postulate that the pairwise pronoun disambiguation is mutually exclusive. we formulate this idea using a contrastive loss and use this to update the language model. the proposed contrastive loss decomposes into two components: here f is the language model parameterized by θ. the first term, l m ex enforces the mutual exclusivity of the answers across pairs. as such, it is a relaxation of the exclusive-or (xor) operator w.r.t. candidates. the second term, l cm constitutes the contrastive margin. it enforces a margin between the candidate likelihoods from the language model. whereas l m ex operates across pairs, l cm considers the candidates of each pair. although both terms encourage the same property (mutual exclusivity of the answers), we empirically observed that adding cm increases stability. it should be noted that the proposed approach does not make use of any class label information explicitly. rather, it solely exploits the structural information of the data. in terms of the language model, we leverage bert for masked token prediction @xcite . this entails replacing the pronoun by a mask, i.e., @xcite . as a result, we yield probabilities for the candidates of each sentence. preliminaries: given an associated pair of training sentences, i.e., (s j , s j+1 ), where the difference between the sentence pairs are the trigger words. let c i and c i+1 be the two answer candidates for the masked pronoun resolution task. then employing bert for masked token prediction @xcite provides p (c i |s j ) and p (c i+1 |s j ), i.e., the likelihood of the first and the second candidate being true in sentence s j , respectively. it should be noted, if a candidate consists of several tokens, the corresponding number of [mask] tokens is used in the masked sentence. the candidate probability then corresponds to the average of log-probabilities of each composing token. since a candidate cannot be the right answer for the first and second sentence in the pair, we yield a logical term that holds true for viable answers. it is worth noting that the logical expression is not unique as many logical equivalents exist: here ⊕ denotes the xor operator and c i,@xmath0, 1} denotes the binary state variable corresponding to candidate c i in sentence s j . mutual-exclusive loss: in order to be differentiable, the discrete logical term of eq. 2 has to be converted into a "soft" version. to this end, we replace the binary variables with their corresponding probabilities. similarly, the logical operators are replaced accordingly to accommodate for the probabilistic equivalent. with a ⊕ @xmath1, we adopt the following replacement scheme: the not operation of a binary variable ¬x i is replaced by 1 -x i . thus, transforming all the logical terms of eq. 2, we yield the following soft-loss equivalent: here p i,@xmath2, 1] denotes the probability of candidate c i being the right answer in sentence s j , γ is a hyperparameter, and n corresponds to the number of training samples. intuitively speaking, as no labels are provided to the model during training, the model seeks to make the answer probabilities less ambiguous, i.e., approximate binary constitution. as the model is forced to leverage the pairwise relationship in order to resolve the ambiguity, it needs to generalize w.r.t. commonsense relationships. as such, the task is inherently more challenging compared to, e.g., supervised cross-entropy minimization. contrastive margin: in order to stabilize optimization and speed-up convergence, it is beneficial to augment the mex loss with some form of regularization. to this end, we add a contrastive margin. it seeks to maximize difference between the individual candidate probabilities of the language model and is defined as, with α, β being hyperparameters. see fig. 1 for a schematic illustration of the proposed method. ## experiment & results in this work, we use the pytorch @xcite implementation of bert. specifically, we employ a pre-trained bert large-uncased architecture. the model is trained for 25 epochs using a batch size of 4 (pairs), hyperparameters α = 0.05, β = 0.02 and γ = 60.0, and adam optimizer at a learning rate of 10 -5 . we approach commonsense reasoning by first fine-tuning the pre-trained bert lm model on the dpr training set @xcite . subsequently, we evaluate the performance on four different tasks. pronoun disambiguation problem: the first evaluation task is on pdp-60 @xcite , which aims the pronoun disambiguation. as can be seen in tab. 1 (top), our method outperforms all previous unsupervised results by a significant margin of at least (+15.0%). next, we have the alternative approaches making use of a supervisory signal during training. here, our method outperforms even the best system (78.3%) by (+11.7%). winograd schema challenge: the second task is wsc-273 @xcite , which is known to be more challenging than pdp-60. here, our method outperforms the current unsupervised state-of-the-art (trinh and le, 2018) (62.6%), as shown in tab. 1 (middle). specifically, our method achieves an accuracy of (69.6%), which is (+7%) above the previous best result. simultaneously, the proposed approach is just slightly lower than the best supervised approach @xcite . ## discussion in contrast to supervised learning, where semantics is directly injected through "labels", the self-pdp-60 (sup.) @xcite 45.0 % nicos issak (ws challenge 2016) 48.3 % quan liu (ws challenge 2016-winner) 58.3 % ussm + supervised deepnet 53.3 % ussm + supervised deepnet + 3 kb 66.7 % bert-ft @xcite 78.3 % pdp-60 (unsupervised) unsupervised sem. similarity (ussm) 55.0 % transformer lm @xcite 58.3 % bert lm @xcite 60.0 % mas @xcite 68.3 % dssm @xcite 75.0 % css (proposed method) 90.0 % wsc-273 (sup.) @xcite ussm + kb 52.0% ussm + supervised deepnet + kb 52.8 % transformer @xcite 54.1 % know. hunter @xcite 57.1 % gpt-ft @xcite 67.4 % bert-ft @xcite 71.4 % wsc-273 (unsupervised) single lms @xcite 54.5 % mas @xcite 60.3 % dssm @xcite 63.0 % ensemble lms @xcite 63.8 % css (proposed method) 69.6 % dpr @xcite ) @xcite 73.0% @xcite 76.4 % css (proposed method) 80.1 % knowref @xcite e2e @xcite 58.0 % bert-ft @xcite 61.0 % css (proposed method) 65.5 % ## conclusion the proposed approach outperforms all approaches on pdp and dpr tasks. at the more challenging wsc task, it outperforms all unsupervised approaches while being comparable in performance to the most recent supervised approaches. additionally, it is less susceptible to gender and number biases as the performance on knowref suggests. all this taken together confirms that selfsupervision is possible for commonsense reasoning tasks. we believe in order to solve commonsense reasoning truly, algorithms should refrain from using labeled data, instead exploit the structure of the task itself. therefore, future work will aim at relaxing the prior of winograd-structured twin-question pairs. possibilities are automatically generating an extensive collection of similar sentences or pre-training in a self-supervised fashion on large-scale winograd-structured datasets, such as the recently published winogrande @xcite . furthermore, we seek to investigate the transferability of the obtained inductive bias to other commonsense-demanding downstream tasks, which are distinct from the winograd-structure.