Attention-based Contrastive Learning for W inograd Schemas (2021)

Abstract: Self-supervised learning has recently attracted considerable attention in the NLP community for its ability to learn discriminative features using a contrastive objective. This paper investigates whether contrastive learning can be extended to Transfomer attention to tackling the Winograd Schema Challenge. To this end, we propose a novel self-supervised framework, leveraging a contrastive loss directly at the level of self-attention. Experimental analysis of our attention-based models on multiple datasets demonstrates superior commonsense reasoning capabilities. The proposed approach outperforms all comparable unsupervised approaches while occasionally surpassing supervised ones.

## introduction pre-trained language models have propelled the domain of nlp to a new era. specifically, transformer-based models are the driving force behind recent breakthroughs. however, despite all the recent success in text understanding, the task of commonsense reasoning is still far from being solved @xcite . in order to assess the commonsense reasoning capabilities of automatic systems, several tasks have been devised. among them is the popular winograd schema challenge (wsc) @xcite . wsc frames commonsense reasoning as a pronoun co-reference resolution problem @xcite , which consists of twin-pair sentences. experts curated the twin pairs manually to be "google-proof", e.g., simple statistical biases from large data should be insufficient to resolve the pronouns. hence, solving wsc was expected to require diverse reasoning capabilities (e.g., 1 the source code can be found at: https://github.com/sap-samples/ emnlp2021-attention-contrastive-learning/ relational, causal). sentences in the twin pairs differ only in the "trigger word". furthermore, trigger words are responsible for switching the correct answer choice between the questions. below is a popular example from wsc. in the example, the trigger word is underlined. the challenge entails resolving the pronoun "it" with a noun from the candidate set ("suitcase", "trophy"): sentence-1: the trophy doesn't fit in the suitcase because it is too small. answers: a) the trophy b) the suitcase sentence-2: the trophy doesn't fit in the suitcase because it is too big. ## attention-based contrastive learning preliminaries: the proposed approach extends the contrastive self-supervised method @xcite to facilitate commonsense reasoning for winograd schemas at the attention level. in the context of data, we assume that d with @xmath0, (s i , s i+1 ) ∈ d c , with c j and c j+1 denoting answer candidates. the difference between the sentence pairs is the so-called "trigger words" responsible for flipping the answer in pronoun disambiguation. thus, this triggerword structure induces a mutual-exclusive candidate answer relationship at the pair level. in the context of the model, we employ a transformerbased lm for masked token prediction @xcite . given a sentence with a [mask] token, the lm provides the likelihood of sentence s i with the token replaced by candidate tokens c @xmath1, [candidate-2]} denoted as p (c j |s i ) = p i,j , assuming that the dataset consists of @xmath2. besides sentence likelihoods, the transformer architecture also provides an attention tensor a(x) ∈ r h×l×c×c , for a given an input x with |x| = c, where l denotes the number of layers, and h the number of heads. then the tensor decomposes into elements a h,l i,j (x), gauging the influence of token i w.r.t. token j in layer l of attention head h. ## conclusion in this paper, we introduce an attention-level selfsupervised learning method for commonsense reasoning. specifically, we propose a method that enforces a contrastive loss on the attentions produced by transformer lm while pushing the likelihood of the candidates towards the extremities. the experimental analysis demonstrates that our proposed system outperforms the previous unsupervised state-of-the-art in multiple datasets.