Commonsense Statements Identification and Explanation with Transformer-based Encoders (2020)

Abstract: In this work, we present our empirical attempt to identify the proper strategy of using Transformer Language Models to identify sentences consistent with commonsense. We tackle the first two tasks from the ComVE competition. The starting point for our work is the BERT assumption according to which a large number of NLP tasks can be solved with pre-trained Transformers with no substantial task-specific changes of the architecture. However, our experiments show that the encoding strategy can have a great impact on the quality of the fine-tuning. The combination between cross-encoding and multi-input models worked better than one cross-encoder and allowed us to achieve comparable results with the state-of-the-art without the use of any external data.

## introduction for human beings, the answer to the question "can we consider a statement consistent with commonsense?" comes natural even in the absence of a certain context. it is not only about understanding the words in the statement, but also about reasoning based on commonsense knowledge. until recently, this was considered difficult for the machines @xcite , since it was considered that it requires a formal representation of an extremely large knowledge base equipped with a general inference mechanism. in the transformers era, the machines' deeper understanding of text has gained increasing attention. transformers brought reduction of losses of semantics and connections in long text through the extensive use of attention mechanisms and the elimination of the recurrent connections and convolutions. their good performance in recognizing complex semantic relations offered a faster starting point for the upcoming stacked layers of trans-formers capable of generalizing with impressive results on downstream tasks. semeval-2020 task 4, commonsense validation and explanation (comve) @xcite addresses commonsense understanding in the problem of identifying the sentences which are inconsistent with commonsense and the reason for which they are inconsistent. on the basis of transfer learning using transformers, our experiments explored the power of different transformer models offered by huggingface pytorch library on the data set provided in this competition. during our experiments, answers to the following questions were searched: our contribution is in identifying a suitable transformer for these specific commonsense tasks, together with the suitable encoding and decoding strategies. our conclusions are supported by empirical experiments, but the deeper understanding behind these conclusions requires further qualitative analysis which is not yet included. ## related work pre-trained transformer language models (lm) are the foundation of our work. given a sequence of words, an lm estimates the probability distribution of the next word, where the latter can be any word from the vocabulary. furthermore, given a fixed sentence, or succession of words, an lm can assign a probability to the whole sentence. being trained on large sets of unlabeled text, lms aim to capture the context dependent semantics of words and phrases. elmo @xcite would be the first to look at from the historical point of view. it runs the input through a bilstm network gathering left and right context. the drawback stands in risking to derive mistaken context from the misplaced expressions. bert @xcite ) is a multi-layer bidirectional transformer encoder with integrated selfattention mechanism. the volume of text on which bert was pre-trained, totalling to 16gb of text, prepared the model for numerous nlp tasks, such as question answering and language understanding. for solving the commonsense tasks, we looked for an lm which takes into consideration both elmo and bert advantages. xlnet @xcite is a generalized autoregressive method (ar). the autoregressive language model is a contextualized method of predicting the next word considering either a forward or backward factorized context. the main flaw of the autoregressive method is that long-distance context is lost in detriment of close surroundings. xlnet solves this by maximizing the expected log-likelihood of a sequence thanks to the token's permutation operation providing context from both the left and right context. reflecting on bert's autoencoding technique and avoiding the [mask] token impediment, xlnet uses the ar bidirectional context to offer meaning to the current token. roberta @xcite is a replication of bert. it is trained on 10× bigger text corpus with larger batches and for a longer period, with elimination of the next sentence prediction objective and a dynamically change in the masking pattern. the vocabulary size is increased by using byte-pair-encoding @xcite instead of character-level encoding. the majority of the participants in the comve competition @xcite used pre-trained lm. the top performing systems used also external knowledge either through the use of formal commonsense knowledge-bases, like conceptnet, either through a second pre-training of the lm on text relevant for commonsense understanding. we obtained comparable results with the top performing systems for the first two of the three tasks of the competition without any external sources. our approach relies on a proper encoding strategy for the sentences or pairs of sentencesexplanations. ## data set the comve data set was inspired by existing commonsense corpora, like choice of plausible alternatives (copa) @xcite and winograd schema challenge (wsc) @xcite . the first two comve tasks involved two balanced corpora, consisting of a total of 10.000 train sentences and 997 dev sentences each. in task a of comve, the validation one, two statements that differ by one or several words are given without any other context (e.g. he was sent to a (restaurant)/(hospital) for treatment after a car crash;, or bob looks up a words in a (dictionary)/(shopping list)). besides the fact that no context is given, the strong resemblance of the inputs increases the request for a good understanding of the subtle meaning of words. in task b, the explanation one, a nonsense premise is given together with three possible explanations and it is asked to identify the most plausible explanation for the fact that the premise is inconsistent with commonsense. the premises are the statements classified as nonsense on the validation task. we give here an example: ## encoder/decoder architectures in order to solve the commonsense tasks, we chose to focus on the final mission rather than on developing a brand-new model trained from scratch cls t s1 1 t s1 2 ... t s1 m sep t s2 1 t s2 2 ... t s2 n transformer encoder decoder c t 1 1 t 1 2 t 1 m s t 2 1 t 2 2 t 2 n figure 1: task a: one single model for 2 statements packed together cls t s1 1 t s1 2 ... t s1 m cls t s2 1 t s2 2 ... t s2 n transformer encoder 1 transformer encoder 2 aggregate the encodings decoder c c t 1 1 t 1 2 t 1 m t 2 1 t 2 2 t 2 n on an enormous corpus of text. therefore, our involvement stands in locating and perfecting an existing transformer using only the comve data set, without any external knowledge base or additional dataset for a two phase training. transformers were built to generalize well over multiple nlp tasks. the common practice involves adding the right decoder on top of the transformer encoder and fine tuning with a proper amount of data. the encoder extracts the semantics and the features from the inputs, while the decoder performs the classification. our first experiments targeted identification of the transformer language model that is capable to extract the most relevant aspects for the classification of text into consistent with commonsense and inconsistent. the rest of the experiments targeted architectures which instead of including only one encoder for all the statements, include two or three encoders, one for each statement in case of task a, respectively one for each pair of (premise, explanation) for task b. task a was reduced to a binary classification problem in which the model must decide the gibberish statement. as for task b, selecting the good explanation was considered a multi-class classification. in the rest of this section, we detail the proposed architectures, while the corresponding experiments are described in section 5. ## discussions the results revealed the importance of the amount of text on which the lms were pre-trained. the model's size also has significance, as larger models are better at generalizations. a single transformer language model delivers good results after fine-tuning on a comparable small amount of data. additionally, the fine-tuning does not seem to alter the previous knowledge accumulated by the model. this raised the question whether it is necessary to fine-tune the model on a specific task if the lm was already trained on a large text corpus, or is it possible to freeze the model. consequently, we used a roberta large model, added a classifier on top and trained only the classifier on the provided corpus. freezing the model obstructed lm's capabilities and it delivered poor results, similar to random guessing. even when we worked with two roberta large models coupled as siamese but with complete freeze, the results were close to randomness, confirming that the lm adjusts its behaviour based on the task. it implies that a transformer may be considered as a good encoder, but it needs adjustments in the form of fine-tuning. another observation regards the needed decoder. a 2 roberta large model is a formidable model with powerful generalization capabilities, but it does not incorporate enough knowledge to correctly classify a sentence as nonsense or consistent with commonsense if on top of it a very simple two layers classifier is added. as the results showed, the lms, especially bidirectional lms are an excellent starting point in solving commonsense tasks (at least as they are formulated in the comve tasks). from our current experiments, multiple model approach works better than single one. multiple cross-encoders with roberta large is a capable architecture which combines and leverages the benefits of both encoder (cross/bi) techniques. our results show that it can be further improved, but it still provides good enough performance. when using the same decoder, freezing the lm resulted in extremely low performance, while fine tuning the lm reached accuracy values higher than 0.9. in the same time, with the same lm, but different decoders, the performance changed. consequently, we could assert that we need a good encoding, which can be attained even with a small dataset from the pre-trained lm, but we also need a good decoder. however, the set of the experiments need to be extended in order to check the presumption that multiple lms work better than single lms not only in case of roberta, but also other lms. ## conclusions in this paper we evaluated three transformer language models, bert, xlnet, and roberta, for the commonsense validation and explanation problems proposed in sem-eval 2020 comve. the experiments have shown that the self-attention mechanism used by bert and roberta models is more suited for the commonsense tasks. we also leveraged the complexity of the model by using two language models to capture the independent sense for each input sentence. two separate models are more capable to emphasize the subtle disparities from the input sentences. furthermore, in order to select the the right explanation for the inconsistency of the premise with commonsense, we used an architecture with three roberta, each one working as a cross-encoder for pairs (premise, explanation).