U o R at S em E val-2020 Task 4: Pre-trained Sentence Transformer Models for Commonsense Validation and Explanation (2020)

Abstract: SemEval Task 4 Commonsense Validation and Explanation Challenge is to validate whether a system can differentiate natural language statements that make sense from those that do not make sense. Two subtasks, A and B, are focused in this work, i.e., detecting against-common-sense statements and selecting explanations of why they are false from the given options. Intuitively, commonsense validation requires additional knowledge beyond the given statements. Therefore, we propose a system utilising pre-trained sentence transformer models based on BERT, RoBERTa and DistillBERT architectures to embed the statements before classification. According to the results, these embeddings can improve the performance of the typical MLP and LSTM classifiers as downstream models of both subtasks compared to regular tokenised statements. These embedded statements are shown to comprise additional information from external resources which help validate common sense in natural language.

## introduction against-common-sense statements are statements which are contradictory, either partly or entirely, to the fact or do not make sense in reality. understanding these statements can be challenging due to the variety and complexity of natural languages, yet it can be highly beneficial for many purposes such as detecting false information or validating statements generated by computer program. to explore this approach, task4: commonsense validation and explanation challenge (comve) is introduced as a challenge in semeval-2020. the main goal of comve is to develop a system that can differentiate and explain counterfactual natural language statements. this task consists of three subtasks, subtask a, subtask b and subtask c. in subtask a, two statements are given, one is against common sense while the other is not. the objective of subtask a is to identify the statement that is contradictory to common sense. for subtask b and c, the objectives are to explain why a given statement is contradict to the fact by using different methods. in subtask b, an explanation is selected from the given options, while, an entire explanation has to be generated from scratch in subtask c. the total detail can be found in @xcite to decide whether a given statement makes sense, additional context may be required. in such case, a sentence embedding model becomes useful for including this additional information by semantically encoding statements before classification. in this work, we study on using pre-trained sentence transformer models @xcite which are state-of-the-art transformer models fine-tuned for sentence embedding to solve the problem in subtask a and subtask b. these models were modified based on three widely used transformer models with different configurations. in our approach, these pre-trained models are used to embed the statements so that the embedded statements can be classified more effectively to detect against-common-sense statements in subtask a and find an explanation for against-common-sense statements in subtask b. ## related work language pre-training methods have been long utilised in nlp tasks and have, indeed, been proven to be effective for developing a successful model @xcite . @xcite proposes the elmo (embedding from language models) representation, a feature-based approach which makes use of pre-trained representations of the data as additional features in the network and promises improved performance over using a long short-term memory (lstm) top layer @xcite . a variation of the hybrid neural network, as explored by @xcite is the knowledge enhanced hybrid neural network (kehnn). this model makes use of prior knowledge representations and multiple channels to improve matching of long text data-sets @xcite . in recent time there have been notable advancements in commonsense validation and reasoning models, many of the new developments stemming from the bert language representation model proposed by @xcite such as the robustly optimised bert (roberta) pre-training approach developed by @xcite and distilbert, which used knowledge reduction, or distillation, techniques whilst retaining the models language understanding capabilities and decreasing its training and computation timings @xcite . another development on the bert model is the commonsense auto-generated explanations (cage) model proposed by @xcite which, in their study, shows an improved accuracy when classifying the cqa (commonsense question answer) dataset when compared with other similar networks. the sbert model, a variation on the bert model, which makes use of siamese and triplet network architectures, has been found to be particularly effective with deriving sentence embedding whilst retaining the underlying semantics within the sentences @xcite . the resultant embeddings can then be further applied to classification networks, or regression models equally. however, there has been no work where this pre-trained embedding model is applied in a commonsense validation and explanation task. therefore, in this work, we propose to use sentence transformer models including sbert and other embedding models based on roberta and distillbert developed by @xcite to embed the statements and classify them in subtask a and subtask b. ## system overview the proposed system consists of two parts: (1) the pre-processing stage, in which the original datasets are adjusted to fit our classification model, and nlp techniques, which are applied for cleaning and preparing the data before further analysis. (2) the classifier with pre-trained sentence transformer models. in this stage the pre-processed statements are embedded by using pre-trained sentence transformer models and then passed to a multilayer perceptron (mlp) model and long short-term memory (lstm) model for classification. these two parts are detailed in the following subsection. ## experimental setup the models were trained with the training and development set provided in both subtasks. however, in subtask b, after conversion, one original sample can be converted to three new samples, one sample labeled with 1 and two samples labeled with 0, causing an imbalance in the dataset. therefore, the data from subtask c was used to compensate the positive samples. in subtask c, for each false statement, there are three examples that can be used to explain how the given statement is false. thus, one of these example distinct from the correct option in subtask b was randomly selected and added to subtask b dataset. as for the pre-processing step, we used existing libraries, i.e., spacy named entity recognition 1 to detect and replace person's names, num2words 2 to convert numbers to words and the punctuation set in python string library to remove punctuation marks. we selected two architectures, mlp and lstm models, to build classifiers for both subtasks. the mlp model consists of four hidden layers with 512, 256, 128, 64 number of neurons and a rectified linear unit (relu) as an activation function. dropout layers were included after the input layer with dropout rate 0.1 and between each hidden layer with dropout rate 0.5 to avoid overfitting. for lstm model, we used an lstm layer with 64 dimensions of an output. then, it is followed by a fully-connected layer consists of 64 neurons. the proposed models were trained with adam optimiser for 20 epochs. a binary cross-entropy and accuracy were used as a loss function and metric respectively. all models were evaluated on both trial and test set by accuracy which will be shown in the next section. ## result the mlp and lstm classifier results of subtask a are shown in for subtask b, as shown in for both subtasks, the results show that using the pre-trained sentence transformer models with the mlp classifier can largely increase the performance of commonsense validation and explanation.on the other hand, the lstm classifier can perform better with the embedded statements only in subtask a whereas there is no outstanding improvement in performance when it was applied in subtask b. considering the sentence transformer models, the bert-base and roberta-base models show to perform better than the distilbert-base models, which consist of fewer parameters, in most cases. for different pooling methods on the same bert-base model, the results obtained from the mean-tokens and cls-tokens methods with the mlp classifier are not significantly different. meanwhile, using max-tokens pooling causes a slight accuracy decline compared to the others which agrees with the result in the original paper @xcite . contrary, with the lstm classifier, the cls-tokens pooling performs worst in both subtasks while mean-tokens and max-tokens pooling methods can perform almost equally better. according to our experiment applying mean-tokens pooling, all token embeddings in a statement are summarised together as well as using the 'cls' token embedding which is treated as a whole statement representation. on the other hand, selecting the maximum value and neglecting the other values may lead to essential information loss causing the lower meaningful embedding performance. ## conclusion commonsense validation and explanation in natural language statements requires additional knowledge which is, occasionally, inadequate when only a single statement is considered independently. to satisfy this condition, this work proposes a system using the pre-trained sentence transformer models to embed the statements before classification. with semantic knowledge these model gained from both nli and sts datasets, the statements can be embedded into vectors comprise the meaning and additional commonsense comprehension based on these natural language resources. in our experiment, these embedding models were proved to be useful for detecting and explaining counterfactual statements. the results show classification models with embedding models have better accuracy than those without embedding models. this suggests that pre-trained sentence transformer models can provide additional prior knowledge for commonsense validation and explanation.