DEEPYANG at S em E val-2020 Task 4: Using the Hidden Layer State of BERT Model for Differentiating Common Sense (2020)

Abstract: Introducing common sense to natural language understanding systems has received increasing research attention. To facilitate the researches on common sense reasoning, the SemEval-2020 Task 4 Commonsense Validation and Explanation(ComVE) is proposed. We participate in sub-task A and try various methods including traditional machine learning methods, deep learning methods, and also recent pre-trained language models. Finally, we concatenate the original output of BERT and the output vector of BERT hidden layer state to obtain more abundant semantic information features, and obtain competitive results. Our model achieves an accuracy of 0.8510 in the final test data and ranks 25th among all the teams.

## introduction humans can utilize a variety of knowledge and reasoning to help understand the meanings of language. for example, it is easy for a human to know that someone can put a turkey into a refrigerator. still, he can never put an elephant into it with basic commonsense reasoning, but it can be non-trivial for a system to tell the difference @xcite . what enables us to arrive at this conclusion is the knowledge we have about the world and the underlying reasoning process, often called commonsense thought @xcite or commonsense reasoning @xcite , that allows us to connect pieces of knowledge to reach the new conclusion. we know that the refrigerator is household appliances and turkeys are the food we usually eat, but elephants cannot be eaten; the elephant is larger than the refrigerator and cannot be put in it. while this kind of knowledge and reasoning becomes so naturally to humans, it is extremely difficult for machines. despite significant advances in natural language processing in the last several decades, machines are still far away from having this type of natural language inference (nli) ability @xcite . therefore, we need an effective automatic method to differentiate natural language statements that make sense from those that do not make sense. in this paper, the used methods and the results obtained by our team, deepyang, on the comve shared task organized at semeval 2020 are presented @xcite . the comve shared task includes three sub-tasks commonsense validation, commonsense explanation (multi-choice), and commonsense explanation (generation). we only participate in subtask a. the main goal in sub-task a is to choose which statement of the two is against common sense. by definition, the sub-task a is to select from two natural language statements with similar wordings which one makes sense and which one does not make sense. for this subtask, we try traditional machine learning methods, deep learning methods, and pre-trained models. after comparison, we choose the bert-base model and concatenate the original output of bert and the output vector of bert hidden layer to obtain more abundant semantic information features. finally, our model achieves an accuracy of 0.8510 on the test set, and ranks 25th. and knowledge of the world. many benchmark tasks and datasets have been presented to support the development and evaluation of such natural language inference ability. for example, using traditional machine learning methods mainly includes support vector machines (svms) @xcite , etc. nowadays, deep neural networks have become mainstream for nlp, such as convolutional neural networks @xcite and long short term memory networks (lstms) @xcite . however, in traditional word embedding models such as word2vec @xcite , glove @xcite or fasttext @xcite , the embedding vectors are contextindependent. despite the target word appears in different contexts, after training, the embedding vector remains the same. consequently, these embeddings lack the capability of modeling different word senses in different contexts, although this phenomenon is prevalent in language. to address this problem, recent works have developed contextual word representation models, e.g., embeddings from language models (elmo) @xcite and bidirectional encoder representations from transformers (bert) @xcite . based on the context they appeared, these models give words different embedding vectors . these pre-trained word representations can be used as features or fine-tuned for downstream tasks. for example, the generative pre-trained transformer (gpt) @xcite and bert introduce minimal task-specific parameters and can be easily fine-tuned on the downstream tasks with modified final layers and loss functions. ## methodology in this section, the methods operated by our team for sub-task a are explained. we use several methods including traditional machine learning methods, deep learning methods, and pretrained language model, such as svm, lstm, attention, bert, and so on. as expected, as the out-of-the-box pre-trained and fine-tuned bert model has shown great performance in many kinds of nlp problems, bert also has achieved a good result by comparison in this work. finally, we choose the bert model as our base model. next, we introduce the methods we use in this work. ## conclusion in this paper, we address the challenge of automatically differentiate natural language statements that make sense from those that do not make sense. we conduct experiments with svm, word embedding model, and bert. after comparison, we obtain more abundant semantic information features by concatenating the original output of bert and the output vector of bert hidden layer state. the result shows that it is helpful to improve the performance of bert to obtain more abundant semantic information features by extracting the hidden state of bert. the code is available online. 5