YNU -oxz at S em E val-2020 Task 4: Commonsense Validation Using BERT with Bidirectional GRU (2020)

Abstract: This paper describes the system and results of our team participated in SemEval-2020 Task4: Commonsense Validation and Explanation (ComVE), which aim to distinguish meaningful natural language statements from unreasonable natural language statements. This task contains three subtasks: Subtask A–Validation, Subtask B–Explanation (Multi-Choice), and Subtask C– Explanation (Generation). In these three subtasks, we only participated in Subtask A, which aims to distinguish whether a given two natural language statements with similar wording are meaningful. To solve this problem, we proposed a method using a combination of BERT with the Bidirectional Gated Recurrent Unit (Bi-GRU). Our model achieved an accuracy of 0.836 in Subtask A (ranked 27/45).

## introduction in recent years, the introduction of commonsense into natural language understanding systems has received more and more research attention. commonsense validation and explanation describes whether the natural language statement is consistent with the facts and explains the reasons for its against common sense. specifically, the purpose of the commonsense verification is to determine that the natural language statement is reasonable and meaningful. this task involves commonsense, understanding, and reasoning, so it is very important for a system to be able to accurately distinguish between meaningful natural language statements and unreasonable natural language statements. in semeval-2020 task 4: verification and interpretation of common sense @xcite . the organizer has designed three subtasks. subtask a selects from two natural language statements with similar grammatical structures (one of which is meaningful and one of which is not). it requires the model to determine which of the two statements violates commonsense; subtask b is to find the key reason why a given statement violates commonsense from three options, this task is a multiple-choice; subtask c requires the system to generate the reason why this statement violates commonsense. among these three subtasks, subtask a and subtask b are evaluated by accuracy, and subtask c is evaluated by bleu. for subtask a, we experimented with different neural network-based models, such as bidirectional long short-term memory (bi-lstm) @xcite , bidirectional gated recurrent unit (bi-gru) @xcite , and bidirectional encoder representation from transformers (bert) @xcite . we found that bert performs best on the validation set. therefore, we chose to use bert to combine different models to optimize the system. the rest of the paper is structured as follows. section 2 introduce the related work of comve; section 3 introduce the data description and methods; in section 4, we analyze the experimental results; in section 5, we summarize and look forward to future work. ## related work commonsense verification aims to require models to determine whether natural language statements are meaningful. one of the key issues in how to build an effective classifier is to find suitable features. through the introduction of word embedding technology, better performance can be achieved. a common way of a neural network is to add more layers in order to learn high-level features, and the stacked residual lstm model can solve the problem of gradient disappearance and deviation between layers @xcite . commonsense reasoning is a challenging task for modern machine learning methods @xcite . many commonsense representations and reasoning processes have explored and developed large knowledge bases of commonsense. the jhu ordinal common-sense inference (joci) @xcite aims to mark the plausibility of human response in certain circumstances from 5 (very likely) to 1 (impossible). the story cloze test (also known as roc story) compares the true ending of the story to the incredible false ending @xcite . some researchers also investigate ways to use common sense to answer questions. commonsenseqa (cqa) @xcite requires crowd workers to create questions from conceptnet @xcite . @xcite show that using google search to extract context from the first 100 result snippets selected for each question and answer does not greatly improve. cqa is even trained using the most advanced reading comprehension model bidaf ++ @xcite , which also adds a self-attention layer and elmo representation @xcite . @xcite generates interpretation and prediction of natural language inference problems. a survey of commonsense data sets and more common natural language data sets show that there are already some data that focus on non-linguistic world knowledge verification @xcite or limited attributes or actions of world knowledge @xcite . the winograd schema challenge (wsc) @xcite is a widely used and important task that requires more commonsense knowledge. in recent years, some large-scale knowledge resources of commonsense reasoning have also been released, which may be helpful for the development of commonsense reasoning tasks. common sense question answering (cqa) is a multiple-choice question and answer datasets proposed for the development of natural language processing (nlp) models with common sense reasoning capabilities @xcite . @xcite proposes a huge knowledge map of commonsense reasoning, which has nine if-then relationship variables, including causes, effects, etc. techniques based on language modeling, such as gpt and bert models, can achieve human-level performance on these datasets @xcite . ## conclusion this year we participated in the sub-task a for commonsense validation and explanation (comve), which is to verify which of the two sentences is against commonsense. this paper proposes a model combining bert with bi-gru. using the bert model to extract more suitable features, and a combination model with a layer of bi-gru behind bert is used for classification. although the overall performance of our model is not ideal, the preliminary results show what we should do next. in future research, we will use multiple methods to improve our model and will consider introducing k-fold cross-validation to integrate the results to improve model performance.