UI at S em E val-2020 Task 4: Commonsense Validation and Explanation by Exploiting Contradiction (2020)

Abstract: This paper describes our submissions into the ComVe challenge, the SemEval 2020 Task 4. This evaluation task consists of three sub-tasks that test commonsense comprehension by identifying sentences that do not make sense and explain why they do not. In subtask A, we use Roberta to find which sentence does not make sense. In subtask B, besides using BERT, we also experiment with replacing the dataset with MNLI when selecting the best explanation from the provided options why the given sentence does not make sense. In subtask C, we utilize the MNLI model from subtask B to evaluate the explanation generated by Roberta and GPT-2 by exploiting the contradiction of the sentence and their explanation. Our system submission records a performance of 88.2%, 80.5%, and BLEU 5.5 for those three subtasks, respectively.

## introduction natural language understanding (nlu) has seen increasing attention in recent times. it requires not only the capability to deduce the semantics of the text but also commonsense knowledge of the world. one of the prominent tasks to test how well a machine recognizes and understands the human language is natural language inference, in which given the pair of texts (a hypothesis and a premise), and the model deduces whether the premise entails or contradicts the hypothesis. winograd scheme challenge @xcite is another task that provides a text and a question regarding the text with two possible answers, and the model must select the correct one. however, the question is designed that both answers are possible, grammatically wise, and the only way to answer is to have access to the implicit commonsense knowledge. while the aforementioned tasks are the application of commonsense, semeval 2020 task 4: commonsense validation and explanation @xcite puts emphasis purely on the commonsense itself. in this paper, we present our submission system that relies on the transformers-based model @xcite . on the other hand, one of the biggest concerns of the comve subtask is its small size. we know that the explanatory nlu model is prone to over-fitting and may generate inconsistent explanation @xcite . a potential solution is to augment the dataset by leveraging the existing commonsense knowledge. ## task description semeval 2020 task 4: commonsense validation and explanation (comve) challenges three subtasks: identifying which member of a pair of a similar sentence does not make sense, selecting the best explanation why a sentence does not make sense, and generate an explanation from scratch why a sentence does not make sense. the dataset for comve task consists of 10k instances in train set and 1k instances in test set ## resources our submission system to comve challenge takes advantage of several resources, i.e. transformers based mode, conceptnet knowledge base, and natural language inference dataset. ## result in overall, performance of our model surpass the baseline @xcite , that are a fine-tuned elmo with an accuracy of 74.1 % for subtask a and the bert model with an accuracy of 45.6 %. for subtask a, the roberta model records an accuracy of 88.2% and finishes in 22nd out of 39 teams. moreover, the roberta model that uses conceptnet to generate an additional dataset, which we submit after the evaluation period, shows a slight improvement regarding accuracy. the bert model that we submit for subtask b reaches 80.5% and positions 16th place out of 27 teams. however, the model that is trained with only snli gets an accuracy of 64.7%. we see a significant accuracy drop from the snli model, but it should be high enough that it should be able to support the model for subtask c. both gpt-2 and roberta achieve the bleu score of 5.5. gpt-2 was evaluated by human grader and marked of 0.73/3, although we discover that the sentences created by roberta should be more coherent than roberta. from its bleu, the model finish in 12th place out of 17 teams. however, sorted by human evaluation, the model places in 14th place. from the submission into the subtask a, we observe that in around 70% our inaccurate prediction involve changing the last word in the instance (shown in ## summary we observe that training transformer model to take the single sentence as an input and classify the sentence within the pair independently can achieve a high performance in subtask a and subtask b. we have explored that the contradiction relationship between the sentence that does not make sense and the explanation may work on commonsense validation and explanation task.