CS - NET at S em E val-2020 Task 4: S iamese BERT for C om VE (2020)

Abstract: In this paper, we describe our system for Task 4 of SemEval 2020, which involves differentiating between natural language statements that conform to common sense and those that do not. The organizers propose three subtasks - first, selecting between two sentences, the one which is against common sense. Second, identifying the most crucial reason why a statement does not make sense. Third, generating novel reasons for explaining the against common sense statement. Out of the three subtasks, this paper reports the system description of subtask A and subtask B. This paper proposes a model based on transformer neural network architecture for addressing the subtasks. The novelty in work lies in the architecture design, which handles the logical implication of contradicting statements and simultaneous information extraction from both sentences. We use a parallel instance of transformers, which is responsible for a boost in the performance. We achieved an accuracy of 94.8% in subtask A and 89% in subtask B on the test set.

## introduction incorporating common sense in natural language understanding systems and evaluating whether a system has sense-making capability remains a fundamental question in the natural language processing field @xcite @xcite . one important difference between human and machine text understanding lies in the fact that humans have access to commonsense knowledge while processing text, which helps them to draw inferences about facts that are not mentioned in a text, but that is assumed to be common ground @xcite . for a computer system, inferring unmentioned facts is a non-trivial challenge @xcite . for our problem, we have proposed methods to include common sense in the validation and reasoning paradigm @xcite . task 4 of semeval 2020 @xcite ) is a common-sense validation and explanation task. it consists of classifying against common sense sentences from sentences that make sense. figure 1 shows examples from subtask a and subtask b. in subtask a, clearly sentence 1 is against common sense. subtask b contains three options for reasons to explain why sentence 1 is against common sense. as orange juice does not taste good on cereal, but milk does, sentence 1 makes less sense than sentence 2. we use the generated embedding from transformer based encoders like bert @xcite , roberta @xcite , albert @xcite which capture in-context semantic information and design a siamese architecture to extract the relational information between the sentences. the implementation for our system is made available via github 1 . ## experimental setup data: dataset that was used to build the models were provided by the organizers as a part of the pilot study @xcite . as a single datapoint, subtask a had an against-common-sense and a correct sentence. for each of the against common sense sentence of subtask a, subtask b had a sensible reason and two confusing reasons. the datasets were provided in three phases. the training data was used to train the language model. it consisted of 10,000 datapoints. the 997 datapoints long development dataset was used to tune the hyperparameters of the language model. the answers generated on the test dataset of 1,000 datapoints were used for submission. from parameter setting: we used held-out validation using the development dataset for validation. the train:dev ratio for the training was nearly 10:1. the hyperparameters were tuned using a grid search around the default setting of the language model. it ensures the highest efficiency of the model. for both the subtasks, the accuracy of the model on the development dataset was used as a measure of performance. our submitted model is trained on batch mode with a batch size of 32 using adamw optimizer. the learning rate of the optimizer was set to be 2e-5, and the adam epsilon was set to be 1e-8. dataset first sentence invalid second sentence invalid training data 4979 5021 validation data 518 479 (a) subtask a dataset details dataset first reason correct second reason correct third reason correct training data 3195 3362 3443 validation data 344 327 336 (b) subtask b dataset details ## conclusion in this paper, we have presented our systems for the commonsense validation and explanation challenge in semeval2020. our approach for subtask a and subtask b achieves close to state-of-the-art results. our approach demonstrates the advantage of using parallel instances of the transformer in terms of a performance gain in classification based tasks.