Lijunyi at S em E val-2020 Task 4: An ALBERT Model Based Maximum Ensemble with Different Training Sizes and Depths for Commonsense Validation and Explanation (2020)

Abstract: This article describes the system submitted to SemEval 2020 Task 4: Commonsense Validation and Explanation. We only participated in the subtask A, which is mainly to distinguish whether the sentence has meaning. To solve this task, we mainly used ALBERT model-based maximum ensemble with different training sizes and depths. To prove the validity of the model to the task, we also used some other neural network models for comparison. Our model achieved the accuracy score of 0.938(ranked 10/41) in subtask A.

## introduction after thousands of years of accumulation, human civilization has undergone tremendous changes. people can accumulate a lot of experience every day in study, work and life. these large amounts of experience have become our common knowledge through summarization and verification. in our daily life, common sense can often tell us some other people's practical experience, so that we can avoid the repetition of some errors. after scientific verification, a lot of wrong common sense is slowly being corrected. however, it takes a lot of human resources to manually correct normal knowledge. at the same time, in recent years, due to the development of natural language processing and neural networks, common sense revision has entered the era of mechanization. semeval 2020 task 4 @xcite ) @xcite is designed for common sense verification and interpretation. this task is to directly test whether the system can distinguish meaningful natural language statements from unreasonable natural language statements. in this way, we can save a lot of time to distinguish between common sense and extraordinary knowledge. subtask a is to choose from two natural language statements with similar wording, one of which is meaningful and the other is meaningless. this subtask is mainly to directly distinguish whether the discourse has common sense. subtask b is to find out the key reason why this sentence violates common sense from the three options that can be selected. subtask c is the reason why generating statements violates common sense. for this task, we mainly participated in subtask a. we want to be able to effectively distinguish meaningful sentences from meaningless sentences through subtask a. in this task, we only participate in subtask a: identify whether the language has meaning. for this task, we use a combination of research and deep learning methods to deal with related tasks. in the task, we mainly use the lightweight albert model based on the transformer mechanism. according to the progress of the latest related research, the albert model has become our preferred model. because it has fewer parameters and better performance than the bert model. after data processing, we input the generated word vector into the trained model. besides, we also used pre-trained models of different training sizes and depths for comparative experiments to optimize the prediction probability of the results. meanwhile, we also adopted the method of maximum voting ensemble to optimize the performance of our model. to prove the excellent performance of our method, we use some other neural networks for comparative experiments. in this task, our method is an effective way to get good performance. the rest of our paper is structured as follows. section 2 introduces related work. model and data preparation are described in section 3. experiments and evaluation are described in section 4. the conclusions are drawn in section 5. ## related work in natural language processing, the problem of common sense verification has always been an important one. the common sense verification problem has a large data set. such as, event2mind @xcite is a crowdsourced corpus of 25,000 event phrases covering a diverse range of everyday events and situations. situations with adversarial generations (swag) @xcite is a dataset consisting of 113k multiple choice questions about a rich spectrum of grounded situations. the winograd schema challenge (trinh and le, 2018) is a dataset for common sense reasoning. it employs winograd schema questions that require the resolution of anaphora: the system must identify the antecedent of an ambiguous pronoun in a statement. reading comprehension with commonsense reasoning dataset (record) @xcite is a large-scale reading comprehension dataset which requires commonsense reasoning. for these data sets, there are some neural networks that can get good performance. for example, convolutional neural network @xcite , bidirectional recurrent neural network @xcite , bert model based on transform mechanism @xcite , multi-task deep neural networks @xcite , xlnet @xcite , etc. based on the above research, we have a clear direction to deal with this task. ## model and data preparation our neural network model is shown in figure 1 . in this task, we mainly use a lightweight albert model based on the transformer mechanism. where the e[cls] and e[sep ] are added at the beginning and end of each instance, respectively, which can separate different sentences. ## experiments and evaluation in this task, we obtained good performance using the albert model. meanwhile, we found that albert models with different pre-training sizes and pre-training depths will get different results. we use dev data set to verify the performance of the model. the standard of our judgment is accuracy, and this standard is the judgment standard used for our task. in our task, the results of our comparison are shown in by comparing the performance of the albert model with different pre-training depths, we can get the following conclusions. in this task, we found that increasing the number of hidden layers by a certain number can improve the performance of the albert model. at the same time, in the task, based on the in this task, we use the albert model to pre-train the task. for the albert model, the main hyper-parameters we pay attention to are the training step size, batch size and learning rate. at the same time, according to relevant work research, we also use some other neural network models for experiments. for other models, the main parameters we pay attention to are batch size learning rate and epoch. the parameters of all models are shown in this sub-task a is to evaluate the classification system by calculating the accuracy. in order to prove the effectiveness of our model to improve performance. we also used other neural network models for comparative experiments. we hope to verify that our model is more effective for our task through comparative experiments. the results of our comparative experiments are shown in ## conclusion in this task, we mainly used albert model-based maximum ensemble method with different training sizes and training depths for experiments. at the same time, we also used some other neural networks for comparative experiments, to prove that our model can obtain excellent performance. the results show that our model can achieve the best performance in this task. in the future, we will continue to adjust the model, improve the hardware configuration of the computer, collect more external data, and conduct more experiments to obtain better results.