TA - MAMC at S em E val-2021 Task 4: Task-adaptive Pretraining and Multi-head Attention for Abstract Meaning Reading Comprehension (2021)

Abstract: This paper describes our system used in the SemEval-2021 Task4 Reading Comprehension of Abstract Meaning, achieving 1st for subtask 1 and 2nd for subtask 2 on the leaderboard. We propose an ensemble of ELECTRA-based models with task-adaptive pretraining and a multi-head attention multiple-choice classifier on top of the pre-trained model. The main contributions of our system are 1) revealing the performance discrepancy of different transformer-based pretraining models on the downstream task, 2) presentation of an efficient method to generate large task-adaptive corpora for pretraining. We also investigated several pretraining strategies and contrastive learning objectives. Our system achieves a test accuracy of 95.11 and 94.89 on subtask 1 and subtask 2 respectively.

## introduction machine reading comprehension (mrc) is one of the key tasks for measuring machines' ability of understanding human languages and reasoning, it can be used broadly in real world applications such as q&a systems and dialogue systems. mrc often comes in a triplet style {passage, question, answer}, given a context passage, questions related with this passage is asked, and the machine is expected to give the answers. the question-answer form can be questionanswer pair, where the answer text is to be provided by machines, or statement form where the answer is to be filled in as cloze or multiple choices selection. by the type of answer formation, mrc can be divided into extractive and generative mrc, the former takes segments from the passage as the answer and the latter requires answer text generation based on the understanding of the passage. generative mrc is harder than extractive mrc, since it requires more on information integration and reasoning besides focusing on relevant information. one of the classic mrc approach focuses on matching networks, various network structures have been proposed to capture the semantic interaction within passages/questions/answers. recent years, pre-trained language models (lms) have brought non-trivial progress to the performance on mrc, and there's a decline of complex matching networks @xcite . plugging matching networks on top of pre-trained lms can see either improvements or degradation in performance @xcite . multiplechoice mrc (mmrc) often lacks abundant training data for deep neural networks (this might be caused by the expensive human labelling cost) and it results in a limitation to take full advantage of the pre-trained lms. the semeval-2021 task 4 reading comprehension of abstract meaning @xcite , is a multiple-choice english mrc task, aiming at investigating the machine's ability to understand abstract concepts in two aspects: subtask 1, nonconcrete concepts, e.g. service/economy compared with trees/red; subtask 2, generalized/summarized concepts, like vertebrate compared with monkey. we propose an approach based on the pre-trained lm electra @xcite , with an ensemble of multi-head attention @xcite multiple-choice classifier, and wae @xcite to get the final prediction. first, we conduct task-adaptive pretraining, which is transfer learning using in-domain data on the electra model. then we fine-tune the recam task using a multi-head attention multiple choice classifier (mamc) on top of the electra model. finally we enhance the system with wae and ensemble them all to get the best generalization capability. ## system overview figure 1 illustrates the overall architecture of our system. the options are substituted into the query to form a complete context, rather than separate query/option segments, in order to get a less semantically ambiguous representation of the query and option. the option-filled query and context tokens are concatenated as in figure 1 , wrapped by @xcite token and [sep] tokens. token embeddings are added up with segment embeddings and positional encodings to form the input for the pretrained encoder. then the representations from the encoder are put through a multi-head attention multiple choice classifier, which consists of 1) a 2 layer multi-head attention feed forward network to further capture the task specific query-context interactions, 2) a pooler and a linear transformation to get the final cross entropy loss. we first conduct task-adaptive pretraining on the system, and then fine-tune on the recam dataset, the final model is an ensemble model by several generalization techniques including wrong answer ensemble. ## conclusion our system takes the large pre-trained lm elec-tra, and enhance it with in-domain transfer learning and a multi-head multiple-choice classifier on top. we compared the benchmark performance of different pre-trained lms (bert, albert and electra) on the semeval-2021 task 4, the result shows that different pretraining objective/dataset can lead to different inclination of model knowledge and large performance discrepancy on the downstream task. task-adaptive pretraining has contributed the main improvement, and multi-head multiple-choice classifier and wae bring marginal improvement. we also investigated into option contrastive learning and multitask learning, the degradation of performance suggests that triplet and n-tuplet contrastive loss is not suitable for this task and nli is not generally beneficial for mmrc tasks.