PINGAN Omini-Sinitic at S em E val-2021 Task 4:Reading Comprehension of Abstract Meaning (2021)

Abstract: This paper describes the winning system for subtask 2 and the second-placed system for subtask 1 in SemEval 2021 Task 4: ReadingComprehension of Abstract Meaning. We propose to use pre-trianed Electra discriminator to choose the best abstract word from five candidates. An upper attention and auto denoising mechanism is introduced to process the long sequences. The experiment results demonstrate that this contribution greatly facilitatesthe contextual language modeling in reading comprehension task. The ablation study is also conducted to show the validity of our proposed methods.

## introduction reading comprehension of abstract meaning (re-cam) @xcite is a cloze-style task, which takes a document and related human written abstract with one word replaced by a placeholder as input. the model is required to choose the best word from five candidates. the recam consists of three subtasks. in subtask 1 and 2, participating systems are required to choose the best imperceptible concept and hyper-nyms concepts word respectively. subtask 3 aims to evaluate performance of a system trained on one definition and test on the other. traditional cloze-style reading comprehension model (sa reader) @xcite uses attention to directly pick the answer from the context, which makes model incapable to answer the question where the answer does not appear in passage. furthermore, ga reader @xcite adopts multi-hop attention mechanism to build query-specific representation of answer for ranking the candidates which is not part of passage. pre-trained language models @xcite @xcite have been widely adopted for context modeling in many natural language processing tasks. these models are pre-trained on huge corpora with plain texts and can better model contextual dependencies of tokens, thus enhance the performance of downstream approaches. as described in @xcite @xcite , they improved the performance of single-choice reading comprehension tasks by introducing pre-trained model, but they takes excessive memory for concatenating each option with the question and the passage. gpt2 @xcite has outperformed the sota result on cloze-style task cbt @xcite . as stated in @xcite , gpt2 computes the probability of each choice and the rest of the sentence conditioned on this choice according to the pre-trained model, the answer is the choice with highest probability. gpt2 outperforms rnn-based models without fine-tuning on cbt task, we assume that pre-trained language model has better potential to address the cloze-style problem than fine-tuning the pre-trained model with an additional ranking network. the most relevant work to our model is pattern-exploiting training (pet) @xcite , which proposed to reformulate the sentence classification task to cloze-style task with defined golden answer word as supervising signal. the comparison between pet and our proposed method is reported in section 5.