ZJUKLAB at S em E val-2021 Task 4: Negative Augmentation with Language Model for Reading Comprehension of Abstract Meaning (2021)

Abstract: This paper presents our systems for the three Subtasks of SemEval Task4: Reading Comprehension of Abstract Meaning (ReCAM). We explain the algorithms used to learn our models and the process of tuning the algorithms and selecting the best model. Inspired by the similarity of the ReCAM task and the language pre-training, we propose a simple yet effective technology, namely, negative augmentation with language model. Evaluation results demonstrate the effectiveness of our proposed approach. Our models achieve the 4th rank on both official test sets of Subtask 1 and Subtask 2 with an accuracy of 87.9% and an accuracy of 92.8%, respectively. We further conduct comprehensive model analysis and observe interesting error cases, which may promote future researches. The code and dataset used in our paper can be found at https://github.com/CheaSim/SemEval2021 . The leaderboard can be found at https://competitions.codalab.org/competitions/26153 .

## introduction past decades have witnessed the huge progress of representation learning in natural language processing (nlp). with pre-trained language models, machine reading comprehension (mrc) models can extract answers from given documents and even yield better performance than humans on benchmark datasets such as squad @xcite . however, these successes sometimes lead to the hype in which these models are being described as "understanding" language or capturing "meaning" @xcite . note that the intention of mrc is letting the systems read a text like human beings, extracting text information and understanding the meaning of a text then answering questions, which means the systems can not only conclude the semantic of the text but also comprehend the abstract concepts under the constraint of general knowledge regarding the world @xcite . nevertheless, little works as well as benchmarks focus on this direction. semeval 2021 task4 @xcite is an mrc task that focuses on evaluating the model's ability to understand abstract words. reading comprehension of abstract meaning (recam) task is divided into three subtasks including subtask 1: recam-imperceptibility, subtask 2: recam-nonspecificity and subtask 3: recam-intersection. unlike previous mrc datasets such as cnn/daily mail @xcite , squad @xcite , and coqa @xcite that request computers to predict concrete concepts, e.g. named entities. this task challenges the model's ability to fill the abstract words removed from human-written summaries based on the english context. note that this task's input format is similar to the mlm pre-training task of bert @xcite , which aims to predict the mask tokens. pretrained language models (plms) such as bert @xcite , roberta @xcite , albert @xcite , deberta @xcite have achieved success on mrc tasks. inspired by this, we introduce a simple yet effective method, namely, negative augmentation with language model (nal) in semeval 2021 task4. specifically, we augment the answer distribution with an additional negative candidate from the mask language model's prediction. previous work @xcite indicates that the pre-trained language model has already captured much world knowledge. thus, we argue that knowledge can help guild the model training and identify those ambiguous abstract meanings. further, we introduce other technologies such as label smoothing, domain-adaptive pre-training in our system. we describe the detailed approaches used for the subtasks in section 3. we conduct comprehensive experiments in section 3, and we achieve the 4th system for subtask 1: recam-imperceptibility and the 4th system for subtask 2: recam-nonspecificity in the leaderboard. in our experiments, we observe that plms without fine-tuning can easily get 60+% accuracy on both subtask 1 and subtask 2, demonstrating that pre-trained language models already capture some abstract meanings. we further find that our negative augmentation with language model can improve the performance with 2.6% in subtask 1 and 4.6% in subtask 2. finally, we conduct error analysis to promote future researches. ## background machine reading comprehension (mrc) has received increasing attention recently, which is a challenging task. according to the type of the answer, reading comprehension tasks can be divided into four categories @xcite ): 1) cloze-style: the question contains a "@placeholder," and the system must choose a word or entity from the set of candidate answers to fill in the "@placeholder" to make the sentence complete. 2) multiple choice: in this type of task, choosing a suitable answer from k sets of given answers. this answer can be one word or a sentence. 3) span prediction: this kind of task is also called (extractive question answering), which requires the system to extract a suitable range of text fragments from a given original text based on the question as to the answer. 4) freeform answer: this task allows the answer to be any type of text, which is necessary to mine deeplevel contextual semantic information according to a given question and a collection of candidate documents, and even combine multiple articles to give the best answer. in semeval 2021 task4, it requires the system to have a strong ability of reading comprehension not only because the task is the cloze-style format as mentioned above but also the abstract words in answers. there are two definitions of abstract words: imperceptibility and nonspecificity. concrete words refer to things, events, and properties that we can perceive directly with our senses @xcite . compared to concrete words like "trees" and "red," abstract words for imperceptibility are created by humans instead of pointing the things in the natural world. for example, as shown in the difference between subtask 1 and subtask 2 is the definition of abstract words. so the input of both subtask 1 and subtask 2 are the same. the input of these tasks are shown in ## conclusion this paper presents our system design for the se-meval 2021 task4. we propose a simple yet effective method called negative augmentation with language model. comprehensive experiments demonstrate the effectiveness of our proposed approach. we also conduct case studies and investigate why the model fails to obtain the correct prediction. note that language models are pre-trained from the huge corpus; recently, researchers have identified the bias in the language model, which may mislead the model prediction. our proposed negative augmentation with language model can help the model better discriminate candidates in finetuning, thus boost the performance. from another perspective, as depicts in section 3.2, the language model without any fine-tuning gets 60+% accuracy in both subtask 1 and subtask 2. this indicates that bias exists in the datasets (part of the abstract meaning can be obtained from the language model). more strong benchmarks should be constructed in the future.