S em E val-2021 Task 4: Reading Comprehension of Abstract Meaning (2021)

Abstract: This paper introduces the SemEval-2021 shared task 4: Reading Comprehension of Abstract Meaning (ReCAM). This shared task is designed to help evaluate the ability of machines in representing and understanding abstract concepts. Given a passage and the corresponding question, a participating system is expected to choose the correct answer from five candidates of abstract concepts in cloze-style machine reading comprehension tasks. Based on two typical definitions of abstractness, i.e., the imperceptibility and nonspecificity, our task provides three subtasks to evaluate models’ ability in comprehending the two types of abstract meaning and the models’ generalizability. Specifically, Subtask 1 aims to evaluate how well a participating system models concepts that cannot be directly perceived in the physical world. Subtask 2 focuses on models’ ability in comprehending nonspecific concepts located high in a hypernym hierarchy given the context of a passage. Subtask 3 aims to provide some insights into models’ generalizability over the two types of abstractness. During the SemEval-2021 official evaluation period, we received 23 submissions to Subtask 1 and 28 to Subtask 2. The participating teams additionally made 29 submissions to Subtask 3. The leaderboard and competition website can be found at https://competitions.codalab.org/competitions/26153 . The data and baseline code are available at https://github.com/boyuanzheng010/SemEval2021-Reading-Comprehension-of-Abstract-Meaning .

## introduction humans use words with abstract meaning in their daily life. in the past, research efforts have been exerted to better understand and model abstract meaning @xcite @xcite . modelling abstract meaning is closely related to many other nlp tasks such as reading comprehension, metaphor modelling, sentiment analysis, summarization, and word sense disambiguation. in the past decade, significant advancement has been seen in developing computational models for semantics, based on deep neural networks. in this shared task, we aim to help assess the capability of the state-of-the-art deep learning models on representing and modelling abstract concepts in a specific reading comprehension setup. we introduce semeval-2021 task 4, reading comprehension of abstract meaning (recam). specifically, we design this shared task by following the machine reading comprehension framework @xcite @xcite , in which computers are given a passage d i as well as a human summary s i to comprehend. if a model can digest the passage as humans do, we expect it to predict the abstract word used in the summary, if the abstract word is masked. unlike the previous work that requires computers to predict concrete concepts, e.g., named entities, in our task we ask models to fill in abstract words removed from human summaries. during the semeval-2021 official evaluation period, we received 23 submissions to subtask 1 and 28 submissions to subtask 2. the participating teams additionally made 29 submissions to subtask 3. in this paper, we induce the shared task and provide a summary for the evaluation. ## data construction we develop our multi-choice machine reading comprehension datasets based on the xsum summarization dataset @xcite . we first locate words with abstract meaning using our abstractness scorers. then we perform data filtering to select our target words to construct our datasets. ## systems and results our shared task received 23 submissions to subtask 1, 28 submissions to subtask 2, and 29 submissions to subtask 3. we use accuracy as the evaluation metric for the three subtasks. in general, most participating teams use pretrained language models in their systems such as bert @xcite , albert @xcite , distilbert @xcite , roberta @xcite , electra @xcite , deberta @xcite , xl-net @xcite , t5 @xcite . data augmentation, external knowledge resources, and/or transfer learning are additionally used by many teams to further enhance their model performance. ## related work there have been tasks being proposed to evaluate machines' ability on reading comprehension, which either require models to find an entity or text span from the source document as the answer @xcite @xcite @xcite , or further generate an answer @xcite @xcite . the cloze-style mrc tasks @xcite @xcite are most similar to ours, in which the missing words in the cloze questions are entities appearing in source documents. unlike previous work, recam questions specifically focus on abstract words unseen in the corresponding source documents. in general, multi-choice questions have been widely used as a tool for language examination to test both humans and machines. in this paper, we follow the multiple-choice framework for our proposed recam task to evaluate computers' ability in comprehending abstract concepts, in which computers are asked to predict the missing abstract words in human-written summaries. ## summary this shared task aims to study the ability of machines in representing and understanding abstract concepts, based on two definitions of abstractness, the imperceptibility and nonspecificity, in a specific machine reading comprehension setup. we provide three subtasks to evaluate models' ability in comprehending the two types of abstract meaning as well as their generalizability. in subtask 1, the top system achieves an accuracy of 0.951, and in subtask 2, an accuracy of 0.953, suggesting the current systems perform well in the specific setup of our share task. in subtask 3, we found that in general the models' performances dropped in both subtask 2-to-1 and subtask 1-to-2 testing. however, some models generalize well, benefiting from technologies such as data augmentation and task adaptive training. we hope the shared task can help shed some light on modelling abstract concepts and help design more challenging tasks in the future.