IIE - NLP -Eyas at S em E val-2021 Task 4: Enhancing PLM for R e CAM with Special Tokens, Re-Ranking, S iamese Encoders and Back Translation (2021)

Abstract: This paper introduces our systems for all three subtasks of SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning. To help our model better represent and understand abstract concepts in natural language, we well-design many simple and effective approaches adapted to the backbone model (RoBERTa). Specifically, we formalize the subtasks into the multiple-choice question answering format and add special tokens to abstract concepts, then, the final prediction of QA is considered as the result of subtasks. Additionally, we employ many finetuning tricks to improve the performance. Experimental results show that our approach gains significant performance compared with the baseline systems. Our system achieves eighth rank (87.51%) and tenth rank (89.64%) on the official blind test set of subtask 1 and subtask 2 respectively.

## introduction the computer's ability in understanding, representing, and expressing abstract meaning is a fundamental problem towards achieving true natural language understanding. semeval-2021 task 4: reading comprehension of abstract meaning (recam) provides a well-formed benchmark that aims to study the machine's ability in representing and understanding abstract concepts @xcite . the reading comprehension of abstract meaning (recam) task is divided into three subtasks, including imperceptibility, nonspecificility, and interaction. please refer to the task description paper @xcite for more details. to address the above challenges in recam, we first formalize all subtasks as a type of multiple-choice § corresponding author. ¶ our code is publicly available at https://github. com/indexfziq/iie-nlp-eyas-semeval2021 . question answering (qa) task like @xcite . recently, the large pre-trained language models (plms), such as gpt-2 @xcite , bert @xcite , roberta @xcite , demonstrate their excellent ability in various natural language understanding tasks @xcite @xcite . so, we employ the state-of-the-art plm, roberta, as our backbone model. moreover, we design many simple and effective approaches to improve the performance of the backbone model, such as adding special tokens, sentence re-ranking, label smoothing and back translation. this paper describes approaches for all subtasks developed by the iie-nlp-eyas team (natural language processing group of institute of information engineering of the chinese academy of sciences). our contributions are summarized as the followings: ## approaches since the format of the tasks in recam is the same, we use the unified framework to address all tasks. the following is the detail of our methods. ## conclusion in this paper, we design many simple and effective approaches to improve the performance of the plms on all three subtasks. experiments demonstrate that the proposed methods achieve significant improvement compared with the plms baseline and we obtain the eighth-place in subtask-1 and tenth-place in subtask-2 on the final official evaluation. moreover, we show that special tokens are useful features contributing to most of the system's boost, which work well in enhancing plms for representating and understanding abstract concepts.