R e CAM @ IITK at S em E val-2021 Task 4: BERT and ALBERT based Ensemble for Abstract Word Prediction (2021)

Abstract: This paper describes our system for Task 4 of SemEval-2021: Reading Comprehension of Abstract Meaning (ReCAM). We participated in all subtasks where the main goal was to predict an abstract word missing from a statement. We fine-tuned the pre-trained masked language models namely BERT and ALBERT and used an Ensemble of these as our submitted system on Subtask 1 (ReCAM-Imperceptibility) and Subtask 2 (ReCAM-Nonspecificity). For Subtask 3 (ReCAM-Intersection), we submitted the ALBERT model as it gives the best results. We tried multiple approaches and found that Masked Language Modeling(MLM) based approach works the best.

## introduction computers' ability to understand, represent, and express text with abstract meaning is a fundamental problem towards achieving true natural language understanding. in past decades, significant advancement has been achieved in representation learning. semeval-2021 task 4 : reading comprehension of abstract meaning (recam) @xcite explores the ability of machines to understand abstract concepts and proposes to predict abstract words just as humans do while writing article summaries. in the shared task, text passages are provided to read and understand abstract meaning. it consists of three subtasks where the first two subtasks are based on two different definitions of abstractness 1) imperceptibility @xcite and 2) non-specificity @xcite and the third subtask discusses their intersection. many cloze-style reading comprehension datasets like cnn/daily mail @xcite and children's book test (cbtest) dataset @xcite and models @xcite similar to this task exist, where a missing word has to be inferred. however, these previous datasets and models have mostly focused on inferring concrete words or concepts like named entities, but this task moves the focus from concreteness to abstractness of words in reading comprehension. this can prove to be quite useful for current ongoing research in the field of abstractive summarization. we participated in all the three subtasks. we mainly used an ensemble of bert and albert as our final model for submission on subtasks 1 and 2. we were ranked 13th on subtask 1 and 11th on subtask 2. we submitted the albert model on subtask 3. all of our code is made publicly available on github foot_0 . we approached this task in two ways. one is a multiple choice question answering (mcq) based approach and other a masked language modeling (mlm) approach. through experiments, we concluded that such tasks are best addressed using a masked language model. the rest of the paper is organised as follows. section 2 describes the problem statement formally and also gives a brief description of the dataset provided by the task organizers. section 3 introduces the related work. section 4 describes our proposed approach and section 5 gives the experimental details. we enlist our results in section 6 with a brief error analysis. finally, we give concluding remarks in section 7. ## system overview we explored multiple models and methodologies. we first experimented using an encoder with an attention based approach. from their results, we observed that an mlm based approach would work better than an mcq based approach. consequently, we tried bert and albert models and their ensemble with the mlm based approach. however, for comparison purposes, we also worked with the mcq method and its system is described below along with our other approaches. ## results and analysis the results of all the transformer based approaches are given in the bert model with the mcq approach uses the embeddings of the '[cls]' token to predict the correct option. but, it doesn't exploit the position of "@placeholder" token and hence it becomes difficult for the model to predict the correct result. we observe that our ensemble models work better on subtask 1 and subtask 2 as compared to the bert and albert models. however, on subtask 3, the albert model, which is not fine-tuned gives the best results. it owes to the fact that subtasks 1 and 2 differ a lot. if we fine-tune our model on one of the subtask, then it performs worse on the other. for our final submission, we submitted our ensemble model (8) ( for understanding the mistakes made by our submitted ensemble system, we analysed the confidence scores of our model's predictions. we performed the analysis of the system on the dev set of subtask 2. we set a threshold factor (tf) of "1.4" for deciding between confident and confused predictions. if the confidence score of the model's predicted option is p and the score for the correct option word is t , then the model is confident in its predicted answer if the following condition holds : @xmath0. also, the model makes 20% wrong predictions confidently, while in 80% of the wrong predictions, model is confused between two options. in many cases, it is unable to understand the context properly and in a few cases, the model lacks the necessary world knowledge ( we also performed a similar post-competition analysis on our ensemble system (9) ( ## conclusion the task of predicting abstract words with context from a question and article is quite novel in itself. we showed that this task can be modelled better as a masked language modeling task rather than multiple choice question answering task. the transformer based approaches worked best, where we used bert and albert models and their ensembles. these models are pretrained models and hence they perform better on our small dataset after fine-tuning. we were able to improve the results of albert model with our ensemble model on subtasks 1 and 2, but on subtask 3, the al-bert model performs better. in future, we shall try to improve our results on subtask 3. in our current approaches, we haven't used options while training the model. we can try using a pairwise ranking loss function to rank the options according to their scores with a linear layer built on top of transformer models. this will help the model to predict answers more confidently and hence might also improve results. moreover, we have used the same approach for subtask 1 and 2. in future, we aim to incorporate common sense knowledge, for example, prototypical knowledge about activities in the form of scripts @xcite @xcite , or in the form of semantic networks like conceptnet @xcite for tackling two different definitions of abstractness and incorporating some knowledge in the two subtasks.