Noobs at S emeval-2021 Task 4: Masked Language Modeling for abstract answer prediction (2021)

Abstract: This paper presents the system developed by our team for Semeval 2021 Task 4: Reading Comprehension of Abstract Meaning. The aim of the task was to benchmark the NLP techniques in understanding the abstract concepts present in a passage, and then predict the missing word in a human written summary of the passage. We trained a Roberta-Large model trained with a masked language modeling objective. In cases where this model failed to predict one of the available options, another Roberta-Large model trained as a binary classifier was used to predict correct and incorrect options. We used passage summary generated by Pegasus model and question as inputs. Our best solution was an ensemble of these 2 systems. We achieved an accuracy of 86.22% on subtask 1 and 87.10% on subtask 2.

## introduction there has been a lot of research in evaluating the performance of machine learning models to identify concrete concepts present in text and answer questions based on it @xcite . the organizers of recam task at semeval 2021 have provided a dataset to benchmark the models' performance on understanding the abstract concepts in the text in english language. the models are required to predict the missing words in a human written summary of the passage. this can help assess if the models can accurately capture the important concepts and meaning in the text. the paper is organized as follows: in section 2, we give a background on the problem and method that has been used, in section 3, we present the proposed system architecture, in section 4, we present the hyperparameters analysis done on the system, in section 5, we present the results of the proposed architecture along with other approaches taken, and in section 6, we conclude the paper. ## system overview the key components in our proposed system are abstractive summarization of context using pegasus, roberta for masked language modelling and roberta for sequence classification. algorithm 1 presents our system's algorithm for predicting the option using the given context and masked question. robertaformaskedlm the task at hand can be converted into predicting the masked token by replacing the @placeholder in question with the [mask] token. we use robertaformaskedlm to predict the masked token. to help the model get the context in question, we prepend the summary from pegasus-xsum to the masked question text. to finetune the model, we train it by passing concatenation of summary and masked question as input and providing concatenation of summary and filling the question with correct blank as output. once the model is trained, we use huggingface's @xcite pipeline to predict the top 5 words for filling the masked token. out of five predictions, word which has the highest probability of filling the blank and which is present in the given options is selected as the answer. roberta for sequence classification in some of the cases, we observed that the predicted words weren't present in the options. for handling such cases, we use roberta for sequence classification. we convert the task into a binary classification task by filling the masked question with correct option and treating it as one class and by filling the mask with wrong options as another class. this leads to a class-imbalanced dataset (1:4 ratio). to handle this, we randomly selected equal number of wrong-option class. on the test set, option which achieves the highest softmax score out of all the given options, is selected as the answer. ## experimental setup all the experiments were conducted in a google colab system with 12gb ram and t4-nvidia gpu. we experimented with different settings in three components in our system: summary extraction, masked language model (mlm) and handling cases when word predicted by mlm is not present in the options. ## results we have mentioned the accuracy of all the systems developed by us for subtasks 1 and 2 in ## conclusion we have described the systems developed by as to solve the reading comprehension challenge at semeval 2021. in our best performing submission, we framed the problem as a masked language modeling task. we used the predictions from a separately trained binary classifier when the above system failed to generate words which were not part of the options. our models were able to achieve high accuracy with a relatively simple setup. we were ranked 11th out of 23 participants in subtask 1 and 12th out of 28 participants in subtask 2. as part of future work, we aim to use information from knowledge bases such as conceptnet. this can help extract broader concepts related to the words predicted by the masked language model.