LRG at S em E val-2021 Task 4: Improving Reading Comprehension with Abstract Words using Augmentation, Linguistic Features and Voting (2021)

Abstract: We present our approaches and methods for SemEval-2021 Task-4 Reading Comprehension of Abstract Meaning. Given a question with a fill-in-the-blank, and a corresponding context, the task is to predict the most suitable word from a list of 5 options. There are three subtasks: Imperceptibility, Non-Specificity and Intersection. We use encoders of transformers-based models pretrained on the MLM task to build our Fill-in-the-blank (FitB) models. Moreover, to model imperceptibility, we define certain linguistic features, and to model non-specificity, we leverage information from hypernyms and hyponyms provided by a lexical database. Specifically, for non-specificity, we try out augmentation techniques, and other statistical techniques. We also propose variants, namely Chunk Voting and Max Context, to take care of input length restrictions for BERT, etc. Additionally, we perform a thorough ablation study, and use Integrated Gradients to explain our predictions on a few samples. Our models achieve accuracies of 75.31% and 77.84%, on the test sets for subtask-I and subtask-II, respectively. For subtask-III, we achieve accuracies of 65.64% and 64.27%.

## introduction a very common assessment in schools is questionanswering based on a given "comprehension passage". students are given a comprehension passage, from which they are supposed to glean necessary information, and answer short questions (such as fill-in-the-blanks-type question) based on what they have garnered from the given passage. while trying to find the most appropriate word for the blank, the children look at the words surrounding the blank * equal contribution. author ordering determined by coin flip. semeval-2021 task-4 @xcite focuses on a similar idea. every sample has an article, and a corresponding question. the question has a blank which the model is supposed to predict from a set of 5 options. the novelty in the task lies in its 3 subtasks: imperceptibility (subtask-i), non-specificity (subtask-ii), and intersection (subtask-iii). a description of these subtasks is given in section 3. in this work, we propose using bert and its derivative models such as distilbert @xcite , albert @xcite and roberta @xcite . further, we propose 2 bert variants: (1) bert voting; (2) bert max. context. most importantly, we also model the concepts of imperceptibility and non-specificity. for imperceptibility, we create statistical embeddings using features that have a high correlation with concreteness. for non-specificity, we propose two approaches: (1) we augment the dataset by replacing some nouns in the article by their hypernyms; and (2) we use the options' hyponyms to decide the most appropriate option. we also experiement with ga-reader @xcite and gsamnbased approaches @xcite by trying out their various combinations with bert. in section 2, we perform a succinct literature survey. section 3 elucidates our approach, including the modelling aspect, the various variants of the base model, and the different ways we model imperceptibility and non-specificity. in section 4, we perform an extensive ablation and comparative study. ## background the advent of large-scale question answering systems began with straightforward tasks, like the one introduced by the simplequestions dataset @xcite , which consisted of knowledgebase fact triples which were later used to answer questions. however, this dataset would only judge a model based on the ability to relate the facts to the question at hand. the purpose of nlp research is to be able to create a generalised model that may answer questions based on any context, thus datasets like the cnn daily mail @xcite and squad @xcite were created. in a typical question-answering dataset, an original and anonymised context is provided before each question. before transformers, methods consisting of lstm/grus were used to achieve good results on the aforementioned tasks. these datasets however, always had answers in the passage. the cloth @xcite dataset focuses on passages from middle-school and high-school text, with multiple fill-in-the-blanks in the passage. the recam @xcite dataset puts a twist to archetypal fill-in-the-blank datasets by providing answer choices that are abstract in some form and which are not available in the passage itself. the models created for the qa task have to take into account semantic relations between the options and the context. ga-reader @xcite , is one such model, which utilises a multi-hop architecture with a novel attention mechanism, that serves as a baseline to this task. ## experimental setup in all our experiments, we use the pytorch implementations of the transformers-based models provided by the huggingface @xcite . the metric for all the 3 subtasks is accuracy. for subtask-i, to obtain the linguistic features mentioned in 3.2, and to obtain the hypernyms and hyponyms for subtask-ii, we use the lexical database, wordnet provided by nltk @xcite , a library in python. for both subtasks, we train our models on train + trial dataset, and evaluate them on the dev set. the training and the evaluation of systems was on google colaboratory's free gpu (nvidia k80/p100). the training time varies with the models. it is around 1-2 hours for the base variants and 2-4 hours for the large models, which is well within the 12 hour limit of colab. distilbert took about half an hour for training. for finetuning the bert fitb hypr aug model on the augmented dataset on the mlm task, we use nvidia-dgx station with the following specifications: four 32 gb tesla v100 gpus, 256 gb ram and forty intel xeon 2.20ghz processors since it is a computationally intensive task. ## conclusion we tried out myriad approaches, taking care to not only focus on the architecture aspect, but also how we can quantify imperceptibility and nonspecificity. although we did not achieve favourable improvements in all approaches, we did observe gains in accuracy on the dev set. we reckon that with more careful tuning of parameters such as the threshold in the difference method, we will be able to achieve these gains on the test set. we further interpreted the outputs of transformers-based models using integrated gradients, and demonstrated that transformer models are able to learn the concepts of imperceptibility and non-specificity. in the future, we intend to solidify our proposed approaches and carry out further research in this interesting field.