NLP - IIS @ UT at S em E val-2021 Task 4: Machine Reading Comprehension using the Long Document Transformer (2021)

Abstract: This paper presents a technical report of our submission to the 4th task of SemEval-2021, titled: Reading Comprehension of Abstract Meaning. In this task, we want to predict the correct answer based on a question given a context. Usually, contexts are very lengthy and require a large receptive field from the model. Thus, common contextualized language models like BERT miss fine representation and performance due to the limited capacity of the input tokens. To tackle this problem, we used the longformer model to better process the sequences. Furthermore, we utilized the method proposed in the longformer benchmark on wikihop dataset which improved the accuracy on our task data from (23.01% and 22.95%) achieved by the baselines for subtask 1 and 2, respectively, to (70.30% and 64.38%).

## introduction reading comprehension is the ability to understand a passage either by human or machine. one of the great benchmarks to evaluate this ability is to try to answer specific questions related to the passage @xcite . generally, this problem can contain single or multiple documents as context (containing relevant information needed to understand and answer the question), a question (a sentence with at least one asking parameter), and an answer (which is the parameter value of the question). in the task of reading comprehension of abstract meaning (recam), we have one passage as a context, one question and five candidate answers @xcite . the goal is to identify the correct answer based on the context and the given question. you can see a sample of the data in ## background and related works many approaches have been presented in the literature, from pipeline-based models to end-to-end ones. each module is also well-investigated from rule-based models to deep learning ones. despite various configurations presented in the literature to model this problem, most of the systems consist of three modules @xcite ): recent studies are provided as follows with respect to these modules that the last two modules have been merged. in the end, the longformer model is presented as our mainstay in this paper. ## method as mentioned in section 1, given a passage, a question, and a set of answers to the question, the goal is to predict the correct answer among the candidates, which can be seen as a benchmark to evaluate how well the model can comprehend the abstract meaning. to do so, we considered an end-to-end deep learning architecture based on the transformer architecture. specifically, we used contextual word embeddings based on the transformer to better discover and encode the information contained in the passage. in our model, both subtasks use the same architecture as shown in figure 1 , although we did not experiment on the possibility of multi-task learning. the word representation models are fine-tuned on the data for better performance. the fine-tuning procedure could allow us to extract additional, taskrelated information which could result in better accuracy in the evaluation phase. to model this problem, let @xmath0, c 2 , ..., c i } denote the passage as the context, where c i corresponds to the i th token (word or subword, depending on the tokenization technique used) and i is the number of tokens in the passage. similarly, the question is considered as @xmath1, q 2 , ..., q k } where k denotes the length of the question, and q k corresponds to the k th token of the question. each answer also denotes as e j which is only one abstract word (@xmath2, 2, ..., 5}). then we concate-nate the question and the candidates as: @xmath3; e 1 ; e 2 ; ...; e 5 ]. the size of this sequence is @xmath4. generally, this can be an arbitrary length based on the dataset. note that we introduce special tokens to separate the context, the question, and the candidates, similar to @xcite . specifically, we introduce the @xmath5, <@xmath6, and the @xmath7. in the case of multiple passages, all passages are concatenated to form a single context. these tokens are randomly initialized and fine-tuned. we used the longformer model introduced in @xcite as the pre-trained contextual embedding model in our method. since the context could be too long, we split the context sequence to separate chunks. each chunk length is equal to maximum sequence length the model could accept appending the sequence a; in fact, model max @xmath8. if c l denote each chunk, this sequence could be showed as: after feeding the input b to the longformer model, we apply a global attention only on a (concatenated question and answer candidates), and the rest is the context. as the longformer model utilizes a base model (like roberta without the self-attention layer, in our case), we denote this as basemodel function that outputs the encoded sequence of the input. if gattn denotes the global attention function, we have: figure 1 : the model architecture. the concatenated input vector will be encoded using the base model (like roberta without the self-attention layer, in our case). a global attention @xcite will be applied to the question and the candidate answers representations with respect to the passage as the context. the logit (score) of each ent token will be calculated using a linear transformation function, then the prediction distribution over the answer candidates (ent tokens) will be outputted using a softmax layer. finally, we obtain the logit of each candidate (<@xmath9, average over different chunks, and apply a linear transformation: ## evaluation although we only participated in the second subtask, we will evaluate our model on both subtasks here. we will explain our configurations for utilizing the model on the task as well as other baselines which are the bert-base as an alternative model and the gate-attention (ga) as our task baseline. finally, a brief discussion will be done based on the results. ## conclusion and future works we have shown how different approaches can be leveraged to machine reading comprehension of abstract meaning. we reformulated the longformer model to learn abstract meaning as a new level of semantic in machine reading comprehension. this method can also be improved by taking advantage of external knowledge and task-specific model architectures that optimize the current baseline.