XRJL - HKUST at S em E val-2021 Task 4: W ord N et-Enhanced Dual Multi-head Co-Attention for Reading Comprehension of Abstract Meaning (2021)

Abstract: This paper presents our submitted system to SemEval 2021 Task 4: Reading Comprehension of Abstract Meaning. Our system uses a large pre-trained language model as the encoder and an additional dual multi-head co-attention layer to strengthen the relationship between passages and question-answer pairs, following the current state-of-the-art model DUMA. The main difference is that we stack the passage-question and question-passage attention modules instead of calculating parallelly to simulate re-considering process. We also add a layer normalization module to improve the performance of our model. Furthermore, to incorporate our known knowledge about abstract concepts, we retrieve the definitions of candidate answers from WordNet and feed them to the model as extra inputs. Our system, called WordNet-enhanced DUal Multi-head Co-Attention (WN-DUMA), achieves 86.67% and 89.99% accuracy on the official blind test set of subtask 1 and subtask 2 respectively.

## introduction recently, there has been an increasing interest on machine reading comprehension (mrc). while most mrc studies such as cnn/daily mail @xcite focus on concrete concepts, se-meval 2021 task 4, reading comprehension on abstract meaning (recam), targets abstract concept understanding, including imperceptibility in subtask 1 and nonspecificity in subtask 2. the former, imperceptibility, highlights the abstract words that refer to ideas and concepts that do not correspond directly to human perception. the latter is for hypernyms and abstract concepts such as the class of vertebrate which includes whales as a concrete subclass @xcite . in this task, given news fragments and incomplete abstracts, the machine needs to select the most suitable abstract words from candidate answers. figure 1 shows one example of recam subtask 1. passage is the news sections and question is a human written summary in which abstract words have been removed. machines are requested to choose abstract words from five candidates for replacing @placeholder. for this shared task, we regard both subtasks as multi-choice mrc tasks. various deep neural networks and attention mechanisms (e.g. @xcite @xcite ) have been proposed to address these tasks. in our work, following the state-of-the-art model duma @xcite , we adopt a pre-trained language model (prlm) as encoder and extend with an additional dual multi-head co-attention layer to strengthen the relationship between passages and questionanswer pairs. for the dual multi-head attention layer, while duma builds passage-question and question-passage attention modules in a parallel way to simulate the transposition thinking process, our model stacks two attention modules in order to simulate the process of re-considering for a deeper understanding of the passage. more details on our attention calculation process can be found in section 2. furthermore, we add an additional layer normalization module immediately after the attention module. from our experiments, we found that this additional normalization module definitely improves our model's performance. our most significant design decision is to use wordnet @xcite to expand on the abstract concepts in the candidate answers. intuitively, expanding an abstract concept according to its definition in a dictionary should help as it helps relate the abstract concept with others that may occur in the text. a key conclusion that we can draw from our experiments is that this is indeed the case. one problem that we encountered when implementing this idea was that most english words have multiple entries in wordnet. for example, bank in wordnet can have noun definitions as well as verb definitions. we addressed this problem by using some heuristics and some additional information such as part-of-speech labels. because of the significant role played by wordnet, we call our system wordnet-enhanced dual multi-head co-attention (wn-duma). we remark that our system did not use any additional training data for the tasks. in the final evaluation, our model is ranked 10 out of 23 and 9 out of 28 on the official subtask 1 and subtask 2 blind test set with 86.67% and 89.99% accuracy, respectively. the code for our model is publicly available 1 . the rest of the paper is organized as follows. section 2 gives the details of our system. section 3 describes our experimental setup including the datasets and hyper parameters used for training. section 4 presents experimental results. section 5 concludes this paper with some final remarks. ## system description in this section, we describe the framework of our end-to-end model wn-duma. figure 2 depicts the detailed architecture of our approach, with inputs at the bottom and outputs at the top. wordnet-enhanced encoder we regard both subtask 1 and 2 as multi-choice mrc problems. such a problem includes a passage, a question with a @placeholder, and 5 candidate answers to choose from. first, we replace @placeholder in the question with the given 5 candidate answers to form 5 options. in the tasks, the candidate answers are all single words with abstract meanings, so we decided to add some extra knowledge from wordnet @xcite to help the system better understanding the abstract meanings. more specifically, for a single candidate answer, we find its part-of-speech tag based on the option it's located in, and extract its definitions under this part-of-speech tag. after tokenization, every instance is cast into the input form: [cls] passage [sep] option + answer definition [sep]. to encode input tokens into representations, we feed them through a prlm based on transformer to obtain sequence embeddings, which draws a global relationship between the passage and the option-definition. dual multi-head co-attention layer based on the above process, we further separate the output representations from transformer encoder to acquire the passage context embeddings e @xmath0, where l p , l od denote the maximum length of passage and optiondefinition respectively. then based on the bidirectional matching network of duma which is quite similar to the multi-head self-attention module in vanilla transformer block @xcite , we first take e od as query, e p as key and value to calculate one of the co-attention representations, which simulates the process of human re-reading the passage with impression of option and definition. the formulas are listed as follows: classifier here the co-attention representations rep 1 and rep 2 are merged and used for final classification: ## experimental setup data and metric we used the official datasets @xcite provided by semeval 2021 task 4 competition. they were collected from bbc news in english language. some basic statistics are listed in hyper parameters all of our codes are written based on pytorch foot_0 . to extract the word definition of candidate answers, we use nltk toolkit @xcite . the transformer encoder we used is pretrained albert-xxlarge-v2 model foot_1 . since the code of duma is not open-source, we reimplement it by only using one co-attention layer where the attention heads are 64 and the dimension of query, key and value are all 64, because it is pointed that more co-attention layers do not improve the performance @xcite . the setting is also applied to our wn-duma for fair comparison. due to limited resources, the maximum sequence length of input tokens is set to 150 for both subtask 1 and subtask 2. in fact, we found that sequence length longer than 150 can only slightly improve the model performance. we choose minibatch size equal to 2, and the adamw optimizer @xcite with an initial learning rate of 5e-06. we use some strategies for more stable training: 1) clip the gradient norm to 10; 2) adopt a linear scheduler with warm up of the first 10% training steps. to avoid overfitting, we apply 0.1 dropout @xcite rate to the co-attention layer. we trained all the models for 3 epochs, evaluate on the dev set at every 200 training steps and save the model with the best dev accuracy. for each single model, we run experiments for 5 times with different random seeds and use the average as the ultimate performance. ## conclusion in this paper, we describe our submitted system in semeval 2021 task 4 recam. unlike previous mrc datasets, recam focus more on machine's ability in understanding and representing abstract concepts. in order to provide more knowledge of abstract word, we extract wordnet definitions for each candidate answer based on part-of-speech tags. in addition, our proposed wn-duma model consists of a prlm as the encoder and a dual multihead co-attention layer to enhance the relationship between passage and question-answer pairs as human's re-considering process. our wn-duma model improves the performance of our baseline model duma on these datasets. there are some limitations in our experiments. firstly, training data size of this task is limited compared to other mrc tasks, with less than 3400 training pairs in both subtasks. this is understandable as collecting labeled data in many natural language processing tasks is expensive. secondly, using albert xxlarge prlm, we only set 150 as the maximum text length in our experiments due to device limitation. important sentences in the passage that are highly relevant to the summary are sometimes not covered. for prlms, their performance always improve as the number of their parameters increase. the use of large pre-trained models sometimes requires the sacrifice of context. for our future work, we plan to explore ways to train models more efficiently with limited amount of labeled data, and to design more cost-effective models to deal with long input texts.