U o R at S em E val-2021 Task 4: Using Pre-trained BERT Token Embeddings for Question Answering of Abstract Meaning (2021)

Abstract: Most question answering tasks focuses on predicting concrete answers, e.g., named entities. These tasks can be normally achieved by understanding the contexts without additional information required. In Reading Comprehension of Abstract Meaning (ReCAM) task, the abstract answers are introduced. To understand abstract meanings in the context, additional knowledge is essential. In this paper, we propose an approach that leverages the pre-trained BERT Token embeddings as a prior knowledge resource. According to the results, our approach using the pre-trained BERT outperformed the baselines. It shows that the pre-trained BERT token embeddings can be used as additional knowledge for understanding abstract meanings in question answering.

## introduction question answering (qa) is one of the machine reading comprehension tasks. the goal is to find an answer of a given question based on a given context. in most qa tasks @xcite , the answers are commonly concrete words appearing in the contexts. abstract words, on the other hand, have usually been ignored in such tasks. unlike concrete words, these abstract words are difficult to understand since they cannot be perceived directly with human senses. to study machine comprehension of abstract meaning, a task called reading comprehension of abstract meaning (recam) @xcite was proposed. unlike other qa tasks, the answers in recam are abstract words that used to summarise the information in the contexts. recam is divided into three subtasks. for subtask 1, the abstract answers are in the form of imperceptible words such as 'objective', 'chance', and 'prospective'. subtask 2 is about nonspecificity. the answers in this subtask are abstract words that represent nonspecific or general concepts such as 'vertebrate'. subtask 3 focuses on generality of the models developed in the first two subtasks. in this subtask, the model in subtask 1 must be evaluated on subtask 2 data and vice versa. based on these abstract answers, a machine has to understand the abstract meaning of certain words. this requires an additional knowledge to fulfill the lack of abstract concept information @xcite . for example, given a context which is a passage and a question shown in in this work, we propose to use a pre-trained word/token embedding model to provide external knowledge for abstract meaning understanding. the pre-trained bert token embedding is chosen in this work due to its good performance in various natural language processing tasks. in our approach, the token embeddings are firstly extracted by the pre-trained bert model. presumably, these embeddings are enriched with additional information for understanding the abstract meanings. these embeddings are used as inputs in our approach to predict answers. in this way, the prior knowledge from the pre-trained model can be leveraged in the task of abstract answer prediction. ## related work a qa task usually requires a machine to understand a context to answer a question. typically, a context is a passage and an answer usually appear somewhere in a given passage. several ap-proaches have been proposed to compute a combined passage-question representation and then use it to find an answer. the deeplstm reader @xcite uses a deep long-term short-term memory (lstm) encoder to find a representation of a concatenated passage-question text. the attentive reader @xcite attentively aggregates words to compute a passage representation based on a question. the attentive reader was modified in @xcite where a deep lstm module with the dot product attention function were replaced by a shallow bi-directional lstm module with the bi-linear attention function. recently, @xcite proposed a model called gated-attention (ga) reader. it combines a multi-hop architecture and a novel attention mechanism to learn the representations. these models have shown good performances in question answering when answers are concrete words. however, in recam task, the answers are abstract. only the given contexts may not suffice to answer the questions. recently, word/token embeddings from pretrained models have been popularly used in many applications. they are capable of providing additional knowledge from the resources they were pre-trained. several pre-trained models have been proposed in the past few years such as glove @xcite and elmo @xcite . many models have also been developed from the transformer architecture @xcite , e.g., bert @xcite , roberta @xcite and distilbert @xcite . they have been highly successful in pre-training token embeddings for various downstream tasks including qa. however, there has been no work where such models are applied in a qa task that focuses on imperceptibility and nonspecificity abstractness. each sample in the dataset of both subtasks consists of three components: passage, question and options. each passage is a long text that is used to provide context for answering a question. each question is a passage-summarized text containing one special token, @placeholder. this token is a representative of a word (an answer) that should be filled to complete the text. the correct answer must be selected from five possible options provided in each sample. given a context (a passage and a question containing a placeholder token), a placeholder embedding extracted from the pre-trained bert model should be able to guide an answer. due to the fact that the bert model considers token contexts to learn token embeddings, a placeholder embedding should comprise information of its context. that means any word or token with a similar embedding should also be in the same context as a placeholder as well. thus, to find the correct answer from a list of options, every option embedding extracted from the pre-trained bert model is compared with a placeholder embedding. then, any option with the most similar embedding compared to a placeholder embedding should be an answer. accordingly, we propose an approach that considers similarity between placeholder and option embeddings to predict answers. several metrics such as dot product, cosine similarity and euclidean distance can be considered to measure the similarity. the selected metric in this work is described in section 4. to extract placeholder and option embeddings in each sample, a question and a passage are firstly concatenated in the following form: ## experimental setup the pre-trained bert model used in this work is bert-base-uncased foot_0 @xcite . all inputs were converted to lower case and tokenized by bert-uncased tokenizer. the prediction model was trained using rmsprop optimizer for 100 epochs with the learning rate set to 0.001. a categorical cross entropy was used as a loss function. to avoid over-fitting, both l1 and l2 regularizers were added at the dense layer in both placeholder and option learning modules. the regularization factors were set to 0.001. to select the similarity metric, we examined three functions, i.e., dot product (s d ), cosine similarity (s c ) and euclideandistance-based similarity (s e ) computed by figure 1 : the input and target generated from an example shown in ## results the proposed approach was evaluated on the trial and test sets of all three subtasks. the evaluation metric is accuracy. the results of subtask 1 and 2 are shown in ## conclusion in this work, we propose to use the pre-trained bert token embeddings for qa of abstract meaning. these embeddings are additional information that help understanding abstract meanings in the tasks. according to the results, our approach outperformed the baselines in every subtask. it means that the pre-trained bert model is effective in qa of abstract meaning that focus on imperceptibility and nonspecificity. for the future work, it is worth to fine-tune the embeddings from the pre-trained model in an end-to-end manner. other resources, e.g., semantic graphs, are also worth considering to provide more information for machine comprehension of abstract meanings.