ANNA : Enhanced Language Representation for Question Answering (2022)

Abstract: Pre-trained language models have brought significant improvements in performance in a variety of natural language processing tasks. Most existing models performing state-of-the-art results have shown their approaches in the separate perspectives of data processing, pre-training tasks, neural network modeling, or fine-tuning. In this paper, we demonstrate how the approaches affect performance individually, and that the language model performs the best results on a specific question answering task when those approaches are jointly considered in pre-training models. In particular, we propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7% F1 and 90.6% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet on the SQuAD 2.0 benchmark.

## introduction question answering (qa) is the task of answering given questions, which demands a high level of language understanding and machine reading comprehension abilities. as pre-trained language models based on a transformer encoder @xcite have brought a huge improvement in performance on a broad range of natural language processing (nlp) tasks including qa tasks, methodologies for qa tasks are widely used to develop applications such as dialog systems @xcite and chat-bots @xcite in a variety of domains. pre-trained language models like bert @xcite are designed to represent individual words for contextualization. however, recent extractive qa tasks such as stanford question answering dataset (squad) benchmarks @xcite figure 1 : example of a passage with a pair of question and answer sampled from the squad 1.1 dataset. @xcite involve reasoning relationships between spans of texts that include a group of two or more words in the evidence document @xcite . in the example, as shown in figure 1 , "a golden statue of the virgin mar", the correct answer for the question "what sits on top of the main building at notre dame?", is a group of words consisting of nouns and other words and is called as a noun phrase, which performs as a noun in a sentence. since predicting a span of answer texts including a start and end positions may be challenging for self-supervised training rather than predicting an individual word, we introduce a novel pre-training approach that extends a standard masking scheme to wider spans of texts such as a nounphrase rather than an entity level and prove that this approach is more effective for an extractive qa task by outperforming existing models. in this paper, we present a new pre-training approach, anna (approach of noun-phrase based language representation with neighboraware attention), which is designed to better under-stand syntactic and contextual information based on comprehensive experimental evaluation of data processing, pre-training tasks, attention mechanisms. first, we extend the conventional pretraining tasks. our models are trained to predict not only individual tokens but also an entire span of noun phrases during the pre-training procedure. this noun-phrase span masking scheme lets models learn contextualized representations in the whole span level, which benefits predicting answer texts for the specific extractive qa tasks. second, we enhance the self-attention approach by incorporating a novel neighbor-aware mechanism in the transformer architecture @xcite . we find that more consideration of relationships between neighboring tokens by masking diagonality in attention matrix is helpful for contextualized representations. additionally, we use a larger volume of corpora for pre-training language models and find that using a lot of additional datasets does not guarantee the best performance. we evaluate our proposed models on the squad datasets which is a major extractive qa benchmarks for pre-trained language models. for squad 1.1 task, anna achieves new state-of-theart results of 90.6% exact match (em) and 95.7% f1-score (f1). when evaluated on the squad 2.0 development dataset, the results show that our proposed approaches obtain competitive performance outperforming self-supervised pre-training models such as bert, albert, roberta, and xlnet models. we summarize our main contributions as follows: ## methodology we introduce a novel transformer encoder architecture integrating a new neighbor-aware mechanism for pre-training a language model. figure 2 demonstrates the architecture of anna model. anna extends the original transformer encoder blocks by including a neighbor-aware self-attention layer stacked on a multi-head self-attention layer. ## experiments in this section, we present the fine-tuning results of anna transferred to specific extractive question answering tasks. we evaluate anna on squad 1.1 and 2.0 tasks that are well-known machine reading comprehension benchmarks in the nlp area, and some nlu tasks. the dataset of squad 1.1 consists of around 100k pairs of a question and an answer along with wikipedia passages where the answers are included. this task is to predict a correct span of an answer text for a given question from the corresponding wikipedia passage @xcite . for squad 2.0, the dataset is extended to the squad 1.1 dataset by combining over 50,000 unanswerable questions, so that systems are required to predict answers to both answerable and unanswerable questions @xcite . we follow the finetuning procedure of bert @xcite , but the provided squad training dataset only is used for fine-tuning, while bert augments its training dataset with other qa datasets available in public. squad 1.1 squad 2.0 anna is evaluated on squad 2.0 development dataset, and the results are compared with the published pre-trained language models @xcite @xcite @xcite in ## model analysis we conduct additional experiments in terms of perspectives such as data processing, pre-training task, and attention mechanisms. we report a detailed analysis of how those approaches affect the performance of anna on a specific downstream task individually. in this study, anna base model is used for these additional experiments due to the limitation of computing resources. ## conclusion in this paper, we present a novel pre-trained language representation model, anna which improves the original transformer encoder architecture by collaborating a neighbor-aware mechanism, and is pre-trained for contextualized representations of words and noun phrases in a span level. model squad1.1 squad1.1 dev em dev f1 @xmath0.9 91.1 @xmath1.5 90.6 @xmath2.5 91.0 @xmath3.1 91.5 @xmath4.1 91.4 @xmath5.4 91.4 a new state-of-the-art on the specific extractive question answering task by outperforming published language model systems including bert baselines, as well as the latest top system on the corresponding leaderboard. there are two main directions for future research: (1) validating the competitiveness of anna to a variety of nlp tasks; and (2) enhancing the robustness of anna in order to apply for real-world question answering tasks in business.