Ensemble ALBERT and R o BERT a for Span Prediction in Question Answering (2021)

Abstract: Retrieving relevant answers from heterogeneous data formats, for given for questions, is a challenging problem. The process of pinpointing relevant information suitable to answer a question is further compounded in large document collections containing documents of substantial length. This paper presents the models designed as part of our submission to the DialDoc21 Shared Task (Document-grounded Dialogue and Conversational Question Answering) for span prediction in question answering. The proposed models leverage the superior predictive power of pretrained transformer models like RoBERTa, ALBERT and ELECTRA, to identify the most relevant information in an associated passage for the next agent turn. To further enhance the performance, the models were fine-tuned on different span selection based question answering datasets like SQuAD2.0 and Natural Questions (NQ) corpus. We also explored ensemble techniques for combining multiple models to achieve enhanced performance for the task. Our team SB_NITK ranked 6th on the leaderboard for the Knowledge Identification task, and our best ensemble model achieved an Exact score of 58.58 and an F1 score of 73.39.

## introduction in recent years, deep learning based transformer models like bert have accelerated research in the natural language processing (nlp) domain, due to their outstanding performance in various nlp tasks like summarization, machine translation etc, against state-of-the-art models. questionanswering is one such text based information retrieval framework, focusing on generating relevant answers to natural language questions presented by humans. extractive question answering models leverage document context to make decisions while identifying the most relevant answer and its location in a given passage or document. the applications of question answering systems include chat bots in medical science, search engines, personal assistants etc. several researchers have addressed the problem of answer generation for a given question, especially focusing on the challenge of dealing with descriptive answers. some works deal with this challenge in a two-phased approach -first, classifying the question into opinion-based or yes/no questions and secondly, dealing with the issue of lengthy questions and generating relevant answers for them using deep neural models like lstms for the question answering task @xcite . @xcite proposed a question answering model built on bilstms pre-trained on the squad dataset, to obtain appropriate ranks for answers corresponding to a given question at hand. additionally, ensemble techniques have proven well-suited due to better prediction performance, while reducing the variance and bias. adopting ensemble techniques to combine multiple models can provide better predictions and boost the performance of question answering systems. as shown in fig. 1 , pretrained encoders such as bert @xcite with an additional linear layer on top to predict spans have been shown to provide the advantage of transfer learning as they are pretrained on large, open datasets. the dialdoc21 shared task aims to encourage the development of models that can detect the most relevant details in the grounding document and predict agent responses close to common human responses. dialdoc21 is composed of two different shared tasks - most recent extractive question answering systems are predominantly bert based models. transformers like bert, roberta @xcite and xlnet @xcite are experimented for extractive question answering task on datasets from multiple domains and languages like stanford squad v1.1 @xcite and v2.0 @xcite , natural questions @xcite , newsqa @xcite and hotpotqa @xcite . @xcite experimented on the scenario of multiple answer spans. in this paper, we describe various models and experiments that were developed and tested for the knowledge identification subtask. the models are built on fine-tuned, pretrained transformers like roberta, albert @xcite and electra @xcite . we adopt ensembling techniques on multiple models to boost the prediction performance further. as part of our approach, we pretrained the transformer models considered on various question-answering datasets like squad2.0, natural questions corpus, and cord-19 dataset (wang et al., 2020) prior to finetuning them for our dataset, and observe their performance. the rest of this article is organized as follows. in section 2, we provide information about the data used such as description of dataset and dataset pre-processing. section 3 gives an overview of models used and their different versions. in section 4, we describe the ensemble technique used and the various ensemble models submitted. section 5 describes the system flow and experimental setup. in section 6, we list the results and compare the performance of our proposed models in detail, followed by conclusion and directions for future work. ## models for the dialdoc21 knowledge identification shared task, we experimented with various versions of three different transformer models by fine-tuning them on the doc2dial dataset. the details of these implementations are discussed in detail in subsequent sections. ## ensemble models to further enhance the performance of the proposed models for the knowledge identification task, we employed ensembling techniques for leverage the predictive power of all the transformer models considered for the experiments. various combinations of the models were designed and experimented with, in order to improve the predictions. the confidence score from different models is used as a measure to combine models. the predictions of the model with maximum confidence score is treated as best prediction and the same is considered for evaluation. initially, various models from same groups of roberta, albert and electra are ensembled together, and based on the predictions on the validation set, few other models are added. the following are the different combinations of models submitted for testset evaluation. 1. roberta-ensemble : roberta-large-squadv2 + roberta-base-squadv2-nq + roberta-base-squadv2-covid 2. albert-ensemble : albert-base-squadv2 + albert-xlarge-squadv2 + albert-xxlarge-squadv2 3. electra-ensemble : electra-base-squadv2 + electra-large-squadv2 4. ensemble1 : alberta-ensemble + roberta-base-squadv2-covid 5. ensemble2 : alberta-ensemble + roberta-base-squadv2-nq + roberta-base-squadv2-covid the performance of all the proposed models was measured using standard metrics, for both the validation and test set, the details of which are presented in section 6. we also employed metrics like exact match and f1 score for individual pretrained models, and also alberta-ensemble with robertabase pretrained on the nq and covid datasets. ## model fine-tuning in order to improve exact match and f1 scores, fine-tuning and ensemble techniques were considered. the dataset provided for the test-dev phase of the shared task is considered as validation set, which shares the same grounding document as training dataset. the test dataset is provided during the test phase of the task and contains 787 end-user questions. during the training phase, the document trunk along with the corresponding dialogue context was input to the encoder model. the output obtained from the linear layer is a tuple representing the probabilities of the position being the start and end of the corresponding span. in case the ground truth span is not inside the considered trunk, the begin and end positions are taken as the start of the sequence. in the decoding phase, the probability tuples of all the trunks are considered by the model to obtain the optimum span. we also conducted experiments by varying hyperparameters such as maximum sequence length (384, 512), batch size (4, 8, 16) and learning rate(3e-5, 1e-4) to select best performance. each model has been trained for 5 epochs and during training, checkpoints were generated for every 2000 steps. loss reduction was examined at every checkpoint to pick the best optimal checkpoint that could potentially generate optimal predictions for each model. all experiments were performed on nvidia v100 gpus with 32gb ram. in section 6, the details of experiments conducted and the observed performance are described. ## experimental results and discussion in this section, we discuss various versions of models submitted for consideration in the final phase on the leaderboard. in case of ensemble1 and ensemble2, applying ensembling techniques on best performing roberta models like roberta-base-squadv2nq and roberta-base-squadv2-covid with albertensemble resulted in further improvements as is model predicted text span roberta-base-squadv2-nq keep away from others who are sick, limit close contact, and wash your hands often. consider steps you can take to stay away from other people. this is especially important for people who are at higher risk of getting very sick. roberta-ensemble keep away from others who are sick, limit close contact, and wash your hands often. ## conclusion and future work in this paper, the application of transfer learning by utilizing transformer models like roberta, al-bert and electra for question answering span prediction task was explored. we also experimented with pretrained models on several other datasets prior to fine-tuning it on the dialdoc21 dataset, provided as part of the dialdoc21 shared task. maximum confidence score based ensemble techniques were employed to combine various base transformer models to further boost the per-formance. we plan to extend our approach and experiment with other ensembling techniques for further enhancing the performance and also explore avenues for improved scalability when applied to larger datasets.