Rethinking the Objectives of Extractive Question Answering (2021)

Abstract: This work demonstrates that using the objective with independence assumption for modelling the span probability P (a_s , a_e ) = P (a_s )P (a_e) of span starting at position a_s and ending at position a_e has adverse effects. Therefore we propose multiple approaches to modelling joint probability P (a_s , a_e) directly. Among those, we propose a compound objective, composed from the joint probability while still keeping the objective with independence assumption as an auxiliary objective. We find that the compound objective is consistently superior or equal to other assumptions in exact match. Additionally, we identified common errors caused by the assumption of independence and manually checked the counterpart predictions, demonstrating the impact of the compound objective on the real examples. Our findings are supported via experiments with three extractive QA models (BIDAF, BERT, ALBERT) over six datasets and our code, individual results and manual analysis are available online.

## introduction the goal of extractive question answering (eqa) is to find the span boundaries -the start and the end of the span from text evidence, which answers a given question. therefore, a natural choice of the objective to this problem is to model the probabilities of the span boundaries. in the last years, there was a lot of effort put into building better neural models underlying the desired probability distributions. however, there has been a little progress seen towards the change of the objective itself. for instance, the "default" choice of objective for modelling the probability over spans in squadv1.1 @xcite -maximization of independent span boundary probabilities p (a s )p (a e ) for answer at position a s ,a e -has stayed the same over the course of years in many influential works @xcite @xcite @xcite since the earliest work on this dataset -the submission of @xcite . based on the myths of worse performance of different objectives, these works adopt the deeply rooted assumption of independence. however, this assumption may lead to obviously wrong predictions, as shown in figure 1 . figure 1 : an example of an error which comes with an independence assumption. the model assigns high probability mass to boundaries around "little boy", and "fat man" answers. however, during decoding, the start of one and the end of another answer is picked up. in addition, this assumption leads to degenerate distribution p (a s , a e ), as high probability mass is assigned to many trivially wrong foot_1 answers. some of the earlier work @xcite and recent approaches including large language representation models (lrms) like xlnet @xcite , albert @xcite or electra @xcite started modelling the span probability via conditional probability factorization p (a e |a s )p (a s ). however, is it unknown whether this objective improves any performance at all, as almost none of the recent works reported results on its effect, not even described its existence (except electra paper). additionally, this objective requires beam search which slows down inference in test time. exceptionally, @xcite proposed one way for modelling p (a s , a e ) directly, but the approach was only sparsely adopted @xcite . this may be caused by the belief, that enumerating all possible spans has a large complexity @xcite . however, in practice we find the complexity to be often similar to assumption of independence, when implementing the objective efficiently. we continue the in-depth discussion on complexity in appendix c. in this work, we try to break the myths about the objectives that have been widely used previously. we experiment with joint objective and we also introduce a new compound objective, that deals with modelling joint probability p (a s , a e ) directly while keeping the traditional independent objective as an auxiliary objective. we experiment with 5 different joint probability function realisations and find that with current lrms, simple dot product works the best. however, we show that this is not a rule, and for some models, other function realisations might be better. the conducted experiments demonstrate that using compound objective is superior to previously used objectives across the various choices of models or datasets. in summary, our work contributions are: ## probabilistic assumptions for the answer span this section describes the common approach to the eqa, with its independent modelling of the answer span start and end positions. secondly, it defines an assumption based on conditional factorization of span probability. finally a function family for computing joint span probability and a combination of independent and joint assumption we call the compound objective are proposed. the eqa can be defined as follows: given a question q and a passage or a set of passages d, find a string a from d such that a answers the question q. this can be expressed by modelling a categorical probability mass function (pmf) that has its maximum in the answer start and end indices @xmath0, a e from the passage d as p (a s , a e |q, d) for each question-passage-answer triplet (q, d, a) from dataset d. the parameters θ of such model can be estimated by minimizing maximum likelihood objective during inference, the most probable answer span a s , a e is predicted. although there are works that were able to model the joint probability explicitly @xcite , modelling it directly results in a number of categories quadratic to the passage's length. optimizing such models may be seen challenging, as there are often more classes than the amount of data points within the current datasets. therefore, state-of-the-art approaches resort to independence assumption p (a s , a e |q, d) = p θ (a s |q, d)p θ (a e |q, d). the factorized pmfs are usually computed by the model with shared parameters θ, as introduced in @xcite . for most of the systems modelling the independent objective with neural networks, the final endpoint probabilities foot_2 are derived from start/end position passage representations computed via shared model h s ,h @xmath1, e}. the passage representations h s ,h e are often presoftmax layer representations from neural network with passage and question at the input. symbols d and l denote the model-specific dimension and the passage length, respectively. occasionally, the conditional factorization p (a s , a e |q, d) = p θ (a s )p θ (a e |a s ) is considered instead. the probabilities of span's start and end are computed the same way as in equation 2. the difference is in the end representations h @xmath2, which now must be the a function of span's start a s . ## experimental setup we use transformers @xcite for language representation model (lrm) implementation. our experiments were done on 16gb gpus using pytorch @xcite . for experiments with lrms, we used adam optimizer with a decoupled weight decay @xcite . the used hyperparameters were the same as the squadv1.1 default hyperparameters as proposed by specific lrm authors through all our datasets. for bidaf, we tuned hyperparameters using hyperopt @xcite separately for independent and compound objectives foot_4 . see appendix d for further details. in all our experiments, we apply length filtering (lf). therefore, probabilities p (a @xmath3, a @xmath4, where ζ is a length threshold. following @xcite , we set ζ = 30 in all of our experiments. ## results and discussion we now show the effectiveness of proposed approaches. each of the presented results is averaged from 10 training runs. ## analysis apart from example in figure 1 , we provide more examples of different predictions foot_6 between models trained with independent and compound objective in to quantify an occurrence of these errors, we study our best bert and albert checkpoint predictions for squadv1.1 validation data. for bert, we found the most frequently occurring is the error type (1), for which we manually annotated 200 random differences between independent and compound model predictions. we found 5% of them to be the case of this error of the independent, and no case of this error for the compound model. interestingly, 4 out of 10 of these cases were questions clearly asking about single entity, while independent model answered multiple entities, e.g., q:which male child of ghengis khan and börte was born last? a:chagatai (1187-1241), ögedei (1189-1241), and tolui. for the error type (2), we filtered all prediction differences (more than 1300 for bert and albert) down to cases, where either independent or compound prediction contained non-alphanumeric paired punctuation marks, which resulted in less than 30 cases for each. for bert, 37% independent predictions from these cases contained an error type (2), while again no paired punctuation marks errors were observed for compound objective. for the error type (3), we filter prediction differences down to cases, where independent or compound prediction contained the same prefix and suffix of length at least 2 (only 9 and 5 cases for bert and albert). from these, error type (3) occurred in 3 cases for bert and in 1 case for albert in case of independent and again we found no case for the compound for both models. note the error type (3) can be fully alleviated by marginalizing over probabilities of top-k answer spans during the inference, as in @xcite (see appendix e for details). interestingly, for albert, we found only negligible amount of errors of type (1) and (2) for both objectives foot_7 . during manual analysis, we observed that, an uncertain models with an independent objective are prone to pick large answer spans. to illustrate, that spans retrieved with approaches modelling joint probability differ, we took the top 20 most probable spans from each model and averaged their length. this was done for each example in the squadv1.1 test data. the histogram of these averages is shown in figure 2 . for a fair comparison, these predictions were filtered via length filtering. ## conclusion the paper closely studies the objectives used within the extractive question answering (eqa). it identifies commonly used independent probability model as a source of trivially wrong answers. as a remedy it experiments with various ways of learning the joint span probability. finally it shows how the compound objective -the combination of independent and joint probability in objective -improves statistical eqa systems across 6 datasets without using any additional data. using the proposed approach, we were able to reach significant improvements through the wide spectrum of datasets, including +1.28 em on adversarial squad and +2.07 em on naturalquestions for bert-base. we performed a thorough manual analysis to understand what happened to trivially wrong answers, and we found most of the cases disappear. we also found that independent models tend to "overfit" to f1 metric by encompassing multiple possible answer spans, which would explain the effect of joint objectives improving the em far more significantly than the f1. we shown the samples from joint model contain the greatest start/end token diversity. we further hypothesize that having diverse answers may be especially beneficial towards answer reranking step commonly used in qa @xcite . in addition, we also identified the reason for performance decrease with compound objective on squadv2.0 -no-answer classifier trained within the same model performs