Learning with Instance Bundles for Reading Comprehension (2021)

Abstract: When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other. However, closely related questions and their corresponding answers are not independent, and leveraging these relationships could provide a strong supervision signal to a model. Drawing on ideas from contrastive estimation, we introduce several new supervision losses that compare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers, adding a cross entropy loss term in addition to traditional maximum likelihood estimation. Our techniques require bundles of related question-answer pairs, which we either mine from within existing data or create using automated heuristics. We empirically demonstrate the effectiveness of training with instance bundles on two datasets—HotpotQA and ROPES—showing up to 9% absolute gains in accuracy.

## introduction machine learning models are typically trained with the assumption that the training instances sampled from some data distribution are independent and identically distributed. however, this assumption can cause the learner to ignore distinguishing cues @xcite between related or minimally different questions associated with a given context, resulting in inconsistent model predictions @xcite @xcite . in cases like ropes, where the dataset contains only minimally different questions, we see that the performance of a competitive model (roberta) is close to random @xcite . one potential reason for this poor performance is that the model considers each question independently, instead of looking at differences between related questions. context: marsilea is a genus of approximately 65 species of aquatic ferns of the family marsileaceae. the name honours italian naturalist luigi ferdinando marsili (1656-1730) .... brabejum is a genus of a single species of large evergreen tree, brabejum stellatifolium in the proteaceae, commonly called wild almond, bitter almond or ghoeboontjie. question, q: is the marsilea or the brabejum the genus of more individual species of plants? answer, a: marsilea question, q: is the marsilea or the brabejum the genus of less individual species of plants? answer, ã: brabejum (a) instance bundle created from hotpotqa. the qa pair, ( q, ã) is heuristically generated from (q, a) in the dataset. normalized over the two questions {q, q} in each instance bundle, in hotpotqa's comparison type dev set. the higher value indicates that correct qa pair has a higher likelihood than the wrong pair (with 0.5 indicating model has the same likelihood for the answer for both the contrastive questions). this problem can be addressed by training models with sets of related question-answer (qa) pairs simultaneously, instead of having a loss function that decomposes over independent examples. we use the term instance bundle to refer to these sets of closely contrasting examples. consider an instance bundle from hotpotqa in figure 1a , containing two contrastive qa pairs, which differ in their input by only one word (changing more to less), resulting in different answers. with both examples in the training set, traditional maximum likelihood estimation will incentivize the model to figure out the difference between the inputs that leads to the difference in the answers, but the instances are likely to be seen far apart from each other during training, giving only a weak and indirect signal about the relationship between the pair. to learn from these instance bundles more effectively, we draw on contrastive estimation @xcite , a method for re-normalizing an unsupervised probabilistic model using a neighborhood of related examples (originally a set of perturbations of some observed text). we extend this technique to apply to supervised reading comprehension problems by carefully selecting appropriate "neighborhoods" from instance bundles. the simplest choice of neighborhood is the set of contrasting answers from the instance bundle, resulting in a method similar to unlikelihood training @xcite or noise-contrastive estimation (gutmann and hyvärinen, 2010). however, there are other choices, including the set of contrasting questions, or combinations of questions and answers. these re-normalized loss functions are not effective on their own, which is likely why they have not been used before for training reading comprehension models, but provide substantial gains when combined with maximum likelihood training. an intuitive explanation for the improved performance is illustrated in figure 1b . when trained on non-contrasting data with maximum likelihood estimation, the model gives roughly equal values for both p(a|q) and p(a| q), even though q and q are opposites. augmenting the contrasting data helps the model differentiate these two probabilities, but not as much as unlikelihood training, which itself is not as effective as contrastive estimation. we empirically demonstrate the utility of this approach on two reading comprehension datasets: hotpotqa @xcite and ropes @xcite . to define the instance bundles on these datasets, we introduce various heuristics for obtaining closely related instances. we show that using contrastive estimation on the instance bundles gives up to a 9% absolute performance improvement over prior training techniques. these results strongly suggest that data should be collected in instance bundles wherever possible, to allow for stronger supervision signals during training. ## contrastive estimation for reading comprehension reading comprehension is the task of producing an answer a given a question q about a context c. the question is tied to a particular passage, so in the discussion that follows we will typically use q as a shorthand to refer to both q and c together. reading comprehension models are typically trained to maximize the likelihood of the answer to each training question. given a model's exponentiated scoring function ψ(q, a) for a qa pair, foot_0 this mle objective normalizes the scores over all possible answer candidates a for a given question: instead of normalizing scores over all possible answer candidates, contrastive estimation (ce, @xcite normalizes scores over some neighborhood of closely related instances. this method was originally introduced for unsupervised linguistic structure prediction, with a neighborhood obtained by permuting observed text to get inputs that had similar content but were ungrammatical. our contribution is to apply this general idea to supervised reading comprehension problems. in our setting, given a neighborhood n(q, a) of related qa pairs, ce can be described as smith and eisner (2005) replace the mle objective with ce, which worked well in their unsupervised learning problem. in supervised learning, mle is a much stronger training signal, and ce on its own severely underperforms mle. this is because ce provides no learning signal for the very large space of alternative answers to a question that are not in the neighborhood. however, ce can provide a much stronger signal than mle for a small set of potentially confusing alternatives, as there are fewer ways for the model to increase the probability of the correct answer. to adapt ce to supervised settings we interpolate between the two losses, instead of replacing mle with ce: interestingly, this can be viewed as forcing the scoring function ψ to capture different probabilistic interpretations, as both losses perform softmaxes over different sets of alternatives. additionally, if ψ has some locally-normalized component, as is true for the generative models we work with and for many other common models (such as bio tagging, or independent span start and span end positions), this interpolation trades off between the locally-focused mle and the more global view of the problem that the normalization in ce provides. the key question in applying ce to reading comprehension is how to choose a neighborhood n for a given training example. we do so by making bundles of related instances, then extracting various combinations of questions and answers from a bundle to use as neighborhood. formally, an instance bundle b is a collection of unique questions b q and unique answers b a , such that there is at least one qa pair where a is the correct answer to q: ans(q) = a. we refer to such pairs as (q g , a g ) in the discussion that follows. our assumption is that the questions in b q and the answers in b a are related to each other in some way-often they differ in only a single word-though we do not characterize this formally. however, a good bundle creation procedure is crucial for effective model learning. we discuss several ways for creating bundles in section 3, and discuss the limitations of ce when effective bundles cannot be created in section 5.2. the following section discusses choices of neighborhood functions given an instance bundle. ## bundling heuristics in this section we discuss how we obtain instance bundles for use with contrastive estimation and other related baselines. a naive way to create a bundle would be to exploit the fact that all the questions associated with a context are likely to be related, and simply make bundles consisting of all qa pairs associated with the context. however, this approach poses two problems. first, there could be many questions associated with any particular context, and smaller, more closely-related bundles are more informative. second, and relatedly, it is likely that bundles obtained this way will have many questions whose answers can be obtained from the bundle by superficial type matching. for instance, a wh-question starting with "where" would most likely align with a location type answer. if this were bundled with a question starting with "how many", with an answer that is a number, the bundle would be largely uninformative. we instead attempt to create bundles with minimally different questions and answers, in several different ways. diverse top-k sampling we first discuss a method for getting alternative answers to a single question. this will result in a bundle that can only be used with answer conditional ce, as there are no alternative questions in the bundle. an easy way to get answer candidates is to employ a pre-trained answering model and sample answers from the posterior distribution. however, since the model has seen all the qa pairs while training, it can easily memorize answers, resulting in a low variance, high confidence distribution. in order to achieve diverse answer samples we need to either overgenerate and prune out the gold answer from the samples or induce a diversity promoting sampling. we adopt a hybrid sampling strategy where we use nucleus sampling for the first few (depending on dataset) timesteps of decoding (without replacement) and then top-k for the remaining timesteps. this forces the answer generator to consider different starting positions in the passage and then generate the best answer span (of an appropriate length) from the token produced at the first step. question mining some datasets, such as ropes, are constructed with very close question pairs already in the data. when these exist, we can create instance bundles by finding natural pairings from the training set. to find these pairings, we cluster the questions with a high lexical bagof-words overlap based on jaccard index (≥ 0.8), ensuring that each question in the cluster has a unique answer. in ropes, these bundles typically result in bundles of two qa pairs that differ in one or a few words. in hotpotqa, the other dataset we focus on in this work, there are very few such pairings naturally occurring in the dataset, so we resort to heuristics to create them. question generation hotpotqa has many questions that are phrased as multiple-choice, with answer options given in the question itself. these multiple choice questions can most often be rephrased to provide qa pairs that can be bundled with the original question. for instance, given the question, "which animal is faster, turtle or hare?", it is straightforward to obtain a minimally different question with the opposite answer: "which animal is slower, turtle or hare?". we adopt three main heuristics to generate such questions whenever possible, applicable to any dataset that has questions of this kind. all of these heuristics require identifying the two plausible answer choices from the question, which can be done with reasonably high precision using simple regular expressions. 1. we replace superlatives with their contrasting counterparts, e.g., taller/smaller, more/less, etc. 2. we negate the main verb, e.g., @xmath0, by inflecting the verbs. foot_1 3. we swap the noun phrases being compared in the question, e.g., "are rock a's wavelengths shorter or longer than rock b's?" can be used to generate, "are rock b's wavelengths shorter or longer than rock a's?" ## experiments we use an encoder-decoder style t5-large model for all our experiments. the baseline models in our experiments are fine-tuned t5 models for the corresponding tasks using the mle objective. we compare them against models that are further fine-tuned with a combination of mle and contrastive estimation objectives, as described in section 2.1. that is, when using various instance bundle techniques, we initialize the model with the weights from the fine-tuned mle model, then continue training with the new loss function. 3 the model takes a concatenated context and question as an input to produce an answer output. we use a learning rate of 2e-5 for ropes and 5e-5 for comparison with lowercased inputs and outputs for further fine-tuning. we truncate the concatenated context and question up to a length of 650 for ropes and 850 for com-parison. all the interpolation hyper-parameters (α l , λ l ) are set to 1. in addition to standard metrics on these datasets, we additionally evaluate using consistency. this metric evaluates to true only if all the questions in a bundle are answered correctly, and is thus a stricter version of em. ## discussion here we try to understand how ce compares to mle and ul and when is it effective. ## related work learning with negative samples has been explored in many natural language tasks, such as dialogue generation @xcite , word embeddings @xcite , language modeling @xcite , etc., and computer vision tasks such as image captioning @xcite , unsupervised representation learning @xcite , etc. similarly, mutual information minimization based learners in question answering @xcite ) and image classification @xcite try to decrease the mutual information between positive and negative samples. natural language applications often sample negative examples either randomly from the data or based on likelihood (or unlikelihood) metrics from a reference model. however, the negative samples extracted in this manner are often unrelated. a growing body of literature is exploring ways to obtain closely-related examples, either manually @xcite or automatically @xcite @xcite . this is complementary to our work, as we show how to make better use of these related examples during training. there is also work on consistent cluster assignments in coreference resolution @xcite ; factually consistent summaries @xcite and language models @xcite . another growing body of literature on training with closely related examples, to which we are contributing, includes methods that make use of logical or domain specific consistency rules, in natural language inference tasks @xcite , reading comprehension @xcite , and visual question answering @xcite @xcite . in open domain qa, re-ranking extracted answer spans from a baseline model has shown promising improvements and shares connections with our answer conditional setup @xcite . instead of training just a ranking model (which is similar to answer conditional ce) on top of a baseline (mle) model, we jointly train a single qa model with both objectives. this promotes better representation learning in the baseline qa model. ## conclusion we have presented a way to use contrastive estimation in a supervised manner to learn from distinguishing cues between multiple related qa pairs, or instance bundles. our experiments with multiple ce-based loss functions, defined over a joint neighborhood of questions and answers, have shown that these models outperform existing methods on two datasets: ropes and hotpotqa. apart from presenting several ways to create instance bundles, we also explore theoretical connections between unlikelihood training and contrastive estimation, and initial exploration into when instance bundles are likely to be effective with these methods. we believe our results give strong motivation for further work in techniques to both create and use instance bundles in nlp datasets. the code is available at https://github.com/ddua/ contrastive-estimation .