Benefits of Intermediate Annotations in Reading Comprehension (2020)

Abstract: Complex compositional reading comprehension datasets require performing latent sequential decisions that are learned via supervision from the final answer. A large combinatorial space of possible decision paths that result in the same answer, compounded by the lack of intermediate supervision to help choose the right path, makes the learning particularly hard for this task. In this work, we study the benefits of collecting intermediate reasoning supervision along with the answer during data collection. We find that these intermediate annotations can provide two-fold benefits. First, we observe that for any collection budget, spending a fraction of it on intermediate annotations results in improved model performance, for two complex compositional datasets: DROP and Quoref. Second, these annotations encourage the model to learn the correct latent reasoning steps, helping combat some of the biases introduced during the data collection process.

## introduction recently many reading comprehension datasets requiring complex and compositional reasoning over text have been introduced, including hotpotqa @xcite , drop @xcite , quoref @xcite , and ropes @xcite . however, models trained on these datasets @xcite only have the final answer as supervision, leaving the model guessing at the correct latent reasoning. figure 1 shows an example from drop, which requires first locating various operands (i.e. relevant spans) in the text and then performing filter and count operations over them to get the final answer "3". however, the correct answer can also be obtained by extracting the span "3" from the passage, or by adding or subtracting various numbers in the passage. the lack of intermediate hints makes learning challenging and can lead the model question: how many touchdown passes did cutler throw in the second half? answer: 3 .....in the third quarter, the vikes started to rally with running back adrian peterson's 1-yard touchdown run (with the extra point attempt blocked). the bears increased their lead over the vikings with cutler's 3-yard td pass to tight end desmond clark. the vikings then closed out the quarter with quarterback brett favre firing a 6-yard td pass to tight end visanthe shiancoe. an exciting .... with kicker ryan longwell's 41-yard field goal, along with adrian peterson's second 1-yard td run. the bears then responded with cutler firing a 20-yard td pass to wide receiver earl bennett. the vikings then completed the remarkable comeback with favre finding wide receiver sidney rice on a 6-yard td pass on 4th-and-goal with 15 seconds left in regulation. the bears then took a knee to force overtime.... the bears then won on jay cutler's game-winning 39-yard td pass to wide receiver devin aromashodu. with the loss, not only did the vikings fall to 11-4, they also surrendered homefield advantage to the saints. to rely on data biases, limiting its ability to perform complex reasoning. in this paper, we present three main contributions. first, we show that annotating relevant context spans, given a question, can provide an easy and low-cost way to learn better latent reasoning. to be precise, we show that under low budget constraints, collecting these annotations for up to 10% of the training data (2-5% of the total budget) can improve the performance by 4-5% in f1. we supervise the current state-of-the-art models for drop and quoref, by jointly predict the relevant spans and the final answer. even though these models were not designed with these annotations in mind, we show that they can still be successfully used to improve model performance. models that explicitly incorporate these annotations might see greater benefits. our results suggest that future dataset collection efforts should set aside a fraction of budget for intermediate annotations, particularly as the reasoning required becomes more complex. question: what record do the children that conroy teaches play back to him? answer: beethoven's fifth symphony conroy tries to teach them about the outside world but comes into conflict both with the principal and mr. skeffington, the superintendent. he teaches them how to brush their teeth, who babe ruth is, and has the children listen to music, including flight of the bumblebee and beethoven's fifth symphony. he explains that the when beethoven wrote the fifth symphony, he was writing about "what death would sound like". he is also astounded they've never even heard of halloween, and he decides to take them to beaufort on the mainland to go trick-or-treating, which the superintendent has forbidden. he also must overcome parental fears of "the river." as he leaves the island for the last time, the children come out to see him leave, all of them lined up on a rickety bridge. as he is about to leave by boat, one of the students then begins playing a record, which is the beginning movement of beethoven's fifth symphony. second, these annotations can help combat biases that are often introduced while collecting data @xcite . this can take the form of label bias-in drop, 18% of questions have answers 1, 2, or 3-or annotator bias, where a small group of crowd workers creates a large dataset with common patterns. by providing intermediate reasoning steps explicitly, the annotations we collect help the model overcome some of these biases in the training data. finally, the intermediate annotations collected in this work, including 8,500 annotations for drop and 2,000 annotations for quoref, will be useful for training further models on these tasks. we have made them available at https://github.com/ddua/ intermediate_annotations . ## intermediate annotations intermediate annotations describe the right set of context spans that should be aggregated to answer a question. we demonstrate their impact on two datasets: drop and quoref. drop often requires aggregating information from various events in the context (figure 1 ). it can be challenging to identify the right set of events directly from an answer when the same answer can be derived from many possible event combinations. we annotate the entire event span including all the attributes associated with the specific event. quoref requires understanding long chains of coreferential reasoning, as shown in figure 2 , which are often hard to disentangle, especially when the context refers to multiple entities. we specifically annotate the coreference chains which lead to the entity being queried. collection process: we used amazon mechanical turk to crowd-source the data collection. we randomly sample 8,500 and 2,000 qa pairs from the training set for drop and quoref respectively. we showed a qa pair and its context to the workers and asked them to highlight "essential spans" in the context. in case of drop, crowd workers were asked to highlight complete events with all their corresponding arguments in each span. for quoref, they were asked to highlight the coreference chains associated with the answer entity in the context. ## experiments and results in this section, we train multiple models for the drop and quoref datasets, and evaluate the benefits of intermediate annotations as compared to traditional qa pairs. in particular, we will focus on the cost vs benefit tradeoff of intermediate annotations, along with evaluating their ability to mitigate bias in the training data. ## related work similar to our work, @xcite studied the impact of providing explicit supervision via rationales, rather than generating them, for varying fractions of training set in text classification. however, we study the benefits of such supervision for complex compositional reading comprehension datasets. in the field of computer vision, @xcite collected similar annotations, for visual recognition, where crowd-workers highlighted relevant regions in images. within reading comprehension, various works like hotpotqa @xcite and coqa @xcite have collected similar reasoning steps for entire dataset. our work shows that collecting intermediate annotations for a fraction of dataset is cost-effective and helps alleviate dataset collection biases to a degree. another line of work @xcite explores the cost vs. benefit of collecting full vs. partial annotations for various structured predictions tasks. however, they do not focus on intermediate reasoning required to learn the task. our auxiliary training with intermediate annotations is inspired by extensive related work on training models using side information or domain knowledge beyond labels @xcite @xcite . especially relevant is work on supervising models using explanations @xcite , which, similar to our annotations, identify parts of the input that are important for prediction @xcite . ## conclusion we show that intermediate annotations are a costeffective way to not only boost model performance but also alleviate certain unanticipated biases introduced during the dataset collection. however, it may be unnecessary to collect these for entire dataset and there is a sweet-spot that works best depending on the task. we proposed a simple semi-supervision technique to expose the model to these annotations. we believe that in future they can be used more directly to yield better performance gains. we have also released these annotations for the research community at https: //github.com/ddua/intermediate_annotations. figure 8: hit interface used for collection annotations question: what is the full name of mary harriette's father? motteux was also without heirs and bequeathed sandringham, together with another norfolk estate and a property in surrey, to the third son of his close friend, emily lamb, the wife of lord palmerston. at the time of his inheritance in 1843, charles spencer cowper was a bachelor diplomat, resident in paris. on succeeding to motteux's estates, he sold the other properties and based himself at sandringham. he undertook extensions to the hall, employing samuel sanders teulon to add an elaborate porch and conservatory. cowper's style of living was extravagant he and his wife spent much of their time on the continent and within 10 years the estate was mortgaged for £89,000. the death of their only child, mary harriette, from cholera in 1854 led the couple to spend even more time abroad, mainly in paris, and by the early 1860s cowper was keen to sell the estate.