A Simple Yet Strong Pipeline for H otpot QA (2020)

Abstract: State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition. However, does their strong performance on popular multi-hop datasets really justify this added design complexity? Our results suggest that the answer may be no, because even our simple pipeline based on BERT, named , performs surprisingly well. Specifically, on HotpotQA, Quark outperforms these models on both question answering and support identification (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially relevant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of Quark resurfaces the importance of carefully exploring simple model designs before using popular benchmarks to justify the value of complex techniques.

## introduction textual multi-hop question answering (qa) is the task of answering questions by combining information from multiple sentences or documents. this is a challenging reasoning task that requires qa systems to identify relevant pieces of information in the given text and learn to compose them to answer a question. to enable progress in this area, many datasets @xcite @xcite and models @xcite @xcite with varying complexities have been proposed over the past few years. our work focuses on hotpotqa @xcite , which contains 105,257 multi-hop questions derived from two wikipedia paragraphs, where the correct answer is a span in these paragraphs or yes/no. due to the multi-hop nature of this dataset, it is natural to assume that the relevance of a sentence for a question would depend on the other sentences considered to be relevant. e.g., the relevance of "obama was born in hawaii." to the question "where was the 44 th president of usa born?" depends on the other relevant sentence: "obama was the 44 th president of us." as a result, many approaches designed for this task focus on jointly identifying the relevant sentences (or paragraphs) via mechanisms such as cross-document attention, graph networks, and entity linking. our results question this basic assumption. we show that a simple model, quark (see fig. 1 ), that first identifies relevant sentences from each paragraph independent of other paragraphs, is surprisingly powerful on this task: by using only the context of the corresponding paragraph, quark can recover all gold supporting sentences within the top-5 sentences. for qa, it uses a standard bert @xcite span prediction model (similar to current published models) on the output of this module. additionally, quark exploits the inherent similarity between the relevant sentence identification task and the task of generating an explanation given the answer from a qa module: it uses the same architecture for both tasks. we show that this independent sentence scoring model results in a simple qa pipeline that outperforms other bert models in both 'distractor' and 'fullwiki' settings of hotpotqa. in the distractor setting (10 paragraphs, including two gold, provided as context), quark achieves joint scores (answer and support prediction) within 0.75% of the current state of the art. even in the fullwiki setting (all 5m wikipedia paragraphs as context), by combining our sentence selection approach with a commonly used paragraph selection approach @xcite ), foot_0 we outperform all previously published bert models. in both settings, the only models scoring higher use roberta @xcite , a more robustly trained language model that is known to outperform bert across various tasks. while our design uses multiple transformer models (now considered a standard starting point in nlp), our contribution is a simple pipeline without any bells and whistles, such as ner, graph networks, entity linking, etc. the closest effort to quark is by @xcite , who also propose a simple qa model for hotpotqa, but don't consider the support task and fall several points short of sota on the qa task due to reasoning over only one paragraph at a time. finally, our ablation study demonstrates that the sentence selection module benefits substantially from using context from the corresponding paragraph. it also shows that running this module a second time, with the chosen answer as input, results in more accurate support identification. ## related work most approaches for hotpotqa attempt to capture the interactions between the paragraphs by either relying on cross-attention between documents or sequentially selecting paragraphs based on the previously selected paragraphs. while @xcite also use a standard reading comprehension (rc) model, they combine it with a special query focused extractor that identifies relevant sentences by updating a rnn state representation in each step, allowing the model to capture dependencies between sentences across time-steps. @xcite propose a dynamically fused graph networks (dfgn) model that first creates an entity graph from paragraphs, dynamically extracts sub-graphs, and fuses them with paragraph representations. the select, answer, explain (sae) model ( tu et al., 2020) also first selects relevant documents and uses them to produce answers and explanations. however, it relies on a self-attention over all document representations to capture potential interactions. additionally, it relies on a graph neural network (gnn) to answer the questions. hierarchical graph network (hgn) model @xcite builds a hierarchical graph with three levels: entities, sentences and paragraphs to allow for joint reasoning. de-comprc @xcite takes a completely different approach of learning to decompose the question (using additional annotations) and then answer the decomposed questions using a standard single-hop rc system. others such as @xcite have noticed that many hotpotqa questions can be answered just based on a single paragraph. however, they did not consider the support identification task (which we show can also be done independently). while they achieve strong (but not quite sota) qa performance by only reasoning over a single paragraph, we show that interaction is actually valuable for qa! specifically, by using relevant sentences spread across multiple paragraphs, our simple model outperforms previous models with more complex interactions. we thus view quark as a different, stronger baseline for multi-hop qa. in the fullwiki setting, each question has no associated context and models are expected to select paragraphs from wikipedia. to be able to scale to such a large corpus, the proposed systems often select the paragraphs independent of each other. a recent retrieval method in this setting is semantic retrieval @xcite where first the paragraphs are selected based on the question, followed by individual sentences from these para-graphs. however, unlike our approach, they do not use the paragraph context to select the sentences, missing key context needed to identify relevance. ## experiments we evaluate on both the distractor and fullwiki settings of hotpotqa with the following goal: can a simple pipeline model outperform more complex approaches? we present the em (exact match) and f1 scores on (1) answer selection, (2) support selection, and (3) joint score. we attribute this improvement to two factors: our sentence selection capitalizing on the sentence's paragraph context leading to better support selection, and a better span selection model leading to improved qa. ## conclusion our work shows that on the hotpotqa tasks, a simple pipeline model can do as well as or better than more complex solutions, such as graph networks, cross-document attention, or ner. powerful pre-trained models allow us to score sentences one at a time, without looking at other paragraphs. by operating jointly over these sentences chosen from multiple paragraphs, we arrive at answers and supporting sentences on par with state-of-theart approaches. this result shows that supporting sentence identification in hotpotqa is itself not a multi-hop problem, and suggests focusing on other multi-hop datasets to demonstrate the value of more complex retrieval techniques.