Break It Down: A Question Understanding Benchmark (2020)

Abstract: Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decomposition Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, showing that quality QDMRs can be annotated at scale, and release the Break dataset, containing over 83K pairs of questions and their QDMRs. We demonstrate the utility of QDMR by showing that (a) it can be used to improve open-domain question answering on the HotpotQA dataset, (b) it can be deterministically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use Break to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially outperforms several natural baselines.

## introduction recently, increasing work has been devoted to models that can reason and integrate information from multiple parts of an input. this includes reasoning over images @xcite @xcite , paragraphs @xcite , documents @xcite @xcite , tables @xcite , and more. question answering (qa) is commonly used to test the ability to reason, where a complex natural language question is posed, and is to be answered given a particular context @xcite . although questions often share structure across tasks and modalities, understanding the language of complex questions has thus far been addressed within each task in isolation. consider the questions in figure 1 , all of which express operations such as fact chaining and counting. additionally, humans can take a complex question and break it down into a sequence of simpler questions even when they are unaware of what or where the answer is. this ability, to compose and decompose questions, lies at the heart of human language @xcite and allows us to tackle previously unseen problems. thus, better question understanding models should improve performance and generalization in tasks that require multi-step reasoning or that do not have access to substantial amounts of data. in this work we propose question understanding as a standalone language understanding task. we introduce a formalism for representing the meaning of questions that relies on question decomposition, and is agnostic to the information source. our formalism, question decomposition meaning representation (qdmr), is inspired by database query languages (sql; sparql), and by semantic parsing @xcite @xcite , in which questions are given full meaning representations. we express complex questions via simple (''atomic'') questions that can be executed in sequence to answer the original question. each atomic question can be mapped into a small set of formal operations, where each operation either selects a set of entities, retrieves information about their attributes, or aggregates information over entities. while this has been formalized in knowledge-base (kb) query languages @xcite , the same intuition can be applied to other modalities, such as images and text. qdmr abstracts away the context needed to answer the question, allowing in principle to query multiple sources for the same question. in contrast to semantic parsing, qdmr operations are expressed through natural language, facilitating annotation at scale by non-experts. figure 1 presents examples of complex questions on three different modalities. the middle box lists the natural language decompositions provided for each question, and the bottom box displays their corresponding formal queries. qdmr serves as the formalism for creating break, a question decomposition dataset of 83,978 questions over ten datasets and three modalities. break is collected via crowdsourcing, with a user interface that allows us to train crowd-workers to produce quality decompositions ( §3). validating the quality of annotated structures reveals 97.4% to be correct ( §4). we demonstrate the utility of qdmr in two setups. first, we regard the task of open-domain qa over multi-hop questions from the hotpotqa dataset. combining qdmr structures in break with a reading comprehension (rc) model @xcite improves f 1 from 43.3 to 52.4 ( §5). second, we show that decompositions in break possess high annotation consistency, which indicates that annotators produce high-quality qdmrs ( §4.3). in §6 we discuss how these qdmrs can be used as a strong proxy for full logical forms in semantic parsing. we use break to train a neural qdmr parser that maps questions into qdmr representations, based on a sequence-to-sequence model with copying @xcite . manual analysis of generated structures reveals an accuracy of 54%, showing that automatic qdmr parsing is possible, though still far from human performance ( §7). to conclude, our contributions are: the break dataset, models, and entire codebase are publicly available at: https://github. com/tomerwolgithub/break . ## question decomposition formalism in this section we define the qdmr formalism for domain agnostic question decomposition. qdmr is primarily inspired by sql @xcite . however, while sql was designed for relational databases, qdmr also aims to capture the meaning of questions over unstructured sources such as text and images. thus, our formalism abstracts away from sql by assuming an underlying ''idealized'' kb, which contains all entities and relations expressed in the question. this abstraction enables qdmr to be unrestricted to a particular modality, with its operators to be executed also against text operator template / signature question decomposition select return [entities] @xmath0. return touchdowns 2. return the number of #1 filter return [ref] [condition] s o ,@xmath1. 1. return flights 2. return #1 from toronto 3. return #2 to san diego project return [relation] of [ref] w,s @xmath2. return the los angeles lakers 2. return the head coach of #1 aggregate return [aggregate] of [ref] w agg ,s @xmath3. return colorado 2. return border states of #1 3. return the number of #2 group return [aggregate] [ref1] for each [ref2] w agg ,s o ,s @xmath4. return clubs 2. return female students of #1 3. return the number of #2 for each #1 superlative return [ref1] where [ref2] is [highest / lowest] s e ,s n ,w @xmath5, which has been contained by the most number of papers? 1. return papers 2. return keywords of #1 3. return the number of #1 for each #2 4. return #2 where #3 is highest comparative return [ref1] where [ref2] [comparison] [number] s e ,s n ,w com ,@xmath6. return authors 2. return papers of #1 3. return the number of #2 for each of #1 4. return #1 where #3 is more than 500 union return [ref1] , [ref2] s o ,s @xmath7. return the president 2. return the vice-president 3. return #1, #2 intersection return [relation] in both [ref1] and [ref2] w,s e ,s @xmath8. 1. return representatives 2. return #1 in new york state 3. return #1 in pennsylvania state 4. return parties in both #2 and #3 discard return [ref1] besides [ref2] s o ,s @xmath9. 1. return professors 2. return #1 playing canoeing 3. return #1 besides #2 sort return [ref1] sorted by [ref2] s e ,s @xmath10...e k find all information about student addresses, and sort by monthly rental. 1. return students 2. return addresses of #1 3. return monthly rental of #2 4. return #2 sorted by #3 boolean return [if / is] [ref1] [condition] [ref2] s o ,w,s @xmath11... 3. return the nationality of #1 4. return the nationality of #2 5. return if #3 is the same as #4 arithmetic return the [arithmetic] of [ref1] and [ref2] w ari ,n,@xmath12... 3. return the number of #1 4. return the number of #2 5. return the difference of #3 and #4 and images, while allowing in principle to query multiple modalities for the same question. 1 ## data collection our annotation pipeline for generating break consisted of three phases. first, we collected complex questions from existing qa benchmarks. second, we crowdsourced the qdmr annotation of these questions. finally, we validated worker annotations in order to maintain their quality. dataset example original break academic (db) return me the total citations of all the papers in the vldb conference. 195 195 atis (db) what is the first flight from atlanta to baltimore that serves lunch? 5,283 4,906 geoquery (db) how high is the highest point in the largest state? 880 877 spider (db) how many transactions correspond to each invoice number? 10,181 7,982 clevr-humans (images) what is the number of cylinders divided by the number of cubes? 32,164 13,935 nlvr2 (images) if there are only two dogs pulling one of the sleds? 29,680 13,517 comqa (text) what was gandhi's occupation before becoming a freedom fighter? 11,214 5,520 cwq (text) robert e jordan is part of the organization started by whom? 34,689 2,988, 2,991 high drop (text) approximately how many years did the churches built in 1909 survive? 96,567 10,230, 10,262 high hotpotqa-hard (text) benjamin halfpenny was a footballer for a club that plays its home matches where? 23,066 10,575 high total: 83,978 question collection questions in break were randomly sampled from ten qa datasets over the following tasks ( all questions collected were composed by human annotators. 3 hotpotqa questions were all sampled from the hard split of the dataset. ## dataset analysis this section examines the properties of collected qdmrs in break and analyzes their quality. ## qdmr for open-domain qa a natural setup for qdmr is in answering complex questions that require multiple reasoning steps. we compare models that exploit question decompositions to baselines that do not. we use the open-domain qa (''full-wiki") setting of the hotpotqa dataset @xcite : given a question, the qa model retrieves the relevant wikipedia paragraphs and answers the question using these paragraphs. ## qdmr for semantic parsing as qdmr structures can be easily annotated at scale, a natural question is how far are they from fully executable queries (known to be expensive to annotate). as shown in §4.3, qdmrs can be mapped to pseudo-logical forms with high precision (93.1%) by extracting formal operators and arguments from their steps. the pseudo-logical form differs from an executable query in the lack of grounding of its arguments (entities and relations) in kb constants. this stems from the design of qdmr as a domain-agnostic meaning representation ( §2). qdmr abstracts away from a concrete kb schema by assuming an underlying ''idealized'' kb, which contains all of its arguments. thus, qdmr can be viewed as an intermediate representation between a natural language question and an executable query. such intermediate representations have already been discussed in prior work on semantic parsing. @xcite and @xcite used underspecified logical forms as an intermediate representation. @xcite proposed a two-stage approach, separating between learning an intermediate text-to-sql representation and the actual mapping to schema items. works in the database community have particularly targeted the mapping of intermediate query representations into db grounded queries, using schema mapping and join path inference @xcite @xcite . we argue that qdmr can be used as an easy-to-annotate representation in such semantic parsers, bridging between natural language and full logical forms. ## qdmr parsing we now present evaluation metrics and models for mapping questions into qdmr structures. task definition given a question x we wish to map it to its qdmr steps, @xmath13, ... , s n . one can frame this as a sequence-to-sequence problem where x is mapped to a string representing its decomposition. we add a special separating token sep , and define the target string to be s 1 1 , ... , s 1 m 1 , sep , s 2 1 , ... , s 2 m 2 , sep , ... , s n m n , where m 1 , ... , m n are the number of tokens in each decomposition step. ## conclusion in this paper, we presented a formalism for question understanding. we have shown it is possible to train crowd-workers to produce such representations with high quality at scale, and created break, a benchmark for question decomposition with over 83k decompositions of questions from 10 datasets and 3 modalities (db, images, text). we presented the utility of qdmr for both open-domain question answering and semantic parsing, and constructed a qdmr parser with reasonable performance. qdmr proposes a promising direction for modeling question understanding, which we believe will be useful for multiple tasks in which reasoning is probed through questions.