Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021)

Abstract: We propose a general framework called Text Modular Networks(TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models. To ensure solvability of simpler tasks, TMNs learn the textual input-output behavior (i.e., language ) of existing models through their datasets. This differs from prior decomposition-based approaches which, besides being designed specifically for each complex task, produce decompositions independent of existing sub-models. Specifically, we focus on Question Answering (QA) and show how to train a next-question generator to sequentially produce sub-questions targeting appropriate sub-models, without additional human annotation. These sub-questions and answers provide a faithful natural language explanation of the model’s reasoning. We use this framework to build ModularQA, a system that can answer multi-hop reasoning questions by decomposing them into sub-questions answerable by a neural factoid single-span QA model and a symbolic calculator. Our experiments show that ModularQA is more versatile than existing explainable systems for DROP and HotpotQA datasets, is more robust than state-of-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work.

## introduction an intuitive way to solve more complex tasks, such as multi-hop question-answering @xcite @xcite and numerical reasoning @xcite , would be to decompose them into already solved simpler problems, e.g., single-fact qa @xcite . besides allowing reuse of existing simpler models, this approach would yield an interpretable system that provides a faithful explanation @xcite hotpotqa question: little big girl was a simpsons episode directed by the animator and artist of what nationality? (answer: american) figure 1 : modularqa learns to ask sub-questions to existing simple qa models, including a symbolic calculator, to answer a given complex question. notably, the approach does not rely on annotated decompositions. despite this, the system learned to add "start to take a dip" in the drop dataset question. goldberg, 2020) of its reasoning as a composition of simpler sub-tasks, as shown in fig. 1 . motivated by this, we ask the following question: given a set of existing qa models, can one leverage them to answer complex questions by communicating with these existing models? we propose a general framework, text modular networks (tmns), that answers this question by learning to decompose complex questions (of any form) into sub-questions that are answerable by existing qa models-symbolic or neural (henceforth referred to as sub-models ). 2 unlike previous approaches @xcite , the decompositions are not based on splits of the complex questions and aren't built independent of the sub-model. instead, our framework learns to generate sub-questions in the scope of existing models. for instance, the second subquestion in the drop dataset example in fig. 1 requires the introduction of a new phrase, "start to take a dip", which is beyond the scope of standard decomposition approaches. additionally, the final sub-question targets a symbolic calculator, which operates over a different input language. the core of our tmn framework is a next-question generator that sequentially produces the next sub-question to ask as well as an appropriate sub-model for answering it. the resulting sequence of sub-questions and their answers provides a human-interpretable description of the model's neuro-symbolic reasoning @xcite , as illustrated in fig. 1 . notably, tmns learn to produce these decompositions using only distant supervision, without the need for any explicit human annotation. one of our key insights is that the capabilities of existing sub-models can be captured by training a text-to-text system to generate the questions in the sub-model's training dataset (e.g., squad), given appropriate hints. in our case, we train a bart model @xcite to generate questions given the context, answer, and preferred vocabulary as hints. we then use these sub-task question models to generate sub-questions (and identify appropriate sub-models) that could lead to the likely intermediate answers extracted for each step of the complex question ("raymond s." and "american" in the hotpotqa example in fig. 1 ). the resulting sub-questions, by virtue of our training, are in the language (i.e., within-scope) of the corresponding sub-models. these sub-question sequences can now be used to train the next-question generator to sequentially produce the next sub-question. we use this trained generator, along with existing qa models, to answer complex questions, without the need for any intermediate answers. we use the tmn framework to develop mod-ularqa, a modular system that explains its reasoning in natural language, by decomposing complex questions into those answerable by two sub-models: a neural factoid single-span qa model and a symbolic calculator. modularqa's implementation 1 covers multi-hop questions that can be answered using these two sub-models via five classes of reasoning found in existing qa datasets: composition, conjunction, comparison, difference, and complementation. we evaluate modularqa on questions from two datasets, drop @xcite and hot-potqa @xcite , resulting in the first cross-dataset decomposition-based interpretable qa system. despite its interpretability and versatility, modularqa scores only 3.7% f1 lower than numnet+v2 @xcite , a state-ofthe-art blackbox model designed for drop. mod-ularqa even outperforms this blackbox model by 2% f1 in a limited data setting and demonstrates higher (+7% f1) robustness @xcite . modularqa is competitive with and can even outperform task-specific neural module networks @xcite while producing textual explanations. further, our human evaluation against a split-point based decomposition model trained on decomposition annotation @xcite for hotpotqa finds our explanations to be more trustworthy, understandable, and preferable in 67%-78% of the cases. ## related work many early qa systems were designed as a combination of distinct modules, often composing outputs of lower-level language tasks to solve higherlevel tasks @xcite . however, much of this prior work is limited to pre-determined composition structures @xcite @xcite . various modular network architectures have been proposed to exploit compositionality @xcite . the closest models to our work are based on neural module networks (nmn) @xcite which compose task-specific simple neural modules. we to as 'bridge' questions. complementation refers to questions such as 'what percentage of x is not y?' modularqa can be easily extended to other reasoning types by defining the corresponding hints ( §4.3). question decomposition has been pursued before for complexwebquestions (talmor and berant, 2018) and hotpotqa. both approaches @xcite focus on directly training a model to produce sub-questions using question spans-an approach not suitable for drop questions (as illustrated in fig. 1 ). our nextquestion generator overcomes this limitation by generating free-form sub-questions in the language of existing models. @xcite also use a text-to-text model to generate sub-questions for hotpotqa. however, they generate simpler questions without capturing the requisite reasoning, and hence use them mainly for evidence retrieval. break @xcite follows an alternative paradigm of collecting full question decomposition meaning representations (qdmr) annotations. while this can be effective, it relies on costly human annotation that may not generalize to domains with new decomposition operations. its decompositions are generated in a model-agnostic way and still need qa systems to answer the subquestions, e.g, high-level qdmr questions such as "which is earlier?" and "which is longer?" would need special systems that can map these to symbolic comparisons. in contrast, tmns start with pre-determined models and learn to generate decompositions in their language. while many multi-hop qa models exist for hotpotqa and drop, these are often equally complex models @xcite @xcite focusing on just one of these datasets. only on hotpotqa, where supporting sentences are annotated, can these models also produce post-hoc explanations, but these explanations are often not faithful and shown to be gameable @xcite . tmns are able to produce explanations for multiple datasets without needing such annotations, making it more generalizable to future datasets. ## text modular networks tmns are a family of architectures consisting of modules that communicate through language learned from these modules, to accomplish a cer- qa s nextgen qc: how many years did it take for the services sector to rebound? qa c q 1 : in what year did the services sector rebound? q 2 : when did the services sector start to take a dip? q 3 : diff(2003, 2002)=? a 3 : 1 a 1 : 2003 a 2 : 2002 answer: 1 figure 2: a sample inference on a drop question using tmns with the text-to-text interactions between the next-question generator d and existing qa models a . building a text modular network. the key challenge in building a text modular networks is developing the next-question generator model. training this model requires a next-question prediction dataset where each example is a step in the iterative progression of sub-question generation. for example, the second step in fig. 2 is: ## modularqa system we next describe a specific instantiation of the text modular network: modularqa -a new qa system that works across hotpotqa and drop. to handle these datasets, we first introduce the two qa sub-models( §4.1), the sub-task question models for these models( §4.2), our approach to build training data ( §4.3), and the inference procedure used for question-answering( §4.4). figure 3 : the overall flow of building the training data for the next-question generator, given a complex question. para, p: ... the sector decreased by 7.8 percent in 2002, before rebounding in 2003 ... question, qc: how many years did it take for the services sector to rebound? answer a: 1 @xmath0, @xmath1, @xmath2, a1, v1): in what year did the services sector rebound? @xmath3, @xmath4, @xmath5, a2, v2): when did the services sector start to take a dip? @xmath6, @xmath7, @xmath8, 2003, 2002} → @xmath9, a3, v3): diff(2003, 2002) → @xmath10 ## experiments to evaluate our modular approach, we use two datasets, drop and hotpotqa, that contain ques10 if an intermediate answer is unused or vocabulary of question chain is too different from the input question. see appendix a.3 for more details.