A Flexible and Extensible Framework for Multiple Answer Modes Question Answering (2021)

Abstract: This paper presents a framework to answer the questions that require various kinds of inference mechanisms (such as Extraction, Entailment-Judgement, and Summarization). Most of the previous approaches adopt a rigid framework which handles only one inference mechanism. Only a few of them adopt several answer generation modules for providing different mechanisms; however, they either lack an aggregation mechanism to merge the answers from various modules, or are too complicated to be implemented with neural networks. To alleviate the problems mentioned above, we propose a divide-and-conquer framework, which consists of a set of various answer generation modules, a dispatch module, and an aggregation module. The answer generation modules are designed to provide different inference mechanisms, the dispatch module is used to select a few appropriate answer generation modules to generate answer candidates, and the aggregation module is employed to select the final answer. We test our framework on the 2020 Formosa Grand Challenge Contest dataset. Experiments show that the proposed framework outperforms the state-of-the-art Roberta-large model by about 11.4%.

## introduction natural language inference (nli) is an important topic in the artificial intelligence (ai) field, and any nli related issue can be checked by asking an appropriate corresponding question @xcite . therefore, the question answering (qa) task has become a very suitable testbed for evaluating nli models and checking the progress of current techniques. accordingly, the ministry of science and technology of taiwan has organized the formosa grand challenge open contest series 1 (fgc) in 2018, which mainly evaluates the reasoning/inference capability on natural texts, to promote the ai progress in taiwan. specifically, this open contest covers a variety of answer modes; that is, it needs different inference mechanisms (such as extraction, entailment-judgement, aggregative-operation, etc.) to get the desired answer. as a result, the system/framework must be able to handle various answer modes at the same time. the previous frameworks for the qa task could be classified into two main categories according to the number of answer modules adopted: (1) single answer generation module @xcite @xcite , which involves only one answer mode, and allows merely one type of replying format (such as identifying a span within the given passage, giving yes/no answer, free text reply, etc.). (2) multiple answer generation modules @xcite @xcite , which adopts several answer generation modules, and each module conducts a specific inference mechanism (or, answer mode) with a specific replying format. since the first category only considers one answer mode, the types of questions that can be handled are quite limited. for example, it is not suitable for handling the fgc-2020 qa task foot_0 , which covers various question types and needs different answer modes to get the desired answers. in contrast, the approaches under the second category adopt the divide-and-conquer strategy, which adopts a different answer generation module for each specific answer mode. since each answer generation module only needs to consider a specific answer mode, it will be easier to design and add new inference mechanisms. among those second category approaches, the framework of watson @xcite is not designed for end-to-end training; therefore, it is not suitable for modern neural-network multi-task learning due to the complicated flow/architecture under its statistics-based architecture. also, the framework adopted in either @xcite or @xcite does not have an aggregation layer/module to merge the answers generated from different answer generation modules (i.e., the output is only picked from a specific module, and merging is not allowed). therefore, their approaches not only have the error accumulation problem foot_1 (i.e., once a wrong module is selected, this error will propagate to the next answer-gener-ation stage), but also lose the advantage of combining the strength of different inference mechanisms. additionally, all modules will be activated in parallel under their frameworks @xcite , so computing resources on those modules that should not be activated for a given question would be wasted. to overcome the problems mentioned above, a flexible and extensible framework is proposed in this paper. it adopts a divide-and-conquer strategy, and possesses the following main modules/functionalities: (1) a supporting evidences locating module, which extracts supporting evidences from the passage to narrow down the searching space. (2) a dispatch module, which would select and activate several appropriate answer generation modules; also, the answer type distribution will be provided to each answer generation module as a reference, based on the answer mode. (3) a set of answer generation modules, each of them generates a few local/module outputs (i.e., possible answers) if it is activated. (4) an aggregation module, which picks the best answer at the final stage by merging the answer candidates from those activated answer generation modules. the strengths of the proposed framework are summarized as follows: (1) with the dispatch module, it is flexible for handling different question types with the same framework; as a result, it is extensible for adding more answer modes in the future. (2) with the aggregation module, it is able to merge the results from various modules; it thus possesses the capability of combining the strength of different inference mechanisms, and also reduces the error accumulation problem. (3) it is designed to fit the neural-network based end-to-end multi-task learning framework; therefore, it can be implemented with an appropriate neural network without much effort. (4) since the dispatch module only activates the corresponding modules according to the given question, it will not waste computing resources on those modules that are irrelevant and should not be activated. in comparison with ibm watson framework, which adopts a complicated flow/architecture with probabilistic models, our proposed framework is usually preferred if the desired answer is explicitly given in the supporting sentence; otherwise, a more complicated mechanism must be adopted). the 33rd conference on computational linguistics and speech processing (rocling 2021) taoyuan, taiwan, october 15-16, 2021. the association for computational linguistics and chinese language processing adopts the neural-network based approach and can be optimized by the end-to-end training strategy. in comparison with the approaches from andor et al. (2019) and hu et al. (2019), which lack the mechanism to merge different answer candidates, our proposed framework only activates several possible/responsible modules and has the ability to aggregate the outputs from various modules. the proposed framework is tested on the fgc-2020 qa dataset, which contains 1,322 questions. this dataset covers eight different answer modes (i.e., single-span-extraction, multi-span-extraction, yes/no, aggregative-operation, arithmetic-operations, date-duration, kinship, and summarization) and ten different answer types (i.e., yes/no, number-measure, kinship, person, date-duration, location, organization, object, event, and misc). the experiment results show that our system outperforms the baseline roberta-large (liu et al., 2019) model by 11.4%. in summary, this paper makes the following contributions: (1) we propose a novel modular framework/model that is more flexible for handling/adding various inference mechanisms. (2) we propose a novel aggregation model to merge various answer candidates. (3) we conduct experiments to show that the proposed framework outperforms the state-of-the-art roberta-large model on the fgc-2020 qa dataset. ## the proposed approach in this section, the proposed divide-and-conquer qa model is first described in section 2.1. the descriptions of the architecture of the proposed model is then presented in section 2.2. afterwards, section 2.3 provides the concepts and principles of designing each answer generation module. ## error analysis and discussion for official final test-set as figure 7 shows, the overall system performance degrades significantly (down 0.314, from 0.705 to 0.391) when we move from fgc-2020-pre test-set to fgc-2020-final test-set. it is mainly because the questions in the fgc-2020-final test-set is generally more difficult than that in the fgc-2020-pre test-set. and it is also because the involved topics (also their associated lexicons), the distributions of the distributions of answer mode and answer type in the fgc-2020-pre and fgc-2020-final test-sets. figure 7 . the overall system accuracy rate on the fgc-2020-pre and fgc-2020-final test-sets. figure 8 . the accuracy rates associated with various answer modes on the fgc-2020-final test-set. the 33rd conference on computational linguistics and speech processing (rocling 2021) taoyuan, taiwan, october 15-16, 2021. the association for computational linguistics and chinese language processing both answer mode and answer type drift significantly from fgc-2020-pre test-set to fgc-2020final test-set (as shown in figure 6 ). since almost all our current answer generation modules adopt bert-based approaches, and it is well-known that bert conducts the inference mainly based on surface-clues/hidden-distributionbias @xcite @xcite , the mismatch of those surface-clues/distributions thus causes serious degradation. on the other hand, it also implies that bert-based approaches, although they have become state-of-the-art models, are still not capable to handle the fgc-2020 kind of tests (which require deep reasoning and cannot be falsely solved simply with surface-clues/distribution-bias). specifically, the performance of the elementary questions drops more (down 0.436, from 0.827 to 0.391) in comparison with that of advanced ones (down 0.158, from 0.549 to 0.391). the performance of the advanced questions is less affected because those questions require deeper reasoning, and is thus less affected by the drift of topics and the distribution of answer mode/answer type mentioned above. if we zoom into various answer modes, it is observed that the multi-span-extraction causes most overall degradation in the fgc-2020-final test-set, which is mainly due to both its low accuracy rate (15.4% in figure 8 ) and its high answer mode portion (28% in figure 6 )). it seems that the tag-based approach (section 2.3) is not capable of handling the multi-span-extraction questions involved in this dataset, as getting a multi-span answer needs to locate various list-members via matching the structures @xcite of the question and the passage, not just regarding it as a sequence-tagging task. ## conclusion we proposed a divide-and-conquer model/framework for answering the questions in fgc-2020 qa dataset, which covers various answer modes. with the proposed dispatch-layer, the proposed framework is flexible for handling various answer modes with different modules simultaneously, and is extensible for adding new answer modes and answer types in the future. also, with the proposed aggregation-layer, the proposed framework can take advantage of different inference mechanisms, and also reduce the error accumulation problem. last, due to its design for fitting the end-to-end multitask learning framework, the proposed framework could be implemented with an appropriate neural network and is thus more suitable for end-to-end optimization without much effort. we have tested the proposed framework on 2020 formosa grand challenge contest qa dataset. the experiment results show that our system outperforms the baseline roberta-large model about 11.4% on the fgc-2020-pre test-set. however, the overall system performance drops significantly (about 31.4%) from the fgc-2020-pre test-set to the fgc-2020-final test-set. on the other hand, together with our another dialog sub-system (tested on the fgc-2020-final dialog test-set), we obtained 44.1 total score (out of 100; the human performance is 68.2), which outperforms that of the official top one system (announced in this contest) 7.4 points.