Table-based Fact Verification with Self-adaptive Mixture of Experts (2022)

Abstract: The table-based fact verification task has recently gained widespread attention and yet remains to be a very challenging problem. It inherently requires informative reasoning over natural language together with different numerical and logical reasoning on tables (e.g., count, superlative, comparative). Considering that, we exploit mixture-of-experts and present in this paper a new method: Self-adaptive Mixture-of-Experts Network (SaMoE). Specifically, we have developed a mixture-of-experts neural network to recognize and execute different types of reasoning—the network is composed of multiple experts, each handling a specific part of the semantics for reasoning, whereas a management module is applied to decide the contribution of each expert network to the verification result. A self-adaptive method is developed to teach the management module combining results of different experts more efficiently without external knowledge. The experimental results illustrate that our framework achieves 85.1% accuracy on the benchmark dataset TabFact, comparable with the previous state-of-the-art models. We hope our framework can serve as a new baseline for table-based verification. Our code is available at https://github.com/THUMLP/SaMoE .

## introduction fact verification, aiming to determine the consistency between a statement and given evidence, has become a crucial part of various applications such as fake news detection, rumor detection @xcite @xcite @xcite . while most existing research focuses on verification based on unstructured text, a new trend is extending the scope to structured evidence (e.g., tables), which is informative and ubiquitous in our daily lives. table-based verification faces different challenges than unstructured-text-based due to the complexity of the requirements, including sophisticated textual, numerical, and logical reasoning across evidence tables; even for some statements, multiple types of reasoning are indispensable to complete the verification. an example is presented in figure 1 . to tackle the challenges above, previous work established two kinds of methods: (1) programenhanced methods @xcite @xcite and (2) table-based pre-trained models @xcite . the program-enhanced methods mainly leverage programs generated by the semantic parser. specifically, statements are parsed into executable programs to extract the logical/numerical semantics, which is further be leveraged together with contextual semantics learned by a language model (e.g., bert) in inference. however, the semantic parsers that generate semanticconsistent programs must be trained in a weak supervision setting, which brings difficulties in training. furthermore, generalizing this method to other datasets is almost impossible without the api set modification according to the reasoning requirements on the new datasets. the table-based pre-trained models leverage elaborate model structure @xcite and pre-training tasks @xcite to enhance the reasoning skills on structured data. nevertheless, two significant shortcomings remain. firstly, the process is demanding due to the tremendous computing resources required by pre-training. moreover, the effectiveness of pretraining to its downstream tasks mainly depends on the adaptability between these two tasks. therefore, implementing pre-training tasks may fail to meet the requirements when facing the unseen reasoning types demanded by new datasets. in this paper, we introduce an innovative framework, self-adaptive mixture of experts (samoe), to address the previously mentioned problems. the entire framework is illustrated in figure 2 . samoe consists of 3 components: feature extractor, experts, and management module, which is the combination of manager and supervisor networks. each expert initially takes the same feature as input and then learns to deal with different parts of the reasoning types (e.g., contextual/logical/numerical) required by table-based verification. a management module is designed to guide the training of experts and combine experts' verification results effectively. the manager network in this module assigns each expert a unique attention score, allowing each individual to focus on different kinds of reasoning types and summarizes experts' entire outputs as the final verification result. however, managers failed to allocate the highest attention score to the expert who performs best on the current reasoning type in most circumstances. to alleviate this problem, we introduce a supervisor network to adjust the attention score given by the manager. the supervisor network is trained selfadaptively (i.e., it learns directly from experts' performance on the train set) without prior knowledge of the task or dataset. extensive experiments are conducted to show that our proposed framework, implemented with a general pre-trained language model roberta @xcite , outperforms previous state-of-the-art methods, including tablebased pre-trained models. the main contributions of this work are as follows: ## research question the table-based verification task expects one to determine whether a statement s is entailed or refuted by an evidence table t . the process above can be regarded as a binary classification task and thus denoted as f (s, t ) = ŷ, where f is the verification model and ŷ ∈ {0, 1} its prediction. ## methods we present the proposed framework (samoe), which leverages a set of experts to deal with differ-ent parts of the reasoning types involved in tablebased verification. this section is organized as follows. sec.3.1 introduces the feature extractor that extracts the joint semantics of the table-statement pair. sec.3.2 describes experts that verify the statements separately based on the same extracted semantics. sec.3.3 describes the management module that guides the experts' training and combines their verification results effectively; two components of this module, the manager and the supervisor, are introduced in this section individually. ## parameter learning parameters in samoe are learned in two consecutive stages: 1) multi-expert training: parameters in the feature extractor, experts and the manager are end-to-end optimized under the supervision of labels; 2) self-adaptive learning: parameters in the supervisor are self-optimized by observing experts' performance on the train set (other parameters are fixed simultaneously). a weighted sum of two losses is minimized in the first stage to achieve diverse and balanced training of experts. for the second stage, we minimize a self-adaptive loss calculated based on the experts' classification loss. subsequent sections introduce these two learning stages in detail. ## related works table-based fact verification most of the current models utilize programs to improve the model's ability to handle various types of numerical and logical reasoning @xcite @xcite @xcite , while eisenschlos et al. (2020); liu et al. (2021) leverage table-based pre-trained models to parse the structural and numerical semantics of tables better. unlike previous works, we use a novel mixture-of-experts framework to handle different logical and numerical semantics without semantic parsing and table-based pre-training. mixture of experts mixture of experts is a special model combining method. @xcite first introduces this method and proposes a loss that encourages competitive learning across expert models. afterwards, it is applied in various fields, including dialog system @xcite , content recommendation @xcite , image classification @xcite , etc. in this paper, we develop a self-adapted mixture-of-experts framework that achieves a more effective combination of experts by learning from the experts' performance on the train set. ## conclusion this paper proposes a new method that exploits the mixture of experts to recognize and execute different types of reasoning required for table-based fact verification. we propose an moe model guided with limited prior knowledge to handle different parts of the reasoning types required by table-based verification with diverse experts. moreover, we design a supervisor network to adjust the imprecise attention score and achieve a more efficient combination across experts. a self-adaptive learning strategy is further applied to train the proposed supervisor network without prior knowledge of the task or dataset. the experiments show that the proposed model achieves a new state-of-the-art performance of 85.1% accuracy on the benchmark dataset tabfact. the ablation studies and analysis further indicate the effectiveness of the proposed moe structure and self-adaptive learning strategy. we hope our work is helpful for those who aim to further exploit the power of mixture-of-experts on table-based reasoning in the future.