Simple Compounded-Label Training for Fact Extraction and Verification (2020)

Abstract: Automatic fact checking is an important task motivated by the need for detecting and preventing the spread of misinformation across the web. The recently released FEVER challenge provides a benchmark task that assesses systems’ capability for both the retrieval of required evidence and the identification of authentic claims. Previous approaches share a similar pipeline training paradigm that decomposes the task into three subtasks, with each component built and trained separately. Although achieving acceptable scores, these methods induce difficulty for practical application development due to unnecessary complexity and expensive computation. In this paper, we explore the potential of simplifying the system design and reducing training computation by proposing a joint training setup in which a single sequence matching model is trained with compounded labels that give supervision for both sentence selection and claim verification subtasks, eliminating the duplicate computation that occurs when models are designed and trained separately. Empirical results on FEVER indicate that our method: (1) outperforms the typical multi-task learning approach, and (2) gets comparable results to top performing systems with a much simpler training setup and less training computation (in terms of the amount of data consumed and the number of model parameters), facilitating future works on the automatic fact checking task and its practical usage.

## introduction the increasing concern with the spread of misinformation has motivated research regarding automatic fact checking datasets and systems @xcite @xcite @xcite @xcite our code will be publicly available on our webpage. several top-ranked approaches on fever @xcite @xcite decompose the task into 3 subtasks: document retrieval, sentence selection, and claim verification, and follow a similar pipeline training setup where sub-components are developed and trained sequentially. although achieving higher scores on benchmarks, pipeline training is timeconsuming and imposes difficulty for fast application development since downstream training relies on data provided by a fully-converged upstream component. the impossibility of parallelization also causes data-inefficiency as training the same input sentence for both sentence selection and claim verification requires twice the computation, whereas humans can learn the task of sentence selection and claim verification jointly. in this work, we simplify the training procedure and increase training efficiency for sentence selection and claim verification by merging redundant components and computation that exist when training the two tasks separately. we propose a joint training setup in which sentence selection and claim verification are tackled by a single neural sequence matching model. this model is trained with a compounded label space in which for a given claim, an input sentence that is labeled as "non-select" for sentence selection module training will also be labeled as "notenoughinfo" for claim verification module training. similarly, input evidence that is labeled as "supports" or "refutes" for claim verification module training will also be labeled as "select" for sentence selection module training. to validate our new setup, we compare with the previous pipeline setup and a multi-task learning setup which trains the two tasks alternately. fig. 1 illustrates differences among these three setups. results indicate that: our method (1) outperforms the multi-task learning setup, and (2) yields comparable results with a top performing pipelinetrained system while consuming less than half the number of data points, reducing the parameter size by one-third, and converging to a functional state much faster than the pipeline-trained system. we argue that the aforementioned design simplification and training acceleration are valuable especially during time-sensitive application development. ## experimental setup we focused on comparing the following five nsmn 1 training setups for sentence selection and claim verification. we obtain upstream document retrieval data using the method in @xcite . training details are in the appendix. pipeline: we train separate sentence selection and claim verification models as in @xcite . multi-task learning: we follow the neural multitask learning setup called alternate training @xcite @xcite , where each batch contains examples from a single task only. we build a single nsmn model for both selection and verification and alternatively optimize the two tasks. ## conclusion we present a simple compounded-label setup for jointly training sentence selection and claim verification. this setup provides higher training efficiency and lower parameter size while still achieving comparable results to the pipeline approach.