Adapting Open Domain Fact Extraction and Verification to COVID - FACT through In-Domain Language Modeling (2020)

Abstract: With the epidemic of COVID-19, verifying the scientifically false online information, such as fake news and maliciously fabricated statements, has become crucial. However, the lack of training data in the scientific domain limits the performance of fact verification models. This paper proposes an in-domain language modeling method for fact extraction and verification systems. We come up with SciKGAT to combine the advantages of open-domain literature search, state-of-the-art fact verification systems and in-domain medical knowledge through language modeling. Our experiments on SCIFACT, a dataset of expert-written scientific fact verification, show that SciKGAT achieves 30% absolute improvement on precision. Our analyses show that such improvement thrives from our in-domain language model by picking up more related evidence pieces and accurate fact verification. Our codes and data are released via Github.

## introduction online contents with false information, such as lies, rumors and conspiracy theories, have been growing significantly and spreading widely during the covid-19 epidemic. an automatic fact-checking system is urgently needed to check these scientific claims, which can avoid undesired consequences. automatic fact-checking has drawn lots of attention from nlp community. researchers mainly focus on stopping misinformation transmission through videos and texts @xcite @xcite . the scientific fact verification task @xcite is come up to deal with covid-fact with high-quality articles of spanning domains from basic science to clinical medicine. nevertheless, the small-scale training data of scifact may limit the performance of covid-fact checking. the state-of-the-art model @xcite achieves only 46.6% precision of fact verification, which is hard to be trusted for users. this paper presents the scientific kgat (scik-gat) to deal with low-resource covid-fact verification. scikgat employs the in-domain language model in the fact extraction and verification pipeline @xcite to adapt fact-checking into covid domain. the in-domain language model transfers covid domain knowledge into pre-trained language models with continuous training and learns medical token semantics towards covid with mask language model based training. the state-of-the-art fact verification model kgat @xcite is also used in scikgat for multi-evidence reasoning in the fact verification module. our experiments show that the in-domain language modelings achieve better performance for various components in the whole fact extraction and verification pipeline by achieving more accurate evidence selection and fact verification. our indomain language modelings improve the fact verification performance with more than 10% absolute f 1 score and 30% absolute precision (from 46.6% to 76%) than previous state-of-the-art on scifact. such improvement shows that our model provides a set of solutions for low-resource fact verification tasks, such as covid-19. ## related work existing fact extraction and verification models usually employ a three-step pipeline system @xcite : document retrieval (abstract retrieval), sentence selection (rationale selection) and fact verification @xcite . the preliminary fact verification methods concatenate all evidence pieces @xcite for fact verification. kgat @xcite conducts fine-grained multiple evidence reasoning with a graph and achieves the stateof-the-art for fact verification @xcite . the reasoning ability of the pre-trained language model is crucial and helps improve fact verification performance @xcite @xcite . some work @xcite transfers medical domain knowledge into pre-trained language models for better medical semantic understanding, which provides a potential way to deal with covid-fact checking problem. ## methodology this section describes our scikgat for fact extraction and verification. we first introduce the pipeline of fact extraction and verification (sec. 3.1) and then continuously train the bert based model (sec. 3.2) for the whole. ## experimental methodology this section describes the dataset, evaluation metrics, baselines, and implementation details. dataset. the recently released dataset sci-fact @xcite ) is leveraged in our experiments. it consists of 1,409 annotated claims with 5,183 scientific articles. all claims are classified as support, contradict or not enough info. the training, development and testing sets contain 809, 300 and 300 claims, respectively. fever @xcite is also used by official baselines to train the fact verification modules of baselines and our models. the fever consists of 185,455 annotated claims with 5,416,537 wikipedia documents. baselines. since the scientific fact verification task is recently released, our baselines are mainly from @xcite . they first use tf-idf for abstract retrieval and then use roberta (large) and sibert for rationale selection. kgat and roberta (large) are leveraged for fact verification. the rationale selection module is trained with scifact and the fact verification module is trained with data from fever and scifact @xcite . ## evaluation result this section first tests the overall performance of scikgat. then it studies the impacts of our indomain language modeling techniques in knowl ## conclusion this paper presents in-domain language modeling methods for open domain fact extraction and verification, which transfer domain knowledge for the covid-fact checking task. our experiments show that our pipeline significantly improves the fact-checking performance of the state-of-the-art model with more than 30% absolute prediction precision. our analyses illustrate that our model has stronger reasoning ability with continuous training and benefits from covid related knowledge.