An Entity-based Claim Extraction Pipeline for Real-world Biomedical Fact-checking (2023)

Abstract: Existing fact-checking models for biomedical claims are typically trained on synthetic or well-worded data and hardly transfer to social media content. This mismatch can be mitigated by adapting the social media input to mimic the focused nature of common training claims. To do so, Wührl and Klinger (2022a) propose to extract concise claims based on medical entities in the text. However, their study has two limitations: First, it relies on gold-annotated entities. Therefore, its feasibility for a real-world application cannot be assessed since this requires detecting relevant entities automatically. Second, they represent claim entities with the original tokens. This constitutes a terminology mismatch which potentially limits the fact-checking performance. To understand both challenges, we propose a claim extraction pipeline for medical tweets that incorporates named entity recognition and terminology normalization via entity linking. We show that automatic NER does lead to a performance drop in comparison to using gold annotations but the fact-checking performance still improves considerably over inputting the unchanged tweets. Normalizing entities to their canonical forms does, however, not improve the performance.

## introduction fact-checking models trained on synthetic, wellworded and atomic claims struggle to transfer to colloquial content @xcite . there are multiple ways to address this problem: we can build custom datasets and models that verify medical content shared online @xcite @xcite and tackle related tasks @xcite . alternatively, we can adapt the input before addressing other fact-checking tasks. @xcite create claim summaries and find that this improves the detection of previously fact-checked claims. similarly, wührl and klinger (2022a) extract concise claims from user-generated text in an effort to mimic the focused, well-structured nature of the claims the fact-checking models were originally trained on. they find that this improves the accuracy of pretrained evidence-based fact-checking models in the biomedical domain. however, the study by wührl and klinger (2022a) is limited in two ways: (1) their claim extraction method relies on gold-annotated, claimrelated entities. for a realistic evaluation, such an oracle needs to be replaced by an entity recognizer. only then it is possible measure the impact of potential error propagation which may ultimately render the method unfeasible. (2) the claim entities are represented by the original token sequence. this is problematic as medical mentions on twitter potentially contain imprecise, abbreviated, or colloquial terminology. this is in contrast to the terminology in the original model input as well as the documents that we provide as evidence (cf. we address both limitations and evaluate a realworld, fully-automatic claim extraction pipeline for medical tweets which incorporates an entity recognizer. it only relies on the original text as input that contains the claim. we further evaluate the impact of an entity linker for normalizing entity mentions to canonical forms based on the unified medical language system (umls, @xcite . our pipeline improves the fact-checking performance over tasking models to check unchanged tweets. normalizing entities to overcome the terminology mismatch does not improve fact-checking, potentially due to limitations of biomedical entity linking for social media. following the condense seq method, we therefore extract the sequence from the character onset of the first entity to the character offset of the second entity for all pairs of entities found by the ner module. entity linking. to normalize entities, we use the entitylinking component in scispacy @xcite . this model compares an entity mention to concepts in an ontology and creates a ranked list of candidates, based on an approximate nearest neighbor search. for text normalization, we retrieve the canonical name of the top concept. for entities which could not be linked, we use the original mention instead. as the knowledge base, we use umls @xcite . ## conclusion & future work we propose a fully automatic claim extraction pipeline that is capable of handling real-world medical content. we show that entity-based claim extraction has a positive effect on the performance of multiple fact-checking models -even after replacing the entity oracle with automatic ner. while we observe a negative impact of error propagation from ner and a performance drop as a result, factchecking the extracted claims is more successful than checking unchanged tweets. future research may therefore focus on improving the pipeline components as this clearly has the potential to further strengthen the verdict prediction performance. in particular, we expect an improved entity recognizer to have a considerable impact. our work focuses on the biomedical domain and builds upon the assumption by wührl and klinger (2022a) that claims in this domain are strongly cen-tered around entities. claims from other domains may share this property which could make entitybased claim extraction applicable for such claims as well. we leave the evaluation for future work. we find that normalizing entity mentions does not improve the fact-checking performance. however, our analysis shows that the off-the-shelf linking module might be too unreliable. to fully gauge the potential of normalizing entities, future work needs to ensure correct mappings (creating gold links or building a reliable linker) before evaluating the downstream fact-checking performance.