C laim C atchers at S em E val-2025 Task 7: Sentence Transformers for Claim Retrieval (2025)

Abstract: Retrieving previously fact-checked claims from verified databases has become a crucial area of research in automated fact-checking, given the impracticality of manual verification of massive online content. To address this challenge, SemEval 2025 Task 7 focuses on multilingual previously fact-checked claim retrieval. This paper presents the experiments conducted for this task, evaluating the effectiveness of various sentence transformer models—ranging from 22M to 9B parameters—in conjunction with retrieval strategies such as nearest neighbor search and reranking techniques. Further, we explore the impact of learning context-specific text representation via finetuning these models. Our results demonstrate that smaller and medium-sized models, when optimized with effective finetuning and reranking, can achieve retrieval accuracy comparable to larger models, highlighting their potential for scalable and efficient misinformation detection.

## introduction given the vast volume of online content, manually fact-checking every claim is impractical and timeconsuming. moreover, many claims are recurrent, which do not need to be verified repeatedly @xcite . to address these challenges, retrieving previously fact-checked claims from a database can greatly support an automated fact-checking pipelines @xcite . research on retrieving previously fact-checked claims (claim retrieval) has evolved from traditional methods, such as bm25 @xcite , to more advanced approaches leveraging sentence transformers @xcite and language models @xcite @xcite . however, given the dearth of research tackling the task from a multilingual angle, semeval 2025 task 7 @xcite focuses on multilingual claim retrieval. given a social media post, the task aims to retrieve a matching claim from a database of fact-checked claims. the task features two challenges: monolingual (posts and claims in the same language) and cross-lingual (posts and claims in different languages) claim retrieval. we present our experiments for both the monolingual and cross-lingual tasks. we investigate the effectiveness of various sentence transformer models, ranging from 22m to 9b parameters, in combination with retrieval strategies such as nearest neighbors and reranking techniques. additionally, we finetune these models to assess the impact of learning context-specific representation on claim retrieval tasks. our findings reveal that smaller and medium-size models can achieve performance comparable to larger models when optimized with effective finetuning and reranking strategies. this suggests that these computationally effective models can still be adapted for accurate retrieval, making them a practical choice for real-world automated fact-checking systems. ## background one of the first studies on claim retrieval was @xcite , who introduced a pipeline using a fast lexical search to select fact-checked claims, followed by language models for reranking. the claim retrieval task also featured in the 2020-2022 editions of the clef-checkthat competition @xcite @xcite . in 2020, the best-performing team, buster.ai, used data augmentation and additional training datasets @xcite . in 2021, the best result in the english task was obtained by team aschern @xcite , which fine-tuned sbert @xcite and then used lambdamart to rank the top 20 matches. finally, in 2022, team riet ranked first (shliselberg and dori-hacohen, 2022) using sentence-t5 for candidate selection, instead of bm25. for final reranking, they utilized gpt-neo, an autoregressive language model @xcite . using llms to rerank selected candidates or match claim/post pairs has also been tried @xcite . the task of retrieving previously fact-checked claims has been carried out primarily using datasets in english. some of the exceptions were the 2021 edition of checkthat @xcite which contained a claim retrieval subtask with arabic tweet/claim pairs and a dataset presented by @xcite with whatsapp and public group messages in english, hindi, bengali, malayalam and tamil. the currently most comprehensive dataset with multilingual previously fact-checked claims is multiclaim @xcite , with 28k posts in 27 languages from social media and 206k fact-checks in 39 languages. ## system overview this section discusses our claim retrieval approaches, nearest neighbors search and reranking. we also enhance retrieval by finetuning sentence transformers on training data. figure 1 illustrates our method. the task organizers provided social media posts with original text, english translations, and ocrextracted text (if available), while fact-checks included original and translated text. given a social media post, we concatenated its translated text and ocr-translated text to form the query for retrieving fact-check translations. ## conclusion this paper presents the experiments conducted by the claimcatchers team for semeval task 7, focusing on multilingual fact-checked claim retrieval. we investigate the performance of various sentence transformer models, categorized into three groups-small, medium, and large-based on the number of parameters. these models are evaluated using a range of approaches, including nearest neighbor, reranking, and fine-tuning. our experiment results show that the largest pretrained model, baai/bge-multilingual-gemma2 yields the best performance when applied with nearest neighbor approach in both monolingual and cross-lingual tasks. furthermore, the pretrained medium-sized and large models are powerful enough to retrieve more accurate results than when combined with reranking strategies like bm25. interestingly, some small and mediumsized pretrained models outperform the larger models, indicating that the number of parameters is not the sole factor in achieving rich sentence representations. additionally, the finetuned medium-sized model, novasearch/stella_en_400m_v5, is competitive with the largest model in train partition, indicating that smaller or medium-sized models, with focused fine-tuning, can achieve performance comparable to larger models. these findings highlight the potential of smaller and fine-tuned models to achieve competitive performance, emphasizing the importance of selecting the appropriate ranking and fine-tuning strategies.