D uluth at S em E val-2025 Task 7: TF - IDF with Optimized Vector Dimensions for Multilingual Fact-Checked Claim Retrieval (2025)

Abstract: This paper presents our approach to the SemEval-2025 Task 7 on Multilingual and Crosslingual Fact-Checked Claim Retrieval. We implemented a TF-IDF-based retrieval system with experimentation on vector dimensions and tokenization strategies. Our best-performing configuration used word-level tokenization with a vocabulary size of 15,000 features, achieving an average success@10 score of 0.78 on the development set and 0.69 on the test set across ten languages. Our system showed stronger performance on higher resource languages with large performance gaps compared to the top-ranked system, which achieved 0.96 average success@10. Our findings suggest that though advanced neural architectures are increasingly dominant in multilingual retrieval tasks, properly optimized traditional methods like TF-IDF remain competitive baselines, especially in limited resource scenarios.

## introduction the semeval-2025 task 7 on multilingual and crosslingual fact-checked claim retrieval addresses the challenge of identifying previously factchecked claims across multiple languages @xcite . this task is important because the global spread of misinformation makes it difficult for professional fact-checkers to manually identify existing fact-checks, as false narratives frequently cross linguistic boundaries. the task covers a diverse set of languages including english, spanish, german, portuguese, french, arabic, malay, thai, and two surprise languages (polish and turkish) in the test phase, creating a broad evaluation framework for multilingual information retrieval systems in the fact-checking domain. our approach uses a tf-idf foot_0 @xcite based retrieval system with optimization of key parameters. we experimented with different vector dimension settings and tokenization to identify an optimal configuration for multilingual retrieval. after testing, we found that a wordlevel tokenization approach with a vocabulary size of 15,000 features provided the best performance across all languages. our pipeline foot_1 includes data cleaning, text preprocessing, vector embedding generation, similarity computation, and ranked retrieval of the most relevant fact-checked claims for each social media post. the optimized tf-idf system achieved an average success@10 score of 0.78 on the development set and 0.69 on the test set, ranking 23rd out of 28 participating teams. the top-performing system outperformed our approach with an average score of 0.96, showing the performance gap between traditional statistical methods and modern neural approaches. our system performed better on languages such as french: 0.814 and arabic: 0.820, but struggled with others such as english: 0.452 and spanish: 0.546. these results suggest that though tf-idf can serve as a reasonable baseline, significant improvements in multilingual retrieval tasks require advanced contextual embedding techniques and language-specific optimizations. ## task description the semeval-2025 task 7 organizers offer the mul-ticlaim dataset, an augmented and modified version of the original multiclaim corpus @xcite . this dataset collects social media posts containing potential misinformation, previously fact-checked claims, and various related information. the following information in the dataset is used in this task: the social media post containing a potential misinformation claim, for example : "[(1525826671.0, 'fb')]","[(""fb/david avocado wolfe flip the bell peppers over to check their gender. the ones with four bumps are female and those with three bumps are male. the female peppers are full of seeds, but sweeter and better for eating raw and the males are better for cooking. i didn't know this!"", ""fb/david avocado wolfe flip the bell peppers over to check their gender. the ones with four bumps are female and those with three bumps are male. the female peppers are full of seeds, but sweeter and better for eating raw and the males are better for cooking. i didn't know this!"", [('eng', 1.0)])]",['false information']," the task is to retrieve relevant fact-checked claims from the fact_check collection that match the given post. retrieved claims are classified into the following two tracks based on language: the provided training data consists of 153,743 samples in the fact_checks.csv. the distribution across languages is: english: 85,734 (55.76%), portuguese: 21,569 (14.03%), arabic: 14,201 (9.24%), spanish: 14,082 (9.16%), malay: 8,424 (5.48%), german: 4,996 (3.25%), french: 4,355 (2.83%), and thai: 382 (0.25%). the test phase introduced two surprise languages: turkish and polish. the agreement between the model's predictions and the ground truths is evaluated based on successat-10 (s@10), which measures if at least one relevant fact-checked claim appears among the top 10 retrieved results. participants may choose to participate in either one or both tracks when making submissions, with a maximum of 5 submissions allowed during the test phase, where only the last submission will be counted for the final leaderboard. in our submission, we participated only in the monolingual track. ## related work fact-checked claim retrieval has seen various approaches over the past few years. early work by @xcite and later, the extensive multi-claim study by @xcite , provided a broad multilingual dataset that expanded the task past english and monolingual settings. their work highlighted the challenges in retrieving factchecked claims across many languages. parallel to the fact-check retrieval, the development of multilingual language models (mllms) is explained in @xcite 's primer. mllms like mbert and xlm-r have shown impressive zero-shot transfer capabilities in a wide range of tasks. these models are designed to learn shared representations across languages, using large-scale multilingual pre-training to enable cross-lingual transfer. ## system overview the section presents a detailed view of our proposed system which generates the monolingual predictions for the given fact checks and posts; based on the given tasks file. ## experiment we stuck to tf-idf because it gave us the best results. however, there were several models that we wanted to experiment with before finalizing tf-idf: tf-idf -within tf-idf, we experimented with different vector sizes and with the analyzer parameter having values: 'word', 'char', 'charwb'. xlm-roberta @xcite ) -we tried two approaches for xlm-roberta foot_3 , the first approach used mean pooling over token embeddings weighted by attention masks, capturing richer contextual information, whereas the second approach relied on the cls token embedding. the first approach processed texts one at a time, making it slower, while the second implemented batch processing (batch @xmath0, which aimed to improve efficiency. multilingual e5 large @xcite ) -we use the e5 retriever, a transformer-based embedding model foot_4 . our retriever encodes text using an average pooling strategy over token embeddings, followed by l2 normalization for better similarity computation. we generated dense embeddings in batches. fasttext @xcite ) -fasttext uses pre-trained word vectors for 157 languages @xcite (cc.en.300.bin) foot_5 . our fasttextretriever preprocesses text by removing special characters and normalizing case before generating sentence embeddings. we compute dense representations for fact-check claims and social media posts, t5-model @xcite ) -the sentence-transformers/gtr-t5-large foot_6 was used to generate dense text embeddings. our gtr retriever preprocesses text by removing special characters and normalizing case before encoding claims and social media posts into 768-dimensional vectors. ## experimental results and discussion the default parameter 'word' along with a vector size of 15,000 gave us the best success@k score of 0.78 (avg) on the dev set and 0.69 (avg) on the test set. the duluth system achieved the highest success@k score of 0.78 (dev set) and 0.69 (test set) with the tf-idf 15k configuration, which outperformed all other methods. the tf-idf 15k configuration showed consistent high performance across all languages, and improved over the tf-idf 10k model (0.7757 vs. 0.7630 avg). tf-idf performed well on english (0.6130) and spanish (0.8358), which indicated that increasing the vocabulary size improved retf-idf was pretty standard across languages, but minor performance drops were observed for german (deu) and thai (tha). character-based analyzers underperformed, more so in german and arabic, mainly due to their complex word forms and stuck together structures. e5 large and t5 model had strong performance, but they had more errors in lower-resource languages (e.g., malay and thai), this could be due to pretraining biases as these models have largely been trained on other languages. bert-based models had very low performance, especially on arabic (ara), malay (msa), and thai (tha), which suggested poor generalization in non-latin scripts. ## future work and conclusions potential modifications to improve the performance of t5 model and e5 large could be fine-tuning these models on a more extensive and diverse set of fact-checking data to help them better capture domain-specific features. models could be generalized more by augmenting and balancing the training data, resulting in a reduced bias observed in pretraining. experimenting with language-specific tokenizers could address difficulty with non-latin scripts and complex word forms. we presented an optimized tf-idf retrieval system for multilingual fact-checked claim retrieval. by fine-tuning vector dimensions and using wordlevel tokenization with a 15,000-feature vocabulary, our approach achieved robust performance-0.78 success@10 on the development set and 0.69 on the test set. these results highlight that, with careful tuning, traditional methods can be competitive even against advanced neural models, especially in high-resource settings. we plan to investigate why our neural methods did not reach the level of tf-idf by doing a caseby-case error analysis of where tf-idf was performant and where it failed, to see if there exist any patterns to discover. future work could incorporate the reviewers' recommendation of a hybrid approach, combining tf-idf and neural embeddings, to better support lower-resource languages and enhance retrieval effectiveness.