CAIDAS at S em E val-2025 Task 7: Enriching Sparse Datasets with LLM -Generated Content for Improved Information Retrieval (2025)

Abstract: The focus of SemEval-2024 Task 7 is the retrieval of relevant fact-checks for social media posts across multiple languages. We approach this task with an enhanced bi-encoder retrieval setup, which is designed to match social media posts with relevant fact-checks using synthetic data from LLMs. We explored and analyzed two main approaches for generating synthetic posts. Either based on existing fact-checks or on existing posts. Our approach achieved an S@10 score of 89.53% for the monolingual task and 74.48% for the crosslingual task, ranking 16th out of 28 and 13th out of 29, respectively. Without data augmentation, scores would have been 88.69 (17th) and 72.93 (15th).

## introduction semeval task 7 @xcite focuses on retrieving relevant fact-checks for social media posts across multiple languages. it comprises two subtasks: (1) monolingual retrieval, where posts and fact-checks share the same language, and (2) crosslingual retrieval, where they differ. our approach involved fine-tuning sentence transformers @xcite as bi-encoders on both the training data and synthetically generated samples. given the dataset's sparsity (there are much more fact-checks than posts), we bridge this gap by generating synthetic posts using llms. we observed slight performance gains when generating posts for fact-checks without assignments, particularly enhancing retrieval for turkish and polish, despite the absence of training data for these languages. however we found that generating variations of existing posts did not enhance retrieval performance. ultimately, we ranked 16th (of 28) in the monolingual and 13th (of 29) in the crosslingual subtask. without data augmentation, scores would have dropped to 88.69 (17th) and 72.93 (15th). our results suggest that † these authors contributed equally to this work. the main limitation was the inability of synthetic posts to fully match the style and content of real posts. ## background the task involves matching fact-checks to social media posts using a modified subset of the multi-claim dataset @xcite . it comprises two tracks: monolingual and crosslingual. in the monolingual track, each post is paired with factchecks in the same language. in the crosslingual track fact-checks are assigned regardless of their language. the dataset spans arabic, english, french, german, malay, polish, portuguese, spanish, thai, turkish. the goal is to retrieve the top-10 most relevant fact-checks for each post. we participated in both tracks, with performance evaluated using the binary success@10 metric, which assigns a score of 1 if at least one of the top 10 retrieved items is relevant and 0 otherwise. matching social media posts with applicable factchecks is a crucial research area, since misinformation spreads rapidly on social media networks.this task aligns with information retrieval, where posts serve as queries and fact-checks as the document corpus. @xcite tested bm25 @xcite and sentence transformers, with the latter performing better. sentence transformers offer two retrieval strategies: biencoders, which encode queries and documents separately before computing similarity, and crossencoders, which evaluate entire sentence pairs directly. while cross-encoders provide higher accuracy, they are computationally expensive @xcite , which makes them less suitable for fact-checking in a fast-paced setting like social-media platforms. to enhance retrieval, we propose enriching the dataset with synthetic training data generated by an llm, following @xcite . ## methodology: retrieval setup the dataset consists of social media posts and factchecks. each post is divided into an ocr-text and a post-text, both available in their original language and an english translation. either or both may contain relevant information: our initial tests showed the best retrieval performance when concatenating the english version of both fields. for fact-checks, each entry includes a title and a claim in both the source language and english. unlike posts, the best performance was achieved using only the english claim. our sole preprocessing step was whitespace stripping. ## methodology: data augmentation the dataset contains only 31,305 fact-check-topost pairs, covering less than 10% of the 205,751 available fact-checks. the authors estimate their dataset could contain up to seven times more pairs than annotated @xcite . our goal was to enhance retrieval performance by enriching the dataset without scraping new posts or manually searching for additional fact-check-to-post pairs. instead, we developed a system to generate new pairs by creating synthetic social media posts matching existing fact-checks. to ensure a relation between the synthetic post and a real fact-check, we devised two approaches detailed below. the first approach, generating synthetic posts from fact-checks, employs an llm to generate a social media post based on each fact-check. the fact-check itself serves as input, naturally maintaining the connection in content. the second approach, generating synthetic posts from real posts, uses an llm and an existing post from a fact-check-to-post pair to generate a new post that references the same content. this ensures that the generated post remains linked to the same fact-check. ## analysis of synthetic data the final retrieval results indicate that augmenting the dataset with synthetic fact-check-to-post pairs only marginally improves performance. this section explores potential reasons by analyzing differences between synthetic and real data. we assess spelling errors as per @xcite and measure vocabulary richness using the mtld-score (measure of textual and lexical diversity) @xcite , accounting for unique words while compensating for text length. grammatical correctness is evaluated via grammatical errors per word @xcite 4 . to highlight key (semantic) differences, we embed posts using all-minilm-l6-v2 @xcite and reduce dimensions via umap (mcinnes et al., 2020) for visualization. cluster comparison is performed using intra-cluster distance, measured by the wcss-score (agrawal and kushwaha, 2018), which computes the mean squared distance of each point to its cluster centroid. ## conclusion we introduced our approach for retrieving relevant fact-checks for social media posts. for retrieval, we utilized a minilm-based bi-encoder and explored synthesizing additional training data with llms. specifically, we followed two main strategies: prompting llms to generate posts based on existing 1) fact-checks, or 2) posts. overall, augmenting the training set using the first strategy improves both monolingual and cross-lingual retrieval. furthermore, we qualitatively identify the main hurdle of our synthetic approaches: llms commonly fail to consistently generate posts with a complexity and diversity comparable to real posts.