LLMT axo: Leveraging Large Language Models for Constructing Taxonomy of Factual Claims from Social Media (2025)

Abstract: With the rapid expansion of content on social media platforms, analyzing and comprehending online discourse has become increasingly complex. This paper introduces LLMTaxo, a novel framework leveraging large language models for the automated construction of taxonomies of factual claims from social media by generating topics at multiple levels of granularity. The resulting hierarchical structure significantly reduces redundancy and improves information accessibility. We also propose dedicated taxonomy evaluation metrics to enable comprehensive assessment. Evaluations conducted on three diverse datasets demonstrate LLMTaxo’s effectiveness in producing clear, coherent, and comprehensive taxonomies. Among the evaluated models, GPT-4o mini consistently outperforms others across most metrics. The framework’s flexibility and low reliance on manual intervention underscore its potential for broad applicability.

## introduction misinformation, also known as false or misleading information @xcite , has the potential to sway public perception, cause confusion, and influence people's decision-making processes @xcite . social media platforms, in particular, facilitate the rapid sharing of vast amounts of content, blending accurate information with falsehoods @xcite . social media's global reach and ease of use have transformed how millions of users exchange opinions, news, and factual claims in real-time, making it fertile ground for misinformation @xcite . factual claims, which are assertions that can be verified as either true or false, are a common vehicle for misinformation @xcite . these claims, whether accurate or not, have a profound societal impact, as the public tends to believe a factual claim is true regardless of its truthfulness @xcite @xcite @xcite . the dynamic nature of social media often leads to repetitive or reformulated claims, complicating the identification and validation of factual content @xcite , making it difficult for non-technical individuals and researchers from different fields to navigate the vast amounts of data @xcite @xcite . this challenge calls for scalable and automated tools to systematically organize and analyze factual claims, helping stakeholders-such as researchers and fact-checkers-more effectively navigate the complex information landscape. one promising approach to addressing this challenge is the use of taxonomies, which provide structured systems for organizing complex information. taxonomy has been applied in misinformation categorization. for example, @xcite classified fake news into distinct categories to capture the diverse forms and manifestations of the phenomenon. @xcite constructed taxonomies of false information on social media based on various characteristics, such as whether the content is intended to deceive and whether it is opinion-based or fact-based. @xcite proposed a taxonomy of misinformation on social media based on falsity level and evidence type. yet, these approaches do not construct topic taxonomies, which are particularly valuable for exploring and organizing factual claims. in practice, many fact-checking websites, such as politifact, 1 snopes, foot_1 and full fact, 3 use topics to organize fact checks. for example, the claim "right to try experimental drug program saved thousands and thousands of lives" in politifact is given topics such as "elections" and "health care." however, these topics are too broad to help users efficiently locate specific claims. motivated by the challenges, prior studies, and practical applications discussed above, this paper introduces llmtaxo, a novel framework utilizing large language models (llms) for automatically constructing a topic taxonomy given a collection of factual claims from social media within a specific topic domain. the taxonomy starts with broad categories and branches into increasingly specific subcategories. in this way, factual claims can be categorized into hierarchical categories, allowing for the efficient exploration of information at multiple levels of granularity. for example, broad categories can group covid-19 vaccine-related claims into overarching topics such as "public health," while more detailed categories can address specific claims about "vaccine safety and effectiveness." our framework, llmtaxo, organizes factual claims into a coherent hierarchical taxonomy by clustering semantically similar claims, identifying distinct ones, and generating topics at multiple levels of granularity-broad, medium, and detailed. leveraging the rich background knowledge of llms and few-shot learning, llmtaxo minimizes human involvement and automates the categorization process. it effectively addresses challenges such as semantic variability (the expression of similar ideas in diverse ways) and ensures scalability and adaptability to social media discourse across various datasets and topic domains. to demonstrate the generalizability of llmtaxo, we conducted evaluations using carefully designed metrics across three distinct datasets spanning different topic domains and data sources. these include social media posts from x (formerly twitter) and facebook, covering domains such as covid-19 vaccines, climate change, and cybersecurity. the evaluation results, showing all models scoring above 3.0 out of 5 across all metrics, with a highest score of 4.9, demonstrate the effectiveness of llmtaxo in generating clear and comprehensive taxonomies. by identifying distinct factual claims, the framework substantially reduces redundancy. the hierarchical structure of the taxonomy produced by llmtaxo allows users to explore claims at multiple levels of detail. moreover, llmtaxo exhibits consistently strong performance across diverse datasets, demonstrating its adaptability and potential for wide-ranging applications. in summary, this paper makes several key contributions: constructing taxonomies from broader, less formatted content, such as social media posts, differs from traditional taxonomy construction. the inherent variability of such content makes it challenging to establish a precise taxonomy. several studies have attempted to address this problem. @xcite explored automatic taxonomy generation from disaster-related tweets using topic modeling techniques @xcite . najem and hadi (2021) proposed semi-automatic ontology construction of tweets based on semantic feature extraction using wordnet and babel-net @xcite . the rise of llms has significantly advanced many natural language processing tasks, but limited effort has been devoted to taxonomy construction. @xcite employed pretrained language models to construct taxonomic trees, while @xcite compared prompting and finetuning approaches for hypernym taxonomy con-struction. additionally, @xcite and @xcite introduced end-to-end pipelines that integrate llms to generate, refine, and apply labels for user intent analysis in log data. our work extends this line of research by applying llms to generate multi-level taxonomies for factual claims and social media posts. like prior efforts, we harness llms' generative and semantic capabilities, but we focus on taxonomy construction in opendomain, noisy, and high-variance contexts where entity boundaries and hierarchical relationships are less clear. topic generation. llms have emerged as a promising alternative to traditional topic generation approaches. they can generate topics from a given set of documents without requiring predefined labels or training data @xcite . this capability allows for more flexible and context-aware topic extraction. for instance, @xcite introduced a framework that prompts llms to generate topics and adhere to human guidelines for refining and merging topics. recent research has also explored different llm-based topic modeling techniques. for example, bertopic (grootendorst, 2022) has shown superior performance in terms of diversity and coherence across multiple datasets @xcite . yet, to the best of our knowledge, no existing work has generated topics at multiple levels of granularity. ## methodology analyzing social media data presents numerous challenges, including the overwhelming volume of content with high repetition and the substantial human effort required to explore the data. our llmtaxo framework is designed to systematize online content through automated construction of a taxonomy of factual claims. this taxonomy serves as a hierarchical classification system. it enhances the accessibility and navigability of information for users by categorizing claims into broad, medium, and detailed topics. llmtaxo initially identifies factual claims from social media posts, and subsequently clusters similar claims to discern and select distinct ones, thereby reducing redundancy. it then leverages llms to generate topics for each distinct claim at multiple levels of granularity, ultimately constructing a structured taxonomy. the overview of our framework is shown in figure 1 . ## experiments 4.1 datasets to evaluate our method, we conducted experiments on three social media datasets, each covering a specific topic: covid-19 vaccine, climate change, and cybersecurity, denoted as cv, cc and cs, respectively. these datasets were collected from two social media platforms to ensure diversity in the content and structural characteristics of the posts. ## evaluation we assessed llmtaxo from two perspectives: 1) the quality of the generated taxonomy, and 2) the relevance of the generated topics to the factual claims. we developed a set of evaluation metrics to assess these two aspects. we engaged both human evaluators and gpt-4 @xcite for the evaluation. the effectiveness of using llms for model performance evaluation has been validated by previous studies @xcite . identical instructions and metrics were provided to both human and gpt-4 evaluators. they were instructed to rate each criterion on a scale from 1 (strongly disagree) to 5 (strongly agree), where 5 indicates the best quality and 1 the worst. to evaluate the quality of taxonomies, we presented taxonomies generated by zephyr, gpt-4o mini, and gemini across the three datasets to the evaluators, resulting in a total of 9 taxonomies. for the evaluation of the claim-topic suitability, we randomly selected 102 factual claims along with their corresponding lowest-level topics (i.e., leaf nodes of the taxonomy) generated by the three models from each dataset, resulting a total of 306 claimtopic pairs for evaluation. we exclusively evaluated the leaf node topics with claims, as the broader topics had already been assessed in taxonomy quality evaluation. note that not every claim has threelevel topics, as mentioned in section 3.1. therefore, we focused our evaluation on the leaf node topics. to mitigate bias, we shuffled claim-topic pairs from different models before presenting them to the evaluators. both human evaluators and gpt-4 reviewed these pairs using our predefined metrics. human evaluators were required to provide rationales for the scores. the evaluation prompt for gpt-4 is detailed in appendix b. ## conclusion in this study, we introduced llmtaxo, a novel framework that leverages llms to construct taxonomies of factual claims from social media. through evaluations across three distinct datasets, our approach demonstrated effectiveness in organizing claims into hierarchical structures with broad, medium, and detailed topics. the results highlight the framework's potential in reducing redundancy, improving information accessibility, and assisting users such as researchers and factcheckers in navigating online factual claims.