On Fake News Detection with LLM Enhanced Semantics Mining (2024)

Abstract: Large language models (LLMs) have emerged as valuable tools for enhancing textual features in various text-related tasks. Despite their superiority in capturing the lexical semantics between tokens for text analysis, our preliminary study on two popular LLMs, i.e., ChatGPT and Llama2, showcases that simply applying the news embeddings from LLMs is ineffective for fake news detection. Such embeddings only encapsulate the language styles between tokens. Meanwhile, the high-level semantics among named entities and topics, which reveal the deviating patterns of fake news, have been ignored. Therefore, we propose a topic model together with a set of specially designed prompts to extract topics and real entities from LLMs and model the relations among news, entities, and topics as a heterogeneous graph to facilitate investigating news semantics. We then propose a Generalized Page-Rank model and a consistent learning criteria for mining the local and global semantics centered on each news piece through the adaptive propagation of features across the graph. Our model shows superior performance on five benchmark datasets over seven baseline methods and the efficacy of the key ingredients has been thoroughly validated.

## introduction the ubiquity of fake news on social media poses a significant threat to public discourse and societal well-being @xcite @xcite . to alleviate the far-reaching consequences, many fake news detection methods probe the information dissemination process or social structure @xcite to detect fake news. unfortunately, despite the impressive detection performance, their applicability is substantially constrained when the social context is unavailable or incomplete due to the * . work done at macquarie university. †. contributed equally evolving nature of social networks and data privacy concerns @xcite . facing limited access to social context, other text-mining methods @xcite investigate the intricacies of news content to uncover hierarchical textual semantics (e.g., sentence and document level semantics) and formulate fake news detection as a classification problem, using only textual content from the social media. following the latter approach, in which news embeddings are critical for providing a discriminatory description of authentic and fake news, we are propelled to enhance them with large language models (llms), which have been renowned for their remarkable capabilities in language understanding, and context modeling @xcite @xcite . a fundamental question that guides our research in this under-explored realm is, "are the llms output news embeddings effective for fake news detection?" to answer this question, we conducted a preliminary study by comparing the detection performance of an mlp classifier trained using news embeddings extracted from gpt-3.5 1 , llama2 2 , @xcite and het-erosgt @xcite , respectively. from the results depicted in fig. 2 ( and p1. how can we apply llms to explore highlevel news semantics? from the above study, we affirm that the exploration of high-level semantics enables the model to acquire a better perception of deeper contextual nuances, which encompass fabricated knowledge among entities with real meaning on a particular topic @xcite , for distinguishing fake news. we identify the keys for high-level semantics exploration using llms are to extract meaningful entities and topics. p2. how can we identify the irregular semantics in fake news? given the llm-derived entities and topics, one can aggregate their features to enhance the centered news embeddings for fake news detection. but this primarily focuses on the information within individual news pieces (local semantics), lacking the ability to explicitly explore the broader range of knowledge across news pieces (global semantics) to identify narrative inconsistencies and manipulations in fake news. for example, in detecting news #2 as fake, we identify the relation between 'covid-19' and 'genetically modified method source of features semantics unlabeled social context news text other sources local global data han textgcn dualemo comments usdefake propagation network hgnnr knowledge graph heterosgt less4fd (ours) cr to address p1, by prompting llms for entity extraction, we first propose a refined topic model that summarizes news topics through llm-generated embeddings. we then construct a heterogeneous graph to model the relationships among news, entities, and topics by representing them as nodes and connecting them with edges, which facilitates further exploration of local and global news semantics. for p2, we apply short-and long-scale feature propagation centered on news nodes to encapsulate the local and global semantics into news representations. with these two scales of feature propagation, we can identify inconsistencies between each individual news text and the broader knowledge across news, and involve unlabeled news for training with our specially designed consistency training criterion. our major contributions are: ## conclusion in this paper, we propose less4fd to take advantage of llms for enhancing semantics mining for fake news detection. we first employ llms as the enhancers to extract news, entities, topics, and their corresponding features using a set of potent prompts. by modeling the extracted data as a heterogeneous graph, we then propose an effective feature propagation algorithm to encode both the local and global semantics into news embeddings to enrich the training of the detector. through extensive experiments on five widely-used datasets, our method demonstrates better performance than seven baseline methods while the efficacy of key ingredients is further validated in the case studies. limitations. in this work, we only adopt the two most popular llms as enhancers to explore the news semantics. extending our method to tuning llms, particularly for fake news detection is an important direction for future efforts. ethical issues. the datasets utilized in our research for detecting fake news are widely accessed and publicly available for academic research. our proposed method exclusively relies on the textual content of news articles from these datasets as input, without requiring any additional user-specific information (e.g., personal identifiers) or user social information (e.g., retweet/comment behavior). we employed publicly accessible apis provided by openai and meta to obtain embeddings. our prompts, which are made publicly available, are used exclusively for extracting entities and topics from llms. therefore, our method ensures minimal risk of privacy infringement. applications. detecting fake news is critical due to its significant implications for society, politics, and individual decision-making. our proposed model demonstrates efficacy in distinguishing authentic and false content, which could contribute to mitigate the spread of false information and public distrust.