Platform-Invariant Topic Modeling via Contrastive Learning to Mitigate Platform-Induced Bias (2024)

Abstract: Cross-platform topic dissemination is one of the research subjects that delved into media analysis; sometimes it fails to grasp the authentic topics due to platform-induced biases, which may be caused by aggregating documents from multiple platforms and running them on an existing topic model. This work deals with the impact of unique platform characteristics on the performance of topic models and proposes a new approach to enhance the effectiveness of topic modeling. The data utilized in this study consisted of a total of 1.5 million posts collected using the keyword ”ChatGPT” on the three social media platforms. The devised model reduces platform influence in topic models by developing a platform-invariant contrastive learning algorithm and removing platform-specific jargon word sets. The proposed approach was thoroughly validated through quantitative and qualitative experiments alongside standard and state-of-the-art topic models and showed its supremacy. This method can mitigate biases arising from platform influences when modeling topics from texts collected across various platforms.

## introduction topic modeling is a well-known method in the realm of natural language processing (nlp) to identify coherent topics within a text corpus. topic modeling is used in a variety of areas, including trends and major events on social media @xcite . in the realm of media analysis, researchers often analyze cross-platform topic (or information) dissemination to understand general trends on social media. their general approach is to aggregate posts, i.e., documents, from multiple platforms rather than analyzing each platform individually @xcite , thereby being exposed to platform-induced bias. in that sense, it is beneficial to devise an integrated topic modeling when investigating the influence of media platforms. there are two ways to extract topics from documents: counting word frequency to determine topics or clustering words with similar meanings. these methods correspond to the bag of words (hereafter, bow) approach and the contextual approach, respectively. algorithms based on bow assume that the words within a document follow a dirichlet distribution and extract the topic distribution within the document @xcite @xcite . latent dirichlet allocation (lda) is one of the representative models @xcite . there is also a large body of research on topic modeling methodologies based on lda @xcite @xcite . conversely, context-based topic modeling algorithms cluster elements with similar contexts before producing topics @xcite . bertopic, which uses the pre-trained model bert, is an example of contextbased topic modeling @xcite . with posts collected from multiple social media platforms, topic modeling algorithms can be an efficient tool to analyze and understand the predominant topics of discussion @xcite @xcite . however, when conducting topic modeling on various internet social media platforms, the unique characteristics of each platform can influence the topic modeling algorithms. these platform-specific characteristics could be differences in writing styles, i.e., semantic traits, within posts on the platform, or they could be inherent features, i.e., syntactic traits, of the platform's functionality. regarding semantic traits, for example, x (i.e., twitter) focuses on quick communication through short messages, reddit is more suited for writing explanatory posts, while youtube is characterized by comments on videos @xcite . due to these differences, topics extracted in topic modeling can vary inconsistently across platforms. regarding syntactic traits, on the other hand, platform-specific jargon (hereafter referred to as jargon), such as 'rt' and 'retweet' predominantly used on x, can influence the results of topic modeling algorithms. if such jargon is included in the process, algorithms might be biased toward extracting topics specific to a particular platform. this study first aims to answer the following questions through preliminary studies: based on these insights, we introduce platform-invariant topic modeling (pitopic), designed to nullify the influence of platform-specific characteristics. by removing jargon and applying a new platform-invariant contrastive learning algorithm, pitopic ensures that the extracted topics are universally coherent, irrespective of the platform. experiments with both real-world and synthetic datasets with multiple platforms (or sources) confirm our model's superiority over contemporary baselines in both topic diversity and coherence. the code and implementation details for the model can be found in a github repository. 1 2 related works there are largely three streams of topic modeling: bow-based methods, clustering methods, and methods that combine these two methods. lda @xcite is a typical bow-based algorithm that assumes a dirichlet before the topic distribution. it follows a probabilistic latent semantic analysis (plsa) assumption that all word tokens in a document are sampled from a mixture of latent topics @xcite . bow-based algorithms do not consider the order of words in sentences when extracting topics from documents. this leads to a limitation in topic modeling with platform data: the frequency of simultaneous word occurrences can be low and influenced by words predominantly used on a specific platform. to address these limitations of lda, autoencoding variational bayes (aevb) can be applied to topic modeling @xcite . the avitm method trains inference networks that map approximate posterior distributions directly from documents. however, this method relies on the quality of the training data, making it difficult to learn the exact posterior distribution in the presence of noise, such as individual characteristics of platform data. moreover, both methods cannot capture the contextual information of sentences in long text documents. bertopic effectively embeds sentences using the pre-trained model bert @xcite and extracts important topic words from documents through the clustering and c-tf-idf algorithm @xcite . yet, this clustering-based method has limitations in representing situations where multiple topics can coexist simultaneously. recently, the contextualized topic model (ctm) was introduced to apply bert @xcite for extracting the embedding vector and reconstructing bow for each sentence @xcite with variational autoencoder (vae) @xcite . since the model still uses bow, it can be influenced by words predominantly used on a single platform, and unique platform characteristics in bert embeddings can yield biased topic modeling results.recent research combines the advantages of both bow-based and clustering methods for topic modeling. this approach uses contrastive learning and term weighting with topic word extraction to effectively integrate these benefits @xcite . when multiple platforms exist, however, the extraction of topic words can become ambiguous due to platform characteristics. in this research, we propose a method to overcome the weakness that existing topic modeling algorithms rely on the influence of platforms and effectively extract topics by utilizing various platform data. this research seeks to reduce the influence of platforms on topic modeling and ensure the performance and consistency of topic modeling algorithms. ## preliminary studies we here provide preliminary studies to answer our research questions: (rq1) whether different platforms have their unique characteristics, and (rq2) these traits differently influence the performance of topic modeling algorithms. ## platform-invariant topic modeling problem formulation: let p @xmath0, each containing a collection of documents x j . given an input document @xmath1, the primary goal of our topic model is to estimate and assign a topic from the predefined set of k topics. from the model, we extract the top-m words representing each topic to identify the theme of the topic and measure its classification quality. we introduce pitopic, a neural topic model that extracts coherent topics from documents collected across various platforms while avoiding bias towards any specific platform. figure 2 illustrates our concept. our model employs a traditional plsa-based approach @xcite , estimating the topic distribution from the text and then reconstructing the bow distribution of the original input from the estimated topic distribution. to eliminate bias introduced by platform grouping, we propose two components. first, during the process of estimating the topic distribution, we use contrastive learning to optimize the conditional mutual information of two similar text pairs concerning the platform, preventing the topic from being predominantly drawn from a single platform (section 4.1). second, in reconstructing the bow distribution from the topic distribution, we minimize platform-dependent jargon in composing the bow, thereby removing bias (section 4.2). these two methods enable our model to handle biases from both the unique textual styles and different word usage patterns inherent in each platform. each component is outlined in the following section. ## conclusion this paper investigates the influence of platform bias on existing topic modeling algorithms by collecting text corpora from multiple platforms sharing the same keywords, "chatgpt." to address the platform bias, we introduced pitopic, which incorporates platform-invariant contrastive learning and a novel jargon-filtering process. figure 3 : clustering results over four topic models. colors represent the three ground-truth class names (i.e., 20newsgroup, bbc, and the new york times), and dots depict the document cluster assignments. as a result, our model extracted high-quality topics that are coherent across all platforms while being less affected by platform-specific biases, demonstrating superior topic coherence and diversity compared to other baselines. we believe our work enables more accurate topic discovery without bias, contributing to a variety of social analyses.