VIBE : Topic-Driven Temporal Adaptation for T witter Classification (2023)

Abstract: Language features are evolving in real-world social media, resulting in the deteriorating performance of text classification in dynamics. To address this challenge, we study temporal adaptation, where models trained on past data are tested in the future. Most prior work focused on continued pretraining or knowledge updating, which may compromise their performance on noisy social media data. To tackle this issue, we reflect feature change via modeling latent topic evolution and propose a novel model, VIBE: Variational Information Bottleneck for Evolutions. Concretely, we first employ two Information Bottleneck (IB) regularizers to distinguish past and future topics. Then, the distinguished topics work as adaptive features via multi-task training with timestamp and class label prediction. In adaptive learning, VIBE utilizes retrieved unlabeled data from online streams created posterior to training data time. Substantial Twitter experiments on three classification tasks show that our model, with only 3% of data, significantly outperforms previous state-of-the-art continued-pretraining methods.

## introduction our ever-changing world leads to the continuous data distribution shifting in social media. as a result, language features, formed by word patterns, will consequently evolve over time. therefore, many text classification models, including the state-of-the-art ones based on pretraining, demonstrate compromised results facing shifted fea-evolving topics over time t + 0: covid, nomask, returntowork, dread t + 1: wearamask, lockdown, immunecompromise t + 2: symptom, panicbuy, imwithfauci, trumpkillus t + 3: therapy, trumpisanationaldisgrace, lyingtrump t + 4: redstates, trumplandslid, trumpliesamericans tures @xcite , when there is a time gap between training and test data. @xcite and @xcite then defined a temporal adaptation task to adapt a model trained on the past data to the shifted future data. to tackle this task, @xcite and @xcite apply continued pretraining on the temporally-updated data to catch the distribution shift yet observe limited enhancement with noisy contexts, meanwhile relying heavily on computational resources. others @xcite @xcite resort to knowledge-based methods, which require high-quality data unavailable on social media. given these concerns, how shall we effectively approach temporal adaptation in noisy social media contexts? inspired by the success of latent topics in many social media tasks @xcite , our solution is to explore latent topic evolution in representing feature change @xcite . to better illustrate our idea, we exemplify the covid-19 stance detection dataset @xcite . here we employ the neural topic model (ntm) @xcite to capture the latent topics in five sequential periods and show the top topic words by likelihoods in figure 1 . as can be seen from the topic evolution, users changed discussion points rapidly over time, where they shifted focus from concerns to the virus itself (e.g., "immune compromise", "symptom") to the disappointment to the former us president trump (e.g., "trump land slid", "lying trump"). from this example, we can learn that topics can help characterize evolving social media environ-ments, which motivates us to propose topic-driven temporal adaptation. specifically, we employ an ntm for featuring latent topics, which is under the control of information bottleneck (ib) regularizers to distinguish past-exclusive, future-exclusive, and time-invariant latent topics to represent evolution. here the time-invariant topic is to encode the task-relevant semantics unchanged over time; the past-and future-exclusive latent topics capture the time-variant topics by modeling the data from the past and future. to further align the topic evolution to a specific classification task, we leverage multitask training to infer timestamps and class labels jointly. our model is then named vibe, short for variational information bottleneck for evolutions. to study how vibe functions on social media data, we utilize the data temporally posterior to training data to learn adaptation (henceforth adaptive data). here we consider two scenarios. first, in an ideal circumstance, the data between training and test time from the original dataset can serve as the golden adaptive data. second, when it is inaccessible in a realistic setup, we retrieve relevant online streams as the retrieved adaptive data. for experiments, we conduct a temporal adaptation setup on three text classification tasks with data from twitter platform and draw the following results. first, the main comparison shows that vibe outperforms the previous state-of-the-art (sota) models with either golden or retrieved adaptive data. for example, vibe brings a relative increase of 5.5% over the baseline for all datasets, using only 3% of the continued pretraining data. then, an ablation study exhibits the positive contribution of vibe's sub-modules, and vibe is insensitive to retrieved adaptive data quality. next, we examine the effects of adaptive data quantity on vibe, and the results show that vibe relies much less on data volume compared to continued pretraining. at last, we present a case study to interpret how vibe learns the evolving topic patterns for adaption. to sum up, we present three-fold contributions: ## preliminaries to prepare readers for understanding vibe's model design (to appear in §4), we present a preliminary section here with the problem formulation in §3.1, information bottleneck principle (vibe's theoretical basis) in §3.2, and the model overview in §3.3. ## vibe model here §4.1 first describes how ntm explores the distribution of past and future tweets. then, we discuss the design of time-oriented ( §4.2) and taskoriented training ( §4.3) to feature the time-invariant and time-variant latent topic variables by ib. next, §4.4 presents how to jointly optimize the ntm and classifier, followed by the application of the topic evolution features for final prediction in §4.5. ## experimental setup and data analysis in the following, we introduce data (including its analysis) and baselines for the experimental setup. data collection. as displayed in ## experimental results in this section, we first discuss the main results ( §6.1). then, an ablation study ( §6.2) discusses the contributions of various vibe modules. next, we quantify vibe's sensitivity to adaptive data scales in §6.3. lastly, a case study ( §6.4) interprets the features learned by vibe in temporal adaptation. ## discussion on application scenarios in this section, we envision more application scenarios where our model vibe can potentially be applied for dynamic language understanding in a temporally evolving environment. in the real world, the rapid changes in language environments challenge language models. previous approaches often resort to continuously updating models to cope with shifted distributions and newly emerging knowledge, resulting in extensive resource consumption. then how about taking advantage of large language models that are highly knowledgeable? we have explored applying a zeroshot large language model (llm) llama @xcite , while exhibiting compromised performance compared with the transformer-based vibe, showing that purely scaled-up language models can not tackle temporal misalignment. moreover, it may be prohibitively expensive to keep updating llm's knowledge via continual training, and the challenge will be exacerbated as models continue to scale up @xcite . inspired by that topic evolution can indicate language evolution in a temporally changing environment @xcite , vibe unveils a lowcost temporal adaptation approach by reflecting on past and future topic statistics simultaneously to infer evolution, which is efficient and practical in the real world. vibe is built upon the general text classification task, and it encodes sentence-level temporal semantics, hence having the potential to generalize over more complex tasks with varying data resources @xcite . many social media applications suffer from temporal misalignment caused by distribution shifts over time. for instance, in a dynamic environment, how to recommend users items that align with their evolving interests @xcite ? furthermore, harmful texts may contain new semantics over time, making non-adapted models fail to detect them. in the context of rumor detection, as rumors are tested by facts over time, learning new facts to reinforce rumor detection is necessary. more language understanding tasks also face the challenge of comprehending changing texts in dynamic language environments, resulting in compromised end-task performance, which is explored by @xcite 's empirical study on the effects of temporal misalignment on social media downstream tasks when there is a time gap between training and test data. we will explore our model vibe in more application scenarios like the aforementioned tasks in the future to enhance performance of language models in the real world. ## conclusion we have presented the first study on the effects of latent topic evolution for temporal adaptation in noisy contexts. a novel model vibe has been proposed leveraging the information bottleneck to capture topic evolution. in experiments, vibe significantly outperforms the continued pretraining methods with much smaller scales of adaptive data.