Generative Deduplication For Socia Media Data Selection (2024)

Abstract: Social media data exhibits severe redundancy caused by its noisy nature. It leads to increased training time and model bias in its processing. To address this issue, we propose a novel Generative Deduplication framework for social media data selection by removing semantically duplicate data. While related work involves data selection in the task-specific training, our model functions as an efficient pre-processing method to universally enhance social media NLP pipelines. Specifically, we train a generative model via self-supervised learning to predict keyword to capture the semantics of noisy social media text for deduplication. Meanwhile, time-dimensional Gaussian noise is added to improve training complexity and avoid learning trivial features. Extensive experiments suggest that our model can better reduce training samples while improving performance than baselines. The results show our model’s potential to broadly advance social media language understanding in effectiveness and efficiency.

## introduction social media is an abundant resource with vast realtime user-generated content, providing valuable insights into the world and society. it has benefited various applications, such as stance detection @xcite and content recommendation @xcite , taken advantage of cutting-edge nlp practices. however, a common challenge nlp models may face is the severe redundancy of social media data @xcite caused by its noisy nature @xcite . here, we define redundancy as semantically similar content that leads to information overload and model bias. the redundant data not only increases the training cost of a model (in time and resources) but also results in the redundancy bias adversely affecting its performance @xcite . to illustrate this point, we show some tweeteval examples in figure 1 . the duplicated tweets containing "pumpkin" and "neutral" sentiment labels bias the model to connect "pumpkin" to "neutral wrongly" sentiment, meanwhile rendering unnecessary training costs. to tackle the redundancy problem, we explore the solution from data selection @xcite @xcite . the goal is to select a subset of relevant data from a larger dataset to benefit the model training. it has profoundly affected the performance of large language model pretraining @xcite and its downstream task finetuning @xcite . in data selection, most work focuses on how to find useful data via augmentation or retrieval for "data addition" @xcite @xcite . in contrast, others center on "data deduplication" to remove semantically duplicated data and show its positive effects in pretraining @xcite . our work aligns with the "deduplication" line. previous work has shown its help in social media nlp, such as shingles @xcite and simhash @xcite . however, these efforts rely on surface linguistic features rather than semantics. while textual semantic similarity models @xcite @xcite can be easily applied to the deduplication, the pairwise comparison would render high complexity. moreover, our work differs from existing efforts @xcite to engage data selection in task-specific training. instead, we aim to provide a pre-processing method for social media data, which can be easily plugged into various nlp pipelines. to that end, we propose a novel semantic deduplication approach, generative deduplication, for social media data selection. specifically, we adopt a generative model as the generative backbone and train it with a self-supervised task to generate a keyword from the input text. here, we train the generative backbone for only one epoch. duplicate text undergoes multiple optimizations, enabling more accurate keyword prediction than non-duplicate text with a single optimization. moreover, we improve the training difficulty with time-dimensional gaussian noise (tgn) to prevent trivial feature learning in one epoch, limiting keyword prediction for non-duplicates. hence, we can consider samples with correct keyword prediction as duplications and remove them. this way, we allow a computational complexity of o(n), where n is the data size, and avoid pairwise comparison. to the best of our knowledge, we are the first to propose generative deduplication for social media data selection and study its broad impact on downstream social media language understanding. in experiments, the deduplication experiment indicates that our model enables the best data quality with less training time. then, the results on the tweeteval benchmark show that our selected data allows performance gains with much shorter training time on varying downstream models and tasks. for example, for llama on sentiment analysis, our model reduces the training set (50.9%) and training time (42.9%) yet improves the macro recall from 73.0 to 73.5. next, the ablation study shows the positive contributions of varying modules. the general short text classification experiments suggest that our method also benefits general scenarios. lastly, we interpret our model's superiority with more analyses. in summary, our contributions are as follows: ## related work the proposed generative deduplication is in line with data selection. it is an essential technique for selecting helpful data to benefit the training of downstream tasks. many previous studies center on domain adaptation @xcite @xcite , where they selected data that aligns with the target distribution from vast data, aiming to improve performance in specific domains. more recently, deep learning techniques @xcite have been used for better data selection. given recent language model advances, some work, such as @xcite , explored retrieval and augmentation to "select and add" relevant data from external resources for taskspecific training. however, prior work observed these practices may result in a significant amount of duplicate data, which introduces redundancy @xcite and adversely affects performance @xcite . to mitigate this redundancy issue, deduplication @xcite can be applied to shortlist duplicate data to allow more effective and efficient training. our work aims to adopt deduplication to address redundancy bias @xcite in social media data. however, existing deduplication methods @xcite @xcite mainly rely on surface linguistic features, unable to handle semantic-level duplication prevalent in noisy social media data. meanwhile, directly applying models based on textual semantic similarity @xcite @xcite ) may involve pairwise comparison and result in high deduplication cost. in addition, existing methods engage data selection in end-to-end task-specific training @xcite . in contrast, our work focuses on data selection for pre-processing, which can be seamlessly integrated into various nlp pipelines. ## generative deduplication this section will elaborate on the proposed generative deduplication with an overall framework in figure 2 . we will first introduce the problem formulation in section 3.1. then, we describe the generative training in section 3.2, followed by the inference as the deduplication stage in section 3.3. in the self-supervised training stage, the model encodes input text and learns to generate the corresponding keyword. the time-dimensional gaussian noise (tgn) will be applied to increase training difficulties and avoid trivial feature learning in the training stage. in the inference as deduplication stage, text with correct keyword prediction is identified as a duplicate. 2) the right part depicts the downstream applications. first, the training set is deduplicated using gd. then, the deduplicated training set is used for training and inference. by doing so, it can reduce training samples and resource consumption while improving accuracy. ## experimental setup datasets. for the deduplication experiment, we use the mrpc dataset @xcite , where texts are labeled as either equivalent or nonequivalent based on semantic duplication. for social media language understanding, we evaluate our model on 7 widely used twitter datasets, including 6 tasks from tweeteval @xcite : emoji predication, hate speech detection, offensive language identification, sentiment analysis, stance detection, and irony detection, and humor detection from semeval-2021 task 7 @xcite . the statistics of adopted datasets are listed in it is more challenging (with 20 labels) and larger than other datasets, making it ideal for evaluating the performance of the proposed model. evaluation metrics. for deduplication, we report the f1 score for equivalent (duplicate text), marked as f1 eq . we also report the deduplication time in seconds. for social media language understanding, we follow prior works @xcite to report the same evaluation metrics from the original tasks. specifically, we report macro f1 for emoji, hate, offensive, and humor, report micro-recall for sentiment, report the average of the f1 of against and favor (marked as f1 a+f /2) for stance, and report the f1 of the ironic label (marked as f1 i ) for irony. baselines and comparisons. for the deduplication experiment, we compare the proposed generative deduplication with two widely used deduplication approaches: shingles @xcite and simhash @xcite . we also compare it with pairwise semantic deduplication using pretrained sentence embeddings @xcite . for social media language understanding, we adopt three popular backbones: roberta @xcite , bertweet (nguyen et al., 2020), and llama @xcite and compare with them using deduplicated data. ## conclusion in this paper, we have introduced a novel preprocessing method called generative deduplication for social media data selection. it tackles the semantic redundancy bias in noisy social media data. extensive experiments have suggested that generative deduplication can significantly reduce the training cost of a model (in time and resources) while improving social media language understanding.