Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research (2024)

Abstract: Research in natural language processing (NLP) for Computational Social Science (CSS) heavily relies on data from social media platforms. This data plays a crucial role in the development of models for analysing socio-linguistic phenomena within online communities. In this work, we conduct an in-depth examination of 20 datasets extensively used in NLP for CSS to comprehensively examine data quality. Our analysis reveals that social media datasets exhibit varying levels of data duplication. Consequently, this gives rise to challenges like label inconsistencies and data leakage, compromising the reliability of models. Our findings also suggest that data duplication has an impact on the current claims of state-of-the-art performance, potentially leading to an overestimation of model effectiveness in real-world scenarios. Finally, we propose new protocols and best practices for improving dataset development from social media data and its usage.

## introduction research in natural language processing (nlp) for computational social science (css) aims to analyze social behavior and sociolinguistic phenomena with computational methods on a large scale @xcite . this relies upon using vast amounts of user-generated content in social media such as twitter/x and weibo @xcite . social media data exhibits distinctive characteristics such as rapid and continual topic evolution @xcite . prior research has focused on introducing and developing novel resources including new tasks and datasets @xcite @xcite . social media posts often contain a significant amount of near-duplicate or even identical content. for example, during social emergencies (e.g., covid-19 pandemic), users tend to repeatedly post about the same topics (e.g., covid-19 vaccination), resulting in a proliferation of near-duplicate content over a very short period of time @xcite . furthermore, the presence of social bots (e.g., automatically generated posts from third-party applications) amplifies the rapid generation of near-duplicate content on these platforms, e.g., in political campaigns @xcite . data quality is paramount for reliable and effective model performance. it includes various dimensions such as accuracy, completeness and consistency @xcite . ensuring high data quality usually involves careful data management and processing such as data validation and cleaning @xcite @xcite . previous works have examined and deduplication of samples in image datasets @xcite , generic real-word datasets (e.g., movie, restaurant) @xcite and language modeling datasets (e.g., wikipedia) @xcite . however, the quality of existing css datasets in nlp has been under-explored. the goal of this paper is to conduct a large scale meta-analysis of current nlp for css datasets by considering the potential noise caused by duplicated text samples. to this end, we perform an indepth re-examination of 20 social media datasets across various tasks such as offensive language detection and misinformation detection. 1 our main contributions are as follows: tweet_id samples label consequence 129839* @user @user donald trump's lessons for republicans: consequences for lying... neutral 129840* @user @user donald trump's lessons for republicans: consequences for lying... against inconsistent labeling 132605* @user fight against tyranny!!! the coronavirus vaccine is a new technology rna vaccine. it literally changes your dna!!!! [emojis] anti. vaxx 132607* @user fight against tyranny!!! the coronavirus vaccine is a new technology rna vaccine. it literally changes your dna!!!! [emojis] anti. vaxx label leakage 131120* (dupli.) † @user trump neutral label leakage 407156* r.i.p to the driver who died with paul walker that no one cares about because he wasn't famous omg:( true rumor 407164* r.i.p to the driver that died with paul walker that no one cares about because he wasn't famous. ## tasks & datasets we evaluate a comprehensive collection of datasets that adequately cover a broad range of prevalent tasks in css. based on recent publications in nlp and css venues (e.g., *acl and icwsm), we choose datasets from four main css tasks following previous work @xcite @xcite ## examining dataset quality we focus on examining duplicate or near-duplicate samples in datasets to assess data quality. ## impact of duplicate and near-duplicate samples we conduct comparative experiments regarding three potential issues (data leakage, model rank-ings and inconsistent labels) on the selected social media datasets to verify the impact of those duplicate and near-duplicate posts on classification performance. ## effective strategies for development and use of social media datasets this section outlines practical recommendations we propose for more effective development and use of datasets in responsible css research. ## conclusion beyond focusing on innovative methodologies to improve model performance in downstream css tasks, we advocate a more comprehensive understanding of the datasets via meta analysis. in this work, we conduct an extensive evaluation of the data duplication issue in 20 selected social media datasets. we find that the majority of these datasets require additional preprocessing before the modeling, a critical step often overlooked in previous css research. furthermore, we explore the potential impact of data duplication on model performance with a focus on label leakage, model rankings and label inconsistency. finally, we offer targeted recommendations for more effective use of existing datasets and the creation of new datasets derived from social media sources.