Improved Near-Duplicate Detection for Aggregated and Paywalled News-Feeds (2025)

Abstract: News aggregators play a key role in the rapidly evolving digital landscape by providing comprehensive and timely news stories aggregated from diverse sources into one feed. As these articles are sourced from different outlets, they often end up covering the same underlying event but differ in phrasing, formatting or supplemented with additional details. It is crucial for the news aggregators to identify these near-duplicates, improving the content quality and user engagement by steering away from redundant information. The problem of near-duplicate news detection has become harder with increasing use of paywalls by the news websites resulting in restricted access to the content. It is now common to get only the headline and a short snippet from the article. Previous works have concentrated on full length versions of documents such as webpages. There is very little work that focuses on this variation of the near-duplicate detection problem in which only headline and a small text blurb is available for each news article. We propose Near-Duplicate Detection Using Metadata Augmented Communities (NDD-MAC) approach that combines embeddings from pretrained language model (PLM) and latent metadata of a news article followed by community detection to identify clusters of near-duplicates. We show the efficacy of proposed approach using 2 different real-world datasets. By integrating metadata with community detection, NDD-MAC is able to detect nuanced similarities and differences in news snippets and offers an industrial scale solution for the near-duplicate detection in scenarios with restricted content availability.

## introduction the digital era has brought both opportunities and challenges to the news industry. the news ecosystem has undergone significant changes, reshaping the way news is produced, distributed and consumed. news aggregator apps and portals have played a significant role in the evolution of the news industry. news aggregators foot_0 provide users with a one-stop platform to access news from various sources, saving time and effort foot_1 in browsing multiple websites or picking up physical newspapers @xcite . one of the key challenges faced by the news aggregators and their subscribers is redundancy due to repetitive content. redundancy problem in news aggregators refers to the issue of users encountering duplicate or highly similar content across multiple articles within the aggregator app, web portal or the news fetched using their apis. it can occur when the aggregators include multiple sources that all cover the same news event or topic. many aggregator apps display content from syndicated foot_2 news services or wire services. these services provide the same articles to multiple news outlets. the news outlets may do a few editorial changes to the input articles. this creates some variations in the content and gives rise to near-duplicates at the news aggregator app or web-portal level. while diversity of sources is valuable, too many similar news items from different sources can undermine the overall quality of the user experience. it affects the news consumers' engagement, retention, and perception of a news aggregator vendor's offerings. this in-turn has a potentially adverse effect on the monetization and the financial viability of the news aggregator app or portal itself. further, news consumers in enterprises typically subscribe to the apis of news aggregator vendors. these enterprises spend valuable compute and storage resources in fetching, archiving and analyzing the news they have paid for. near-duplicate news items not only provide a cluttered user experience for them, but also introduces multiple inefficiencies in the enterprise infrastructure for procuring and disseminating news within their organizations. thus, redundancy due to near-duplicate content affects the overall quality and operational efficiency of news ecosystem. enterprise solutions as well as the research literature for the near-duplicate detection problem have predominantly focused on input consisting of entire documents such as webpages as well as full-length news articles. however, with increasing use of paywalls by the newspaper websites and proliferation of news aggregator apps and apis for large, enterprise-scale news procurement, it is now common to get only the headline and a small snippet of few lines from the article. as shown in in this paper, we propose an unsupervised approach, near-duplicate detection using metadata augmented communities (ndd-mac) to improve the efficiency for news aggregators, enterprise users, as well as the user experience for the end consumers. using this method, we have been able to create an enterprise-wide positive impact by enabling retention and analysis of older news. earlier this data was purged due to infrastructural and process inefficiencies. the improved system now obviates the need for data purging, provides historical continuity and empowers business analysts to observe evolution of events across longer timelines and refine their insights with contextually richer evidence. rest of the paper is organized as follows. in section §2 we describe ndd-mac approach and show how it can be used for the problem of nearduplicate news detection. sections §3 covers the experimental setup and results. in section §4, we briefly describe the related work. finally, we conclude in section §5. comcast and time warner yesterday announced they will make a joint bid for adelphia communications, jumping to the front of the pack in the widely watched auction ## ndd-mac: proposed approach our approach, near-duplicate detection with metadata augmented communities (ndd-mac) is motivated by the observation if a pair of news articles are indeed near-duplicates of each other, then the metadata related to the news content also needs to be matching. we also use sentence-transformers foot_3 based semantically meaningful neural-embeddings as one of the key signals to capture the similarity between a pair of news articles. further, we use a community detection-based graph partitioning technique to identify subsets of articles which are more cohesive within a cluster. figure 1 gives a high-level overview of our proposed approach. we infer and extract the metadata about each news article such as news-type (i.e., type of the key event) described (e.g., product launches, mergeracquisition, awards, financial reporting etc.), (ii) industry domain (e.g., finance, telecom, agriculture, healthcare etc.), (iii) technology (e.g., ai, cloud computing, blockchain, cybersecurity, 5g networking etc.), (iv) types of products, services and organizations based on the content of an article. appendix table a contains a sample of the metadata information extracted from the news articles using specific classifiers for each dimension. we highlight that this metadata can contain information that is not readily mentioned in the surface form of the news content. for instance, the news-type of news-items 1 and 2 in in contrast to the locality sensitivity hashing (lsh) based approaches, the proposed approach does not restrict its focus only on the surface form of the content. to the best of our knowledge, there are no existing methods which are unsupervised and make use this metadata for the near duplicate detection task. we now describe the key steps in the proposed approach in detail. input pre-processing: firstly, the news articles are pre-processed to remove any noisy characters to ensure consistent character encoding and date formatting issues are resolved. the entire news corpus is partitioned into multiple sliding windows from the start date and end date of the input. this helps to ensure that the approach can be adapted even when the resources such as compute power and memory are constrained. then for the cleaned content of each news article snippet within each sliding window is passed through two components and discussed in detail in the following sections. neural embedding computation and preliminary cluster formation: we map an input news snippet (d i ) to a high-dimensional vector embedding c @xmath0. this enables us to get the news snippets with similar meaning closer in the embedding space. this spatial relationship enables the detection of news articles on the same topic and similar content. sentence-bert uses of siamese and triplet neural network to modify the standard pretrained bert network and capture better contextual embeddings compared to prior approaches. we make use of all-mpnet-base-v2 (denoted as mpnet) foot_4 from the sentence-transformers foot_5 1 library. the all-mpnet-base-v2 model transforms input sentences into a 768-dimensional dense vector, providing semantically rich representations. it has achieved the best overall performance across semantic search and sentence embeddings benchmarks. for every pair (d i , d j ) of articles, we check for cosine similarity of their embedding (c i , c j ) and form clusters. each cluster is then represented as a graph in which the news articles are represented as nodes and the edges connecting two articles are initialized with weights as the cosine similarity among the embeddings. ## experimental evaluation datasets: we evaluate our proposed approach using two different datasets: (i) newsaggregator-vendor dataset: a large, private dataset of 34801 real-life news collected from a leading news aggregator using its subscription api, (ii) ndd-ns (kumar et al., 2025): a sample of 1205 news articles extracted from the publicly available agnews dataset foot_6 . as shown in ## related work locality sensitive hashing (lsh) @xcite has been a cornerstone of the techniques used for near-duplicate detection. multiple web search engines have applied lsh variants such as minhash for near duplicate detection and related applications. lsh focuses on the surface form of the content. it uses only the words mentioned in the input text to form the n-grams or shingles while identifying the near-duplicate articles. most recent adaptation of lsh based approach for the problem of near duplicate detection has been proposed by @xcite . their approach uses minhash and is based on shingling proposed by @xcite . shingling technique translates a document into a set of n-grams (i.e., shingles, a contiguous sequence of n words). similarity of two documents can be then measured by computing set similarity. if the similarity is greater than a threshold value, documents are considered as near duplicates of one another. but this practice is costly as the number of shingles generated for a document is too large. to resolve this, they prepare document's sketch (small signatures) using minhash technique proposed by @xcite . using these document sketches they identify near duplicate news articles. in spite of being such a well-known technique, the recent adaptation of lsh based approach proposed by @xcite (minhash-lsh) performs quite poorly on the shortened news data addressed in this paper. in @xcite , authors implement simhash (charikar, 2002) fingerprint technique to identify near duplicate for web documents in an online settings or offline (batch) settings. they propose an algorithmic technique for detecting existing f -bit fingerprints that differ from a given fingerprint in at most m bit-positions, for small m. they have experimentally validated their approach on a corpus of 8b webpages. @xcite employed supervised training to develop their bi-encoder (with mpnet as base) and cross-encoder models, using a dataset consisting of ocr-processed text from newspa-pers published between 1920 and 1977. their approach generates article representations through the bi-encoder model. these representations are then used to construct a graph and identify communities to reduce the computation required for duplicate identification. the cross-encoder model then works with these clusters or communities to identify near-duplicates. we note the reliance of their method on a historical dataset with large number of articles which have significantly more content per article than a news snippet available from the paywalled sources. due to the more recent paywall constraints and evolving nature of news, using their approach may require an updated dataset that reflects the changes in news reporting style. further, one may have to retrain the models using the updated dataset to use their approach. in contrast, the proposed approach only uses off-the-shelf mpnet embeddings to form preliminary clusters. then the edge weights between pairs of news articles in these preliminary clusters are updated based on metadata augmentation. after that we perform community detection on individual clusters. in addition to this difference, we also highlight that the proposed approach does not need to train any supervised model. novo and gedikli (2023) have proposed a supervised learning based approach to identify nearduplicates in which common named entities in a pair of documents are used as the key features. firstly, they assume that if there are no common named entities in a pair of documents they are nonduplicates. then a bert model was fine-tuned to classify whether a given pair of articles are nearduplicates. they have evaluated their approach on a small dataset of 100 business energy news articles. out of the resulting 4950 article pairs in their dataset, only 88 of such pairs are near-duplicates. the pairwise evaluation strategy leads to inconsistent evaluation as transitivity property among the near-duplicate documents gets violated. further, due to the supervised learning approach, they have additional overhead of requiring labeled training data. due to drift in the news topics and changes in the named entities mentioned in news over time, this approach tends require repeated labeling of data to update the supervised learning models. unsupervised methods for near duplicate detection are more realistic given the practical constraints in industrial usage. hence, we focus on unsupervised learning methods such as minhash, simhash etc. as relevant baselines for the near-duplicate detection task. near duplicate detection is an important task not only for news snippets but also it has multiple other applications @xcite , especially where short text snippets are common @xcite . the metadata augmentation idea discussed in this paper can be useful for identifying duplicate questions in technical @xcite as well as non-technical domains @xcite . detecting duplicate defect reports @xcite @xcite is another important application in software maintenance lifecycle. ## conclusion with rise of paywalls on news websites and proliferation of news aggregators, it is now common get only the headline and a small snippet of a news article. this makes the problem of near-duplicate detection more challenging compared to when the full article was readily available. current research has largely overlooked this problem. we introduced near duplicate detection using metadata augmented communities (ndd-mac) to address this issue. unlike the lsh-based approaches, the proposed approach does not rely solely on the surface form of the content or full article availability. it effectively detects near-duplicates using small text excerpts and incorporates multi-dimensional metadata augmentation along with community detection. to the best of our knowledge, the prior work does not use this type of metadata for the near duplicate detection task. evaluation on real-world datasets from a news aggregator and the agnews dataset demonstrates that ndd-mac significantly outperforms established baselines like minhash-lsh, simhash as well as a recent supervised learning based approach.