D 2 CS - Documents Graph Clustering using LLM supervision (2025)

Abstract: Knowledge discovery from large-scale, heterogeneous textual corpora presents a significant challenge. Document clustering offers a practical solution by organizing unstructured texts into coherent groups based on content and thematic similarity. However, clustering does not inherently ensure thematic consistency. Here, we propose a novel framework that constructs a similarity graph over document embeddings and applies iterative graph-based clustering algorithms to partition the corpus into initial clusters. To overcome the limitations of conventional methods in producing semantically consistent clusters, we incorporate iterative feedback from a large language model (LLM) to guide the refinement process. The LLM is used to assess cluster quality and adjust edge weights within the graph, promoting better intra-cluster cohesion and inter-cluster separation. The LLM guidance is based on a set of success Rate metrics that we developed to measure the semantic coherence of clusters. Experimental results on multiple benchmark datasets demonstrate that the iterative process and additional user-supplied a priori edges improve the summaries’ consistency and fluency, highlighting the importance of known connections among the documents. The removal of very rare or very frequent sentences has a mixed effect on the quality scores.Our full code is available here: https://github.com/D2CS-sub/D2CS

## introduction document clustering and summarization are fundamental tasks in natural language processing (nlp) with numerous applications in information retrieval, content organization, and knowledge discovery @xcite . as the volume of digital text continues to grow exponentially, efficient methods for organizing and extracting meaningful insights from large document collections have become increasingly important @xcite . however, text cohorts are often heterogeneous. as such, two parallel tasks should be performed for content extraction: text clustering and multi-text summarization. the first task aims to combine similar texts into clusters, and the second aims to summarize all the texts within each cluster. traditional approaches to document clustering typically rely on conventional distance metrics in high-dimensional vector spaces @xcite , often struggling with the semantic complexity of natural language. for the summarization, abstraction (shortening and condensing) and extractive (extracting the most important sentences) number of methods were proposed @xcite . however, these methods often fail to combine documents that deal with different topics. recent advancements in neural embeddings @xcite @xcite and large language models (llms) @xcite @xcite have opened new possibilities for improving both clustering quality and summary generation. given the capacity of llms, a simple approach for the summarization of heterogeneous cohorts is to first cluster the document and then summarize each cluster. however, clustering methods are not optimized for thematic consistency. here, we present documentto-cluster summary with llms (d2cs), a novel algorithm that leverages the strengths of modern embedding techniques, density-based filtering, energy distance metrics, and community detection algorithms to create semantically coherent document clusters. our approach uniquely combines clustering and the generative capabilities of llms to produce coherent and fluent cluster summaries, which are iteratively refined through a contrastive evaluation framework. the primary contributions of this work are: a) a comprehensive pipeline integrating sentence-level embeddings with kernel density estimation (kde) for feature selection, energy distance for document similarity, and the leiden algorithm for community detection, b)a novel iterative refinement mechanism leveraging llm-generated summaries and contrastive document pair evaluations to improve cluster coherence, and c) a multi-faceted evaluation framework that assesses the quality of the clusters and the semantic relevance of the generated summaries. community detection algorithms have gained prominence in document clustering, with methods like louvain @xcite and @xcite showing superior performance in identifying natural clusters. these approaches model documents as graphs, where similarities define edge weights, and have demonstrated effectiveness in capturing complex relationships @xcite . recent advances by @xcite and @xcite have significantly improved the scalability and interpretability of community detection for large document collections. many measures were defined for the quality of the clusters. however, to the best of our knowledge, no clustering algorithm uses the consistency of texts with their summary as a measure. text summarization as mentioned, text summarization techniques fall broadly into extractive and abstractive categories @xcite . extractive methods @xcite ) select important sentences from documents, while abstractive approaches @xcite generate novel text that captures the essence of the content. a different yet related approach is topic detection, where each sentence/document is assigned a domain, and then each domain is summarized by itself by combining either documents belonging to the same domain, or all sentences from multiple documents belonging to the same domain @xcite . notable recent contributions include @xcite , who developed a pretraining approach specifically for multi-document summarization. the advent of llms has transformed summarization capabilities @xcite , with few-shot and zero-shot approaches demonstrating remarkable efficacy. recent work by @xcite has shown that guided prompting strategies can substantially improve llm-based summarization quality, while @xcite have explored the use of llms as judges for evaluating summarization outputs. however, to the best of our knowledge, there is no feedback approach to improve the clustering based on the summary evaluation. ## methods d2cs leverages an energy distance metric applied to document representations to construct a graph. the weights of the graph's edges are iteratively refined using supervision from a large language model (llm), with the goal of enhancing cluster coherence and forming homogeneous subgroups of semantically similar documents. ## results evaluation of d2cs on multiple datasets d2cs clusters documents and then summarizes each cluster. specifically, d2cs first projects each sentence to an embedding space and defines each text/abstract as a bag of projected sentences (the order is ignored). then, it creates a graph based on the energy distance between bags and clusters the bags using the leiden algorithm @xcite . each cluster is then summarized using an llm (different llms can be chosen). to test that d2cs can produce a coherent summary, we analyzed multiple datasets (as you can see in the appendix beyond that 4 measures on the quality of the abstract were proposed in section 4.4. we first tested 8 sets of 100 abstracts each with different keywords ( iterative clustering improves the scores the vanilla flavor above reaches a high (but not always perfect) rs score, and intermediate scores in the fluency and consistency of text scores. these scores can be iteratively improved by strengthening the edges between texts that are consistent with their common summary and reducing the weights of non-consistent edges. similarly, the summary can be enhanced by adding refinement queries to improve the text quality. we tested the effect of such queries, and they indeed improve the scores on average(unless they are already close to 1 -figure 2c ). we stopped the iterations when the clusters stopped changing. the iterative process creates a clique of highly consistent documents, and the clustering typically converges after 2 iterations. interesintgly, in some cases, d2cs can detect "junk" texts (e.g., advertisements, placeholder strings, etc.) and clusters them together, purifying the rest of the summaries. ## discussion we have here presented an llm-guided graph refinement framework that addresses key challenges in document clustering by enhancing semantic coherence through iterative feedback. d2cs combines computational efficiency with the semantic understanding capabilities of llms @xcite . the combination of clustering and summarizing requires appropriate success rate metrics. we used state-of-the-art metrics and added a clustering coherence metric to ensure, on the one hand, good summarization and, on the other hand, coherence between the summary and the clustering. while the approach is based on text analysis, we found that user-defined connections significantly guide the clustering process in domain-specific applications. fully automated methods often struggle to incorporate domain expertise @xcite . this semi-supervised approach aligns with recent work @xcite showing that minimal human guidance enhances language model performance on complex tasks. removing very frequent sentences and very rare sentences improved cluster quality in some types of problems, but not in all. a possible reason for that may be that for specialized corpora, rare sentences contain crucial domain-specific terminology @xcite . to summarize, d2cs represents a significant advancement in knowledge discovery from heterogeneous textual corpora. the iterative refinement process, guided by llm feedback and augmented with user-supplied knowledge, effectively overcomes limitations of conventional clustering methods and reaches an approximately 100 % coherence in most datasets. beyond d2cs, we propose a composite evaluation approach that ensures all aspects of document clustering and summarizing are ensured.