YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Semantic Message Clustering & Visualization
This project demonstrates how to convert message text into semantic embeddings, cluster them into meaningful groups, and visualize them using UMAP and HDBSCAN.
The result is a clean, interactive, visualization of your dataset.
I plan to use something like this for my cluster data dashboard project.
Features
- Embeds text using Hugging Face Sentence Transformers
- Projects embeddings to 2D using UMAP
- Finds natural clusters with HDBSCAN
- Automatically labels clusters using TF-IDF keyword extraction
- Draws cluster boundaries using convex hulls
- Interactive Plotly visualization with hover data
- Loads message data from
input.json
What I did
1. Embedding Text
We use the Hugging Face embedding model: sentence-transformers/all-MiniLM-L6-v2
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeddings = model.encode(texts)
2. UMAP
Embeddings are high-dimensional (usually 384 dimensions), so we reduce them to 2D for visualization.
3. HDBSCAN
HDBSCAN finds clusters automatically based on density, without needing to pick a fixed number of clusters.
4. Automatic Cluster Labels
We use TF-IDF keyword extraction to summarize each cluster’s dominant topics.
5. Interactive Visualization
We render everything using Plotly:
- Points colored by cluster
- Optional “noise” cluster
- Convex hull shading around cluster regions
- Hover tooltips showing text, user, and channel
- Clean layout with hidden axes This creates a highly readable “semantic map” of your dataset.