7
Repo Graph
👁
A byte-level map of the Hugging Face Hub
Jared Sulzdorf PRO
jsulz
AI & ML interests
Infrastructure, law, policy
Recent Activity
updated
a collection
about 4 hours ago
Papers (I want) To Read
liked
a Space
6 days ago
xet-team/repo-graph
posted
an
update
6 days ago
As https://huggingface.co/xet-team infrastructure begins backing hundreds of repositories on the Hugging Face Hub, we’re getting to put on our researcher hats and peer into the bytes. 👀 🤓
IMO, one of the most interesting ideas Xet storage introduces is a globally shared store of data.
When you upload a file through Xet, the contents are split into ~64KB chunks and deduplicated, but what if those same chunks already exist in another repo on the Hub?
If we can detect and reuse them, we skip them as well saving time and bandwidth for AI builders. More on how that works here:
🔗 https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation
Because of this, different repositories can share bytes we store. That opens up something cool - we can draw a graph of which repos actually share data at the chunk level, where:
- Nodes = repositories
- Edges = shared chunks
- Edge thickness = how much they overlap
https://huggingface.co/spaces/xet-team/repo-graph
Come find the many BERT islands. Or see how datasets relate in practice, not just in theory. See how libraries or tasks can tie repositories together. You can play around with node size using storage/likes/downloads too.
The result is a super fun visualization from @saba9 and @znation that I’ve already lost way too much time to. I'm excited to see how the networks grow as we add more repositories!
Organizations
jsulz's activity
liked
a
Space
6 days ago
liked
a
model
6 days ago
liked
2
models
12 days ago
liked
a
model
18 days ago
liked
a
Space
26 days ago
liked
a
dataset
26 days ago
liked
a
Space
27 days ago
liked
a
Space
28 days ago
liked
a
dataset
about 2 months ago
liked
a
Space
about 2 months ago
liked
2
models
about 2 months ago
liked
3
Spaces
2 months ago
liked
2
Spaces
4 months ago
liked
2
Spaces
5 months ago