Garreth Lee's picture

Garreth Lee PRO

garrethlee

·

AI & ML interests

None yet

Recent Activity

Reacted to jsulz's post with 🔥 3 days ago

When the XetHub crew joined Hugging Face this fall, @erinys and I started brainstorming how to share our work to replace Git LFS on the Hub. Uploading and downloading large models and datasets takes precious time. That’s where our chunk-based approach comes in. Instead of versioning files (like Git and Git LFS), we version variable-sized chunks of data. For the Hugging Face community, this means: ⏩ Only upload the chunks that changed. 🚀 Download just the updates, not the whole file. 🧠 We store your file as deduplicated chunks In our benchmarks, we found that using CDC to store iterative model and dataset version led to transfer speedups of ~2x, but this isn’t just a performance boost. It’s a rethinking of how we manage models and datasets on the Hub. We're planning on our new storage backend to the Hub in early 2025 - check out our blog to dive deeper, and let us know: how could this improve your workflows? https://huggingface.co/blog/from-files-to-chunks

liked a Space 3 days ago

xet-team/lfs-analysis

New activity 3 days ago

garrethlee/comprehensive-arithmetic-problems-carries:Convert dataset to Parquet

View all activity

Organizations

garrethlee's activity

New activity in garrethlee/comprehensive-arithmetic-problems-carries 3 days ago

Convert dataset to Parquet

#5 opened 3 days ago by

Convert dataset to Parquet

#1 opened 3 days ago by

Convert dataset to Parquet

#2 opened 3 days ago by

Convert dataset to Parquet

#3 opened 3 days ago by

Convert dataset to Parquet

#4 opened 3 days ago by

New activity in garrethlee/comprehensive-arithmetic-problems 5 days ago

Convert dataset to Parquet

#2 opened 5 days ago by

Convert dataset to Parquet

#1 opened 5 days ago by

New activity in indolem/IndoMMLU 6 days ago

Add subsets based on level

#4 opened 6 days ago by

Convert dataset to Parquet

#3 opened 6 days ago by

New activity in garrethlee/simple-arithmetic-problems about 1 month ago

Convert dataset to Parquet

#1 opened about 1 month ago by

New activity in deepmind/math_dataset about 1 month ago

Convert dataset to Parquet

#5 opened about 1 month ago by

Convert dataset to Parquet

#4 opened about 1 month ago by

Convert dataset to Parquet

#5 opened about 1 month ago by

Convert dataset to Parquet

#4 opened about 1 month ago by