Support sharded safetensors size

#63
by tomaarsen HF staff - opened
Massive Text Embedding Benchmark org
edited Jan 16

Hello!

Pull Request overview

  • Add tqdm for external models
  • Add model.safetensors.index.json support

Details

I've added TQDM support for easier progresstracking when building. Additionally, the model.safetensors.index.json now works, allowing model sizes to be captured for e.g. mixedbread-ai/mxbai-mistral-7b-reprev-v1.

  • Tom Aarsen
Massive Text Embedding Benchmark org

Amazing! I tried it here https://huggingface.co/spaces/Muennighoff/tmp but it doesn't show -- does it work for you?

Massive Text Embedding Benchmark org

It did; although the model that inspired this PR has been made private, so it's a bit hard to verify. I don't think there's another model that uses model.safetensors.index.json at this time.

Massive Text Embedding Benchmark org

Okay great, let's merge then!

Muennighoff changed pull request status to open
Muennighoff changed pull request status to merged

Sign up or log in to comment