Hugging Face's logo

nvidia
/
Lyra-2.0

Image-to-3D
Lyra-2.0
world
Model card Files
xet
 Community
9

Shards __32_0 through __63_0 (half of model/model/) are 0 bytes — LFS blobs not uploaded?

#6
by Labona - opened
Discussion
Labona

Hi NVIDIA Lyra team — thanks for the release!

I'm setting up inference on a fresh machine and hit a consistent download failure for exactly half of the DCP shards in checkpoints/model/model/. The failure pattern is perfectly contiguous:

Working: __0_0.distcp … __31_0.distcp — each ~2.13–2.14 GB, download fine
Broken: __32_0.distcp … __63_0.distcp — each exactly 0 bytes

Evidence it's server-side (not my network or the huggingface_hub client):

  1. HF API repo_info(files_metadata=True) reports size: 0 for all 32 broken shards and the correct 2.1 GB size for the working ones.
  2. HEAD https://huggingface.co/nvidia/Lyra-2.0/resolve/main/checkpoints/model/model/__0_0.distcp returns content-length: 2139054201 and redirects to cas-bridge.xethub.hf.co/....
  3. HEAD https://huggingface.co/nvidia/Lyra-2.0/resolve/main/checkpoints/model/model/__32_0.distcp returns content-length: 0 with no location: redirect to CAS.
  4. The raw/main/... URL for __32_0.distcp returns a valid LFS pointer with oid sha256:3769d293b42a17b48cb1a1eb24be5ec9aebb74de34799ee9b02fb5706ee6c831, but the pointer's backing LFS blob is not accessible.
  5. hf_hub_download(..., force_download=True) "completes" in ~0.3 s and produces a 0-byte file — the client is downloading the empty server response correctly; there's just nothing there.

It looks like the LFS pointers for all 64 shards were committed, but only the first 32 LFS blobs were actually uploaded to the HF/CAS backend. Could someone re-push the missing blobs for __32_0.distcp through __63_0.distcp?

Happy to provide any additional diagnostic output. Thanks!
frshen

Thanks for the reminder! The checkpoints are correct—those 0-byte files are expected due to DCP allocation, no worries.
Labona

Thanks @frshen , much appreciated! Makes sense, DCP allocation with sparse ranks explains the pattern perfectly. I'll verify end-to-end with torch.distributed.checkpoint on my side and close this out once inference runs clean. Sorry for the false alarm!
Labona changed discussion status to closed

Sign up or log in to comment