No space left on device when running ZeroGpu

#132
by xiaozaa - opened

I created a new space https://huggingface.co/spaces/xiaozaa/catvton-flux-try-on
It used the ZeroGPU. It's loading the flux1-dev-fill pipeline. Trying to figure out how much space does the container have? I will came across the

ZeroGPU tensors packing: 0%| | 0.00/35.1G [00:00<?, ?B/s]
ZeroGPU tensors packing: 0%| | 0.00/35.1G [00:00<?, ?B/s]
Traceback (most recent call last):
File "/home/user/app/app.py", line 225, in
demo.launch()
File "/usr/local/lib/python3.10/site-packages/spaces/zero/gradio.py", line 142, in launch
task(*task_args, **task_kwargs)
File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 348, in pack
_pack(Config.zerogpu_offload_dir)
File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 340, in _pack
pack = pack_tensors(originals, fakes, offload_dir, callback=update)
File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/packing.py", line 114, in pack_tensors
os.posix_fallocate(fd, 0, total_asize)
OSError: [Errno 28] No space left on device

Issue when packing the tensor.

save problem met

Solved this problem by adding

subprocess.run("rm -rf /data-nvme/zerogpu-offload/*", env={}, shell=True)

xiaozaa changed discussion status to closed

Sign up or log in to comment