Stuck When Running Python Inference as Described in Model Card?

#4
by fffiend - opened

I basically get stuck on :
Fetching 1 files: 100%|█████████████████████████████████| 1/1 [00:00<00:00, 1.55it/s]

And then nothing happens. I'm running a Macbook Pro with an M2 chip and 8GB RAM.

Resolved! It was because I didn't have enough CPU mem available for it to generate a completion :D

fffiend changed discussion status to closed
rustformers org

@fffiend Yeah, 8GB RAM isn't that much, i would recommend to run the q4_0 versions of the models, those should fit. Or maybe try different models like redpajama-3B which should fit more easily into your available RAM.

Sign up or log in to comment