Request: oQ4 (MLX) quantization for Apple Silicon?
#1
by 0xtimi2233 - opened
Hi,
Great model! Is anyone planning to convert this model to the oQ4 (MLX) format using omlx?
(Ref: https://github.com/jundot/omlx/blob/main/docs/oQ_Quantization.md)
This format would be perfect for running it locally on Mac.
Thanks!
Hi @bearzi , could you help create a standard oQ4 (and maybe oQ6) quant for this new Hy-MT2-30B model? I really want to run it locally on Mac, but my own hardware just can't handle a 30B calibration.
You've done amazing work with oMLX quants for the community, so it would be awesome if you could look into this one when you have some time. Thanks a lot!
0xtimi2233 changed discussion status to closed