Can you quantize this model?
#5
by
KatyTheCutie
- opened
teknium/Phi-Hermes-1.3B
https://huggingface.co/teknium/Phi-Hermes-1.3B
Would it be okay to request if you could do this model in GGUF? I'd like to test the creativity compared to other models.
If you can, could you do it in Q8 and Q4?
Thank you.
I would do it myself but my internet is currently throttled to a slow speed.
Well I have quantized the models locally and adapted the code to support it but I'm also at a place where internet is not ideal so haven't been able to upload the quantized files yet. Will try to do so again.
The q4k version should be available now.
Thank you!
KatyTheCutie
changed discussion status to
closed