GGUF seemingly broken for L3 in general; EXL2 version to test?
#1
by
Magiwarriorx
- opened
Title. GGUF is apparently broken for Llama 3 in general, and might interfere with testing finetunes. Would it be possible to get an EXL2 quant for testing v2?
@Magiwarriorx I'm uploading 8bpw exl2 right now.
Will be here in a few minutes: https://huggingface.co/JayhC/L3-ChaoticSoliloquy-v2-4x8B-test-8bpw-h8-exl2
@JayhC Doing god's work, thank you so much! Sadly, I still haven't gotten to test it yet; 8bpw is a little too large for my 24GB. Any plans on a <=6.5bpw?