fireworks-ai

/

llama-3-firefunction-v2

like 107

@MaziyarPanahi Would love GPTQ or exl2 quants too! I am getting AttributeError: 'LlamaCppModel' object has no attribute 'model' errors with this I think because of the tokenizer not being found.

I'll do my best for the GPTQ. For the Llama models, you need the latest Llama.cpp to make it work :)

Thank you! I made an exl2 quant, but I still haven't found a way to do gptq quants on 4x3090s. Last thing I tried was an AutoGPTQ example file but that seems to make the quant but give an error at saving time.