BF16 GGUF to hopefully generate lossless, or as close to that for now, Llama-3 model quantizations avoiding the recent talked about issues on that topic, it is more resource intensive and will generate more writes in the drive as there's a whole additional conversion step that isn't performed in the previous version. This should only be necessary until we have GPU support for BF16 to run directly without conversion. 

 

 

Pull Requests with your own features and improvements to this script are always welcome. 

 

ized GGUF version of [Felladrin/Llama-160M-Chat-v1](https://huggingface.co/Felladrin/Llama-160M-Chat-v1). 

 

GGUF version of [Josephgflowers/Tinyllama-616M-Cinder](https://huggingface.co/Josephgflowers/Tinyllama-616M-Cinder). 

GGUF version of [Qwen/Qwen1.5-0.5B-Chat](https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat). 

rded GGUF version of [anakin87/gemma-2b-orpo](https://huggingface.co/anakin87/gemma-2b-orpo). 

 

GGUF version of [anakin87/gemma-2b-orpo](https://huggingface.co/anakin87/gemma-2b-orpo). 

 

ivqn/GGUF-metamath-llemma",  

model_file="metamath-llemma.gguf",  

model_type="llama", gpu_layers=0, 

context_length=768) 

model('''AI will ''', temperature=0.1) 

ivqn/GGUF-openchat",  

model_file="openchat.gguf",  

model_type="llama", gpu_layers=0, 

context_length=768) 

model('''AI will ''', temperature=0.1) 

GGUF version of [Felladrin/TinyMistral-248M-SFT-v4](https://huggingface.co/Felladrin/TinyMistral-248M-SFT-v4). 

 

ized GGUF version of [JackFram/llama-160m](https://huggingface.co/JackFram/llama-160m). 

 

a 7B GGUF VERSION here is the orignal [TokenBender/Navarna_v0_1_OpenHermes_Hindi](https://huggingface.co/TokenBender/Navarna_v0_1_OpenHermes_Hindi)