Post
295
Experimental
eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF is now available.
Got close to achieve a 10% reduction in size but the quality started to deteriorate so this version has been pruned more conservatively. Sizes, on average, are about 8% smaller with only a very small penalty (< 1%) in quality.
After trying with different models and parameter count, I suspect the best I'll be able to do with the current process is between 6 and 8% reduction so have decided to try a different approach. I'll publish process and findings next.
For background: https://huggingface.co/posts/eaddario/832567461491467
Got close to achieve a 10% reduction in size but the quality started to deteriorate so this version has been pruned more conservatively. Sizes, on average, are about 8% smaller with only a very small penalty (< 1%) in quality.
After trying with different models and parameter count, I suspect the best I'll be able to do with the current process is between 6 and 8% reduction so have decided to try a different approach. I'll publish process and findings next.
For background: https://huggingface.co/posts/eaddario/832567461491467