Perplexity appears high versus casper hansen version
#3
by
RonanMcGovern
- opened
Through testing out Nicolas Carlini's benchmark, this model is scoring 17% versus 24% for casper hansen's quant. I'm not sure why there's a difference as the group size seems the same...
hi
@RonanMcGovern
Thanks for testing out ! I suspect I used an old version of autoawq code that didn't included a fix for RoPE theta .. :/ I think yes you should use the casperhansen version of the model for better results