Spaces:
Running
Running
Gemma-2-27b-it giving nonsense answers
#493
by
cmcmaster
- opened
You'll probably have to trust me on this, but the responses in this chat are just completely absurd (https://hf.co/chat/r/VrLqa1f). I notice on the GH repo that the temperature has been turned down form 0.6 to 0.1 and certainly on my local tests these models really go off the rails with temps > 0.3. But there's clearly something else going on. Even the 8bit MLX quant of the 9B model gives vastly superior answers to the huggingchat version of the 27B model. Something is very off.
Compared with temp, top_p has a greater impact for the quality - I usually set top_p=0.3 temp=0.1