Temperate for 3-bit quantized model
#16
by
WinstonChen
- opened
I'm using a 3-bit quantized version of this to run an iPhone. What do you suggest for temperature? I heard different thoughts on this. Some believe the temperature needs to be low (<0.2) because of quantization.
I'm using a 3-bit quantized version of this to run an iPhone. What do you suggest for temperature? I heard different thoughts on this. Some believe the temperature needs to be low (<0.2) because of quantization.
I've never used sub-4bit but the same settings worked for all quants Ive used