Repository: EleutherAI/lm-evaluation-harness
Issue: Same result on GPTQ 8bit and 4bit model, normal?
Root cause: Configuration Error

Hi team, I have been trying to evaluate how the LLM performances were downgraded in different quantization setups. Thus, I chose the Llama-7b-chat model as a reference. Runs evaluation on 
`
1. Full resolution (fp16)
2. GPTQ-8bit (int8)
3. GPTQ-4bin(int4)
`
The score dropped significantly from FP16 to INT8 in HellaSwag and mmlu. However, there was nothing different from INT8 to INT4. If anyone facing the same issue or has any questions about this, please let me know! Thanks.

That's not expected, no. Could you try looking at the actual generations and tell us if they're identical between the two models?

if so, maybe adding some print statements will reveal if the model is just not being loaded in int4

Hi Stella, thanks for your reply. This question may sound silly, but how can I print out the model's response for each evaluation? I'm using the big-refactor branch and running the evaluation with the command: `python -m lm_eval --model hf --model_args pretrained="/root/model/LLMs/llama2/Llama-2-7b-chat-hf" --tasks hellaswag,mmlu --device cuda:4 --batch_size 8`. Any command line flags to control it?

@Chrisz236 you can control this via logging to a directory with `--log_samples` enabled, e.g. `--output_path ./results_folder --log_samples`. 

I would also suggest testing on `lambada_openai` to see if the perplexities reported are different between 4bit and 8bit. If the perplexities differ, it may be that going from 8bit to 4bit changes loglikelihoods, but simply not enough to affect the highest-probability answer in most/all cases for your model.

> @Chrisz236 you can control this via logging to a directory with `--log_samples` enabled, e.g. `--output_path ./results_folder --log_samples`.
> 
> I would also suggest testing on `lambada_openai` to see if the perplexities reported are different between 4bit and 8bit. If the perplexities differ, it may be that going from 8bit to 4bit changes loglikelihoods, but simply not enough to affect the highest-probability answer in most/all cases for your model.

Thanks for your reply, I am trying now

> @Chrisz236 you can control this via logging to a directory with `--log_samples` enabled, e.g. `--output_path ./results_folder --log_samples`.
> 
> I would also suggest testing on `lambada_openai` to see if the perplexities reported are different between 4bit and 8bit. If the perplexities differ, it may be that going from 8bit to 4bit changes loglikelihoods, but simply not enough to affect the highest-probability answer in most/all cases for your model.

Would you suggest use `gptq_model-8bit-128g.safetensors` or keep it as `pytorch_model.bin` for quantized model?

> Would you suggest use gptq_model-8bit-128g.safetensors or keep it as pytorch_model.bin for quantized model?

I am not especially familiar with GPTQ--I would suggest trying either (or preferably both if this choice of file makes a functional difference?) as if either does not work that would be important to know.

> > Would you suggest use gptq_model-8bit-128g.safetensors or keep it as pytorch_model.bin for quantized model?
> 
> I am not especially familiar with GPTQ--I would suggest trying either (or preferably both if this choice of file makes a functional difference?) as if either does not work that would be important to know.

Alright, now trying on `safetensor` format

> > @Chrisz236 you can control this via logging to a directory with `--log_samples` enabled, e.g. `--output_path ./results_folder --log_samples`.
> > I would also suggest testing on `lambada_openai` to see if the perplexities reported are different between 4bit and 8bit. If the perplexities differ, it may be that going from 8bit to 4bit changes loglikelihoods, but simply not enough to affect the highest-probability answer in most/all cases for your model.
> 
> Would you suggest use `gptq_model-8bit-128g.safetensors` or keep it as `pytorch_model.bin` for quantized model?

Hi Hailey, it's just like magic but this time it works :). I have the test result as expected. Thank you!