Repository: huggingface/lighteval
Issue: quantized model not loading
Root cause: Validation Gap

I aim to evaluate a 8-bit quantized model. for some reason lighteval asks me to provide data for quantization:

ValueError: You need to pass `dataset` in order to quantize your model

I started with this:
 accelerate launch run_evals_accelerate.py --model_args="pretrained=/teamspace/studios/this_studio/models/Meta-Llama-3-8B-LoRA,dtype=auto" --tasks "leaderboard|mmlu:astronomy|0|0" --output_dir ./scratch


Interesting, could you provide the full stacktrace?

This is the complete output:

Could not initialize the JudgeOpenAI model:
The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
Could not initialize the JudgeOpenAI model:
The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
INFO:absl:Using default tokenizer.
INFO:absl:Using default tokenizer.
INFO:absl:Using default tokenizer.
INFO:absl:Using default tokenizer.
INFO:absl:Using default tokenizer.
WARNING:lighteval.logging.hierarchical_logger:main: (0, Namespace(model_config_path=None, model_args='pretrained=/teamspace/studios/this_studio/models/Meta-Llama-3-8B-LoRA,dtype=auto', max_samples=None, override_batch_size=-1, job_id='', output_dir='./scratch', push_results_to_hub=False, save_details=False, push_details_to_hub=False, public_run=False, cache_dir=None, results_org=None, use_chat_template=False, system_prompt=None, dataset_loading_processes=1, custom_tasks=None, tasks='leaderboard|mmlu:astronomy|0|0', num_fewshot_seeds=1)),  {
WARNING:lighteval.logging.hierarchical_logger:  Test all gather {
WARNING:lighteval.logging.hierarchical_logger:    Test gather tensor
WARNING:lighteval.logging.hierarchical_logger:    gathered_tensor tensor([0], device='cuda:0'), should be [0]
WARNING:lighteval.logging.hierarchical_logger:  } [0:00:00.000630]
WARNING:lighteval.logging.hierarchical_logger:  Creating model configuration {
WARNING:lighteval.logging.hierarchical_logger:  } [0:00:00.000013]
WARNING:lighteval.logging.hierarchical_logger:  Model loading {
WARNING:lighteval.logging.hierarchical_logger:    `quantization_config` is None but was found in the model's config, using the one found in config.json
WARNING:lighteval.logging.hierarchical_logger:  } [0:00:00.000896]
WARNING:lighteval.logging.hierarchical_logger:} [0:00:00.002690]
Traceback (most recent call last):
  File "/teamspace/studios/this_studio/lighteval/run_evals_accelerate.py", line 82, in <module>
    main(args)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lighteval/logging/hierarchical_logger.py", line 166, in wrapper
    return fn(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lighteval/main_accelerate.py", line 77, in main
    model, model_info = load_model(config=model_config, env_config=env_config)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lighteval/models/model_loader.py", line 83, in load_model
    return load_model_with_accelerate_or_default(config=config, env_config=env_config)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lighteval/models/model_loader.py", line 125, in load_model_with_accelerate_or_default
    model = BaseModel(config=config, env_config=env_config)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lighteval/models/base_model.py", line 73, in __init__
    self._config = config.init_configs(env_config)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lighteval/models/model_config.py", line 158, in init_configs
    return self._init_configs(self.pretrained, env_config=env_config)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lighteval/models/model_config.py", line 153, in _init_configs
    self.quantization_config = GPTQConfig(**auto_config.quantization_config, disable_exllama=True)
TypeError: GPTQConfig.__init__() missing 1 required positional argument: 'bits'
Traceback (most recent call last):
  File "/home/zeus/miniconda3/envs/cloudspace/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 46, in main
    args.func(args)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1082, in launch_command
    simple_launcher(args)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/accelerate/commands/launch.py", line 688, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/zeus/miniconda3/envs/cloudspace/bin/python', 'run_evals_accelerate.py', '--model_args=pretrained=/teamspace/studios/this_studio/models/Meta-Llama-3-8B-LoRA,dtype=auto', '--tasks', 'leaderboard|mmlu:astronomy|0|0', '--output_dir', './scratch']' returned non-zero exit status 1.

The error is here `TypeError: GPTQConfig.init() missing 1 required positional argument: 'bits'`.
Can you check that your GPTQConfig is correct?

My understanding was that since the model is stored with 8bit (bitsandbytes) I do not have a GPTQConfig (?) Why would lighteval ask for a GPTQConfig?

The config.json contains this:

"quantization_config": {
    "_load_in_4bit": false,
    "_load_in_8bit": true,
    "bnb_4bit_compute_dtype": "float32",
    "bnb_4bit_quant_storage": "uint8",
    "bnb_4bit_quant_type": "fp4",
    "bnb_4bit_use_double_quant": false,
    "llm_int8_enable_fp32_cpu_offload": false,
    "llm_int8_has_fp16_weight": false,
    "llm_int8_skip_modules": null,
    "llm_int8_threshold": 6.0,
    "load_in_4bit": false,
    "load_in_8bit": true,
    "quant_method": "bitsandbytes"
  }


You are entirely correct, a bits and bytes model should not need a GPTQ config - this feels like a bug when creating quantized models, I'll investigate. Thanks for all the infos!

Not sure why really it is needed, but it helped me to set 

"pretrained=/teamspace/studios/this_studio/models/Meta-Llama-3-8B-LoRA,quantization_config="load_in_8bit=True""