Repository: EleutherAI/lm-evaluation-harness
Issue: [BUG] Fail to evaluate `gptqmodel`; `TypeError: LlamaForCausalLM.__init__() got an unexpected keyword argument 'gptqmodel'`
Root cause: Validation Gap

**Describe the bug**

Hi, I want to evaluate a gptqmodel as follows, but there are some error. 
```bash
lm_eval --model hf \
    --model_args pretrained=ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v2.5,gptqmodel=True \
    --tasks hellaswag
```
The error:
```
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/eval/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 204, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/api/model.py", line 147, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/huggingface.py", line 182, in __init__
    self._create_model(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/huggingface.py", line 572, in _create_model
    self._model = self.AUTO_MODEL_CLASS.from_pretrained(
  File "/home/ubuntu/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/ubuntu/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4130, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: LlamaForCausalLM.__init__() got an unexpected keyword argument 'gptqmodel'
```


**GPU Info**
```
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10G                    On  |   00000000:00:1B.0 Off |                    0 |
|  0%   18C    P8             16W /  300W |       4MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A10G                    On  |   00000000:00:1C.0 Off |                    0 |
|  0%   18C    P8             16W /  300W |       4MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A10G                    On  |   00000000:00:1D.0 Off |                    0 |
|  0%   19C    P8             16W /  300W |       4MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
|  0%   19C    P8             16W /  300W |       4MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
```

**Software Info**

Operation System/Version + Python Version
```
Name: lm_eval
Version: 0.4.4
Summary: A framework for evaluating language models
Home-page: https://github.com/EleutherAI/lm-evaluation-harness
Author: 
Author-email: EleutherAI <contact@eleuther.ai>
License: MIT
Location: /home/ubuntu/miniconda3/envs/eval/lib/python3.10/site-packages
Editable project location: /home/ubuntu/lm-evaluation-harness
Requires: accelerate, datasets, dill, evaluate, jsonlines, more-itertools, numexpr, peft, pybind11, pytablewriter, rouge-score, sacrebleu, scikit-learn, sqlitedict, torch, tqdm-multiprocess, transformers, word2number, zstandard
Required-by: 
---
Name: gptqmodel
Version: 1.4.6.dev0
Summary: A LLM quantization package with user-friendly apis. Based on GPTQ algorithm.
Home-page: https://github.com/ModelCloud/GPTQModel
Author: ModelCloud
Author-email: qubitium@modelcloud.ai
License: 
Location: /home/ubuntu/miniconda3/envs/eval/lib/python3.10/site-packages
Requires: accelerate, datasets, device-smi, numpy, packaging, protobuf, safetensors, sentencepiece, setuptools, threadpoolctl, torch, transformers
Required-by: 
---
Name: torch
Version: 2.4.0
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: /home/ubuntu/miniconda3/envs/eval/lib/python3.10/site-packages
Requires: filelock, fsspec, jinja2, networkx, nvidia-cublas-cu12, nvidia-cuda-cupti-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12, nvidia-cudnn-cu12, nvidia-cufft-cu12, nvidia-curand-cu12, nvidia-cusolver-cu12, nvidia-cusparse-cu12, nvidia-nccl-cu12, nvidia-nvtx-cu12, sympy, triton, typing-extensions
Required-by: accelerate, auto_gptq, autoawq, bitsandbytes, exllamav2, gptqmodel, lm_eval, optimum, peft, torchvision, vllm, xformers
---
Name: transformers
Version: 4.47.1
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: transformers@huggingface.co
License: Apache 2.0 License
Location: /home/ubuntu/miniconda3/envs/eval/lib/python3.10/site-packages
Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm
Required-by: auto_gptq, autoawq, evalplus, gptqmodel, lm_eval, optimum, optimum-benchmark, peft, stop-sequencer, vllm
---
Name: accelerate
Version: 1.2.1
Summary: Accelerate
Home-page: https://github.com/huggingface/accelerate
Author: The HuggingFace team
Author-email: zach.mueller@huggingface.co
License: Apache
Location: /home/ubuntu/miniconda3/envs/eval/lib/python3.10/site-packages
Requires: huggingface-hub, numpy, packaging, psutil, pyyaml, safetensors, torch
Required-by: auto_gptq, autoawq, gptqmodel, lm_eval, optimum-benchmark, peft
---
Name: triton
Version: 3.0.0
Summary: A language and compiler for custom Deep Learning operations
Home-page: https://github.com/triton-lang/triton/
Author: Philippe Tillet
Author-email: phil@openai.com
License: 
Location: /home/ubuntu/miniconda3/envs/eval/lib/python3.10/site-packages
Requires: filelock
Required-by: autoawq, torch
```

**If you are reporting an inference bug of a post-quantized model, please post the content of `config.json` and `quantize_config.json`.**

**To Reproduce**

Just run the command I mentioned above. 

The problem is solved, thanks! 

May I ask how the problem was solved?

> May I ask how the problem was solved?

I just updated the version of both `lm-eval-harness` and `gptqmodel` to the latest. 

Could you please tell me what versions they are respectively? I attempted to upgrade today, but still encountered this error. Currently, gptqmodel=1.7.4, lm_eval=0.4.7, transformers=4.47.1, torch=torch 2.1.0a0+32f93b1, python=3.10.12. 

Still getting an error message:
Traceback (most recent call last):
  File "/host_yjq/LLMSuperWeight/my_evaluate.py", line 5, in <module>
    cli_evaluate()
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 401, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 204, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/model.py", line 147, in create_from_arg_string
    return cls(**args, **args2)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/huggingface.py", line 185, in __init__
    self._create_model(
  File "/host_yjq/LLMSuperWeight/outliers/model.py", line 182, in _create_model
    self._model = self.AUTO_MODEL_CLASS.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4130, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: LlamaForCausalLM.__init__() got an unexpected keyword argument 'gptqmodel'

> Could you please tell me what versions they are respectively? I attempted to upgrade today, but still encountered this error. Currently, gptqmodel=1.7.4, lm_eval=0.4.7, transformers=4.47.1, torch=torch 2.1.0a0+32f93b1, python=3.10.12.
> 
> Still getting an error message: Traceback (most recent call last): File "/host_yjq/LLMSuperWeight/my_evaluate.py", line 5, in cli_evaluate() File "/usr/local/lib/python3.10/dist-packages/lm_eval/**main**.py", line 382, in cli_evaluate results = evaluator.simple_evaluate( File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 401, in _wrapper return fn(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 204, in simple_evaluate lm = lm_eval.api.registry.get_model(model).create_from_arg_string( File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/model.py", line 147, in create_from_arg_string return cls(**args, **args2) File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/huggingface.py", line 185, in **init** self._create_model( File "/host_yjq/LLMSuperWeight/outliers/model.py", line 182, in _create_model self._model = self.AUTO_MODEL_CLASS.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained return model_class.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4130, in from_pretrained model = cls(config, *model_args, **model_kwargs) TypeError: LlamaForCausalLM.**init**() got an unexpected keyword argument 'gptqmodel'

Unfortunately, my old environment is not available now. I suggest creating a new conda environment and pip installing the packages from scratch. 

OK. Thank you.