Repository: EleutherAI/lm-evaluation-harness
Issue: Loading LoRA models in 4bit
Root cause: Validation Gap

I am trying to run the harness with these settings. My base model is the new Falcon model and I've trained a LoRA adapter on top of that.  I am receiving an error when trying to load it. I am a novice so I apologize if this is something very obvious. The environment is  a colab notebook.

```python
model_name = "ybelkada/falcon-7b-sharded-bf16"
repo_name = f"nolestock/{model_name.split('/')[-1]}-finetuned-guanaco-lora"
```

```
!python ./lm-evaluation-harness/main.py \
    --model hf-causal-experimental \
    --model_args pretrained={model_name},peft={repo_name},torch_dtype=torch.bfloat16,trust_remote_code=True,load_in_4bit=True,device_map="auto" \
    --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
    --device cuda:0
```

```
2023-06-13 23:07:47.619930: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Selected Tasks: ['arc_challenge', 'arc_easy', 'boolq', 'hellaswag', 'openbookqa', 'piqa', 'winogrande']
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/./lm-evaluation-harness/main.py:93 in <module>                      │
│                                                                              │
│   90                                                                         │
│   91                                                                         │
│   92 if __name__ == "__main__":                                              │
│ ❱ 93 │   main()                                                              │
│   94                                                                         │
│                                                                              │
│ /content/./lm-evaluation-harness/main.py:59 in main                          │
│                                                                              │
│   56 │   │   with open(args.description_dict_path, "r") as f:                │
│   57 │   │   │   description_dict = json.load(f)                             │
│   58 │                                                                       │
│ ❱ 59 │   results = evaluator.simple_evaluate(                                │
│   60 │   │   model=args.model,                                               │
│   61 │   │   model_args=args.model_args,                                     │
│   62 │   │   tasks=task_names,                                               │
│                                                                              │
│ /content/lm-evaluation-harness/lm_eval/utils.py:243 in _wrapper              │
│                                                                              │
│   240 │   │   │   │   "deprecated and will be disallowed in a future version │
│   241 │   │   │   │   "lm-evaluation-harness!"                               │
│   242 │   │   │   )                                                          │
│ ❱ 243 │   │   return fn(*args, **kwargs)                                     │
│   244 │                                                                      │
│   245 │   return _wrapper                                                    │
│   246                                                                        │
│                                                                              │
│ /content/lm-evaluation-harness/lm_eval/evaluator.py:72 in simple_evaluate    │
│                                                                              │
│    69 │   if isinstance(model, str):                                         │
│    70 │   │   if model_args is None:                                         │
│    71 │   │   │   model_args = ""                                            │
│ ❱  72 │   │   lm = lm_eval.models.get_model(model).create_from_arg_string(   │
│    73 │   │   │   model_args, {"batch_size": batch_size, "max_batch_size": m │
│    74 │   │   )                                                              │
│    75 │   else:                                                              │
│                                                                              │
│ /content/lm-evaluation-harness/lm_eval/base.py:115 in create_from_arg_string │
│                                                                              │
│   112 │   │   additional_config = {} if additional_config is None else addit │
│   113 │   │   args = utils.simple_parse_args_string(arg_string)              │
│   114 │   │   args2 = {k: v for k, v in additional_config.items() if v is no │
│ ❱ 115 │   │   return cls(**args, **args2)                                    │
│   116 │                                                                      │
│   117 │   def set_cache_hook(self, cache_hook):                              │
│   118 │   │   self.cache_hook = cache_hook                                   │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: HuggingFaceAutoLM.__init__() got an unexpected keyword argument 
'torch_dtype'

```

Can you change `torch_dtype` to `dtype`? You can also try merging the model before evaluating it. Check [this](https://github.com/lm-sys/FastChat/blob/main/fastchat/model/apply_lora.py) script.

> Can you change `torch_dtype` to `dtype`? You can also try merging the model before evaluating it. Check [this](https://github.com/lm-sys/FastChat/blob/main/fastchat/model/apply_lora.py) script.

The tests seem to be running now but they look like they'll take quite a bit so I wanted to reply.

Thank you for the tip! I changed it and I also had to change the value to `float16`. I had to remove device map Auto. And finally, needed to install all of the required libraries for falcon (go figure). Here is all the code that I used to get this to run successfully for completeness.

```python

!git clone https://github.com/EleutherAI/lm-evaluation-harness
!cd lm-evaluation-harness && pip install -e ".[auto-gptq]"

!pip install -q datasets bitsandbytes einops git+https://github.com/huggingface/peft

model_name = "ybelkada/falcon-7b-sharded-bf16"
repo_name = f"nolestock/{model_name.split('/')[-1]}-finetuned-guanaco-lora"

!python ./lm-evaluation-harness/main.py \
    --model hf-causal-experimental \
    --model_args pretrained={model_name},peft={repo_name},dtype=float16,trust_remote_code=True,load_in_4bit=True \
    --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
    --device cuda:0
```

Glad you got it to work. Sorry my tip was not complete, I oversaw that you had `torch.bfloat16`. 

I'm not sure about the `load_in_4bit` flag since this will evaluate the quantized model. The QLoRA weights are saved in `bf16` and Falcon weights are probably `bf16` too. Usually `load_in_4bit` is used if you're tight on GPU memory and you're willing to sacrifice performance.

No problem at all your tip was just what I needed to let me poke at the problem and get it rolling. I really appreciate it thank you!

I'm trying to benchmark in QLoRA because I was following this hugging face[ blogpost](https://huggingface.co/blog/falcon) to use falcon. I was interested to see if I could get a fine tuning loop with benchmarks going with as little resources used as possible in colab.

Because the [QLoRA paper ](https://arxiv.org/abs/2305.14314) stated that fine tuning a quantized model 4 in bits with a LoRA adapter could match performance of a fine-tuned unquantized model, I wanted to see if I could match the results and play around with it myself.

I really appreciate that merge link as well. I was just wondering how to do that in the back of my mind. 

Side note:
I have tests running but it's extremely slow and I don't know if that's typical. 
Is 4-bit inference slower than normal or could it be the architecture of my model?
``` 74% 606/817 [3:41:29<1:23:12, 23.66s/it]Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.```


Sure! Yes, this is what QLoRA people mention in their GitHub repo: "4-bit inference is slow. Currently, our 4-bit inference implementation is not yet integrated with the 4-bit matrix multiplication".

Closing as addressed