Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError: HFLM.__init__() got an unexpected keyword argument 'use_accelerate'
Root cause: Interface Contract Mismatch

Error with fresh install.

(lmeh) shaunst@c4140-500g-01:~/repo/lm-evaluation-harness$ python main.py --model=hf-causal --model_args="pretrained=/home/shaunst/models/,use_accelerate=True" --tasks=gsm8k --num_fewshot=1 --batch_size=2 --output_path=output
Selected Tasks: ['gsm8k']
Traceback (most recent call last):
  File "/home/shaunst/repo/lm-evaluation-harness/main.py", line 93, in <module>
    main()
  File "/home/shaunst/repo/lm-evaluation-harness/main.py", line 59, in main
    results = evaluator.simple_evaluate(
  File "/home/shaunst/repo/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/home/shaunst/repo/lm-evaluation-harness/lm_eval/evaluator.py", line 72, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/shaunst/repo/lm-evaluation-harness/lm_eval/base.py", line 115, in create_from_arg_string
    return cls(**args, **args2)
TypeError: HFLM.__init__() got an unexpected keyword argument 'use_accelerate'


Please use the big-refactor branch or `hf-causal-experimental` on the main branch.

Hi here! Does that mean that the command mentioned within the About page at https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard (as shown in the screenshot below) is incorrect?

<img width="1624" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/36760800/32c52d49-8a7a-4cd3-bac8-2e0572614333">

Asking in case that needs and edit because maybe HuggingFace is not using that revision but another? Or maybe because they missed the `-experimental` within the `--model` arg?

Thanks in advance @StellaAthena!

> Hi here! Does that mean that the command mentioned within the About page at https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard (as shown in the screenshot below) is incorrect?
> 
> <img alt="image" width="1624" src="https://user-images.githubusercontent.com/36760800/284638622-32c52d49-8a7a-4cd3-bac8-2e0572614333.png">
> Asking in case that needs and edit because maybe HuggingFace is not using that revision but another? Or maybe because they missed the `-experimental` within the `--model` arg?
> 
> Thanks in advance @StellaAthena!

They have an older version of the library pinned, but also I'm currently working with them to get the refactor branch up and running and confirm that nothing has changed.

> They have an older version of the library pinned, but also I'm currently working with them to get the refactor branch up and running and confirm that nothing has changed.

That's great news! I've seen some mismatches between the DROP benchmark from `main` compared to `big-refactor`, currently investigating that, I'll report if applicable; everything else has been great so far from `big-refactor` 🤗 Congrats!

> > They have an older version of the library pinned, but also I'm currently working with them to get the refactor branch up and running and confirm that nothing has changed.
> 
> That's great news! I've seen some mismatches between the DROP benchmark from `main` compared to `big-refactor`, currently investigating that, I'll report if applicable; everything else has been great so far from `big-refactor` 🤗 Congrats!

Interesting. I would love to hear more when you trace it down.

These might be relevant: https://github.com/EleutherAI/lm-evaluation-harness/pull/213 https://github.com/EleutherAI/lm-evaluation-harness/issues/978


> They have an older version of the library pinned, but also I'm currently working with them to get the refactor branch up and running and confirm that nothing has changed.

Thanks @StellaAthena! I'm curious if there has some progress in the front? in particular, is there a simple script to replicate (at least approximately) the open_lm_leaderboard with lm_eval? thanks!