Repository: EleutherAI/lm-evaluation-harness
Issue: I get this error whenever I try to run an eval: ImportError: cannot import name 'HfApi' from 'huggingface_hub'
Root cause: External Dependency Breakage

Hello!

Within the past week, I've been getting this error on Google Colab whenever I try to run any evals:
 
```
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/content/lm-evaluation-harness/lm_eval/__main__.py", line 3, in <module>
    import logging
  File "/content/lm-evaluation-harness/lm_eval/logging/__init__.py", line 1, in <module>
    from .evaluation_tracker import EvaluationTracker
  File "/content/lm-evaluation-harness/lm_eval/logging/evaluation_tracker.py", line 8, in <module>
    from huggingface_hub import HfApi
ImportError: cannot import name 'HfApi' from 'huggingface_hub' (/usr/local/lib/python3.10/dist-packages/huggingface_hub/__init__.py)
```

I note the following:
- It is probably related to this PR, as the error occurs at the changed line, and no issues occurred for me back around April 15th: https://github.com/EleutherAI/lm-evaluation-harness/pull/1766#issuecomment-2093190929
- As implied, it seems to be an error for importing HfApi from huggingface_hub (the Colab is running 0.23.0)
- I have tried the following fixes: installing huggingface_hub, uninstalling and reinstalling huggingface_hub, logging into huggingface cli, troubleshooting by running models via hf and vllm etc.
- Simply removing the line `from huggingface_hub import HfApi` leads to a different error: 

```
File "/content/lm-evaluation-harness/lm_eval/evaluator.py", line 10, in <module>
    import torch
  File "/usr/local/lib/python3.10/dist-packages/torch/__init__.py", line 27, in <module>
    from ._utils_internal import get_file_path, prepare_multiprocessing_environment, \
  File "/usr/local/lib/python3.10/dist-packages/torch/_utils_internal.py", line 8, in <module>
    log = logging.getLogger(__name__)
AttributeError: partially initialized module 'logging' has no attribute 'getLogger' (most likely due to a circular import)
```

I genuinely have no idea what's causing this, because literally no one else has reported a problem with HfApi. I have no idea how to make it usable after about 5-6 hours of debugging attempts.

I may try again tomorrow or Tuesday, removing as much of my setup as possible. For replication on colab: https://colab.research.google.com/drive/1URpXmPqIQWttwBZANo1Eud3_jWkQYC6e

cc @KonradSzafer , I'll look into this ASAP if he doesn't have a fix though

(perhaps it's a version issue with huggingface_hub ?)

@haileyschoelkopf if you ctrl+f "huggingface_hub" or "0.23.0" on the Colab you'll find that I was running 0.23.0, which does seem to be a new version updated May 2nd.

It's really hard for me to pinpoint what broke, but I tried a few times simply running the cells for installing the harness + running `!pip install lm-eval[vllm]` +running model args on colab and that didn't work. I'll have another swing at stripped-down runs today or tomorrow, but for now it really is a complete mystery why that broke.

And even stranger, why I'd be the only one to have reported that error.

Hello @menhguin! It seems I can't access your Colab. Could you update the sharing settings to "Anyone with the link"? Also, could you run the following command and share the output?
```bash
pip show huggingface-hub
```

@KonradSzafer can you access it now?

Output:

```
Name: huggingface-hub
Version: 0.23.0
Summary: Client library to download and publish models, datasets and other repos on the huggingface.co hub
Home-page: https://github.com/huggingface/huggingface_hub
Author: Hugging Face, Inc.
Author-email:  _[his email]_
License: Apache
Location: /usr/local/lib/python3.10/dist-packages
Requires: filelock, fsspec, packaging, pyyaml, requests, tqdm, typing-extensions
Required-by: accelerate, datasets, evaluate, peft, tokenizers, transformers
```

Thanks, I can access it now. Will look later today or tomorrow at the latest.

Hi ! I was able to make it work by running:

```
!pip install lm_eval[wandb,vllm] # skip if you don't want to use wandb to log results
!huggingface-cli login

!lm_eval \
    --model vllm \
    --model_args pretrained=mistralai/Mistral-7B-v0.1,dtype=auto \
    --batch_size "auto" \
    --tasks gsm8k_cot \
    --num_fewshot 8 \
    --log_samples \
    --output_path ./lm-eval-output/ \
    --device cuda
    #--gen_kwargs min_p=0.1,top_p=1,temperature=0,do_sample=True \
    #--wandb_args project=lm-eval-harness-integration 
```

there just seem to be an issue with `--gen_kwargs` and the fact that I don't have any GPU on my collab setup. Tell me if it still does not work for you ! (don't forget to restart the env)

@KonradSzafer so this sorta works!

so it appears to be a problem with `gen_kwargs`, since `wandb_args` works fine. I tried it with every `gen_kwargs`, including `top_p=1` and `temperature=1` and none of those were recognised.

I note that I also tried running on HF instead of VLLM, and HF also was unable to work. (`lm_eval: error: argument --gen_kwargs: invalid dict value: 'temperature=0'`)

So at the very least, I figured out it's not a huggingface_hub issue anymore, now it's a gen_kwargs being unusable issue ... which I suppose is progress, if completely baffling

Hey @menhguin , #1858 should fix this--apologies for the bug. 

@menhguin did you manage to pin point what precisely fixed the import error?

@clefourrier it's due to `lm_eval.logging` shadowing the python `logging` module when importing from it in `__main__` (naming it this was my suggestion, embarrassing :S)

Oh, I meant the initial HfApi import though

(But I get how tempting it is to call logging submodules `logging` haha)

Hi @haileyschoelkopf  ! This still doesn't work. Same gen_kwargs error as before: `lm_eval: error: argument --gen_kwargs: invalid dict value: 'temperature=0`

I'm only using temp = 0 which should work? I'm not sure if anyone else has managed to get it to work

Replication: https://colab.research.google.com/drive/1URpXmPqIQWttwBZANo1Eud3_jWkQYC6e#scrollTo=F4Xrw41nxz0P

Thanks for reporting this! 

#1607 seems like it fixed this issue already? I ran your colab but installing `lm_eval` from github, and it seemed to run.

After doing a clean reinstall, it indeed fixed it for me, thanks