Repository: stanford-crfm/helm
Issue: "--enable-huggingface-models" seems broken
Root cause: External Dependency Breakage

Running [this from the docs](https://crfm-helm.readthedocs.io/en/latest/huggingface_models/) doesn't work:

```bash
helm-run \
    --run-specs boolq:model=stanford-crfm/BioMedLM \
    --enable-huggingface-models stanford-crfm/BioMedLM \
    --suite v1 \
    --max-eval-instances 10
```

```
Traceback (most recent call last):
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/benchmark/executor.py", line 84, in process
    result: RequestResult = self.service.make_request(self.execution_spec.auth, state.request)
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/proxy/services/server_service.py", line 103, in make_request
    request_result: RequestResult = self.client.make_request(request)
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 249, in make_request
    client: Client = self._get_client(request.model)
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 72, in _get_client
    tokenizer: Tokenizer = self._get_tokenizer(organization)
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 354, in _get_tokenizer
    raise ValueError(f"Could not find tokenizer for model: {tokenizer_name}")
ValueError: Could not find tokenizer for model: stanford-crfm
...
Traceback (most recent call last):
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/bin/helm-run", line 8, in <module>
    sys.exit(main())
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/benchmark/run.py", line 309, in main
    run_benchmarking(
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/benchmark/run.py", line 111, in run_benchmarking
    runner.run_all(run_specs)
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/benchmark/runner.py", line 182, in run_all
    raise RunnerError(f"Failed runs: [{failed_runs_str}]")
helm.benchmark.runner.RunnerError: Failed runs: ["boolq:model=stanford-crfm_BioMedLM"]
```


I am also getting the same issues (#2058 ). However, when I run the evaluation in local setting, I got a similar error of missing tokenizer.

I tried this on a smaller model (so that it runs on my laptop) and it runs fine:
`helm-run  --enable-huggingface-models EleutherAI/pythia-70m-deduped --run-specs "mmlu:subject=anatomy,model=EleutherAI/pythia-70m-deduped" --suite v1 --max-eval-instances 20`
Any suggestions @dlwh  on why we are getting different behaviours ?

what release/branch?

On Wed, Nov 29, 2023 at 1:36 PM Josselin Somerville Roberts <
***@***.***> wrote:

> I tried this on a smaller model (so that it runs on my laptop) and it runs
> fine:
> helm-run --enable-huggingface-models EleutherAI/pythia-70m-deduped
> --run-specs "mmlu:subject=anatomy,model=EleutherAI/pythia-70m-deduped"
> --suite v1 --max-eval-instances 20
> Any suggestions @dlwh <https://github.com/dlwh> on why we are getting
> different behaviours ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/stanford-crfm/helm/issues/2056#issuecomment-1832738163>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAACLIJR7PIANRUINZNLPZ3YG6TGXAVCNFSM6AAAAAA74XSVWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZSG4ZTQMJWGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


> what release/branch?
> […](#)
> On Wed, Nov 29, 2023 at 1:36 PM Josselin Somerville Roberts < ***@***.***> wrote: I tried this on a smaller model (so that it runs on my laptop) and it runs fine: helm-run --enable-huggingface-models EleutherAI/pythia-70m-deduped --run-specs "mmlu:subject=anatomy,model=EleutherAI/pythia-70m-deduped" --suite v1 --max-eval-instances 20 Any suggestions @dlwh <https://github.com/dlwh> on why we are getting different behaviours ? — Reply to this email directly, view it on GitHub <[#2056 (comment)](https://github.com/stanford-crfm/helm/issues/2056#issuecomment-1832738163)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACLIJR7PIANRUINZNLPZ3YG6TGXAVCNFSM6AAAAAA74XSVWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZSG4ZTQMJWGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

on main

Ok after some investigating, this seems to be an issue with `0.3.0` but it is solved on `main`. I will close this when `0.4.0` comes out (https://github.com/stanford-crfm/helm/pull/2023)

Thanks @JosselinSomervilleRoberts ! @KaiserWhoLearns I also verified this works. #2057 doesn't seem fixed just yet, but it could be i'm using it wrong.

Thanks @dlwh I tested it with my local HF model (#2058 ), and it was able to run through with that release. I think #2057 may be a different issue.

I tested this and it should be working now.

Perhaps we should add an integration test for the Hugging Face integration, but I'm reluctant to add integration tests with external dependencies in general because it increases flakiness.