Repository: stanford-crfm/helm
Issue: Requests failed when using `--enable-local-huggingface-models`
Root cause: Environment Incompatibility

Thank you for the awesome work!

When using `--enable-local-huggingface-models` option 1 described in #1505, I followed the instruction. Also, I modified the GPU to "cuda:5" due to the occupied other GPUs with the instruction from this post #1683. When using it and running the command below, it throws some error saying "ExecutorError: Failed to make requests to huggingface/<local_model_name> after retrying 5 times. Error: HuggingFace error: CUDA error: no kernel image is available for execution on the device."
```
python 3 -m helm.benchmark.run \
    --enable-local-huggingface-models /dir-to-model-folder/mymodel \
    --run-specs boolq:model=huggingface/mymodel \
    --suite v1 \
    --max-eval-instances 1
```

My question is do I need to change other parts of the code in the repo to make it work, like building up some server? Or is it the device issue leading to this problem?

And also it throws warnings like "UserWarning: NVIDIA RTX A5000 with CUDA capability sm_85 is not compatible with the current PyTorch installation." But the current Pytorch version 1.21.1 seems to be required by the helm repo. Do I need to try using other types of GPUs instead?

Above problem is solved when upgrading Pytorch to the latest version. But when I was trying to evaluate the local model with `--max-eval-instances 10`. It got stuck after the evaluation loop.
<img width="930" alt="image" src="https://github.com/stanford-crfm/helm/assets/67442207/6b1d3c86-2f03-4be2-b539-126dde7b28fd">


Required PyTorch version: I'd suggest manually installing a version of PyTorch and CUDA that is compatible with your GPU and GPU drivers. You can ignore the HELM dependency conflict errors. I am planning to widen the range of PyTorch dependency versions to reduce dependency conflict issues: #1745

Stuck in evaluation loop: Not too sure why this is happening. It could be a deadlock inside or outside HELM. I'd suggest trying again with `--num-threads 1`.

> Required PyTorch version: I'd suggest manually installing a version of PyTorch and CUDA that is compatible with your GPU and GPU drivers. You can ignore the HELM dependency conflict errors. I am planning to widen the range of PyTorch dependency versions to reduce dependency conflict issues: #1745
> 
> Stuck in evaluation loop: Not too sure why this is happening. It could be a deadlock inside or outside HELM. I'd suggest trying again with `--num-threads 1`.

Yeah, setting the `num_thread` works for me. Is there any existing `.conf` file for the [core scenario](https://crfm.stanford.edu/helm/latest/?group=core_scenarios) only on this webpage? The [run_specs.conf](https://github.com/stanford-crfm/helm/blob/b344db17ee6bf3787cfbe90598c0dfbb39ffa208/src/helm/benchmark/presentation/run_specs.conf) and [run_specs_gpu.conf](https://github.com/stanford-crfm/helm/blob/b344db17ee6bf3787cfbe90598c0dfbb39ffa208/src/helm/benchmark/presentation/run_specs_gpu.conf) seem heavy for me now.

We don't have that yet, but I'll make add next week. Seems like a good thing to have.

@linguo4 You can download the conf files from PR #1754.

`run_specs_core_scenarios.conf` is the core scenarios for the webpage. `run_specs_lite_core_scenarios.conf` has some additional expensive scenarios removed (summarization and MS MARCO) and would be my recommendation if the former is too heavy.

We usually run these with `--priority 2`.

> @linguo4 You can download the conf files from PR #1754.
> 
> `run_specs_core_scenarios.conf` is the core scenarios for the webpage. `run_specs_lite_core_scenarios.conf` has some additional expensive scenarios removed (summarization and MS MARCO) and would be my recommendation if the former is too heavy.
> 
> We usually run these with `--priority 2`.

Thank you so much!