Repository: EleutherAI/lm-evaluation-harness
Issue: Module 'exact_match' doesn't exist on the Hugging Face Hub either.
Root cause: Architectural Constraint

![eroor](https://github.com/EleutherAI/lm-evaluation-harness/assets/113493692/53cedfd7-bcb5-40e7-839a-2cdb2ebd0ba1)
Module 'exact_match' doesn't exist on the Hugging Face Hub either.

What's wrong with this error? When I run the code on Google Colab, it worked. But running on the local server, this error occured.

I solved that problem. 
But one more problem, My server can't access to the Hub.
What should i do?
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/113493692/6a4c78dc-c3ec-4685-b7d8-9d97837d16aa)


try reaching it with a vpn like 1.1.1.1, i too faced this strange issue regarding SSL, ill share if i find any real solution like clearning up DNS cache and try or something.

Using a vpn worked for me

@BBaekdabang 

If you are able to access `https://hf-mirror.com`, try
```shell
export HF_ENDPOINT=https://hf-mirror.com
```
before running your script.

The problem is that the first `import lm_eval` in a Python process makes a HTTPS connection to Hugging Face Hub. The import fails unless you have a working internet connection or `exact_match` in your local HF cache directory.

I find the behavior rather concerning. It's not just slow and breaks offline use. `lm_eval` inadvertently behave like a tracking spy, that pings a remote server on every use. You can easily reproduce the issue on Linux by setting the XDG cache root and using `unshare` to create an unconnected network namespace.

```
$ XDG_CACHE_HOME=$(pwd)/cache unshare -rn python3 -c 'import lm_eval'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".tox/py311-unit/lib/python3.11/site-packages/lm_eval/__init__.py", line 1, in <module>
    from .evaluator import evaluate, simple_evaluate
  File ".tox/py311-unit/lib/python3.11/site-packages/lm_eval/evaluator.py", line 11, in <module>
    import lm_eval.api.metrics
  File ".tox/py311-unit/lib/python3.11/site-packages/lm_eval/api/metrics.py", line 168, in <module>
    exact_match = hf_evaluate.load("exact_match")
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".tox/py311-unit/lib/python3.11/site-packages/evaluate/loading.py", line 748, in load
    evaluation_module = evaluation_module_factory(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".tox/py311-unit/lib/python3.11/site-packages/evaluate/loading.py", line 681, in evaluation_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a module script at exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.
```

```
$ unshare -rn python3 -c 'import lm_eval'
2024-07-01:07:34:10,281 WARNING  [loading.py:546] Using the latest cached version of the module from /home/heimes/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/9d3b67e0c429cd7460b2b05aab53419b48eea369b73e1d9f185a56ca90c373d4 (last modified on Mon Jul  1 07:02:26 2024) since it couldn't be found locally at evaluate-metric--exact_match, or remotely on the Hugging Face Hub.
```

Just pushed a fix for this behavior in #2045 . It'll be part of v0.4.3 on PyPI which will be released later today!

I was about to ask for this exact thing - thank you @haileyschoelkopf!