Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Feature Request] Update Datasets Version, so that lmms-eval can be used in Offline Environment
Root cause: External Dependency Breakage

I encountered a similar issue as [This one](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/15#issuecomment-2373169492), when running lmms-eval with an **offline** machine(no Internet). `load_dataset` method still tries to reach Hugging Face Hub when I set `HF_DATASETS_OFFLINE ` to `1`.

I looked into [this issue from huggingface Datasets](https://github.com/huggingface/datasets/issues/4760) and found it is a bug from datasets library, where `load_dataset` method still tries to reach Hugging Face Hub after setting`HF_DATASETS_OFFLINE ` to `1`.

And the bug is fixed with [this PR](https://github.com/huggingface/datasets/pull/6741) since Datasets Version **2.19.0**.     And it has been verified [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/15#issuecomment-2373251290) that updating Datasets to newer version ACTUALLY enables lmms-eval to run without bug in offline environment.

So I suggest to update Datasets Version to >= 2.19.0 so that lmms-eval can be used in fully offline environment. Any future plans for that?

(Although there are currently some workarounds for running lmms-eval in offline environment [179](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/179) [21](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/21), I think them  inconvenient when you need to evaluate MANY tasks. And I think supporting lmms-eval in offline environment will help a lot of users.)



Thanks for this nice suggestion!

Are you often use in offline environment? It's much appreciated that you can send a PR to modify the version and also give us some guidance by adding to `./docs/xxx.md` to introduce the usage in offline environment.

Hi @jungle-gym-ac , have you tried at your side that once exporting the `HF_DATASET_OFFLINE` env var with the newest dataset version, you no longer need internet access using `lmms-eval`? If that is the case, I think we will update the dependency in the pyproject.toml and update the docs

Hi @kcz358, sorry for the late reply! I have tried with datasets version `2.20.0`, and SUCCESSFULLY run lmms-eval without internet access with `HF_DATASET_OFFLINE=1`.

The main steps are:
- Suppose you have successfully run lmms-eval on a `source` machine with Internet access.  Transfer the datasets you need to test from `source` machine to the `target` machine without internet(under `$HF_HOME/datasets/` for both machines), and also pack&move the conda environment with **updated** datasets version.
-  Run lmms-eval on the `target` machine with the following command:
```bash
export HF_DATASET_OFFLINE=1
python3 -m accelerate.commands.launch ... #The exact same command you run on `sourse machine`
```
And sure I can create a PR for that as @Luodian mentioned.

Great to hear that, this helps a lot!

> I encountered a similar issue as [This one](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/15#issuecomment-2373169492), when running lmms-eval with an **offline** machine(no Internet). `load_dataset` method still tries to reach Hugging Face Hub when I set `HF_DATASETS_OFFLINE ` to `1`.
> 
> I looked into [this issue from huggingface Datasets](https://github.com/huggingface/datasets/issues/4760) and found it is a bug from datasets library, where `load_dataset` method still tries to reach Hugging Face Hub after setting`HF_DATASETS_OFFLINE ` to `1`.
> 
> And the bug is fixed with [this PR](https://github.com/huggingface/datasets/pull/6741) since Datasets Version **2.19.0**. And it has been verified [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/15#issuecomment-2373251290) that updating Datasets to newer version ACTUALLY enables lmms-eval to run without bug in offline environment.
> 
> So I suggest to update Datasets Version to >= 2.19.0 so that lmms-eval can be used in fully offline environment. Any future plans for that?
> 
> (Although there are currently some workarounds for running lmms-eval in offline environment [179](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/179) [21](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/21), I think them inconvenient when you need to evaluate MANY tasks. And I think supporting lmms-eval in offline environment will help a lot of users.)

哈喽你好，在datasets 2.16 的时候，我设置HF_DATASETS_OFFLINE=1，第一次运行很慢，然后第二次运行就很快，是因为第一次仍然试图从hf在线加载数据吗~

> > I encountered a similar issue as [This one](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/15#issuecomment-2373169492), when running lmms-eval with an **offline** machine(no Internet). `load_dataset` method still tries to reach Hugging Face Hub when I set `HF_DATASETS_OFFLINE ` to `1`.
> > I looked into [this issue from huggingface Datasets](https://github.com/huggingface/datasets/issues/4760) and found it is a bug from datasets library, where `load_dataset` method still tries to reach Hugging Face Hub after setting`HF_DATASETS_OFFLINE ` to `1`.
> > And the bug is fixed with [this PR](https://github.com/huggingface/datasets/pull/6741) since Datasets Version **2.19.0**. And it has been verified [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/15#issuecomment-2373251290) that updating Datasets to newer version ACTUALLY enables lmms-eval to run without bug in offline environment.
> > So I suggest to update Datasets Version to >= 2.19.0 so that lmms-eval can be used in fully offline environment. Any future plans for that?
> > (Although there are currently some workarounds for running lmms-eval in offline environment [179](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/179) [21](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/21), I think them inconvenient when you need to evaluate MANY tasks. And I think supporting lmms-eval in offline environment will help a lot of users.)
> 
> 哈喽你好，在datasets 2.16 的时候，我设置HF_DATASETS_OFFLINE=1，第一次运行很慢，然后第二次运行就很快，是因为第一次仍然试图从hf在线加载数据吗~

牛逼，确实是datasets版本的问题


Thanks for this suggestion, I've made this change to the default requirement.