Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG] name 'get_task_dict' is not defined when using --tasks list_with_num command
Root cause: Validation Gap


### Description

When running the command `python -m lmms_eval --tasks list_with_num --verbosity=DEBUG`, I encounter a `NameError` because the `get_task_dict` function is not properly imported or defined in the scope where it's being called.

### Error Details

**Command:**
```bash
python -m lmms_eval --tasks list_with_num --verbosity=DEBUG
```

**Error Output:**
```
  File "/.../lmms-eval/lmms_eval/__main__.py", line 435, in cli_evaluate_single
    task_dict = get_task_dict([task_name], model_name="llava")
                ^^^^^^^^^^^^^
NameError: name 'get_task_dict' is not defined. Did you mean: 'task_dict'?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
...
Task :   fail to load
Exception :
name 'get_task_dict' is not defined
```

### Root Cause Analysis

1. **Missing Import:** The `get_task_dict` function is not imported in `lmms_eval/__main__.py`
2. **Function Signature Mismatch:** The function is called with `model_name="llava"` parameter, but the actual function definition in `lmms_eval/tasks/__init__.py` has a different signature:

**Current call in `__main__.py`:**
```python
task_dict = get_task_dict([task_name], model_name="llava")
```

**Actual function definition in `lmms_eval/tasks/__init__.py`:**
```python
def get_task_dict(
    task_name_list: Union[str, List[Union[str, Dict, Task]]],
    task_manager: Optional[TaskManager] = None,
):
```

### Environment

- **Installation:** `pip install -e .`
- **Commit:** [c4a3128](https://github.com/EvolvingLMMs-Lab/lmms-eval/commit/c4a31289c56a673aba99d2015a1db1decf5308e7)



I think this one is deprecated and will be removed in the later version. With so many tasks integrated, it is kind of unnecessary to list all tasks with num. Tried with `list` or `list_group` to check all the task and check the dataset detail on the huggingface would be better.

> I think this one is deprecated and will be removed in the later version. With so many tasks integrated, it is kind of unnecessary to list all tasks with num. Tried with `list` or `list_group` to check all the task and check the dataset detail on the huggingface would be better.

Thanks for the clarification! I understand that list_with_num might be deprecated, but I have a specific use case where it would be very helpful.

My situation:
* I'm working with an offline compute cluster that doesn't have internet access
* I need to pre-download all datasets on a machine that has network access, then transfer them to the offline cluster
* Some datasets are not hosted on HuggingFace, so I can't use the HuggingFace CLI to download them
* I'm following the approach mentioned in https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/15#issuecomment-2016694472 for offline dataset preparation

Is there any updated recommended way to bulk download datasets for offline use? 

I would suggest that you can first list down the huggingface dataset path that you needed. I am assuming that you don't actually need every dataset in lmms-eval?

Then you can write a simple python script and use `load_dataset` the cache all the files. Then you can transfer the `~/.cache/huggingface` on your local machine to your compute machine. The you can follow #335 to set offline mode to load from compute cluster

> I would suggest that you can first list down the huggingface dataset path that you needed. I am assuming that you don't actually need every dataset in lmms-eval?
> 
> Then you can write a simple python script and use `load_dataset` the cache all the files. Then you can transfer the `~/.cache/huggingface` on your local machine to your compute machine. The you can follow [#335](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/335) to set offline mode to load from compute cluster

Thank you for the suggestion! You're right that I don't need every dataset in lmms-eval.

However, I've noticed that some datasets like Video-MME ([https://huggingface.co/datasets/lmms-lab/Video-MME ](https://huggingface.co/datasets/lmms-lab/Video-MME)) contain YouTube links rather than direct video files. When I use datasets.load_dataset("lmms-lab/Video-MME"), it only downloads the metadata (including YouTube URLs) to HF_HOME, but doesn't actually download the video content.

I observed that there's video downloading logic in lmms-eval/lmms_eval/api/task.py. Is there an elegant way to invoke this functionality to pre-download all the videos on a machine that has internet access but no GPU?