Repository: huggingface/lighteval
Issue: [BUG] Cannot import lighteval.pipeline using vllm
Root cause: External Dependency Breakage

## Describe the bug
I want to evaluate models on AIME24 or MATH-500 benchmark but encounter the following error when using lighteval
/lighteval/src/lighteval/main_vllm.py:105 in vllm                                       │
│                                                                                                  │
│   102 │   from lighteval.logging.evaluation_tracker import EvaluationTracker                     │
│   103 │   from lighteval.models.model_input import GenerationParameters                          │
│   104 │   from lighteval.models.vllm.vllm_model import VLLMModelConfig                           │
│ ❱ 105 │   from lighteval.pipeline import EnvConfig, ParallelismManager, Pipeline, PipelinePara   │
│   106 │                                                                                          │
│   107 │   TOKEN = os.getenv("HF_TOKEN")                                                          │
│   108                                                                                            │
│                                                                                                  │
│ /lighteval/src/lighteval/pipeline.py:48 in <module>                                     │
│                                                                                                  │
│    45 │   ModelResponse,                                                                         │
│    46 )                                                                                          │
│    47 from lighteval.tasks.lighteval_task import LightevalTask, create_requests_from_tasks       │
│ ❱  48 from lighteval.tasks.registry import Registry, taskinfo_selector                           │
│    49 from lighteval.tasks.requests import RequestType, SampleUid                                │
│    50 from lighteval.utils.imports import (                                                      │
│    51 │   NO_ACCELERATE_ERROR_MSG,                                                               │
│                                                                                                  │
│ /lighteval/src/lighteval/tasks/registry.py:36 in <module>                               │
│                                                                                                  │
│    33 from datasets.load import dataset_module_factory                                           │
│    34                                                                                            │
│    35 import lighteval.tasks.default_tasks as default_tasks                                      │
│ ❱  36 from lighteval.tasks.extended import AVAILABLE_EXTENDED_TASKS_MODULES                      │
│    37 from lighteval.tasks.lighteval_task import LightevalTask, LightevalTaskConfig              │
│    38 from lighteval.utils.imports import CANNOT_USE_EXTENDED_TASKS_MSG, can_load_extended_tas   │
│    39                                                                                            │
│                                                                                                  │
│ /lighteval/src/lighteval/tasks/extended/__init__.py:29 in <module>                      │
│                                                                                                  │
│   26 if can_load_extended_tasks():                                                               │
│   27 │   import lighteval.tasks.extended.hle.main as hle                                         │
│   28 │   import lighteval.tasks.extended.ifeval.main as ifeval                                   │
│ ❱ 29 │   import lighteval.tasks.extended.lcb.main as lcb                                         │
│   30 │   import lighteval.tasks.extended.mix_eval.main as mix_eval                               │
│   31 │   import lighteval.tasks.extended.mt_bench.main as mt_bench                               │
│   32 │   import lighteval.tasks.extended.olympiade_bench.main as olympiad_bench                  │
│                                                                                                  │
│ /lighteval/src/lighteval/tasks/extended/lcb/main.py:118 in <module>                     │
│                                                                                                  │
│   115                                                                                            │
│   116 extend_enum(Metrics, "lcb_codegen_metric", lcb_codegen_metric)                             │
│   117                                                                                            │
│ ❱ 118 configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_co   │
│   119                                                                                            │
│   120 tasks = []                                                                                 │
│   121                                                                                            │
│                                                                                                  │
│ open-r1/openr1/lib/python3.11/site-packages/datasets/inspect.py:174 in                 │
│ get_dataset_config_names                                                                         │
│                                                                                                  │
│   171 │   │   **download_kwargs,                                                                 │
│   172 │   )                                                                                      │
│   173 │   builder_cls = get_dataset_builder_class(dataset_module, dataset_name=os.path.basenam   │
│ ❱ 174 │   return list(builder_cls.builder_configs.keys()) or [                                   │
│   175 │   │   dataset_module.builder_kwargs.get("config_name", builder_cls.DEFAULT_CONFIG_NAME   │
│   176 │   ]                                                                                      │
│   177                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'NoneType' object has no attribute 'builder_configs'

## To Reproduce
Error when evaluating models in huggingface/openr1 project:
```
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

# AIME 2024
TASK=aime24
lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```
Environment (cuda 12.4)
```
absl-py                           2.1.0
accelerate                        1.4.0
aenum                             3.1.15
aiohappyeyeballs                  2.4.6
aiohttp                           3.11.13
aiohttp-cors                      0.7.0
aiosignal                         1.3.2
airportsdata                      20250224
annotated-types                   0.7.0
antlr4-python3-runtime            4.13.2
anyio                             4.8.0
astor                             0.8.1
attrs                             25.1.0
bitsandbytes                      0.45.3
blake3                            1.0.4
blis                              0.7.11
cachetools                        5.5.2
catalogue                         2.0.10
certifi                           2025.1.31
cffi                              1.17.1
cfgv                              3.4.0
chardet                           5.2.0
charset-normalizer                3.4.1
click                             8.1.8
cloudpathlib                      0.16.0
cloudpickle                       3.1.1
colorama                          0.4.6
colorful                          0.5.6
colorlog                          6.9.0
compressed-tensors                0.9.1
confection                        0.1.5
cymem                             2.0.11
DataProperty                      1.1.0
datasets                          3.3.2
deepspeed                         0.15.4
depyf                             0.18.0
dill                              0.3.8
diskcache                         5.6.3
distlib                           0.3.9
distro                            1.9.0
docker-pycreds                    0.4.0
einops                            0.8.1
emoji                             2.14.1
fastapi                           0.115.10
filelock                          3.17.0
flash_attn                        2.7.4.post1
frozenlist                        1.5.0
fsspec                            2024.12.0
gguf                              0.10.0
gitdb                             4.0.12
GitPython                         3.1.44
google-api-core                   2.24.1
google-auth                       2.38.0
googleapis-common-protos          1.68.0
grpcio                            1.70.0
h11                               0.14.0
hf_transfer                       0.1.9
hjson                             3.1.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.28.1
huggingface                       0.0.1
huggingface-hub                   0.29.1
identify                          2.6.8
idna                              3.10
importlib_metadata                8.6.1
iniconfig                         2.0.0
inquirerpy                        0.3.4
interegular                       0.3.3
jieba                             0.42.1
Jinja2                            3.1.5
jiter                             0.8.2
joblib                            1.4.2
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
langcodes                         3.5.0
langdetect                        1.0.9
language_data                     1.3.0
lark                              1.2.2
latex2sympy2_extended             1.0.6
liger_kernel                      0.5.3
lighteval                         0.6.0.dev0    /root/hzk/lighteval
lm-format-enforcer                0.10.11
lxml                              5.3.1
marisa-trie                       1.2.1
markdown-it-py                    3.0.0
MarkupSafe                        3.0.2
math-verify                       0.5.2
mbstrdecoder                      1.1.4
mdurl                             0.1.2
mistral_common                    1.5.3
more-itertools                    10.6.0
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.19.0
multidict                         6.1.0
multiprocess                      0.70.16
murmurhash                        1.0.12
natto-py                          1.0.1
nest-asyncio                      1.6.0
networkx                          3.4.2
ninja                             1.11.1.3
nltk                              3.9.1
nodeenv                           1.9.1
numpy                             1.26.4
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-ml-py                      12.570.86
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
open-r1                           0.1.0.dev0    /root/hzk/open-r1
openai                            1.65.1
opencensus                        0.11.4
opencensus-context                0.1.3
opencv-python-headless            4.11.0.86
outlines                          0.1.11
outlines_core                     0.1.26
packaging                         24.2
pandas                            2.2.3
partial-json-parser               0.2.1.1.post5
pathvalidate                      3.2.3
pfzy                              0.3.4
pillow                            11.1.0
pip                               25.0.1
platformdirs                      4.3.6
pluggy                            1.5.0
portalocker                       3.1.1
pre_commit                        4.1.0
preshed                           3.0.9
prometheus_client                 0.21.1
prometheus-fastapi-instrumentator 7.0.2
prompt_toolkit                    3.0.50
propcache                         0.3.0
proto-plus                        1.26.0
protobuf                          3.20.3
psutil                            7.0.0
py-cpuinfo                        9.0.0
py-spy                            0.4.0
pyarrow                           19.0.1
pyasn1                            0.6.1
pyasn1_modules                    0.4.1
pycountry                         24.6.1
pycparser                         2.22
pydantic                          2.10.6
pydantic_core                     2.27.2
Pygments                          2.19.1
pytablewriter                     1.2.1
pytest                            7.4.0
pythainlp                         5.1.0
python-crfsuite                   0.9.11
python-dateutil                   2.9.0.post0
python-dotenv                     1.0.1
pytz                              2025.1
pyvi                              0.1.1
PyYAML                            6.0.2
pyzmq                             26.2.1
ray                               2.43.0
referencing                       0.36.2
regex                             2024.11.6
requests                          2.32.3
rich                              13.9.4
rouge_score                       0.1.2
rpds-py                           0.23.1
rsa                               4.9
ruff                              0.2.2
sacrebleu                         2.5.1
safetensors                       0.5.3
scikit-learn                      1.6.1
scipy                             1.15.2
sentencepiece                     0.2.0
sentry-sdk                        2.22.0
setproctitle                      1.3.5
setuptools                        75.8.2
six                               1.17.0
sklearn-crfsuite                  0.5.0
smart-open                        6.4.0
smmap                             5.0.2
sniffio                           1.3.1
spacy                             3.7.2
spacy-legacy                      3.0.12
spacy-loggers                     1.0.5
srsly                             2.5.1
stanza                            1.10.1
starlette                         0.46.0
SudachiDict-core                  20250129
SudachiPy                         0.6.10
sympy                             1.13.1
tabledata                         1.3.4
tabulate                          0.9.0
tcolorpy                          0.1.7
termcolor                         2.3.0
thinc                             8.2.5
threadpoolctl                     3.5.0
tiktoken                          0.9.0
tokenizers                        0.21.0
torch                             2.5.1+cu124
torchaudio                        2.5.1
torchvision                       0.20.1
tqdm                              4.67.1
transformers                      4.49.0
triton                            3.1.0
trl                               0.16.0.dev0
typepy                            1.3.4
typer                             0.9.4
typing_extensions                 4.12.2
tzdata                            2025.1
urllib3                           2.3.0
uvicorn                           0.34.0
uvloop                            0.21.0
virtualenv                        20.29.2
vllm                              0.7.2
wandb                             0.19.7
wasabi                            1.1.3
watchfiles                        1.0.4
wcwidth                           0.2.13
weasel                            0.3.4
websockets                        15.0
wrapt                             1.17.2
xformers                          0.0.28.post3
xgrammar                          0.1.14
xxhash                            3.5.0
yarl                              1.18.3
zipp                              3.21.0
```

Thanks!

I have encountered the same problem.

Look at this line:
`--custom-tasks src/open_r1/evaluate.py \`

It seems that this issue is related to open-r1.

+1

+1

It seems to be a problem related to datasets. It can not read from the local cache files correctly when in offline mode.

I am a user, same bug.
I try to trace and get that：

![Image](https://github.com/user-attachments/assets/79284e9c-6dbb-4715-8cab-850c2f24cf8c)
this bug look like from the feature of livecodebench.
Anyone could talk me: what is desigh for the feature? i don‘t know how to avoid it. 

> I am a user, same bug. I try to trace and get that：
> 
> ![Image](https://github.com/user-attachments/assets/79284e9c-6dbb-4715-8cab-850c2f24cf8c) this bug look like from the feature of livecodebench. Anyone could talk me: what is desigh for the feature? i don‘t know how to avoid it.

@plaguss 
I noticed that the recent submitter of this code is you. Do you have any ideas on how to solve this problem?

hi ! all those task are in the newest version of lighteval (`pip install -U lighteval[vllm]`)
you can therefore run:

```
lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```