Repository: huggingface/lighteval
Issue: [BUG] Cannot run LiveCodeBench with `lighteval==0.11.0` (`ValueError: Incorrect type for <function codegen_metric at 0x7f96264c9440>, should be a SampleLevelComputation or Preparator`)
Root cause: External Dependency Breakage

## Describe the bug
It is not possible to run LiveCodeBench with the latest lighteval release and highest version of `vllm` that's compatible with it.

## To Reproduce

Running the following:

```sh
lighteval vllm 'model_name=Qwen/Qwen3-0.6B,revision=main,trust_remote_code=False,system_prompt=/think,dtype=bfloat16,data_parallel_size=8,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}' 'extended|lcb:codegeneration_v4|0|0' --remove-reasoning-tags --output-dir eval_results/Qwen/Qwen3-0.6B/main/lcb_v4/2025-09-22T13-04-21 --save-details
```

produces

```
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /fsx/lewis/git/hf/dev/trl-internal/trl-internal/lib/python3.11/site-packages/lighteval/main_vllm.py:129 in vllm                                                                                                                                                                           │
│                                                                                                                                                                                                                                                                                           │
│   126 │   │   metric_options=metric_options,                                                                                                                                                                                                                                              │
│   127 │   )                                                                                                                                                                                                                                                                               │
│   128 │                                                                                                                                                                                                                                                                                   │
│ ❱ 129 │   pipeline.evaluate()                                                                                                                                                                                                                                                             │
│   130 │                                                                                                                                                                                                                                                                                   │
│   131 │   pipeline.show_results()                                                                                                                                                                                                                                                         │
│   132                                                                                                                                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                                           │
│ /fsx/lewis/git/hf/dev/trl-internal/trl-internal/lib/python3.11/site-packages/lighteval/pipeline.py:286 in evaluate                                                                                                                                                                        │
│                                                                                                                                                                                                                                                                                           │
│   283 │   │                                                                                                                                                                                                                                                                               │
│   284 │   │   if self.is_main_process():                                                                                                                                                                                                                                                  │
│   285 │   │   │   self._post_process_outputs(outputs)                                                                                                                                                                                                                                     │
│ ❱ 286 │   │   │   self._compute_metrics(outputs)                                                                                                                                                                                                                                          │
│   287 │   │   │                                                                                                                                                                                                                                                                           │
│   288 │   │   │   self.evaluation_tracker.general_config_logger.log_end_time()                                                                                                                                                                                                            │
│   289 │   │   │   self.evaluation_tracker.metrics_logger.aggregate(                                                                                                                                                                                                                       │
│                                                                                                                                                                                                                                                                                           │
│ /fsx/lewis/git/hf/dev/trl-internal/trl-internal/lib/python3.11/site-packages/lighteval/pipeline.py:386 in _compute_metrics                                                                                                                                                                │
│                                                                                                                                                                                                                                                                                           │
│   383 │   │   │   │   docs = [doc for doc, _ in samples]                                                                                                                                                                                                                                  │
│   384 │   │   │   │   responses = [response for _, response in samples]                                                                                                                                                                                                                   │
│   385 │   │   │   │                                                                                                                                                                                                                                                                       │
│ ❱ 386 │   │   │   │   outputs = apply_metric(                                                                                                                                                                                                                                             │
│   387 │   │   │   │   │   docs=docs,                                                                                                                                                                                                                                                      │
│   388 │   │   │   │   │   responses=responses,                                                                                                                                                                                                                                            │
│   389 │   │   │   │   │   metrics=metric_category_metrics,                                                                                                                                                                                                                                │
│                                                                                                                                                                                                                                                                                           │
│ /fsx/lewis/git/hf/dev/trl-internal/trl-internal/lib/python3.11/site-packages/lighteval/metrics/__init__.py:54 in apply_metric                                                                                                                                                             │
│                                                                                                                                                                                                                                                                                           │
│   51 │   │   # Add non-batched metric results for this sample                                                                                                                                                                                                                             │
│   52 │   │   for metric in non_batched_metrics:                                                                                                                                                                                                                                           │
│   53 │   │   │   output.update(                                                                                                                                                                                                                                                           │
│ ❱ 54 │   │   │   │   metric.compute_sample(                                                                                                                                                                                                                                               │
│   55 │   │   │   │   │   model_response=responses[i],                                                                                                                                                                                                                                     │
│   56 │   │   │   │   │   doc=docs[i],                                                                                                                                                                                                                                                     │
│   57 │   │   │   │   )                                                                                                                                                                                                                                                                    │
│                                                                                                                                                                                                                                                                                           │
│ /fsx/lewis/git/hf/dev/trl-internal/trl-internal/lib/python3.11/site-packages/lighteval/metrics/utils/metric_utils.py:53 in compute_sample                                                                                                                                                 │
│                                                                                                                                                                                                                                                                                           │
│    50 │   │   elif isinstance(self.sample_level_fn, Preparator):                                                                                                                                                                                                                          │
│    51 │   │   │   sample_level_fn = self.sample_level_fn.prepare                                                                                                                                                                                                                          │
│    52 │   │   else:                                                                                                                                                                                                                                                                       │
│ ❱  53 │   │   │   raise ValueError(                                                                                                                                                                                                                                                       │
│    54 │   │   │   │   f"Incorrect type for {self.sample_level_fn}, should be a SampleLevelComp                                                                                                                                                                                            │
│    55 │   │   │   )                                                                                                                                                                                                                                                                       │
│    56                                                                                                                                                                                                                                                                                     │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Incorrect type for <function codegen_metric at 0x7f96264c9440>, should be a SampleLevelComputation or Preparator
```

## Expected behavior

That LiveCodeBench works :)

## Version info

Output from `uv pip freeze`

```
absl-py==2.3.1
accelerate==1.9.0
addict==2.4.0
aenum==3.1.15
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.12.15
aiosignal==1.4.0
annotated-types==0.7.0
antlr4-python3-runtime==4.13.2
anyio==4.10.0
appdirs==1.4.4
argcomplete==3.6.2
astor==0.8.1
async-lru==2.0.5
attrs==25.3.0
backoff==2.2.1
bfcl-eval @ git+https://github.com/huggingface/gorilla@e8e22e3d55943d258f67b60d99a9a7287a791171#subdirectory=berkeley-function-call-leaderboard
bitsandbytes==0.47.0
black==25.9.0
blake3==1.0.6
blis==1.3.0
boto3==1.40.35
botocore==1.40.35
brotli==1.1.0
cachetools==6.2.0
catalogue==2.0.10
cbor2==5.7.0
certifi==2025.8.3
cffi==2.0.0
cfgv==3.4.0
chardet==5.2.0
charset-normalizer==3.4.3
click==8.3.0
cloudpathlib==0.22.0
cloudpickle==3.1.1
colorama==0.4.6
colorlog==6.9.0
compressed-tensors==0.10.2
confection==0.1.5
contourpy==1.3.3
coverage==7.10.7
cryptography==46.0.1
cupy-cuda12x==13.6.0
cycler==0.12.1
cymem==2.0.11
dashscope==1.24.6
dataclasses-json==0.6.7
datamodel-code-generator==0.25.7
dataproperty==1.1.0
datasets==4.1.1
datatrove==0.6.0
deepdiff==8.6.1
deepspeed==0.17.4
depyf==0.19.0
diffusers==0.35.1
dill==0.4.0
diskcache==5.6.3
distlib==0.4.0
distro==1.9.0
dnspython==2.8.0
docopt==0.6.2
docstring-parser==0.17.0
einops==0.8.1
email-validator==2.3.0
emoji==2.15.0
eval-type-backport==0.2.2
execnet==2.1.1
fastapi==0.117.1
fastapi-cli==0.0.13
fastapi-cloud-cli==0.2.0
fastrlock==0.8.3
fastuuid==0.12.0
ffmpy==0.6.1
filelock==3.19.1
fonttools==4.60.0
frozenlist==1.7.0
fs==2.4.16
fsspec==2025.9.0
ftfy==6.3.1
genson==1.3.0
gguf==0.17.1
gitdb==4.0.12
gitpython==3.1.45
googleapis-common-protos==1.70.0
gradio==5.46.1
gradio-client==1.13.1
groovy==0.1.2
h11==0.16.0
hf-transfer==0.1.9
hf-xet==1.1.10
hjson==3.1.0
httpcore==1.0.9
httptools==0.6.4
httpx==0.28.1
huggingface-hub==0.35.0
humanize==4.13.0
identify==2.6.14
idna==3.10
importlib-metadata==8.7.0
inflect==5.6.2
iniconfig==2.1.0
inquirerpy==0.3.4
interegular==0.3.3
isort==5.13.2
jieba==0.42.1
jinja2==3.1.6
jiter==0.11.0
jmespath==1.0.1
joblib==1.5.2
json5==0.12.1
jsonlines==4.0.0
jsonschema==4.25.1
jsonschema-specifications==2025.9.1
kernels==0.9.0
kiwisolver==1.4.9
langcodes==3.5.0
langdetect==1.0.9
langfuse==3.5.0
language-data==1.3.0
lark==1.2.2
latex2sympy2-extended==1.0.6
libtmux==0.46.2
liger-kernel==0.6.2
lighteval==0.11.0
litellm==1.77.3
llguidance==0.7.30
llm-blender==0.0.2
llvmlite==0.44.0
lm-format-enforcer==0.10.12
loguru==0.7.3
lxml==6.0.2
madoka==0.7.1
marisa-trie==1.3.1
markdown-it-py==4.0.0
markupsafe==3.0.2
marshmallow==3.26.1
matplotlib==3.10.6
mbstrdecoder==1.1.4
mdurl==0.1.2
mistral-common==1.8.5
more-itertools==10.8.0
mpmath==1.3.0
msgpack==1.1.1
msgspec==0.19.0
multidict==6.6.4
multiprocess==0.70.16
murmurhash==1.0.13
mypy-extensions==1.1.0
narwhals==2.5.0
natto-py==1.0.1
networkx==3.5
ninja==1.13.0
nltk==3.9.1
nodeenv==1.9.1
num2words==0.5.14
numba==0.61.2
numpy==2.2.6
nvidia-cublas-cu12==12.6.4.1
nvidia-cuda-cupti-cu12==12.6.80
nvidia-cuda-nvrtc-cu12==12.6.77
nvidia-cuda-runtime-cu12==12.6.77
nvidia-cudnn-cu12==9.5.1.17
nvidia-cufft-cu12==11.3.0.4
nvidia-cufile-cu12==1.11.1.6
nvidia-curand-cu12==10.3.7.77
nvidia-cusolver-cu12==11.7.1.2
nvidia-cusparse-cu12==12.5.4.2
nvidia-cusparselt-cu12==0.6.3
nvidia-nccl-cu12==2.26.2
nvidia-nvjitlink-cu12==12.6.85
nvidia-nvtx-cu12==12.6.77
openai==1.108.1
openai-harmony==0.0.4
opencv-python-headless==4.12.0.88
opentelemetry-api==1.37.0
opentelemetry-exporter-otlp-proto-common==1.37.0
opentelemetry-exporter-otlp-proto-http==1.37.0
opentelemetry-proto==1.37.0
opentelemetry-sdk==1.37.0
opentelemetry-semantic-conventions==0.58b0
orderly-set==5.5.0
orjson==3.11.3
outlines-core==0.2.10
overrides==7.7.0
packaging==25.0
pandas==2.3.2
parameterized==0.9.0
partial-json-parser==0.2.1.1.post6
pathlib==1.0.1
pathspec==0.12.1
pathvalidate==3.3.1
peft==0.17.1
pfzy==0.3.4
pillow==11.3.0
pip==25.2
platformdirs==4.4.0
plotly==6.3.0
pluggy==1.6.0
pondpond==1.4.1
portalocker==3.2.0
pre-commit==4.3.0
preshed==3.0.10
prometheus-client==0.23.1
prometheus-fastapi-instrumentator==7.1.0
prompt-toolkit==3.0.52
propcache==0.3.2
protobuf==6.32.1
psutil==7.1.0
py-cpuinfo==9.0.0
pyarrow==21.0.0
pybase64==1.4.2
pycountry==24.6.1
pycparser==2.23
pydantic==2.11.9
pydantic-argparse==0.10.0
pydantic-core==2.33.2
pydantic-extra-types==2.10.5
pydub==0.25.1
pygments==2.19.2
pyparsing==3.2.5
pytablewriter==1.2.1
pytest==8.4.2
pytest-cov==7.0.0
pytest-rerunfailures==15.1
pytest-xdist==3.8.0
pythainlp==5.1.2
python-crfsuite==0.9.11
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
python-json-logger==3.3.0
python-multipart==0.0.20
pytokens==0.1.10
pytz==2025.2
pyvi==0.1.1
pyyaml==6.0.2
pyzmq==27.1.0
qwen-agent==0.0.29
ray==2.49.2
redis==6.4.0
referencing==0.36.2
regex==2025.9.18
requests==2.32.5
rich==14.1.0
rich-toolkit==0.15.1
rignore==0.6.4
rouge-score==0.1.2
rpds-py==0.27.1
ruff==0.13.1
s3transfer==0.14.0
sacrebleu==2.5.1
safehttpx==0.1.6
safetensors==0.6.2
scikit-learn==1.7.2
scipy==1.16.2
seaborn==0.13.2
semantic-version==2.10.0
sentencepiece==0.2.1
sentry-sdk==2.38.0
setproctitle==1.3.7
setuptools==80.9.0
shellingham==1.5.4
six==1.17.0
sklearn-crfsuite==0.5.0
smart-open==7.3.1
smmap==5.0.2
sniffio==1.3.1
soundfile==0.13.1
soxr==1.0.0
spacy==3.8.7
spacy-legacy==3.0.12
spacy-loggers==1.0.5
srsly==2.5.1
stanza==1.10.1
starlette==0.48.0
sudachidict-core==20250825
sudachipy==0.6.10
syllapy==0.7.2
sympy==1.14.0
tabledata==1.3.4
tabulate==0.9.0
tau2 @ git+https://github.com/huggingface/tau2-bench.git@05f4e743beda8354d6e1bfb611bf00d3faab31ab
tcolorpy==0.1.7
tenacity==9.1.2
termcolor==2.3.0
thinc==8.3.6
threadpoolctl==3.6.0
tiktoken==0.11.0
tokenizers==0.21.0
toml==0.10.2
tomlkit==0.13.3
torch==2.7.1
torchaudio==2.7.1
torchvision==0.22.1
tqdm==4.67.1
trackio==0.4.0
transformers==4.55.0
tree-sitter==0.21.3
tree-sitter-java==0.21.0
tree-sitter-javascript==0.21.4
triton==3.3.1
-e file:///fsx/lewis/git/hf/dev/trl-internal
typepy==1.3.4
typer==0.19.1
typing-extensions==4.15.0
typing-inspect==0.9.0
typing-inspection==0.4.1
tzdata==2025.2
urllib3==2.5.0
uvicorn==0.36.0
uvloop==0.21.0
virtualenv==20.34.0
vllm==0.10.1.1
wandb==0.22.0
wasabi==1.1.3
watchdog==6.0.0
watchfiles==1.1.0
wcwidth==0.2.13
weasel==0.4.1
websocket-client==1.8.0
websockets==15.0.1
wrapt==1.17.3
writer-sdk==2.3.1
xformers==0.0.31
xgrammar==0.1.21
xxhash==3.5.0
yarl==1.20.1
zipp==3.23.0
```