Repository: EleutherAI/lm-evaluation-harness
Issue: Unable to use accelerate for multi-gpu evaluation
Root cause: External Dependency Breakage

Hello all,

first of all thanks to the authors for maintaining this awesome resource for benchmarking!

I'd like to evaluate a few models on squadv2 using data-parallel (7 GPUs), for this I'm following the instructions as provided by the readme by running:
```
accelerate launch -m lm_eval --model hf --model_args pretrained=Qwen/Qwen3-1.7B --tasks squadv2 --batch_size 32
```

However, this fails with:
```
[rank5]:   File "[...]/anaconda3/envs/ellmc_peft/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4284, in from_pretrained
[rank5]:     raise ValueError("DeepSpeed Zero-3 is not compatible with passing a `device_map`.")
[rank5]: ValueError: DeepSpeed Zero-3 is not compatible with passing a `device_map`.

```
same errors for the other ranks.

Running 
```
accelerate launch -m lm_eval --model hf --model_args pretrained=Qwen/Qwen3-1.7B,device_map=None --tasks squadv2 --batch_size 32
```
instead fails with:
```
[rank1]:   File "[...]/anaconda3/envs/ellmc_peft/lib/python3.11/site-packages/torch/nn/functional.py", line 2551, in embedding
[rank1]:     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:1! (when checking argument for argument index in method wrapper_CUDA__index_select)
Running generate_until requests:   0%|   
```

I'd really appreciate any help in this matter.


Here is the enviroment I'm using:
```
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_python_abi3_support      1.0                  hd8ed1ab_2    conda-forge
absl-py                   2.3.1                    pypi_0    pypi
accelerate                1.9.0                    pypi_0    pypi
aiohappyeyeballs          2.6.1              pyhd8ed1ab_0    conda-forge
aiohttp                   3.12.14         py311h3778330_0    conda-forge
aiosignal                 1.4.0              pyhd8ed1ab_0    conda-forge
annotated-types           0.7.0                    pypi_0    pypi
attrs                     25.3.0             pyh71513ae_0    conda-forge
aws-c-auth                0.9.0               h0fbd49f_19    conda-forge
aws-c-cal                 0.9.2                he7b75e1_1    conda-forge
aws-c-common              0.12.4               hb03c661_0    conda-forge
aws-c-compression         0.3.1                h92c474e_6    conda-forge
aws-c-event-stream        0.5.5                h149bd38_3    conda-forge
aws-c-http                0.10.4               h37a7233_0    conda-forge
aws-c-io                  0.21.2               h6252d9a_1    conda-forge
aws-c-mqtt                0.13.3               h19deb91_3    conda-forge
aws-c-s3                  0.8.6                h800fcd2_2    conda-forge
aws-c-sdkutils            0.2.4                h92c474e_1    conda-forge
aws-checksums             0.2.7                h92c474e_2    conda-forge
aws-crt-cpp               0.33.1               hb4fd278_2    conda-forge
aws-sdk-cpp               1.11.606             h31ade35_1    conda-forge
azure-core-cpp            1.16.0               h3a458e0_0    conda-forge
azure-identity-cpp        1.12.0               ha729027_0    conda-forge
azure-storage-blobs-cpp   12.14.0              hb1c9500_1    conda-forge
azure-storage-common-cpp  12.10.0              hebae86a_2    conda-forge
azure-storage-files-datalake-cpp 12.12.0              h8b27e44_3    conda-forge
brotli-python             1.1.0           py311hfdbb021_3    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.5               hb9d3cd8_0    conda-forge
ca-certificates           2025.7.14            hbd8a1cb_0    conda-forge
certifi                   2025.7.14          pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py311hf29c0ef_0    conda-forge
chardet                   5.2.0                    pypi_0    pypi
charset-normalizer        3.4.2              pyhd8ed1ab_0    conda-forge
click                     8.2.1                    pypi_0    pypi
colorama                  0.4.6              pyhd8ed1ab_1    conda-forge
cpython                   3.11.13         py311hd8ed1ab_0    conda-forge
cuda                      12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-cccl                 12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-command-line-tools   12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-compiler             12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-cudart               12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-cudart-dev           12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-cudart-static        12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-cuobjdump            12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-cupti                12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-cupti-static         12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-cuxxfilt             12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-demo-suite           12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-documentation        12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-driver-dev           12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-gdb                  12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-libraries            12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-libraries-dev        12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-libraries-static     12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-nsight               12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nsight-compute       12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-nvcc                 12.4.131                      0    nvidia/label/cuda-12.4.1
cuda-nvdisasm             12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nvml-dev             12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nvprof               12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nvprune              12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nvrtc                12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nvrtc-dev            12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nvrtc-static         12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nvtx                 12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-nvvp                 12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-opencl               12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-opencl-dev           12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-profiler-api         12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-sanitizer-api        12.4.127                      0    nvidia/label/cuda-12.4.1
cuda-toolkit              12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-tools                12.4.1                        0    nvidia/label/cuda-12.4.1
cuda-visual-tools         12.4.1                        0    nvidia/label/cuda-12.4.1
dataproperty              1.1.0                    pypi_0    pypi
datasets                  3.6.0              pyhd8ed1ab_0    conda-forge
deepspeed                 0.17.3                   pypi_0    pypi
dill                      0.3.8              pyhd8ed1ab_0    conda-forge
einops                    0.8.1                    pypi_0    pypi
evaluate                  0.4.5                    pypi_0    pypi
filelock                  3.18.0             pyhd8ed1ab_0    conda-forge
flash-attn                2.7.4.post1              pypi_0    pypi
frozenlist                1.7.0           py311h52bc045_0    conda-forge
fsspec                    2025.3.0           pyhd8ed1ab_0    conda-forge
gds-tools                 1.9.1.3                       0    nvidia/label/cuda-12.4.1
gflags                    2.2.2             h5888daf_1005    conda-forge
glog                      0.7.1                hbabe93e_0    conda-forge
h2                        4.2.0              pyhd8ed1ab_0    conda-forge
hf-xet                    1.1.5            py39h260a9e5_3    conda-forge
hjson                     3.1.0                    pypi_0    pypi
hpack                     4.1.0              pyhd8ed1ab_0    conda-forge
huggingface_hub           0.33.4             pyhd8ed1ab_0    conda-forge
hyperframe                6.1.0              pyhd8ed1ab_0    conda-forge
icu                       75.1                 he02047a_0    conda-forge
idna                      3.10               pyhd8ed1ab_1    conda-forge
jinja2                    3.1.4                    pypi_0    pypi
joblib                    1.5.1                    pypi_0    pypi
jsonlines                 4.0.0                    pypi_0    pypi
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
ld_impl_linux-64          2.44                 h1423503_1    conda-forge
libabseil                 20250512.1      cxx17_hba17884_0    conda-forge
libarrow                  21.0.0           hd5bb725_0_cpu    conda-forge
libarrow-acero            21.0.0           h635bf11_0_cpu    conda-forge
libarrow-compute          21.0.0           he319acf_0_cpu    conda-forge
libarrow-dataset          21.0.0           h635bf11_0_cpu    conda-forge
libarrow-substrait        21.0.0           h3f74fd7_0_cpu    conda-forge
libblas                   3.9.0           32_h59b9bed_openblas    conda-forge
libbrotlicommon           1.1.0                hb9d3cd8_3    conda-forge
libbrotlidec              1.1.0                hb9d3cd8_3    conda-forge
libbrotlienc              1.1.0                hb9d3cd8_3    conda-forge
libcblas                  3.9.0           32_he106b2a_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcublas                 12.4.5.8                      0    nvidia/label/cuda-12.4.1
libcublas-dev             12.4.5.8                      0    nvidia/label/cuda-12.4.1
libcublas-static          12.4.5.8                      0    nvidia/label/cuda-12.4.1
libcufft                  11.2.1.3                      0    nvidia/label/cuda-12.4.1
libcufft-dev              11.2.1.3                      0    nvidia/label/cuda-12.4.1
libcufft-static           11.2.1.3                      0    nvidia/label/cuda-12.4.1
libcufile                 1.9.1.3                       0    nvidia/label/cuda-12.4.1
libcufile-dev             1.9.1.3                       0    nvidia/label/cuda-12.4.1
libcufile-static          1.9.1.3                       0    nvidia/label/cuda-12.4.1
libcurand                 10.3.5.147                    0    nvidia/label/cuda-12.4.1
libcurand-dev             10.3.5.147                    0    nvidia/label/cuda-12.4.1
libcurand-static          10.3.5.147                    0    nvidia/label/cuda-12.4.1
libcurl                   8.14.1               h332b0f4_0    conda-forge
libcusolver               11.6.1.9                      0    nvidia/label/cuda-12.4.1
libcusolver-dev           11.6.1.9                      0    nvidia/label/cuda-12.4.1
libcusolver-static        11.6.1.9                      0    nvidia/label/cuda-12.4.1
libcusparse               12.3.1.170                    0    nvidia/label/cuda-12.4.1
libcusparse-dev           12.3.1.170                    0    nvidia/label/cuda-12.4.1
libcusparse-static        12.3.1.170                    0    nvidia/label/cuda-12.4.1
libedit                   3.1.20250104    pl5321h7949ede_0    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.7.1                hecca717_0    conda-forge
libffi                    3.4.6                h2dba641_1    conda-forge
libgcc                    15.1.0               h767d61c_3    conda-forge
libgcc-ng                 15.1.0               h69a702a_3    conda-forge
libgfortran               15.1.0               h69a702a_3    conda-forge
libgfortran5              15.1.0               hcea5267_3    conda-forge
libgomp                   15.1.0               h767d61c_3    conda-forge
libgoogle-cloud           2.39.0               hdb79228_0    conda-forge
libgoogle-cloud-storage   2.39.0               hdbdcf42_0    conda-forge
libgrpc                   1.73.1               h1e535eb_0    conda-forge
libiconv                  1.18                 h4ce23a2_1    conda-forge
liblapack                 3.9.0           32_h7ac8fdf_openblas    conda-forge
liblzma                   5.8.1                hb9d3cd8_2    conda-forge
libnghttp2                1.64.0               h161d5f1_0    conda-forge
libnpp                    12.2.5.30                     0    nvidia/label/cuda-12.4.1
libnpp-dev                12.2.5.30                     0    nvidia/label/cuda-12.4.1
libnpp-static             12.2.5.30                     0    nvidia/label/cuda-12.4.1
libnsl                    2.0.1                hb9d3cd8_1    conda-forge
libnvfatbin               12.4.127                      0    nvidia/label/cuda-12.4.1
libnvfatbin-dev           12.4.127                      0    nvidia/label/cuda-12.4.1
libnvjitlink              12.4.127                      0    nvidia/label/cuda-12.4.1
libnvjitlink-dev          12.4.127                      0    nvidia/label/cuda-12.4.1
libnvjpeg                 12.3.1.117                    0    nvidia/label/cuda-12.4.1
libnvjpeg-dev             12.3.1.117                    0    nvidia/label/cuda-12.4.1
libnvjpeg-static          12.3.1.117                    0    nvidia/label/cuda-12.4.1
libopenblas               0.3.30          pthreads_h94d23a6_1    conda-forge
libopentelemetry-cpp      1.21.0               hb9b0907_1    conda-forge
libopentelemetry-cpp-headers 1.21.0               ha770c72_1    conda-forge
libparquet                21.0.0           h790f06f_0_cpu    conda-forge
libprotobuf               6.31.1               h9ef548d_1    conda-forge
libre2-11                 2025.07.22           h7b12aa8_0    conda-forge
libsqlite                 3.50.3               hee844dc_1    conda-forge
libssh2                   1.11.1               hcf80075_0    conda-forge
libstdcxx                 15.1.0               h8f9b012_3    conda-forge
libstdcxx-ng              15.1.0               h4852527_3    conda-forge
libthrift                 0.22.0               h454ac66_1    conda-forge
libutf8proc               2.10.0               h202a827_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.13.8               h4bc477f_0    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
llvmlite                  0.44.0                   pypi_0    pypi
lm-eval                   0.4.8                    pypi_0    pypi
lxml                      6.0.0                    pypi_0    pypi
lz4-c                     1.10.0               h5888daf_1    conda-forge
markupsafe                2.1.5                    pypi_0    pypi
mbstrdecoder              1.1.4                    pypi_0    pypi
more-itertools            10.7.0                   pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
msgpack                   1.1.1                    pypi_0    pypi
multidict                 6.6.3           py311h2dc5d0c_0    conda-forge
multiprocess              0.70.16         py311h9ecbd09_1    conda-forge
ncurses                   6.5                  h2d0b736_3    conda-forge
networkx                  3.3                      pypi_0    pypi
ninja                     1.11.1.4                 pypi_0    pypi
nlohmann_json             3.12.0               h3f2d84a_0    conda-forge
nltk                      3.9.1                    pypi_0    pypi
nsight-compute            2024.1.1.4                    0    nvidia/label/cuda-12.4.1
numba                     0.61.2                   pypi_0    pypi
numexpr                   2.11.0                   pypi_0    pypi
numpy                     2.2.6                    pypi_0    pypi
nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
nvidia-cusparselt-cu12    0.6.2                    pypi_0    pypi
nvidia-ml-py              12.575.51                pypi_0    pypi
nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
openssl                   3.5.1                h7b32b05_0    conda-forge
orc                       2.1.3                h61e0c1e_0    conda-forge
packaging                 25.0               pyh29332c3_1    conda-forge
pandas                    2.3.1           py311hed34c8f_0    conda-forge
pathvalidate              3.3.1                    pypi_0    pypi
peft                      0.16.0                   pypi_0    pypi
pillow                    11.0.0                   pypi_0    pypi
pip                       25.1.1             pyh8b19718_0    conda-forge
portalocker               3.2.0                    pypi_0    pypi
prometheus-cpp            1.3.0                ha5d0236_0    conda-forge
propcache                 0.3.1           py311h2dc5d0c_0    conda-forge
protobuf                  6.31.1                   pypi_0    pypi
psutil                    7.0.0                    pypi_0    pypi
py-cpuinfo                9.0.0                    pypi_0    pypi
pyarrow                   21.0.0          py311h38be061_0    conda-forge
pyarrow-core              21.0.0          py311h342b5a4_0_cpu    conda-forge
pybind11                  3.0.0                    pypi_0    pypi
pycparser                 2.22               pyh29332c3_1    conda-forge
pydantic                  2.11.7                   pypi_0    pypi
pydantic-core             2.33.2                   pypi_0    pypi
pysocks                   1.7.1              pyha55dd90_7    conda-forge
pytablewriter             1.2.1                    pypi_0    pypi
python                    3.11.13         h9e4cc4f_0_cpython    conda-forge
python-dateutil           2.9.0.post0        pyhe01879c_2    conda-forge
python-gil                3.11.13              hd8ed1ab_0    conda-forge
python-tzdata             2025.2             pyhd8ed1ab_0    conda-forge
python-xxhash             3.5.0           py311h9ecbd09_2    conda-forge
python_abi                3.11                    8_cp311    conda-forge
pytz                      2025.2             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.2           py311h2dc5d0c_2    conda-forge
re2                       2025.07.22           h5a314c3_0    conda-forge
readline                  8.2                  h8c095d6_2    conda-forge
regex                     2024.11.6       py311h9ecbd09_0    conda-forge
requests                  2.32.4             pyhd8ed1ab_0    conda-forge
rouge-score               0.1.2                    pypi_0    pypi
s2n                       1.5.23               h8e187f5_0    conda-forge
sacrebleu                 2.5.1                    pypi_0    pypi
safetensors               0.5.3           py311h9e33e62_0    conda-forge
scikit-learn              1.7.1                    pypi_0    pypi
scipy                     1.16.1                   pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
setuptools                80.9.0             pyhff2d567_0    conda-forge
six                       1.17.0             pyhe01879c_1    conda-forge
snappy                    1.2.2                h03e3b7b_0    conda-forge
sqlitedict                2.1.0                    pypi_0    pypi
sympy                     1.13.1                   pypi_0    pypi
tabledata                 1.3.4                    pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
tcolorpy                  0.1.7                    pypi_0    pypi
threadpoolctl             3.6.0                    pypi_0    pypi
tinybenchmarks            1.0.0                    pypi_0    pypi
tk                        8.6.13          noxft_hd72426e_102    conda-forge
tokenizers                0.21.4          py311hffbc7eb_0    conda-forge
torch                     2.6.0+cu124              pypi_0    pypi
torchaudio                2.6.0+cu124              pypi_0    pypi
torchvision               0.21.0+cu124             pypi_0    pypi
tqdm                      4.67.1             pyhd8ed1ab_1    conda-forge
tqdm-multiprocess         0.0.11                   pypi_0    pypi
transformers              4.52.4             pyhd8ed1ab_0    conda-forge
triton                    3.2.0                    pypi_0    pypi
typepy                    1.3.4                    pypi_0    pypi
typing-extensions         4.14.1               h4440ef1_0    conda-forge
typing-inspection         0.4.1                    pypi_0    pypi
typing_extensions         4.14.1             pyhe01879c_0    conda-forge
tzdata                    2025b                h78e105d_0    conda-forge
urllib3                   2.5.0              pyhd8ed1ab_0    conda-forge
uv                        0.8.3                heb9285d_0    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
word2number               1.1                      pypi_0    pypi
xxhash                    0.8.3                hb47aa4a_0    conda-forge
yaml                      0.2.5                h280c20c_3    conda-forge
yarl                      1.20.1          py311h2dc5d0c_0    conda-forge
zlib                      1.3.1                hb9d3cd8_2    conda-forge
zstandard                 0.23.0          py311h9ecbd09_2    conda-forge
zstd                      1.5.7                hb8e6e7a_2    conda-forge

```


Nvm, it seems there was an update a short while ago that fixed this issue. New clone and reinstall of the enviroment fixed it.