Repository: prometheus-eval/prometheus-eval
Issue: Question about RuntimeError
Root cause: Environment Incompatibility

Hi, at firs, I want to say thank for you guys great work.
It is very impressive.

When I run 
`from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("prometheus-eval/prometheus-7b-v2.0")
tokenizer = AutoTokenizer.from_pretrained("prometheus-eval/prometheus-7b-v2.0")`

I faced with 
RuntimeError: Failed to import transformers.models.mistral.modeling_mistral because of the following error (look up to see its traceback): 'My workspace path' : flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

I installed flash_atten with 'pip install flash-attn --no-build-isolation'
I think it is because version of pytorch and flash-atten are not match.
Or the version of cuda and flash-atten are not match..

Of course, when I delete flash-atten, it is work, but super slow..

Anyway, below is my env

-------------------------------------------------------------------------------------------------
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aiofiles                  23.2.1                   pypi_0    pypi
aiohttp                   3.9.3                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
altair                    5.2.0                    pypi_0    pypi
annotated-types           0.6.0                    pypi_0    pypi
antlr4-python3-runtime    4.9.3                    pypi_0    pypi
anyio                     4.3.0                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
argon2-cffi               23.1.0                   pypi_0    pypi
argon2-cffi-bindings      21.2.0                   pypi_0    pypi
arrow                     1.3.0                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
async-lru                 2.0.4                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
av                        12.0.0                   pypi_0    pypi
babel                     2.15.0                   pypi_0    pypi
beautifulsoup4            4.12.3                   pypi_0    pypi
black                     23.7.0                   pypi_0    pypi
blas                      1.0                         mkl  
bleach                    6.1.0                    pypi_0    pypi
blinker                   1.7.0                    pypi_0    pypi
braceexpand               0.1.7                    pypi_0    pypi
brotli-python             1.0.9           py310h6a678d5_7  
bzip2                     1.0.8                h5eee18b_5  
ca-certificates           2024.3.11            h06a4308_0  
cachetools                5.3.3                    pypi_0    pypi
certifi                   2024.2.2        py310h06a4308_0  
cffi                      1.16.0                   pypi_0    pypi
chardet                   5.1.0                    pypi_0    pypi
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.1.7                    pypi_0    pypi
clip                      1.0                      pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
cmake                     3.28.4                   pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
comm                      0.2.2                    pypi_0    pypi
contourpy                 1.2.0                    pypi_0    pypi
cryptography              42.0.5                   pypi_0    pypi
cuda-cudart               11.8.89                       0    nvidia
cuda-cupti                11.8.87                       0    nvidia
cuda-libraries            11.8.0                        0    nvidia
cuda-nvrtc                11.8.89                       0    nvidia
cuda-nvtx                 11.8.86                       0    nvidia
cuda-runtime              11.8.0                        0    nvidia
cycler                    0.12.1                   pypi_0    pypi
debugpy                   1.8.1                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
diskcache                 5.6.3                    pypi_0    pypi
distlib                   0.3.8                    pypi_0    pypi
distro                    1.9.0                    pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
editables                 0.5                      pypi_0    pypi
einops                    0.7.0                    pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
fairscale                 0.4.13                   pypi_0    pypi
fastapi                   0.110.0                  pypi_0    pypi
fastchat                  0.1.0                    pypi_0    pypi
fastjsonschema            2.19.1                   pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
ffmpy                     0.3.2                    pypi_0    pypi
filelock                  3.13.1          py310h06a4308_0  
fire                      0.6.0                    pypi_0    pypi
flatbuffers               24.3.25                  pypi_0    pypi
fonttools                 4.50.0                   pypi_0    pypi
fqdn                      1.5.1                    pypi_0    pypi
freetype                  2.12.1               h4a9f257_0  
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.3.1                 pypi_0    pypi
ftfy                      6.2.0                    pypi_0    pypi
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.42                   pypi_0    pypi
gmp                       6.2.1                h295c915_3  
gmpy2                     2.1.2           py310heeb90bb_0  
gnutls                    3.6.15               he1e5248_0  
gradio                    4.23.0                   pypi_0    pypi
gradio-client             0.14.0                   pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
hatch                     1.9.4                    pypi_0    pypi
hatchling                 1.21.1                   pypi_0    pypi
httpcore                  1.0.4                    pypi_0    pypi
httptools                 0.6.1                    pypi_0    pypi
httpx                     0.27.0                   pypi_0    pypi
huggingface-hub           0.22.0                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
hyperlink                 21.0.0                   pypi_0    pypi
idna                      3.4             py310h06a4308_0  
imageio                   2.34.0                   pypi_0    pypi
imageio-ffmpeg            0.4.9                    pypi_0    pypi
imhist                    0.0.4                    pypi_0    pypi
importlib-metadata        7.1.0                    pypi_0    pypi
importlib-resources       6.4.0                    pypi_0    pypi
imwatermark               0.0.2                    pypi_0    pypi
install                   1.3.5                    pypi_0    pypi
intel-openmp              2023.1.0         hdb19cb5_46306  
interegular               0.3.3                    pypi_0    pypi
invisible-watermark       0.2.0                    pypi_0    pypi
ipykernel                 6.29.4                   pypi_0    pypi
ipython                   8.24.0                   pypi_0    pypi
isoduration               20.11.0                  pypi_0    pypi
jaraco-classes            3.3.1                    pypi_0    pypi
jaraco-context            4.3.0                    pypi_0    pypi
jaraco-functools          4.0.0                    pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jeepney                   0.8.0                    pypi_0    pypi
jinja2                    3.1.3           py310h06a4308_0  
joblib                    1.4.2                    pypi_0    pypi
jpeg                      9e                   h5eee18b_1  
json5                     0.9.25                   pypi_0    pypi
jsonpointer               2.4                      pypi_0    pypi
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
jupyter-client            8.6.1                    pypi_0    pypi
jupyter-core              5.7.2                    pypi_0    pypi
jupyter-events            0.10.0                   pypi_0    pypi
jupyter-lsp               2.2.5                    pypi_0    pypi
jupyter-server            2.14.0                   pypi_0    pypi
jupyter-server-terminals  0.5.3                    pypi_0    pypi
jupyterlab                4.1.8                    pypi_0    pypi
jupyterlab-pygments       0.3.0                    pypi_0    pypi
jupyterlab-server         2.27.1                   pypi_0    pypi
keyring                   25.0.0                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
kornia                    0.6.9                    pypi_0    pypi
kornia-rs                 0.1.2                    pypi_0    pypi
lame                      3.100                h7b6447c_0  
lark                      1.1.9                    pypi_0    pypi
lazy-loader               0.3                      pypi_0    pypi
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      3.0                  h295c915_0  
libcublas                 11.11.3.6                     0    nvidia
libcufft                  10.9.0.58                     0    nvidia
libcufile                 1.9.0.20                      0    nvidia
libcurand                 10.3.5.119                    0    nvidia
libcusolver               11.4.1.48                     0    nvidia
libcusparse               11.7.5.86                     0    nvidia
libdeflate                1.17                 h5eee18b_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libiconv                  1.16                 h7f8727e_2  
libidn2                   2.3.4                h5eee18b_0  
libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
libnpp                    11.8.0.86                     0    nvidia
libnsl                    2.0.1                hd590300_0    conda-forge
libnvjpeg                 11.9.0.86                     0    nvidia
libpng                    1.6.39               h5eee18b_0  
libsqlite                 3.45.2               h2797004_0    conda-forge
libstdcxx-ng              11.2.0               h1234567_1  
libtasn1                  4.19.0               h5eee18b_0  
libtiff                   4.5.1                h6a678d5_0  
libunistring              0.9.10               h27cfd23_0  
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                h5eee18b_0  
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lightning-utilities       0.11.1                   pypi_0    pypi
lit                       18.1.2                   pypi_0    pypi
llvm-openmp               14.0.6               h9e868ea_0  
llvmlite                  0.42.0                   pypi_0    pypi
lm-format-enforcer        0.9.8                    pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_0  
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.3           py310h5eee18b_0  
matplotlib                3.8.3                    pypi_0    pypi
matplotlib-inline         0.1.7                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mistune                   3.0.2                    pypi_0    pypi
mkl                       2023.1.0         h213fc3f_46344  
mkl-service               2.4.0           py310h5eee18b_1  
mkl_fft                   1.3.8           py310h5eee18b_0  
mkl_random                1.2.4           py310hdb19cb5_0  
more-itertools            10.2.0                   pypi_0    pypi
mpc                       1.1.0                h10f8cd9_1  
mpfr                      4.0.2                hb69a4c5_1  
mpmath                    1.3.0           py310h06a4308_0  
msgpack                   1.0.8                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
natsort                   8.4.0                    pypi_0    pypi
nbclient                  0.10.0                   pypi_0    pypi
nbconvert                 7.16.4                   pypi_0    pypi
nbformat                  5.10.4                   pypi_0    pypi
ncurses                   6.4.20240210         h59595ed_0    conda-forge
nest-asyncio              1.6.0                    pypi_0    pypi
nettle                    3.7.3                hbbd107a_1  
networkx                  3.1             py310h06a4308_0  
ninja                     1.11.1.1                 pypi_0    pypi
notebook                  7.1.3                    pypi_0    pypi
notebook-shim             0.2.4                    pypi_0    pypi
numba                     0.59.1                   pypi_0    pypi
numpy                     1.26.4          py310h5f9d8c6_0  
numpy-base                1.26.4          py310hb5e798b_0  
nvidia-cublas-cu11        11.10.3.66               pypi_0    pypi
nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
nvidia-cuda-cupti-cu11    11.7.101                 pypi_0    pypi
nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-nvrtc-cu11    11.7.99                  pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-runtime-cu11  11.7.99                  pypi_0    pypi
nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
nvidia-cudnn-cu11         8.5.0.96                 pypi_0    pypi
nvidia-cudnn-cu12         8.9.2.26                 pypi_0    pypi
nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi
nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
nvidia-curand-cu11        10.2.10.91               pypi_0    pypi
nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
nvidia-cusolver-cu11      11.4.0.1                 pypi_0    pypi
nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
nvidia-cusparse-cu11      11.7.4.91                pypi_0    pypi
nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
nvidia-ml-py              12.550.52                pypi_0    pypi
nvidia-nccl-cu11          2.14.3                   pypi_0    pypi
nvidia-nccl-cu12          2.19.3                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.4.99                  pypi_0    pypi
nvidia-nvtx-cu11          11.7.91                  pypi_0    pypi
nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
omegaconf                 2.3.0                    pypi_0    pypi
onnxruntime               1.17.1                   pypi_0    pypi
open-clip-torch           2.24.0                   pypi_0    pypi
openai                    1.26.0                   pypi_0    pypi
opencv-python             4.6.0.66                 pypi_0    pypi
opencv-python-headless    4.9.0.80                 pypi_0    pypi
openh264                  2.1.1                h4ff587b_0  
openjpeg                  2.4.0                h3ad879b_0  
openssl                   3.2.1                hd590300_1    conda-forge
orjson                    3.9.15                   pypi_0    pypi
outlines                  0.0.34                   pypi_0    pypi
overrides                 7.7.0                    pypi_0    pypi
packaging                 23.2                     pypi_0    pypi
pandas                    2.2.1                    pypi_0    pypi
pandocfilters             1.5.1                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
pathspec                  0.12.1                   pypi_0    pypi
pexpect                   4.9.0                    pypi_0    pypi
pillow                    10.2.0          py310h5eee18b_0  
pip                       23.3.1          py310h06a4308_0  
platformdirs              4.2.0                    pypi_0    pypi
pluggy                    1.4.0                    pypi_0    pypi
pooch                     1.8.1                    pypi_0    pypi
prometheus-client         0.20.0                   pypi_0    pypi
prometheus-eval           0.1.13                   pypi_0    pypi
prometheus-fastapi-instrumentator 7.0.0                    pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
protobuf                  3.20.3                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pudb                      2024.1                   pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
py-cpuinfo                9.0.0                    pypi_0    pypi
pyarrow                   15.0.2                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pydantic                  2.6.4                    pypi_0    pypi
pydantic-core             2.16.3                   pypi_0    pypi
pydeck                    0.8.1b0                  pypi_0    pypi
pydub                     0.25.1                   pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pymatting                 1.1.12                   pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pysocks                   1.7.1           py310h06a4308_0  
python                    3.10.14         hd12c33a_0_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
python-dotenv             1.0.1                    pypi_0    pypi
python-json-logger        2.0.7                    pypi_0    pypi
python-multipart          0.0.9                    pypi_0    pypi
pytorch-cuda              11.8                 h7e8668a_5    pytorch
pytorch-lightning         2.0.1                    pypi_0    pypi
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2024.1                   pypi_0    pypi
pywavelets                1.5.0                    pypi_0    pypi
pyyaml                    6.0.1           py310h5eee18b_0  
pyzmq                     26.0.3                   pypi_0    pypi
ray                       2.20.0                   pypi_0    pypi
readline                  8.2                  h5eee18b_0  
referencing               0.34.0                   pypi_0    pypi
regex                     2023.12.25               pypi_0    pypi
rembg                     2.0.56                   pypi_0    pypi
requests                  2.31.0          py310h06a4308_1  
rfc3339-validator         0.1.4                    pypi_0    pypi
rfc3986-validator         0.1.1                    pypi_0    pypi
rich                      13.7.1                   pypi_0    pypi
rpds-py                   0.18.0                   pypi_0    pypi
ruff                      0.3.4                    pypi_0    pypi
safetensors               0.4.2                    pypi_0    pypi
scikit-image              0.22.0                   pypi_0    pypi
scipy                     1.12.0                   pypi_0    pypi
sdata                     0.0.1                     dev_0    <develop>
secretstorage             3.3.3                    pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
send2trash                1.8.3                    pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
sentry-sdk                1.43.0                   pypi_0    pypi
setproctitle              1.3.3                    pypi_0    pypi
setuptools                68.2.2          py310h06a4308_0  
sgm                       0.1.0                    pypi_0    pypi
shellingham               1.5.4                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
smmap                     5.0.1                    pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
soupsieve                 2.5                      pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0  
stack-data                0.6.3                    pypi_0    pypi
starlette                 0.36.3                   pypi_0    pypi
streamlit                 1.32.2                   pypi_0    pypi
streamlit-keyup           0.2.0                    pypi_0    pypi
sympy                     1.12            py310h06a4308_0  
tbb                       2021.8.0             hdb19cb5_0  
tenacity                  8.2.3                    pypi_0    pypi
tensorboardx              2.6                      pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
terminado                 0.18.1                   pypi_0    pypi
tifffile                  2024.2.12                pypi_0    pypi
tiktoken                  0.6.0                    pypi_0    pypi
timm                      0.9.16                   pypi_0    pypi
tinycss2                  1.3.0                    pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tokenizers                0.19.1                   pypi_0    pypi
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
tomli-w                   1.0.0                    pypi_0    pypi
tomlkit                   0.12.0                   pypi_0    pypi
toolz                     0.12.1                   pypi_0    pypi
torch                     2.2.1                    pypi_0    pypi
torchaudio                2.1.1               py310_cu118    pytorch
torchdata                 0.6.1                    pypi_0    pypi
torchmetrics              1.3.2                    pypi_0    pypi
torchvision               0.16.1              py310_cu118    pytorch
tornado                   6.4                      pypi_0    pypi
tqdm                      4.66.2                   pypi_0    pypi
traitlets                 5.14.3                   pypi_0    pypi
transformers              4.40.2                   pypi_0    pypi
triton                    2.3.0                    pypi_0    pypi
trove-classifiers         2024.3.25                pypi_0    pypi
typer                     0.10.0                   pypi_0    pypi
types-python-dateutil     2.9.0.20240316           pypi_0    pypi
typing_extensions         4.9.0           py310h06a4308_1  
tzdata                    2024.1                   pypi_0    pypi
uri-template              1.3.0                    pypi_0    pypi
urllib3                   1.26.18                  pypi_0    pypi
urwid                     2.6.10                   pypi_0    pypi
urwid-readline            0.14                     pypi_0    pypi
userpath                  1.9.2                    pypi_0    pypi
uvicorn                   0.29.0                   pypi_0    pypi
uvloop                    0.19.0                   pypi_0    pypi
virtualenv                20.25.1                  pypi_0    pypi
vllm                      0.4.2                    pypi_0    pypi
vllm-nccl-cu12            2.18.1.0.4.0             pypi_0    pypi
wandb                     0.16.5                   pypi_0    pypi
watchdog                  4.0.0                    pypi_0    pypi
watchfiles                0.21.0                   pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
webcolors                 1.13                     pypi_0    pypi
webdataset                0.2.86                   pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.8.0                    pypi_0    pypi
websockets                11.0.3                   pypi_0    pypi
wheel                     0.41.2          py310h06a4308_0  
xformers                  0.0.26.post1             pypi_0    pypi
xz                        5.4.6                h5eee18b_0  
yaml                      0.2.5                h7b6447c_0  
yarl                      1.9.4                    pypi_0    pypi
zipp                      3.18.1                   pypi_0    pypi
zlib                      1.2.13               hd590300_5    conda-forge
zstandard                 0.22.0                   pypi_0    pypi
zstd                      1.5.5                hc292b87_0  



Hi @JinSeoung-Oh! Thanks for your interest in our work! 

I haven't thoroughly tested Prometheus model on `transformers` since we mainly utilized `vllm`, so it might take a while to inspect on this issue 😞 Meanwhile, if speed is important for you, can you tryout using our library or use`vllm` instead? It should be much faster!

The model starts (and works) using Huggingface's text-generation-inference server, making it available as an API endpoint. This clearly suggests that it can work with Huggingface transformers, which is used by the server.

However, the library unfortunately doesn't currently allow you to specify an API endpoint as a model, only running it locally (see #26).

This doesn't solve your specific issue, but it means there's no fundamental problem with running Prometheus using `transformers`.