Repository: pytorch/benchmark
Issue: Subprocess terminates with code 11
Root cause: External Dependency Breakage

Hi,
I'm running the benchmark on a computer cluster and constantly getting the following error with `python test.py -k "test_<model>_train_cuda"` for all the models.
```
ERROR: test_DALLE2_pytorch_eval_cuda (__main__.TestBenchmark)
----------------------------------------------------------------------
RuntimeError: Failed to bootstrap worker (timeout):
    working_dir: /tmp/tmpc2hlzl0j
    stdout:


    stderr:
        Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux
        Type "help", "copyright", "credits" or "license" for more information.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/firoozh1/w/cuda118-benchmark/benchmark/test.py", line 86, in eval_fn
    task = ModelTask(path, timeout=TIMEOUT)
  File "/home/firoozh1/w/cuda118-benchmark/benchmark/torchbenchmark/__init__.py", line 254, in __init__
    self._worker = Worker(timeout=timeout, extra_env=extra_env, save_output_dir=save_output_dir)
  File "/home/firoozh1/w/cuda118-benchmark/benchmark/components/_impl/workers/subprocess_worker.py", line 138, in __init__
    self._bootstrap_worker()
  File "/home/firoozh1/w/cuda118-benchmark/benchmark/components/_impl/workers/subprocess_worker.py", line 265, in _bootstrap_worker
    raise e from RuntimeError(
  File "/home/firoozh1/w/cuda118-benchmark/benchmark/components/_impl/workers/subprocess_worker.py", line 254, in _bootstrap_worker
    result = bootstrap_pipe.read()
  File "/home/firoozh1/w/cuda118-benchmark/benchmark/components/_impl/workers/subprocess_rpc.py", line 238, in read
    msg_size = struct.unpack(_ULL, self._read(_ULL_SIZE))[0]
  File "/home/firoozh1/w/cuda118-benchmark/benchmark/components/_impl/workers/subprocess_rpc.py", line 227, in _read
    raise IOError(f"Subprocess terminates with code {int.from_bytes(msg, sys.byteorder)}")
OSError: Subprocess terminates with code 11

----------------------------------------------------------------------
Ran 1 test in 1.108s

FAILED (errors=1)
```
### Setup
I'm using Anaconda with the following packages installed:
```
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
aiohttp                   3.9.1                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
annotated-types           0.6.0                    pypi_0    pypi
antlr4-python3-runtime    4.9.3                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
astroid                   3.0.1                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
audioread                 3.0.1                    pypi_0    pypi
babel                     2.13.1                   pypi_0    pypi
beartype                  0.15.0                   pypi_0    pypi
beautifulsoup4            4.12.2                   pypi_0    pypi
bert-pytorch              0.0.1a4                   dev_0    <develop>
bibtexparser              2.0.0b4                  pypi_0    pypi
black                     23.11.0                  pypi_0    pypi
blas                      1.0                         mkl
blinker                   1.7.0                    pypi_0    pypi
blis                      0.7.11                   pypi_0    pypi
blobfile                  2.1.1                    pypi_0    pypi
bokeh                     3.3.2                    pypi_0    pypi
braceexpand               0.1.7                    pypi_0    pypi
brotli-python             1.0.9           py310h6a678d5_7
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2023.11.17           hbcca054_0    conda-forge
cachetools                5.3.2                    pypi_0    pypi
cairo                     1.16.0               hb05425b_5
cardboardlint             1.3.1                    pypi_0    pypi
catalogue                 2.0.10                   pypi_0    pypi
cerberus                  1.3.5                    pypi_0    pypi
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310h5eee18b_0
charset-normalizer        2.0.4              pyhd3eb1b0_0
click                     8.1.7                    pypi_0    pypi
clip-anytorch             2.5.2                    pypi_0    pypi
clldutils                 3.22.0                   pypi_0    pypi
cloudpathlib              0.16.0                   pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
coca-pytorch              0.1.0                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
colorlog                  6.8.0                    pypi_0    pypi
confection                0.1.4                    pypi_0    pypi
contourpy                 1.2.0                    pypi_0    pypi
cryptography              41.0.7          py310hdda0065_0
cssselect2                0.7.0                    pypi_0    pypi
csvw                      3.2.1                    pypi_0    pypi
cuda-cudart               11.8.89                       0    nvidia
cuda-cupti                11.8.87                       0    nvidia
cuda-libraries            11.8.0                        0    nvidia
cuda-nvrtc                11.8.89                       0    nvidia
cuda-nvtx                 11.8.86                       0    nvidia
cuda-runtime              11.8.0                        0    nvidia
cycler                    0.12.1                   pypi_0    pypi
cymem                     2.0.8                    pypi_0    pypi
dall-e                    0.1                      pypi_0    pypi
dalle2-pytorch            1.14.2                   pypi_0    pypi
datasets                  2.15.0                   pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
detectron2                0.6                      pypi_0    pypi
diffusers                 0.20.2                   pypi_0    pypi
dill                      0.3.7                    pypi_0    pypi
distlib                   0.3.7                    pypi_0    pypi
distro                    1.8.0                    pypi_0    pypi
dlinfo                    1.2.1                    pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
dominate                  2.9.0                    pypi_0    pypi
effdet                    0.3.0                    pypi_0    pypi
einops                    0.7.0                    pypi_0    pypi
ema-pytorch               0.3.1                    pypi_0    pypi
embedding-reader          1.5.1                    pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
expat                     2.5.0                hcb278e6_1    conda-forge
expecttest                0.1.6              pyhd8ed1ab_0    conda-forge
fastnlp                   0.6.0                    pypi_0    pypi
ffmpeg                    4.2.2                h20bf706_0
ffmpeg-python             0.2.0                    pypi_0    pypi
filelock                  3.13.1          py310h06a4308_0
flask                     3.0.0                    pypi_0    pypi
fontconfig                2.14.1               h52c9d5c_1
fonttools                 4.46.0                   pypi_0    pypi
freetype                  2.12.1               h4a9f257_0
fribidi                   1.0.10               h36c2ea0_0    conda-forge
frozenlist                1.4.0                    pypi_0    pypi
fsspec                    2023.10.0                pypi_0    pypi
ftfy                      6.1.3                    pypi_0    pypi
functorch                 1.14.0a0+b71aa0b          pypi_0    pypi
future                    0.18.3                   pypi_0    pypi
fvcore                    0.1.5.post20221221          pypi_0    pypi
gdown                     4.7.1                    pypi_0    pypi
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h5eee18b_3
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.40                   pypi_0    pypi
glib                      2.78.3               hfc55251_0    conda-forge
glib-tools                2.78.3               hfc55251_0    conda-forge
gmp                       6.2.1                h295c915_3
gmpy2                     2.1.2           py310heeb90bb_0
gnutls                    3.6.15               he1e5248_0
google-auth               2.25.2                   pypi_0    pypi
google-auth-oauthlib      1.1.0                    pypi_0    pypi
graphite2                 1.3.13            h58526e2_1001    conda-forge
grpcio                    1.60.0                   pypi_0    pypi
gym                       0.26.2                   pypi_0    pypi
gym-notices               0.0.8                    pypi_0    pypi
h5py                      3.10.0                   pypi_0    pypi
harfbuzz                  4.4.1                hf9f4e7c_0    conda-forge
higher                    0.2.1                    pypi_0    pypi
html5lib                  1.1                      pypi_0    pypi
huggingface-hub           0.19.4                   pypi_0    pypi
hydra-core                1.3.2                    pypi_0    pypi
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4             py310h06a4308_0
imageio                   2.33.1                   pypi_0    pypi
importlib-metadata        7.0.0                    pypi_0    pypi
inflect                   7.0.0                    pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
intel-openmp              2023.1.0         hdb19cb5_46306
iopath                    0.1.9                    pypi_0    pypi
isodate                   0.6.1                    pypi_0    pypi
isort                     5.13.0                   pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jinja2                    3.1.2           py310h06a4308_0
joblib                    1.3.2                    pypi_0    pypi
jpeg                      9e                   h5eee18b_1
jsonpatch                 1.33                     pypi_0    pypi
jsonpointer               2.4                      pypi_0    pypi
jsonschema                4.20.0                   pypi_0    pypi
jsonschema-specifications 2023.11.2                pypi_0    pypi
kaldi-io                  0.9.8                    pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
kornia                    0.7.0                    pypi_0    pypi
lame                      3.100                h7b6447c_0
lameenc                   1.7.0                    pypi_0    pypi
langcodes                 3.3.0                    pypi_0    pypi
langdetect                1.0.9                    pypi_0    pypi
language-tags             1.2.0                    pypi_0    pypi
lazy-loader               0.3                      pypi_0    pypi
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.38                 h1181459_1
lerc                      3.0                  h295c915_0
libcublas                 11.11.3.6                     0    nvidia
libcufft                  10.9.0.58                     0    nvidia
libcufile                 1.8.1.2                       0    nvidia
libcurand                 10.3.4.101                    0    nvidia
libcusolver               11.4.1.48                     0    nvidia
libcusparse               11.7.5.86                     0    nvidia
libdeflate                1.17                 h5eee18b_1
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.4                h6a678d5_0
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libglib                   2.78.3               h783c2da_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libidn2                   2.3.4                h5eee18b_0
libjpeg-turbo             2.0.0                h9bf148f_0    pytorch-nightly
libnpp                    11.8.0.86                     0    nvidia
libnvjpeg                 11.9.0.86                     0    nvidia
libopus                   1.3.1                h7b6447c_0
libpng                    1.6.39               h5eee18b_0
librosa                   0.9.2                    pypi_0    pypi
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libtasn1                  4.19.0               h5eee18b_0
libtiff                   4.5.1                h6a678d5_0
libunistring              0.9.10               h27cfd23_0
libuuid                   1.41.5               h5eee18b_0
libvpx                    1.7.0                h439df22_0
libwebp                   1.3.2                h11a3e52_0
libwebp-base              1.3.2                h5eee18b_0
libxcb                    1.16                 hd590300_0    conda-forge
libxml2                   2.9.14               h22db469_4    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lightning-utilities       0.10.0                   pypi_0    pypi
llvm-openmp               14.0.6               h9e868ea_0
llvmlite                  0.41.1                   pypi_0    pypi
lxml                      4.9.3                    pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_0
magma-cuda118             2.6.1                         1    pytorch
markdown                  3.5.1                    pypi_0    pypi
markupsafe                2.1.1           py310h7f8727e_0
matplotlib                3.8.2                    pypi_0    pypi
mccabe                    0.7.0                    pypi_0    pypi
mkl                       2023.1.0         h213fc3f_46344
mkl-service               2.4.0           py310h5eee18b_1
mkl_fft                   1.3.8           py310h5eee18b_0
mkl_random                1.2.4           py310hdb19cb5_0
mpc                       1.1.0                h10f8cd9_1
mpfr                      4.0.2                hb69a4c5_1
mplcursors                0.5.2                    pypi_0    pypi
mpmath                    1.3.0           py310h06a4308_0
multidict                 6.0.4                    pypi_0    pypi
multiprocess              0.70.15                  pypi_0    pypi
murmurhash                1.0.10                   pypi_0    pypi
musdb                     0.4.2                    pypi_0    pypi
museval                   0.4.1                    pypi_0    pypi
mypy                      1.7.1                    pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0
nettle                    3.7.3                hbbd107a_1
networkx                  3.1             py310h06a4308_0
nose                      1.3.7                    pypi_0    pypi
numba                     0.58.1                   pypi_0    pypi
numpy                     1.26.2          py310h5f9d8c6_0
numpy-base                1.26.2          py310hb5e798b_0
oauthlib                  3.2.2                    pypi_0    pypi
omegaconf                 2.3.0                    pypi_0    pypi
onnx                      1.15.0                   pypi_0    pypi
opacus                    1.4.0                    pypi_0    pypi
open-clip-torch           2.23.0                   pypi_0    pypi
openh264                  2.1.1                h4ff587b_0
openjpeg                  2.4.0                h3ad879b_0
openssl                   3.2.0                hd590300_1    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 23.2                     pypi_0    pypi
pandas                    2.1.4                    pypi_0    pypi
pango                     1.50.7               h05da053_0
pathspec                  0.12.1                   pypi_0    pypi
pcre2                     10.42                hcad00b1_0    conda-forge
pep517                    0.13.1                   pypi_0    pypi
phonemizer                3.2.1                    pypi_0    pypi
pillow                    10.0.1          py310ha6cbd5a_0
pip                       23.3.1          py310h06a4308_0
pip-api                   0.0.30                   pypi_0    pypi
pipreqs                   0.4.13                   pypi_0    pypi
pixman                    0.42.2               h59595ed_0    conda-forge
platformdirs              4.1.0                    pypi_0    pypi
plette                    0.4.4                    pypi_0    pypi
pluggy                    1.3.0                    pypi_0    pypi
pooch                     1.8.0                    pypi_0    pypi
portalocker               2.8.2                    pypi_0    pypi
preshed                   3.0.9                    pypi_0    pypi
prettytable               3.9.0                    pypi_0    pypi
psutil                    5.9.6                    pypi_0    pypi
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
py-cpuinfo                9.0.0                    pypi_0    pypi
pyaml                     23.9.7                   pypi_0    pypi
pyarrow                   14.0.1                   pypi_0    pypi
pyarrow-hotfix            0.6                      pypi_0    pypi
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pyclipper                 1.3.0.post5              pypi_0    pypi
pycocotools               2.0.6                    pypi_0    pypi
pycparser                 2.21               pyhd3eb1b0_0
pycryptodomex             3.19.0                   pypi_0    pypi
pydantic                  2.5.2                    pypi_0    pypi
pydantic-core             2.14.5                   pypi_0    pypi
pydot                     1.4.2                    pypi_0    pypi
pydyf                     0.8.0                    pypi_0    pypi
pygame                    2.5.2                    pypi_0    pypi
pylatexenc                2.10                     pypi_0    pypi
pylint                    3.0.2                    pypi_0    pypi
pynvml                    11.5.0                   pypi_0    pypi
pyopenssl                 23.2.0          py310h06a4308_0
pyparsing                 3.1.1                    pypi_0    pypi
pypdfium2                 3.21.1                   pypi_0    pypi
pyphen                    0.14.0                   pypi_0    pypi
pysbd                     0.3.4                    pypi_0    pypi
pysocks                   1.7.1           py310h06a4308_0
pytest                    7.4.3                    pypi_0    pypi
python                    3.10.13              h955ad1f_0
python-dateutil           2.8.2                    pypi_0    pypi
python-doctr              0.6.1a0                  pypi_0    pypi
pytorch                   2.2.0.dev20231210 py3.10_cuda11.8_cudnn8.7.0_0    pytorch-nightly
pytorch-cuda              11.8                 h7e8668a_5    pytorch-nightly
pytorch-mutex             1.0                        cuda    pytorch-nightly
pytorch-warmup            0.1.1                    pypi_0    pypi
pytz                      2023.3.post1             pypi_0    pypi
pyyaml                    6.0.1           py310h5eee18b_0
rapidfuzz                 2.15.1                   pypi_0    pypi
rdflib                    7.0.0                    pypi_0    pypi
readline                  8.2                  h5eee18b_0
referencing               0.32.0                   pypi_0    pypi
regex                     2023.10.3                pypi_0    pypi
requests                  2.31.0          py310h06a4308_0
requests-oauthlib         1.3.1                    pypi_0    pypi
requirementslib           3.0.0                    pypi_0    pypi
resampy                   0.4.2                    pypi_0    pypi
resize-right              0.0.2                    pypi_0    pypi
rfc3986                   1.5.0                    pypi_0    pypi
rotary-embedding-torch    0.3.3                    pypi_0    pypi
rpds-py                   0.13.2                   pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
safetensors               0.4.1                    pypi_0    pypi
scikit-image              0.22.0                   pypi_0    pypi
scikit-learn              1.3.2                    pypi_0    pypi
scipy                     1.11.4                   pypi_0    pypi
segments                  2.2.1                    pypi_0    pypi
sentencepiece             0.1.99                   pypi_0    pypi
sentry-sdk                1.38.0                   pypi_0    pypi
setproctitle              1.3.3                    pypi_0    pypi
setuptools                68.0.0          py310h06a4308_0
shapely                   1.8.5.post1              pypi_0    pypi
simplejson                3.19.2                   pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
smart-open                6.4.0                    pypi_0    pypi
smmap                     5.0.1                    pypi_0    pypi
soundfile                 0.12.1                   pypi_0    pypi
soupsieve                 2.5                      pypi_0    pypi
spacy                     3.7.2                    pypi_0    pypi
spacy-legacy              3.0.12                   pypi_0    pypi
spacy-loggers             1.0.5                    pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0
srsly                     2.4.8                    pypi_0    pypi
stempeg                   0.2.3                    pypi_0    pypi
sympy                     1.12            py310h06a4308_0
tabulate                  0.9.0                    pypi_0    pypi
tbb                       2021.8.0             hdb19cb5_0
tensorboard               2.15.1                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
tensorboardx              2.6.2.2                  pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
thinc                     8.2.1                    pypi_0    pypi
threadpoolctl             3.2.0                    pypi_0    pypi
tifffile                  2023.12.9                pypi_0    pypi
tinycss2                  1.2.1                    pypi_0    pypi
tk                        8.6.12               h1ccaba5_0
tomli                     2.0.1                    pypi_0    pypi
tomlkit                   0.12.3                   pypi_0    pypi
torch-fidelity            0.3.0                    pypi_0    pypi
torch-geometric           2.4.0                    pypi_0    pypi
torchaudio                2.2.0.dev20231210     py310_cu118    pytorch-nightly
torchmetrics              1.2.1                    pypi_0    pypi
torchmultimodal           0.1.0b0                  pypi_0    pypi
torchtriton               2.1.0+bcad9dabe1           py310    pytorch-nightly
torchvision               0.17.0.dev20231210     py310_cu118    pytorch-nightly
tornado                   6.4                      pypi_0    pypi
tqdm                      4.66.1                   pypi_0    pypi
treetable                 0.2.5                    pypi_0    pypi
typer                     0.9.0                    pypi_0    pypi
typing-extensions         4.9.0                    pypi_0    pypi
typing-inspect            0.9.0                    pypi_0    pypi
tzdata                    2023.3                   pypi_0    pypi
unidecode                 1.3.7                    pypi_0    pypi
uritemplate               4.1.1                    pypi_0    pypi
urllib3                   1.26.18         py310h06a4308_0
vector-quantize-pytorch   1.12.0                   pypi_0    pypi
visdom                    0.2.4                    pypi_0    pypi
wandb                     0.16.1                   pypi_0    pypi
wasabi                    1.1.2                    pypi_0    pypi
wcwidth                   0.2.12                   pypi_0    pypi
weasel                    0.3.4                    pypi_0    pypi
weasyprint                60.1                     pypi_0    pypi
webdataset                0.2.86                   pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.7.0                    pypi_0    pypi
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.41.2          py310h06a4308_0
x-clip                    0.14.4                   pypi_0    pypi
x264                      1!157.20191217       h7b6447c_0
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xxhash                    3.4.1                    pypi_0    pypi
xyzservices               2023.10.1                pypi_0    pypi
xz                        5.4.5                h5eee18b_0
yacs                      0.1.8                    pypi_0    pypi
yaml                      0.2.5                h7b6447c_0
yarg                      0.1.9                    pypi_0    pypi
yarl                      1.9.4                    pypi_0    pypi
zipp                      3.17.0                   pypi_0    pypi
zlib                      1.2.13               hd590300_5    conda-forge
zopfli                    0.2.3                    pypi_0    pypi
zstd                      1.5.5                hc292b87_0
```
I'm also using `gcc 11.3.0` and CUDA `Build cuda_11.8.r11.8/compiler.31833905_0` on a machine with `Tesla V100`.

I'm able to run `python run.py`, but `python test.py` raises an error. Installing every model with `python install.py` give me 'OK' messages; and I don't think the installation is part of the problem.

Yeah we observe similar issue from our CI: https://github.com/pytorch/benchmark/actions/runs/7159070221/job/19491917312. We are looking into it.

Thanks @xuzhao9 for the headsup.
Is there an older version which is working without any problems?

@hsnfirooz It happened ~2weeks ago, so nightly build before Nov 27 should be good

I tried to install the dev builds from Oct 16 and Nov 16 with the following commands for CUDA `11.8` and `12.1` respectively. However the subprocess problem is still there.
`pip install --pre torch==2.2.0.dev20231016 torchvision==0.17.0.dev20231016 torchaudio==2.2.0.dev20231016 -i https://download.pytorch.org/whl/nightly/cu118`
and,
`pip install --pre torch==2.2.0.dev20231116 torchvision==0.17.0.dev20231116 torchaudio==2.2.0.dev20231116 -i https://download.pytorch.org/whl/nightly/cu121`

EDIT: I also used the repository from this commit `361a7756fdf91a29abc4d2b7040e0edae0be3d05` where the build was successful and all the test has been passed from the CLI output.

Do you have any other tips?

I'm facing the same issue.
I fetched the commit fa41cb26a39c112f72f55b3ccacc2bbc502e2649 on Nov 15, which was a successful build and passed all the CLI tests.
I also used `torch==2.2.0.dev20231115 torchvision==0.17.0.dev20231115 torchaudio==2.2.0.dev20231115` which are the nightly builds from the same date.

These are the `conda list` packages:
```
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
absl-py                   2.0.0                    pypi_0    pypi
accelerate                0.25.0                   pypi_0    pypi
annotated-types           0.6.0                    pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
beartype                  0.15.0                   pypi_0    pypi
beautifulsoup4            4.12.2                   pypi_0    pypi
bert-pytorch              0.0.1a4                   dev_0    <develop>
boto3                     1.34.0                   pypi_0    pypi
botocore                  1.34.0                   pypi_0    pypi
braceexpand               0.1.7                    pypi_0    pypi
bs4                       0.0.1                    pypi_0    pypi
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2023.08.22           h06a4308_0  
cachetools                5.3.2                    pypi_0    pypi
certifi                   2022.12.7                pypi_0    pypi
cffi                      1.16.0                   pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
clip-anytorch             2.5.2                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
coca-pytorch              0.1.0                    pypi_0    pypi
dalle2-pytorch            1.14.2                   pypi_0    pypi
distro                    1.8.0                    pypi_0    pypi
einops                    0.7.0                    pypi_0    pypi
ema-pytorch               0.3.1                    pypi_0    pypi
embedding-reader          1.5.1                    pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
ffmpeg-python             0.2.0                    pypi_0    pypi
filelock                  3.9.0                    pypi_0    pypi
fsspec                    2023.12.2                pypi_0    pypi
ftfy                      6.1.3                    pypi_0    pypi
future                    0.18.3                   pypi_0    pypi
google-auth               2.25.2                   pypi_0    pypi
google-auth-oauthlib      1.2.0                    pypi_0    pypi
grpcio                    1.60.0                   pypi_0    pypi
huggingface-hub           0.19.4                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
imageio                   2.33.1                   pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
iopath                    0.1.10                   pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
joblib                    1.3.2                    pypi_0    pypi
jsonschema                4.20.0                   pypi_0    pypi
jsonschema-specifications 2023.11.2                pypi_0    pypi
kornia                    0.7.0                    pypi_0    pypi
lameenc                   1.7.0                    pypi_0    pypi
lazy-loader               0.3                      pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libcst                    1.1.0                    pypi_0    pypi
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
lightning-utilities       0.10.0                   pypi_0    pypi
magma-cuda118             2.6.1                         1    pytorch
markdown                  3.5.1                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
monkeytype                23.3.0                   pypi_0    pypi
mpmath                    1.2.1                    pypi_0    pypi
musdb                     0.4.2                    pypi_0    pypi
museval                   0.4.1                    pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.0rc1                   pypi_0    pypi
numpy                     1.23.5                   pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
open-clip-torch           2.23.0                   pypi_0    pypi
opencv-python             4.8.1.78                 pypi_0    pypi
openssl                   3.0.12               h7f8727e_0  
packaging                 23.2                     pypi_0    pypi
pandas                    2.1.4                    pypi_0    pypi
patch                     1.16                     pypi_0    pypi
pillow                    9.3.0                    pypi_0    pypi
pip                       23.3.1          py310h06a4308_0  
pluggy                    1.3.0                    pypi_0    pypi
portalocker               2.8.2                    pypi_0    pypi
protobuf                  4.23.4                   pypi_0    pypi
psutil                    5.9.6                    pypi_0    pypi
py-cpuinfo                9.0.0                    pypi_0    pypi
pyaml                     23.9.7                   pypi_0    pypi
pyarrow                   14.0.1                   pypi_0    pypi
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pydantic                  2.5.2                    pypi_0    pypi
pydantic-core             2.14.5                   pypi_0    pypi
pynvml                    11.5.0                   pypi_0    pypi
pyparsing                 3.1.1                    pypi_0    pypi
pytest                    7.4.3                    pypi_0    pypi
pytest-benchmark          4.0.0                    pypi_0    pypi
python                    3.10.13              h955ad1f_0  
python-dateutil           2.8.2                    pypi_0    pypi
pytorch-triton            2.1.0+6e4932cda8          pypi_0    pypi
pytorch-warmup            0.1.1                    pypi_0    pypi
pytz                      2023.3.post1             pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h5eee18b_0  
referencing               0.32.0                   pypi_0    pypi
regex                     2023.10.3                pypi_0    pypi
requests                  2.28.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
resize-right              0.0.2                    pypi_0    pypi
rotary-embedding-torch    0.5.0                    pypi_0    pypi
rpds-py                   0.13.2                   pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
s3transfer                0.9.0                    pypi_0    pypi
safetensors               0.4.1                    pypi_0    pypi
scikit-image              0.22.0                   pypi_0    pypi
scikit-learn              1.3.2                    pypi_0    pypi
scipy                     1.11.4                   pypi_0    pypi
sentencepiece             0.1.99                   pypi_0    pypi
setuptools                68.2.2          py310h06a4308_0  
simplejson                3.19.2                   pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
soundfile                 0.12.1                   pypi_0    pypi
soupsieve                 2.5                      pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0  
stempeg                   0.2.3                    pypi_0    pypi
submitit                  1.5.1                    pypi_0    pypi
sympy                     1.11.1                   pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
tensorboard               2.15.1                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
tensorboardx              2.6.2.2                  pypi_0    pypi
threadpoolctl             3.2.0                    pypi_0    pypi
tifffile                  2023.12.9                pypi_0    pypi
timm                      0.9.7                    pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
tokenizers                0.13.3                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
torch                     2.2.0.dev20231115+cu118          pypi_0    pypi
torch-fidelity            0.3.0                    pypi_0    pypi
torch-geometric           2.4.0                    pypi_0    pypi
torchaudio                2.2.0.dev20231115+cu118          pypi_0    pypi
torchmetrics              1.2.1                    pypi_0    pypi
torchvision               0.17.0.dev20231115+cu118          pypi_0    pypi
tqdm                      4.66.1                   pypi_0    pypi
transformers              4.34.0.dev0              pypi_0    pypi
treetable                 0.2.5                    pypi_0    pypi
typing-extensions         4.8.0                    pypi_0    pypi
typing-inspect            0.9.0                    pypi_0    pypi
tzdata                    2023.3                   pypi_0    pypi
urllib3                   1.26.13                  pypi_0    pypi
vector-quantize-pytorch   1.12.1                   pypi_0    pypi
wcwidth                   0.2.12                   pypi_0    pypi
webdataset                0.2.86                   pypi_0    pypi
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.41.2          py310h06a4308_0  
x-clip                    0.14.4                   pypi_0    pypi
xz                        5.4.5                h5eee18b_0  
zlib                      1.2.13               h5eee18b_0  
```
Any tips @xuzhao9 to solve the issue?

For anyone else facing this issue, I've resolved the problem with bypassing the codna environment as using python virtual env. Installed the December 14 build from Pytorch and used the commit f09b12112dd18d4da0503ba82214da778b830cfa.