Repository: pytorch/benchmark
Issue: More issues in setup with incompatible pyarrow and dill versions
Root cause: Configuration Error

I'm already up to date as of now

```
$ git pull
Already up to date.
$ git branch
* main
$ git branch --show-current
main
$ git rev-parse HEAD
90f07fd6cac33a66ab2f8451328ef81b676f4535
```
But still more errors are cropping up

```
python install.py
checking packages torch, torchvision, torchtext, torchaudio are installed...OK
checking out input files from Amazon S3 ...Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/Background_Matting_inputs.tar.gz to /home/user/benchmark/torchbenchmark/data/Background_Matting_inputs.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/coco128.tar.gz to /home/user/benchmark/torchbenchmark/data/coco128.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/multi30k.tar.gz to /home/user/benchmark/torchbenchmark/data/multi30k.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/tacotron2-minimal.tar.gz to /home/user/benchmark/torchbenchmark/data/tacotron2-minimal.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/coco2017-minimal.tar.gz to /home/user/benchmark/torchbenchmark/data/coco2017-minimal.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/pytorch_stargan_inputs.tar.gz to /home/user/benchmark/torchbenchmark/data/pytorch_stargan_inputs.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/LearningToPaint_inputs.tar.gz to /home/user/benchmark/torchbenchmark/data/LearningToPaint_inputs.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/pytorch_CycleGAN_and_pix2pix_inputs.tar.gz to /home/user/benchmark/torchbenchmark/data/pytorch_CycleGAN_and_pix2pix_inputs.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/Super_SloMo_inputs.tar.gz to /home/user/benchmark/torchbenchmark/data/Super_SloMo_inputs.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/speech_transformer_inputs.tar.gz to /home/user/benchmark/torchbenchmark/data/speech_transformer_inputs.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/data/Reddit_minimal.tar.gz to /home/user/benchmark/torchbenchmark/data/Reddit_minimal.tar.gz
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/models/drq/obs.pkl to /home/user/benchmark/torchbenchmark/models/drq/obs.pkl
Checking out https://ossci-datasets.s3.amazonaws.com/torchbench/models/maml_omniglot/batch.pt to /home/user/benchmark/torchbenchmark/models/maml_omniglot/batch.pt
OK
decompressing input tarball: coco2017-minimal.tar.gz...OK
decompressing input tarball: Background_Matting_inputs.tar.gz...OK
decompressing input tarball: multi30k.tar.gz...OK
decompressing input tarball: coco128.tar.gz...OK
decompressing input tarball: Super_SloMo_inputs.tar.gz...OK
decompressing input tarball: tacotron2-minimal.tar.gz...OK
decompressing input tarball: pytorch_stargan_inputs.tar.gz...OK
decompressing input tarball: pytorch_CycleGAN_and_pix2pix_inputs.tar.gz...OK
decompressing input tarball: Reddit_minimal.tar.gz...OK
decompressing input tarball: speech_transformer_inputs.tar.gz...OK
decompressing input tarball: LearningToPaint_inputs.tar.gz...OK
running setup for /home/user/benchmark/torchbenchmark/models/BERT_pytorch...OK
running setup for /home/user/benchmark/torchbenchmark/models/Background_Matting...OK
running setup for /home/user/benchmark/torchbenchmark/models/DALLE2_pytorch...OK
running setup for /home/user/benchmark/torchbenchmark/models/LearningToPaint...OK
running setup for /home/user/benchmark/torchbenchmark/models/Super_SloMo...OK
running setup for /home/user/benchmark/torchbenchmark/models/alexnet...OK
running setup for /home/user/benchmark/torchbenchmark/models/attention_is_all_you_need_pytorch...FAIL
Error for /home/user/benchmark/torchbenchmark/models/attention_is_all_you_need_pytorch:
---------------------------------------------------------------------------
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
multiprocess 0.70.14 requires dill>=0.3.6, but you have dill 0.3.5.1 which is incompatible.
datasets 2.11.0 requires pyarrow>=8.0.0, but you have pyarrow 7.0.0 which is incompatible.
Collecting en-core-web-sm==3.5.0
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.5.0/en_core_web_sm-3.5.0-py3-none-any.whl (12.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 5.2 MB/s eta 0:00:00
Requirement already satisfied: spacy<3.6.0,>=3.5.0 in /home/user/env/lib/python3.10/site-packages (from en-core-web-sm==3.5.0) (3.5.1)
Requirement already satisfied: typer<0.8.0,>=0.3.0 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (0.7.0)
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.0.12)
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.3.0)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (4.65.0)
Requirement already satisfied: numpy>=1.15.0 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.21.2)
Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (8.1.9)
Requirement already satisfied: packaging>=20.0 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (23.0)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.0.8)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (2.0.7)
Requirement already satisfied: pathy>=0.10.0 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (0.10.1)
Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.1.1)
Requirement already satisfied: jinja2 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.1.2)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /home/user/env/lib/python3.10/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.10.7)
...
  File "/home/user/env/lib/python3.10/site-packages/torchdata/__init__.py", line 9, in <module>
    from . import datapipes
  File "/home/user/env/lib/python3.10/site-packages/torchdata/datapipes/__init__.py", line 9, in <module>
    from . import iter, map, utils
  File "/home/user/env/lib/python3.10/site-packages/torchdata/datapipes/iter/__init__.py", line 46, in <module>
    from torchdata.datapipes.iter.load.huggingface import HuggingFaceHubReaderIterDataPipe as HuggingFaceHubReader
  File "/home/user/env/lib/python3.10/site-packages/torchdata/datapipes/iter/load/huggingface.py", line 15, in <module>
    import datasets
  File "/home/user/env/lib/python3.10/site-packages/datasets/__init__.py", line 34, in <module>
    raise ImportWarning(
ImportWarning: To use `datasets`, the module `pyarrow>=8.0.0` is required, and the current version of `pyarrow` doesn't match this condition.
If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.
Traceback (most recent call last):
  File "/home/user/benchmark/torchbenchmark/models/attention_is_all_you_need_pytorch/install.py", line 25, in <module>
    preprocess()
  File "/home/user/benchmark/torchbenchmark/models/attention_is_all_you_need_pytorch/install.py", line 17, in preprocess
    subprocess.check_call([sys.executable, 'preprocess.py', '-lang_src', 'de_core_news_sm', '-lang_trg', 'en_core_web_sm', '-share_vocab',
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/user/env/bin/python', 'preprocess.py', '-lang_src', 'de_core_news_sm', '-lang_trg', 'en_core_web_sm', '-share_vocab', '-save_data', '/home/user/benchmark/torchbenchmark/models/attention_is_all_you_need_pytorch/.data/m30k_deen_shr.pkl', '-data_path', '/home/user/benchmark/torchbenchmark/data/.data/multi30k']' returned non-zero exit status 1.


---------------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/user/benchmark/install.py", line 114, in <module>
    raise RuntimeError("Failed to complete setup")
RuntimeError: Failed to complete setup
```

Strange, our CI workflow doesn't have this issue: https://github.com/pytorch/benchmark/actions/runs/4640727333/jobs/8212965327
so I can't reproduce this problem.

Can you try upgrade the dill version at https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/attention_is_all_you_need_pytorch/requirements.txt#L1 to see if it fixes the problem?

> Strange, our CI workflow doesn't have this issue: https://github.com/pytorch/benchmark/actions/runs/4640727333/jobs/8212965327 so I can't reproduce this problem.
> 
> Can you try upgrade the dill version at https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/attention_is_all_you_need_pytorch/requirements.txt#L1 to see if it fixes the problem?

just tried, the same problem.

$ pip install dill==0.3.5.1
Requirement already satisfied: dill==0.3.5.1 in /home/user/env/lib/python3.10/site-packages (0.3.5.1)

@dany-nonstop Can you try install dill==0.3.6 ?

the problem came from multiple pulls in the last several days, which have conflicting requirements to the packages. After cleaning and reinstalling, this bug disappeared. I'll have it closed. Thanks.