Repository: pytorch/benchmark
Issue: Can't seem to run test in docker with A5500
Root cause: External Dependency Breakage

Pulled the latest docker version like so:

```
docker run -it --gpus all ghcr.io/pytorch/torchbench:latest bash
```



```bash
(torchbench) runner@1d84519ae75f:/workspace/benchmark$ python run_benchmark.py torch-nightly -d cuda -t eval --model densenet121
Running TorchBenchModelConfig(name='densenet121', test='eval', device='cuda', batch_size=None, extra_args=[], extra_env=None, output_dir=None) ...Traceback (most recent call last):
  File "/workspace/benchmark/run_benchmark.py", line 36, in <module>
    run()
  File "/workspace/benchmark/run_benchmark.py", line 30, in run
    benchmark.run(bm_args)
  File "/workspace/benchmark/userbenchmark/torch-nightly/run.py", line 211, in run
    metrics = run_config(config, dryrun=args.dryrun)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/benchmark/userbenchmark/torch-nightly/run.py", line 162, in run_config
    result: TorchBenchModelMetrics = get_model_test_metrics(model, metrics=metrics)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/benchmark/torchbenchmark/util/experiment/metrics.py", line 126, in get_model_test_metrics
    latencies = get_latencies(model.invoke, device, nwarmup=nwarmup, num_iter=num_iter)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/benchmark/torchbenchmark/util/experiment/metrics.py", line 33, in get_latencies
    func()
  File "/workspace/benchmark/torchbenchmark/__init__.py", line 399, in invoke
    self.worker.run("""
  File "/workspace/benchmark/components/_impl/workers/subprocess_worker.py", line 158, in run
    self._run(snippet)
  File "/workspace/benchmark/components/_impl/workers/subprocess_worker.py", line 307, in _run
    result = marshal.loads(self._output_pipe.read())
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/benchmark/components/_impl/workers/subprocess_rpc.py", line 238, in read
    msg_size = struct.unpack(_ULL, self._read(_ULL_SIZE))[0]
                                   ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/benchmark/components/_impl/workers/subprocess_rpc.py", line 227, in _read
    raise IOError(f"Subprocess terminates with code {int.from_bytes(msg, sys.byteorder)}")
OSError: Subprocess terminates with code 11
(torchbench) runner@1d84519ae75f:/workspace/benchmark$

```


using:
```
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal
```

with

```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12             Driver Version: 535.104.12   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A5500               Off | 00000000:3B:00.0 Off |                  Off |
| 30%   37C    P3              61W / 230W |     19MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A5500               Off | 00000000:AF:00.0 Off |                  Off |
| 30%   37C    P0              45W / 230W |     19MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

```


This seems like a fairly straightforward run, so I assume I'm missing something? Spent a couple of hours and can't quite seem to crack it..

Duplicate of https://github.com/pytorch/benchmark/issues/2081

Yes, we can reproduce this in our CI.
There is an upstream packaging change in nightly (big wheel -> small wheel), which broke the nightly docker run environment.

cc @malfet 

@xuzhao9 can you please elaborate how the docker container for benchmarks are build? What I find a bit suspicious is that it installs NVIDIA driver into the container, whereas in all other containers that we shipped driver is absent and is mirrored from the host OS when  `--gpus all` is invoked

@malfet here is the code pointer: https://github.com/pytorch/benchmark/blob/main/docker/gcp-a100-runner-dind.dockerfile

We have to pre-install NVIDIA driver into the container, because the NVIDIA driver version must match the version used by the underlying Google Cloud K8s host, otherwise there will be driver version compatibility issue.

We also used host-mirroring instead of installing the driver about 6 months ago, but that triggered internal security tasks claiming non-container so files are loaded in the container, so we switched to installing the same version of NVIDIA driver as host instead.

From the stack backtrace, this looks more like a CUDA library issue instead of NVIDIA driver issue.

cc: @atalman 

Closed as this has been workarounded in our Docker image, follow-up in https://github.com/pytorch/benchmark/issues/2081

yup, works as expected, thanks!