Repository: pytorch
Issue: Unit testing failures when porting PyTorch wheel
Labels: module: binaries,module: build,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

I built and installed a PyTorch wheel inside a custom container, and ran the unit test with the `run_test.py` command with a few exclusions. The unit testing completed successful inside this container.

However, when I start a second container (the same container as above), clone the pytorch repo with the same version tag, and run the `run_test.py` command with the same set of exclusions. Some tests started to fail, for example, the `test_linalg` test.

Then, I investigated the unit testing logs, and found out that the `test_linalg` wasn't in the first test suite, but showed up in the second one. I suspect this is due to some environmental settings during the wheel building phase and thus caused the test suites to be different in the two containers. Could you provide some ideas on how to resolve this issue?

Another more obvious difference could be the OMPI difference, where the first container is good and the second one fails:
```
Executing ['mpiexec', '-n', '3', '', '', '/opt/conda/bin/python3.6', 'distributed/test_distributed_fork.py'] ... [2021-03-19 16:04:46.610181]
--
1005 | --------------------------------------------------------------------------
1006 | mpiexec has detected an attempt to run as root.
1007 |  
1008 | Running as root is *strongly* discouraged as any mistake (e.g., in
1009 | defining TMPDIR) or bug can result in catastrophic damage to the OS
1010 | file system, leaving your system in an unusable state.
1011 |  
1012 | We strongly suggest that you run mpiexec as a non-root user.
1013 |  
1014 | You can override this protection by adding the --allow-run-as-root option
1015 | to the cmd line or by setting two environment variables in the following way:
1016 | the variable OMPI_ALLOW_RUN_AS_ROOT=1 to indicate the desire to override this
1017 | protection, and OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 to confirm the choice and
1018 | add one more layer of certainty that you want to do so.
1019 | We reiterate our advice against doing so - please proceed at your own risk.
1020 | --------------------------------------------------------------------------
```

## To Reproduce

Steps to reproduce the behavior:

1. Build and install a PyTorch wheel in a container
1. Unit testing PyTorch to succeed
1. Install the PyTorch wheel in a second container (the same one as step1), unit testing PyTorch to fail

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

The same set of unit testing should be in the two same containers, regardless of building or not building PyTorch inside it. 

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```
I'm using a CPU image with a locally built wheel.

PyTorch version: N/A
--
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.19.6

Python version: 3.6 (64-bit runtime)
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
CuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] sagemaker-pytorch-training==2.4.0
[pip3] torch==1.8.1
[pip3] torchvision==0.9.0
[conda] mkl                       2020.2                      256
[conda] mkl-include               2020.2                      256
[conda] numpy                     1.19.1           py36h30dfecb_0    anaconda
[conda] numpy-base                1.19.1           py36h75fe3a5_0    anaconda
[conda] sagemaker-pytorch-training 2.4.0                    pypi_0    pypi
[conda] torch                     1.8.1                    pypi_0    pypi
[conda] torchvision               0.9.0                    pypi_0    pypi



## Additional context

<!-- Add any other context about the problem here. -->


cc @ezyang @seemethere @malfet @walterddr