Repository: pytorch
Issue: Cant compile from source 1.8.0a0 because 'magma_v2.h' file not found and undefined references to `magma_*'
Labels: high priority,module: build,module: cuda,triaged,module: linear algebra,module: magma

## 🐛 Bug

## To Reproduce

I got latest 11.2 drivers and updated required libs


```
22:51 $ conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses
Collecting package metadata (current_repodata.json): done
Solving environment: - 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - <develop>/pypi::fastai==2.1.11=dev_0
  - <develop>/pypi::torch==1.8.0a0=dev_0
  - <develop>/pypi::fastai-xla-extensions==0.0.7=dev_0
  - <develop>/pypi::torchvision==0.9.0a0+3ee34eb=dev_0
done


==> WARNING: A newer version of conda exists. <==
  current version: 4.9.1
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /home/tyoc213/miniconda3/envs/xla

  added / updated specs:
    - cffi
    - cmake
    - dataclasses
    - future
    - mkl
    - mkl-include
    - ninja
    - numpy
    - pyyaml
    - requests
    - setuptools
    - six
    - typing_extensions


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2021.1.19  |       h06a4308_0         121 KB
    certifi-2020.12.5          |   py36h06a4308_0         140 KB
    cffi-1.14.4                |   py36h261ae71_0         223 KB
    ninja-1.10.2               |   py36hff7bd54_0         1.4 MB
    openssl-1.1.1i             |       h27cfd23_0         2.5 MB
    pyyaml-5.4.1               |   py36h27cfd23_1         170 KB
    requests-2.25.1            |     pyhd3eb1b0_0          52 KB
    setuptools-52.0.0          |   py36h06a4308_0         724 KB
    typing-extensions-3.7.4.3  |       hd3eb1b0_0          12 KB
    typing_extensions-3.7.4.3  |     pyh06a4308_0          28 KB
    ------------------------------------------------------------
                                           Total:         5.4 MB

The following NEW packages will be INSTALLED:

  typing-extensions  pkgs/main/noarch::typing-extensions-3.7.4.3-hd3eb1b0_0

The following packages will be UPDATED:

  ca-certificates                              2020.10.14-0 --> 2021.1.19-h06a4308_0
  certifi                          2020.11.8-py36h06a4308_0 --> 2020.12.5-py36h06a4308_0
  cffi                                1.14.3-py36h261ae71_2 --> 1.14.4-py36h261ae71_0
  ninja                               1.10.1-py36hfd86e86_0 --> 1.10.2-py36hff7bd54_0
  openssl                                 1.1.1h-h7b6447c_0 --> 1.1.1i-h27cfd23_0
  pyyaml                               5.3.1-py36h7b6447c_1 --> 5.4.1-py36h27cfd23_1
  requests                                      2.24.0-py_0 --> 2.25.1-pyhd3eb1b0_0
  setuptools                          50.3.1-py36h06a4308_1 --> 52.0.0-py36h06a4308_0

The following packages will be DOWNGRADED:

  typing_extensions                            3.7.4.3-py_0 --> 3.7.4.3-pyh06a4308_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
typing_extensions-3. | 28 KB     | ######################################################################################################################################################################## | 100% 
requests-2.25.1      | 52 KB     | ######################################################################################################################################################################## | 100% 
ca-certificates-2021 | 121 KB    | ######################################################################################################################################################################## | 100% 
cffi-1.14.4          | 223 KB    | ######################################################################################################################################################################## | 100% 
setuptools-52.0.0    | 724 KB    | ######################################################################################################################################################################## | 100% 
ninja-1.10.2         | 1.4 MB    | ######################################################################################################################################################################## | 100% 
typing-extensions-3. | 12 KB     | ######################################################################################################################################################################## | 100% 
pyyaml-5.4.1         | 170 KB    | ######################################################################################################################################################################## | 100% 
openssl-1.1.1i       | 2.5 MB    | ######################################################################################################################################################################## | 100% 
certifi-2020.12.5    | 140 KB    | ######################################################################################################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
```
# and updated magma to magma-cuda112

```
(xla) ✔ ~/Documents/github/pytorch [master ↓·2|✚ 1…3⚑ 4] 
22:51 $ conda install -c pytorch magma-cuda110^C
(xla) ✘-INT ~/Documents/github/pytorch [master ↓·2|✚ 1…3⚑ 4] 
22:52 $ nvidia-smi 
Thu Jan 28 22:52:10 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:02:00.0  On |                  N/A |
| 34%   32C    P8    23W / 350W |   7278MiB / 24267MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1082      G   /usr/lib/xorg/Xorg                147MiB |
|    0   N/A  N/A      1472      G   /usr/bin/gnome-shell               50MiB |
|    0   N/A  N/A      2321      G   ...mviewer/tv_bin/TeamViewer        4MiB |
|    0   N/A  N/A      5959      C   ...a3/envs/fastai/bin/python     3431MiB |
|    0   N/A  N/A      9291      G   ...gAAAAAAAAA --shared-files       37MiB |
|    0   N/A  N/A     11141      C   ...a3/envs/fastai/bin/python     1565MiB |
|    0   N/A  N/A     11272      G   ...AAAAAAAA== --shared-files       42MiB |
|    0   N/A  N/A     12783      C   ...a3/envs/fastai/bin/python     1991MiB |
|    0   N/A  N/A     13299      G   /usr/lib/firefox/firefox            4MiB |
+-----------------------------------------------------------------------------+
(xla) ✔ ~/Documents/github/pytorch [master ↓·2|✚ 1…3⚑ 4] 
22:52 $ conda install -c pytorch magma-cuda112
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.9.1
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /home/tyoc213/miniconda3/envs/xla

  added / updated specs:
    - magma-cuda112


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    magma-cuda112-2.5.2        |                1        79.5 MB  pytorch
    ------------------------------------------------------------
                                           Total:        79.5 MB

The following NEW packages will be INSTALLED:

  magma-cuda112      pytorch/linux-64::magma-cuda112-2.5.2-1


Proceed ([y]/n)? y


Downloading and Extracting Packages
magma-cuda112-2.5.2  | 79.5 MB   | ######################################################################################################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
```

and updated pytorch to latest master

```
(xla) ✔ ~/Documents/github/pytorch [master ↓·2|✚ 1…3⚑ 4] 
22:53 $ git submodule sync
Synchronizing submodule url for 'android/libs/fbjni'
Synchronizing submodule url for 'third_party/FP16'
Synchronizing submodule url for 'third_party/FXdiv'
Synchronizing submodule url for 'third_party/NNPACK'
Synchronizing submodule url for 'third_party/QNNPACK'
Synchronizing submodule url for 'third_party/XNNPACK'
Synchronizing submodule url for 'third_party/benchmark'
Synchronizing submodule url for 'third_party/cpuinfo'
Synchronizing submodule url for 'third_party/cub'
Synchronizing submodule url for 'third_party/eigen'
Synchronizing submodule url for 'third_party/fbgemm'
Synchronizing submodule url for 'third_party/fmt'
Synchronizing submodule url for 'third_party/foxi'
Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
Synchronizing submodule url for 'third_party/gloo'
Synchronizing submodule url for 'third_party/googletest'
Synchronizing submodule url for 'third_party/ideep'
Synchronizing submodule url for 'third_party/ios-cmake'
Synchronizing submodule url for 'third_party/kineto'
Synchronizing submodule url for 'third_party/nccl/nccl'
Synchronizing submodule url for 'third_party/neon2sse'
Synchronizing submodule url for 'third_party/onnx'
Synchronizing submodule url for 'third_party/onnx-tensorrt'
Synchronizing submodule url for 'third_party/protobuf'
Synchronizing submodule url for 'third_party/psimd'
Synchronizing submodule url for 'third_party/pthreadpool'
Synchronizing submodule url for 'third_party/pybind11'
Synchronizing submodule url for 'third_party/python-enum'
Synchronizing submodule url for 'third_party/python-peachpy'
Synchronizing submodule url for 'third_party/python-six'
Synchronizing submodule url for 'third_party/sleef'
Synchronizing submodule url for 'third_party/tbb'
Synchronizing submodule url for 'third_party/tensorpipe'
Synchronizing submodule url for 'third_party/zstd'
(xla) ✔ ~/Documents/github/pytorch [master ↓·2|✚ 1…3⚑ 4] 
22:53 $ git submodule update --init --recursive
```
finally tried to install like always

```
(xla) ✔ ~/Documents/github/pytorch [master ↓·2|✚ 1…3⚑ 4] 
22:53 $ pip install -e . --verbose
Using pip 20.2.4 from /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/pip (python 3.6)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-xd73aayr
Created temporary directory: /tmp/pip-req-tracker-fmdiyeuw
Initialized build tracking at /tmp/pip-req-tracker-fmdiyeuw
Created build tracker: /tmp/pip-req-tracker-fmdiyeuw
Entered build tracker: /tmp/pip-req-tracker-fmdiyeuw
Created temporary directory: /tmp/pip-install-sgj326q5
Obtaining file:///home/tyoc213/Documents/github/pytorch
  Added file:///home/tyoc213/Documents/github/pytorch to build tracker '/tmp/pip-req-tracker-fmdiyeuw'
    Running setup.py (path:/home/tyoc213/Documents/github/pytorch/setup.py) egg_info for package from file:///home/tyoc213/Documents/github/pytorch
    Created temporary directory: /tmp/pip-pip-egg-info-f3icyzo6
    Running command python setup.py egg_info
    Building wheel torch-1.8.0a0+Unknown
    running egg_info
    creating /tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info
    writing /tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info/dependency_links.txt
    writing entry points to /tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info/entry_points.txt
    writing requirements to /tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-f3icyzo6/torch.egg-info/SOURCES.txt'
    /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/setuptools/dist.py:461: UserWarning: Normalizing '1.8.0a0+Unknown' to '1.8.0a0+unknown'
      warnings.warn(tmpl.format(**locals()))
  Source in /home/tyoc213/Documents/github/pytorch has version 1.8.0a0+unknown, which satisfies requirement torch==1.8.0a0+unknown from file:///home/tyoc213/Documents/github/pytorch
  Removed torch==1.8.0a0+unknown from file:///home/tyoc213/Documents/github/pytorch from build tracker '/tmp/pip-req-tracker-fmdiyeuw'
Requirement already satisfied: typing_extensions in /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages (from torch==1.8.0a0+unknown) (3.7.4.3)
Requirement already satisfied: numpy in /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages (from torch==1.8.0a0+unknown) (1.19.2)
Requirement already satisfied: dataclasses in /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages (from torch==1.8.0a0+unknown) (0.7)
Installing collected packages: torch
  Attempting uninstall: torch
    Found existing installation: torch 1.8.0a0
    Uninstalling torch-1.8.0a0:
      Created temporary directory: /tmp/pip-uninstall-o9jciy57
      Removing file or directory /home/tyoc213/miniconda3/envs/xla/bin/convert-caffe2-to-onnx
      Removing file or directory /home/tyoc213/miniconda3/envs/xla/bin/convert-onnx-to-caffe2
      Created temporary directory: /tmp/pip-uninstall-6_g5wb2b
      Removing file or directory /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/torch.egg-link
      Removing pth entries from /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/easy-install.pth:
      Removing entry: /home/tyoc213/Documents/github/pytorch
      Successfully uninstalled torch-1.8.0a0
  Running setup.py develop for torch
    Running command /home/tyoc213/miniconda3/envs/xla/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/tyoc213/Documents/github/pytorch/setup.py'"'"'; __file__='"'"'/home/tyoc213/Documents/github/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
    [0/1] Re-running CMake...
    /bin/sh: 1: /usr/bin/cmake: not found
    FAILED: build.ninja
    /usr/bin/cmake -S/home/tyoc213/Documents/github/pytorch -B/home/tyoc213/Documents/github/pytorch/build
    ninja: error: rebuilding 'build.ninja': subcommand failed
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/tyoc213/Documents/github/pytorch/setup.py", line 773, in <module>
        build_deps()
      File "/home/tyoc213/Documents/github/pytorch/setup.py", line 320, in build_deps
        cmake=cmake)
      File "/home/tyoc213/Documents/github/pytorch/tools/build_pytorch_libs.py", line 58, in build_caffe2
        cmake.build(my_env)
      File "/home/tyoc213/Documents/github/pytorch/tools/setup_helpers/cmake.py", line 345, in build
        self.run(build_args, my_env)
      File "/home/tyoc213/Documents/github/pytorch/tools/setup_helpers/cmake.py", line 140, in run
        check_call(command, cwd=self.build_dir, env=env)
      File "/home/tyoc213/miniconda3/envs/xla/lib/python3.6/subprocess.py", line 311, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '4']' returned non-zero exit status 1.
    Building wheel torch-1.8.0a0+Unknown
    -- Building version 1.8.0a0+Unknown
    cmake --build . --target install --config Release -- -j 4
  Rolling back uninstall of torch
  Moving to /home/tyoc213/miniconda3/envs/xla/bin/convert-caffe2-to-onnx
   from /tmp/pip-uninstall-o9jciy57/convert-caffe2-to-onnx
  Moving to /home/tyoc213/miniconda3/envs/xla/bin/convert-onnx-to-caffe2
   from /tmp/pip-uninstall-o9jciy57/convert-onnx-to-caffe2
  Moving to /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/torch.egg-link
   from /tmp/pip-uninstall-6_g5wb2b/torch.egg-link
  Replacing /home/tyoc213/miniconda3/envs/xla/bin/convert-caffe2-to-onnx from /tmp/pip-uninstall-o9jciy57/convert-caffe2-to-onnx
  Replacing /home/tyoc213/miniconda3/envs/xla/bin/convert-onnx-to-caffe2 from /tmp/pip-uninstall-o9jciy57/convert-onnx-to-caffe2
  Replacing /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/torch.egg-link from /tmp/pip-uninstall-6_g5wb2b/torch.egg-link
  Rolling /home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/easy-install.pth back to previous state
ERROR: Command errored out with exit status 1: /home/tyoc213/miniconda3/envs/xla/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/tyoc213/Documents/github/pytorch/setup.py'"'"'; __file__='"'"'/home/tyoc213/Documents/github/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 406, in run
    pycompile=options.compile,
  File "/home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 90, in install_given_reqs
    pycompile=pycompile,
  File "/home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 800, in install
    unpacked_source_directory=self.unpacked_source_directory,
  File "/home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/pip/_internal/operations/install/editable_legacy.py", line 51, in install_editable
    cwd=unpacked_source_directory,
  File "/home/tyoc213/miniconda3/envs/xla/lib/python3.6/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/tyoc213/miniconda3/envs/xla/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/tyoc213/Documents/github/pytorch/setup.py'"'"'; __file__='"'"'/home/tyoc213/Documents/github/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
Removed build tracker: '/tmp/pip-req-tracker-fmdiyeuw'
(xla) ✘-1 ~/Documents/github/pytorch [master ↓·2|✚ 1…3⚑ 4] 

```


## Expected behavior

Be able to install with `pip install -e .`

## Environment

```
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.19.2

Python version: 3.6 (64-bit runtime)
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 3090
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch-xla==1.6
[pip3] torchvision==0.9.0a0+3ee34eb
[conda] blas                      1.0                         mkl  
[conda] magma-cuda110             2.5.2                         1    pytorch
[conda] magma-cuda112             2.5.2                         1    pytorch
[conda] mkl                       2020.2                      256  
[conda] mkl-include               2020.2                      256  
[conda] mkl-service               2.3.0            py36he904b0f_0  
[conda] mkl_fft                   1.2.0            py36h23d657b_0  
[conda] mkl_random                1.1.1            py36h0573a6f_0  
[conda] numpy                     1.19.2           py36h54aff64_0  
[conda] numpy-base                1.19.2           py36hfa32c7d_0  
[conda] torch                     1.8.0a0                  pypi_0    pypi
[conda] torch-xla                 1.6                       dev_0    <develop>
[conda] torchvision               0.9.0a0+3ee34eb           dev_0    <develop>
```

 - PyTorch Version: previous install was 1.7 from source more than 100 commits a go

## Additional context

Previously I have compiled pytorch from source without problems, also can't see from logs what is the problem.

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @malfet @seemethere @walterddr @ngimel @jianyuh @nikitaved @pearu @mruberry @heitorschueroff @IvanYashchuk