Repository: pytorch
Issue: pytorch test failed
Labels: module: autograd,module: cuda,module: tests,triaged

Hi, 

I built pytorch from source successfully but  running the tests produces these errors
```
.................................s..........s....................................................................................................................................................................................................................test_autograd.py:7408: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  self.assertIsNone(a.grad)
.............s.................................................................sE.E...................................................................................................................................................................................................................................................................................................................s......................................................................................................................................................................................................................................................................................................................................................................................................s..................................................................................................
======================================================================
ERROR: test_lu_square_batch_no_info_cuda (__main__.TestAutogradDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 874, in wrapper
    method(*args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 295, in instantiated_test
    raise rte
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 290, in instantiated_test
    result = test_fn(self, *args)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "test_autograd.py", line 5176, in do_test
    check(name)
  File "test_autograd.py", line 5082, in check
    run_grad_and_gradgrad_checks(self, name, test_name, fn,
  File "test_autograd.py", line 4947, in run_grad_and_gradgrad_checks
    test_case.assertTrue(gradcheck(apply_method, input_variables, eps=1e-6, atol=PRECISION,
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1911, in gradcheck
    return torch.autograd.gradcheck(fn, inputs, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 479, in gradcheck
    return not_reentrant_error()
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 476, in not_reentrant_error
    return fail_test(error_msg)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 367, in fail_test
    raise RuntimeError(msg)
RuntimeError: Backward is not reentrant, i.e., running backward with same                         input and grad_output multiple times gives different values,                         although analytical gradient matches numerical gradient.                         The tolerance for nondeterminism was 0.0.

======================================================================
ERROR: test_lu_square_many_batches_no_info_cuda (__main__.TestAutogradDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 874, in wrapper
    method(*args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 295, in instantiated_test
    raise rte
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 290, in instantiated_test
    result = test_fn(self, *args)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "test_autograd.py", line 5176, in do_test
    check(name)
  File "test_autograd.py", line 5082, in check
    run_grad_and_gradgrad_checks(self, name, test_name, fn,
  File "test_autograd.py", line 4959, in run_grad_and_gradgrad_checks
    test_case.assertTrue(gradgradcheck(apply_method, input_variables,
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1925, in gradgradcheck
    return torch.autograd.gradgradcheck(fn, inputs, grad_outputs, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 674, in gradgradcheck
    return gradcheck(
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 468, in gradcheck
    checkIfNumericalAnalyticAreClose(a, n, j)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 449, in checkIfNumericalAnalyticAreClose
    return fail_test(error_str + 'Jacobian mismatch for output %d with respect to input %d,\n'
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 367, in fail_test
    raise RuntimeError(msg)
RuntimeError: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[ 3.6142e+00,  7.0091e+00,  5.3443e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 7.0091e+00,  1.1945e+01,  9.4077e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 5.3443e+00,  9.4077e+00,  7.4169e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        ...,
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  1.4775e+00,
          1.3568e+00, -9.9772e-02],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  1.3568e+00,
          1.2459e+00, -9.1998e-02],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -9.9772e-02,
         -9.1998e-02, -1.1102e-10]], device='cuda:0')
analytical:tensor([[ 3.6142,  7.0091,  5.3443,  ...,  0.0000,  0.0000,  0.0000],
        [ 7.0091, 11.9455,  9.4077,  ...,  0.0000,  0.0000,  0.0000],
        [ 5.3443,  9.4077,  7.4169,  ...,  0.0000,  0.0000,  0.0000],
        ...,
        [ 0.0000,  0.0000,  0.0000,  ...,  1.4775,  1.3568, -0.0998],
        [ 0.0000,  0.0000,  0.0000,  ...,  1.3568,  1.2459, -0.0920],
        [ 0.0000,  0.0000,  0.0000,  ..., -0.0998, -0.0920,  0.0000]],
       device='cuda:0')


----------------------------------------------------------------------
Ran 2715 tests in 1295.592s

FAILED (errors=2, skipped=23, expected failures=1)
Traceback (most recent call last):
  File "test/run_test.py", line 926, in <module>
    main()
  File "test/run_test.py", line 905, in main
    raise RuntimeError(err_message)
RuntimeError: test_autograd failed!
```
System is Ubuntu 20.04
python 3.8.10
CUDA 11.0
cudnn 8
GPU Nvidia GTX1070

I built pytorch with these commands

```
export BUILD_ONNX_PYTHON=1 USE_CUDA=1 USE_CUDNN=1 USE_MKLDNN=1 USE_TENSORRT=1 USE_OPENCV=1 USE_FFMPEG=1 USE_LMDB=1 GPU_ARCH=sm_61
 git clone https://github.com/pytorch/pytorch.git
cd pytorch
git checkout -b v1.6 origin/release/1.8
git submodule sync
git submodule update --init --recursive
 python setup.py install
```

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @ngimel @mruberry @VitalyFedyunin @walterddr