Repository: pytorch
Issue: Calling emptyCache in Conv_v7.cpp causes performance degradation
Labels: module: cudnn,module: convolution,triaged

## 🐛 Bug

In `aten/src/ATen/native/cudnn/Conv_v7.cpp`, there are calls to `CUDACachingAllocator::emptyCache` when running an algorithm search during benchmarking. [The justification for the calls is](https://github.com/pytorch/pytorch/blob/25a8397bf3760d60bd754517d590f15cbd041e25/aten/src/ATen/native/cudnn/Conv_v7.cpp#L280-L283):

```cpp
      // Free the cached blocks in our caching allocator. They are
      // needed here because the above benchmarking uses a huge amount of memory,
      // e.g. a few GBs.
      c10::cuda::CUDACachingAllocator::emptyCache();
```

There are a few issues with this call:
1. It's not necessary because `CUDACachingAllocator` [flushes its own cache internally](https://github.com/pytorch/pytorch/blob/8aad66a7bd35782a0e662d133276830204499656/c10/cuda/CUDACachingAllocator.cpp#L243) on a memory allocation when required.
2. The workspace memory is managed by a `Workspace` instance ([using RAII pattern](https://github.com/pytorch/pytorch/blob/25a8397bf3760d60bd754517d590f15cbd041e25/aten/src/ATen/native/cudnn/Conv_v7.cpp#L101-L120)) which is [scope-local with the `emptyCache` call](https://github.com/pytorch/pytorch/blob/25a8397bf3760d60bd754517d590f15cbd041e25/aten/src/ATen/native/cudnn/Conv_v7.cpp#L267-L283). So we're attempting to empty the cache before releasing the memory, defeating the justification in the comment.
3. In the case where `emptyCache` is able to find segments to release back to the driver, the internal `cudaFree` calls will cause device synchronizations, limiting parallelism. But as explained in the previous two points, there is no benefit to this call but we could end up paying a performance penalty nonetheless.

The performance degradation is particularly notable when training with `torch.backends.cudnn.benchmarking=True` and variable batch sizes since this code path is executed for each new input size.

I'm happy to submit a PR for this change.

## Environment

```
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti

Nvidia driver version: 440.33.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.1
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] haste-pytorch==0.5.0rc0
[pip3] numpy==1.18.5
[pip3] torch==1.6.0
[pip3] torchaudio==0.6.0
[pip3] torchvision==0.7.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              hfd86e86_1
[conda] haste-pytorch             0.5.0rc0                 pypi_0    pypi
[conda] mkl                       2020.1                      217
[conda] mkl-service               2.3.0            py38he904b0f_0
[conda] mkl_fft                   1.1.0            py38h23d657b_0
[conda] mkl_random                1.1.1            py38h0573a6f_0
[conda] numpy                     1.18.5                   pypi_0    pypi
[conda] pytorch                   1.6.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] torchaudio                0.6.0                    pypi_0    pypi
[conda] torchvision               0.7.0                    pypi_0    pypi
```

cc @csarofeen @ptrblck @xwang233