Repository: pytorch
Issue: Memory usage of torch.nn.functional.interpolate increased with v1.5.0 when run on numpy input
Labels: module: nn,module: cuda,module: memory usage,triaged,module: numpy

## 🐛 Bug

The code below, which downscales an image by 2x, used to use 1GB of GPU memory with pytorch-1.3.0. Switching to 1.5.0, it now uses 5GB to do the same job, which makes it run out of memory on smaller GPUs. 

## To Reproduce

Run the test below using pytest (`pytest -v -s test_downscale_plain.py`)

    import torch
    import numpy as np

    def test_downscale_plain():

        def get_image():
            # Pixels would typically come from some image reader
            pixels = np.random.random((17173, 20400, 3)).astype(np.float32)
            # Turn pixels into a tensor bach
            return torch.from_numpy(np.moveaxis(pixels, -1, -3)).to(torch.device("cuda:0"))[None, ...]
        image = get_image()

        print('\nInput: shape {}, type {}/{}'.format(image.shape, image.type(), image.dtype))

        # Force garbage collect, just in case
        torch.cuda.ipc_collect()
        # Reset peak measurement
        torch.cuda.reset_max_memory_allocated()
        peak_start = torch.cuda.memory_allocated()/1024/1024

        with torch.no_grad():
            image = torch.nn.functional.interpolate(image, scale_factor=0.5, mode='area')
        
        peak_end = torch.cuda.max_memory_allocated()/1024/1024
        memory_usage = peak_end-peak_start
        print('pytorch {}, {}: {:.2f}MB ({:.2f}MB -> {:.2f}MB)'.format(torch.__version__, torch.cuda.get_device_name(), memory_usage, peak_start, peak_end))
        assert 900 <= memory_usage <= 1100

Running with 1.3.0, the output is:

    test_downscale_plain.py::test_downscale_plain 
    Input: shape torch.Size([1, 3, 17173, 20400]), type torch.cuda.FloatTensor/torch.float32
    pytorch 1.3.0a0+ee77ccb, Quadro RTX 8000: 1002.24MB (4010.00MB -> 5012.24MB)
    PASSED

Running with 1.5.0:

    test_downscale_plain.py::test_downscale_plain 
    Input: shape torch.Size([1, 3, 17173, 20400]), type torch.cuda.FloatTensor/torch.float32
    pytorch 1.5.0a0+4ff3872, Quadro RTX 8000: 5012.24MB (4010.00MB -> 9022.24MB)
    FAILED

## Expected behavior

I would have expected the memory usage to stay the same compared to previous versions.

Note that replacing the `get_image()`method by a method returning a tensor created directly from torch (e.g., `return torch.Tensor(1, 3, 17173, 20400).normal_().to(torch.device("cuda:0"))`) gets the behaviour back to using 1GB of memory. In addition, removing the `np.moveaxis()` call(and swapping the dimensions in the line above to keep the end shape the same), or removing the `[None, ...]`(and adding an extra dimension in the line above to keep the end shape the same) also removes the bug. This bug seems tied to the way I am converting the numpy array into a torch tensor. The `pixels` object is of a form often returned by image readers (i.e., a 3D numpy array of shape HxWxC).

## Environment

PyTorch version: 1.5.0a0+4ff3872
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.105
GPU models and configuration: 
GPU 0: Quadro M4000
GPU 1: Quadro RTX 8000

Nvidia driver version: 435.21
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] numpydoc==0.7.0
[pip] torch==1.5.0a0+4ff3872
[conda] Could not collect


cc @ngimel @albanD @mruberry @rgommers