Repository: pytorch
Issue: Significant Memory Increase with Conv2d using padding_mode 'replicate', 'reflect', or 'circular'
Labels: module: nn,module: memory usage,module: convolution,triaged,actionable

### 🐛 Describe the bug

I've observed that using `padding_mode='replicate'` or `padding_mode='reflect'` or `padding_mode='circular'` in `nn.Conv2d` layers leads to significantly higher memory consumption compared to `padding_mode='zeros'`. This increase in memory usage could be problematic for models with limited GPU memory or when scaling to larger architectures.


Reproducible Example:
Here's a minimal script that demonstrates the issue:

```python
import torch
import torch.nn as nn

def check_memory_usage(padding_mode):
    model = nn.Conv2d(64, 64, kernel_size=3, padding=1, padding_mode=padding_mode).cuda()
    optimizer = torch.optim.Adam(model.parameters())
    
    input_tensor = torch.randn(32, 64, 256, 256).cuda()
    
    torch.cuda.reset_peak_memory_stats()
    torch.cuda.synchronize()
    
    output = model(input_tensor)
    loss = output.sum()
    loss.backward()
    optimizer.step()
    
    torch.cuda.synchronize()
    max_memory = torch.cuda.max_memory_allocated() / 1e6  # Convert to MB
    
    print(f"Max memory used with padding_mode='{padding_mode}': {max_memory:.2f} MB")

check_memory_usage('zeros')
torch.cuda.empty_cache()
check_memory_usage('replicate')
torch.cuda.empty_cache()
check_memory_usage('reflect')
torch.cuda.empty_cache()
check_memory_usage('circular')
```
Results:
Running this script produces the following output:
```
Max memory used with padding_mode='zeros': 2689.22 MB
Max memory used with padding_mode='replicate': 3242.94 MB
Max memory used with padding_mode='reflect': 3242.94 MB
Max memory used with padding_mode='circular': 3242.94 MB
```
As we can see, there's about 20% increase in memory usage when switching from 'zeros' to other padding modes.

Environment:

PyTorch Version: 2.3.1
CUDA Version: 12.1
Operating System: Windows 10

Questions:

1. Is this behavior expected? If so, why do other padding modes consume significantly more memory?
2. Are there any known workarounds or best practices to mitigate this increased memory usage when 'replicate' or any other padding is necessary for a model's performance?
3. Is this something that could be optimized in future PyTorch releases?


Thank you for your time and assistance in addressing this issue.


### Versions

PyTorch version: 2.3.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.17 (main, Jul  5 2023, 20:47:11) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version: 555.99
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3400
DeviceID=CPU0
Family=198
L2CacheSize=8192
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=3400
Name=13th Gen Intel(R) Core(TM) i7-13700K
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] onnx==1.13.1
[pip3] onnxruntime-openvino==1.13.1
[pip3] torch==2.3.1+cu121
[pip3] torch_geometric==2.5.3
[pip3] torch_scatter==2.1.2+pt23cu121
[pip3] torchaudio==2.3.1+cu121
[pip3] torchvision==0.18.1+cu121
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0                 pypi_0    pypi
[conda] mkl-service               2.4.0            py39h2bbff1b_0
[conda] mkl_fft                   1.3.1            py39h277e83a_0
[conda] mkl_random                1.2.2            py39hf11a4ad_0
[conda] numpy                     1.23.5                   pypi_0    pypi
[conda] numpy-base                1.24.3           py39h005ec55_0
[conda] torch                     2.3.1+cu121              pypi_0    pypi
[conda] torch-geometric           2.5.3                    pypi_0    pypi
[conda] torch-scatter             2.1.2+pt23cu121          pypi_0    pypi
[conda] torchaudio                2.3.1+cu121              pypi_0    pypi
[conda] torchvision               0.18.1+cu121             pypi_0    pypi

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki