Repository: pytorch
Issue: Is nn.Conv2d equivalent with Unfold + Matrix Multiplication + Fold ?
Labels: needs reproduction,module: convolution,triaged,module: correctness (silent)

## Is nn.Conv2d equivalent with Unfold + Matrix Multiplication + Fold

I have implemented a custom convolutional operator with Unfold+matrix multiplication+Fold as described in the pytorch documentation, under Unfold examples [here](https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html):

```py
    inp = torch.randn(1, 3, 10, 12)
    w = torch.randn(2, 3, 4, 5)
    inp_unf = torch.nn.functional.unfold(inp, (4, 5))
    out_unf = inp_unf.transpose(1, 2).matmul(w.view(w.size(0), -1).t()).transpose(1, 2)
    out = torch.nn.functional.fold(out_unf, (7, 8), (1, 1))
    # or equivalently (and avoiding a copy),
    out = out_unf.view(1, 2, 7, 8)
    (torch.nn.functional.conv2d(inp, w) - out).abs().max()       
```

Ideally, this should be the same thing with the nn.Conv2d, however my simple experiments on a vision understanding task does not say so. What can be the reason? Are there any interesting optimizations happening on cuda implementation?

## To Reproduce

Here is my custom convolution:

```py
class CustomConv(nn.Module):

    def __init__(self, in_channel, out_channel, kernel_size=3, stride=1, padding=0, bias=True):
        super(CustomConv, self).__init__()
        self.k = kernel_size
        self.in_c = in_channel
        self.out_c = out_channel
        self.stride = stride
        self.padding = padding
        self.conv = nn.Parameter(torch.ones(out_channel, in_channel, kernel_size, kernel_size))
        nn.init.xavier_uniform_(self.conv, gain=nn.init.calculate_gain('relu'))
        #nn.init.kaiming_normal_(self.conv, mode='fan_out', nonlinearity='relu')

    def forward(self, inp):
        k = self.k
        stride = self.stride
        h_in, w_in = inp.shape[2], inp.shape[3]

        padding = self.padding  # + k//2
        batch_size = inp.shape[0]

        h_out = (h_in + 2 * padding - (k - 1) - 1) / stride + 1
        w_out = (w_in + 2 * padding - (k - 1) - 1) / stride + 1
        h_out, w_out = int(h_out), int(w_out)

        inp_unf = torch.nn.functional.unfold(inp, (k, k), padding=padding)
        out_unf = inp_unf.transpose(1, 2).matmul(self.conv.view(self.conv.size(0), -1).t()).transpose(1, 2)
        #out_ = out_unf.view(batch_size, self.out_c, h_out, w_out) # this is equivalent with fold.
        out_ = torch.nn.functional.fold(out_unf, (h_out, w_out), (1, 1))
```


## Expected behavior

I expected to be able to train my network with this CustomConv, instead of nn.Conv2d. But I cannot replicate the results.

## Environment

PyTorch version: 1.5.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.17.3

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti

Nvidia driver version: 450.51.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.0
[pip3] torch==1.5.1
[pip3] torchfile==0.1.0
[pip3] torchnet==0.0.5.1
[pip3] torchvision==0.6.1