Repository: pytorch
Issue: 4D tensor support for torch.nn.functionnal.fold() (col2im)
Labels: module: nn,triaged,function request

## 🚀 Feature
Implementation or modification of col2im function/method for 4D tensor support in torch.nn.functionnal.fold().

## Motivation
I came to this "issue" when i was working on a custom locally connected decoder.
(the layer behave like a transposed convolution but without the weight sharing)
I initially build a locally connected network for encoding purpose using the torch.nn.functional.unfold() function; I then assume that the torch.nn.functional.fold() work in an inverse manner with 4D tensor, until i find out that it only support 3D tensor which is misleading and not consistent with unfold() and other function.
This issue halt the development of custom convolution / transposed convolution in my opinion.

I am open to alternative to build my decoder !

here is the current code : 

```
import numpy as np
import torch
import torch.nn as nn
from torch.nn import init


# Main decoder locally connected linear layer
class LocallyConnected2dTranspose(nn.Module):
    def calculate_spatial_transposed_output_shape(self, input_shape, kernel_size, dilation, input_padding, out_padding, stride):
        return [np.floor((input_shape[index]-1)*stride[index]-2*input_padding[index]+dilation[index]*kernel_size[index]-1+out_padding[index]+1).astype(int) for index in range(len(input_shape))]
    def __init__(self, input_shape, in_channels, out_channels, kernel_size, dilation, input_padding, out_padding, stride):
        super().__init__()
        self.kernel_size = kernel_size
        self.out_channels = out_channels
        self.dilation = dilation
        self.input_padding = input_padding
        self.out_padding = out_padding
        self.stride = stride
        # calculate desired output shape and generate weight/bias matrix
        self.output_height, self.output_width = self.calculate_spatial_transposed_output_shape(input_shape, kernel_size, dilation, input_padding, out_padding, stride)
        # weight and spatial block
        self.weight_tensor_depth = in_channels * kernel_size[0] * kernel_size[1]
        self.spatial_blocks_size = self.output_height * self.output_width
        self.weights = nn.Parameter(torch.empty((1, self.weight_tensor_depth, self.spatial_blocks_size, out_channels),requires_grad=True, dtype=torch.float))
        print(self.weights.shape)
        self.bias = nn.Parameter(torch.empty((1, out_channels, self.output_height, self.output_width),requires_grad=True, dtype=torch.float))
        # init weight and bias
        torch.nn.init.xavier_uniform_(self.weights)
        torch.nn.init.xavier_uniform_(self.bias)

    def forward(self, input):
        input_f = torch.nn.functional.fold(input=input, output_size=[self.output_height, self.output_width], kernel_size=self.kernel_size, dilation=self.dilation, padding=self.out_padding, stride=self.stride)
        local_conv_f = (input_f.view((*input_f.shape, 1)) * self.weights)
        print(local_conv_f.shape)
        return local_conv_f.sum(dim=1).transpose(2, 1).reshape((-1, self.out_channels, self.output_height, self.output_width)) + self.bias


layer_ = LocallyConnected2dTranspose([64,64], 49, 3, [5,5], (1,1), (2,2), (1,1), (2,2))
ht = torch.randn(1,49,64,64)
print(layer_(ht).shape)

```

best regards ^^

PS : I am not sure the last 2 lines of my code work...



cc @albanD @mruberry