Repository: pytorch
Issue: Performance bugs exists in multiple convolution operations(e.g., `Convtranspose2d`) when useing the `groups` argument
Labels: module: cudnn,module: docs,module: convolution,triaged

### 🐛 Describe the bug

This issue should be related to [#70954](https://github.com/pytorch/pytorch/issues/70954).

The convolution performance problem of `Conv2d` has been mentioned in 70954 will actually affect multiple convolution operations, **including `Convtranspose2d`, `LazyConv2d`, `LazyConvTranspose2d`**.
As can be seen in the reproducible code provided below, even on the 1.13.0 version, after using the `group=4`, the running time of the model is significantly higher than expected.
To make matters worse, even in version 1.13.0, the relevant documentation (e.g., [ConvTranspose2d](https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html?highlight=convtranspose2d#torch.nn.ConvTranspose2d)) has **not added any hints and explanations for this performance problem**.

### To Reproduce
```python
import math
import time
import torch
import numpy as np
import torch.nn as nn

torch.backends.cudnn.benchmark = True

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
steps = 25
batch_size = 16
input_w = 224
dtype= torch.float32#
ch0=128
groups=4

# func_cls=torch.nn.LazyConvTranspose2d
# func_cls=torch.nn.LazyConv2d
func_cls=torch.nn.ConvTranspose2d

# # For LazyConv2d/LazyConvTranspose2d
# def build_model(ch: int, groups: int = 1) -> nn.Module:
#     return nn.Sequential(
#         func_cls(out_channels=ch, kernel_size=(1, 1)),
#         func_cls(out_channels=ch, kernel_size=(3, 3), padding=1, groups=groups),  # 0
#         func_cls(out_channels=ch, kernel_size=(3, 3), padding=1, groups=groups),  # 1
#         func_cls(out_channels=ch, kernel_size=(3, 3), padding=1, groups=groups),  # 2
#         func_cls(out_channels=ch, kernel_size=(3, 3), padding=1, groups=groups),  # 3
#         func_cls(out_channels=ch, kernel_size=(3, 3), padding=1, groups=groups),  # 4
#         func_cls(out_channels=ch, kernel_size=(3, 3), padding=1, groups=groups),  # 5
#         func_cls(out_channels=ch, kernel_size=(3, 3), padding=1, groups=groups),  # 6
#         func_cls(out_channels=ch, kernel_size=(3, 3), padding=1, groups=groups),  # 7
#         nn.MaxPool2d((input_w, input_w)),
#         nn.Flatten(),
#         nn.Linear(ch, 2),
#         nn.Softmax(1))

def build_model(ch: int, groups: int = 1) -> nn.Module:
    return nn.Sequential(
        func_cls(3, ch, (1, 1)),
        func_cls(ch, ch, (3, 3), padding=1, groups=groups),  # 0
        func_cls(ch, ch, (3, 3), padding=1, groups=groups),  # 1
        func_cls(ch, ch, (3, 3), padding=1, groups=groups),  # 2
        func_cls(ch, ch, (3, 3), padding=1, groups=groups),  # 3
        func_cls(ch, ch, (3, 3), padding=1, groups=groups),  # 4
        func_cls(ch, ch, (3, 3), padding=1, groups=groups),  # 5
        func_cls(ch, ch, (3, 3), padding=1, groups=groups),  # 6
        func_cls(ch, ch, (3, 3), padding=1, groups=groups),  # 7
        nn.MaxPool2d((input_w, input_w)),
        nn.Flatten(),
        nn.Linear(ch, 2),
        nn.Softmax(1))


def train_model(model: nn.Module, data) -> float:
    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
    loss_fn = nn.BCELoss()
    model.to(device)
    model.train()
    t0 = time.time()
    for (X, y) in data:
        X, y = X.to(device), y.to(device)

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return time.time() - t0


def perf_test(ch: int, groups: int, dtype) -> float:
    model = build_model(ch, groups).to(dtype)
    images = torch.from_numpy(np.full((batch_size, 3, input_w, input_w), 0.5)).to(dtype)
    labels = torch.from_numpy(np.full((batch_size, 2), 1)).to(dtype)
    train_model(model, [(images, labels)] * 2)  # Warmup
    return train_model(model, [(images, labels)] * steps)

dtype_s = str(dtype).replace("torch.", "")
dt0_last = float("nan")
dt0 = perf_test(ch0, 1, dtype)
dt0_ratio = f"{dt0 / dt0_last:.3f}" if not math.isnan(dt0_last) else ""
dt0_last = dt0
ch = ch0 * groups
dt = perf_test(ch, groups, dtype)
expected = dt0 * groups
ratio = dt / expected
print(f'expected time cost: {expected}')
print(f'real time cost: {dt}')
print(f'radio: {ratio}')
```

### Error Results

PyTorch 1.13.0, testing on `LazyConvTranspose2d`:
```
/opt/conda/lib/python3.9/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
  warnings.warn('Lazy modules are a new feature under heavy development '
expected time cost: 20.092201232910156
real time cost: 46.74727201461792
radio: 2.3266376577021295
```

PyTorch 1.13.0, testing on `LazyConv2d`:
```
/opt/conda/lib/python3.9/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
  warnings.warn('Lazy modules are a new feature under heavy development '
expected time cost: 20.051240921020508
real time cost: 46.57210874557495
radio: 2.322654689004887
```

PyTorch 1.13.0, testing on `ConvTranspose2d`:
```
expected time cost: 20.15345001220703
real time cost: 46.877477169036865
radio: 2.3260274117157596
```


### Expected behavior

Using the `groups` argument should speed up the models (as shown in many other experiments in [#70954](https://github.com/pytorch/pytorch/issues/70954)).
If it is not possible to speed up for larger channel settings, this unexpected performance bug **should be explained in the documentation of the relevant APIs**.
The current documentation (such as [ConvTranspose2d](https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html)) does not seem to have any clarification of the potential performance problems on the `groups` argument.

### Versions

```
[pip3] numpy==1.22.3
[pip3] torch==1.13.0
[pip3] torchtext==0.14.0
[pip3] torchvision==0.14.0
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.22.3           py39he7a7128_0  
[conda] numpy-base                1.22.3           py39hf524024_0  
[conda] pytorch                   1.13.0          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchtext                 0.14.0                     py39    pytorch
[conda] torchvision               0.14.0               py39_cu116    pytorch
```

cc @csarofeen @ptrblck @xwang233 @svekars @carljparker @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh