Repository: pytorch
Issue: Batchnorm2D does behaves differently with different shapes when batch_size=1
Labels: module: nn,triaged,module: norms and normalization

## 🐛 Bug

Batchnorm2D does behaves differently with different shapes when batch_size=1. Throwing an error when batch_size=1 makes sense because batch statistics (std dev) cannot be calculated using only one sample. However, the bug report describes the case when it does or doesn't throw error based on the input shape (other than batch_size).

## To Reproduce

Steps to reproduce the behavior:

1.
```
import torch
import torch.nn as nn
import torch.nn.functional as F

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find("Conv") != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find("BatchNorm2d") != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)


class BaseBlock(nn.Module):
    def __init__(
        self,
        in_channels,
        out_channels,
        kernel_size,
        stride,
        padding,
        residual=False,
        *args,
        **kwargs
    ):
        super().__init__(*args, **kwargs)
        self.conv = nn.Sequential(
            nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding),
            nn.BatchNorm2d(out_channels),
        )
        self.act = nn.ReLU()
        self.residual = residual
        self.apply(weights_init)

    def forward(self, x):
        out = self.conv(x)
        if self.residual:
            out += x
        return self.act(out)


class ConvSyncNet(nn.Module):
    def __init__(self):
        super(ConvSyncNet, self).__init__()

        self.image_encoder = nn.Sequential(
            BaseBlock(15, 32, kernel_size=(7, 7), stride=1, padding=3),
            BaseBlock(32, 64, kernel_size=5, stride=(1, 2), padding=1),
            BaseBlock(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(64, 128, kernel_size=3, stride=2, padding=1),
            BaseBlock(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(128, 256, kernel_size=3, stride=2, padding=1),
            BaseBlock(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(256, 512, kernel_size=3, stride=2, padding=1),
            BaseBlock(512, 512, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(512, 512, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(512, 512, kernel_size=3, stride=2, padding=1),
            BaseBlock(512, 512, kernel_size=3, stride=1, padding=0),
            BaseBlock(512, 512, kernel_size=1, stride=1, padding=0),
        )

        self.audio_encoder = nn.Sequential(
            BaseBlock(1, 32, kernel_size=3, stride=1, padding=1),
            BaseBlock(32, 32, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(32, 32, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(32, 64, kernel_size=3, stride=(3, 1), padding=1),
            BaseBlock(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(64, 128, kernel_size=3, stride=3, padding=1),
            BaseBlock(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(128, 256, kernel_size=3, stride=(3, 2), padding=1),
            BaseBlock(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
            BaseBlock(256, 512, kernel_size=3, stride=1, padding=0),
            BaseBlock(512, 512, kernel_size=1, stride=1, padding=0),
        )

    def forward(self, audio_sequences, image_sequences):
        image_embedding = self.image_encoder(image_sequences)
        audio_embedding = self.audio_encoder(audio_sequences)

        # audio_embedding = audio_embedding.view(audio_embedding.size(0), -1)
        # image_embedding = image_embedding.view(image_embedding.size(0), -1)

        audio_embedding = F.normalize(audio_embedding)
        image_embedding = F.normalize(image_embedding)

        return audio_embedding, image_embedding
```
2. Using the following code throws an error:
```
syncnet = networks.ConvSyncNet()
batch_size = 1
audio_sequences = torch.zeros((batch_size, 1, 80, 16))
image_sequences = torch.zeros((batch_size, 15, 48, 96))
audio_embedding, image_embedding = syncnet(audio_sequences, image_sequences)
print(audio_embedding.shape, image_embedding.shape)
```
3. Using the following code does not throws an error:

```
syncnet = networks.ConvSyncNet()
batch_size = 1
audio_sequences = torch.zeros((batch_size, 1, 80, 80))
image_sequences = torch.zeros((batch_size, 15, 96, 96))
audio_embedding, image_embedding = syncnet(audio_sequences, image_sequences)
print(audio_embedding.shape, image_embedding.shape)

```
output: 
torch.Size([1, 512, 1, 12]) torch.Size([1, 512, 4, 1])

env info:

```
Collecting environment information...
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.8 (default, Feb 24 2021, 21:46:12)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-135-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000

Nvidia driver version: 455.32.00
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] gpytorch==1.4.1
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] pytorch-ignite==0.4.4
[pip3] pytorch-lightning==1.2.10
[pip3] pytorch-msssim==0.2.1
[pip3] torch==1.9.0
[pip3] torchaudio==0.9.0a0+33b2469
[pip3] torchcontrib==0.0.2
[pip3] torchmetrics==0.2.0
[pip3] torchvision==0.10.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.1               h6406543_8    conda-forge
[conda] gpytorch                  1.4.1                    pypi_0    pypi
[conda] mkl                       2021.3.0           h06a4308_520
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.0            py38h42c9631_2
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.21.2           py38h20f2e39_0
[conda] numpy-base                1.21.2           py38h79a1101_0
[conda] pytorch                   1.9.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
[conda] pytorch-ignite            0.4.4                    pypi_0    pypi
[conda] pytorch-lightning         1.2.10                   pypi_0    pypi
[conda] pytorch-msssim            0.2.1                    pypi_0    pypi
[conda] torchaudio                0.9.0                      py38    pytorch
[conda] torchcontrib              0.0.2                    pypi_0    pypi
[conda] torchmetrics              0.2.0                    pypi_0    pypi
[conda] torchvision               0.10.0               py38_cu111    pytorch
```

cc @albanD @mruberry @jbschlosser @walterddr