Repository: pytorch
Issue: InstanceNorm doesn't preserve memory format
Labels: triaged,module: memory format,module: norms and normalization

### 🐛 Describe the bug

As pointed out here https://github.com/pytorch/pytorch/issues/67976, if the input of instance norm is in channels_last format the output is not in channels_last format. This basically means instance norm pays the cost of memory format conversion from channels_last => non-channels_last and the preceding operation(s) (usually conv) pays the cost of non-channels_last => channels_last. 

The reason for not preserving memory format is the current implementation of InstanceNorm. Current implementation converts N,C,H,W => contiguous =>  1, N*C, H, W => BatchNorm. If input is in channels_last format, this method doesn't work as N and C dim are not contiguous in memory so may be we need a new implementation of instanceNorm that works on channels_last memory format.  

```
import torch
import torch.nn as nn


m = nn.InstanceNorm2d(100)

input = torch.randn(20, 100, 35, 45)
input_channels_last = torch.randn(20, 100, 35, 45).to(memory_format=torch.channels_last)

print("#"*20)
print(m)
print(f"input channels last: {input.is_contiguous(memory_format=torch.channels_last)}")
output = m(input)
print(f"output channels last: {output.is_contiguous(memory_format=torch.channels_last)}")
print("#"*10)
print(f"input channels last: {input_channels_last.is_contiguous(memory_format=torch.channels_last)}")
output = m(input_channels_last)
print(f"output channels last: {output.is_contiguous(memory_format=torch.channels_last)}")
```

Output: 
```
####################
InstanceNorm2d(100, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
input channels last: False
output channels last: False
##########
input channels last: True
output channels last: False
```

### Versions

Collecting environment information...
PyTorch version: 1.10.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.27

Python version: 3.8.12 (default, Sep 10 2021, 00:16:05)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.11.0-34-generic-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080
Nvidia driver version: 470.57.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] pytorch-ranger==0.1.1
[pip3] torch==1.10.1+cu113
[pip3] torch-optimizer==0.1.0
[pip3] torch-tb-profiler==0.3.1
[pip3] torchaudio==0.10.1+cu113
[pip3] torchmetrics==0.6.2
[pip3] torchvision==0.11.2+cu113
[conda] Could not collect

cc @VitalyFedyunin @jamesr66a