Repository: pytorch
Issue: Channels last doesn't improve speed when using SyncBatchNorm
Labels: module: performance,triaged,module: memory format

## 🐛 Bug

Training with channels_last improves training speed significantly when using DistributedDataParallel.

But many projects benefit from SyncBatchNorm which allows training with smaller batch sizes because batch statistics can be pooled from multiple GPUs. Sometimes this improvement in model prediction quality can be extremely important.

Unfortunately, SyncBatchNorm doesn't work well with channels_last. Using channels_last actually causes training epochs to take several percent longer when SyncBatchNorm is being applied.

## To Reproduce

I've noticed this performance issue across a number of projects, but a good example can be found here:

https://github.com/tstandley/imagenet_training

here we train resnet152 with and without channels last, but with SBN.

`CUDA_VISIBLE_DEVICES=1,0 python train_imagenet.py -a=resnet152 -t=/data2/imagenet_data/train/ -v=/data2/imagenet_data/val/ -b=256 -j=4 --fp16 -sbn` (sync batch norm, but no channels_last)
results in epochs that take about 59 minutes

`CUDA_VISIBLE_DEVICES=1,0 python train_imagenet.py -a=resnet152 -t=/data2/imagenet_data/train/ -v=/data2/imagenet_data/val/ -b=256 -j=4 --fp16 -sbn -cl` (sync batch norm, and channels_last)
results in epochs that take about 65 minutes

For what it's worth, using channels last and no sbn results in epochs that take 40 minutes, and using neither results in epochs that take 47 minutes.
## Expected behavior

Ideally, channels last memory format should improve the speed even when using SyncBatchNorm. Thanks!

## Environment
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: TITAN RTX
GPU 1: TITAN RTX
GPU 2: TITAN RTX

Nvidia driver version: 460.32.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] numpydoc==1.1.0
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1  
[conda] pytorch                   1.7.1           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] torchaudio                0.7.2                      py38    pytorch
[conda] torchvision               0.8.2                py38_cu102    pytorch

cc @VitalyFedyunin @jamesr66a @ppwwyyxx @ngimel