Repository: pytorch
Issue: No speedup from channels_last with DataParallel
Labels: module: performance,triaged,module: data parallel,module: memory format

## 🐛 Bug

For single GPU training, channels last improves performance for many networks that I have tried, but I get no speedup at all when running the same code with DataParallel and two Titan RTX's with nvlink. 

## To Reproduce
Steps to reproduce the behavior:

Unfortunately, apex's imagenet example does not support DataParallel, and Pytorch's imagenet example doesn't use fp16 training (channels last seems to be a bad idea for 32 bit training at the moment, with an epoch of resnet50 taking several hours). So I have to point to my own implementation, which should be very straightforward and based on the aforementioned examples. You can get the repo with: 

`
git clone https://github.com/tstandley/imagenet_training.git
cd imagenet_training
`

On a single Titan RTX, resnet50, nvidia apex level O1...

`
CUDA_VISIBLE_DEVICES=1 python train_imagenet.py -a=resnet50 -t=imagenet/train -v=imagenet/val  --b=512 --workers=4 -n=DL_test --fp16 --channels_last
`

...finishes an epoch about 28 minutes.

The same code without --channels_last... 

`
CUDA_VISIBLE_DEVICES=1 python train_imagenet.py -a=resnet50 -t=imagenet/train -v=imagenet/val  --b=512 --workers=4 -n=DL_test --fp16
`

...takes 37 minutes or so. This is like a 25% speedup. Awesome!

Resnet101 takes about an hour without, and 45 minutes with. Also Awesome.

Xception (which uses grouped convolution heavily) takes about 54 minutes without and 41 minutes with. A similar speedup.

But with DataParallel on 2 Titan RTX w/ nvlink, resnet50 takes...

`
CUDA_VISIBLE_DEVICES=1,0 python train_imagenet.py -a=resnet50 -t=imagenet/train -v=imagenet/val  --b=512 --workers=8 -n=DL_test --fp16 --channels_last
`

... 20 minutes with channels last and 20 minutes without. It's the same.

resnet101 takes 32 minutes with and 32 minutes without. Also the same

Xception takes 32 minutes with and 30 without. It's actually slightly worse. 

## Expected behavior

A similar speedup for DataParallel models

## Environment
Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 19.10
GCC version: (Ubuntu 9.2.1-9ubuntu2) 9.2.1 20191008
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect (not installed outside pytorch)
GPU models and configuration:
GPU 0: TITAN RTX
GPU 1: TITAN RTX

Nvidia driver version: 440.82
cuDNN version: Could not collect (not installed outside pytorch)

Versions of relevant libraries:
[pip3] numpy==1.18.3
[pip3] torch==1.5.0
[pip3] torchvision==0.6.0
[conda] blas 1.0 mkl
[conda] cudatoolkit 10.2.89 hfd86e86_0
[conda] mkl 2020.0 166
[conda] mkl-service 2.3.0 py37he904b0f_0
[conda] mkl_fft 1.0.15 py37ha843d7b_0
[conda] mkl_random 1.1.0 py37hd6b4f25_0
[conda] numpy 1.18.1 py37h4f9e942_0
[conda] numpy-base 1.18.1 py37hde5b4d6_1
[conda] numpydoc 0.9.2 py_0 conda-forge
[conda] pytorch 1.5.0 py3.7_cuda10.2.89_cudnn7.6.5_0 pytorch
[conda] torchvision 0.6.0 py37_cu102 pytorch

cc @VitalyFedyunin @jamesr66a @ngimel