Repository: pytorch
Issue: Batchnorm2d in PT1.6 couldn't turn off track_running_stats
Labels: module: docs,module: nn,triaged

## 🐛 Bug

I have a model trained with BN layers and tracking running parameters ON. After training, I found I couldn't turn the tracking_running_stats off for inference in PT1.6. Everything worked expectedly in PT1.5. 
The results in PT1.6 with tracking_running_stats ON or OFF are the same after Batchnorm which means it always used tracked mean&variance. But PT1.5 can give different outputs for tracking_running_stats ON or OFF which is expected.

Steps to reproduce the behavior:

```
import torch as th
import numpy as np 

def turn_off_running_param(m):

    if isinstance(m, (th.nn.BatchNorm2d, th.nn.InstanceNorm2d)):
        m.track_running_stats=False

class sample_module(th.nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()

        self.conv1_1 = th.nn.Conv2d(in_ch, out_ch, 7, padding=3)
        self.bn = th.nn.BatchNorm2d(out_ch)
        self.nonlinear = th.nn.LeakyReLU(negative_slope=0.2, inplace=True)

        ### rand buffer 
        self.bn.running_mean = (th.ones((out_ch))*100.0)
        self.bn.running_var = (th.ones(out_ch)*1.0)

    def forward(self, x):
        
        x_conv = self.conv1_1(x)
        x = self.bn(x_conv)

        return x_conv, x


if __name__ == "__main__":

    model = sample_module(3, 4).eval().cuda()
    image = th.rand((1, 3, 4, 4)).cuda()

    ### BN track_running_stats ON
    conv_out_tracking_on, bn_out_tracking_on = model(image)

    ### BN track_running_stats OFF
    model.apply(turn_off_running_param)
    conv_out_tracking_off, bn_out_tracking_off = model(image)

    print("Tracking off model:\n", model)
    print("test pytorch:", th.__version__)
    if th.mean(th.abs(conv_out_tracking_on - conv_out_tracking_off)) != 0:
        print("?????? Different conv output, unexpected!!!")
        print("conv_out_tracking_on:", conv_out_tracking_on.cpu().data.numpy()[0,0,:,:].tolist())
        print("conv_out_tracking_off:", conv_out_tracking_off.cpu().data.numpy()[0,0,:,:].tolist())
    else:
        print("### Same conv output, expected!!!")

    if th.mean(th.abs(bn_out_tracking_on - bn_out_tracking_off)) != 0:
        print("### Different BN outputs, expected!!!")
        print("bn_out_tracking_on:", bn_out_tracking_on.cpu().data.numpy()[0,0,:,:].tolist())
        print("bn_out_tracking_off:", bn_out_tracking_off.cpu().data.numpy()[0,0,:,:].tolist())
    else:
        print("?????? Same BN output, unexpected!!!")
```


/************************ TEST *********************************/

/*********** PT1.6 results ************/
(dgxenv) [yuecheng@pit101-dgx123 care_garage]$ python bn_test.py 
```
Tracking off model:
 sample_module(
  (conv1_1): Conv2d(3, 4, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3))
  (bn): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
  (nonlinear): LeakyReLU(negative_slope=0.2, inplace=True)
)
test pytorch: 1.6.0
### Same conv output, expected!!!
?????? Same BN output, unexpected!!!
```


/*********** PT1.5 results ************/
(pt1.5_for_yuecheng) [yuecheng@pit101-dgx123 care_garage]$ python bn_test.py 
```
Tracking off model:
 sample_module(
  (conv1_1): Conv2d(3, 4, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3))
  (bn): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
  (nonlinear): LeakyReLU(negative_slope=0.2, inplace=True)
)
test pytorch: 1.5.1
### Same conv output, expected!!!
### Different BN outputs, expected!!!
bn_out_tracking_on: [[-100.20112609863281, -100.19847869873047, -99.9124526977539, -99.82685089111328], [-100.03643798828125, -100.18742370605469, -100.01443481445312, -100.0298843383789], [-100.31416320800781, -100.27574920654297, -100.29280090332031, -99.8615493774414], [-100.10736083984375, -100.09064483642578, -100.03619384765625, -100.07273864746094]]
bn_out_tracking_off: [[-0.7678689956665039, -0.7493481636047363, 1.24749755859375, 1.8450984954833984], [0.3819170296192169, -0.6721907258033752, 0.5355216860771179, 0.4277057647705078], [-1.557004690170288, -1.2887893915176392, -1.4078713655471802, 1.6028516292572021], [-0.11317643523216248, 0.0034803077578544617, 0.38364145159721375, 0.12853538990020752]]

```

 - PyTorch Version (e.g., 1.0): PT1.5, PT1.6
 - OS (e.g., Linux): Linux
 - How you installed PyTorch (`conda`, `pip`, source): conda
 - Build command you used (if compiling from source): conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
 - Python version:3.7.8 
 - CUDA/cuDNN version: CUDA 10.1
 - GPU models and configuration: 
 - Any other relevant information:



cc @jlin27 @albanD @mruberry