Repository: pytorch
Issue: Learning rate change is not applied at designated iteration with a scheduler
Labels: module: optimizer,triaged

## 🐛 Bug

With a scheduler, the learning rate changes at the designated iteration, but it seems the same iteration still applies the learning rate before change.

## To Reproduce

A minimal example is attached:
[test_lr.zip](https://github.com/pytorch/pytorch/files/4823279/test_lr.zip)

Steps to reproduce the behavior:

1. Run `python main_steplr.py`; this applies a `torch.optim.lr_scheduler.StepLR` with `step_size=5` and `gamma=0.1`. The logs on my computer are as follows:

> iteration: 0
learning rate: 0.1
loss = 0.23123088479042053
iteration: 1
learning rate: 0.1        
loss = 0.11247935891151428
iteration: 2      
learning rate: 0.1
loss = 0.12026116997003555
iteration: 3
learning rate: 0.1
loss = 0.11437922716140747
iteration: 4
learning rate: 0.1
loss = 0.11330760270357132
iteration: 5
learning rate: 0.010000000000000002
loss = 0.11355291306972504
iteration: 6
learning rate: 0.010000000000000002
loss = 0.1208689734339714

2. Run `python main_fixlr.py`; this applies a fixed learning rate. The logs on my computer are as follows:

> iteration: 0
learning rate: 0.1
loss = 0.23123088479042053
iteration: 1      
learning rate: 0.1
loss = 0.11247935891151428
iteration: 2
learning rate: 0.1        
loss = 0.12026116997003555
iteration: 3
learning rate: 0.1
loss = 0.11437922716140747
iteration: 4
learning rate: 0.1
loss = 0.11330760270357132
iteration: 5
learning rate: 0.1
loss = 0.11355291306972504
iteration: 6
learning rate: 0.1
loss = 0.12065751850605011


<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

The above steps compare between a learning rate decreased at iteration 5 and fixing this learning rate.

In step 1, according the [documentation](https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.StepLR), the learning rate should be `0.1` if `iteration < 5`, and be `0.01` if `5 <= iteration < 10`. However, although learning rate changes in iteration 5 in step 1, the loss is the same in the iteration 5 of step 1 and step 2. In other words, in iteration 5, different learning rates lead to the same loss. I think the changed learn rate might not be correctly applied in the designated iteration in step 1.

## Environment

PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Microsoft Windows 10 专业版 (Pro)
GCC version: (Rev5, Built by MSYS2 project) 5.3.0
CMake version: version 3.15.5

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.2.89
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] numpydoc==0.9.1
[pip] torch==1.5.0
[pip] torchvision==0.6.0
[conda] blas                      1.0                         mkl    defaults
[conda] mkl                       2019.4                      245    defaults
[conda] mkl-service               2.3.0            py37hb782905_0    defaults
[conda] mkl_fft                   1.0.14           py37h14836fe_0    defaults
[conda] mkl_random                1.1.0            py37h675688f_0    defaults
[conda] pytorch                   1.5.0           py3.7_cuda102_cudnn7_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] torchvision               0.6.0                py37_cu102    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch


cc @vincentqb