Repository: pytorch
Issue: LR scheduler `get_lr()` bug
Labels: module: optimizer,triaged

## 🐛 Bug

I first found the possible bug when using `StepwiseLR` but then I found it in many `LR` modules. It seemed pretty much the same as [this issue](https://github.com/pytorch/pytorch/issues/22107). It is not affecting performance but may cause confusion.

I think at least for [`StepwiseLR`](https://github.com/pytorch/pytorch/blob/963f7629b591dc9750476faf1513bc7f1fb4d6de/torch/optim/lr_scheduler.py#L369) , it can be fixed by 
`
return [base_lr * self.gamma ** (self.last_epoch // self.step_size) for base_lr in self.base_lrs]
`
directly, but there might be other concerns? 

## To Reproduce

```py
import torch
from torch.nn import Parameter

score_table = {0: 'WRONG', 1: 'correct'}
scheduler_table = {'LambdaLR': torch.optim.lr_scheduler.LambdaLR(
                       torch.optim.SGD([Parameter(torch.randn(2, 2, requires_grad=True))], 0.1), 
                       lambda epoch: 0.1 ** epoch),
                   'MultiplicativeLR': torch.optim.lr_scheduler.MultiplicativeLR(
                       torch.optim.SGD([Parameter(torch.randn(2, 2, requires_grad=True))], 0.1), 
                       lambda epoch: 0.1),
                   'ExponentialLR': torch.optim.lr_scheduler.ExponentialLR(
                       torch.optim.SGD([Parameter(torch.randn(2, 2, requires_grad=True))], 0.1), 
                       0.1),
                   'StepLR': torch.optim.lr_scheduler.StepLR(
                       torch.optim.SGD([Parameter(torch.randn(2, 2, requires_grad=True))], 0.1), 
                       3, gamma=0.1),
                   'MultiStepLR': torch.optim.lr_scheduler.MultiStepLR(
                       torch.optim.SGD([Parameter(torch.randn(2, 2, requires_grad=True))], 0.1), 
                       milestones=[3, 6, 9], gamma=0.1),
                  }
for name, scheduler in scheduler_table.items():
    print(f'--------Evaluate scheduler {name}----------')

    for epoch in range(10):
        try:
            print('| epoch {:2d} | get_lr {:8.2e} | '
                  '_get_closed_form_lr {:8.2e} | optimizer.param.lr {:8.2e} | '
                  ' {:7s} |'.format(
                      epoch, scheduler.get_lr()[0], 
                      scheduler._get_closed_form_lr()[0], 
                      scheduler.optimizer.param_groups[0]['lr'],
                      score_table[scheduler.get_lr()[0]==scheduler.optimizer.param_groups[0]['lr']]
                  ))
        except AttributeError:
            print('| epoch {:3d} | get_lr {:8.2e} |  optimizer.param.lr {:8.2e} |' 
                  ' {:7s} |'.format(
                epoch, scheduler.get_lr()[0], 
                scheduler.optimizer.param_groups[0]['lr'],
                score_table[scheduler.get_lr()[0]==scheduler.optimizer.param_groups[0]['lr']], 
            ))

        scheduler.step()
```

## Actual behavior

```
--------Evaluate scheduler LambdaLR----------
| epoch   0 | get_lr 1.00e-01 |  optimizer.param.lr 1.00e-01 | correct |
| epoch   1 | get_lr 1.00e-02 |  optimizer.param.lr 1.00e-02 | correct |
| epoch   2 | get_lr 1.00e-03 |  optimizer.param.lr 1.00e-03 | correct |
| epoch   3 | get_lr 1.00e-04 |  optimizer.param.lr 1.00e-04 | correct |
| epoch   4 | get_lr 1.00e-05 |  optimizer.param.lr 1.00e-05 | correct |
| epoch   5 | get_lr 1.00e-06 |  optimizer.param.lr 1.00e-06 | correct |
| epoch   6 | get_lr 1.00e-07 |  optimizer.param.lr 1.00e-07 | correct |
| epoch   7 | get_lr 1.00e-08 |  optimizer.param.lr 1.00e-08 | correct |
| epoch   8 | get_lr 1.00e-09 |  optimizer.param.lr 1.00e-09 | correct |
| epoch   9 | get_lr 1.00e-10 |  optimizer.param.lr 1.00e-10 | correct |
--------Evaluate scheduler MultiplicativeLR----------
| epoch   0 | get_lr 1.00e-01 |  optimizer.param.lr 1.00e-01 | correct |
| epoch   1 | get_lr 1.00e-03 |  optimizer.param.lr 1.00e-02 | WRONG   |
| epoch   2 | get_lr 1.00e-04 |  optimizer.param.lr 1.00e-03 | WRONG   |
| epoch   3 | get_lr 1.00e-05 |  optimizer.param.lr 1.00e-04 | WRONG   |
| epoch   4 | get_lr 1.00e-06 |  optimizer.param.lr 1.00e-05 | WRONG   |
| epoch   5 | get_lr 1.00e-07 |  optimizer.param.lr 1.00e-06 | WRONG   |
| epoch   6 | get_lr 1.00e-08 |  optimizer.param.lr 1.00e-07 | WRONG   |
| epoch   7 | get_lr 1.00e-09 |  optimizer.param.lr 1.00e-08 | WRONG   |
| epoch   8 | get_lr 1.00e-10 |  optimizer.param.lr 1.00e-09 | WRONG   |
| epoch   9 | get_lr 1.00e-11 |  optimizer.param.lr 1.00e-10 | WRONG   |
--------Evaluate scheduler ExponentialLR----------
| epoch  0 | get_lr 1.00e-01 | _get_closed_form_lr 1.00e-01 | optimizer.param.lr 1.00e-01 |  correct |
| epoch  1 | get_lr 1.00e-03 | _get_closed_form_lr 1.00e-02 | optimizer.param.lr 1.00e-02 |  WRONG   |
| epoch  2 | get_lr 1.00e-04 | _get_closed_form_lr 1.00e-03 | optimizer.param.lr 1.00e-03 |  WRONG   |
| epoch  3 | get_lr 1.00e-05 | _get_closed_form_lr 1.00e-04 | optimizer.param.lr 1.00e-04 |  WRONG   |
| epoch  4 | get_lr 1.00e-06 | _get_closed_form_lr 1.00e-05 | optimizer.param.lr 1.00e-05 |  WRONG   |
| epoch  5 | get_lr 1.00e-07 | _get_closed_form_lr 1.00e-06 | optimizer.param.lr 1.00e-06 |  WRONG   |
| epoch  6 | get_lr 1.00e-08 | _get_closed_form_lr 1.00e-07 | optimizer.param.lr 1.00e-07 |  WRONG   |
| epoch  7 | get_lr 1.00e-09 | _get_closed_form_lr 1.00e-08 | optimizer.param.lr 1.00e-08 |  WRONG   |
| epoch  8 | get_lr 1.00e-10 | _get_closed_form_lr 1.00e-09 | optimizer.param.lr 1.00e-09 |  WRONG   |
| epoch  9 | get_lr 1.00e-11 | _get_closed_form_lr 1.00e-10 | optimizer.param.lr 1.00e-10 |  WRONG   |
--------Evaluate scheduler StepLR----------
| epoch  0 | get_lr 1.00e-01 | _get_closed_form_lr 1.00e-01 | optimizer.param.lr 1.00e-01 |  correct |
| epoch  1 | get_lr 1.00e-01 | _get_closed_form_lr 1.00e-01 | optimizer.param.lr 1.00e-01 |  correct |
| epoch  2 | get_lr 1.00e-01 | _get_closed_form_lr 1.00e-01 | optimizer.param.lr 1.00e-01 |  correct |
| epoch  3 | get_lr 1.00e-03 | _get_closed_form_lr 1.00e-02 | optimizer.param.lr 1.00e-02 |  WRONG   |
| epoch  4 | get_lr 1.00e-02 | _get_closed_form_lr 1.00e-02 | optimizer.param.lr 1.00e-02 |  correct |
| epoch  5 | get_lr 1.00e-02 | _get_closed_form_lr 1.00e-02 | optimizer.param.lr 1.00e-02 |  correct |
| epoch  6 | get_lr 1.00e-04 | _get_closed_form_lr 1.00e-03 | optimizer.param.lr 1.00e-03 |  WRONG   |
| epoch  7 | get_lr 1.00e-03 | _get_closed_form_lr 1.00e-03 | optimizer.param.lr 1.00e-03 |  correct |
| epoch  8 | get_lr 1.00e-03 | _get_closed_form_lr 1.00e-03 | optimizer.param.lr 1.00e-03 |  correct |
| epoch  9 | get_lr 1.00e-05 | _get_closed_form_lr 1.00e-04 | optimizer.param.lr 1.00e-04 |  WRONG   |
--------Evaluate scheduler MultiStepLR----------
| epoch  0 | get_lr 1.00e-01 | _get_closed_form_lr 1.00e-01 | optimizer.param.lr 1.00e-01 |  correct |
| epoch  1 | get_lr 1.00e-01 | _get_closed_form_lr 1.00e-01 | optimizer.param.lr 1.00e-01 |  correct |
| epoch  2 | get_lr 1.00e-01 | _get_closed_form_lr 1.00e-01 | optimizer.param.lr 1.00e-01 |  correct |
| epoch  3 | get_lr 1.00e-03 | _get_closed_form_lr 1.00e-02 | optimizer.param.lr 1.00e-02 |  WRONG   |
| epoch  4 | get_lr 1.00e-02 | _get_closed_form_lr 1.00e-02 | optimizer.param.lr 1.00e-02 |  correct |
| epoch  5 | get_lr 1.00e-02 | _get_closed_form_lr 1.00e-02 | optimizer.param.lr 1.00e-02 |  correct |
| epoch  6 | get_lr 1.00e-04 | _get_closed_form_lr 1.00e-03 | optimizer.param.lr 1.00e-03 |  WRONG   |
| epoch  7 | get_lr 1.00e-03 | _get_closed_form_lr 1.00e-03 | optimizer.param.lr 1.00e-03 |  correct |
| epoch  8 | get_lr 1.00e-03 | _get_closed_form_lr 1.00e-03 | optimizer.param.lr 1.00e-03 |  correct |
| epoch  9 | get_lr 1.00e-05 | _get_closed_form_lr 1.00e-04 | optimizer.param.lr 1.00e-04 |  WRONG   |
```
(I didn't test `CosineAnnealingLR` or `CyclicLR`)

## Expected behavior

`get_lr()[0] == scheduler.optimizer.param_groups[0]['lr']`

## Environment

 - PyTorch Version 1.7.1



cc @vincentqb