Repository: pytorch
Issue: Minumul LR is never reached in 
Labels: module: optimizer,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

The learning rate in `CosineAnnealingWarmRestarts` is calculated as follows:

 `self.eta_min + (base_lr - self.eta_min) * (1 + math.cos(math.pi * self.T_cur / self.T_i)) / 2` ([lr_scheduler.py#L974](https://github.com/pytorch/pytorch/blob/5b0f40048899e398d286fe7b55f297991f93ba2c/torch/optim/lr_scheduler.py#L974))

It means that the learning rate will be equal to the minumum value set by user (`self.eta_min`) if and only if `self.T_cur ==   self.T_i`. But this even will never happen, becaues in [lr_scheduler.py#L1017](https://github.com/pytorch/pytorch/blob/5b0f40048899e398d286fe7b55f297991f93ba2c/torch/optim/lr_scheduler.py#L1017) we have `self.T_cur = epoch % self.T_0` , i.e. `self.T_cur < self.T_0`. Analougously from [lr_scheduler.py#L102](https://github.com/pytorch/pytorch/blob/5b0f40048899e398d286fe7b55f297991f93ba2c/torch/optim/lr_scheduler.py#L1020) we can see  `self.T_cur` is always `< self.T_i`.


## To Reproduce

Steps to reproduce the behavior:

```
import torch
import matplotlib.pyplot as plt

x = torch.zeros((10,), requires_grad=True)
optimizer = torch.optim.Adam([x], lr=0.1)
scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(
             optimizer, 
             T_0=10, # iters before the first restart
             T_mult=1, 
             eta_min=3.2e-6,  #min lr
)
lrs = []
iters = 21
for i in range(iters):
    lrs.append(scheduler.get_last_lr())
    print(i, scheduler.get_last_lr())
    scheduler.step()
    
plt.figure(figsize=(10,4))
plt.plot(lrs)
```
Output:

> 0 [0.1]
1 [0.09755290412433161]
2 [0.09045115529155638]
3 [0.07938992215822]
4 [0.06545195529155637]
5 [0.05000160000000001]
6 [0.03455124470844364]
7 [0.020613277841780017]
8 [0.009552044708443634]
9 [0.002450295875668396]
10 [0.1]
11 [0.09755290412433161]
12 [0.09045115529155638]
13 [0.07938992215822]
14 [0.06545195529155637]
15 [0.05000160000000001]
16 [0.03455124470844364]
17 [0.020613277841780017]
18 [0.009552044708443634]
19 [0.002450295875668396]
20 [0.1]


![image](https://user-images.githubusercontent.com/1690869/98445001-7a035280-2115-11eb-9626-2b79ea1c21c0.png)


<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

From documentation one would expect that the learning rate swings to the minimum value `eta_min`every T iterations, which actually works like this in `CosineAnnealingLR`, for example.

> 0 [0.1]
> 1 [0.09698472753110217]
> 2 [0.08830259648483992]
> 3 [0.0750008]
> 4 [0.05868373104626227]
> 5 [0.04131946895373776]
> 6 [0.02500240000000001]
> 7 [0.011700603515160097]
> 8 [0.003018472468897842]
> 9 [3.2e-06]
> 10 [0.1]
> 11 [0.09698472753110217]
> 12 [0.08830259648483992]
> 13 [0.0750008]
> 14 [0.05868373104626227]
> 15 [0.04131946895373776]
> 16 [0.02500240000000001]
> 17 [0.011700603515160097]
> 18 [0.003018472468897842]
> 19 [3.2e-06]
> 20 [0.1]

<!-- A clear and concise description of what you expected to happen. -->

## Environment
 - PyTorch master f23a2a11153a7feba0c3c9aa01f68996569144af


cc @vincentqb