Repository: pytorch
Issue: `CosineAnnealingWarmRestarts` does not update parameters added with `add_param_group`
Labels: triaged,module: LrScheduler

### 🐛 Describe the bug

## Background

The `torch.optim.Optimizer.add_param_group` method is often used with transfer learning in order to unfreeze layers during training.  For example, this is the approach used in the PyTorch Lightning fine tuning callbacks:
https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.callbacks.BackboneFinetuning.html#pytorch_lightning.callbacks.BackboneFinetuning

## Issue

When using the `torch.optim.lr_scheduler.CosineAnnealingWarmRestarts` learning rate scheduler, any parameter groups that are added during training are not updated by the learning rate scheduler.

## Minimal example

```
## create a simple model with two layers

> model = th.nn.Sequential(collections.OrderedDict(a=th.nn.Linear(4, 8), b=th.nn.Linear(8, 2)))

## create an optimizer with a single parameter group, the first layer only

> opt = th.optim.Adam([dict(params=model.a.parameters(), lr=0.001),])

## create a learning rate scheduler tied to the optimizer

> sched = th.optim.lr_scheduler.CosineAnnealingWarmRestarts(opt, T_0=10)

## initial lr is as expected

> sched.get_last_lr()
[0.001]

## stepping updates the first param group as expected

> sched.step()

> sched.get_last_lr()
[0.0009755282581475768]

## add a new parameter group

> opt.add_param_group(dict(params=model.b.parameters(), lr=0.0001))

## after a single step operation, the lr for the second param group appears

> sched.get_last_lr()
[0.0009045084971874737]

> sched.step()

> sched.get_last_lr()
[0.0007938926261462366, 0.0001]

## after subsequent steps, the lr is not updated for the second parameter group!!

> sched.step()

> sched.get_last_lr()
[0.0006545084971874737, 0.0001]
```

## Insights

The `base_lrs` attribute on the `_LRScheduler` base class is created in the class initializer but is not updated after `add_param_group` is called:
https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L41

This attribute is later used by `CosineAnnealingWarmRestarts` in the `get_lr` method:
https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L1272

## Suggested fix

The `base_lrs` attribute could be changed to a property that is updated if the number of parameter groups does not match it's current length.

## Related issues

There seems to be a similar issue in the `ReduceLROnPlateau` scheduler described in issue #62475

Similar issues are also described in #53712

### Versions

Collecting environment information...
PyTorch version: 1.10.2+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000

Nvidia driver version: 470.103.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.6.3
[pip3] numpy==1.22.3
[pip3] pytorch-lightning==1.5.10
[pip3] segmentation-models-pytorch==0.2.1
[pip3] torch==1.10.2+cu113
[pip3] torchmetrics==0.7.2
[pip3] torchvision==0.11.3+cu113
[conda] Could not collect