Repository: pytorch
Issue: LambdaLR changes the learning rate in an undesired way
Labels: triaged,module: LrScheduler

### 🐛 Describe the bug

I found the **LambdaLR** scheduler changes the preset learning rate to 1e-6 rather than my preset during the initialization of the optimizer, as the following example shows:

```python
import torch
import torch.nn as nn
from torch.optim.lr_scheduler import LambdaLR

model = nn.Sequential(
            nn.Linear(1, 16,bias=True),
            nn.ReLU(),
            nn.Linear(16, 16, bias=True),
            nn.ReLU()
)
lr = 0.001
lamda = 0.0001
optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=lamda)
print(optimizer)

# Before we apply the LambdaLR, the lr is 0.001, just as what I set before.
#Adam (
#Parameter Group 0
#    amsgrad: False
#    betas: (0.9, 0.999)
#    eps: 1e-08
#    lr: 0.001
#    maximize: False
#    weight_decay: 0.0001
#)

update_func = lambda epoch: lr  # as the simplest example, we don't change lr with epoch.
scheduler = LambdaLR(optimizer, lr_lambda=update_func)   
print(optimizer)

# After wrapped by the scheduler, the lr changes to 1e-6, and the initial_lr stores the user preset lr.
#Adam (
#Parameter Group 0
#    amsgrad: False
#    betas: (0.9, 0.999)
#    eps: 1e-08
#    initial_lr: 0.001
#    lr: 1e-06
#    maximize: False
#    weight_decay: 0.0001
#)



print(scheduler.get_lr())
print(scheduler.get_last_lr())
# Using scheduler function to print learning rate scheduled by the scheduler, it shows 1e-6 rather than 1e-3 as we set.
# UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
#  warnings.warn("To get the last learning rate computed by the scheduler,
#[1e-06]
#[1e-06]

```

More importantly, I have tried to train by network, and I found it conversed just as setting lr=1e-6 without the scheduler. So I wondering there are something wrong in the LambdaLR function.

### Versions

I found the issue on a Linux system with torch-1.10 and GPU, and this simple test script is performed on my mac with latest version of pytorch: torch-1.11.0. I have compared with the source code of ``torch.optim.lr_scheduler.py`` of current main branch with my local file, and they are basically the same. So I believe this problem may be an undiscovered bug.

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.4 (x86_64)
GCC version: Could not collect
Clang version: 11.0.0 (clang-1100.0.33.16)
CMake version: version 3.23.2
Libc version: N/A

Python version: 3.7.7 (default, Mar 26 2020, 10:32:53)  [Clang 4.0.1 (tags/RELEASE_401/final)] (64-bit runtime)
Python platform: Darwin-21.5.0-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.11.0
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==2.0.3
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.12
[pip3] torch-spline-conv==1.2.1
[pip3] torch-summary==1.4.3
[pip3] torchfile==0.1.0
[pip3] torchvision==0.12.0
[pip3] torchviz==0.0.2
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] torch                     1.11.0                   pypi_0    pypi
[conda] torch-cluster             1.5.9                    pypi_0    pypi
[conda] torch-geometric           2.0.3                    pypi_0    pypi
[conda] torch-scatter             2.0.9                    pypi_0    pypi
[conda] torch-sparse              0.6.12                   pypi_0    pypi
[conda] torch-spline-conv         1.2.1                    pypi_0    pypi
[conda] torch-summary             1.4.3                    pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.12.0                   pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi