Repository: pytorch
Issue: ReduceLROnPlateau fails for negative input
Labels: module: docs,module: optimizer,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

The formula in `torch.optim.lr_scheduler.ReduceLROnPlateau` ([documentation](https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau)) that measures improvement is wrong for a negative metric in relative mode. This forumla is in the method `is_better`.

Specifically in relative mode, if there is a worse performance than before, the formula may return that the performance improved.

## To Reproduce

Steps to reproduce the behavior:

1. Initialize scheduler with options `mode='min'`, `threshold=0.1`, `patience=0` and `threshold_mode='rel'`
1. call `step` with initial negative value `a`. E.g. `scheduler.step(-1)`
1. call `step` with higher value `b` than before. `b`should be in the interval `(0.9*a, a)`. E.g. `scheduler.step(-.95)`
1. Notice no change in the learning rate despite worse performance and no patience.

```
import torch

model = torch.nn.Linear(1, 1)
opt = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(opt, mode="min", factor=0.1, threshold=0.1, patience=0, threshold_mode='rel', verbose=True)

# status
print(opt.param_groups[0]["lr"])
# >>> 0.1

# do epoch 1
scheduler.step(-1.0)

# do epoch 2
scheduler.step(-0.95)

# should be 0.01
print(opt.param_groups[0]["lr"])
# >>> 0.1
```

(Analogous for `mode=max`)

## Expected behavior

`ReduceLROnPlateau.is_better(self, -0.95, -1)` should return `False`. Which would result in `opt.param_groups[0]["lr"] == 0.01` in the example above (barring numeric inaccuracies).

## Environment

PyTorch version: 1.7.0+cpu
Is debug build: True
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Home
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.7 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce GTX 950M
Nvidia driver version: 388.57
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.17.0+mkl
[pip3] torch==1.7.0+cpu
[pip3] torchaudio==0.7.0
[pip3] torchtext==0.6.0
[pip3] torchvision==0.8.1+cpu
[conda] Could not collect



## Additional context

For positive numbers relative increase can be described by `(current - previous)/previous` (multiply by -1 for decrease in case of minimization), from which the formular in `is_better` is derived, however this does not work if `previous` is negative. The relative change should instead be: `(current - previous)/|previous|`.

In the [source code (line 662)](https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py) this is represented by (Case `mode='min' and threshold_mode == 'rel'`)
```
rel_epsilon = 1. - self.threshold
return a < best * rel_epsilon
```
which should rather be
```
return best - a > self.threshold * abs(best)
```

And analogous for the case `mode='max' and threshold_mode == 'rel'`
```
return a - best > self.threshold * abs(best)
```

This behavior was already noticed in #38622 which believes this to be an error in the documentation rather than the code.

cc @jlin27 @mruberry @vincentqb