Repository: pytorch
Issue: [Feature Pitch] Full-batch optimization toolkit
Labels: module: optimizer,triaged

## 🚀 Feature
It would be great to have a library of optimization routines for the deterministic setting (a la `scipy.optimize`) using PyTorch autograd mechanics. I have written a [prototype library](https://github.com/rfeinman/pytorch-minimize) of PyTorch-based function minimizers and I'd like to guage the interest of the pytorch community. This could perhaps be a submodule of `torch.optim`, or it could be included elsewhere.

## Motivation

For a variety of optimization problems, deterministic (or "full-batch") routines such as Newton and Quasi-Newton methods are preferrable over vanilla gradient descent. Although PyTorch offers many routines for stochastic optimization, utilities for deterministic optimization are scarce; only L-BFGS is included in the `torch.optim` package, and it's modified for mini-batch training.

MATLAB and SciPy are the industry standards for deterministic optimization. These libraries have a comprehensive set of routines; however, automatic differentiation is not supported. Therefore, the user must specify 1st- and 2nd-order gradients explicitly (they must be known) or use finite-difference approximations. This limits the applications considerably.

It would be wonderful to have a library of deterministic optimization routines in PyTorch. As a standalone entity, the library would be preferable over SciPy/MATLAB: the user only has to provide a function (not jacobian/hessian functions), and analytical derivatives are always used. But perhaps more importantly, it would be great to have these resources available inside of PyTorch to use alongside all of its other great tools.

## Pitch

See complete pitch at [pytorch-minimize](https://github.com/rfeinman/pytorch-minimize).

My current library offers 8 different methods for unconstrained minimization. A complete description of these algorithms is provided in the readme. Constrained minimization is not currently implemented, but could perhaps be included in the future.

```python
import torch
from torchmin import minimize  # <- perhaps ``from torch.optim import minimize``

def rosen(x):
    return torch.sum(100*(x[..., 1:] - x[..., :-1]**2)**2 
                     + (1 - x[..., :-1])**2)

# initial point
x0 = torch.tensor([1., 8.])

# BFGS
result = minimize(rosen, x0, method='bfgs')

# Newton Conjugate Gradient
result = minimize(rosen, x0, method='newton-cg')
```

__methods:__ BFGS, L-BFGS, Conjugate Gradient (CG), Newton Conjugate Gradient (NCG), Newton Exact, Dogleg, Trust-Region Exact, Trust-Region NCG

## Additional context

`torch.optim` is tailored to training neural networks with stochastic (mini-batch) gradient descent. I envision a `minimize` package that is tailored to a different set of problems, namely those of the deterministic or "full-batch" setting. For example, we may want to do various manipulations with a trained neural net such as: search for adversarial examples, traverse a latent space (e.g. styleGAN), or find a MAP estimate of latent variables. In addition, we may want to solve an optimization sub-problem inside the loop of our neural network optimization, as for example in EM algorithms like sparse dictionary learning.

cc @mruberry @kurtamohler @vincentqb