pytorch_optimizer 3.9.0
optimizer & lr scheduler & objective function collections in PyTorch
License: Apache-2.0
Requires Python: >=3.8
Keywords: pytorch, deep-learning, optimizer, lr scheduler, A2Grad, Alice, ASGD, AccSGD, AdaBelief, AdaBound, AdaDelta, AdaFactor, AdaGC, AdaMax, AdaMuon, AdamG, AdaMod, AdaNorm, AdaPNM, AdaSmooth, AdEMAMix, Simplified-AdEMAMix, ADOPT, AdaHessian, Adai, Adalite, AdaLomo, AdamMini, AdamP, AdamS, Adan, AggMo, Aida, AliG, Amos, Apollo, APOLLO, AvaGrad, bSAM, BCOS, CAME, Conda, DAdaptAdaGrad, DAdaptAdam, DAdaptAdan, DAdaptSGD, DAdaptLion, DeMo, DiffGrad, EmoFact, EmoLynx, EmoNavi, EmoNeco, EmoZeal, EXAdam, FAdam, Fira, FOCUS, FriendlySAM, Fromage, FTRL, GaLore, Grams, Gravity, GrokFast, GSAM, Kate, Lamb, LaProp, LARS, Lion, LOMO, Lookahead, MADGRAD, MARS, MSVAG, Muno, Nero, NovoGrad, OrthoGrad, PAdam, PCGrad, PID, PNM, Prodigy, PSGD, QHAdam, QHM, RACS, RAdam, Ranger, Ranger21, RotoGrad, SAM, GCSAM, LookSAM, ScheduleFreeSGD, ScheduleFreeAdamW, ScheduleFreeRAdam, SCION, SGDP, Shampoo, ScalableShampoo, SGDW, SignSGD, SM3, SOAP, SopihaH, SPAM, StableSPAM, SPlus, SRMM, StableAdamW, SWATS, TAM, Tiger, TRAC, VSGD, WSAM, Yogi, BCE, BCEFocal, Focal, FocalCosine, SoftF1, Dice, LDAM, Jaccard, Bi-Tempered, Tversky, FocalTversky, LovaszHinge, bitsandbytes, WSD, QGaLore

# pytorch-optimizer

|         |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Build   | ![workflow](https://github.com/kozistr/pytorch_optimizer/actions/workflows/ci.yml/badge.svg?branch=main) [![Documentation Status](https://readthedocs.org/projects/pytorch-optimizers/badge/?version=latest)](https://pytorch-optimizers.readthedocs.io/en/latest/?badge=latest)                                                                                                                                                                                                                                                                                                        |
| Quality | [![codecov](https://codecov.io/gh/kozistr/pytorch_optimizer/branch/main/graph/badge.svg?token=L4K00EA0VD)](https://codecov.io/gh/kozistr/pytorch_optimizer) [![CodeQL](https://github.com/kozistr/pytorch_optimizer/actions/workflows/pr_codeql.yml/badge.svg)](https://github.com/kozistr/pytorch_optimizer/actions/workflows/pr_codeql.yml) ![black](https://img.shields.io/badge/code%20style-black-000000.svg) [![ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v1.json)](https://github.com/charliermarsh/ruff) |
| Package | [![PyPI version](https://badge.fury.io/py/pytorch-optimizer.svg)](https://badge.fury.io/py/pytorch-optimizer) [![PyPI pyversions](https://img.shields.io/pypi/pyversions/pytorch-optimizer.svg)](https://pypi.python.org/pypi/pytorch-optimizer/)                                                                                                                                                                                                                                                                                                                                       |
| Status  | [![PyPi download](https://static.pepy.tech/badge/pytorch-optimizer)](https://pepy.tech/project/pytorch-optimizer) [![PyPi month download](https://static.pepy.tech/badge/pytorch-optimizer/month)](https://pepy.tech/project/pytorch-optimizer)                                                                                                                                                                                                                                                                                                                                         |
| License | [![apache](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

## The reasons why you use `pytorch-optimizer`.

* Wide range of supported optimizers. Currently, **131 optimizers (+ `bitsandbytes`, `qgalore`, `torchao`)**, **16 lr schedulers**, and **13 loss functions** are supported!
* Including many variants such as `ADOPT`, `Cautious`, `AdamD`, `StableAdamW`, and `Gradient Centrailiaztion`
* Easy to use, clean, and tested codes
* Active maintenance
* Somewhat a bit more optimized compared to the original implementation

Highly inspired by [pytorch-optimizer](https://github.com/jettify/pytorch-optimizer).

## Getting Started

For more, see the [stable documentation](https://pytorch-optimizers.readthedocs.io/en/stable/) or [latest documentation](https://pytorch-optimizers.readthedocs.io/en/latest/).

Most optimizers are under MIT or Apache 2.0 license, but a few optimizers like `Fromage`, `Nero` have `CC BY-NC-SA 4.0 license`, which is non-commercial. 
So, please double-check the license before using it at your work.

### Installation

```bash
$ pip3 install pytorch-optimizer
```

From `v2.12.0`, `v3.1.0`, you can use `bitsandbytes`, `q-galore-torch`, `torchao` optimizers respectively!
please check [the bnb requirements](https://github.com/TimDettmers/bitsandbytes?tab=readme-ov-file#tldr), [q-galore-torch installation](https://github.com/VITA-Group/Q-GaLore?tab=readme-ov-file#install-q-galore-optimizer), [torchao installation](https://github.com/pytorch/ao?tab=readme-ov-file#installation)
 before installing it.

From `v3.0.0`, drop `Python 3.7` support. However, you can still use this package with `Python 3.7` by installing with `--ignore-requires-python` option.

### Simple Usage

```python
from pytorch_optimizer import AdamP

model = YourModel()
optimizer = AdamP(model.parameters())

# or you can use optimizer loader, simply passing a name of the optimizer.

from pytorch_optimizer import load_optimizer

optimizer = load_optimizer(optimizer='adamp')(model.parameters())

# if you install `bitsandbytes` optimizer, you can use `8-bit` optimizers from `pytorch-optimizer`.

optimizer = load_optimizer(optimizer='bnb_adamw8bit')(model.parameters())
```

Also, you can load the optimizer via `torch.hub`.

```python
import torch

model = YourModel()

opt = torch.hub.load('kozistr/pytorch_optimizer', 'adamp')
optimizer = opt(model.parameters())
```

If you want to build the optimizer with parameters & configs, there's `create_optimizer()` API.

```python
from pytorch_optimizer import create_optimizer

optimizer = create_optimizer(
    model,
    'adamp',
    lr=1e-3,
    weight_decay=1e-3,
    use_gc=True,
    use_lookahead=True,
    use_orthograd=False,
)
```

## Supported Optimizers

You can check the supported optimizers with below code.

```python
from pytorch_optimizer import get_supported_optimizers

supported_optimizers = get_supported_optimizers()
```

or you can also search them with the filter(s).

```python
from pytorch_optimizer import get_supported_optimizers

get_supported_optimizers('adam*')
# ['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw']

get_supported_optimizers(['adam*', 'ranger*'])
# ['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw', 'ranger', 'ranger21']
```

| Optimizer             | Description                                                                                                    | Official Code                                                                                                  | Paper                                                                                                                | Citation                                                                                                                            |
|-----------------------|----------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| AdaBelief             | *Adapting Step-sizes by the Belief in Observed Gradients*                                                      | [github](https://github.com/juntang-zhuang/Adabelief-Optimizer)                                                | <https://arxiv.org/abs/2010.07468>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2020arXiv201007468Z/exportcitation)                                                        |
| AdaBound              | *Adaptive Gradient Methods with Dynamic Bound of Learning Rate*                                                | [github](https://github.com/Luolc/AdaBound/blob/master/adabound/adabound.py)                                   | <https://openreview.net/forum?id=Bkg3g2R9FX>                                                                         | [cite](https://github.com/Luolc/AdaBound#citing)                                                                                    |
| AdaHessian            | *An Adaptive Second Order Optimizer for Machine Learning*                                                      | [github](https://github.com/amirgholami/adahessian)                                                            | <https://arxiv.org/abs/2006.00719>                                                                                   | [cite](https://github.com/amirgholami/adahessian#citation)                                                                          |
| AdamD                 | *Improved bias-correction in Adam*                                                                             |                                                                                                                | <https://arxiv.org/abs/2110.10828>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2021arXiv211010828S/exportcitation)                                                        |
| AdamP                 | *Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights*                                 | [github](https://github.com/clovaai/AdamP)                                                                     | <https://arxiv.org/abs/2006.08217>                                                                                   | [cite](https://github.com/clovaai/AdamP#how-to-cite)                                                                                |
| diffGrad              | *An Optimization Method for Convolutional Neural Networks*                                                     | [github](https://github.com/shivram1987/diffGrad)                                                              | <https://arxiv.org/abs/1909.11015v3>                                                                                 | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv190911015D/exportcitation)                                                        |
| MADGRAD               | *A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic*                                       | [github](https://github.com/facebookresearch/madgrad)                                                          | <https://arxiv.org/abs/2101.11075>                                                                                   | [cite](https://github.com/facebookresearch/madgrad#tech-report)                                                                     |
| RAdam                 | *On the Variance of the Adaptive Learning Rate and Beyond*                                                     | [github](https://github.com/LiyuanLucasLiu/RAdam)                                                              | <https://arxiv.org/abs/1908.03265>                                                                                   | [cite](https://github.com/LiyuanLucasLiu/RAdam#citation)                                                                            |
| Ranger                | *a synergistic optimizer combining RAdam and LookAhead, and now GC in one optimizer*                           | [github](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer)                                          | <https://bit.ly/3zyspC3>                                                                                             | [cite](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer#citing-this-work)                                                |
| Ranger21              | *a synergistic deep learning optimizer*                                                                        | [github](https://github.com/lessw2020/Ranger21)                                                                | <https://arxiv.org/abs/2106.13731>                                                                                   | [cite](https://github.com/lessw2020/Ranger21#referencing-this-work)                                                                 |
| Lamb                  | *Large Batch Optimization for Deep Learning*                                                                   | [github](https://github.com/cybertronai/pytorch-lamb)                                                          | <https://arxiv.org/abs/1904.00962>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv190400962Y/exportcitation)                                                        |
| Shampoo               | *Preconditioned Stochastic Tensor Optimization*                                                                | [github](https://github.com/moskomule/shampoo.pytorch)                                                         | <https://arxiv.org/abs/1802.09568>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2018arXiv180209568G/exportcitation)                                                        |
| Nero                  | *Learning by Turning: Neural Architecture Aware Optimisation*                                                  | [github](https://github.com/jxbz/nero)                                                                         | <https://arxiv.org/abs/2102.07227>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2021arXiv210207227L/exportcitation)                                                        |
| Adan                  | *Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models*                                       | [github](https://github.com/sail-sg/Adan)                                                                      | <https://arxiv.org/abs/2208.06677>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2022arXiv220806677X/exportcitation)                                                        |
| Adai                  | *Disentangling the Effects of Adaptive Learning Rate and Momentum*                                             | [github](https://github.com/zeke-xie/adaptive-inertia-adai)                                                    | <https://arxiv.org/abs/2006.15815>                                                                                   | [cite](https://github.com/zeke-xie/adaptive-inertia-adai#citing)                                                                    |
| SAM                   | *Sharpness-Aware Minimization*                                                                                 | [github](https://github.com/davda54/sam)                                                                       | <https://arxiv.org/abs/2010.01412>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2020arXiv201001412F/exportcitation)                                                        |
| ASAM                  | *Adaptive Sharpness-Aware Minimization*                                                                        | [github](https://github.com/davda54/sam)                                                                       | <https://arxiv.org/abs/2102.11600>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2021arXiv210211600K/exportcitation)                                                        |
| GSAM                  | *Surrogate Gap Guided Sharpness-Aware Minimization*                                                            | [github](https://github.com/juntang-zhuang/GSAM)                                                               | <https://openreview.net/pdf?id=edONMAnhLu->                                                                          | [cite](https://github.com/juntang-zhuang/GSAM#citation)                                                                             |
| D-Adaptation          | *Learning-Rate-Free Learning by D-Adaptation*                                                                  | [github](https://github.com/facebookresearch/dadaptation)                                                      | <https://arxiv.org/abs/2301.07733>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2023arXiv230107733D/exportcitation)                                                        |
| AdaFactor             | *Adaptive Learning Rates with Sublinear Memory Cost*                                                           | [github](https://github.com/DeadAt0m/adafactor-pytorch)                                                        | <https://arxiv.org/abs/1804.04235>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2018arXiv180404235S/exportcitation)                                                        |
| Apollo                | *An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization*                | [github](https://github.com/XuezheMax/apollo)                                                                  | <https://arxiv.org/abs/2009.13586>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2020arXiv200913586M/exportcitation)                                                        |
| NovoGrad              | *Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks*                   | [github](https://github.com/lonePatient/NovoGrad-pytorch)                                                      | <https://arxiv.org/abs/1905.11286>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv190511286G/exportcitation)                                                        |
| Lion                  | *Symbolic Discovery of Optimization Algorithms*                                                                | [github](https://github.com/google/automl/tree/master/lion)                                                    | <https://arxiv.org/abs/2302.06675>                                                                                   | [cite](https://github.com/google/automl/tree/master/lion#citation)                                                                  |
| Ali-G                 | *Adaptive Learning Rates for Interpolation with Gradients*                                                     | [github](https://github.com/oval-group/ali-g)                                                                  | <https://arxiv.org/abs/1906.05661>                                                                                   | [cite](https://github.com/oval-group/ali-g#adaptive-learning-rates-for-interpolation-with-gradients)                                |
| SM3                   | *Memory-Efficient Adaptive Optimization*                                                                       | [github](https://github.com/google-research/google-research/tree/master/sm3)                                   | <https://arxiv.org/abs/1901.11150>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv190111150A/exportcitation)                                                        |
| AdaNorm               | *Adaptive Gradient Norm Correction based Optimizer for CNNs*                                                   | [github](https://github.com/shivram1987/AdaNorm)                                                               | <https://arxiv.org/abs/2210.06364>                                                                                   | [cite](https://github.com/shivram1987/AdaNorm/tree/main#citation)                                                                   |
| RotoGrad              | *Gradient Homogenization in Multitask Learning*                                                                | [github](https://github.com/adrianjav/rotograd)                                                                | <https://openreview.net/pdf?id=T8wHz4rnuGL>                                                                          | [cite](https://github.com/adrianjav/rotograd#citing)                                                                                |
| A2Grad                | *Optimal Adaptive and Accelerated Stochastic Gradient Descent*                                                 | [github](https://github.com/severilov/A2Grad_optimizer)                                                        | <https://arxiv.org/abs/1810.00553>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2018arXiv181000553D/exportcitation)                                                        |
| AccSGD                | *Accelerating Stochastic Gradient Descent For Least Squares Regression*                                        | [github](https://github.com/rahulkidambi/AccSGD)                                                               | <https://arxiv.org/abs/1704.08227>                                                                                   | [cite](https://github.com/rahulkidambi/AccSGD#citation)                                                                             |
| SGDW                  | *Decoupled Weight Decay Regularization*                                                                        | [github](https://github.com/loshchil/AdamW-and-SGDW)                                                           | <https://arxiv.org/abs/1711.05101>                                                                                   | [cite](https://github.com/loshchil/AdamW-and-SGDW#contact)                                                                          |
| ASGD                  | *Adaptive Gradient Descent without Descent*                                                                    | [github](https://github.com/ymalitsky/adaptive_GD)                                                             | <https://arxiv.org/abs/1910.09529>                                                                                   | [cite](https://github.com/ymalitsky/adaptive_GD#reference)                                                                          |
| Yogi                  | *Adaptive Methods for Nonconvex Optimization*                                                                  |                                                                                                                | [NIPS 2018](https://papers.nips.cc/paper/8186-adaptive-methods-for-nonconvex-optimization)                           | [cite](https://proceedings.neurips.cc/paper_files/paper/2018/hash/90365351ccc7437a1309dc64e4db32a3-Abstract.html)                   |
| SWATS                 | *Improving Generalization Performance by Switching from Adam to SGD*                                           |                                                                                                                | <https://arxiv.org/abs/1712.07628>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2017arXiv171207628S/exportcitation)                                                        |
| Fromage               | *On the distance between two neural networks and the stability of learning*                                    | [github](https://github.com/jxbz/fromage)                                                                      | <https://arxiv.org/abs/2002.03432>                                                                                   | [cite](https://github.com/jxbz/fromage#citation)                                                                                    |
| MSVAG                 | *Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients*                                    | [github](https://github.com/lballes/msvag)                                                                     | <https://arxiv.org/abs/1705.07774>                                                                                   | [cite](https://github.com/lballes/msvag#citation)                                                                                   |
| AdaMod                | *An Adaptive and Momental Bound Method for Stochastic Learning*                                                | [github](https://github.com/lancopku/AdaMod)                                                                   | <https://arxiv.org/abs/1910.12249>                                                                                   | [cite](https://github.com/lancopku/AdaMod#citation)                                                                                 |
| AggMo                 | *Aggregated Momentum: Stability Through Passive Damping*                                                       | [github](https://github.com/AtheMathmo/AggMo)                                                                  | <https://arxiv.org/abs/1804.00325>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2018arXiv180400325L/exportcitation)                                                        |
| QHAdam                | *Quasi-hyperbolic momentum and Adam for deep learning*                                                         | [github](https://github.com/facebookresearch/qhoptim)                                                          | <https://arxiv.org/abs/1810.06801>                                                                                   | [cite](https://github.com/facebookresearch/qhoptim#reference)                                                                       |
| PID                   | *A PID Controller Approach for Stochastic Optimization of Deep Networks*                                       | [github](https://github.com/tensorboy/PIDOptimizer)                                                            | [CVPR 18](http://www4.comp.polyu.edu.hk/~cslzhang/paper/CVPR18_PID.pdf)                                              | [cite](https://github.com/tensorboy/PIDOptimizer#citation)                                                                          |
| Gravity               | *a Kinematic Approach on Optimization in Deep Learning*                                                        | [github](https://github.com/dariush-bahrami/gravity.optimizer)                                                 | <https://arxiv.org/abs/2101.09192>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2021arXiv210109192B/exportcitation)                                                        |
| AdaSmooth             | *An Adaptive Learning Rate Method based on Effective Ratio*                                                    |                                                                                                                | <https://arxiv.org/abs/2204.00825v1>                                                                                 | [cite](https://ui.adsabs.harvard.edu/abs/2022arXiv220400825L/exportcitation)                                                        |
| SRMM                  | *Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates*              | [github](https://github.com/HanbaekLyu/SRMM)                                                                   | <https://arxiv.org/abs/2201.01652>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2022arXiv220101652L/exportcitation)                                                        |
| AvaGrad               | *Domain-independent Dominance of Adaptive Methods*                                                             | [github](https://github.com/lolemacs/avagrad)                                                                  | <https://arxiv.org/abs/1912.01823>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv191201823S/exportcitation)                                                        |
| PCGrad                | *Gradient Surgery for Multi-Task Learning*                                                                     | [github](https://github.com/tianheyu927/PCGrad)                                                                | <https://arxiv.org/abs/2001.06782>                                                                                   | [cite](https://github.com/tianheyu927/PCGrad#reference)                                                                             |
| AMSGrad               | *On the Convergence of Adam and Beyond*                                                                        |                                                                                                                | <https://openreview.net/pdf?id=ryQu7f-RZ>                                                                            | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv190409237R/exportcitation)                                                        |
| Lookahead             | *k steps forward, 1 step back*                                                                                 | [github](https://github.com/pytorch/examples/tree/main/imagenet)                                               | <https://arxiv.org/abs/1907.08610>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv190708610Z/exportcitation)                                                        |
| PNM                   | *Manipulating Stochastic Gradient Noise to Improve Generalization*                                             | [github](https://github.com/zeke-xie/Positive-Negative-Momentum)                                               | <https://arxiv.org/abs/2103.17182>                                                                                   | [cite](https://github.com/zeke-xie/Positive-Negative-Momentum#citing)                                                               |
| GC                    | *Gradient Centralization*                                                                                      | [github](https://github.com/Yonghongwei/Gradient-Centralization)                                               | <https://arxiv.org/abs/2004.01461>                                                                                   | [cite](https://github.com/Yonghongwei/Gradient-Centralization#citation)                                                             |
| AGC                   | *Adaptive Gradient Clipping*                                                                                   | [github](https://github.com/deepmind/deepmind-research/tree/master/nfnets)                                     | <https://arxiv.org/abs/2102.06171>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2021arXiv210206171B/exportcitation)                                                        |
| Stable WD             | *Understanding and Scheduling Weight Decay*                                                                    | [github](https://github.com/zeke-xie/stable-weight-decay-regularization)                                       | <https://arxiv.org/abs/2011.11152>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2020arXiv201111152X/exportcitation)                                                        |
| Softplus T            | *Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM*                                        |                                                                                                                | <https://arxiv.org/abs/1908.00700>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv190800700T/exportcitation)                                                        |
| Un-tuned w/u          | *On the adequacy of untuned warmup for adaptive optimization*                                                  |                                                                                                                | <https://arxiv.org/abs/1910.04209>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2019arXiv191004209M/exportcitation)                                                        |
| Norm Loss             | *An efficient yet effective regularization method for deep neural networks*                                    |                                                                                                                | <https://arxiv.org/abs/2103.06583>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2021arXiv210306583G/exportcitation)                                                        |
| AdaShift              | *Decorrelation and Convergence of Adaptive Learning Rate Methods*                                              | [github](https://github.com/MichaelKonobeev/adashift)                                                          | <https://arxiv.org/abs/1810.00143v4>                                                                                 | [cite](https://ui.adsabs.harvard.edu/abs/2018arXiv181000143Z/exportcitation)                                                        |
| AdaDelta              | *An Adaptive Learning Rate Method*                                                                             |                                                                                                                | <https://arxiv.org/abs/1212.5701v1>                                                                                  | [cite](https://ui.adsabs.harvard.edu/abs/2012arXiv1212.5701Z/exportcitation)                                                        |
| Amos                  | *An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale*                              | [github](https://github.com/google-research/jestimator)                                                        | <https://arxiv.org/abs/2210.11693>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2022arXiv221011693T/exportcitation)                                                        |
| SignSGD               | *Compressed Optimisation for Non-Convex Problems*                                                              | [github](https://github.com/jxbz/signSGD)                                                                      | <https://arxiv.org/abs/1802.04434>                                                                                   | [cite](https://ui.adsabs.harvard.edu/abs/2018arXiv180204434B/exportcitation)                                                        |
| Sophia                | *A Scalable Stochastic Second-order Optimizer for Language Model Pre-training*                                 | [github](https://github.com/Liuhong99/Sophia)                                                                  | <https://arxiv.org/abs/2305.14342>                                                                                   | [cite](https://github.com/Liuhong99/Sophia)                                                                                         |
| Prodigy               | *An Expeditiously Adaptive Parameter-Free Learner*                                                             | [github](https://github.com/konstmish/prodigy)                                                                 | <https://arxiv.org/abs/2306.06101>                                                                                   | [cite](https://github.com/konstmish/prodigy#how-to-cite)                                                                            |
| PAdam                 | *Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks*                 | [github](https://github.com/uclaml/Padam)                                                                      | <https://arxiv.org/abs/1806.06763>                                                                                   | [cite](https://github.com/uclaml/Padam#citation)                                                                                    |
| LOMO                  | *Full Parameter Fine-tuning for Large Language Models with Limited Resources*                                  | [github](https://github.com/OpenLMLab/LOMO)                                                                    | <https://arxiv.org/abs/2306.09782>                                                                                   | [cite](https://github.com/OpenLMLab/LOMO#citation)                                                                                  |
| AdaLOMO               | *Low-memory Optimization with Adaptive Learning Rate*                                                          | [github](https://github.com/OpenLMLab/LOMO)                                                                    | <https://arxiv.org/abs/2310.10195>                                                                                   | [cite](https://github.com/OpenLMLab/LOMO#citation)                                                                                  |
| Tiger                 | *A Tight-fisted Optimizer, an optimizer that is extremely budget-conscious*                                    | [github](https://github.com/bojone/tiger)                                                                      |                                                                                                                      | [cite](