Repository: pytorch
Issue: Integration of AdamCPR Optimizer into PyTorch
Labels: module: optimizer,triaged

### 🚀 The feature, motivation and pitch

# Proposal: Integration of AdamCPR Optimizer into PyTorch

**Authors:**
- @ZiadHelal

## **Summary**
We propose the integration of AdamCPR, a novel deep learning optimizer developed at the University of Freiburg, into PyTorch's core optimizer library. AdamCPR builds upon the widely adopted AdamW (also originating from our lab) and introduces Constrained Parameter Regularization (CPR) to improve optimization dynamics and generalization. CPR enforces adaptive and individualized regularization constraints across parameter matrices, requiring minimal hyperparameter tuning while outperforming AdamW on diverse tasks such as language modeling, image classification, and medical image segmentation.

For details, see our paper: [Improving Deep Learning Optimization through Constrained Parameter Regularization](https://arxiv.org/abs/2311.09058).

## **Motivation**
AdamCPR addresses key limitations of uniform regularization in traditional optimizers, offering:
- **Dynamic Regularization:** CPR adapts the penalty strength during training, eliminating the need for manual weight decay scheduling.
- **Improved Performance:** Demonstrated gains in multiple benchmarks, including CIFAR100 (+1.5% accuracy), ImageNet (+2-3% accuracy on DeiT models), and GPT-2 pretraining (33% reduced training time to achieve comparable perplexity).
- **Wide Applicability:** Suitable for diverse tasks, including fine-tuning large-scale models and training robust classifiers in noisy settings.

### Experimental Highlights
1. **GPT-2 Pretraining:** CPR achieved the same perplexity as AdamW with **33% fewer training steps**, translating into significant computational savings.
2. **Image Classification:** CPR outperformed AdamW in training ResNet18 on CIFAR100 with +1.5% accuracy and DeiT-Small on ImageNet with +2% top-1 accuracy.
3. **Medical Image Segmentation:** CPR improved Dice scores in tasks such as Brain Tumor Segmentation (+0.43%) and Multi-Atlas Labeling (+0.24%) compared to SGD with weight decay.

### Addressing Concerns

1. While AdamCPR is implemented in our [lab's GitHub repository](https://github.com/automl/CPR) and PyTorch encourages the exploration of optimizers in third-party libraries, we believe AdamCPR merits inclusion in the core library due to its foundational improvements, broad applicability, and lineage from AdamW, which is a widely used and trusted optimizer in PyTorch. Integrating AdamCPR would provide the community with a robust, efficient, and ready-to-use tool, fostering adoption and reducing the need for users to implement or maintain custom solutions.
2. CPR has been tested extensively across diverse domains, achieving consistent performance gains with minimal to zero hyperparameter tuning.
3. Our lab has pioneered impactful contributions like AdamW, and AdamCPR continues this trajectory, representing the cutting edge of optimizer research.

## **Proposed Implementation**
AdamCPR builds on PyTorch’s existing optimizer framework, ensuring compatibility and ease of integration:
1. **Single Tensor & Multi (foreach) Tensor Implementation :** This is already implemented in our repo.
4. **Fused Implementation (Planned):** Targeted for CUDA optimization, offering significant speedup in large-scale deployments.

## **Metrics**
- Performance improvement over AdamW on diverse tasks:
  - CIFAR100: +1.5% accuracy on ResNet18.
  - ImageNet: +2-3% accuracy on DeiT models.
  - GPT-2: 33% reduction in training budget for equivalent perplexity.
- Reduction in hyperparameter tuning effort (e.g., weight decay).
- Computational efficiency compared to baseline optimizers (runtime increase <6%).

## **Drawbacks**
1. **Runtime Overhead:** Minor increase in training time (~0.5–5% in most settings) due to additional computations for constraints.


We look forward to feedback from the PyTorch team, specifically Optimizers maintainers @janeyx99 @albanD @jbschlosser.

### Alternatives

_No response_

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar