Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'b3874abf04e0579fdc0025195593d90b97178f0e'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8c684663042a61de20f163ab6a292cd43c67e294

Affected PyTorch commit: b3874abf04e0579fdc0025195593d90b97178f0e

Affected Tests:
- opacus_cifar10, RMSprop, cuda, differentiable: -33.04042%
- hf_T5_base, Adadelta, cuda, no_foreach: -31.77567%
- moco, RMSprop, cuda, differentiable: -33.15609%
- hf_T5_large, Rprop, cuda, no_foreach: -30.47008%
- phlippe_densenet, RMSprop, cuda, differentiable: -34.88892%
- resnet18, Adadelta, cuda, (pt2) default: +32.43076%
- resnet18, Adadelta, cuda, (pt2) foreach: +34.04744%
- mobilenet_v3_large, Adadelta, cuda, differentiable: -30.75869%
- mobilenet_v3_large, AdamW, cuda, no_foreach: -33.62357%
- mobilenet_v3_large, RMSprop, cuda, no_foreach: -35.38221%
- hf_Whisper, ASGD, cuda, foreach: -31.32677%
- timm_vision_transformer, Adadelta, cuda, (pt2) default: +43.13403%
- timm_vision_transformer, Adadelta, cuda, (pt2) foreach: +41.38473%
- hf_Reformer, AdamW, cuda, no_foreach: -30.64935%


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 0.025980619597248733
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: 0.03816349866489569
- timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 0.025505209458060563
- timm_vision_transformer_large, Adadelta, cuda, foreach: 0.08896098705008626
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 0.023394695638368528
- timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 0.030054887889751365
- timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 0.02354929715188013
- timm_vision_transformer_large, Adam, cuda, (pt2) default: 0.030716839712113142
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 0.05961259175091982
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.032608476267861466
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03212276832865817
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.057868547388352454
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.03198382956907153
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.06907040004928906
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.07519767759367824
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.02410224508494139
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.028891901884760176
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03146686464814203
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.02874406999243157
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.11178017850033939
- timm_vision_transformer_large, NAdam, cuda, (pt2) no_foreach: 0.06153443653602153


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6032886557

cc @janeyx99


Seems related to https://github.com/pytorch/benchmark/issues/1839

I will disable the ASGD overnight to prevent these signals