Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'f89c73be3a3e8274d025ac46a33a780853841c9e'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 0.0951602056932946
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: 0.09252047864720225
- timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 0.09491070068906993
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 0.053263344219885767
- timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 0.03849658001369486
- timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 0.053046090266434476
- timm_vision_transformer_large, Adam, cuda, (pt2) default: 0.07798325305338949
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 0.10975804599002004
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.07716963234512757
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.029113865008444657
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.08456354499018441
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.11291515408083797
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.08372209667383383
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.02972213114013097
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.0844246189808473
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.0719871489952008
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08412744395900518
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.06807168666273355
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 2.773358018952422
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.06788783671800047
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011088976281155882
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.012959409512404818
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.011047100170368427
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 2.3635660170111805
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 2.3436323270434514
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07616552334123601
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09571686709532515
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07585081364959478
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.12120055104605854
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.11923762504011393
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.11998195800697431
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.018473655175925655
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.032448556989298334
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.01892975279638036
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08747267401001106
- timm_vision_transformer_large, NAdam, cuda, (pt2) no_foreach: 0.08850490332891543
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.08704260030450921


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5816800777

cc @janeyx99


this is good news and to be expected!