Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'f89c73be3a3e8274d025ac46a33a780853841c9e'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected Tests:
- fastNLP_Bert, Adamax, cuda, maximize: +37.26383%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 0.095686426696678
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: 0.09252084496741493
- timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 0.09529085402997832
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 0.05411515780724585
- timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 0.03856756569196781
- timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 0.05337622918887064
- timm_vision_transformer_large, Adam, cuda, (pt2) default: 0.07844693562947214
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 0.10795844462700188
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.07812690796951453
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.029391022731683085
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.08539535392386217
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.1112025025067851
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.08424012432806194
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.029976316395082643
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.08461967739276588
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.07219307357445359
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08489398371117811
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.06970656539003055
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 2.814707956975326
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.06819242076016963
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011096650710035311
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.013031482070800848
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.01101787798841925
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 2.380784230073914
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 2.3920460438821465
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07632963499054313
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09557676471304148
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07647624867968261
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.12139839748851955
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.1199733285466209
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.12068748101592064
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.018544336738572878
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.03244561226373272
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.01898510390045968
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08772059134207666
- timm_vision_transformer_large, NAdam, cuda, (pt2) no_foreach: 0.08912980339179437
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.0886083620134741


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5813640590

cc @janeyx99


Came from a test run--closng