Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '2baa4c49288efeded2fad677b2f28570b0ce858b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 7b14f0ff8fabe2185e4ff9551e858e46c7784536

Affected PyTorch commit: 2baa4c49288efeded2fad677b2f28570b0ce858b

Affected Tests:
- resnet18, Adam, cuda, (pt2) no_foreach: +622765.57535%
- resnet18, Adam, cuda, (pt2) foreach: +140.14200%
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: -99.97948%
- timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: +733042.60363%
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: -28552.94408%
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) foreach: +1758.67868%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0869290226449569
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03175068939370768
- timm_vision_transformer_large, AdamW, cuda, default: 0.06932406810422738
- timm_vision_transformer_large, AdamW, cuda, amsgrad, maximize: 0.0872920857121547
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.05601157795172185
- timm_vision_transformer_large, AdamW, cuda, no_foreach: 0.07040293297419946
- timm_vision_transformer_large, AdamW, cuda, differentiable: 0.0879575782455504
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.03073425085416862
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.06969046344359715
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable: 0.08410646300762892
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.09322425117716193
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.030847575249416486
- timm_vision_transformer_large, AdamW, cuda, fused: 0.0290705854339259
- timm_vision_transformer_large, AdamW, cuda, fused, amsgrad, maximize: 0.03259885810049517
- timm_vision_transformer_large, AdamW, cuda, fused, capturable: 0.027284858794882894
- timm_vision_transformer_large, AdamW, cuda, fused, capturable, amsgrad: 0.03058700142428279
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.08413271031652887
- timm_vision_transformer_large, Adamax, cuda, default: 0.08233604347333312
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.07465639725948374
- timm_vision_transformer_large, Adamax, cuda, no_foreach: 0.07512599943826596
- timm_vision_transformer_large, Adamax, cuda, differentiable: 0.07935263402760029
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08465497987344861
- timm_vision_transformer_large, Adamax, cuda, foreach: 0.0820629281612734
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.02444471237767074
- timm_vision_transformer_large, ASGD, cuda, default: 0.06830813890943925
- timm_vision_transformer_large, ASGD, cuda, maximize: 0.07563973714907964
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 207.5385332792066
- timm_vision_transformer_large, ASGD, cuda, no_foreach: 0.08514666790142655
- timm_vision_transformer_large, ASGD, cuda, differentiable: 0.08403141579280297
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.07492035099615653
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.06629862589761615
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011391635803091858
- timm_vision_transformer_large, SGD, cuda, default: 0.00966262126651903
- timm_vision_transformer_large, SGD, cuda, maximize: 0.01632520130369812
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.012641768931644037
- timm_vision_transformer_large, SGD, cuda, no_foreach: 0.010729903914034368
- timm_vision_transformer_large, SGD, cuda, differentiable: 0.009738165435070792
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.012373015394105631
- timm_vision_transformer_large, SGD, cuda, foreach: 0.009628307710712155
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.03530352911911905
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9, nesterov: 0.035355928866192696
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.02572231963276863
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9: 0.02574166771955788
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07645971871291597
- timm_vision_transformer_large, RAdam, cuda, default: 0.07677281554788351
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09790640433008473
- timm_vision_transformer_large, RAdam, cuda, no_foreach: 0.09602568680420517
- timm_vision_transformer_large, RAdam, cuda, differentiable: 0.09651239118538797
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.0756945136624078
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.07592145275945465
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.026615014008712023
- timm_vision_transformer_large, Rprop, cuda, default: 0.11160928593017161
- timm_vision_transformer_large, Rprop, cuda, maximize: 0.12481727101840079
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03589176464204987
- timm_vision_transformer_large, Rprop, cuda, no_foreach: 0.11884426046162844
- timm_vision_transformer_large, Rprop, cuda, differentiable: 0.1260000306647271
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.026017103460617363
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.1119668809697032
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.018390231850472363
- timm_vision_transformer_large, RMSprop, cuda, default: 0.04178516492247582
- timm_vision_transformer_large, RMSprop, cuda, maximize: 0.04828381752595305
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.028041683486662805
- timm_vision_transformer_large, RMSprop, cuda, no_foreach: 0.04357688371092081
- timm_vision_transformer_large, RMSprop, cuda, differentiable: 0.04439799422398209
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.01831430590457537
- timm_vision_transformer_large, RMSprop, cuda, foreach: 0.04176201350055635
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08396156085655093
- timm_vision_transformer_large, NAdam, cuda, default: 0.07467362921064098
- timm_vision_transformer_large, NAdam, cuda, no_foreach: 0.07535927835851908
- timm_vision_transformer_large, NAdam, cuda, differentiable: 0.09274014225229621
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.0769132636487484
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.0742937622902294
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) default: 393.5387691630361
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 304.5033280878949
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 377.6236660216625
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 7.621729400940239
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) default: 44.93352451687679
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 314.7724319839229
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 11.282057445806762
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) default: 435.5381307664017
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 29.847454712726176
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 434.99561914103106
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) default: 17.29115696903318
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 25.538984360949446
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 2.0613928974295654
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 1.9299136108408372
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.04032344277948141
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.08235049278785786
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.15067638084292412
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.12210170443480214
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) default: 596.7998580016816
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 250.26361396598318
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 550.9581254231744
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 170.4870657340313
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 136.1424836854761
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 155.98032941063866
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) default: 96.2991318871888
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 20.374289763780933


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6180774078

cc @janeyx99


Closing, nothing new reported. Will be implicitly fixed by https://github.com/pytorch/benchmark/pull/1931 and https://github.com/pytorch/benchmark/pull/1932