Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '90c406a3a198b8f45682a9979b4c091ec5dc647e'
Root cause: Configuration Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 90c406a3a198b8f45682a9979b4c091ec5dc647e

Affected PyTorch commit: 90c406a3a198b8f45682a9979b4c091ec5dc647e

Affected Tests:


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 0.027871192374732345
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: 0.03819466635468416
- timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 0.02795512262673583
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 0.022725802678097453
- timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 0.029646509992224828
- timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 0.022995856448283628
- timm_vision_transformer_large, Adam, cuda, (pt2) default: 0.029697702698675648
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 0.05562696150445845
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.035275981664502375
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.030822903574776968
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03358603299905857
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.05519938173529226
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.0317666807178674
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.032115386275108904
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.08758739065766956
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.026906910752586555
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 0.03785212831765724
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.02630254274845356
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011696824499975063
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.012826792688429123
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.011354171596596845
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.03536363369785249
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.025794999103527515
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.0778292083220246
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09921788166199501
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07692374632460997
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.026457899613888003
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03681451115213955
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.027554474632779602
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.019014869552021
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.02860535171514909
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.018843417563898998
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.033988536471345775
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 248.61190934997285
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 122.75679718838849
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: 253.76483566568155
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 4.276255259018697
- compile_time, timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 437.82634686502087
- compile_time, timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 295.18518474700005
- compile_time, timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 404.26137325627496
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) default: 392.17158135430265
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 303.0597914183163
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 377.18724522334134
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) fused: 18.05178999902758
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) default: 388.2261883699878
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 311.7175408756593
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 388.4673964563408
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 7.775373226943581
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) default: 46.46972810465377
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) default: 468.2135736087027
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 299.83706035129336
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 420.0995416629594
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) default: 32.00695933236663
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 26.441780135714605
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 2.0807738492731005
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 1.9773468980177615
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.03772908499619613
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07180045096902177
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.06853794331739968
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.08359461401899657
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) default: 615.9181620766758
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 253.93369893626854
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 572.1461876780182
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 186.22239013935905
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 118.14182434700585
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 172.4768807446429
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) default: 343.80542263066553
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 4.679760386313603
- resnet18, Adadelta, cuda, (pt2) default: 0.0013849302604473357
- resnet18, Adadelta, cuda, (pt2) no_foreach: 0.002489492927396233
- resnet18, Adadelta, cuda, (pt2) foreach: 0.001268741461415952
- resnet18, Adagrad, cuda, (pt2) default: 0.0013517210326107168
- resnet18, Adagrad, cuda, (pt2) no_foreach: 0.0025864938566556727
- resnet18, Adagrad, cuda, (pt2) foreach: 0.0013445981257851393
- resnet18, Adam, cuda, (pt2) default: 0.0014627396484265906
- resnet18, Adam, cuda, (pt2) no_foreach: 0.004349287285509965
- resnet18, Adam, cuda, (pt2) foreach: 0.0014900801936164499
- resnet18, Adam, cuda, (pt2) fused: 0.001074303221393297
- resnet18, AdamW, cuda, (pt2) default: 0.0014612448421293311
- resnet18, AdamW, cuda, (pt2) no_foreach: 0.004170043064126124
- resnet18, AdamW, cuda, (pt2) foreach: 0.0015194982772685983
- resnet18, AdamW, cuda, (pt2) fused: 0.0011054586788937862
- resnet18, Adamax, cuda, (pt2) default: 0.006047654147122931
- resnet18, Adamax, cuda, (pt2) no_foreach: 0.006701210897881537
- resnet18, Adamax, cuda, (pt2) foreach: 0.006141477996469334
- resnet18, ASGD, cuda, (pt2) default: 0.0016146909967515498
- resnet18, ASGD, cuda, (pt2) no_foreach: 0.004047301058890298
- resnet18, ASGD, cuda, (pt2) foreach: 0.0016286973013330585
- resnet18, SGD, cuda, (pt2) default: 0.00048248945968225597
- resnet18, SGD, cuda, (pt2) no_foreach: 0.001318960929654024
- resnet18, SGD, cuda, (pt2) foreach: 0.00047294179103656475
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.00042315869295271114
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9: 0.000386777302948758
- resnet18, RAdam, cuda, (pt2) default: 0.0015353749349014834
- resnet18, RAdam, cuda, (pt2) no_foreach: 0.008656031467641394
- resnet18, RAdam, cuda, (pt2) foreach: 0.0014482632049475797
- resnet18, Rprop, cuda, (pt2) default: 0.001042742687483648
- resnet18, Rprop, cuda, (pt2) no_foreach: 0.0020954703377356054
- resnet18, Rprop, cuda, (pt2) foreach: 0.001050766473343469
- resnet18, RMSprop, cuda, (pt2) default: 0.0008793570938123913
- resnet18, RMSprop, cuda, (pt2) no_foreach: 0.002356695564573302
- resnet18, RMSprop, cuda, (pt2) foreach: 0.0008745840890434729
- resnet18, NAdam, cuda, (pt2) default: 0.001930738991807
- resnet18, NAdam, cuda, (pt2) no_foreach: 0.006308519630692899
- resnet18, NAdam, cuda, (pt2) foreach: 0.0019455657112276525
- compile_time, resnet18, Adadelta, cuda, (pt2) default: 16.22221521229949
- compile_time, resnet18, Adadelta, cuda, (pt2) no_foreach: 24.70006023166934
- compile_time, resnet18, Adadelta, cuda, (pt2) foreach: 0.6808852215957213
- compile_time, resnet18, Adagrad, cuda, (pt2) default: 26.780733531651396
- compile_time, resnet18, Adagrad, cuda, (pt2) no_foreach: 23.368895179320436
- compile_time, resnet18, Adagrad, cuda, (pt2) foreach: 15.34439043229213
- compile_time, resnet18, Adam, cuda, (pt2) default: 19.48542708263267
- compile_time, resnet18, Adam, cuda, (pt2) no_foreach: 34.60411929331409
- compile_time, resnet18, Adam, cuda, (pt2) foreach: 15.534126948002571
- compile_time, resnet18, Adam, cuda, (pt2) fused: 2.2364914576367787
- compile_time, resnet18, AdamW, cuda, (pt2) default: 18.232776801024254
- compile_time, resnet18, AdamW, cuda, (pt2) no_foreach: 30.461371169309132
- compile_time, resnet18, AdamW, cuda, (pt2) foreach: 16.21247648401186
- compile_time, resnet18, AdamW, cuda, (pt2) fused: 2.4945041363243945
- compile_time, resnet18, Adamax, cuda, (pt2) default: 1.7759393683130231
- compile_time, resnet18, Adamax, cuda, (pt2) no_foreach: 32.83035925106378
- compile_time, resnet18, Adamax, cuda, (pt2) foreach: 0.834159471327439
- compile_time, resnet18, ASGD, cuda, (pt2) default: 19.01353378663771
- compile_time, resnet18, ASGD, cuda, (pt2) no_foreach: 25.691120516043156
- compile_time, resnet18, ASGD, cuda, (pt2) foreach: 16.432597442006227
- compile_time, resnet18, SGD, cuda, (pt2) default: 0.9524604009736019
- compile_time, resnet18, SGD, cuda, (pt2) no_foreach: 6.186718019986681
- compile_time, resnet18, SGD, cuda, (pt2) foreach: 0.2700683106474268
- compile_time, resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.257638832690039
- compile_time, resnet18, SGD, cuda, (pt2) foreach, momentum=0.9: 0.001772049309996267
- compile_time, resnet18, RAdam, cuda, (pt2) default: 0.004371844329095135
- compile_time, resnet18, RAdam, cuda, (pt2) no_foreach: 0.003739607927855104
- compile_time, resnet18, RAdam, cuda, (pt2) foreach: 0.003995360360325624
- compile_time, resnet18, Rprop, cuda, (pt2) default: 21.635895813330233
- compile_time, resnet18, Rprop, cuda, (pt2) no_foreach: 22.964949152296565
- compile_time, resnet18, Rprop, cuda, (pt2) foreach: 21.02231108533063
- compile_time, resnet18, RMSprop, cuda, (pt2) default: 9.105106203292962
- compile_time, resnet18, RMSprop, cuda, (pt2) no_foreach: 16.23375906900037
- compile_time, resnet18, RMSprop, cuda, (pt2) foreach: 6.254216334025841
- compile_time, resnet18, NAdam, cuda, (pt2) default: 26.508149536675774
- compile_time, resnet18, NAdam, cuda, (pt2) no_foreach: 52.19905022199964
- compile_time, resnet18, NAdam, cuda, (pt2) foreach: 24.578717776030924


Tests that were newly added on affected commit:
- timm_vision_transformer_large, Adamax, cuda, default: 0.08263983998525266
- timm_vision_transformer_large, Adamax, cuda, maximize: 0.08745352268063773
- timm_vision_transformer_large, Adamax, cuda, no_foreach: 0.06963475689990446
- timm_vision_transformer_large, Adamax, cuda, differentiable: 0.07884616967445861
- timm_vision_transformer_large, Adamax, cuda, foreach: 0.08218685966373111


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6307904487

cc @janeyx99


this is expected. i removed all pt2 tests yesterday.