Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'd8fd7f96b9537ce5a88601e7cf410b790fdd12c1'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 00ae5fa31130b0c97b541c18a773d35405d46bb2

Affected PyTorch commit: d8fd7f96b9537ce5a88601e7cf410b790fdd12c1

Affected Tests:
- resnet18, NAdam, cuda, (pt2) no_foreach: -99.98541%
- resnet18, NAdam, cuda, (pt2) foreach: -59.12438%
- compile_time, resnet18, Adamax, cuda, (pt2) foreach: +235.47222%
- compile_time, resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +724.74784%
- compile_time, resnet18, NAdam, cuda, (pt2) no_foreach: +1168.06480%
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: -99.97854%
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: -99.97696%
- timm_vision_transformer_large, Adam, cuda, differentiable: -32.10192%
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: -8482.87051%
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: +8299.61413%


Tests that were no longer run on affected commit:
- nanogpt_generate, Adadelta, cuda, default: 0.010745404940098523
- nanogpt_generate, Adadelta, cuda, maximize: 0.011615067860111595
- nanogpt_generate, Adadelta, cuda, no_foreach: 0.013780321180820464
- nanogpt_generate, Adadelta, cuda, differentiable: 0.015408747317269444
- nanogpt_generate, Adadelta, cuda, foreach: 0.010799648053944112
- nanogpt_generate, Adagrad, cuda, default: 0.005111926309764385
- nanogpt_generate, Adagrad, cuda, maximize: 0.005925975842401386
- nanogpt_generate, Adagrad, cuda, no_foreach: 0.006921281879767776
- nanogpt_generate, Adagrad, cuda, differentiable: 0.007732009813189507
- nanogpt_generate, Adagrad, cuda, foreach: 0.005107993483543396
- nanogpt_generate, Adam, cuda, default: 0.007125466288998723
- nanogpt_generate, Adam, cuda, amsgrad, maximize: 0.009183243413766225
- nanogpt_generate, Adam, cuda, no_foreach: 0.012139546172693371
- nanogpt_generate, Adam, cuda, differentiable: 0.025665631238371133
- nanogpt_generate, Adam, cuda, foreach: 0.007236206596717238
- nanogpt_generate, Adam, cuda, foreach, maximize, capturable: 0.009957487850139538
- nanogpt_generate, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.01108565120957792
- nanogpt_generate, Adam, cuda, fused: 0.0033424507454037668
- nanogpt_generate, Adam, cuda, fused, amsgrad, maximize: 0.003916975492611528
- nanogpt_generate, Adam, cuda, fused, capturable: 0.0033447014447301627
- nanogpt_generate, Adam, cuda, fused, capturable, amsgrad: 0.003920432534068823
- nanogpt_generate, AdamW, cuda, default: 0.007953106136992573
- nanogpt_generate, AdamW, cuda, amsgrad, maximize: 0.009975294241060813
- nanogpt_generate, AdamW, cuda, no_foreach: 0.013420871878042816
- nanogpt_generate, AdamW, cuda, differentiable: 0.02501159906387329
- nanogpt_generate, AdamW, cuda, foreach: 0.007941020121797919
- nanogpt_generate, AdamW, cuda, foreach, maximize, capturable: 0.010740908700972796
- nanogpt_generate, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.011879066564142705
- nanogpt_generate, AdamW, cuda, fused: 0.00344310418702662
- nanogpt_generate, AdamW, cuda, fused, amsgrad, maximize: 0.0039763770066201685
- nanogpt_generate, AdamW, cuda, fused, capturable: 0.003445221558213234
- nanogpt_generate, AdamW, cuda, fused, capturable, amsgrad: 0.003982354374602437
- nanogpt_generate, Adamax, cuda, default: 0.012325893016532063
- nanogpt_generate, Adamax, cuda, maximize: 0.01199896549805999
- nanogpt_generate, Adamax, cuda, no_foreach: 0.016217599716037513
- nanogpt_generate, Adamax, cuda, differentiable: 0.01911850082688034
- nanogpt_generate, Adamax, cuda, foreach: 0.01195826749317348
- nanogpt_generate, ASGD, cuda, default: 0.013114939164370298
- nanogpt_generate, ASGD, cuda, maximize: 0.014196955086663365
- nanogpt_generate, ASGD, cuda, no_foreach: 0.02040277039632201
- nanogpt_generate, ASGD, cuda, differentiable: 0.020736534520983695
- nanogpt_generate, ASGD, cuda, foreach: 0.013152044406160713
- nanogpt_generate, SGD, cuda, default: 0.0011700871167704464
- nanogpt_generate, SGD, cuda, maximize: 0.001967772925272584
- nanogpt_generate, SGD, cuda, no_foreach: 0.0015968857612460852
- nanogpt_generate, SGD, cuda, differentiable: 0.001170610273256898
- nanogpt_generate, SGD, cuda, foreach: 0.00116692038718611
- nanogpt_generate, SGD, cuda, foreach, momentum=0.9, nesterov: 0.004322548927739263
- nanogpt_generate, SGD, cuda, foreach, momentum=0.9: 0.0031431357748806475
- nanogpt_generate, RAdam, cuda, default: 0.008908860323329767
- nanogpt_generate, RAdam, cuda, no_foreach: 0.01997820199467242
- nanogpt_generate, RAdam, cuda, differentiable: 0.020438154879957437
- nanogpt_generate, RAdam, cuda, foreach: 0.008895253886779149
- nanogpt_generate, Rprop, cuda, default: 0.017443695152178404
- nanogpt_generate, Rprop, cuda, maximize: 0.016377014853060243
- nanogpt_generate, Rprop, cuda, no_foreach: 0.025826830044388772
- nanogpt_generate, Rprop, cuda, differentiable: 0.02775728441774845
- nanogpt_generate, Rprop, cuda, foreach: 0.015553068881854416
- nanogpt_generate, RMSprop, cuda, default: 0.005125271528959275
- nanogpt_generate, RMSprop, cuda, maximize: 0.005907063726335764
- nanogpt_generate, RMSprop, cuda, no_foreach: 0.006649174951016903
- nanogpt_generate, RMSprop, cuda, differentiable: 0.007095048418268562
- nanogpt_generate, RMSprop, cuda, foreach: 0.0051189082860946655
- nanogpt_generate, NAdam, cuda, default: 0.00879433611407876
- nanogpt_generate, NAdam, cuda, no_foreach: 0.014423584938049315
- nanogpt_generate, NAdam, cuda, differentiable: 0.0269351770170033
- nanogpt_generate, NAdam, cuda, foreach: 0.008766268100589515


Tests that were newly added on affected commit:
- nanogpt, Adadelta, cuda, default: 0.01077794820012059
- nanogpt, Adadelta, cuda, maximize: 0.011593460350195529
- nanogpt, Adadelta, cuda, no_foreach: 0.01281544884950563
- nanogpt, Adadelta, cuda, differentiable: 0.015104905199768837
- nanogpt, Adadelta, cuda, foreach: 0.010781067949938005
- nanogpt, Adagrad, cuda, default: 0.005107058740031789
- nanogpt, Adagrad, cuda, maximize: 0.005906512040019152
- nanogpt, Adagrad, cuda, no_foreach: 0.006838373930004309
- nanogpt, Adagrad, cuda, differentiable: 0.007374173499993048
- nanogpt, Adagrad, cuda, foreach: 0.005108291069991538
- nanogpt, Adam, cuda, default: 0.00711702606997278
- nanogpt, Adam, cuda, amsgrad, maximize: 0.009144164466609558
- nanogpt, Adam, cuda, no_foreach: 0.01193982264958322
- nanogpt, Adam, cuda, differentiable: 0.02334842709969962
- nanogpt, Adam, cuda, foreach: 0.0071180013299454005
- nanogpt, Adam, cuda, foreach, maximize, capturable: 0.009880794366714933
- nanogpt, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.011030649400345283
- nanogpt, Adam, cuda, fused: 0.0033407328999601306
- nanogpt, Adam, cuda, fused, amsgrad, maximize: 0.003915998540032888
- nanogpt, Adam, cuda, fused, capturable: 0.0033441742799914208
- nanogpt, Adam, cuda, fused, capturable, amsgrad: 0.003919026570001733
- nanogpt, AdamW, cuda, default: 0.008072878933065415
- nanogpt, AdamW, cuda, amsgrad, maximize: 0.009952447366716417
- nanogpt, AdamW, cuda, no_foreach: 0.013095879899628926
- nanogpt, AdamW, cuda, differentiable: 0.024233282399654853
- nanogpt, AdamW, cuda, foreach: 0.007943105660015135
- nanogpt, AdamW, cuda, foreach, maximize, capturable: 0.01070222489979642
- nanogpt, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.011829398649933865
- nanogpt, AdamW, cuda, fused: 0.003442324489951716
- nanogpt, AdamW, cuda, fused, amsgrad, maximize: 0.003975794049983961
- nanogpt, AdamW, cuda, fused, capturable: 0.0034422899699711705
- nanogpt, AdamW, cuda, fused, capturable, amsgrad: 0.003977089350009919
- nanogpt, Adamax, cuda, default: 0.011492194599850337
- nanogpt, Adamax, cuda, maximize: 0.011918183750094614
- nanogpt, Adamax, cuda, no_foreach: 0.015193840999927488
- nanogpt, Adamax, cuda, differentiable: 0.018316630399931456
- nanogpt, Adamax, cuda, foreach: 0.011385077700106196
- nanogpt, ASGD, cuda, default: 0.0132773001503665
- nanogpt, ASGD, cuda, maximize: 0.013912990700191585
- nanogpt, ASGD, cuda, no_foreach: 0.021151272600400263
- nanogpt, ASGD, cuda, differentiable: 0.02072043139996822
- nanogpt, ASGD, cuda, foreach: 0.013266287199803627
- nanogpt, SGD, cuda, default: 0.001171429280002485
- nanogpt, SGD, cuda, maximize: 0.0019746228250005515
- nanogpt, SGD, cuda, no_foreach: 0.0015605083650007145
- nanogpt, SGD, cuda, differentiable: 0.001175119215004088
- nanogpt, SGD, cuda, foreach: 0.0011747530950015062
- nanogpt, SGD, cuda, foreach, momentum=0.9, nesterov: 0.004306317239970667
- nanogpt, SGD, cuda, foreach, momentum=0.9: 0.0031353792200388853
- nanogpt, RAdam, cuda, default: 0.00890788260003319
- nanogpt, RAdam, cuda, no_foreach: 0.019330000549962277
- nanogpt, RAdam, cuda, differentiable: 0.020311697599390756
- nanogpt, RAdam, cuda, foreach: 0.00889724096681069
- nanogpt, Rprop, cuda, default: 0.01595672610019392
- nanogpt, Rprop, cuda, maximize: 0.01605077159983921
- nanogpt, Rprop, cuda, no_foreach: 0.025794410399976188
- nanogpt, Rprop, cuda, differentiable: 0.027507797999714968
- nanogpt, Rprop, cuda, foreach: 0.01581537969977944
- nanogpt, RMSprop, cuda, default: 0.005116785739955958
- nanogpt, RMSprop, cuda, maximize: 0.005910304979988723
- nanogpt, RMSprop, cuda, no_foreach: 0.006448513159994036
- nanogpt, RMSprop, cuda, differentiable: 0.006859910810017027
- nanogpt, RMSprop, cuda, foreach: 0.0051152215999900365
- nanogpt, NAdam, cuda, default: 0.008774318766518263
- nanogpt, NAdam, cuda, no_foreach: 0.013960380349817569
- nanogpt, NAdam, cuda, differentiable: 0.0248437374997593
- nanogpt, NAdam, cuda, foreach: 0.00875810103364832
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.03037142414021738
- timm_vision_transformer_large, Adam, cuda, foreach: 0.058088700599910226
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable: 0.07666310199905031
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.08652852899831487
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.03062119942894372
- timm_vision_transformer_large, Adam, cuda, fused: 0.026526771799399284
- timm_vision_transformer_large, Adam, cuda, fused, amsgrad, maximize: 0.03229225800064991
- timm_vision_transformer_large, Adam, cuda, fused, capturable: 0.026538445899495854
- timm_vision_transformer_large, Adam, cuda, fused, capturable, amsgrad: 0.030205007600306998
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.029500227427758676
- timm_vision_transformer_large, AdamW, cuda, default: 0.0689285859989468
- timm_vision_transformer_large, AdamW, cuda, amsgrad, maximize: 0.08397529933426995
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.053669593999075005
- timm_vision_transformer_large, AdamW, cuda, no_foreach: 0.06996076299886529
- timm_vision_transformer_large, AdamW, cuda, differentiable: 0.08377987533215976
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.0303844564290817
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.06924339166774492
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable: 0.08340722800009341
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.09323500766186044
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.03059844814249248
- timm_vision_transformer_large, AdamW, cuda, fused: 0.02893573542678496
- timm_vision_transformer_large, AdamW, cuda, fused, amsgrad, maximize: 0.03247972599949987
- timm_vision_transformer_large, AdamW, cuda, fused, capturable: 0.027305130899912912
- timm_vision_transformer_large, AdamW, cuda, fused, capturable, amsgrad: 0.030621268500544828
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.08400659800099675
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.07364798500202596
- timm_vision_transformer_large, Adamax, cuda, no_foreach: 0.07103520232825151
- timm_vision_transformer_large, Adamax, cuda, differentiable: 0.07847083699986494
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08491807133395923
- timm_vision_transformer_large, Adamax, cuda, foreach: 0.08215971399719517
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.02444924266617616
- timm_vision_transformer_large, ASGD, cuda, default: 0.06773832633189159
- timm_vision_transformer_large, ASGD, cuda, maximize: 0.0745235999978225
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 0.03519561483335565
- timm_vision_transformer_large, ASGD, cuda, no_foreach: 0.08242445033586894
- timm_vision_transformer_large, ASGD, cuda, differentiable: 0.08252020900302644
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.02459440477751842
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.06724088933454671
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011434022445529182
- timm_vision_transformer_large, SGD, cuda, default: 0.009653083099692594
- timm_vision_transformer_large, SGD, cuda, maximize: 0.016248807500232942
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.012393954882367431
- timm_vision_transformer_large, SGD, cuda, no_foreach: 0.010676628899454955
- timm_vision_transformer_large, SGD, cuda, differentiable: 0.00965293089999856
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.011200564722659893
- timm_vision_transformer_large, SGD, cuda, foreach: 0.009603387533328109
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.035258158299984646
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9, nesterov: 0.035269423000136156
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.025734332499996525
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9: 0.025703920300293247
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07136290790003841
- timm_vision_transformer_large, RAdam, cuda, default: 0.07624273700154542
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.0973075889996835
- timm_vision_transformer_large, RAdam, cuda, no_foreach: 0.09558186620051856
- timm_vision_transformer_large, RAdam, cuda, differentiable: 0.09599556429966469
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07126068450015736
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.07573827500164043
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.02582953387445741
- timm_vision_transformer_large, Rprop, cuda, default: 0.11194099700151128
- timm_vision_transformer_large, Rprop, cuda, maximize: 0.12944606100063538
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03547571416856954
- timm_vision_transformer_large, Rprop, cuda, no_foreach: 0.11842697149768355
- timm_vision_transformer_large, Rprop, cuda, differentiable: 0.12577304750084295
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.026355014877481153
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.11062629850130179
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.018432794273063668
- timm_vision_transformer_large, RMSprop, cuda, default: 0.04178965289975167
- timm_vision_transformer_large, RMSprop, cuda, maximize: 0.04829796100020758
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.0275902448756824
- timm_vision_transformer_large, RMSprop, cuda, no_foreach: 0.04333710400023847
- timm_vision_transformer_large, RMSprop, cuda, differentiable: 0.044118003599578515
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.01802456416711114
- timm_vision_transformer_large, RMSprop, cuda, foreach: 0.041723809899849584
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.03183615500061673
- timm_vision_transformer_large, NAdam, cuda, default: 0.07522356666837975
- timm_vision_transformer_large, NAdam, cuda, no_foreach: 0.07842755000213704
- timm_vision_transformer_large, NAdam, cuda, differentiable: 0.0937397646678922
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.03193559699866455
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.07403553966772354
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 26.415420099663606
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) fused: 12.8508649546687
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) default: 378.33945136932743
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 309.5370932336712
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 369.874446133336
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 6.658124051003445
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) default: 43.21063250133253
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 305.23439153633564
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 10.8444436226661
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) default: 423.4224982899905
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 286.77042015566985
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 398.97083522732765
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) default: 17.617356398329623
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 38.96718988233139
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 1.99857355432808
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 1.9314858893339988
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.03672711400092036
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.06709784766765856
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.06492891566692074
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.0666123206683551
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) default: 572.2060853773315
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 232.67014034766666
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 558.2168006196686
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 180.75296617433924
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 111.35861489133094
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 166.967233489333
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) default: 328.95508564933453
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 297.19574147433497


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6269943849

cc @janeyx99


Closing, nothing new reported. Will be implicitly fixed by https://github.com/pytorch/benchmark/pull/1931 and https://github.com/pytorch/benchmark/pull/1932