Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '3316374d14f2e488b65dfb4d0a2560d8414fa19f'
Root cause: Configuration Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 3316374d14f2e488b65dfb4d0a2560d8414fa19f

Affected PyTorch commit: 3316374d14f2e488b65dfb4d0a2560d8414fa19f

Affected Tests:
- resnet18, Adam, cuda, (pt2) no_foreach: +569078.84959%
- resnet18, Adam, cuda, (pt2) foreach: +130.65408%
- resnet18, AdamW, cuda, (pt2) no_foreach: -99.98369%
- resnet18, AdamW, cuda, (pt2) foreach: -63.86188%
- compile_time, resnet18, AdamW, cuda, (pt2) no_foreach: +1325.69542%
- nanogpt_generate, Rprop, cuda, foreach: +31.65346%
- resnet50_quantized_qat, Adadelta, cuda, differentiable: +31.27610%
- timm_resnest, Adadelta, cuda, no_foreach: +47.33533%
- timm_vision_transformer, SGD, cuda, no_foreach: +41.96672%
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: +377607.03734%
- basic_gnn_gin, Rprop, cuda, differentiable: +31.04468%
- mobilenet_v3_large, RMSprop, cuda, no_foreach: +37.35933%


Tests that were no longer run on affected commit:
- stable_diffusion, Adadelta, cuda, default: 0.029211996868252755
- stable_diffusion, Adadelta, cuda, maximize: 0.03144053071737289
- stable_diffusion, Adadelta, cuda, no_foreach: 0.03275497579015792
- stable_diffusion, Adadelta, cuda, differentiable: 0.0383044668007642
- stable_diffusion, Adadelta, cuda, foreach: 0.029261475428938866
- stable_diffusion, Adagrad, cuda, default: 0.014298676163889468
- stable_diffusion, Adagrad, cuda, maximize: 0.01638124920427799
- stable_diffusion, Adagrad, cuda, no_foreach: 0.017552289576269683
- stable_diffusion, Adagrad, cuda, differentiable: 0.018583468860015273
- stable_diffusion, Adagrad, cuda, foreach: 0.014268244989216328
- stable_diffusion, Adam, cuda, default: 0.019714004104025663
- stable_diffusion, Adam, cuda, amsgrad, maximize: 0.024958647927269338
- stable_diffusion, Adam, cuda, no_foreach: 0.03145223390311003
- stable_diffusion, Adam, cuda, differentiable: 0.06257174960337579
- stable_diffusion, Adam, cuda, foreach: 0.01969234184361994
- stable_diffusion, Adam, cuda, foreach, maximize, capturable: 0.02653818656690419
- stable_diffusion, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.029752470878884198
- stable_diffusion, Adam, cuda, fused: 0.009145725580553215
- stable_diffusion, Adam, cuda, fused, amsgrad, maximize: 0.010630130651406944
- stable_diffusion, Adam, cuda, fused, capturable: 0.009131059842184186
- stable_diffusion, Adam, cuda, fused, capturable, amsgrad: 0.010607561119832099
- stable_diffusion, AdamW, cuda, default: 0.021917135827243327
- stable_diffusion, AdamW, cuda, amsgrad, maximize: 0.027138571301475168
- stable_diffusion, AdamW, cuda, no_foreach: 0.03581410041078925
- stable_diffusion, AdamW, cuda, differentiable: 0.06629443811252714
- stable_diffusion, AdamW, cuda, foreach: 0.021883258828893304
- stable_diffusion, AdamW, cuda, foreach, maximize, capturable: 0.028762374026700854
- stable_diffusion, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.03196243792772293
- stable_diffusion, AdamW, cuda, fused: 0.009415908871839443
- stable_diffusion, AdamW, cuda, fused, amsgrad, maximize: 0.010778019647113979
- stable_diffusion, AdamW, cuda, fused, capturable: 0.009394206308449308
- stable_diffusion, AdamW, cuda, fused, capturable, amsgrad: 0.010795521759428085
- stable_diffusion, Adamax, cuda, default: 0.030437493231147528
- stable_diffusion, Adamax, cuda, maximize: 0.03142344933003187
- stable_diffusion, Adamax, cuda, no_foreach: 0.03960654330439865
- stable_diffusion, Adamax, cuda, differentiable: 0.047807925334200266
- stable_diffusion, Adamax, cuda, foreach: 0.02952753808349371
- stable_diffusion, ASGD, cuda, default: 0.036990666715428236
- stable_diffusion, ASGD, cuda, maximize: 0.03743238858878613
- stable_diffusion, ASGD, cuda, no_foreach: 0.05464004427194595
- stable_diffusion, ASGD, cuda, differentiable: 0.054535699309781194
- stable_diffusion, ASGD, cuda, foreach: 0.03419628762640059
- stable_diffusion, SGD, cuda, default: 0.003207672229036689
- stable_diffusion, SGD, cuda, maximize: 0.005360989151522517
- stable_diffusion, SGD, cuda, no_foreach: 0.004158718073740602
- stable_diffusion, SGD, cuda, differentiable: 0.0032076316885650156
- stable_diffusion, SGD, cuda, foreach: 0.003203385816887021
- stable_diffusion, SGD, cuda, foreach, momentum=0.9, nesterov: 0.011885387054644526
- stable_diffusion, SGD, cuda, foreach, momentum=0.9: 0.008681908017024398
- stable_diffusion, RAdam, cuda, default: 0.02415823293849826
- stable_diffusion, RAdam, cuda, no_foreach: 0.05154481488279998
- stable_diffusion, RAdam, cuda, differentiable: 0.052448800392448904
- stable_diffusion, RAdam, cuda, foreach: 0.02412687982432544
- stable_diffusion, Rprop, cuda, default: 0.04043200029991567
- stable_diffusion, Rprop, cuda, maximize: 0.04460880178958178
- stable_diffusion, Rprop, cuda, no_foreach: 0.06543957996182144
- stable_diffusion, Rprop, cuda, differentiable: 0.07102253302000464
- stable_diffusion, Rprop, cuda, foreach: 0.04203118500299752
- stable_diffusion, RMSprop, cuda, default: 0.014083079248666763
- stable_diffusion, RMSprop, cuda, maximize: 0.016325421258807184
- stable_diffusion, RMSprop, cuda, no_foreach: 0.01623686486855149
- stable_diffusion, RMSprop, cuda, differentiable: 0.017984884488396347
- stable_diffusion, RMSprop, cuda, foreach: 0.014058515056967737
- stable_diffusion, NAdam, cuda, default: 0.023864134773612024
- stable_diffusion, NAdam, cuda, no_foreach: 0.036827271804213524
- stable_diffusion, NAdam, cuda, differentiable: 0.06433440954424441
- stable_diffusion, NAdam, cuda, foreach: 0.023803378781303765
- timm_vision_transformer, Adadelta, cuda, (pt2) default: 0.002712108540927639
- timm_vision_transformer, Adadelta, cuda, (pt2) no_foreach: 44.06596795702353
- timm_vision_transformer, Adadelta, cuda, (pt2) foreach: 0.002823313822786153
- timm_vision_transformer, Adagrad, cuda, (pt2) default: 0.0029521924785940964
- timm_vision_transformer, Adagrad, cuda, (pt2) no_foreach: 0.0056321588603572715
- timm_vision_transformer, Adagrad, cuda, (pt2) foreach: 0.003166805392538663
- timm_vision_transformer, Adam, cuda, (pt2) default: 0.003305534557363049
- timm_vision_transformer, Adam, cuda, (pt2) no_foreach: 62.26269617304206
- timm_vision_transformer, Adam, cuda, (pt2) foreach: 0.0073212268429675275
- timm_vision_transformer, Adam, cuda, (pt2) fused: 0.001978367147054158
- timm_vision_transformer, AdamW, cuda, (pt2) default: 0.003360987198539078
- timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: 0.01124863034217722
- timm_vision_transformer, AdamW, cuda, (pt2) foreach: 0.0033683553027609985
- timm_vision_transformer, AdamW, cuda, (pt2) fused: 0.002028638252405205
- timm_vision_transformer, Adamax, cuda, (pt2) default: 0.01280005375156179
- timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: 0.015530530040940413
- timm_vision_transformer, Adamax, cuda, (pt2) foreach: 0.012902791815577075
- timm_vision_transformer, ASGD, cuda, (pt2) default: 0.0037061901573367693
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: 0.009270681025968357
- timm_vision_transformer, ASGD, cuda, (pt2) foreach: 0.003971774450631118
- timm_vision_transformer, SGD, cuda, (pt2) default: 0.0008818606337294442
- timm_vision_transformer, SGD, cuda, (pt2) no_foreach: 0.002679050583392382
- timm_vision_transformer, SGD, cuda, (pt2) foreach: 0.0008654403761189816
- timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.0008240219624713064
- timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9: 0.0007759198602288962
- timm_vision_transformer, RAdam, cuda, (pt2) default: 0.0034665066329762338
- timm_vision_transformer, RAdam, cuda, (pt2) no_foreach: 0.021078725485131143
- timm_vision_transformer, RAdam, cuda, (pt2) foreach: 0.0034249312430620193
- timm_vision_transformer, Rprop, cuda, (pt2) default: 0.002109380326184787
- timm_vision_transformer, Rprop, cuda, (pt2) no_foreach: 0.004515694112827381
- timm_vision_transformer, Rprop, cuda, (pt2) foreach: 0.002066022298807643
- timm_vision_transformer, RMSprop, cuda, (pt2) default: 0.0016633667444333066
- timm_vision_transformer, RMSprop, cuda, (pt2) no_foreach: 0.005090450227726251
- timm_vision_transformer, RMSprop, cuda, (pt2) foreach: 0.0016492811130878868
- timm_vision_transformer, NAdam, cuda, (pt2) default: 0.007767329995448773
- timm_vision_transformer, NAdam, cuda, (pt2) foreach: 0.004371171604598994
- compile_time, timm_vision_transformer, Adadelta, cuda, (pt2) default: 38.21865296348309
- compile_time, timm_vision_transformer, Adadelta, cuda, (pt2) no_foreach: 2.7914228619386776
- compile_time, timm_vision_transformer, Adadelta, cuda, (pt2) foreach: 30.410832598184545
- compile_time, timm_vision_transformer, Adagrad, cuda, (pt2) default: 65.75495957940196
- compile_time, timm_vision_transformer, Adagrad, cuda, (pt2) no_foreach: 57.70802932201574
- compile_time, timm_vision_transformer, Adagrad, cuda, (pt2) foreach: 53.953061491561435
- compile_time, timm_vision_transformer, Adam, cuda, (pt2) default: 57.142535119162254
- compile_time, timm_vision_transformer, Adam, cuda, (pt2) no_foreach: 3.872051712280765
- compile_time, timm_vision_transformer, Adam, cuda, (pt2) foreach: 98.323569103765
- compile_time, timm_vision_transformer, Adam, cuda, (pt2) fused: 1.6590988584794104
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) default: 57.185963986907154
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: 79.01147969532758
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) foreach: 52.660663636401296
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) fused: 6.771795474303265
- compile_time, timm_vision_transformer, Adamax, cuda, (pt2) default: 6.371959067725887
- compile_time, timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: 69.83283673770104
- compile_time, timm_vision_transformer, Adamax, cuda, (pt2) foreach: 2.3349986077907183
- compile_time, timm_vision_transformer, ASGD, cuda, (pt2) default: 57.60032261939099
- compile_time, timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: 64.79580250785996
- compile_time, timm_vision_transformer, ASGD, cuda, (pt2) foreach: 51.67593231688564
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) default: 2.90187039024507
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) no_foreach: 7.390005302305023
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) foreach: 0.6237640753388405
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 6.069555832849194
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9: 0.003257651502887408
- compile_time, timm_vision_transformer, RAdam, cuda, (pt2) default: 0.010133593187977871
- compile_time, timm_vision_transformer, RAdam, cuda, (pt2) no_foreach: 0.008010049971441427
- compile_time, timm_vision_transformer, RAdam, cuda, (pt2) foreach: 0.008666031993925571
- compile_time, timm_vision_transformer, Rprop, cuda, (pt2) default: 73.1346482699737
- compile_time, timm_vision_transformer, Rprop, cuda, (pt2) no_foreach: 57.08071564650163
- compile_time, timm_vision_transformer, Rprop, cuda, (pt2) foreach: 71.72582017537206
- compile_time, timm_vision_transformer, RMSprop, cuda, (pt2) default: 23.904946741492797
- compile_time, timm_vision_transformer, RMSprop, cuda, (pt2) no_foreach: 34.80092734998713
- compile_time, timm_vision_transformer, RMSprop, cuda, (pt2) foreach: 19.88486252988999
- compile_time, timm_vision_transformer, NAdam, cuda, (pt2) default: 9.22676733120655
- compile_time, timm_vision_transformer, NAdam, cuda, (pt2) foreach: 3.060648381554832
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.03356804993624488
- timm_vision_transformer_large, Adam, cuda, foreach: 0.06478645326569676
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable: 0.0789040297580262
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0883752228692174
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.031125162595084736
- timm_vision_transformer_large, Adam, cuda, fused: 0.028625637958092347
- timm_vision_transformer_large, Adam, cuda, fused, amsgrad, maximize: 0.03278018979887877
- timm_vision_transformer_large, Adam, cuda, fused, capturable: 0.02662907149642706
- timm_vision_transformer_large, Adam, cuda, fused, capturable, amsgrad: 0.03033686359412968
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03382382596222063
- timm_vision_transformer_large, AdamW, cuda, default: 0.0701235206797719
- timm_vision_transformer_large, AdamW, cuda, amsgrad, maximize: 0.08474854767943422
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.056810773676261306
- timm_vision_transformer_large, AdamW, cuda, no_foreach: 0.07123727692912023
- timm_vision_transformer_large, AdamW, cuda, differentiable: 0.09262692431608836
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.03113644290715456
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.0695552531008919
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable: 0.08782339747995138
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.0946183066504697
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.03186459227332047
- timm_vision_transformer_large, AdamW, cuda, fused: 0.02986132273716586
- timm_vision_transformer_large, AdamW, cuda, fused, amsgrad, maximize: 0.033066607878676484
- timm_vision_transformer_large, AdamW, cuda, fused, capturable: 0.02739070188254118
- timm_vision_transformer_large, AdamW, cuda, fused, capturable, amsgrad: 0.03066851939074695
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.0876787262968719
- timm_vision_transformer_large, Adamax, cuda, default: 0.08422216990341742
- timm_vision_transformer_large, Adamax, cuda, maximize: 0.0890400898642838
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.07725029143815239
- timm_vision_transformer_large, Adamax, cuda, no_foreach: 0.07206005835905671
- timm_vision_transformer_large, Adamax, cuda, differentiable: 0.07993812067434192
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08653057894359033
- timm_vision_transformer_large, Adamax, cuda, foreach: 0.08355475837985675
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.025777868344448507
- timm_vision_transformer_large, ASGD, cuda, default: 0.07035867838809888
- timm_vision_transformer_large, ASGD, cuda, maximize: 0.08082291406268875
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 0.035276241367682815
- timm_vision_transformer_large, ASGD, cuda, no_foreach: 0.08386183312783639
- timm_vision_transformer_large, ASGD, cuda, differentiable: 0.08435341125975053
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.025373123469762504
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.0695100431330502
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011775839662946323
- timm_vision_transformer_large, SGD, cuda, default: 0.009699096530675888
- timm_vision_transformer_large, SGD, cuda, maximize: 0.016365201491862533
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.013232005119789392
- timm_vision_transformer_large, SGD, cuda, no_foreach: 0.010736266151070595
- timm_vision_transformer_large, SGD, cuda, differentiable: 0.009714197600260377
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.011367018981319334
- timm_vision_transformer_large, SGD, cuda, foreach: 0.00966554076100389
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.035384559305384755
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9, nesterov: 0.035327375307679175
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.025804742192849516
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9: 0.025801977980881928
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07145727560855449
- timm_vision_transformer_large, RAdam, cuda, default: 0.07714040810242295
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.0985592589713633
- timm_vision_transformer_large, RAdam, cuda, no_foreach: 0.09622989180497825
- timm_vision_transformer_large, RAdam, cuda, differentiable: 0.09667122131213546
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07153001660481095
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.07685103127732873
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.026720819936599582
- timm_vision_transformer_large, Rprop, cuda, default: 0.11243451060727239
- timm_vision_transformer_large, Rprop, cuda, maximize: 0.12557692546397448
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03668309305794537
- timm_vision_transformer_large, Rprop, cuda, no_foreach: 0.11992698186077178
- timm_vision_transformer_large, Rprop, cuda, differentiable: 0.12903232616372406
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.02901526666911585
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.11218255804851651
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.018566257108680227
- timm_vision_transformer_large, RMSprop, cuda, default: 0.041818868229165675
- timm_vision_transformer_large, RMSprop, cuda, maximize: 0.04837957886047661
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.028586483227887323
- timm_vision_transformer_large, RMSprop, cuda, no_foreach: 0.043635558197274806
- timm_vision_transformer_large, RMSprop, cuda, differentiable: 0.0443780308123678
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.018664675087414005
- timm_vision_transformer_large, RMSprop, cuda, foreach: 0.04183132802136243
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08480326070760687
- timm_vision_transformer_large, NAdam, cuda, default: 0.07528974472855528
- timm_vision_transformer_large, NAdam, cuda, no_foreach: 0.0765995626958708
- timm_vision_transformer_large, NAdam, cuda, differentiable: 0.09580032977585991
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.07904205455755194
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.07537029823288321
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 382.6510220368703
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) fused: 6.927720667018245
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) default: 401.70119742148864
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 310.1783001703831
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 388.1106026849399
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 6.812987149848293
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) default: 55.44522200462719
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 315.61308366137865
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 11.481411597846696
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) default: 466.24670391716063
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 276.57570099070045
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 424.4855759392182
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) default: 32.5662074649396
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 25.925997568760067
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 2.0685733660745123
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 1.983915849744032
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.04012265428900719
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.0762285434951385
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.08169700112193823
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.0760740015345315
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) default: 604.6000226841619
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 253.70887342452383
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 562.6008276826082
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 176.63322075642645
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 137.00288541413224
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 160.98809494885305
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) default: 96.77892238196607
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 22.95934549951926


Tests that were newly added on affected commit:
- stable_diffusion_text_encoder, Adadelta, cuda, default: 0.029382059909403326
- stable_diffusion_text_encoder, Adadelta, cuda, maximize: 0.0314158686902374
- stable_diffusion_text_encoder, Adadelta, cuda, no_foreach: 0.031257828278467056
- stable_diffusion_text_encoder, Adadelta, cuda, differentiable: 0.03714951579459012
- stable_diffusion_text_encoder, Adadelta, cuda, foreach: 0.02917856741696596
- stable_diffusion_text_encoder, Adagrad, cuda, default: 0.014260623138397932
- stable_diffusion_text_encoder, Adagrad, cuda, maximize: 0.0163313252851367
- stable_diffusion_text_encoder, Adagrad, cuda, no_foreach: 0.020617480902001262
- stable_diffusion_text_encoder, Adagrad, cuda, differentiable: 0.017865607398562133
- stable_diffusion_text_encoder, Adagrad, cuda, foreach: 0.014174006646499038
- stable_diffusion_text_encoder, Adam, cuda, default: 0.01966417774092406
- stable_diffusion_text_encoder, Adam, cuda, amsgrad, maximize: 0.024930310994386674
- stable_diffusion_text_encoder, Adam, cuda, no_foreach: 0.029405520297586918
- stable_diffusion_text_encoder, Adam, cuda, differentiable: 0.0619691182859242
- stable_diffusion_text_encoder, Adam, cuda, foreach: 0.019637402589432897
- stable_diffusion_text_encoder, Adam, cuda, foreach, maximize, capturable: 0.02664154558442533
- stable_diffusion_text_encoder, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.029842779459431767
- stable_diffusion_text_encoder, Adam, cuda, fused: 0.009134036644051472
- stable_diffusion_text_encoder, Adam, cuda, fused, amsgrad, maximize: 0.010647020302712917
- stable_diffusion_text_encoder, Adam, cuda, fused, capturable: 0.009157481603324413
- stable_diffusion_text_encoder, Adam, cuda, fused, capturable, amsgrad: 0.010659442842006685
- stable_diffusion_text_encoder, AdamW, cuda, default: 0.021879847906529904
- stable_diffusion_text_encoder, AdamW, cuda, amsgrad, maximize: 0.027088148426264523
- stable_diffusion_text_encoder, AdamW, cuda, no_foreach: 0.03208028660155833
- stable_diffusion_text_encoder, AdamW, cuda, differentiable: 0.06361192492768168
- stable_diffusion_text_encoder, AdamW, cuda, foreach: 0.02177798319607973
- stable_diffusion_text_encoder, AdamW, cuda, foreach, maximize, capturable: 0.0288213315885514
- stable_diffusion_text_encoder, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.031971775367856024
- stable_diffusion_text_encoder, AdamW, cuda, fused: 0.009402392230307062
- stable_diffusion_text_encoder, AdamW, cuda, fused, amsgrad, maximize: 0.010772573063150048
- stable_diffusion_text_encoder, AdamW, cuda, fused, capturable: 0.009350394985328117
- stable_diffusion_text_encoder, AdamW, cuda, fused, capturable, amsgrad: 0.010738645982928573
- stable_diffusion_text_encoder, Adamax, cuda, default: 0.028842285694554447
- stable_diffusion_text_encoder, Adamax, cuda, maximize: 0.0320940466132015
- stable_diffusion_text_encoder, Adamax, cuda, no_foreach: 0.03842047331854701
- stable_diffusion_text_encoder, Adamax, cuda, differentiable: 0.0465311462059617
- stable_diffusion_text_encoder, Adamax, cuda, foreach: 0.029086037212982772
- stable_diffusion_text_encoder, ASGD, cuda, default: 0.034524241602048276
- stable_diffusion_text_encoder, ASGD, cuda, maximize: 0.037073550699278715
- stable_diffusion_text_encoder, ASGD, cuda, no_foreach: 0.05297914473339915
- stable_diffusion_text_encoder, ASGD, cuda, differentiable: 0.052451896201819184
- stable_diffusion_text_encoder, ASGD, cuda, foreach: 0.033235152205452326
- stable_diffusion_text_encoder, SGD, cuda, default: 0.003192891329526901
- stable_diffusion_text_encoder, SGD, cuda, maximize: 0.005360640338622034
- stable_diffusion_text_encoder, SGD, cuda, no_foreach: 0.0042665710486471655
- stable_diffusion_text_encoder, SGD, cuda, differentiable: 0.0031983218900859356
- stable_diffusion_text_encoder, SGD, cuda, foreach: 0.0032002294529229402
- stable_diffusion_text_encoder, SGD, cuda, foreach, momentum=0.9, nesterov: 0.011851823749020697
- stable_diffusion_text_encoder, SGD, cuda, foreach, momentum=0.9: 0.008652830636128784
- stable_diffusion_text_encoder, RAdam, cuda, default: 0.024118149001151324
- stable_diffusion_text_encoder, RAdam, cuda, no_foreach: 0.04899447141215205
- stable_diffusion_text_encoder, RAdam, cuda, differentiable: 0.050845457473769784
- stable_diffusion_text_encoder, RAdam, cuda, foreach: 0.02402159022167325
- stable_diffusion_text_encoder, Rprop, cuda, default: 0.04061168162152171
- stable_diffusion_text_encoder, Rprop, cuda, maximize: 0.04610411389730871
- stable_diffusion_text_encoder, Rprop, cuda, no_foreach: 0.06360162338241934
- stable_diffusion_text_encoder, Rprop, cuda, differentiable: 0.06917493180371821
- stable_diffusion_text_encoder, Rprop, cuda, foreach: 0.04268956547603011
- stable_diffusion_text_encoder, RMSprop, cuda, default: 0.014076950028538704
- stable_diffusion_text_encoder, RMSprop, cuda, maximize: 0.016239141533151268
- stable_diffusion_text_encoder, RMSprop, cuda, no_foreach: 0.015717347455210983
- stable_diffusion_text_encoder, RMSprop, cuda, differentiable: 0.017904604645445944
- stable_diffusion_text_encoder, RMSprop, cuda, foreach: 0.01403156090527773
- stable_diffusion_text_encoder, NAdam, cuda, default: 0.02381817949935794
- stable_diffusion_text_encoder, NAdam, cuda, no_foreach: 0.03481138730421662
- stable_diffusion_text_encoder, NAdam, cuda, differentiable: 0.06258335863240064
- stable_diffusion_text_encoder, NAdam, cuda, foreach: 0.023752334993332624
- stable_diffusion_unet, Adadelta, cuda, default: 0.07701420194158952
- stable_diffusion_unet, Adadelta, cuda, maximize: 0.08906794898211956
- stable_diffusion_unet, Adadelta, cuda, no_foreach: 0.0873325193921725
- stable_diffusion_unet, Adadelta, cuda, differentiable: 0.09802836490174134
- stable_diffusion_unet, Adadelta, cuda, foreach: 0.07646466046571732
- stable_diffusion_unet, Adagrad, cuda, default: 0.036154484981670974
- stable_diffusion_unet, Adagrad, cuda, maximize: 0.04154024501331151
- stable_diffusion_unet, Adagrad, cuda, no_foreach: 0.034867606963962315
- stable_diffusion_unet, Adagrad, cuda, differentiable: 0.035925067588686944
- stable_diffusion_unet, Adagrad, cuda, foreach: 0.03613904099911451
- stable_diffusion_unet, Adam, cuda, default: 0.05003477069549263
- stable_diffusion_unet, Adam, cuda, amsgrad, maximize: 0.0671836226247251
- stable_diffusion_unet, Adam, cuda, no_foreach: 0.060824281768873334
- stable_diffusion_unet, Adam, cuda, differentiable: 0.1174429808743298
- stable_diffusion_unet, Adam, cuda, foreach: 0.0499873480759561
- stable_diffusion_unet, Adam, cuda, foreach, maximize, capturable: 0.06908366006488602
- stable_diffusion_unet, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.07710577516506116
- stable_diffusion_unet, Adam, cuda, fused: 0.023279389599338174
- stable_diffusion_unet, Adam, cuda, fused, amsgrad, maximize: 0.026657382817938925
- stable_diffusion_unet, Adam, cuda, fused, capturable: 0.02324977838434279
- stable_diffusion_unet, Adam, cuda, fused, capturable, amsgrad: 0.026677310699597
- stable_diffusion_unet, AdamW, cuda, default: 0.05565505269914865
- stable_diffusion_unet, AdamW, cuda, amsgrad, maximize: 0.07310214514533679
- stable_diffusion_unet, AdamW, cuda, no_foreach: 0.07124639752631386
- stable_diffusion_unet, AdamW, cuda, differentiable: 0.11650068615563214
- stable_diffusion_unet, AdamW, cuda, foreach: 0.055603304971009494
- stable_diffusion_unet, AdamW, cuda, foreach, maximize, capturable: 0.07442698969195287
- stable_diffusion_unet, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.08259949926286936
- stable_diffusion_unet, AdamW, cuda, fused: 0.023909953190013766
- stable_diffusion_unet, AdamW, cuda, fused, amsgrad, maximize: 0.026950037525966763
- stable_diffusion_unet, AdamW, cuda, fused, capturable: 0.023913463996723294
- stable_diffusion_unet, AdamW, cuda, fused, capturable, amsgrad: 0.026981590688228606
- stable_diffusion_unet, Adamax, cuda, default: 0.07693998174120982
- stable_diffusion_unet, Adamax, cuda, maximize: 0.08030656399205327
- stable_diffusion_unet, Adamax, cuda, no_foreach: 0.07533638644963503
- stable_diffusion_unet, Adamax, cuda, differentiable: 0.08994652734448512
- stable_diffusion_unet, Adamax, cuda, foreach: 0.07789745662982266
- stable_diffusion_unet, ASGD, cuda, default: 0.07106006021300952
- stable_diffusion_unet, ASGD, cuda, maximize: 0.07678500221421321
- stable_diffusion_unet, ASGD, cuda, no_foreach: 0.10526762413792312
- stable_diffusion_unet, ASGD, cuda, differentiable: 0.10210018930956721
- stable_diffusion_unet, ASGD, cuda, foreach: 0.07083075462530057
- stable_diffusion_unet, SGD, cuda, default: 0.008358683421586952
- stable_diffusion_unet, SGD, cuda, maximize: 0.014056275878101588
- stable_diffusion_unet, SGD, cuda, no_foreach: 0.009909124424060185
- stable_diffusion_unet, SGD, cuda, differentiable: 0.008332858638217052
- stable_diffusion_unet, SGD, cuda, foreach: 0.008155739870853723
- stable_diffusion_unet, SGD, cuda, foreach, momentum=0.9, nesterov: 0.030167986592277884
- stable_diffusion_unet, SGD, cuda, foreach, momentum=0.9: 0.02205287027172744
- stable_diffusion_unet, RAdam, cuda, default: 0.06755822415774067
- stable_diffusion_unet, RAdam, cuda, no_foreach: 0.1048328595701605
- stable_diffusion_unet, RAdam, cuda, differentiable: 0.10745692960917949
- stable_diffusion_unet, RAdam, cuda, foreach: 0.06701219050834577
- stable_diffusion_unet, Rprop, cuda, default: 0.10693463846109807
- stable_diffusion_unet, Rprop, cuda, maximize: 0.11404015962034464
- stable_diffusion_unet, Rprop, cuda, no_foreach: 0.13914034003391862
- stable_diffusion_unet, Rprop, cuda, differentiable: 0.14883160684257746
- stable_diffusion_unet, Rprop, cuda, foreach: 0.10229448857717216
- stable_diffusion_unet, RMSprop, cuda, default: 0.035796820372343066
- stable_diffusion_unet, RMSprop, cuda, maximize: 0.04143846509978175
- stable_diffusion_unet, RMSprop, cuda, no_foreach: 0.03962955153547228
- stable_diffusion_unet, RMSprop, cuda, differentiable: 0.03960798499174416
- stable_diffusion_unet, RMSprop, cuda, foreach: 0.03578191688284278
- stable_diffusion_unet, NAdam, cuda, default: 0.06560669594909996
- stable_diffusion_unet, NAdam, cuda, no_foreach: 0.07596312168364723
- stable_diffusion_unet, NAdam, cuda, differentiable: 0.11663283640518785
- stable_diffusion_unet, NAdam, cuda, foreach: 0.06470448256004602


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6154735668

cc @janeyx99


fixed by https://github.com/pytorch/benchmark/pull/1931