Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on 'a86aae1504e8051898ebf3cc7ec5284ffa531d11'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch commit: 73b995ec2d10718602011d2cb831cd64847e6af4

Affected PyTorch commit: a86aae1504e8051898ebf3cc7ec5284ffa531d11

Affected Tests:


Tests that were no longer run on affected commit:
- resnet18, Adam, cuda, (pt2) default: 0.001520964870878467
- resnet18, Adam, cuda, default: 0.001244314849609509
- resnet18, Adam, cuda, (pt2) amsgrad, maximize: 0.0017898450252167614
- resnet18, Adam, cuda, amsgrad, maximize: 0.0014367541647516192
- resnet18, Adam, cuda, (pt2) no_foreach: 0.0020428020103207354
- resnet18, Adam, cuda, no_foreach: 0.00540045402944088
- resnet18, Adam, cuda, (pt2) differentiable: 0.0021132766661283216
- resnet18, Adam, cuda, differentiable: 0.009787360904738309
- resnet18, Adam, cuda, (pt2) foreach: 0.0015364095643304915
- resnet18, Adam, cuda, foreach: 0.0012657534237951041
- resnet18, Adam, cuda, (pt2) foreach, maximize, capturable: 0.0029404235065685234
- resnet18, Adam, cuda, foreach, maximize, capturable: 0.0026863575913012028
- resnet18, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0029449363732162643
- resnet18, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0027106082113459704
- resnet18, Adam, cuda, (pt2) fused: 0.0010018647857941688
- resnet18, Adam, cuda, fused: 0.0005248893781099468
- resnet18, Adam, cuda, (pt2) fused, amsgrad, maximize: 0.00122525909398796
- resnet18, Adam, cuda, fused, amsgrad, maximize: 0.0007147796780336648
- resnet18, Adam, cuda, (pt2) fused, capturable: 0.000993627177708146
- resnet18, Adam, cuda, fused, capturable: 0.000523147723870352
- resnet18, Adam, cuda, (pt2) fused, capturable, amsgrad: 0.0012358245626666847
- resnet18, Adam, cuda, fused, capturable, amsgrad: 0.0007133349569048733
- resnet18, Adam, cpu, (pt2) default: 0.1487299669533968
- resnet18, Adam, cpu, default: 0.05387116316705942
- resnet18, Adam, cpu, (pt2) amsgrad, maximize: 0.13431243505328894
- resnet18, Adam, cpu, amsgrad, maximize: 0.06812230365661283
- resnet18, Adam, cpu, (pt2) no_foreach: 0.145155648002401
- resnet18, Adam, cpu, no_foreach: 0.055559802590869366
- resnet18, Adam, cpu, (pt2) differentiable: 0.1334845715900883
- resnet18, Adam, cpu, differentiable: 0.05828429840039462
- resnet18, Adam, cpu, (pt2) foreach: 0.059038830921053886
- resnet18, Adam, cpu, foreach: 0.05979954247595742


Tests that were newly added on affected commit:
- BERT_pytorch, Adadelta, cuda, (pt2) default: 0.00962172666666967
- BERT_pytorch, Adadelta, cuda, default: 0.008773201700000755
- BERT_pytorch, Adadelta, cuda, (pt2) no_foreach: 0.003467933034487252
- BERT_pytorch, Adadelta, cuda, no_foreach: 0.017338795899991054
- BERT_pytorch, Adadelta, cuda, (pt2) foreach: 0.009617351571424002
- BERT_pytorch, Adadelta, cuda, foreach: 0.008682980710000265
- BERT_pytorch, Adagrad, cuda, (pt2) default: 0.011987735941201047
- BERT_pytorch, Adagrad, cuda, default: 0.009992720999995678
- BERT_pytorch, Adagrad, cuda, (pt2) no_foreach: 0.004720054186051595
- BERT_pytorch, Adagrad, cuda, no_foreach: 0.008999632300007457
- BERT_pytorch, Adagrad, cuda, (pt2) foreach: 0.012169795529389911
- BERT_pytorch, Adagrad, cuda, foreach: 0.010078284550013451
- BERT_pytorch, Adam, cuda, (pt2) default: 0.008542439875014907
- BERT_pytorch, Adam, cuda, default: 0.0064225045099988165
- BERT_pytorch, Adam, cuda, (pt2) no_foreach: 0.005264524421052461
- BERT_pytorch, Adam, cuda, no_foreach: 0.017266481449985348
- BERT_pytorch, Adam, cuda, (pt2) foreach: 0.008361212541691051
- BERT_pytorch, Adam, cuda, foreach: 0.006415272919998643
- BERT_pytorch, Adam, cuda, (pt2) fused: 0.004269644936159161
- BERT_pytorch, Adam, cuda, fused: 0.0031561564100002215
- BERT_pytorch, AdamW, cuda, (pt2) default: 0.009025819652181424
- BERT_pytorch, AdamW, cuda, default: 0.007092428710000149
- BERT_pytorch, AdamW, cuda, (pt2) no_foreach: 0.006190198242434649
- BERT_pytorch, AdamW, cuda, no_foreach: 0.018629924300012132
- BERT_pytorch, AdamW, cuda, (pt2) foreach: 0.008990291956518811
- BERT_pytorch, AdamW, cuda, foreach: 0.007072333840001193
- BERT_pytorch, AdamW, cuda, (pt2) fused: 0.00433918187233777
- BERT_pytorch, AdamW, cuda, fused: 0.0032537993899995855
- BERT_pytorch, Adamax, cuda, (pt2) default: 0.016965578250032348
- BERT_pytorch, Adamax, cuda, default: 0.015449643450006078
- BERT_pytorch, Adamax, cuda, (pt2) no_foreach: 0.012189945176487527
- BERT_pytorch, Adamax, cuda, no_foreach: 0.022037975399985043
- BERT_pytorch, Adamax, cuda, (pt2) foreach: 0.016940853833337616
- BERT_pytorch, Adamax, cuda, foreach: 0.015562952300001598
- BERT_pytorch, ASGD, cuda, (pt2) default: 0.006823908833340889
- BERT_pytorch, ASGD, cuda, default: 0.00639665346999891
- BERT_pytorch, ASGD, cuda, (pt2) no_foreach: 0.006611688516115112
- BERT_pytorch, ASGD, cuda, no_foreach: 0.011683287800008203
- BERT_pytorch, ASGD, cuda, (pt2) foreach: 0.007011862310327129
- BERT_pytorch, ASGD, cuda, foreach: 0.006440045499998632
- BERT_pytorch, SGD, cuda, (pt2) default: 0.0017288053189680607
- BERT_pytorch, SGD, cuda, default: 0.0009499480200004957
- BERT_pytorch, SGD, cuda, (pt2) no_foreach: 0.002483085654336528
- BERT_pytorch, SGD, cuda, no_foreach: 0.00150933847499914
- BERT_pytorch, SGD, cuda, (pt2) foreach: 0.001714074709391356
- BERT_pytorch, SGD, cuda, foreach: 0.000947825709999203
- BERT_pytorch, RAdam, cuda, (pt2) default: 0.008563225499998831
- BERT_pytorch, RAdam, cuda, default: 0.008625170033330203
- BERT_pytorch, RAdam, cuda, (pt2) no_foreach: 0.0268983955000067
- BERT_pytorch, RAdam, cuda, no_foreach: 0.02713073139998414
- BERT_pytorch, RAdam, cuda, (pt2) foreach: 0.008606009200002518
- BERT_pytorch, RAdam, cuda, foreach: 0.008609968166676177
- BERT_pytorch, Rprop, cuda, (pt2) default: 0.028946369000030763
- BERT_pytorch, Rprop, cuda, default: 0.026360538599988104
- BERT_pytorch, Rprop, cuda, (pt2) no_foreach: 0.003536558473701677
- BERT_pytorch, Rprop, cuda, no_foreach: 0.03486244870000519
- BERT_pytorch, Rprop, cuda, (pt2) foreach: 0.027635104500006946
- BERT_pytorch, Rprop, cuda, foreach: 0.026770645700003114
- BERT_pytorch, RMSprop, cuda, (pt2) default: 0.005199925025635821
- BERT_pytorch, RMSprop, cuda, default: 0.004151451689999704
- BERT_pytorch, RMSprop, cuda, (pt2) no_foreach: 0.0028737936856974297
- BERT_pytorch, RMSprop, cuda, no_foreach: 0.008396157799999552
- BERT_pytorch, RMSprop, cuda, (pt2) foreach: 0.005200442076941796
- BERT_pytorch, RMSprop, cuda, foreach: 0.004151996309999504
- BERT_pytorch, NAdam, cuda, (pt2) default: 0.010658506421087191
- BERT_pytorch, NAdam, cuda, default: 0.007913249566657517
- BERT_pytorch, NAdam, cuda, (pt2) no_foreach: 0.0076945263076945245
- BERT_pytorch, NAdam, cuda, no_foreach: 0.02006397969998943
- BERT_pytorch, NAdam, cuda, (pt2) foreach: 0.010628394000006761
- BERT_pytorch, NAdam, cuda, foreach: 0.0079308481000074
- DALLE2_pytorch, Adadelta, cuda, (pt2) default: 0.036807544499955235
- DALLE2_pytorch, Adadelta, cuda, default: 0.029761869100002513
- DALLE2_pytorch, Adadelta, cuda, (pt2) foreach: 0.03727890666668069
- DALLE2_pytorch, Adadelta, cuda, foreach: 0.029759211900000082
- DALLE2_pytorch, Adagrad, cuda, (pt2) default: 0.07956654133333056
- DALLE2_pytorch, Adagrad, cuda, default: 0.0818935556666626
- DALLE2_pytorch, Adagrad, cuda, (pt2) foreach: 0.08064130533330172
- DALLE2_pytorch, Adagrad, cuda, foreach: 0.07707465266662439
- DALLE2_pytorch, Adam, cuda, (pt2) default: 0.036385470833351974
- DALLE2_pytorch, Adam, cuda, default: 0.03074709609998081
- DALLE2_pytorch, Adam, cuda, (pt2) foreach: 0.03622840916667277
- DALLE2_pytorch, Adam, cuda, foreach: 0.031234412899993912
- DALLE2_pytorch, Adam, cuda, (pt2) fused: 0.022125152800003888
- DALLE2_pytorch, Adam, cuda, fused: 0.013835438650005472
- DALLE2_pytorch, AdamW, cuda, (pt2) default: 0.03809678833340513
- DALLE2_pytorch, AdamW, cuda, default: 0.030230931500000224
- DALLE2_pytorch, AdamW, cuda, (pt2) foreach: 0.037821064500046
- DALLE2_pytorch, AdamW, cuda, foreach: 0.03177214719999029
- DALLE2_pytorch, AdamW, cuda, (pt2) fused: 0.02296345700005986
- DALLE2_pytorch, AdamW, cuda, fused: 0.01430538214998478
- DALLE2_pytorch, Adamax, cuda, (pt2) default: 0.12291767299996081
- DALLE2_pytorch, Adamax, cuda, default: 0.12005537999993976
- DALLE2_pytorch, Adamax, cuda, (pt2) foreach: 0.12424498600000788
- DALLE2_pytorch, Adamax, cuda, foreach: 0.12104704500006847
- DALLE2_pytorch, ASGD, cuda, (pt2) default: 0.05516879549998066
- DALLE2_pytorch, ASGD, cuda, default: 0.050723895499970695
- DALLE2_pytorch, ASGD, cuda, (pt2) foreach: 0.059752176250015054
- DALLE2_pytorch, ASGD, cuda, foreach: 0.05548667350001324
- DALLE2_pytorch, SGD, cuda, (pt2) default: 0.00871184265219713
- DALLE2_pytorch, SGD, cuda, default: 0.005852273010000317
- DALLE2_pytorch, SGD, cuda, (pt2) foreach: 0.00845708583333741
- DALLE2_pytorch, SGD, cuda, foreach: 0.005562555470000916
- DALLE2_pytorch, RAdam, cuda, (pt2) default: 0.045936035599970636
- DALLE2_pytorch, RAdam, cuda, default: 0.045698763600057646
- DALLE2_pytorch, RAdam, cuda, (pt2) foreach: 0.04741300059990863
- DALLE2_pytorch, RAdam, cuda, foreach: 0.04680633640000451
- DALLE2_pytorch, Rprop, cuda, (pt2) default: 0.20524999500003105
- DALLE2_pytorch, Rprop, cuda, default: 0.19724737200010622
- DALLE2_pytorch, Rprop, cuda, (pt2) foreach: 0.20301387800009252
- DALLE2_pytorch, Rprop, cuda, foreach: 0.1978611129999308
- DALLE2_pytorch, RMSprop, cuda, (pt2) default: 0.02164417790004336
- DALLE2_pytorch, RMSprop, cuda, default: 0.014701300350009206
- DALLE2_pytorch, RMSprop, cuda, (pt2) foreach: 0.021506182499956596
- DALLE2_pytorch, RMSprop, cuda, foreach: 0.014649646399993799
- DALLE2_pytorch, NAdam, cuda, (pt2) default: 0.047182796600009166
- DALLE2_pytorch, NAdam, cuda, default: 0.045299057600004744
- DALLE2_pytorch, NAdam, cuda, (pt2) foreach: 0.046830740399991556
- DALLE2_pytorch, NAdam, cuda, foreach: 0.04470139600002767
- hf_GPT2_large, Adadelta, cuda, (pt2) default: 0.06934376366666584
- hf_GPT2_large, Adadelta, cuda, default: 0.06850807999997717
- hf_GPT2_large, Adadelta, cuda, (pt2) no_foreach: 0.01962334727276853
- hf_GPT2_large, Adadelta, cuda, no_foreach: 0.06912866633335095
- hf_GPT2_large, Adadelta, cuda, (pt2) foreach: 0.0690435903333461
- hf_GPT2_large, Adadelta, cuda, foreach: 0.0682793966667153
- hf_GPT2_large, Adagrad, cuda, (pt2) default: 0.05321712274997026
- hf_GPT2_large, Adagrad, cuda, default: 0.03916514889999689
- hf_GPT2_large, Adagrad, cuda, (pt2) no_foreach: 0.01738531974996249
- hf_GPT2_large, Adagrad, cuda, no_foreach: 0.027785243599987554
- hf_GPT2_large, Adagrad, cuda, (pt2) foreach: 0.05349756325000499
- hf_GPT2_large, Adagrad, cuda, foreach: 0.03911344950001876
- hf_GPT2_large, Adam, cuda, (pt2) default: 0.05452470049999647
- hf_GPT2_large, Adam, cuda, default: 0.04946058860000448
- hf_GPT2_large, Adam, cuda, (pt2) no_foreach: 0.02299137622220668
- hf_GPT2_large, Adam, cuda, no_foreach: 0.05050570719999996
- hf_GPT2_large, Adam, cuda, (pt2) foreach: 0.05385924200010095
- hf_GPT2_large, Adam, cuda, foreach: 0.0494354200999851
- hf_GPT2_large, Adam, cuda, (pt2) fused: 0.023322262333320194
- hf_GPT2_large, Adam, cuda, fused: 0.02060030729999198
- hf_GPT2_large, AdamW, cuda, (pt2) default: 0.059762232499963375
- hf_GPT2_large, AdamW, cuda, default: 0.054455595100012036
- hf_GPT2_large, AdamW, cuda, (pt2) no_foreach: 0.023234304111156234
- hf_GPT2_large, AdamW, cuda, no_foreach: 0.05597759600000245
- hf_GPT2_large, AdamW, cuda, (pt2) foreach: 0.06087398175009184
- hf_GPT2_large, AdamW, cuda, foreach: 0.05451825939999253
- hf_GPT2_large, AdamW, cuda, (pt2) fused: 0.02385415511111609
- hf_GPT2_large, AdamW, cuda, fused: 0.021147247599992626
- hf_GPT2_large, Adamax, cuda, (pt2) default: 0.07115580433355717
- hf_GPT2_large, Adamax, cuda, default: 0.07045061899998473
- hf_GPT2_large, Adamax, cuda, (pt2) no_foreach: 0.04037174980003329
- hf_GPT2_large, Adamax, cuda, no_foreach: 0.059406322599988924
- hf_GPT2_large, Adamax, cuda, (pt2) foreach: 0.07105973833343644
- hf_GPT2_large, Adamax, cuda, foreach: 0.07066378833330116
- hf_GPT2_large, ASGD, cuda, (pt2) default: 0.022700176333273703
- hf_GPT2_large, ASGD, cuda, default: 0.018788147499981277
- hf_GPT2_large, ASGD, cuda, (pt2) foreach: 0.02288187766675441
- hf_GPT2_large, ASGD, cuda, foreach: 0.01875984575001439
- hf_GPT2_large, SGD, cuda, (pt2) default: 0.009008867695703648
- hf_GPT2_large, SGD, cuda, default: 0.0072782833200017195
- hf_GPT2_large, SGD, cuda, (pt2) no_foreach: 0.00959177180956907
- hf_GPT2_large, SGD, cuda, no_foreach: 0.008435246066680216
- hf_GPT2_large, SGD, cuda, (pt2) foreach: 0.00887476399991457
- hf_GPT2_large, SGD, cuda, foreach: 0.007264424970003347
- hf_GPT2_large, RAdam, cuda, (pt2) default: 0.06670187300005637
- hf_GPT2_large, RAdam, cuda, default: 0.06205447249999452
- hf_GPT2_large, RAdam, cuda, (pt2) no_foreach: 0.07683038999998644
- hf_GPT2_large, RAdam, cuda, no_foreach: 0.07679096259998915
- hf_GPT2_large, RAdam, cuda, (pt2) foreach: 0.06724487833344028
- hf_GPT2_large, RAdam, cuda, foreach: 0.06695845266661611
- hf_GPT2_large, Rprop, cuda, (pt2) default: 0.09241982233334056
- hf_GPT2_large, Rprop, cuda, default: 0.09170367500003825
- hf_GPT2_large, Rprop, cuda, (pt2) no_foreach: 0.020281163500021647
- hf_GPT2_large, Rprop, cuda, no_foreach: 0.08628524866693017
- hf_GPT2_large, Rprop, cuda, (pt2) foreach: 0.09244102433331136
- hf_GPT2_large, Rprop, cuda, foreach: 0.09154527566670367
- hf_GPT2_large, RMSprop, cuda, (pt2) default: 0.03446557316669896
- hf_GPT2_large, RMSprop, cuda, default: 0.03194652070001212
- hf_GPT2_large, RMSprop, cuda, (pt2) no_foreach: 0.014781859714275924
- hf_GPT2_large, RMSprop, cuda, no_foreach: 0.03237478340001872
- hf_GPT2_large, RMSprop, cuda, (pt2) foreach: 0.03447103049992014
- hf_GPT2_large, RMSprop, cuda, foreach: 0.03194476399999076
- hf_GPT2_large, NAdam, cuda, (pt2) default: 0.06613130574999104
- hf_GPT2_large, NAdam, cuda, default: 0.06538253049984633
- hf_GPT2_large, NAdam, cuda, (pt2) foreach: 0.06629151249990173
- hf_GPT2_large, NAdam, cuda, foreach: 0.0653313062499592
- hf_T5_large, Adadelta, cuda, (pt2) default: 0.06662916349978332
- hf_T5_large, Adadelta, cuda, default: 0.06328694099997847
- hf_T5_large, Adadelta, cuda, (pt2) no_foreach: 0.020148731800099996
- hf_T5_large, Adadelta, cuda, no_foreach: 0.051837431300009486
- hf_T5_large, Adadelta, cuda, (pt2) foreach: 0.06637891324976408
- hf_T5_large, Adadelta, cuda, foreach: 0.06338914590000969
- hf_T5_large, Adagrad, cuda, (pt2) default: 0.05514272550010446
- hf_T5_large, Adagrad, cuda, default: 0.038697494699999876
- hf_T5_large, Adagrad, cuda, (pt2) no_foreach: 0.017954640583297987
- hf_T5_large, Adagrad, cuda, no_foreach: 0.024310906600021553
- hf_T5_large, Adagrad, cuda, (pt2) foreach: 0.05689695575006226
- hf_T5_large, Adagrad, cuda, foreach: 0.037795345799986536
- hf_T5_large, Adam, cuda, (pt2) default: 0.052682795250007075
- hf_T5_large, Adam, cuda, default: 0.04708135190003304
- hf_T5_large, Adam, cuda, (pt2) no_foreach: 0.02351113388886006
- hf_T5_large, Adam, cuda, no_foreach: 0.04774327549998816
- hf_T5_large, Adam, cuda, (pt2) foreach: 0.05292176424995887
- hf_T5_large, Adam, cuda, foreach: 0.047058552699991194
- hf_T5_large, Adam, cuda, (pt2) fused: 0.022602716222233237
- hf_T5_large, Adam, cuda, fused: 0.019452825749999646
- hf_T5_large, AdamW, cuda, (pt2) default: 0.05714248325000426
- hf_T5_large, AdamW, cuda, default: 0.051757548300020065
- hf_T5_large, AdamW, cuda, (pt2) no_foreach: 0.02481715088879355
- hf_T5_large, AdamW, cuda, no_foreach: 0.05042751290002343
- hf_T5_large, AdamW, cuda, (pt2) foreach: 0.05798515149990635
- hf_T5_large, AdamW, cuda, foreach: 0.05175785399997039
- hf_T5_large, AdamW, cuda, (pt2) fused: 0.023365205888745147
- hf_T5_large, AdamW, cuda, fused: 0.01995826530001068
- hf_T5_large, Adamax, cuda, (pt2) default: 0.06801151266684731
- hf_T5_large, Adamax, cuda, default: 0.06727688899976177
- hf_T5_large, Adamax, cuda, (pt2) no_foreach: 0.044243549999919196
- hf_T5_large, Adamax, cuda, no_foreach: 0.059764059500002985
- hf_T5_large, Adamax, cuda, (pt2) foreach: 0.06822193000001182
- hf_T5_large, Adamax, cuda, foreach: 0.06717742533328419
- hf_T5_large, ASGD, cuda, (pt2) default: 0.02350883644440829
- hf_T5_large, ASGD, cuda, default: 0.01851765695000722
- hf_T5_large, ASGD, cuda, (pt2) foreach: 0.02318493999998989
- hf_T5_large, ASGD, cuda, foreach: 0.018479029199988874
- hf_T5_large, SGD, cuda, (pt2) default: 0.009000371739128477
- hf_T5_large, SGD, cuda, default: 0.006924702510000316
- hf_T5_large, SGD, cuda, (pt2) no_foreach: 0.010022049700000934
- hf_T5_large, SGD, cuda, no_foreach: 0.008567631633331985
- hf_T5_large, SGD, cuda, (pt2) foreach: 0.008908676130411786
- hf_T5_large, SGD, cuda, foreach: 0.006913781040002505
- hf_T5_large, RAdam, cuda, (pt2) default: 0.06456084550006835
- hf_T5_large, RAdam, cuda, default: 0.0648156279999057
- hf_T5_large, RAdam, cuda, (pt2) no_foreach: 0.07163401520001571
- hf_T5_large, RAdam, cuda, no_foreach: 0.07185867560001498
- hf_T5_large, RAdam, cuda, (pt2) foreach: 0.06482928874993377
- hf_T5_large, RAdam, cuda, foreach: 0.06445269450011892
- hf_T5_large, Rprop, cuda, (pt2) default: 0.09037757499982035
- hf_T5_large, Rprop, cuda, default: 0.08878275766664956
- hf_T5_large, Rprop, cuda, (pt2) no_foreach: 0.020424853100030303
- hf_T5_large, Rprop, cuda, no_foreach: 0.09394019000001208
- hf_T5_large, Rprop, cuda, (pt2) foreach: 0.09239896899998712
- hf_T5_large, Rprop, cuda, foreach: 0.08976033466645579
- hf_T5_large, RMSprop, cuda, (pt2) default: 0.033280925714409805
- hf_T5_large, RMSprop, cuda, default: 0.030429445400022813
- hf_T5_large, RMSprop, cuda, (pt2) no_foreach: 0.015398476000007997
- hf_T5_large, RMSprop, cuda, no_foreach: 0.025542735099998028
- hf_T5_large, RMSprop, cuda, (pt2) foreach: 0.03336872666667053
- hf_T5_large, RMSprop, cuda, foreach: 0.030433957500008546
- hf_T5_large, NAdam, cuda, (pt2) default: 0.06449742749987308
- hf_T5_large, NAdam, cuda, default: 0.06349770100018759
- hf_T5_large, NAdam, cuda, (pt2) foreach: 0.06480039850009689
- hf_T5_large, NAdam, cuda, foreach: 0.06350255325014587
- resnet50, Adadelta, cuda, (pt2) default: 0.003370048583284794
- resnet50, Adadelta, cuda, default: 0.002496191489999546
- resnet50, Adadelta, cuda, (pt2) no_foreach: 0.002917285942045134
- resnet50, Adadelta, cuda, no_foreach: 0.014294921700002305
- resnet50, Adadelta, cuda, (pt2) foreach: 0.0033675472999978714
- resnet50, Adadelta, cuda, foreach: 0.0024955961200021195
- resnet50, Adagrad, cuda, (pt2) default: 0.008610426375033361
- resnet50, Adagrad, cuda, default: 0.00832092249999429
- resnet50, Adagrad, cuda, (pt2) no_foreach: 0.004362194369536076
- resnet50, Adagrad, cuda, no_foreach: 0.007780439650000517
- resnet50, Adagrad, cuda, (pt2) foreach: 0.008791758391235699
- resnet50, Adagrad, cuda, foreach: 0.008691224033327671
- resnet50, Adam, cuda, (pt2) default: 0.0037373691666373445
- resnet50, Adam, cuda, default: 0.0034446661700030744
- resnet50, Adam, cuda, (pt2) no_foreach: 0.0048644215714180716
- resnet50, Adam, cuda, no_foreach: 0.014892815799976233
- resnet50, Adam, cuda, (pt2) foreach: 0.0037025335272780715
- resnet50, Adam, cuda, foreach: 0.00338542033000067
- resnet50, Adam, cuda, (pt2) fused: 0.0022660258089958556
- resnet50, Adam, cuda, fused: 0.0011638661950019013
- resnet50, AdamW, cuda, (pt2) default: 0.003842859698116232
- resnet50, AdamW, cuda, default: 0.0033717578599998885
- resnet50, AdamW, cuda, (pt2) no_foreach: 0.004814355428585432
- resnet50, AdamW, cuda, no_foreach: 0.01605234090000067
- resnet50, AdamW, cuda, (pt2) foreach: 0.003885977865383421
- resnet50, AdamW, cuda, foreach: 0.0033096003299988297
- resnet50, AdamW, cuda, (pt2) fused: 0.00224358026667283
- resnet50, AdamW, cuda, fused: 0.0012008988649995444
- resnet50, Adamax, cuda, (pt2) default: 0.015109480500021684
- resnet50, Adamax, cuda, default: 0.01476738590001787
- resnet50, Adamax, cuda, (pt2) no_foreach: 0.01094646931580045
- resnet50, Adamax, cuda, no_foreach: 0.0199079160500105
- resnet50, Adamax, cuda, (pt2) foreach: 0.015118842642972854
- resnet50, Adamax, cuda, foreach: 0.014542689500012784
- resnet50, ASGD, cuda, (pt2) default: 0.005943960205941574
- resnet50, ASGD, cuda, default: 0.005786808060001931
- resnet50, ASGD, cuda, (pt2) no_foreach: 0.005984875117706799
- resnet50, ASGD, cuda, no_foreach: 0.010009751133323638
- resnet50, ASGD, cuda, (pt2) foreach: 0.005974770235284536
- resnet50, ASGD, cuda, foreach: 0.005710818659999859
- resnet50, SGD, cuda, (pt2) default: 0.0009777372731594191
- resnet50, SGD, cuda, default: 0.0005964835049999238
- resnet50, SGD, cuda, (pt2) no_foreach: 0.002327492232568549
- resnet50, SGD, cuda, no_foreach: 0.0013040866350002033
- resnet50, SGD, cuda, (pt2) foreach: 0.0009531805190623085
- resnet50, SGD, cuda, foreach: 0.0005600364500000978
- resnet50, RAdam, cuda, (pt2) default: 0.005458850470004109
- resnet50, RAdam, cuda, default: 0.005144753579997996
- resnet50, RAdam, cuda, (pt2) no_foreach: 0.0230753328999981
- resnet50, RAdam, cuda, no_foreach: 0.02297050990000571
- resnet50, RAdam, cuda, (pt2) foreach: 0.005215110160002041
- resnet50, RAdam, cuda, foreach: 0.005105098869998983
- resnet50, Rprop, cuda, (pt2) default: 0.022961821333259042
- resnet50, Rprop, cuda, default: 0.022391063300028692
- resnet50, Rprop, cuda, (pt2) no_foreach: 0.0035238642631236616
- resnet50, Rprop, cuda, no_foreach: 0.029415250700003524
- resnet50, Rprop, cuda, (pt2) foreach: 0.023337336666726414
- resnet50, Rprop, cuda, foreach: 0.022908328099993013
- resnet50, RMSprop, cuda, (pt2) default: 0.002205241835181968
- resnet50, RMSprop, cuda, default: 0.0014571624600012
- resnet50, RMSprop, cuda, (pt2) no_foreach: 0.002583741935874665
- resnet50, RMSprop, cuda, no_foreach: 0.0071215302799964774
- resnet50, RMSprop, cuda, (pt2) foreach: 0.002184295413030668
- resnet50, RMSprop, cuda, foreach: 0.0014343149600017568
- resnet50, NAdam, cuda, (pt2) default: 0.004760159255807213
- resnet50, NAdam, cuda, default: 0.004191039110000929
- resnet50, NAdam, cuda, (pt2) no_foreach: 0.0071270348965707195
- resnet50, NAdam, cuda, no_foreach: 0.018179516450004485
- resnet50, NAdam, cuda, (pt2) foreach: 0.004783486738114377
- resnet50, NAdam, cuda, foreach: 0.004538918690000174
- timm_vision_transformer, Adadelta, cuda, (pt2) default: 0.0030733869545284424
- timm_vision_transformer, Adadelta, cuda, default: 0.002095810079999865
- timm_vision_transformer, Adadelta, cuda, (pt2) no_foreach: 0.00273649170268269
- timm_vision_transformer, Adadelta, cuda, no_foreach: 0.013536372700014
- timm_vision_transformer, Adadelta, cuda, (pt2) foreach: 0.003114797384562105
- timm_vision_transformer, Adadelta, cuda, foreach: 0.002072904329997982
- timm_vision_transformer, Adagrad, cuda, (pt2) default: 0.008280653760029963
- timm_vision_transformer, Adagrad, cuda, default: 0.008008523300001494
- timm_vision_transformer, Adagrad, cuda, (pt2) no_foreach: 0.004150254714306579
- timm_vision_transformer, Adagrad, cuda, no_foreach: 0.007260814300002494
- timm_vision_transformer, Adagrad, cuda, (pt2) foreach: 0.008285542840003473
- timm_vision_transformer, Adagrad, cuda, foreach: 0.008737059333331368
- timm_vision_transformer, Adam, cuda, (pt2) default: 0.0033572912833430262
- timm_vision_transformer, Adam, cuda, default: 0.0030484830200020953
- timm_vision_transformer, Adam, cuda, (pt2) no_foreach: 0.004556660931836053
- timm_vision_transformer, Adam, cuda, no_foreach: 0.013667644150018532
- timm_vision_transformer, Adam, cuda, (pt2) foreach: 0.0033562027333042957
- timm_vision_transformer, Adam, cuda, foreach: 0.0031026997200024197
- timm_vision_transformer, Adam, cuda, (pt2) fused: 0.0019168562000058521
- timm_vision_transformer, Adam, cuda, fused: 0.0009117639666677254
- timm_vision_transformer, AdamW, cuda, (pt2) default: 0.003351730483313986
- timm_vision_transformer, AdamW, cuda, default: 0.0030866951000007247
- timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: 0.004440200652168088
- timm_vision_transformer, AdamW, cuda, no_foreach: 0.0149100374499767
- timm_vision_transformer, AdamW, cuda, (pt2) foreach: 0.0034155387966397243
- timm_vision_transformer, AdamW, cuda, foreach: 0.0031254458000012164
- timm_vision_transformer, AdamW, cuda, (pt2) fused: 0.0019105941143087485
- timm_vision_transformer, AdamW, cuda, fused: 0.0009408055366657208
- timm_vision_transformer, Adamax, cuda, (pt2) default: 0.012906030624947107
- timm_vision_transformer, Adamax, cuda, default: 0.013193479599999591
- timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: 0.01002705850005441
- timm_vision_transformer, Adamax, cuda, no_foreach: 0.01832815949999258
- timm_vision_transformer, Adamax, cuda, (pt2) foreach: 0.012895265625019192
- timm_vision_transformer, Adamax, cuda, foreach: 0.012246697599994149
- timm_vision_transformer, ASGD, cuda, (pt2) default: 0.00551293667562326
- timm_vision_transformer, ASGD, cuda, default: 0.005325746429998617
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: 0.005644493638909707
- timm_vision_transformer, ASGD, cuda, no_foreach: 0.009299729966657348
- timm_vision_transformer, ASGD, cuda, (pt2) foreach: 0.005676492861109889
- timm_vision_transformer, ASGD, cuda, foreach: 0.005198019029999159
- timm_vision_transformer, SGD, cuda, (pt2) default: 0.0009006766547138025
- timm_vision_transformer, SGD, cuda, default: 0.0005642057350000869
- timm_vision_transformer, SGD, cuda, (pt2) no_foreach: 0.0020984944687540974
- timm_vision_transformer, SGD, cuda, no_foreach: 0.0012372960350012364
- timm_vision_transformer, SGD, cuda, (pt2) foreach: 0.0008793127763155688
- timm_vision_transformer, SGD, cuda, foreach: 0.0005328243160001875
- timm_vision_transformer, RAdam, cuda, (pt2) default: 0.004897146690000227
- timm_vision_transformer, RAdam, cuda, default: 0.004822518390001278
- timm_vision_transformer, RAdam, cuda, (pt2) no_foreach: 0.02146951419999823
- timm_vision_transformer, RAdam, cuda, no_foreach: 0.021315015700020014
- timm_vision_transformer, RAdam, cuda, (pt2) foreach: 0.004880752059998486
- timm_vision_transformer, RAdam, cuda, foreach: 0.004795951490000334
- timm_vision_transformer, Rprop, cuda, (pt2) default: 0.021753818999968644
- timm_vision_transformer, Rprop, cuda, default: 0.020810174100006407
- timm_vision_transformer, Rprop, cuda, (pt2) no_foreach: 0.0030666991969858577
- timm_vision_transformer, Rprop, cuda, no_foreach: 0.029021585800001047
- timm_vision_transformer, Rprop, cuda, (pt2) foreach: 0.02153402359995198
- timm_vision_transformer, Rprop, cuda, foreach: 0.021178491000000577
- timm_vision_transformer, RMSprop, cuda, (pt2) default: 0.0019227700762236228
- timm_vision_transformer, RMSprop, cuda, default: 0.0013368141549972277
- timm_vision_transformer, RMSprop, cuda, (pt2) no_foreach: 0.002361906070564146
- timm_vision_transformer, RMSprop, cuda, no_foreach: 0.006728599029997895
- timm_vision_transformer, RMSprop, cuda, (pt2) foreach: 0.0019306905672963764
- timm_vision_transformer, RMSprop, cuda, foreach: 0.0013396049199991466
- timm_vision_transformer, NAdam, cuda, (pt2) default: 0.004349217086972045
- timm_vision_transformer, NAdam, cuda, default: 0.00394970664000084
- timm_vision_transformer, NAdam, cuda, (pt2) no_foreach: 0.006401454718698574
- timm_vision_transformer, NAdam, cuda, no_foreach: 0.015793144999997823
- timm_vision_transformer, NAdam, cuda, (pt2) foreach: 0.004454412333315203
- timm_vision_transformer, NAdam, cuda, foreach: 0.004006119720002061
- yolov3, Adadelta, cuda, (pt2) default: 0.007066389551725698
- yolov3, Adadelta, cuda, default: 0.005606049390003136
- yolov3, Adadelta, cuda, (pt2) no_foreach: 0.004193954854182873
- yolov3, Adadelta, cuda, no_foreach: 0.020772784100017816
- yolov3, Adadelta, cuda, (pt2) foreach: 0.006977468310369345
- yolov3, Adadelta, cuda, foreach: 0.005600217809997048
- yolov3, Adagrad, cuda, (pt2) default: 0.013397303400051897
- yolov3, Adagrad, cuda, default: 0.013286687600020742
- yolov3, Adagrad, cuda, (pt2) no_foreach: 0.006513576161289107
- yolov3, Adagrad, cuda, no_foreach: 0.011297070999989955
- yolov3, Adagrad, cuda, (pt2) foreach: 0.013349902799988437
- yolov3, Adagrad, cuda, foreach: 0.013049775700005739
- yolov3, Adam, cuda, (pt2) default: 0.006834508466681655
- yolov3, Adam, cuda, default: 0.004965095219999966
- yolov3, Adam, cuda, (pt2) no_foreach: 0.00713657431040325
- yolov3, Adam, cuda, no_foreach: 0.022176384500016865
- yolov3, Adam, cuda, (pt2) foreach: 0.007044630137944955
- yolov3, Adam, cuda, foreach: 0.005027222959997743
- yolov3, Adam, cuda, (pt2) fused: 0.0037329471296271986
- yolov3, Adam, cuda, fused: 0.002146876599999814
- yolov3, AdamW, cuda, (pt2) default: 0.0072385171428745055
- yolov3, AdamW, cuda, default: 0.005057759650003391
- yolov3, AdamW, cuda, (pt2) no_foreach: 0.007257156571394522
- yolov3, AdamW, cuda, no_foreach: 0.023741306200008694
- yolov3, AdamW, cuda, (pt2) foreach: 0.007262738607112494
- yolov3, AdamW, cuda, foreach: 0.0051244486500036145
- yolov3, AdamW, cuda, (pt2) fused: 0.0038248569244816915
- yolov3, AdamW, cuda, fused: 0.002214097609999044
- yolov3, Adamax, cuda, (pt2) default: 0.02305837322203135
- yolov3, Adamax, cuda, default: 0.02124359210001785
- yolov3, Adamax, cuda, (pt2) no_foreach: 0.017553693500076406
- yolov3, Adamax, cuda, no_foreach: 0.029659050999998725
- yolov3, Adamax, cuda, (pt2) foreach: 0.02323327900012373
- yolov3, Adamax, cuda, foreach: 0.02121420120001858
- yolov3, ASGD, cuda, (pt2) default: 0.008871535347917297
- yolov3, ASGD, cuda, default: 0.00833274840000134
- yolov3, ASGD, cuda, (pt2) no_foreach: 0.008552148374936527
- yolov3, ASGD, cuda, no_foreach: 0.01453947990003144
- yolov3, ASGD, cuda, (pt2) foreach: 0.00885337995667056
- yolov3, ASGD, cuda, foreach: 0.0083531942666923
- yolov3, SGD, cuda, (pt2) default: 0.0016813183109118982
- yolov3, SGD, cuda, default: 0.0009724291833360137
- yolov3, SGD, cuda, (pt2) no_foreach: 0.003173684484309547
- yolov3, SGD, cuda, no_foreach: 0.0020208630599972822
- yolov3, SGD, cuda, (pt2) foreach: 0.0016563235289110655
- yolov3, SGD, cuda, foreach: 0.0009187111566673897
- yolov3, RAdam, cuda, (pt2) default: 0.007696943539995118
- yolov3, RAdam, cuda, default: 0.0075823861400022
- yolov3, RAdam, cuda, (pt2) no_foreach: 0.033198082200033244
- yolov3, RAdam, cuda, no_foreach: 0.03280774080003539
- yolov3, RAdam, cuda, (pt2) foreach: 0.007733729110004788
- yolov3, RAdam, cuda, foreach: 0.007503650839998955
- yolov3, Rprop, cuda, (pt2) default: 0.03352489549994667
- yolov3, Rprop, cuda, default: 0.03283858019995023
- yolov3, Rprop, cuda, (pt2) no_foreach: 0.004564009977250614
- yolov3, Rprop, cuda, no_foreach: 0.04279789359998176
- yolov3, Rprop, cuda, (pt2) foreach: 0.03407738199969875
- yolov3, Rprop, cuda, foreach: 0.03299983960005193
- yolov3, RMSprop, cuda, (pt2) default: 0.004145512367358606
- yolov3, RMSprop, cuda, default: 0.0027024147500014804
- yolov3, RMSprop, cuda, (pt2) no_foreach: 0.0036820234908769966
- yolov3, RMSprop, cuda, no_foreach: 0.010888933949991042
- yolov3, RMSprop, cuda, (pt2) foreach: 0.00412749846936447
- yolov3, RMSprop, cuda, foreach: 0.0027061433200015017
- yolov3, NAdam, cuda, (pt2) default: 0.009057493826058699
- yolov3, NAdam, cuda, default: 0.00641724893999708
- yolov3, NAdam, cuda, no_foreach: 0.02438441670001339
- yolov3, NAdam, cuda, (pt2) foreach: 0.008881912347770476
- yolov3, NAdam, cuda, foreach: 0.006515105100006622
- yolov3, Adadelta, cpu, default: 0.3629307190003601
- yolov3, Adadelta, cpu, no_foreach: 0.3630052139997133
- yolov3, Adadelta, cpu, (pt2) foreach: 0.41201709100005246
- yolov3, Adadelta, cpu, foreach: 0.4428538590000244
- yolov3, Adagrad, cpu, default: 0.15644733099952646
- yolov3, Adagrad, cpu, no_foreach: 0.1612696369998048
- yolov3, Adagrad, cpu, (pt2) foreach: 0.49267349100045976
- yolov3, Adagrad, cpu, foreach: 0.4865847920000306
- yolov3, Adam, cpu, default: 0.29059941200011963
- yolov3, Adam, cpu, no_foreach: 0.2912993890004145
- yolov3, Adam, cpu, (pt2) foreach: 0.3161686070006908
- yolov3, Adam, cpu, foreach: 0.31547816900001635
- yolov3, AdamW, cpu, default: 0.31674174100044183
- yolov3, AdamW, cpu, no_foreach: 0.3172231260004992
- yolov3, AdamW, cpu, (pt2) foreach: 0.3386438019997513
- yolov3, AdamW, cpu, foreach: 0.3372226570008934
- yolov3, Adamax, cpu, default: 0.5204450080000242
- yolov3, Adamax, cpu, no_foreach: 0.49818044399944483
- yolov3, Adamax, cpu, (pt2) foreach: 0.8870704199998727
- yolov3, Adamax, cpu, foreach: 0.892644293999183
- yolov3, ASGD, cpu, default: 0.11624405449992992
- yolov3, ASGD, cpu, no_foreach: 0.11504810000042198
- yolov3, ASGD, cpu, (pt2) foreach: 0.11884203099998558
- yolov3, ASGD, cpu, foreach: 0.1177751359996364
- yolov3, SGD, cpu, (pt2) default: 0.04171192419998988
- yolov3, SGD, cpu, default: 0.04331184149996261
- yolov3, SGD, cpu, (pt2) no_foreach: 0.04231681220007886
- yolov3, SGD, cpu, no_foreach: 0.0452383142999679
- yolov3, SGD, cpu, (pt2) foreach: 0.044279348599775406
- yolov3, SGD, cpu, foreach: 0.04355653459997484
- yolov3, RAdam, cpu, (pt2) default: 0.25475907399959397
- yolov3, RAdam, cpu, default: 0.2556008110004768
- yolov3, RAdam, cpu, (pt2) no_foreach: 0.2571922090000953
- yolov3, RAdam, cpu, no_foreach: 0.2560090510005466
- yolov3, RAdam, cpu, (pt2) foreach: 0.41218263900009333
- yolov3, RAdam, cpu, foreach: 0.408470987000328
- yolov3, Rprop, cpu, default: 0.7807191330002752
- yolov3, Rprop, cpu, no_foreach: 0.7857798269997147
- yolov3, Rprop, cpu, (pt2) foreach: 0.8130313279998518
- yolov3, Rprop, cpu, foreach: 0.8051206310001362
- yolov3, RMSprop, cpu, default: 0.17241126699991582
- yolov3, RMSprop, cpu, no_foreach: 0.1747381754998969
- yolov3, RMSprop, cpu, (pt2) foreach: 0.20387007600038487
- yolov3, RMSprop, cpu, foreach: 0.20139784749972023
- yolov3, NAdam, cpu, default: 0.3549834289997307
- yolov3, NAdam, cpu, no_foreach: 0.34434368799975346
- yolov3, NAdam, cpu, (pt2) foreach: 0.3792224390008414
- yolov3, NAdam, cpu, foreach: 0.3775586869996914


cc @janeyx99


Teehee this is a test issue that shows my PR worked!