Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'f89c73be3a3e8274d025ac46a33a780853841c9e'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected Tests:
- DALLE2_pytorch, AdamW, cuda, amsgrad, maximize: +30.52377%
- DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +36.31309%


Tests that were no longer run on affected commit:
- nvidia_deeprecommender, Adadelta, cuda, default: 0.01745093901408836
- nvidia_deeprecommender, Adadelta, cuda, maximize: 0.018761409039143473
- nvidia_deeprecommender, Adadelta, cuda, no_foreach: 0.016440155950840564
- nvidia_deeprecommender, Adadelta, cuda, differentiable: 0.017713632003869863
- nvidia_deeprecommender, Adadelta, cuda, foreach: 0.01744002734776586
- nvidia_deeprecommender, Adagrad, cuda, default: 0.008352283008086185
- nvidia_deeprecommender, Adagrad, cuda, maximize: 0.009671672930320103
- nvidia_deeprecommender, Adagrad, cuda, no_foreach: 0.006650899981614202
- nvidia_deeprecommender, Adagrad, cuda, differentiable: 0.006638368340209127
- nvidia_deeprecommender, Adagrad, cuda, foreach: 0.008338176100514828
- nvidia_deeprecommender, Adam, cuda, default: 0.01188443680293858
- nvidia_deeprecommender, Adam, cuda, amsgrad, maximize: 0.014804353110957891
- nvidia_deeprecommender, Adam, cuda, no_foreach: 0.01092154704965651
- nvidia_deeprecommender, Adam, cuda, differentiable: 0.010922222945373504
- nvidia_deeprecommender, Adam, cuda, foreach: 0.011612737202085555
- nvidia_deeprecommender, Adam, cuda, foreach, maximize, capturable: 0.014103882305789739
- nvidia_deeprecommender, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.01600768860662356
- nvidia_deeprecommender, Adam, cuda, fused: 0.005278976678382606
- nvidia_deeprecommender, Adam, cuda, fused, amsgrad, maximize: 0.006058671961072832
- nvidia_deeprecommender, Adam, cuda, fused, capturable: 0.005267991679720581
- nvidia_deeprecommender, Adam, cuda, fused, capturable, amsgrad: 0.006088066308293492
- nvidia_deeprecommender, AdamW, cuda, default: 0.012918454047758132
- nvidia_deeprecommender, AdamW, cuda, amsgrad, maximize: 0.016120433202013375
- nvidia_deeprecommender, AdamW, cuda, no_foreach: 0.012135507445782422
- nvidia_deeprecommender, AdamW, cuda, differentiable: 0.01213691228767857
- nvidia_deeprecommender, AdamW, cuda, foreach: 0.012914810492657126
- nvidia_deeprecommender, AdamW, cuda, foreach, maximize, capturable: 0.01541902874596417
- nvidia_deeprecommender, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.017333652242086828
- nvidia_deeprecommender, AdamW, cuda, fused: 0.005408490300178528
- nvidia_deeprecommender, AdamW, cuda, fused, amsgrad, maximize: 0.006143090170808136
- nvidia_deeprecommender, AdamW, cuda, fused, capturable: 0.005414818720892072
- nvidia_deeprecommender, AdamW, cuda, fused, capturable, amsgrad: 0.006118739729281515
- nvidia_deeprecommender, Adamax, cuda, default: 0.01448306064121425
- nvidia_deeprecommender, Adamax, cuda, maximize: 0.015779947035480288
- nvidia_deeprecommender, Adamax, cuda, no_foreach: 0.012202140351291746
- nvidia_deeprecommender, Adamax, cuda, differentiable: 0.01346922239754349
- nvidia_deeprecommender, Adamax, cuda, foreach: 0.014492701448034495
- nvidia_deeprecommender, ASGD, cuda, default: 0.004502540768589825
- nvidia_deeprecommender, ASGD, cuda, maximize: 0.005792599909473211
- nvidia_deeprecommender, ASGD, cuda, no_foreach: 0.004321238710545004
- nvidia_deeprecommender, ASGD, cuda, differentiable: 0.0043213865486904975
- nvidia_deeprecommender, ASGD, cuda, foreach: 0.0045003504818305375
- nvidia_deeprecommender, SGD, cuda, default: 0.0019237174990121275
- nvidia_deeprecommender, SGD, cuda, maximize: 0.003216073198709637
- nvidia_deeprecommender, SGD, cuda, no_foreach: 0.0018395552947185934
- nvidia_deeprecommender, SGD, cuda, differentiable: 0.001924230760196224
- nvidia_deeprecommender, SGD, cuda, foreach: 0.0019199880352243782
- nvidia_deeprecommender, SGD, cuda, foreach, momentum=0.9, nesterov: 0.007059172401204706
- nvidia_deeprecommender, SGD, cuda, foreach, momentum=0.9: 0.005143785830587149
- nvidia_deeprecommender, RAdam, cuda, default: 0.014239443140104412
- nvidia_deeprecommender, RAdam, cuda, no_foreach: 0.017014512105379256
- nvidia_deeprecommender, RAdam, cuda, differentiable: 0.016997190902475267
- nvidia_deeprecommender, RAdam, cuda, foreach: 0.01424026385648176
- nvidia_deeprecommender, Rprop, cuda, default: 0.01989994279574603
- nvidia_deeprecommender, Rprop, cuda, maximize: 0.02241833701264113
- nvidia_deeprecommender, Rprop, cuda, no_foreach: 0.02075554239563644
- nvidia_deeprecommender, Rprop, cuda, differentiable: 0.02200894649140537
- nvidia_deeprecommender, Rprop, cuda, foreach: 0.019895165448542685
- nvidia_deeprecommender, RMSprop, cuda, default: 0.008349503964806597
- nvidia_deeprecommender, RMSprop, cuda, maximize: 0.009653881766522925
- nvidia_deeprecommender, RMSprop, cuda, no_foreach: 0.007874869361209373
- nvidia_deeprecommender, RMSprop, cuda, differentiable: 0.007862158197288712
- nvidia_deeprecommender, RMSprop, cuda, foreach: 0.00835210046110054
- nvidia_deeprecommender, NAdam, cuda, default: 0.015442586748395114
- nvidia_deeprecommender, NAdam, cuda, no_foreach: 0.014511539705563337
- nvidia_deeprecommender, NAdam, cuda, differentiable: 0.01690885954303667
- nvidia_deeprecommender, NAdam, cuda, foreach: 0.015444840944837779
- attention_is_all_you_need_pytorch, Adadelta, cuda, default: 0.005173015119507909
- attention_is_all_you_need_pytorch, Adadelta, cuda, maximize: 0.005510958097875118
- attention_is_all_you_need_pytorch, Adadelta, cuda, no_foreach: 0.016932335193268956
- attention_is_all_you_need_pytorch, Adadelta, cuda, differentiable: 0.02011216514511034
- attention_is_all_you_need_pytorch, Adadelta, cuda, foreach: 0.005120210240129382
- attention_is_all_you_need_pytorch, Adagrad, cuda, default: 0.0033538520010188222
- attention_is_all_you_need_pytorch, Adagrad, cuda, maximize: 0.003431912309024483
- attention_is_all_you_need_pytorch, Adagrad, cuda, no_foreach: 0.00904447006372114
- attention_is_all_you_need_pytorch, Adagrad, cuda, differentiable: 0.009751387763147552
- attention_is_all_you_need_pytorch, Adagrad, cuda, foreach: 0.0032049424690194428
- attention_is_all_you_need_pytorch, Adam, cuda, default: 0.0034980594995431603
- attention_is_all_you_need_pytorch, Adam, cuda, amsgrad, maximize: 0.00436450184090063
- attention_is_all_you_need_pytorch, Adam, cuda, no_foreach: 0.016219678102061152
- attention_is_all_you_need_pytorch, Adam, cuda, differentiable: 0.030747037497349083
- attention_is_all_you_need_pytorch, Adam, cuda, foreach: 0.00341347805922851
- attention_is_all_you_need_pytorch, Adam, cuda, foreach, maximize, capturable: 0.005790465329773724
- attention_is_all_you_need_pytorch, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.006105316770263016
- attention_is_all_you_need_pytorch, Adam, cuda, fused: 0.0020594131015241146
- attention_is_all_you_need_pytorch, Adam, cuda, fused, amsgrad, maximize: 0.0021297453599981963
- attention_is_all_you_need_pytorch, Adam, cuda, fused, capturable: 0.0020610262383706866
- attention_is_all_you_need_pytorch, Adam, cuda, fused, capturable, amsgrad: 0.0021332465298473835
- attention_is_all_you_need_pytorch, AdamW, cuda, default: 0.0038053672504611315
- attention_is_all_you_need_pytorch, AdamW, cuda, amsgrad, maximize: 0.004756716948468238
- attention_is_all_you_need_pytorch, AdamW, cuda, no_foreach: 0.01758569690864533
- attention_is_all_you_need_pytorch, AdamW, cuda, differentiable: 0.03288948810659349
- attention_is_all_you_need_pytorch, AdamW, cuda, foreach: 0.003805448510684073
- attention_is_all_you_need_pytorch, AdamW, cuda, foreach, maximize, capturable: 0.005941328241024166
- attention_is_all_you_need_pytorch, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.006589848708827049
- attention_is_all_you_need_pytorch, AdamW, cuda, fused: 0.0021288999705575407
- attention_is_all_you_need_pytorch, AdamW, cuda, fused, amsgrad, maximize: 0.002166094200219959
- attention_is_all_you_need_pytorch, AdamW, cuda, fused, capturable: 0.002124761890154332
- attention_is_all_you_need_pytorch, AdamW, cuda, fused, capturable, amsgrad: 0.0021662067784927784
- attention_is_all_you_need_pytorch, Adamax, cuda, default: 0.015167566703166813
- attention_is_all_you_need_pytorch, Adamax, cuda, maximize: 0.01559729449218139
- attention_is_all_you_need_pytorch, Adamax, cuda, no_foreach: 0.019956838351208716
- attention_is_all_you_need_pytorch, Adamax, cuda, differentiable: 0.02448316749650985
- attention_is_all_you_need_pytorch, Adamax, cuda, foreach: 0.014997324149589986
- attention_is_all_you_need_pytorch, ASGD, cuda, default: 0.006881456370465457
- attention_is_all_you_need_pytorch, ASGD, cuda, maximize: 0.007435156120918691
- attention_is_all_you_need_pytorch, ASGD, cuda, no_foreach: 0.011767951049841942
- attention_is_all_you_need_pytorch, ASGD, cuda, differentiable: 0.011871023045387119
- attention_is_all_you_need_pytorch, ASGD, cuda, foreach: 0.006709828758612275
- attention_is_all_you_need_pytorch, SGD, cuda, default: 0.0007098730779252947
- attention_is_all_you_need_pytorch, SGD, cuda, maximize: 0.0013008749566506593
- attention_is_all_you_need_pytorch, SGD, cuda, no_foreach: 0.0016376342496369034
- attention_is_all_you_need_pytorch, SGD, cuda, differentiable: 0.0007137386200483888
- attention_is_all_you_need_pytorch, SGD, cuda, foreach: 0.0005539580779150129
- attention_is_all_you_need_pytorch, SGD, cuda, foreach, momentum=0.9, nesterov: 0.002049780657980591
- attention_is_all_you_need_pytorch, SGD, cuda, foreach, momentum=0.9: 0.0015018489549402146
- attention_is_all_you_need_pytorch, RAdam, cuda, default: 0.004210930648259819
- attention_is_all_you_need_pytorch, RAdam, cuda, no_foreach: 0.0253820100100711
- attention_is_all_you_need_pytorch, RAdam, cuda, differentiable: 0.026187645783647896
- attention_is_all_you_need_pytorch, RAdam, cuda, foreach: 0.004210075340233743
- attention_is_all_you_need_pytorch, Rprop, cuda, default: 0.0216818236047402
- attention_is_all_you_need_pytorch, Rprop, cuda, maximize: 0.024044667091220617
- attention_is_all_you_need_pytorch, Rprop, cuda, no_foreach: 0.03276849370449782
- attention_is_all_you_need_pytorch, Rprop, cuda, differentiable: 0.035078099183738234
- attention_is_all_you_need_pytorch, Rprop, cuda, foreach: 0.021425667917355895
- attention_is_all_you_need_pytorch, RMSprop, cuda, default: 0.0024511136999353766
- attention_is_all_you_need_pytorch, RMSprop, cuda, maximize: 0.0028292648610658945
- attention_is_all_you_need_pytorch, RMSprop, cuda, no_foreach: 0.0085133501716579
- attention_is_all_you_need_pytorch, RMSprop, cuda, differentiable: 0.009111827301482361
- attention_is_all_you_need_pytorch, RMSprop, cuda, foreach: 0.002449127449654043
- attention_is_all_you_need_pytorch, NAdam, cuda, default: 0.0049076829012483355
- attention_is_all_you_need_pytorch, NAdam, cuda, no_foreach: 0.020419197413139044
- attention_is_all_you_need_pytorch, NAdam, cuda, differentiable: 0.034456249605864284
- attention_is_all_you_need_pytorch, NAdam, cuda, foreach: 0.004737464259378612
- pytorch_struct, Adadelta, cuda, default: 0.00043349122395738957
- pytorch_struct, Adadelta, cuda, maximize: 0.000501799687044695
- pytorch_struct, Adadelta, cuda, no_foreach: 0.0023117761011235416
- pytorch_struct, Adadelta, cuda, differentiable: 0.002718108110129833
- pytorch_struct, Adadelta, cuda, foreach: 0.00039541194681078193
- pytorch_struct, Adagrad, cuda, default: 0.0005487439460121096
- pytorch_struct, Adagrad, cuda, maximize: 0.0005619873271789402
- pytorch_struct, Adagrad, cuda, no_foreach: 0.001275284484727308
- pytorch_struct, Adagrad, cuda, differentiable: 0.0013490179192740472
- pytorch_struct, Adagrad, cuda, foreach: 0.0005171015439555049
- pytorch_struct, Adam, cuda, default: 0.0005993680220562964
- pytorch_struct, Adam, cuda, amsgrad, maximize: 0.0006986413577881952
- pytorch_struct, Adam, cuda, no_foreach: 0.0021685667405836285
- pytorch_struct, Adam, cuda, differentiable: 0.004147054250352084
- pytorch_struct, Adam, cuda, foreach: 0.0005800098499748855
- pytorch_struct, Adam, cuda, foreach, maximize, capturable: 0.0009619580333431561
- pytorch_struct, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0009822770875568191
- pytorch_struct, Adam, cuda, fused: 0.00017320067388936877
- pytorch_struct, Adam, cuda, fused, amsgrad, maximize: 0.000178504669922404
- pytorch_struct, Adam, cuda, fused, capturable: 0.00017504276242107152
- pytorch_struct, Adam, cuda, fused, capturable, amsgrad: 0.0001784943254897371
- pytorch_struct, AdamW, cuda, default: 0.000616413104813546
- pytorch_struct, AdamW, cuda, amsgrad, maximize: 0.0007284312550909817
- pytorch_struct, AdamW, cuda, no_foreach: 0.0023627216811291873
- pytorch_struct, AdamW, cuda, differentiable: 0.004337908171582967
- pytorch_struct, AdamW, cuda, foreach: 0.0005934682500083
- pytorch_struct, AdamW, cuda, foreach, maximize, capturable: 0.0009670432672525446
- pytorch_struct, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.000994558793803056
- pytorch_struct, AdamW, cuda, fused: 0.00017993173841387034
- pytorch_struct, AdamW, cuda, fused, amsgrad, maximize: 0.00018480480602011085
- pytorch_struct, AdamW, cuda, fused, capturable: 0.00017997459287289532
- pytorch_struct, AdamW, cuda, fused, capturable, amsgrad: 0.00018484839750453828
- pytorch_struct, Adamax, cuda, default: 0.002187983151525259
- pytorch_struct, Adamax, cuda, maximize: 0.002339574780780822
- pytorch_struct, Adamax, cuda, no_foreach: 0.0027562174783088266
- pytorch_struct, Adamax, cuda, differentiable: 0.003307119009550661
- pytorch_struct, Adamax, cuda, foreach: 0.0021475877007469534
- pytorch_struct, ASGD, cuda, default: 0.0010021736146882177
- pytorch_struct, ASGD, cuda, maximize: 0.0010986573598347604
- pytorch_struct, ASGD, cuda, no_foreach: 0.0015776543144602328
- pytorch_struct, ASGD, cuda, differentiable: 0.0016160876594949513
- pytorch_struct, ASGD, cuda, foreach: 0.0009725843323394656
- pytorch_struct, SGD, cuda, default: 0.000135497689479962
- pytorch_struct, SGD, cuda, maximize: 0.00021547755203209817
- pytorch_struct, SGD, cuda, no_foreach: 0.00023205082793720065
- pytorch_struct, SGD, cuda, differentiable: 0.00013365930609870704
- pytorch_struct, SGD, cuda, foreach: 0.00010584530956111848
- pytorch_struct, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0001603530809516087
- pytorch_struct, SGD, cuda, foreach, momentum=0.9: 0.00014436074905097485
- pytorch_struct, RAdam, cuda, default: 0.0006760093318298459
- pytorch_struct, RAdam, cuda, no_foreach: 0.00348755155922845
- pytorch_struct, RAdam, cuda, differentiable: 0.003594823449384421
- pytorch_struct, RAdam, cuda, foreach: 0.0006434878620784729
- pytorch_struct, Rprop, cuda, default: 0.0030286261602304877
- pytorch_struct, Rprop, cuda, maximize: 0.0033651637402363124
- pytorch_struct, Rprop, cuda, no_foreach: 0.004453612500801682
- pytorch_struct, Rprop, cuda, differentiable: 0.004788550909142941
- pytorch_struct, Rprop, cuda, foreach: 0.0029935695603489874
- pytorch_struct, RMSprop, cuda, default: 0.0003063608449883759
- pytorch_struct, RMSprop, cuda, maximize: 0.00038125641387887297
- pytorch_struct, RMSprop, cuda, no_foreach: 0.0011589198000729083
- pytorch_struct, RMSprop, cuda, differentiable: 0.0012700973346363754
- pytorch_struct, RMSprop, cuda, foreach: 0.00027423402713611724
- pytorch_struct, NAdam, cuda, default: 0.0007951281336136162
- pytorch_struct, NAdam, cuda, no_foreach: 0.002777596958912909
- pytorch_struct, NAdam, cuda, differentiable: 0.004571745279245079
- pytorch_struct, NAdam, cuda, foreach: 0.0007493818278890103


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5886617956

cc @janeyx99


This should be fixed