Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '981d4c2578d10d8a96d173471802fc2812541fb1'
Root cause: Unimplemented Feature Gap


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 871fc7bb76f05c3c487214404f687cf7a6a8e453

Affected PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected Tests:
- BERT_pytorch, Adam, cuda, (pt2) differentiable: +21.69033%
- BERT_pytorch, Adamax, cuda, (pt2) default: -22.13005%
- BERT_pytorch, Adamax, cuda, foreach: +27.75430%
- BERT_pytorch, ASGD, cuda, default: +51.55750%
- basic_gnn_gcn, Adam, cuda, (pt2) amsgrad, maximize: +12.66619%
- basic_gnn_gcn, ASGD, cuda, default: +38.07286%
- basic_gnn_gcn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +10.15557%
- basic_gnn_gcn, RMSprop, cuda, default: +20.02033%
- basic_gnn_gcn, NAdam, cuda, (pt2) default: +13.78051%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +409.87643%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9: +149.43584%
- timm_resnest, ASGD, cuda, (pt2) default: -15.02599%
- timm_resnest, Rprop, cuda, (pt2) maximize: +19.77447%
- moco, Adam, cuda, (pt2) foreach, maximize, capturable: +17.90010%
- moco, AdamW, cuda, (pt2) foreach, maximize, capturable: +10.90383%
- moco, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +22.37379%
- moco, Rprop, cuda, maximize: -17.77188%
- moco, Rprop, cuda, (pt2) foreach: +20.16803%
- moco, NAdam, cuda, (pt2) foreach: +10.72376%
- timm_vision_transformer, Adam, cuda, foreach, maximize, capturable, amsgrad: +12.07377%
- timm_vision_transformer, AdamW, cuda, differentiable: +11.49212%
- timm_vision_transformer, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +15.83428%
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: +12.25932%
- timm_vision_transformer, ASGD, cuda, (pt2) differentiable: +16.18824%
- timm_vision_transformer, SGD, cuda, no_foreach: -11.53384%
- yolov3, Adam, cuda, foreach, maximize, capturable, amsgrad: -14.98050%
- yolov3, ASGD, cuda, differentiable: -21.73505%
- yolov3, ASGD, cuda, foreach: -25.09982%
- yolov3, SGD, cuda, (pt2) no_foreach: -11.77061%
- yolov3, RAdam, cuda, no_foreach: -28.96947%
- yolov3, Rprop, cuda, no_foreach: -19.06485%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +12.59348%
- hf_T5, Rprop, cuda, default: +14.15496%
- hf_Albert, Adamax, cuda, (pt2) default: +10.39788%
- hf_Albert, ASGD, cuda, (pt2) default: -12.54815%
- resnet50, Adam, cuda, (pt2) amsgrad, maximize: +10.87688%
- resnet50, Adam, cuda, (pt2) foreach, maximize, capturable: +10.87947%
- resnet50, AdamW, cuda, no_foreach: +30.65357%
- resnet50, NAdam, cuda, no_foreach: -11.33855%
- attention_is_all_you_need_pytorch, Adagrad, cuda, (pt2) default: +41.80306%
- attention_is_all_you_need_pytorch, Adagrad, cuda, (pt2) maximize: +11.05780%
- attention_is_all_you_need_pytorch, ASGD, cuda, default: +12.20633%
- phlippe_resnet, Rprop, cuda, foreach: +15.38462%
- hf_DistilBert, Rprop, cuda, default: +40.61273%
- maml, NAdam, cuda, (pt2) default: -15.92943%
- resnext50_32x4d, Adam, cuda, default: +13.60721%
- tts_angular, AdamW, cuda, foreach, maximize, capturable: +14.14655%
- tts_angular, Adamax, cuda, (pt2) default: -10.48688%
- tts_angular, ASGD, cuda, default: -21.24919%
- tts_angular, ASGD, cuda, (pt2) foreach: -23.78262%
- tts_angular, Rprop, cuda, (pt2) default: +18.23624%
- hf_BigBird, ASGD, cuda, (pt2) maximize: +10.57055%
- doctr_det_predictor, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +20.79829%
- pytorch_struct, Adam, cuda, (pt2) default: +16.52394%
- pytorch_struct, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -13.81840%
- pytorch_struct, ASGD, cuda, default: +11.12090%
- basic_gnn_gin, Adagrad, cuda, (pt2) default: +10.29865%
- basic_gnn_gin, Adamax, cuda, maximize: +12.78327%
- basic_gnn_gin, Adamax, cuda, foreach: +11.43978%
- basic_gnn_gin, Rprop, cuda, (pt2) default: +11.16851%
- demucs, SGD, cuda, (pt2) foreach, momentum=0.9: +23.18819%
- resnet152, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +14.53919%
- resnet152, ASGD, cuda, (pt2) default: +11.00256%
- detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, (pt2) maximize: +41.31947%
- timm_efficientdet, Adam, cuda, (pt2) amsgrad, maximize: +14.99190%
- timm_efficientdet, Adam, cuda, amsgrad, maximize: +10.29101%
- timm_efficientdet, Adam, cuda, (pt2) foreach, maximize, capturable: +15.47636%
- timm_efficientdet, Adam, cuda, foreach, maximize, capturable, amsgrad: +16.41972%
- timm_efficientdet, AdamW, cuda, (pt2) amsgrad, maximize: +10.25701%
- timm_efficientdet, AdamW, cuda, (pt2) foreach: +10.57276%
- timm_efficientdet, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +16.38967%
- timm_efficientdet, Adamax, cuda, maximize: +19.56140%
- timm_efficientdet, ASGD, cuda, maximize: +11.03710%
- timm_efficientdet, NAdam, cuda, (pt2) foreach: +12.84483%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, maximize: +14.34199%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, foreach: +11.64014%
- timm_vovnet, Adagrad, cuda, (pt2) maximize: -16.52880%
- timm_vovnet, ASGD, cuda, (pt2) maximize: -19.91515%
- pytorch_stargan, SGD, cuda, (pt2) foreach, momentum=0.9: +11.74738%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -18.22559%
- phlippe_densenet, ASGD, cuda, foreach: -12.93930%
- fambench_xlmr, Rprop, cuda, maximize: -14.76695%
- squeezenet1_1, Adam, cuda, (pt2) default: -10.33984%
- squeezenet1_1, AdamW, cuda, (pt2) foreach, maximize, capturable: -15.06394%
- squeezenet1_1, AdamW, cuda, foreach, maximize, capturable, amsgrad: -18.76172%
- squeezenet1_1, SGD, cuda, (pt2) foreach, momentum=0.9: +11.12846%
- Super_SloMo, Adamax, cuda, default: +18.19838%
- mobilenet_v2_quantized_qat, Adagrad, cuda, (pt2) maximize: -12.11791%
- shufflenet_v2_x1_0, Adagrad, cuda, (pt2) default: +12.38311%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +71.57118%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9: +41.94932%
- hf_GPT2, Adamax, cuda, default: +16.19808%
- dcgan, Adagrad, cuda, default: +10.18182%
- basic_gnn_edgecnn, Adadelta, cuda, default: -17.70717%
- basic_gnn_edgecnn, ASGD, cuda, (pt2) default: +11.46643%
- basic_gnn_edgecnn, ASGD, cuda, maximize: +19.02736%
- fastNLP_Bert, Adagrad, cuda, (pt2) foreach: +19.82932%
- fastNLP_Bert, ASGD, cuda, maximize: +10.90793%
- vision_maskrcnn, ASGD, cuda, maximize: +24.78923%
- mnasnet1_0, AdamW, cuda, (pt2) foreach, maximize, capturable: +12.77249%
- opacus_cifar10, ASGD, cuda, maximize: -13.63788%
- opacus_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +19.41780%
- mobilenet_v3_large, ASGD, cuda, foreach: -12.14421%
- timm_regnet, Adagrad, cuda, foreach: +13.34642%
- hf_T5_large, Adagrad, cuda, no_foreach: -11.91637%
- hf_T5_large, AdamW, cuda, differentiable: +13.02915%
- hf_T5_large, RAdam, cuda, (pt2) differentiable: -37.92844%
- densenet121, Adagrad, cuda, (pt2) default: -27.20356%
- densenet121, Adam, cuda, (pt2) default: -14.88187%
- densenet121, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -20.13763%
- densenet121, Adam, cuda, foreach, maximize, capturable, amsgrad: -10.33239%
- densenet121, Adamax, cuda, (pt2) foreach: -13.91150%
- densenet121, Rprop, cuda, (pt2) maximize: -14.44242%
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +24.55167%
- resnet18, RAdam, cuda, (pt2) foreach: +12.53583%
- resnet18, Rprop, cuda, (pt2) default: -12.44119%
- DALLE2_pytorch, Adagrad, cuda, (pt2) maximize: -16.08898%
- DALLE2_pytorch, Adam, cuda, foreach: +20.13436%
- DALLE2_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable: +11.09930%
- DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable: +12.83904%
- DALLE2_pytorch, AdamW, cuda, default: +11.33147%
- DALLE2_pytorch, AdamW, cuda, foreach: +12.95092%
- DALLE2_pytorch, AdamW, cuda, foreach, maximize, capturable, amsgrad: +12.26479%
- DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: -28.60510%
- DALLE2_pytorch, ASGD, cuda, (pt2) default: -15.29780%
- DALLE2_pytorch, ASGD, cuda, default: -20.05703%
- DALLE2_pytorch, ASGD, cuda, maximize: -28.86709%
- DALLE2_pytorch, RAdam, cuda, default: -11.31502%
- DALLE2_pytorch, Rprop, cuda, default: +23.20192%
- DALLE2_pytorch, Rprop, cuda, maximize: +19.68210%
- functorch_maml_omniglot, Rprop, cuda, maximize: +13.21961%
- basic_gnn_sage, Adadelta, cuda, maximize: +18.61135%
- basic_gnn_sage, AdamW, cuda, foreach, maximize, capturable, amsgrad: +11.50857%
- basic_gnn_sage, AdamW, cuda, fused, amsgrad, maximize: +16.97784%
- basic_gnn_sage, ASGD, cuda, default: +19.69918%
- basic_gnn_sage, Rprop, cuda, default: -11.31805%
- basic_gnn_sage, Rprop, cuda, foreach: +25.47171%
- basic_gnn_sage, RMSprop, cuda, maximize: +11.04688%
- doctr_reco_predictor, Rprop, cuda, (pt2) default: -10.62767%
- LearningToPaint, Adam, cuda, foreach, maximize, capturable, amsgrad: +10.75858%
- LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -11.72063%
- LearningToPaint, Rprop, cuda, default: +36.56857%
- detectron2_fasterrcnn_r_101_c4, Adamax, cuda, (pt2) maximize: +17.81302%
- detectron2_fasterrcnn_r_101_c4, ASGD, cuda, foreach: -26.84518%
- Background_Matting, AdamW, cuda, (pt2) foreach, maximize, capturable: +10.04748%
- Background_Matting, RAdam, cuda, (pt2) default: +20.52269%
- maml_omniglot, Adam, cuda, foreach, maximize, capturable, amsgrad: -14.75210%
- drq, NAdam, cuda, default: +11.00263%
- resnet50_quantized_qat, ASGD, cuda, maximize: -16.90590%
- detectron2_maskrcnn_r_101_fpn, Adagrad, cuda, default: +39.71326%
- detectron2_maskrcnn_r_101_fpn, Rprop, cuda, (pt2) maximize: -14.24839%
- hf_Bert_large, Rprop, cuda, (pt2) default: +10.68786%
- timm_nfnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +11.85689%
- tacotron2, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -10.51143%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- sam, Adadelta, cpu, (pt2) foreach: 4.005513823591173
- sam, Adadelta, cpu, foreach: 4.050980799365789
- sam, Adagrad, cpu, (pt2) foreach: 2.7941528619267046
- sam, Adagrad, cpu, foreach: 2.8002054300159216
- sam, Adam, cpu, (pt2) foreach: 3.0794076467864215
- sam, Adam, cpu, foreach: 3.0970330317504704
- sam, AdamW, cpu, (pt2) foreach: 3.314660478848964
- sam, AdamW, cpu, foreach: 3.338030522223562
- sam, Adamax, cpu, (pt2) foreach: 6.724740885663778
- sam, Adamax, cpu, foreach: 6.647923167794943
- sam, ASGD, cpu, (pt2) foreach: 1.1005898648872972
- sam, ASGD, cpu, foreach: 1.1407688208855689
- sam, SGD, cpu, (pt2) foreach: 0.43704793974757195
- sam, SGD, cpu, foreach: 0.41818635119125247
- sam, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: 3.0039971582591534
- sam, SGD, cpu, foreach, momentum=0.9, nesterov: 1.5139519949443638
- sam, SGD, cpu, (pt2) foreach, momentum=0.9: 2.56168697681278
- sam, SGD, cpu, foreach, momentum=0.9: 1.078660893253982
- sam, RAdam, cpu, (pt2) foreach: 3.9494041218422353
- sam, RAdam, cpu, foreach: 3.9667405588552356
- sam, Rprop, cpu, (pt2) foreach: 8.040245674084872
- sam, Rprop, cpu, foreach: 8.004174078814685
- sam, RMSprop, cpu, (pt2) foreach: 1.9686353588476777
- sam, RMSprop, cpu, foreach: 1.9972618967294693
- sam, NAdam, cpu, (pt2) foreach: 3.6810210258699954
- sam, NAdam, cpu, foreach: 3.660273721907288
- sam, Adadelta, cuda, (pt2) default: 0.05888357863295823
- sam, Adadelta, cuda, default: 0.05542351040057838
- sam, Adadelta, cuda, (pt2) maximize: 0.06422379286959767
- sam, Adadelta, cuda, maximize: 0.059605956776067615
- sam, Adadelta, cuda, (pt2) foreach: 0.05877963046077639
- sam, Adadelta, cuda, foreach: 0.05539705241099
- sam, Adagrad, cuda, (pt2) default: 0.054126784787513316
- sam, Adagrad, cuda, default: 0.03586018178611994
- sam, Adagrad, cuda, (pt2) maximize: 0.05520518939010799
- sam, Adagrad, cuda, maximize: 0.03785021468065679
- sam, Adagrad, cuda, (pt2) foreach: 0.05371510225813836
- sam, Adagrad, cuda, foreach: 0.03858707281760872
- sam, Adam, cuda, (pt2) default: 0.04817295130342245
- sam, Adam, cuda, default: 0.04136993158608675
- sam, Adam, cuda, (pt2) amsgrad, maximize: 0.05666171375196427
- sam, Adam, cuda, amsgrad, maximize: 0.05144104510545731
- sam, Adam, cuda, (pt2) foreach: 0.04801015080884099
- sam, Adam, cuda, foreach: 0.04143497901968658
- sam, Adam, cuda, (pt2) foreach, maximize, capturable: 0.05390442826319486
- sam, Adam, cuda, foreach, maximize, capturable: 0.04968732330016792
- sam, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.06104071869049221
- sam, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.06095919548533857
- sam, Adam, cuda, (pt2) fused: 0.022032301127910613
- sam, Adam, cuda, fused: 0.017975162458606066
- sam, Adam, cuda, (pt2) fused, amsgrad, maximize: 0.024884938521103725
- sam, Adam, cuda, fused, amsgrad, maximize: 0.02012060289271176
- sam, Adam, cuda, (pt2) fused, capturable: 0.021581195667386054
- sam, Adam, cuda, fused, capturable: 0.017977170390076937
- sam, Adam, cuda, (pt2) fused, capturable, amsgrad: 0.02416496930850877
- sam, Adam, cuda, fused, capturable, amsgrad: 0.020128069818019866
- sam, AdamW, cuda, (pt2) default: 0.052489361609332263
- sam, AdamW, cuda, default: 0.04561384436674416
- sam, AdamW, cuda, (pt2) amsgrad, maximize: 0.060752661316655576
- sam, AdamW, cuda, amsgrad, maximize: 0.059577220818027854
- sam, AdamW, cuda, (pt2) foreach: 0.05216499278321862
- sam, AdamW, cuda, foreach: 0.04559037489816546
- sam, AdamW, cuda, (pt2) foreach, maximize, capturable: 0.058284342056140304
- sam, AdamW, cuda, foreach, maximize, capturable: 0.053812416596338154
- sam, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.06547115463763475
- sam, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.06539025786332786
- sam, AdamW, cuda, (pt2) fused: 0.0222733854316175
- sam, AdamW, cuda, fused: 0.018440845306031406
- sam, AdamW, cuda, (pt2) fused, amsgrad, maximize: 0.024024007479763694
- sam, AdamW, cuda, fused, amsgrad, maximize: 0.0203817673958838
- sam, AdamW, cuda, (pt2) fused, capturable: 0.022076109936460853
- sam, AdamW, cuda, fused, capturable: 0.018409591051749885
- sam, AdamW, cuda, (pt2) fused, capturable, amsgrad: 0.025003961520269513
- sam, AdamW, cuda, fused, capturable, amsgrad: 0.020366889098659156
- sam, Adamax, cuda, (pt2) default: 0.06515488575678319
- sam, Adamax, cuda, default: 0.05908757844008505
- sam, Adamax, cuda, (pt2) maximize: 0.06720135329912107
- sam, Adamax, cuda, maximize: 0.06684334964180987
- sam, Adamax, cuda, (pt2) foreach: 0.06496463937219232
- sam, Adamax, cuda, foreach: 0.06462717021349818
- sam, ASGD, cuda, (pt2) default: 0.021872150525450707
- sam, ASGD, cuda, default: 0.020836778543889524
- sam, ASGD, cuda, (pt2) maximize: 0.02419306689666377
- sam, ASGD, cuda, maximize: 0.02334059472195804
- sam, ASGD, cuda, (pt2) foreach: 0.023057176938487425
- sam, ASGD, cuda, foreach: 0.021254874393343927
- sam, SGD, cuda, (pt2) default: 0.00847905631720399
- sam, SGD, cuda, default: 0.006028773533180356
- sam, SGD, cuda, (pt2) maximize: 0.013370103544245164
- sam, SGD, cuda, maximize: 0.010540734184905885
- sam, SGD, cuda, (pt2) foreach: 0.008498353650793433
- sam, SGD, cuda, foreach: 0.0060289609432220455
- sam, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 1.5293976413086057
- sam, SGD, cuda, foreach, momentum=0.9, nesterov: 0.02251190710812807
- sam, SGD, cuda, (pt2) foreach, momentum=0.9: 1.5464262911118567
- sam, SGD, cuda, foreach, momentum=0.9: 0.016489191167056563
- sam, RAdam, cuda, (pt2) default: 0.060008589993231
- sam, RAdam, cuda, default: 0.059377501020208
- sam, RAdam, cuda, (pt2) foreach: 0.05987725651357323
- sam, RAdam, cuda, foreach: 0.052524110022932294
- sam, Rprop, cuda, (pt2) default: 0.09303694420183699
- sam, Rprop, cuda, default: 0.09221767106403907
- sam, Rprop, cuda, (pt2) maximize: 0.09683224325999618
- sam, Rprop, cuda, maximize: 0.09449603253354628
- sam, Rprop, cuda, (pt2) foreach: 0.09358126871908705
- sam, Rprop, cuda, foreach: 0.09374586958438158
- sam, RMSprop, cuda, (pt2) default: 0.03011131359796439
- sam, RMSprop, cuda, default: 0.026793044805526734
- sam, RMSprop, cuda, (pt2) maximize: 0.035427935499077044
- sam, RMSprop, cuda, maximize: 0.03094885586760938
- sam, RMSprop, cuda, (pt2) foreach: 0.030377698796136037
- sam, RMSprop, cuda, foreach: 0.02675313870422542
- sam, NAdam, cuda, (pt2) default: 0.05966704827733338
- sam, NAdam, cuda, default: 0.058746659313328564
- sam, NAdam, cuda, (pt2) foreach: 0.05835566692985594
- sam, NAdam, cuda, foreach: 0.05862642521969974


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5064560349

cc @janeyx99


Good to know that sam got added. These results should be different once my PR #1681 lands