Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '61239df555df02e8c60a2ad63363878a2a57c161'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 329bb2a33e40f4bc76b2e061b180d3234984c91b

Affected PyTorch commit: 61239df555df02e8c60a2ad63363878a2a57c161

Affected Tests:
- timm_vision_transformer, ASGD, cuda, no_foreach: -14.24807%
- tts_angular, Adamax, cuda, maximize: +24.60240%
- tts_angular, Rprop, cuda, (pt2) default: +23.27589%
- maml_omniglot, Adagrad, cuda, (pt2) default: +31.82708%
- maml_omniglot, AdamW, cuda, foreach, maximize, capturable: +21.76059%
- basic_gnn_sage, Adagrad, cuda, (pt2) maximize: -10.09131%
- basic_gnn_sage, Adam, cuda, (pt2) foreach, maximize, capturable: -12.74579%
- basic_gnn_sage, Adam, cuda, fused, amsgrad, maximize: +14.72234%
- basic_gnn_sage, Adamax, cuda, maximize: +19.29499%
- basic_gnn_sage, SGD, cuda, foreach, momentum=0.9, nesterov: +21.54991%
- basic_gnn_sage, RMSprop, cuda, maximize: +16.86868%
- dcgan, Adagrad, cuda, (pt2) default: +14.25200%
- dcgan, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.68271%
- dcgan, AdamW, cuda, foreach: +11.97053%
- dcgan, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -10.50942%
- dcgan, Adamax, cuda, (pt2) foreach: -18.80016%
- dcgan, Rprop, cuda, maximize: +22.79441%
- hf_BigBird, Adadelta, cuda, (pt2) maximize: -12.65230%
- hf_BigBird, Rprop, cuda, (pt2) maximize: +11.64122%
- mnasnet1_0, Adamax, cuda, (pt2) maximize: +31.00295%
- mnasnet1_0, Adamax, cuda, maximize: +12.48048%
- mnasnet1_0, Adamax, cuda, (pt2) foreach: +10.18290%
- timm_resnest, Adadelta, cuda, (pt2) maximize: -15.01904%
- fambench_xlmr, ASGD, cuda, default: +47.53834%
- detectron2_maskrcnn_r_50_fpn, Rprop, cuda, (pt2) maximize: -19.53644%
- hf_T5_generate, Rprop, cuda, maximize: -14.34243%
- hf_T5_generate, Rprop, cuda, (pt2) foreach: +20.38681%
- resnet18, Adagrad, cuda, (pt2) foreach: +44.49974%
- resnet18, ASGD, cuda, default: +16.82580%
- demucs, SGD, cuda, (pt2) foreach, momentum=0.9: +45.53792%
- Super_SloMo, Adamax, cuda, (pt2) default: -18.55081%
- basic_gnn_edgecnn, Rprop, cuda, maximize: -14.43464%
- mobilenet_v2, Adamax, cuda, (pt2) default: +24.13227%
- detectron2_fasterrcnn_r_50_c4, Adam, cuda, (pt2) foreach, maximize, capturable: +11.89622%
- detectron2_fasterrcnn_r_50_c4, Adamax, cuda, default: +10.05998%
- detectron2_fasterrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +68.80708%
- detectron2_fasterrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +41.47280%
- detectron2_fasterrcnn_r_50_c4, Rprop, cuda, (pt2) default: +28.66249%
- resnet152, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +12.50016%
- resnet152, Adamax, cuda, (pt2) maximize: +23.73746%
- soft_actor_critic, Adam, cuda, (pt2) foreach, maximize, capturable: +19.12549%
- soft_actor_critic, Adamax, cuda, default: +25.67363%
- soft_actor_critic, Adamax, cuda, maximize: +10.54824%
- soft_actor_critic, ASGD, cuda, maximize: +35.37363%
- soft_actor_critic, RAdam, cuda, (pt2) default: +13.69827%
- soft_actor_critic, RAdam, cuda, default: +20.45974%
- soft_actor_critic, NAdam, cuda, foreach: -10.13565%
- timm_nfnet, ASGD, cuda, maximize: +20.84364%
- densenet121, Adadelta, cuda, (pt2) default: -15.06396%
- densenet121, Adagrad, cuda, foreach: +15.82723%
- densenet121, ASGD, cuda, (pt2) default: +19.19490%
- densenet121, ASGD, cuda, (pt2) maximize: +37.66891%
- llama, ASGD, cuda, default: +29.15334%
- llama, ASGD, cuda, (pt2) foreach: +26.88641%
- llama, NAdam, cuda, default: +10.17445%
- mobilenet_v2_quantized_qat, Adagrad, cuda, (pt2) default: -10.93704%
- mobilenet_v2_quantized_qat, ASGD, cuda, foreach: +12.05786%
- timm_efficientdet, Rprop, cuda, (pt2) default: -21.71366%
- timm_efficientdet, NAdam, cuda, foreach: +11.28872%
- moco, Adagrad, cuda, (pt2) foreach: +42.58246%
- resnext50_32x4d, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +23.25351%
- fastNLP_Bert, Adamax, cuda, maximize: +42.22569%
- fastNLP_Bert, ASGD, cuda, (pt2) default: -13.87686%
- LearningToPaint, Adam, cuda, (pt2) foreach: -14.04691%
- LearningToPaint, Adamax, cuda, (pt2) maximize: -12.33982%
- LearningToPaint, Adamax, cuda, (pt2) foreach: -16.48568%
- hf_Bart, Rprop, cuda, (pt2) foreach: +11.49632%
- tacotron2, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +13.11122%
- tacotron2, SGD, cuda, (pt2) foreach, momentum=0.9: -10.18566%
- detectron2_fasterrcnn_r_101_fpn, Adam, cuda, (pt2) foreach, maximize, capturable: +12.08076%
- detectron2_fasterrcnn_r_101_fpn, ASGD, cuda, default: +14.21628%
- detectron2_fasterrcnn_r_101_fpn, ASGD, cuda, (pt2) maximize: +13.60922%
- detectron2_fasterrcnn_r_101_fpn, ASGD, cuda, maximize: +14.85916%
- functorch_dp_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +21.93161%
- squeezenet1_1, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -15.19192%
- squeezenet1_1, Adamax, cuda, (pt2) default: +10.69735%
- squeezenet1_1, ASGD, cuda, (pt2) maximize: +14.12808%
- squeezenet1_1, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +21.74120%
- squeezenet1_1, SGD, cuda, (pt2) foreach, momentum=0.9: +18.32257%
- detectron2_fasterrcnn_r_101_c4, Adamax, cuda, (pt2) foreach: +17.37893%
- hf_Reformer, Adam, cuda, foreach, maximize, capturable: +12.33301%
- hf_Reformer, Adamax, cuda, (pt2) default: +12.25354%
- hf_Reformer, ASGD, cuda, default: +10.00311%
- hf_Reformer, ASGD, cuda, (pt2) foreach: +49.38184%
- hf_Reformer, ASGD, cuda, foreach: +10.78887%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +36.46602%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9: +111.65087%
- hf_Reformer, Rprop, cuda, (pt2) maximize: +14.16118%
- phlippe_resnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -13.41974%
- phlippe_resnet, ASGD, cuda, (pt2) maximize: -23.38989%
- phlippe_resnet, ASGD, cuda, (pt2) foreach: +13.21774%
- phlippe_resnet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +22.20053%
- functorch_maml_omniglot, Adagrad, cuda, (pt2) maximize: -10.21418%
- functorch_maml_omniglot, Adam, cuda, foreach, maximize, capturable, amsgrad: -12.10147%
- functorch_maml_omniglot, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +25.09202%
- functorch_maml_omniglot, Adamax, cuda, (pt2) default: -10.60412%
- pytorch_unet, Adagrad, cuda, (pt2) maximize: +38.58308%
- pytorch_unet, AdamW, cuda, (pt2) foreach, maximize, capturable: +22.14754%
- pytorch_unet, ASGD, cuda, maximize: +26.88668%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -36.35266%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9: +37.46804%
- pytorch_unet, Rprop, cuda, default: +11.79049%
- detectron2_fasterrcnn_r_50_fpn, Adamax, cuda, (pt2) default: +14.44289%
- pytorch_struct, Adadelta, cuda, (pt2) foreach: +12.24809%
- pytorch_struct, Adagrad, cuda, (pt2) default: +16.91471%
- hf_Bert, Adagrad, cuda, maximize: +11.44904%
- hf_Bert, Adagrad, cuda, (pt2) foreach: -10.28149%
- hf_Bert, ASGD, cuda, default: +34.47691%
- hf_Bert, Rprop, cuda, foreach: +28.07731%
- hf_GPT2, Adamax, cuda, (pt2) default: +12.25438%
- hf_GPT2, ASGD, cuda, default: +25.57166%
- detectron2_maskrcnn_r_101_c4, Adamax, cuda, (pt2) foreach: +10.78815%
- hf_T5_base, Adamax, cuda, foreach: +15.22846%
- hf_T5_base, ASGD, cuda, default: +12.83045%
- hf_T5_base, Rprop, cuda, (pt2) foreach: +36.96994%
- basic_gnn_gin, Adadelta, cuda, (pt2) default: +15.57300%
- basic_gnn_gin, Adadelta, cuda, default: +12.43132%
- basic_gnn_gin, Adadelta, cuda, (pt2) maximize: +14.87097%
- basic_gnn_gin, Adam, cuda, (pt2) amsgrad, maximize: -11.35964%
- shufflenet_v2_x1_0, Adadelta, cuda, (pt2) maximize: -13.65211%
- shufflenet_v2_x1_0, Adadelta, cuda, foreach: -14.16242%
- shufflenet_v2_x1_0, ASGD, cuda, (pt2) foreach: +14.16277%
- hf_Bert_large, Adamax, cuda, default: +10.39152%
- hf_Bert_large, ASGD, cuda, (pt2) foreach: +10.09232%
- hf_Bert_large, ASGD, cuda, foreach: +10.33511%
- hf_Bert_large, Rprop, cuda, (pt2) maximize: +15.70764%
- resnet50_quantized_qat, Adadelta, cuda, (pt2) foreach: -11.54096%
- resnet50_quantized_qat, Adam, cuda, (pt2) amsgrad, maximize: +10.04102%
- resnet50_quantized_qat, Adam, cuda, (pt2) foreach: +11.83375%
- resnet50_quantized_qat, ASGD, cuda, maximize: +11.31897%
- resnet50_quantized_qat, Rprop, cuda, (pt2) foreach: -15.49006%
- resnet50_quantized_qat, NAdam, cuda, (pt2) default: +10.79666%
- detectron2_maskrcnn_r_50_c4, Adam, cuda, (pt2) foreach, maximize, capturable: +13.31336%
- detectron2_maskrcnn_r_50_c4, Adamax, cuda, foreach: +10.39805%
- detectron2_maskrcnn_r_50_c4, ASGD, cuda, (pt2) default: +13.96567%
- detectron2_maskrcnn_r_50_c4, ASGD, cuda, (pt2) maximize: +13.48465%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +35.35311%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +15.35485%
- detectron2_maskrcnn_r_50_c4, Rprop, cuda, (pt2) foreach: +33.49349%
- alexnet, Adadelta, cuda, (pt2) default: -11.13233%
- doctr_det_predictor, Adagrad, cuda, (pt2) default: +25.69032%
- doctr_det_predictor, Adagrad, cuda, default: +11.82782%
- doctr_det_predictor, ASGD, cuda, foreach: -12.68963%
- detectron2_maskrcnn, Adagrad, cuda, (pt2) maximize: -10.75546%
- detectron2_maskrcnn, Adamax, cuda, maximize: +29.49249%
- detectron2_maskrcnn, Rprop, cuda, default: +38.36526%
- resnet50, Adadelta, cuda, (pt2) default: -11.05108%
- resnet50, Adagrad, cuda, differentiable: +14.16812%
- resnet50, Adam, cuda, default: +17.67970%
- resnet50, Adam, cuda, (pt2) amsgrad, maximize: +10.01293%
- resnet50, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -17.97744%
- resnet50, AdamW, cuda, (pt2) foreach, maximize, capturable: +12.68206%
- resnet50, Adamax, cuda, (pt2) no_foreach: +14.64251%
- resnet50, Adamax, cuda, (pt2) differentiable: +10.90155%
- resnet50, RAdam, cuda, (pt2) default: +10.49229%
- resnet50, Rprop, cuda, maximize: +21.98761%
- resnet50, Rprop, cuda, differentiable: +14.65993%
- resnet50, NAdam, cuda, (pt2) default: +18.47667%
- basic_gnn_gcn, Adadelta, cuda, (pt2) default: -11.44053%
- basic_gnn_gcn, Adadelta, cuda, maximize: +17.53098%
- basic_gnn_gcn, Adagrad, cuda, (pt2) default: -10.35370%
- basic_gnn_gcn, Adam, cuda, foreach: -10.01857%
- basic_gnn_gcn, ASGD, cuda, maximize: +10.04844%
- basic_gnn_gcn, Rprop, cuda, (pt2) default: +36.57083%
- doctr_reco_predictor, Adagrad, cuda, (pt2) default: +13.83370%
- doctr_reco_predictor, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.07922%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -28.37507%
- attention_is_all_you_need_pytorch, Adadelta, cuda, (pt2) default: -10.05562%
- drq, Adagrad, cuda, foreach: -10.08893%
- drq, AdamW, cuda, (pt2) default: +15.25471%
- drq, AdamW, cuda, (pt2) foreach, maximize, capturable: +24.18874%
- drq, Adamax, cuda, (pt2) default: +20.36760%
- drq, Adamax, cuda, default: +37.20133%
- drq, ASGD, cuda, foreach: +47.24192%
- yolov3, Adadelta, cuda, no_foreach: +24.38213%
- yolov3, Adagrad, cuda, (pt2) foreach: +21.32970%
- yolov3, AdamW, cuda, (pt2) no_foreach: -12.33099%
- yolov3, AdamW, cuda, (pt2) foreach, maximize, capturable: -12.53978%
- yolov3, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -18.24636%
- yolov3, AdamW, cuda, foreach, maximize, capturable, amsgrad: -12.70650%
- yolov3, Adamax, cuda, (pt2) default: +28.15061%
- yolov3, Adamax, cuda, default: +31.44431%
- yolov3, ASGD, cuda, (pt2) no_foreach: -16.85710%
- yolov3, ASGD, cuda, (pt2) differentiable: +21.60345%
- detectron2_fasterrcnn_r_101_dc5, Adadelta, cuda, (pt2) maximize: -12.58444%
- detectron2_fasterrcnn_r_101_dc5, Rprop, cuda, default: +12.80306%
- lennard_jones, Adadelta, cuda, (pt2) default: -10.41880%
- lennard_jones, Adagrad, cuda, (pt2) default: -11.29166%
- lennard_jones, Rprop, cuda, (pt2) maximize: +16.84256%
- mobilenet_v3_large, Adagrad, cuda, default: +37.21348%
- mobilenet_v3_large, Adam, cuda, foreach, maximize, capturable, amsgrad: +11.58356%
- mobilenet_v3_large, ASGD, cuda, maximize: +17.17873%
- mobilenet_v3_large, Rprop, cuda, maximize: +11.80958%
- DALLE2_pytorch, AdamW, cuda, default: +11.46603%
- DALLE2_pytorch, SGD, cuda, maximize: +14.09157%
- maml, Adam, cuda, (pt2) foreach: +18.07617%
- maml, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +21.49871%
- maml, Rprop, cuda, (pt2) default: +33.25284%
- maml, Rprop, cuda, foreach: +11.78776%
- maml, RMSprop, cuda, default: +14.67773%
- hf_Longformer, Adamax, cuda, foreach: +20.41121%
- hf_Longformer, ASGD, cuda, (pt2) maximize: -31.18586%
- hf_T5_large, RAdam, cuda, foreach: +12.69158%
- hf_T5_large, Rprop, cuda, no_foreach: -14.25599%
- hf_T5_large, NAdam, cuda, foreach: +14.69657%
- hf_GPT2_large, Adamax, cuda, (pt2) differentiable: -10.30114%
- hf_GPT2_large, ASGD, cuda, default: +33.48964%
- pytorch_stargan, Adadelta, cuda, (pt2) maximize: -12.04019%
- pytorch_stargan, ASGD, cuda, maximize: +12.95297%
- opacus_cifar10, Adagrad, cuda, (pt2) maximize: -11.79329%
- opacus_cifar10, Adam, cuda, foreach, maximize, capturable, amsgrad: -13.59628%
- opacus_cifar10, Adamax, cuda, (pt2) foreach: +15.40379%
- opacus_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +11.31121%
- opacus_cifar10, Rprop, cuda, maximize: +23.73883%
- hf_T5, Adadelta, cuda, (pt2) maximize: -11.00914%
- hf_T5, Rprop, cuda, (pt2) foreach: +23.05564%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -18.36517%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9: +16.59287%
- timm_vovnet, Adadelta, cuda, (pt2) maximize: -10.80304%
- timm_vovnet, Adadelta, cuda, (pt2) foreach: -10.05891%
- timm_vovnet, Adagrad, cuda, (pt2) maximize: +12.92363%
- timm_vovnet, Adagrad, cuda, maximize: +35.40675%
- timm_vovnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.04576%
- vision_maskrcnn, Adagrad, cuda, (pt2) maximize: -21.64781%
- vision_maskrcnn, Adamax, cuda, maximize: +32.68084%
- BERT_pytorch, Adamax, cuda, maximize: +12.61527%
- BERT_pytorch, ASGD, cuda, (pt2) maximize: -15.28958%
- timm_regnet, Adadelta, cuda, (pt2) default: -11.90396%
- timm_regnet, Adadelta, cuda, (pt2) maximize: -15.29357%
- timm_regnet, Adadelta, cuda, (pt2) foreach: -13.50832%
- speech_transformer, AdamW, cuda, foreach, maximize, capturable: +31.80974%
- speech_transformer, Rprop, cuda, (pt2) default: +14.24435%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5020694796

cc @janeyx99


This was part of a rerun, stats are taken account of here https://docs.google.com/spreadsheets/d/1r3a7UczDofej8p6KhAcbTf8moxnlbyQ1AWZKOo0kbh8/edit#gid=418753160