Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '61239df555df02e8c60a2ad63363878a2a57c161'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 329bb2a33e40f4bc76b2e061b180d3234984c91b

Affected PyTorch commit: 61239df555df02e8c60a2ad63363878a2a57c161

Affected Tests:
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: -10.01248%
- timm_vision_transformer, ASGD, cuda, no_foreach: -16.87992%
- timm_vision_transformer, RMSprop, cuda, maximize: -10.30099%
- tts_angular, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.38692%
- maml_omniglot, Adagrad, cuda, (pt2) foreach: -11.92976%
- maml_omniglot, Adam, cuda, (pt2) foreach, maximize, capturable: -12.09070%
- maml_omniglot, Adam, cuda, (pt2) fused, capturable, amsgrad: -12.79054%
- basic_gnn_sage, Adagrad, cuda, (pt2) maximize: -13.30274%
- basic_gnn_sage, Adam, cuda, (pt2) foreach, maximize, capturable: -12.02412%
- basic_gnn_sage, AdamW, cuda, (pt2) foreach, maximize, capturable: -10.13127%
- basic_gnn_sage, SGD, cuda, (pt2) foreach: -23.68424%
- basic_gnn_sage, SGD, cuda, (pt2) foreach, momentum=0.9: -37.01781%
- dcgan, Adam, cuda, (pt2) foreach, maximize, capturable: -11.98465%
- dcgan, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -14.13365%
- dcgan, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -14.84607%
- dcgan, Adamax, cuda, (pt2) foreach: -21.70068%
- hf_BigBird, Adadelta, cuda, (pt2) maximize: -12.63795%
- hf_BigBird, ASGD, cuda, (pt2) maximize: +11.62701%
- mnasnet1_0, ASGD, cuda, (pt2) foreach: -11.96628%
- timm_resnest, Adadelta, cuda, (pt2) maximize: -15.72213%
- phlippe_densenet, ASGD, cuda, (pt2) foreach: -11.72892%
- detectron2_maskrcnn_r_50_fpn, Rprop, cuda, default: -11.27111%
- detectron2_maskrcnn_r_50_fpn, Rprop, cuda, (pt2) maximize: -20.23396%
- hf_T5_generate, Rprop, cuda, maximize: -15.74429%
- resnet18, Adagrad, cuda, foreach: -10.25527%
- Super_SloMo, Adagrad, cuda, (pt2) default: -10.46961%
- Super_SloMo, Adamax, cuda, (pt2) default: -23.37142%
- basic_gnn_edgecnn, Adadelta, cuda, (pt2) foreach: -14.00214%
- basic_gnn_edgecnn, Adagrad, cuda, (pt2) maximize: -12.57185%
- basic_gnn_edgecnn, Adam, cuda, (pt2) amsgrad, maximize: -13.79396%
- basic_gnn_edgecnn, AdamW, cuda, amsgrad, maximize: -10.12495%
- basic_gnn_edgecnn, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -10.49103%
- basic_gnn_edgecnn, AdamW, cuda, (pt2) fused: -11.74624%
- basic_gnn_edgecnn, Rprop, cuda, maximize: -14.93946%
- resnet152, ASGD, cuda, (pt2) default: -22.51713%
- soft_actor_critic, Adadelta, cuda, default: -10.84086%
- soft_actor_critic, NAdam, cuda, foreach: -11.38998%
- timm_nfnet, AdamW, cuda, (pt2) foreach, maximize, capturable: -11.70121%
- densenet121, Adadelta, cuda, (pt2) default: -16.45142%
- mobilenet_v2_quantized_qat, Adagrad, cuda, (pt2) default: -10.04491%
- timm_efficientdet, Adadelta, cuda, (pt2) default: -12.39400%
- timm_efficientdet, Adagrad, cuda, (pt2) foreach: -10.66051%
- timm_efficientdet, Rprop, cuda, (pt2) default: -24.57834%
- resnext50_32x4d, Adam, cuda, (pt2) foreach: -10.80371%
- resnext50_32x4d, Adam, cuda, (pt2) foreach, maximize, capturable: -11.03400%
- fastNLP_Bert, ASGD, cuda, (pt2) default: -11.94000%
- LearningToPaint, Adagrad, cuda, (pt2) default: -11.88596%
- LearningToPaint, Adagrad, cuda, (pt2) maximize: -10.67240%
- LearningToPaint, Adam, cuda, (pt2) foreach: -15.88313%
- LearningToPaint, Adamax, cuda, (pt2) maximize: -16.48529%
- LearningToPaint, Adamax, cuda, (pt2) foreach: -19.50493%
- LearningToPaint, ASGD, cuda, (pt2) maximize: -11.58741%
- LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -13.59273%
- LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9: -10.52415%
- tacotron2, Adamax, cuda, default: -10.15236%
- tacotron2, SGD, cuda, (pt2) foreach, momentum=0.9: -26.59653%
- squeezenet1_1, Adam, cuda, (pt2) default: -10.24314%
- squeezenet1_1, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -18.78281%
- hf_Reformer, Adam, cuda, (pt2) amsgrad, maximize: -10.62071%
- hf_Reformer, Adam, cuda, (pt2) fused, amsgrad, maximize: +10.38674%
- hf_Reformer, SGD, cuda, (pt2) maximize: -10.04941%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +25.11483%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9: -23.84247%
- phlippe_resnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.87088%
- phlippe_resnet, ASGD, cuda, (pt2) maximize: -22.01429%
- functorch_maml_omniglot, Adadelta, cuda, (pt2) default: -13.44756%
- functorch_maml_omniglot, Adagrad, cuda, (pt2) maximize: -11.95325%
- functorch_maml_omniglot, Adam, cuda, (pt2) amsgrad, maximize: -11.58347%
- functorch_maml_omniglot, Adam, cuda, foreach, maximize, capturable, amsgrad: -14.55326%
- functorch_maml_omniglot, Adamax, cuda, (pt2) default: -13.36964%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -46.01557%
- pytorch_unet, Rprop, cuda, (pt2) foreach: -10.16360%
- hf_Bert, Adagrad, cuda, (pt2) foreach: -13.13066%
- hf_GPT2, ASGD, cuda, (pt2) default: -13.27675%
- basic_gnn_gin, Adam, cuda, (pt2) amsgrad, maximize: -14.89322%
- basic_gnn_gin, AdamW, cuda, (pt2) default: -10.51046%
- shufflenet_v2_x1_0, Adadelta, cuda, (pt2) maximize: -13.90149%
- shufflenet_v2_x1_0, Adadelta, cuda, foreach: -15.55485%
- resnet50_quantized_qat, Adadelta, cuda, (pt2) foreach: -11.40155%
- resnet50_quantized_qat, Rprop, cuda, (pt2) foreach: -16.87012%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +265.50289%
- alexnet, Adadelta, cuda, (pt2) default: -11.25098%
- alexnet, Adagrad, cuda, (pt2) maximize: -10.27147%
- doctr_det_predictor, ASGD, cuda, foreach: -12.56931%
- resnet50, Adadelta, cuda, (pt2) default: -11.75909%
- resnet50, Adagrad, cuda, differentiable: -11.88873%
- resnet50, Adam, cuda, (pt2) foreach, maximize, capturable: -10.72240%
- resnet50, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -20.53246%
- resnet50, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.46658%
- resnet50, NAdam, cuda, (pt2) no_foreach: -11.50416%
- resnet50, NAdam, cuda, (pt2) foreach: -13.24629%
- basic_gnn_gcn, Adadelta, cuda, (pt2) default: -15.68482%
- basic_gnn_gcn, Adadelta, cuda, (pt2) maximize: -14.48354%
- basic_gnn_gcn, Adagrad, cuda, (pt2) default: -13.95354%
- basic_gnn_gcn, Adam, cuda, default: -10.08247%
- basic_gnn_gcn, Adam, cuda, foreach: -12.62749%
- basic_gnn_gcn, Adam, cuda, foreach, maximize, capturable: -10.73199%
- basic_gnn_gcn, AdamW, cuda, foreach: -11.07548%
- basic_gnn_gcn, SGD, cuda, (pt2) default: -10.19325%
- basic_gnn_gcn, SGD, cuda, (pt2) maximize: -12.32765%
- basic_gnn_gcn, Rprop, cuda, default: -11.52703%
- doctr_reco_predictor, Adam, cuda, (pt2) foreach: +10.29696%
- doctr_reco_predictor, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -12.51187%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -48.24580%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9: -34.29776%
- drq, Adadelta, cuda, (pt2) default: -10.13709%
- drq, Adagrad, cuda, foreach: -10.72158%
- drq, Adam, cuda, (pt2) foreach: -11.84423%
- drq, Adam, cuda, foreach, maximize, capturable: -11.30988%
- yolov3, Adam, cuda, (pt2) no_foreach: -12.56054%
- yolov3, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -15.23312%
- yolov3, AdamW, cuda, (pt2) no_foreach: -16.66322%
- yolov3, AdamW, cuda, (pt2) foreach, maximize, capturable: -15.29478%
- yolov3, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -20.88576%
- yolov3, AdamW, cuda, foreach, maximize, capturable, amsgrad: -16.25440%
- yolov3, AdamW, cuda, (pt2) fused, capturable, amsgrad: -11.43903%
- yolov3, Adamax, cuda, maximize: -11.04424%
- yolov3, Adamax, cuda, (pt2) no_foreach: -10.89921%
- yolov3, ASGD, cuda, (pt2) no_foreach: -19.10440%
- yolov3, SGD, cuda, maximize: -13.63384%
- detectron2_fasterrcnn_r_101_dc5, Adadelta, cuda, (pt2) maximize: -13.45554%
- lennard_jones, Adadelta, cuda, (pt2) default: -15.24061%
- lennard_jones, Adadelta, cuda, (pt2) maximize: -12.35836%
- lennard_jones, Adadelta, cuda, (pt2) foreach: -10.86898%
- lennard_jones, Adagrad, cuda, (pt2) default: -13.06779%
- mobilenet_v3_large, Adadelta, cuda, (pt2) foreach: -14.75961%
- DALLE2_pytorch, Adam, cuda, amsgrad, maximize: -12.64253%
- maml, Adadelta, cuda, (pt2) maximize: -11.64258%
- maml, Adagrad, cuda, (pt2) maximize: -10.18347%
- hf_Longformer, ASGD, cuda, (pt2) maximize: -32.43380%
- hf_Longformer, Rprop, cuda, default: -11.42023%
- Background_Matting, Adadelta, cuda, (pt2) maximize: -10.94875%
- Background_Matting, ASGD, cuda, foreach: -12.02201%
- hf_T5_large, Rprop, cuda, no_foreach: -17.83301%
- hf_T5_large, NAdam, cuda, default: -12.33328%
- hf_T5_large, NAdam, cuda, foreach: +12.33019%
- hf_GPT2_large, AdamW, cuda, differentiable: -10.81785%
- hf_GPT2_large, Adamax, cuda, (pt2) differentiable: -13.59470%
- pytorch_stargan, Adadelta, cuda, (pt2) maximize: -12.02932%
- opacus_cifar10, Adadelta, cuda, (pt2) default: -10.65306%
- opacus_cifar10, Adagrad, cuda, default: -14.09334%
- opacus_cifar10, Adagrad, cuda, (pt2) maximize: -16.45381%
- opacus_cifar10, Adam, cuda, foreach, maximize, capturable, amsgrad: -12.40345%
- hf_T5, Adadelta, cuda, (pt2) maximize: -11.46756%
- hf_T5, Rprop, cuda, (pt2) foreach: -16.82742%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -31.77383%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9: -12.34998%
- timm_vovnet, Adadelta, cuda, (pt2) maximize: -11.72112%
- timm_vovnet, Adadelta, cuda, (pt2) foreach: -10.99124%
- vision_maskrcnn, Adagrad, cuda, (pt2) maximize: -23.71952%
- BERT_pytorch, AdamW, cuda, (pt2) differentiable: -11.16219%
- BERT_pytorch, ASGD, cuda, (pt2) maximize: -16.04003%
- BERT_pytorch, Rprop, cuda, (pt2) default: -12.36974%
- BERT_pytorch, RMSprop, cuda, (pt2) maximize: -10.15131%
- BERT_pytorch, RMSprop, cuda, (pt2) no_foreach: -10.38742%
- timm_regnet, Adadelta, cuda, (pt2) default: -12.25981%
- timm_regnet, Adadelta, cuda, (pt2) maximize: -16.44374%
- timm_regnet, Adadelta, cuda, (pt2) foreach: -13.97731%
- timm_regnet, ASGD, cuda, (pt2) foreach: -10.70455%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5026870034

cc @janeyx99


This was part of a rerun, stats are taken account of here https://docs.google.com/spreadsheets/d/1r3a7UczDofej8p6KhAcbTf8moxnlbyQ1AWZKOo0kbh8/edit#gid=418753160