Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '61239df555df02e8c60a2ad63363878a2a57c161'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 329bb2a33e40f4bc76b2e061b180d3234984c91b

Affected PyTorch commit: 61239df555df02e8c60a2ad63363878a2a57c161

Affected Tests:
- timm_vision_transformer, Adagrad, cuda, (pt2) foreach: -12.25431%
- timm_vision_transformer, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -10.90398%
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: -10.89500%
- timm_vision_transformer, ASGD, cuda, no_foreach: -18.54688%
- timm_vision_transformer, Rprop, cuda, (pt2) default: -11.35276%
- timm_vision_transformer, Rprop, cuda, no_foreach: -10.58137%
- timm_vision_transformer, RMSprop, cuda, maximize: -13.91704%
- timm_vision_transformer, NAdam, cuda, foreach: -12.22315%
- detectron2_maskrcnn_r_101_fpn, Rprop, cuda, (pt2) maximize: +10.34279%
- tts_angular, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -10.04711%
- maml_omniglot, Adagrad, cuda, (pt2) foreach: -11.91487%
- maml_omniglot, Adam, cuda, (pt2) foreach, maximize, capturable: -10.89331%
- maml_omniglot, Adam, cuda, (pt2) fused, capturable, amsgrad: -11.27803%
- basic_gnn_sage, Adagrad, cuda, (pt2) maximize: -10.24735%
- basic_gnn_sage, Adam, cuda, (pt2) foreach, maximize, capturable: -15.42742%
- basic_gnn_sage, AdamW, cuda, (pt2) foreach, maximize, capturable: -13.34122%
- dcgan, Adam, cuda, (pt2) default: -11.77369%
- dcgan, Adam, cuda, (pt2) foreach, maximize, capturable: -12.27459%
- dcgan, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -15.98277%
- dcgan, Adam, cuda, (pt2) fused, amsgrad, maximize: -10.00113%
- dcgan, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -13.97522%
- dcgan, Adamax, cuda, (pt2) foreach: -21.05679%
- hf_BigBird, Adadelta, cuda, (pt2) maximize: -13.17801%
- timm_resnest, Adadelta, cuda, (pt2) maximize: -15.93802%
- phlippe_densenet, Adadelta, cuda, maximize: -11.47477%
- phlippe_densenet, ASGD, cuda, (pt2) foreach: -11.75574%
- detectron2_maskrcnn_r_50_fpn, Rprop, cuda, default: -12.89528%
- detectron2_maskrcnn_r_50_fpn, Rprop, cuda, (pt2) maximize: -20.33641%
- hf_T5_generate, Rprop, cuda, maximize: -15.58734%
- resnet18, Adagrad, cuda, foreach: -11.14034%
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -12.03426%
- Super_SloMo, Adamax, cuda, (pt2) default: -21.18235%
- basic_gnn_edgecnn, Adadelta, cuda, (pt2) foreach: -12.84016%
- basic_gnn_edgecnn, Adam, cuda, (pt2) amsgrad, maximize: -10.71441%
- basic_gnn_edgecnn, AdamW, cuda, (pt2) foreach, maximize, capturable: -10.04365%
- basic_gnn_edgecnn, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -10.05515%
- basic_gnn_edgecnn, Rprop, cuda, maximize: -13.74764%
- resnet152, ASGD, cuda, (pt2) default: -23.80873%
- soft_actor_critic, Adadelta, cuda, default: -10.64331%
- soft_actor_critic, AdamW, cuda, (pt2) foreach: -10.45520%
- soft_actor_critic, NAdam, cuda, foreach: -12.48280%
- timm_nfnet, AdamW, cuda, (pt2) foreach, maximize, capturable: -10.57903%
- densenet121, Adadelta, cuda, (pt2) default: -16.87342%
- densenet121, Adagrad, cuda, foreach: -10.65450%
- timm_efficientdet, Adadelta, cuda, (pt2) default: -11.64642%
- timm_efficientdet, Adagrad, cuda, (pt2) foreach: -11.49732%
- timm_efficientdet, Rprop, cuda, (pt2) default: -24.51705%
- resnext50_32x4d, Adam, cuda, (pt2) foreach: -10.84702%
- resnext50_32x4d, Adam, cuda, (pt2) foreach, maximize, capturable: -10.26377%
- resnext50_32x4d, Rprop, cuda, (pt2) maximize: -11.04222%
- fastNLP_Bert, ASGD, cuda, (pt2) default: -15.35421%
- LearningToPaint, Adam, cuda, (pt2) foreach: -15.15483%
- LearningToPaint, Adamax, cuda, (pt2) maximize: -16.02569%
- LearningToPaint, Adamax, cuda, (pt2) foreach: -18.92310%
- LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -15.22607%
- LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9: -17.87073%
- tacotron2, Adamax, cuda, default: -11.48568%
- tacotron2, SGD, cuda, (pt2) foreach, momentum=0.9: -22.02537%
- functorch_dp_cifar10, Adadelta, cuda, (pt2) maximize: -11.12401%
- squeezenet1_1, Adam, cuda, (pt2) default: -10.92421%
- squeezenet1_1, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -19.22051%
- hf_Reformer, Adam, cuda, (pt2) amsgrad, maximize: -12.11402%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -21.12171%
- phlippe_resnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -15.28392%
- phlippe_resnet, ASGD, cuda, (pt2) maximize: -23.86797%
- phlippe_resnet, SGD, cuda, (pt2) foreach, momentum=0.9: -12.13962%
- functorch_maml_omniglot, Adagrad, cuda, (pt2) maximize: -11.47287%
- functorch_maml_omniglot, Adam, cuda, foreach, maximize, capturable, amsgrad: -11.46357%
- functorch_maml_omniglot, Adamax, cuda, (pt2) default: -11.48803%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -64.89727%
- pytorch_unet, Rprop, cuda, (pt2) foreach: -11.00420%
- detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, (pt2) default: -11.67182%
- pytorch_struct, ASGD, cuda, foreach: -10.30790%
- hf_Bert, Adagrad, cuda, (pt2) foreach: -14.05178%
- hf_GPT2, ASGD, cuda, (pt2) default: -11.68062%
- basic_gnn_gin, Adam, cuda, (pt2) amsgrad, maximize: -15.39684%
- basic_gnn_gin, RMSprop, cuda, (pt2) foreach: -10.34168%
- shufflenet_v2_x1_0, Adadelta, cuda, (pt2) maximize: -15.19358%
- shufflenet_v2_x1_0, Adadelta, cuda, foreach: -15.76674%
- resnet50_quantized_qat, Adadelta, cuda, (pt2) foreach: -11.33654%
- resnet50_quantized_qat, Rprop, cuda, (pt2) foreach: -17.28228%
- detectron2_maskrcnn_r_50_c4, Adadelta, cuda, (pt2) maximize: -10.17724%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: -12.37983%
- alexnet, Adadelta, cuda, (pt2) default: -11.43466%
- alexnet, Adagrad, cuda, (pt2) maximize: -10.68209%
- doctr_det_predictor, Adadelta, cuda, maximize: -10.20750%
- doctr_det_predictor, ASGD, cuda, foreach: -12.30961%
- pytorch_CycleGAN_and_pix2pix, Adadelta, cuda, (pt2) default: -10.74675%
- detectron2_maskrcnn, Adagrad, cuda, (pt2) maximize: -12.31377%
- resnet50, Adadelta, cuda, (pt2) default: -11.84826%
- resnet50, Adadelta, cuda, (pt2) maximize: -10.10910%
- resnet50, Adadelta, cuda, (pt2) no_foreach: -10.73397%
- resnet50, Adam, cuda, (pt2) foreach, maximize, capturable: -11.84617%
- resnet50, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -22.49039%
- resnet50, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -12.82654%
- resnet50, NAdam, cuda, (pt2) no_foreach: -11.95918%
- resnet50, NAdam, cuda, (pt2) foreach: -13.52005%
- basic_gnn_gcn, Adadelta, cuda, (pt2) default: -15.13545%
- basic_gnn_gcn, Adadelta, cuda, (pt2) maximize: -11.74896%
- basic_gnn_gcn, Adagrad, cuda, (pt2) default: -12.97061%
- basic_gnn_gcn, Adam, cuda, (pt2) foreach: -10.97466%
- basic_gnn_gcn, Adam, cuda, foreach: -12.56298%
- basic_gnn_gcn, Rprop, cuda, default: -10.01399%
- doctr_reco_predictor, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -12.60705%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -39.82926%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9: -34.17495%
- attention_is_all_you_need_pytorch, Adadelta, cuda, (pt2) default: -11.43244%
- drq, Adadelta, cuda, (pt2) default: -10.29417%
- drq, Adagrad, cuda, foreach: -10.04570%
- drq, Adam, cuda, (pt2) foreach: -11.47032%
- drq, Adam, cuda, foreach, maximize, capturable: -11.50873%
- yolov3, Adam, cuda, (pt2) no_foreach: -13.80637%
- yolov3, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -15.64157%
- yolov3, AdamW, cuda, default: -10.26325%
- yolov3, AdamW, cuda, amsgrad, maximize: -11.24594%
- yolov3, AdamW, cuda, (pt2) no_foreach: -17.92312%
- yolov3, AdamW, cuda, (pt2) foreach, maximize, capturable: -15.97823%
- yolov3, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -21.26918%
- yolov3, AdamW, cuda, foreach, maximize, capturable, amsgrad: -17.75699%
- yolov3, AdamW, cuda, (pt2) fused, capturable, amsgrad: -11.98933%
- yolov3, ASGD, cuda, (pt2) no_foreach: -20.76376%
- yolov3, SGD, cuda, (pt2) maximize: -10.21234%
- yolov3, SGD, cuda, maximize: -10.86766%
- detectron2_fasterrcnn_r_101_dc5, Adadelta, cuda, (pt2) maximize: -13.84887%
- lennard_jones, Adadelta, cuda, (pt2) default: -13.50798%
- lennard_jones, Adadelta, cuda, (pt2) maximize: -10.61586%
- lennard_jones, Adadelta, cuda, (pt2) foreach: -10.01306%
- lennard_jones, Adagrad, cuda, (pt2) default: -12.30475%
- mobilenet_v3_large, Adadelta, cuda, (pt2) foreach: -13.98718%
- DALLE2_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable: -11.06814%
- DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable: -10.06501%
- DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable, amsgrad: -10.68019%
- DALLE2_pytorch, AdamW, cuda, foreach: -11.95239%
- maml, Adadelta, cuda, (pt2) maximize: -12.13246%
- maml, Adagrad, cuda, (pt2) maximize: -11.29478%
- hf_Longformer, ASGD, cuda, (pt2) maximize: -33.78827%
- hf_Longformer, Rprop, cuda, default: -12.19290%
- Background_Matting, Adadelta, cuda, (pt2) maximize: -12.50586%
- Background_Matting, ASGD, cuda, foreach: -10.49496%
- hf_T5_large, AdamW, cuda, (pt2) no_foreach: -10.01123%
- hf_T5_large, RAdam, cuda, default: -10.34683%
- hf_T5_large, RAdam, cuda, foreach: +10.72656%
- hf_T5_large, Rprop, cuda, no_foreach: -18.56174%
- hf_T5_large, NAdam, cuda, foreach: +13.01240%
- hf_GPT2_large, Adamax, cuda, (pt2) differentiable: -12.20673%
- pytorch_stargan, Adadelta, cuda, (pt2) maximize: -11.70615%
- opacus_cifar10, Adadelta, cuda, (pt2) default: -10.67391%
- opacus_cifar10, Adagrad, cuda, default: -11.38452%
- opacus_cifar10, Adagrad, cuda, (pt2) maximize: -16.35154%
- opacus_cifar10, Adam, cuda, foreach, maximize, capturable, amsgrad: -12.60229%
- hf_T5, Adadelta, cuda, (pt2) maximize: -11.70368%
- hf_T5, Rprop, cuda, (pt2) foreach: -17.17438%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -33.26679%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9: -13.56270%
- timm_vovnet, Adadelta, cuda, (pt2) maximize: -11.63745%
- timm_vovnet, Adadelta, cuda, (pt2) foreach: -10.94906%
- timm_vovnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.66452%
- vision_maskrcnn, Adagrad, cuda, (pt2) maximize: -22.90491%
- BERT_pytorch, AdamW, cuda, (pt2) differentiable: -10.29936%
- BERT_pytorch, ASGD, cuda, (pt2) maximize: -18.40478%
- BERT_pytorch, Rprop, cuda, (pt2) default: -13.44318%
- BERT_pytorch, Rprop, cuda, differentiable: -10.11964%
- BERT_pytorch, RMSprop, cuda, (pt2) maximize: -10.06195%
- timm_regnet, Adadelta, cuda, (pt2) default: -13.00043%
- timm_regnet, Adadelta, cuda, (pt2) maximize: -16.90887%
- timm_regnet, Adadelta, cuda, (pt2) foreach: -14.50509%
- timm_regnet, ASGD, cuda, (pt2) foreach: -10.23792%
- timm_regnet, RAdam, cuda, (pt2) default: -10.22619%
- timm_regnet, RAdam, cuda, (pt2) foreach: -11.01031%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5026871474

cc @janeyx99


This was part of a rerun, stats are taken account of here https://docs.google.com/spreadsheets/d/1r3a7UczDofej8p6KhAcbTf8moxnlbyQ1AWZKOo0kbh8/edit#gid=418753160