Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '871fc7bb76f05c3c487214404f687cf7a6a8e453'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 61239df555df02e8c60a2ad63363878a2a57c161

Affected PyTorch commit: 871fc7bb76f05c3c487214404f687cf7a6a8e453

Affected Tests:
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +11.41520%
- resnet18, Rprop, cuda, (pt2) default: +25.87685%
- pytorch_unet, AdamW, cuda, foreach, maximize, capturable, amsgrad: +12.80943%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -33.54267%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9: -52.88524%
- hf_T5_large, Adagrad, cuda, no_foreach: +13.49038%
- hf_T5_large, RMSprop, cuda, (pt2) differentiable: -11.71451%
- basic_gnn_edgecnn, Adadelta, cuda, default: +27.62941%
- basic_gnn_edgecnn, Adamax, cuda, (pt2) default: -17.00284%
- basic_gnn_edgecnn, ASGD, cuda, maximize: -14.39832%
- basic_gnn_edgecnn, ASGD, cuda, foreach: -10.28837%
- basic_gnn_edgecnn, RAdam, cuda, (pt2) default: -10.87078%
- basic_gnn_edgecnn, Rprop, cuda, default: -19.74968%
- basic_gnn_edgecnn, RMSprop, cuda, foreach: -14.68945%
- basic_gnn_gcn, Adam, cuda, (pt2) foreach: -11.63740%
- basic_gnn_gcn, RMSprop, cuda, default: -13.55987%
- basic_gnn_gcn, NAdam, cuda, (pt2) default: -15.55362%
- Background_Matting, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.04879%
- Background_Matting, ASGD, cuda, foreach: -12.42433%
- Background_Matting, Rprop, cuda, default: +11.45717%
- hf_T5_generate, Adagrad, cuda, (pt2) maximize: -10.98973%
- hf_T5_generate, Adagrad, cuda, maximize: -13.58743%
- hf_T5_generate, AdamW, cuda, foreach, maximize, capturable: -11.60198%
- hf_T5_generate, ASGD, cuda, foreach: -11.09561%
- mnasnet1_0, Adamax, cuda, foreach: -12.28675%
- functorch_maml_omniglot, Adadelta, cuda, default: -19.76700%
- functorch_maml_omniglot, Adagrad, cuda, foreach: -17.83221%
- functorch_maml_omniglot, AdamW, cuda, fused, amsgrad, maximize: -10.00804%
- functorch_maml_omniglot, ASGD, cuda, (pt2) foreach: -23.05221%
- functorch_maml_omniglot, NAdam, cuda, default: -14.32785%
- drq, Adagrad, cuda, maximize: -11.62235%
- drq, Adagrad, cuda, (pt2) foreach: -12.77733%
- drq, Adagrad, cuda, foreach: -11.04229%
- drq, ASGD, cuda, maximize: -17.28606%
- drq, ASGD, cuda, (pt2) foreach: -23.73887%
- drq, ASGD, cuda, foreach: -10.34485%
- drq, RAdam, cuda, default: -13.12559%
- drq, Rprop, cuda, default: -12.59875%
- hf_Albert, Adamax, cuda, maximize: -14.00691%
- hf_Albert, ASGD, cuda, (pt2) default: +17.91252%
- hf_Albert, ASGD, cuda, default: -20.32586%
- hf_Albert, ASGD, cuda, (pt2) maximize: -26.37436%
- hf_Albert, ASGD, cuda, maximize: -25.98432%
- hf_Albert, Rprop, cuda, default: -14.52606%
- hf_Albert, NAdam, cuda, foreach: -14.11669%
- timm_resnest, ASGD, cuda, (pt2) default: +18.41259%
- detectron2_maskrcnn_r_50_c4, Adagrad, cuda, default: -13.43967%
- detectron2_maskrcnn_r_50_c4, Adamax, cuda, maximize: -11.68103%
- detectron2_maskrcnn_r_50_c4, ASGD, cuda, default: -29.29136%
- detectron2_maskrcnn_r_50_c4, ASGD, cuda, (pt2) foreach: +14.55121%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +20.85418%
- mobilenet_v2, Adamax, cuda, (pt2) maximize: +11.54524%
- fambench_xlmr, ASGD, cuda, (pt2) maximize: -11.54265%
- fambench_xlmr, Rprop, cuda, maximize: +19.83784%
- functorch_dp_cifar10, Adam, cuda, foreach: -11.43764%
- functorch_dp_cifar10, ASGD, cuda, (pt2) default: -27.84063%
- functorch_dp_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -12.62369%
- functorch_dp_cifar10, RAdam, cuda, default: -14.47291%
- mobilenet_v2_quantized_qat, Adagrad, cuda, (pt2) maximize: +16.80807%
- doctr_reco_predictor, Adagrad, cuda, (pt2) maximize: -16.99350%
- doctr_reco_predictor, AdamW, cuda, (pt2) foreach, maximize, capturable: -16.39528%
- doctr_reco_predictor, ASGD, cuda, default: -24.47826%
- doctr_reco_predictor, ASGD, cuda, (pt2) foreach: -16.80870%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +36.15068%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9: +31.58992%
- doctr_reco_predictor, Rprop, cuda, (pt2) default: +11.16326%
- tts_angular, Adamax, cuda, (pt2) default: +11.74664%
- tts_angular, ASGD, cuda, (pt2) default: +13.92050%
- tts_angular, ASGD, cuda, default: +25.63797%
- tts_angular, ASGD, cuda, (pt2) foreach: +30.72166%
- squeezenet1_1, Adam, cuda, (pt2) default: +11.56718%
- squeezenet1_1, Adam, cuda, foreach, maximize, capturable: +13.76706%
- squeezenet1_1, AdamW, cuda, (pt2) foreach, maximize, capturable: +22.63357%
- squeezenet1_1, AdamW, cuda, foreach, maximize, capturable, amsgrad: +25.74663%
- squeezenet1_1, SGD, cuda, (pt2) foreach, momentum=0.9: -16.74339%
- hf_Bert_large, Adamax, cuda, (pt2) default: -10.67257%
- pytorch_struct, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +17.04121%
- pytorch_struct, Rprop, cuda, (pt2) default: +11.41385%
- timm_vovnet, Adagrad, cuda, (pt2) default: +13.21080%
- timm_vovnet, Adagrad, cuda, (pt2) maximize: +26.66634%
- timm_vovnet, ASGD, cuda, (pt2) maximize: +30.05321%
- mobilenet_v3_large, Adam, cuda, (pt2) foreach: -13.65489%
- mobilenet_v3_large, Adam, cuda, (pt2) foreach, maximize, capturable: -12.75379%
- mobilenet_v3_large, AdamW, cuda, (pt2) amsgrad, maximize: -23.15867%
- mobilenet_v3_large, Adamax, cuda, maximize: -22.70216%
- mobilenet_v3_large, ASGD, cuda, foreach: +13.71498%
- LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +16.17799%
- LearningToPaint, RAdam, cuda, default: -19.24997%
- LearningToPaint, NAdam, cuda, (pt2) foreach: -15.68235%
- speech_transformer, ASGD, cuda, (pt2) foreach: -11.48065%
- moco, Rprop, cuda, maximize: +23.86199%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +24.88031%
- lennard_jones, ASGD, cuda, maximize: -12.59141%
- maml, AdamW, cuda, foreach, maximize, capturable, amsgrad: -15.26627%
- maml, NAdam, cuda, (pt2) default: +17.95387%
- detectron2_maskrcnn_r_101_c4, Adamax, cuda, maximize: -16.36851%
- detectron2_maskrcnn_r_101_c4, Adamax, cuda, foreach: -11.12601%
- detectron2_maskrcnn_r_101_c4, ASGD, cuda, (pt2) maximize: -27.97603%
- detectron2_maskrcnn_r_101_c4, Rprop, cuda, (pt2) maximize: -21.57462%
- shufflenet_v2_x1_0, Rprop, cuda, (pt2) maximize: +10.06785%
- hf_DistilBert, Adamax, cuda, default: -11.70594%
- resnext50_32x4d, ASGD, cuda, default: +11.62242%
- detectron2_fasterrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +11.38885%
- detectron2_fasterrcnn_r_50_c4, Rprop, cuda, (pt2) default: -24.54690%
- detectron2_fasterrcnn_r_50_c4, Rprop, cuda, default: -23.70392%
- detectron2_fasterrcnn_r_50_c4, Rprop, cuda, maximize: -29.02776%
- resnet50_quantized_qat, ASGD, cuda, maximize: +21.82345%
- timm_vision_transformer, Adam, cuda, (pt2) default: -10.72565%
- timm_vision_transformer, SGD, cuda, no_foreach: +13.68268%
- maml_omniglot, Adagrad, cuda, (pt2) maximize: +10.11173%
- maml_omniglot, Adam, cuda, foreach, maximize, capturable, amsgrad: +19.16425%
- dcgan, Adagrad, cuda, (pt2) foreach: -11.14975%
- dcgan, AdamW, cuda, foreach, maximize, capturable, amsgrad: -20.94614%
- dcgan, Adamax, cuda, (pt2) maximize: -12.48798%
- dcgan, Rprop, cuda, maximize: -26.97705%
- dcgan, Rprop, cuda, foreach: -20.43218%
- dcgan, RMSprop, cuda, default: -12.63642%
- detectron2_fasterrcnn_r_50_fpn, Adamax, cuda, foreach: -10.82519%
- detectron2_fasterrcnn_r_50_fpn, Rprop, cuda, (pt2) default: -27.84361%
- hf_Bart, Adagrad, cuda, default: -21.86626%
- hf_Bart, ASGD, cuda, (pt2) foreach: +10.62810%
- pytorch_CycleGAN_and_pix2pix, RAdam, cuda, foreach: +12.05411%
- resnet152, AdamW, cuda, amsgrad, maximize: +10.61317%
- detectron2_maskrcnn_r_101_fpn, ASGD, cuda, (pt2) maximize: -29.52622%
- detectron2_maskrcnn_r_101_fpn, ASGD, cuda, maximize: -24.85662%
- detectron2_maskrcnn_r_101_fpn, Rprop, cuda, (pt2) maximize: +20.18690%
- basic_gnn_sage, Adam, cuda, foreach, maximize, capturable: -11.64165%
- basic_gnn_sage, AdamW, cuda, (pt2) foreach: -10.29728%
- basic_gnn_sage, AdamW, cuda, fused: -11.31960%
- basic_gnn_sage, Adamax, cuda, (pt2) foreach: +10.51443%
- basic_gnn_sage, SGD, cuda, (pt2) foreach: +29.63014%
- basic_gnn_sage, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +51.44615%
- basic_gnn_sage, RAdam, cuda, foreach: -19.92461%
- basic_gnn_sage, Rprop, cuda, default: +16.74887%
- basic_gnn_sage, RMSprop, cuda, default: -11.27915%
- basic_gnn_sage, RMSprop, cuda, maximize: -16.44336%
- basic_gnn_sage, RMSprop, cuda, foreach: -16.47061%
- fastNLP_Bert, Adagrad, cuda, (pt2) foreach: -11.82513%
- fastNLP_Bert, ASGD, cuda, (pt2) default: -19.66079%
- fastNLP_Bert, ASGD, cuda, default: -16.44422%
- fastNLP_Bert, Rprop, cuda, maximize: +13.11318%
- hf_T5, Adamax, cuda, default: -13.94056%
- hf_T5, ASGD, cuda, default: -13.89447%
- detectron2_maskrcnn_r_50_fpn, RAdam, cuda, (pt2) foreach: -13.80514%
- attention_is_all_you_need_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable: -13.30341%
- attention_is_all_you_need_pytorch, AdamW, cuda, default: -11.33796%
- attention_is_all_you_need_pytorch, ASGD, cuda, (pt2) maximize: -20.82695%
- attention_is_all_you_need_pytorch, ASGD, cuda, (pt2) foreach: -14.90072%
- attention_is_all_you_need_pytorch, RAdam, cuda, (pt2) default: -10.28897%
- attention_is_all_you_need_pytorch, RAdam, cuda, (pt2) foreach: -15.64493%
- detectron2_fasterrcnn_r_101_dc5, Adamax, cuda, (pt2) maximize: -15.49404%
- hf_Reformer, Adam, cuda, (pt2) foreach: -13.37742%
- hf_Reformer, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -17.76225%
- hf_Reformer, AdamW, cuda, (pt2) default: -11.05009%
- hf_Reformer, Adamax, cuda, (pt2) default: -19.42769%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9: -79.93768%
- hf_Reformer, RAdam, cuda, (pt2) default: -13.25600%
- pytorch_stargan, AdamW, cuda, default: -13.08601%
- tacotron2, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +15.58870%
- detectron2_fasterrcnn_r_101_fpn, Adagrad, cuda, maximize: -25.29280%
- detectron2_fasterrcnn_r_101_fpn, RAdam, cuda, (pt2) default: -15.81657%
- detectron2_maskrcnn, Adam, cuda, (pt2) foreach, maximize, capturable: -14.52487%
- detectron2_maskrcnn, ASGD, cuda, (pt2) default: -30.51490%
- detectron2_maskrcnn, ASGD, cuda, foreach: -10.32753%
- detectron2_maskrcnn, Rprop, cuda, (pt2) foreach: -10.47181%
- timm_efficientdet, AdamW, cuda, amsgrad, maximize: +12.02063%
- timm_efficientdet, ASGD, cuda, (pt2) foreach: -11.83748%
- hf_Bert, Adagrad, cuda, default: -10.03686%
- hf_Bert, ASGD, cuda, (pt2) foreach: -18.30955%
- DALLE2_pytorch, Adagrad, cuda, (pt2) maximize: +17.51612%
- DALLE2_pytorch, Adagrad, cuda, foreach: -13.34157%
- DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +42.18940%
- DALLE2_pytorch, Adamax, cuda, foreach: -11.42276%
- DALLE2_pytorch, ASGD, cuda, (pt2) default: +22.86124%
- DALLE2_pytorch, ASGD, cuda, default: +28.06001%
- DALLE2_pytorch, ASGD, cuda, maximize: +47.15444%
- DALLE2_pytorch, RAdam, cuda, default: +16.04534%
- DALLE2_pytorch, RAdam, cuda, (pt2) foreach: +15.72546%
- DALLE2_pytorch, Rprop, cuda, maximize: +10.48051%
- detectron2_fasterrcnn_r_101_c4, ASGD, cuda, foreach: +16.03458%
- timm_efficientnet, Adadelta, cuda, default: +10.02855%
- timm_efficientnet, Adadelta, cuda, maximize: +10.18581%
- timm_efficientnet, Adadelta, cuda, foreach: +10.37548%
- basic_gnn_gin, Adadelta, cuda, maximize: -16.15098%
- basic_gnn_gin, Adadelta, cuda, foreach: -16.10535%
- basic_gnn_gin, Adagrad, cuda, default: -28.78077%
- basic_gnn_gin, Adam, cuda, (pt2) foreach: +10.38736%
- basic_gnn_gin, Adamax, cuda, (pt2) default: -21.41468%
- basic_gnn_gin, Adamax, cuda, (pt2) foreach: -21.18892%
- basic_gnn_gin, SGD, cuda, (pt2) maximize: +11.71315%
- basic_gnn_gin, RMSprop, cuda, maximize: -10.53477%
- basic_gnn_gin, NAdam, cuda, default: -17.80739%
- densenet121, Adagrad, cuda, (pt2) default: +43.34466%
- densenet121, Adagrad, cuda, maximize: -14.35915%
- densenet121, Adam, cuda, (pt2) default: +15.76329%
- densenet121, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +24.02817%
- densenet121, Adam, cuda, foreach, maximize, capturable, amsgrad: +14.60496%
- densenet121, Adamax, cuda, (pt2) foreach: +15.62144%
- densenet121, Rprop, cuda, (pt2) maximize: +16.32581%
- doctr_det_predictor, Adagrad, cuda, default: -25.64803%
- doctr_det_predictor, Adagrad, cuda, maximize: +10.33747%
- doctr_det_predictor, RAdam, cuda, (pt2) foreach: +10.82149%
- resnet50, Adamax, cuda, foreach: +10.60714%
- resnet50, ASGD, cuda, (pt2) default: -19.69770%
- resnet50, NAdam, cuda, no_foreach: +17.39944%
- Super_SloMo, AdamW, cuda, (pt2) foreach, maximize, capturable: -12.28276%
- BERT_pytorch, Adadelta, cuda, differentiable: -15.88877%
- BERT_pytorch, AdamW, cuda, (pt2) differentiable: -14.86777%
- BERT_pytorch, Adamax, cuda, (pt2) default: +29.76614%
- BERT_pytorch, Rprop, cuda, maximize: -19.18513%
- BERT_pytorch, RMSprop, cuda, differentiable: -12.07830%
- phlippe_densenet, Adagrad, cuda, (pt2) default: +17.12253%
- phlippe_densenet, ASGD, cuda, foreach: +16.40025%
- yolov3, Adagrad, cuda, maximize: -10.76614%
- yolov3, Adam, cuda, foreach, maximize, capturable, amsgrad: +11.01631%
- yolov3, AdamW, cuda, foreach, maximize, capturable, amsgrad: -12.22652%
- yolov3, Adamax, cuda, (pt2) differentiable: -10.60947%
- yolov3, ASGD, cuda, differentiable: +21.89534%
- yolov3, ASGD, cuda, foreach: +32.95665%
- yolov3, SGD, cuda, (pt2) no_foreach: +15.16500%
- yolov3, RAdam, cuda, no_foreach: +14.42403%
- yolov3, Rprop, cuda, no_foreach: +19.89802%
- opacus_cifar10, Adagrad, cuda, (pt2) maximize: -17.40338%
- opacus_cifar10, AdamW, cuda, foreach, maximize, capturable: -17.16266%
- hf_T5_base, Adagrad, cuda, default: -16.21315%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5053018355

cc @janeyx99


We've severely increased the threshold for noise.