Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on 'f3e13d956729d4fa9ce35dca5beed588c914797d'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 1a3d3669efa55e3360060c9b81f87900ae0c906c

Affected PyTorch commit: f3e13d956729d4fa9ce35dca5beed588c914797d

Affected Tests:
- tts_angular, Adagrad, cuda, (pt2) default: -10.46251%
- tts_angular, Adam, cuda, foreach, maximize, capturable, amsgrad: -11.04357%
- tts_angular, SGD, cuda, (pt2) foreach: +52.06824%
- timm_resnest, Adadelta, cuda, foreach: +11.74446%
- timm_resnest, SGD, cuda, (pt2) foreach, momentum=0.9: +70.71842%
- fambench_xlmr, RAdam, cuda, foreach: +10.87049%
- hf_Longformer, Adadelta, cuda, (pt2) default: +10.76962%
- hf_Longformer, Adamax, cuda, default: +11.14688%
- drq, Adagrad, cuda, (pt2) default: +11.70833%
- attention_is_all_you_need_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +18.16752%
- attention_is_all_you_need_pytorch, Adamax, cuda, default: +13.12034%
- attention_is_all_you_need_pytorch, ASGD, cuda, maximize: +23.92673%
- attention_is_all_you_need_pytorch, RAdam, cuda, default: +10.46599%
- attention_is_all_you_need_pytorch, Rprop, cuda, (pt2) maximize: +32.44805%
- yolov3, Adagrad, cuda, (pt2) maximize: +19.16331%
- yolov3, AdamW, cuda, (pt2) differentiable: +10.70490%
- yolov3, AdamW, cuda, (pt2) foreach: +12.02235%
- yolov3, AdamW, cuda, foreach, maximize, capturable, amsgrad: +15.91373%
- yolov3, Adamax, cuda, no_foreach: +11.33158%
- detectron2_fasterrcnn_r_101_dc5, Adamax, cuda, (pt2) maximize: +18.00754%
- functorch_dp_cifar10, Adam, cuda, (pt2) default: -11.65951%
- functorch_dp_cifar10, AdamW, cuda, (pt2) foreach, maximize, capturable: +10.57721%
- functorch_dp_cifar10, Adamax, cuda, foreach: +16.57436%
- functorch_dp_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9: +17.94764%
- DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable: -10.31806%
- DALLE2_pytorch, AdamW, cuda, foreach: -13.69122%
- DALLE2_pytorch, AdamW, cuda, (pt2) foreach, maximize, capturable: +14.20550%
- DALLE2_pytorch, Adamax, cuda, (pt2) maximize: +12.95911%
- DALLE2_pytorch, ASGD, cuda, (pt2) default: +13.62224%
- mobilenet_v3_large, Adam, cuda, (pt2) foreach, maximize, capturable: +16.52068%
- lennard_jones, Adam, cuda, (pt2) default: +15.94807%
- lennard_jones, AdamW, cuda, fused, capturable, amsgrad: +15.21457%
- lennard_jones, SGD, cuda, (pt2) foreach: -23.19778%
- moco, Adamax, cuda, default: +10.47802%
- hf_Bert_large, ASGD, cuda, maximize: +11.16883%
- hf_Bert_large, ASGD, cuda, foreach: +40.30873%
- demucs, Adadelta, cuda, (pt2) default: +12.98533%
- demucs, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -11.53270%
- demucs, SGD, cuda, (pt2) foreach, momentum=0.9: +12.75307%
- detectron2_fasterrcnn_r_50_fpn, Adadelta, cuda, (pt2) foreach: +10.21988%
- detectron2_fasterrcnn_r_50_fpn, Adagrad, cuda, (pt2) maximize: +27.79988%
- detectron2_fasterrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +25.22801%
- detectron2_fasterrcnn_r_50_fpn, Rprop, cuda, foreach: +18.96838%
- resnet18, Adam, cuda, (pt2) amsgrad, maximize: +11.65925%
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +12.17278%
- pytorch_stargan, Adagrad, cuda, (pt2) maximize: -13.05234%
- opacus_cifar10, Adagrad, cuda, default: -12.08689%
- opacus_cifar10, Adagrad, cuda, foreach: -12.45665%
- opacus_cifar10, Adam, cuda, (pt2) amsgrad, maximize: +16.90901%
- opacus_cifar10, Adam, cuda, (pt2) fused, amsgrad, maximize: +11.60628%
- detectron2_maskrcnn_r_50_c4, Adagrad, cuda, (pt2) foreach: -15.37595%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +26.97115%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +20.63903%
- timm_vision_transformer, Adadelta, cuda, (pt2) default: +13.82816%
- timm_vision_transformer, Adadelta, cuda, (pt2) maximize: -10.93112%
- timm_vision_transformer, Adadelta, cuda, (pt2) no_foreach: -10.17750%
- timm_vision_transformer, Adam, cuda, (pt2) differentiable: +12.17247%
- timm_vision_transformer, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +11.07975%
- timm_vision_transformer, ASGD, cuda, no_foreach: -12.47197%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -36.93441%
- doctr_reco_predictor, Adadelta, cuda, (pt2) default: +11.21220%
- doctr_reco_predictor, Adamax, cuda, (pt2) default: +11.68022%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, maximize: +10.77886%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, foreach: +12.20168%
- LearningToPaint, Adadelta, cuda, (pt2) foreach: -10.43314%
- LearningToPaint, Adagrad, cuda, foreach: +33.21967%
- LearningToPaint, Adam, cuda, default: +10.07031%
- LearningToPaint, Adam, cuda, (pt2) foreach, maximize, capturable: +17.32920%
- detectron2_fasterrcnn_r_101_c4, Adadelta, cuda, (pt2) default: -12.45531%
- detectron2_fasterrcnn_r_101_c4, Adagrad, cuda, (pt2) foreach: +31.17623%
- detectron2_fasterrcnn_r_101_c4, Adam, cuda, (pt2) foreach, maximize, capturable: +15.55113%
- detectron2_fasterrcnn_r_101_c4, Rprop, cuda, (pt2) default: +22.47928%
- shufflenet_v2_x1_0, Adagrad, cuda, (pt2) default: -13.72542%
- shufflenet_v2_x1_0, Adagrad, cuda, (pt2) maximize: -10.82730%
- shufflenet_v2_x1_0, AdamW, cuda, amsgrad, maximize: +11.54402%
- shufflenet_v2_x1_0, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +12.92823%
- hf_Albert, AdamW, cuda, foreach, maximize, capturable: +11.40469%
- functorch_maml_omniglot, Adam, cuda, (pt2) foreach: +13.35354%
- BERT_pytorch, Adadelta, cuda, maximize: +10.81568%
- BERT_pytorch, Adagrad, cuda, (pt2) differentiable: +13.52833%
- BERT_pytorch, Adam, cuda, no_foreach: +22.33205%
- BERT_pytorch, AdamW, cuda, (pt2) no_foreach: +10.30382%
- BERT_pytorch, AdamW, cuda, (pt2) foreach: +10.24822%
- BERT_pytorch, Adamax, cuda, (pt2) differentiable: +17.38904%
- BERT_pytorch, SGD, cuda, no_foreach: -11.19230%
- BERT_pytorch, Rprop, cuda, default: +41.47950%
- BERT_pytorch, NAdam, cuda, (pt2) no_foreach: +29.89862%
- BERT_pytorch, NAdam, cuda, (pt2) differentiable: +12.83842%
- hf_T5_large, Adagrad, cuda, no_foreach: +11.40383%
- mobilenet_v2_quantized_qat, Adadelta, cuda, (pt2) maximize: -10.50446%
- mobilenet_v2_quantized_qat, Adagrad, cuda, (pt2) foreach: -11.25243%
- basic_gnn_sage, Adadelta, cuda, (pt2) foreach: +28.85868%
- basic_gnn_sage, Adagrad, cuda, (pt2) foreach: +22.66569%
- basic_gnn_sage, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +16.10882%
- basic_gnn_sage, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -28.60134%
- basic_gnn_sage, RAdam, cuda, (pt2) default: +11.54619%
- basic_gnn_sage, RMSprop, cuda, foreach: +23.27570%
- llama, Adam, cuda, (pt2) fused: +10.20845%
- llama, Adamax, cuda, (pt2) foreach: +13.53166%
- llama, Adamax, cuda, foreach: +16.88973%
- llama, SGD, cuda, default: +12.41492%
- llama, SGD, cuda, maximize: +11.32208%
- llama, SGD, cuda, foreach: +17.29112%
- soft_actor_critic, Adadelta, cuda, (pt2) maximize: -13.88616%
- soft_actor_critic, Adam, cuda, (pt2) fused, capturable: +12.07351%
- soft_actor_critic, ASGD, cuda, (pt2) default: +17.14380%
- soft_actor_critic, ASGD, cuda, (pt2) maximize: +15.88123%
- soft_actor_critic, SGD, cuda, (pt2) default: +28.11855%
- soft_actor_critic, SGD, cuda, (pt2) maximize: +24.09511%
- hf_DistilBert, Adadelta, cuda, (pt2) maximize: +11.12279%
- hf_DistilBert, Adamax, cuda, (pt2) default: +10.46481%
- torchrec_dlrm, Adagrad, cuda, (pt2) foreach: +14.69701%
- basic_gnn_gin, Adadelta, cuda, (pt2) default: +16.05748%
- basic_gnn_gin, Adadelta, cuda, (pt2) maximize: +10.23072%
- basic_gnn_gin, Adagrad, cuda, (pt2) maximize: +11.07926%
- basic_gnn_gin, Adam, cuda, (pt2) amsgrad, maximize: +19.94562%
- basic_gnn_gin, SGD, cuda, (pt2) foreach: +26.94995%
- basic_gnn_gin, Rprop, cuda, default: +50.18351%
- doctr_det_predictor, Adadelta, cuda, (pt2) maximize: +13.42525%
- doctr_det_predictor, Adam, cuda, (pt2) fused: -11.34177%
- doctr_det_predictor, Rprop, cuda, (pt2) foreach: +24.06454%
- hf_GPT2_large, NAdam, cuda, default: +10.66186%
- hf_GPT2_large, NAdam, cuda, foreach: -14.48495%
- hf_T5, ASGD, cuda, (pt2) default: +21.00818%
- hf_T5, Rprop, cuda, default: +32.01358%
- basic_gnn_edgecnn, Adam, cuda, (pt2) foreach: +12.20459%
- basic_gnn_edgecnn, Adam, cuda, (pt2) fused, capturable, amsgrad: -10.26099%
- basic_gnn_edgecnn, RAdam, cuda, (pt2) default: +19.65621%
- timm_efficientdet, Adadelta, cuda, (pt2) default: -13.97209%
- timm_efficientdet, Adam, cuda, foreach: -11.30912%
- timm_efficientdet, AdamW, cuda, foreach, maximize, capturable, amsgrad: +14.73630%
- hf_T5_base, ASGD, cuda, (pt2) default: +23.83899%
- hf_T5_base, Rprop, cuda, maximize: +10.88302%
- detectron2_fasterrcnn_r_50_dc5, Adagrad, cuda, (pt2) default: -10.38151%
- detectron2_fasterrcnn_r_50_dc5, Adagrad, cuda, (pt2) maximize: +13.04144%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +12.32768%
- timm_vovnet, Adadelta, cuda, (pt2) maximize: -11.65793%
- timm_vovnet, Adadelta, cuda, (pt2) foreach: -11.11922%
- timm_vovnet, Adam, cuda, (pt2) fused, capturable: -11.31555%
- timm_vovnet, Adamax, cuda, maximize: +12.58324%
- resnet50_quantized_qat, Adagrad, cuda, foreach: -14.02534%
- detectron2_maskrcnn_r_101_c4, Adadelta, cuda, (pt2) maximize: +12.28325%
- detectron2_maskrcnn_r_101_c4, Adagrad, cuda, (pt2) default: +10.48974%
- detectron2_maskrcnn_r_101_c4, Adamax, cuda, maximize: +15.65151%
- detectron2_maskrcnn_r_101_c4, Adamax, cuda, foreach: +29.96370%
- detectron2_maskrcnn_r_101_c4, Rprop, cuda, (pt2) foreach: +16.18244%
- pytorch_struct, Adagrad, cuda, (pt2) maximize: +14.78066%
- pytorch_struct, Adagrad, cuda, (pt2) foreach: +10.20707%
- pytorch_struct, Adam, cuda, (pt2) default: +13.07405%
- pytorch_struct, Adam, cuda, (pt2) foreach: -11.06076%
- detectron2_maskrcnn, Adam, cuda, foreach, maximize, capturable, amsgrad: +10.63017%
- detectron2_maskrcnn, Adamax, cuda, (pt2) default: +21.44766%
- detectron2_maskrcnn, Adamax, cuda, (pt2) maximize: +15.58823%
- detectron2_maskrcnn, Adamax, cuda, (pt2) foreach: +20.42230%
- detectron2_maskrcnn, ASGD, cuda, (pt2) default: +51.35736%
- detectron2_maskrcnn, ASGD, cuda, foreach: +38.27164%
- detectron2_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: +113.82209%
- vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +102.36813%
- vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: +303.34670%
- vision_maskrcnn, Rprop, cuda, (pt2) foreach: +12.31371%
- resnet50, SGD, cuda, no_foreach: -13.69982%
- resnet50, SGD, cuda, (pt2) differentiable: +13.01114%
- resnet50, Rprop, cuda, (pt2) default: +10.57720%
- resnet50, Rprop, cuda, (pt2) no_foreach: +10.32224%
- basic_gnn_gcn, Adadelta, cuda, (pt2) default: +11.70343%
- basic_gnn_gcn, AdamW, cuda, (pt2) amsgrad, maximize: +15.90780%
- basic_gnn_gcn, SGD, cuda, (pt2) default: +29.28198%
- basic_gnn_gcn, SGD, cuda, (pt2) maximize: +19.46360%
- basic_gnn_gcn, Rprop, cuda, default: +16.26707%
- detectron2_fasterrcnn_r_101_fpn, Adadelta, cuda, (pt2) default: +11.16024%
- detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, (pt2) default: +12.49722%
- detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, (pt2) maximize: +35.33941%
- detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, foreach: +48.50182%
- timm_nfnet, Adadelta, cuda, (pt2) default: -11.86980%
- timm_nfnet, AdamW, cuda, foreach, maximize, capturable: +11.13546%
- timm_efficientnet, Adadelta, cuda, (pt2) default: -11.21506%
- dcgan, Adagrad, cuda, default: +22.52012%
- dcgan, Adam, cuda, (pt2) amsgrad, maximize: +10.01100%
- dcgan, ASGD, cuda, (pt2) foreach: +18.25329%
- dcgan, SGD, cuda, (pt2) foreach, momentum=0.9: +11.24113%
- dcgan, Rprop, cuda, maximize: +20.19830%
- dcgan, NAdam, cuda, foreach: +10.10435%
- resnext50_32x4d, ASGD, cuda, default: +16.12513%
- maml_omniglot, Adadelta, cuda, default: -10.44347%
- maml_omniglot, Adadelta, cuda, (pt2) maximize: +11.62072%
- squeezenet1_1, Adadelta, cuda, (pt2) default: -11.22805%
- squeezenet1_1, Adagrad, cuda, (pt2) maximize: +10.12932%
- squeezenet1_1, AdamW, cuda, (pt2) fused, capturable, amsgrad: +10.78286%
- squeezenet1_1, NAdam, cuda, (pt2) foreach: +14.15389%
- tacotron2, Adadelta, cuda, (pt2) foreach: +12.67616%
- tacotron2, Adam, cuda, (pt2) amsgrad, maximize: +10.71282%
- tacotron2, Adam, cuda, (pt2) foreach: +20.20838%
- tacotron2, Rprop, cuda, (pt2) foreach: -11.94840%
- detectron2_fasterrcnn_r_50_c4, Adadelta, cuda, (pt2) default: +13.21956%
- detectron2_fasterrcnn_r_50_c4, Adagrad, cuda, foreach: +33.08381%
- detectron2_fasterrcnn_r_50_c4, Adamax, cuda, (pt2) maximize: +11.81436%
- detectron2_fasterrcnn_r_50_c4, ASGD, cuda, maximize: +22.38761%
- Super_SloMo, Adam, cuda, (pt2) foreach, maximize, capturable: +14.70308%
- Super_SloMo, Adamax, cuda, (pt2) default: +15.09632%
- Super_SloMo, Rprop, cuda, default: +14.46986%
- Super_SloMo, Rprop, cuda, (pt2) maximize: +13.92365%
- fastNLP_Bert, Adadelta, cuda, (pt2) maximize: +10.84616%
- phlippe_resnet, Adam, cuda, (pt2) amsgrad, maximize: +10.70157%
- Background_Matting, Adagrad, cuda, default: -10.70374%
- Background_Matting, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +22.07745%
- detectron2_maskrcnn_r_101_fpn, AdamW, cuda, foreach, maximize, capturable, amsgrad: +14.43637%
- detectron2_maskrcnn_r_101_fpn, Adamax, cuda, (pt2) foreach: +26.39369%
- mobilenet_v2, Adagrad, cuda, maximize: +11.75417%
- alexnet, Adam, cuda, (pt2) amsgrad, maximize: +10.88810%
- densenet121, Adagrad, cuda, (pt2) maximize: -12.71072%
- densenet121, AdamW, cuda, (pt2) default: +11.86226%
- densenet121, Adamax, cuda, maximize: +10.96854%
- densenet121, Rprop, cuda, default: +63.98474%
- densenet121, Rprop, cuda, foreach: +11.38508%
- hf_GPT2, Adadelta, cuda, (pt2) default: +12.03175%
- nvidia_deeprecommender, SGD, cuda, (pt2) foreach: +10.78713%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/4987743943

cc @janeyx99


Some things, like 
attention_is_all_you_need_pytorch, Rprop, cuda, (pt2) maximize: +32.44805%
llama
BERT Adam no_foreach

are gradually increasing.

Also note that we have 217 total regressions (173 slowdowns) and 150 of them are pt2 related, so I'm guessing tracing is a culprit.

Closing this, but will keep observing trends