Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '174d01bc939c7bdf390113c75d5ec2ce84cfa1d2'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: ed0910596db6fe7e40eff2d6681535cfe68cad4e

Affected PyTorch commit: 174d01bc939c7bdf390113c75d5ec2ce84cfa1d2

Affected Tests:
- hf_DistilBert, Adagrad, cuda, (pt2) default: +11.44019%
- hf_DistilBert, Adagrad, cuda, foreach: -15.77345%
- hf_DistilBert, ASGD, cuda, (pt2) foreach: -27.21812%
- mobilenet_v2_quantized_qat, Adamax, cuda, default: -13.44763%
- mobilenet_v2_quantized_qat, Rprop, cuda, default: +30.59675%
- pytorch_struct, Adagrad, cuda, (pt2) default: +14.90362%
- pytorch_struct, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +14.85907%
- pytorch_struct, AdamW, cuda, (pt2) foreach: +10.31053%
- cm3leon_generate, Adagrad, cuda, (pt2) maximize: -10.84079%
- timm_vision_transformer, Adagrad, cuda, (pt2) differentiable: -11.31640%
- timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: +12.36779%
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: -10.35291%
- timm_vision_transformer, Rprop, cuda, (pt2) differentiable: -11.60903%
- timm_vision_transformer, NAdam, cuda, (pt2) no_foreach: -11.56540%
- timm_vision_transformer, NAdam, cuda, (pt2) differentiable: -11.04861%
- doctr_reco_predictor, Adadelta, cuda, (pt2) foreach: -11.50001%
- doctr_reco_predictor, Adagrad, cuda, (pt2) foreach: -21.50743%
- doctr_reco_predictor, AdamW, cuda, amsgrad, maximize: -13.46900%
- doctr_reco_predictor, Adamax, cuda, maximize: -14.89705%
- doctr_reco_predictor, ASGD, cuda, foreach: -12.19047%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9: -10.95921%
- doctr_reco_predictor, Rprop, cuda, (pt2) maximize: -10.58682%
- doctr_reco_predictor, Rprop, cuda, (pt2) foreach: -23.04015%
- lennard_jones, Adagrad, cuda, (pt2) maximize: +10.50547%
- lennard_jones, Adam, cuda, (pt2) foreach, maximize, capturable: +11.13280%
- lennard_jones, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -24.61189%
- hf_GPT2, Adagrad, cuda, (pt2) default: +11.31336%
- hf_GPT2, Adagrad, cuda, default: +26.96005%
- hf_GPT2, Adagrad, cuda, foreach: +16.79542%
- hf_GPT2, Adamax, cuda, default: +10.44057%
- hf_GPT2, Adamax, cuda, (pt2) maximize: -11.54638%
- hf_GPT2, ASGD, cuda, (pt2) foreach: -11.13298%
- tacotron2, Adadelta, cuda, (pt2) foreach: +11.21368%
- tacotron2, Adam, cuda, foreach, maximize, capturable: +11.19995%
- dcgan, SGD, cuda, (pt2) foreach, momentum=0.9: +66.67345%
- resnet50, Adagrad, cuda, (pt2) maximize: +13.54261%
- resnet50, Adam, cuda, (pt2) foreach, maximize, capturable: +13.13301%
- resnet50, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -10.35943%
- resnet50, Adamax, cuda, (pt2) no_foreach: +14.65233%
- resnet50, RAdam, cuda, (pt2) no_foreach: +10.00015%
- resnet50, RMSprop, cuda, (pt2) no_foreach: +11.97915%
- functorch_maml_omniglot, ASGD, cuda, (pt2) maximize: +14.09178%
- detectron2_fasterrcnn_r_50_c4, Adagrad, cuda, (pt2) default: +13.99402%
- detectron2_fasterrcnn_r_50_c4, Adamax, cuda, foreach: +17.10353%
- resnet50_quantized_qat, Rprop, cuda, (pt2) default: +11.60758%
- timm_resnest, Adadelta, cuda, foreach: +13.89789%
- timm_resnest, Adamax, cuda, (pt2) foreach: +17.16982%
- timm_resnest, ASGD, cuda, maximize: -15.76468%
- timm_resnest, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +10.25832%
- resnet18, Adagrad, cuda, (pt2) maximize: +10.94371%
- resnet18, Adam, cuda, (pt2) default: +12.99758%
- squeezenet1_1, AdamW, cuda, (pt2) fused, capturable, amsgrad: +14.34188%
- squeezenet1_1, ASGD, cuda, maximize: +10.33168%
- nvidia_deeprecommender, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -16.94127%
- opacus_cifar10, Adagrad, cuda, (pt2) maximize: -11.66605%
- opacus_cifar10, Adagrad, cuda, (pt2) foreach: +18.17750%
- hf_Bart, Rprop, cuda, (pt2) foreach: +12.80568%
- BERT_pytorch, Adadelta, cuda, differentiable: +12.46660%
- BERT_pytorch, Adagrad, cuda, default: -26.29347%
- BERT_pytorch, Adagrad, cuda, no_foreach: -10.52090%
- BERT_pytorch, Adam, cuda, no_foreach: -25.69157%
- BERT_pytorch, AdamW, cuda, no_foreach: -11.61005%
- BERT_pytorch, Adamax, cuda, (pt2) foreach: -12.69868%
- BERT_pytorch, ASGD, cuda, default: -24.98338%
- BERT_pytorch, ASGD, cuda, (pt2) maximize: -14.87003%
- BERT_pytorch, SGD, cuda, no_foreach: +11.31799%
- BERT_pytorch, SGD, cuda, (pt2) foreach: +11.47848%
- BERT_pytorch, RAdam, cuda, no_foreach: -11.92768%
- BERT_pytorch, RMSprop, cuda, (pt2) no_foreach: -10.32308%
- BERT_pytorch, RMSprop, cuda, no_foreach: -22.29748%
- BERT_pytorch, NAdam, cuda, (pt2) no_foreach: +10.24669%
- BERT_pytorch, NAdam, cuda, differentiable: -14.80776%
- yolov3, Adagrad, cuda, (pt2) foreach: +18.16726%
- yolov3, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +12.89068%
- yolov3, ASGD, cuda, (pt2) default: +12.17468%
- yolov3, ASGD, cuda, foreach: +14.05925%
- mobilenet_v3_large, Adadelta, cuda, (pt2) foreach: -10.41295%
- mobilenet_v3_large, Adamax, cuda, (pt2) maximize: -16.67920%
- Background_Matting, Adagrad, cuda, default: -15.87330%
- Background_Matting, Adagrad, cuda, foreach: -14.26189%
- Background_Matting, AdamW, cuda, foreach, maximize, capturable, amsgrad: -10.82371%
- Background_Matting, Adamax, cuda, maximize: +27.73143%
- Background_Matting, Adamax, cuda, (pt2) foreach: -27.79918%
- Background_Matting, Rprop, cuda, default: -17.47232%
- Background_Matting, Rprop, cuda, (pt2) foreach: -26.06122%
- Background_Matting, NAdam, cuda, (pt2) default: -14.61613%
- Background_Matting, NAdam, cuda, (pt2) foreach: -19.34629%
- detectron2_maskrcnn, Adadelta, cuda, (pt2) foreach: +10.44050%
- detectron2_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +98.42973%
- hf_Reformer, Adadelta, cuda, (pt2) default: -10.32564%
- hf_Reformer, Adam, cuda, (pt2) foreach, maximize, capturable: +11.08907%
- hf_Reformer, AdamW, cuda, (pt2) default: +11.12443%
- hf_Reformer, Adamax, cuda, maximize: -21.25780%
- hf_Reformer, ASGD, cuda, (pt2) default: +13.51248%
- detectron2_fasterrcnn_r_50_fpn, Adagrad, cuda, (pt2) default: -13.85905%
- detectron2_fasterrcnn_r_50_fpn, Adam, cuda, (pt2) foreach: +11.86373%
- detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, default: +29.73257%
- detectron2_fasterrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +224.57796%
- detectron2_fasterrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9: -31.63412%
- densenet121, Adagrad, cuda, (pt2) maximize: -23.05084%
- densenet121, Adamax, cuda, (pt2) default: +13.90762%
- densenet121, ASGD, cuda, (pt2) foreach: -11.81781%
- hf_GPT2_large, AdamW, cuda, foreach, maximize, capturable: +11.28527%
- hf_GPT2_large, RAdam, cuda, no_foreach: -12.13548%
- vision_maskrcnn, Adadelta, cuda, (pt2) maximize: +10.26731%
- vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: +200.17893%
- Super_SloMo, Adagrad, cuda, (pt2) maximize: +10.06024%
- Super_SloMo, Adamax, cuda, default: -21.61038%
- Super_SloMo, Adamax, cuda, maximize: -11.99506%
- Super_SloMo, Rprop, cuda, (pt2) default: +11.34098%
- phlippe_resnet, Adagrad, cuda, (pt2) default: +11.06192%
- phlippe_resnet, Adam, cuda, default: -10.42449%
- phlippe_resnet, Adam, cuda, (pt2) amsgrad, maximize: +10.83352%
- phlippe_resnet, ASGD, cuda, (pt2) default: -11.98517%
- phlippe_resnet, RAdam, cuda, foreach: -12.69618%
- mnasnet1_0, Adadelta, cuda, (pt2) default: +12.02650%
- mnasnet1_0, Adadelta, cuda, (pt2) foreach: +14.12543%
- mnasnet1_0, Adagrad, cuda, maximize: -15.92132%
- mnasnet1_0, Adagrad, cuda, (pt2) foreach: -11.23017%
- mnasnet1_0, Adamax, cuda, (pt2) maximize: -19.36675%
- mnasnet1_0, ASGD, cuda, (pt2) maximize: +29.59627%
- mnasnet1_0, Rprop, cuda, foreach: -21.76390%
- doctr_det_predictor, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -11.43679%
- doctr_det_predictor, RAdam, cuda, (pt2) foreach: -11.41263%
- timm_regnet, Adadelta, cuda, (pt2) default: -10.67494%
- soft_actor_critic, Adadelta, cuda, (pt2) foreach: -10.23380%
- soft_actor_critic, Adagrad, cuda, (pt2) maximize: +11.09811%
- soft_actor_critic, NAdam, cuda, (pt2) default: +13.41758%
- detectron2_maskrcnn_r_101_fpn, Rprop, cuda, maximize: -19.14393%
- detectron2_maskrcnn_r_50_fpn, Adadelta, cuda, (pt2) foreach: -10.12242%
- detectron2_maskrcnn_r_50_fpn, Adagrad, cuda, (pt2) default: -17.50401%
- detectron2_maskrcnn_r_50_fpn, Adam, cuda, (pt2) default: +10.12975%
- detectron2_maskrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9: +103.17949%
- detectron2_maskrcnn_r_50_fpn, Rprop, cuda, foreach: +15.87509%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, (pt2) default: +10.64164%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, maximize: -11.29587%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cuda, (pt2) foreach: +11.48239%
- pytorch_CycleGAN_and_pix2pix, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +15.36373%
- pytorch_CycleGAN_and_pix2pix, Adamax, cuda, (pt2) foreach: +19.69543%
- hf_Bert_large, Rprop, cuda, foreach: -16.62718%
- detectron2_fasterrcnn_r_101_c4, Adadelta, cuda, (pt2) foreach: +12.08974%
- detectron2_fasterrcnn_r_101_c4, Rprop, cuda, (pt2) default: -31.46893%
- detectron2_fasterrcnn_r_101_c4, Rprop, cuda, default: +10.07354%
- timm_vovnet, Adadelta, cuda, (pt2) foreach: +14.60306%
- timm_vovnet, Adagrad, cuda, (pt2) default: +13.89099%
- timm_vovnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +10.03495%
- timm_nfnet, Adadelta, cuda, (pt2) default: +11.45996%
- timm_nfnet, Adagrad, cuda, (pt2) maximize: +11.30892%
- timm_nfnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +11.60591%
- timm_nfnet, AdamW, cuda, (pt2) foreach, maximize, capturable: +10.44573%
- hf_T5, Adagrad, cuda, maximize: +17.81140%
- hf_T5, Adagrad, cuda, foreach: -11.83294%
- hf_T5, ASGD, cuda, (pt2) default: -24.96334%
- hf_T5, ASGD, cuda, (pt2) maximize: -13.50209%
- hf_T5, ASGD, cuda, (pt2) foreach: -22.83463%
- hf_T5, Rprop, cuda, foreach: +10.53935%
- moco, Adagrad, cuda, (pt2) default: +12.70790%
- moco, Adam, cuda, foreach, maximize, capturable: +10.00313%
- basic_gnn_gin, Adadelta, cuda, default: -12.66464%
- basic_gnn_gin, Adagrad, cuda, (pt2) foreach: +13.04267%
- basic_gnn_gin, Adam, cuda, (pt2) default: +12.06320%
- basic_gnn_gin, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -13.91200%
- phlippe_densenet, Adagrad, cuda, (pt2) default: -15.55310%
- phlippe_densenet, Adagrad, cuda, maximize: -18.31282%
- phlippe_densenet, Adagrad, cuda, (pt2) foreach: -24.67811%
- phlippe_densenet, ASGD, cuda, (pt2) maximize: -13.71857%
- phlippe_densenet, Rprop, cuda, (pt2) default: +10.49679%
- phlippe_densenet, Rprop, cuda, maximize: +17.37264%
- detectron2_fasterrcnn_r_101_fpn, Adadelta, cuda, (pt2) foreach: +10.21761%
- detectron2_fasterrcnn_r_101_fpn, Adamax, cuda, (pt2) foreach: +13.43584%
- detectron2_fasterrcnn_r_101_fpn, ASGD, cuda, (pt2) foreach: +19.59154%
- detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, (pt2) maximize: -11.73692%
- fastNLP_Bert, Adamax, cuda, (pt2) maximize: +10.27269%
- resnet152, Adagrad, cuda, (pt2) foreach: +13.18992%
- resnet152, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -16.32786%
- DALLE2_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable: +11.24682%
- DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable: +15.59484%
- DALLE2_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +11.97726%
- DALLE2_pytorch, Adam, cuda, foreach, maximize, capturable, amsgrad: +10.38102%
- DALLE2_pytorch, Adam, cuda, fused, amsgrad, maximize: -29.48309%
- DALLE2_pytorch, AdamW, cuda, foreach, maximize, capturable, amsgrad: +13.53641%
- DALLE2_pytorch, ASGD, cuda, (pt2) maximize: +10.83251%
- DALLE2_pytorch, ASGD, cuda, maximize: -19.81649%
- DALLE2_pytorch, Rprop, cuda, foreach: -17.91527%
- functorch_dp_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -15.32707%
- functorch_dp_cifar10, Rprop, cuda, (pt2) foreach: +40.27380%
- nanogpt_generate, Adam, cuda, (pt2) amsgrad, maximize: +11.48019%
- nanogpt_generate, ASGD, cuda, default: -17.19729%
- pytorch_unet, Adagrad, cuda, default: +13.68395%
- pytorch_unet, Adagrad, cuda, (pt2) maximize: +11.81237%
- pytorch_unet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +15.78766%
- pytorch_unet, Rprop, cuda, (pt2) default: +12.04401%
- basic_gnn_sage, ASGD, cuda, foreach: -16.38525%
- basic_gnn_sage, RMSprop, cuda, maximize: -17.95595%
- speech_transformer, Adagrad, cuda, maximize: -11.40314%
- LearningToPaint, Adam, cuda, (pt2) foreach, maximize, capturable: -11.58159%
- LearningToPaint, AdamW, cuda, foreach: -13.17162%
- LearningToPaint, Adamax, cuda, default: +10.60830%
- LearningToPaint, ASGD, cuda, (pt2) foreach: -34.73130%
- LearningToPaint, Rprop, cuda, (pt2) default: -30.26255%
- LearningToPaint, Rprop, cuda, foreach: +14.32854%
- LearningToPaint, NAdam, cuda, (pt2) default: -10.79366%
- mobilenet_v2, Adadelta, cuda, maximize: +12.76338%
- mobilenet_v2, Adam, cuda, (pt2) foreach: +15.42330%
- mobilenet_v2, AdamW, cuda, (pt2) foreach, maximize, capturable: -16.71741%
- mobilenet_v2, Rprop, cuda, default: +12.20731%
- mobilenet_v2, Rprop, cuda, (pt2) foreach: -14.85331%
- alexnet, Adadelta, cuda, (pt2) foreach: -10.84682%
- alexnet, Adagrad, cuda, (pt2) foreach: +15.83993%
- alexnet, Adam, cuda, (pt2) default: +10.58798%
- shufflenet_v2_x1_0, Adagrad, cuda, maximize: -11.84846%
- shufflenet_v2_x1_0, Rprop, cuda, (pt2) maximize: +12.93627%
- hf_T5_base, ASGD, cuda, default: -18.34828%
- attention_is_all_you_need_pytorch, Adagrad, cuda, (pt2) default: -14.97689%
- attention_is_all_you_need_pytorch, Adagrad, cuda, (pt2) maximize: -16.31889%
- attention_is_all_you_need_pytorch, Adagrad, cuda, (pt2) foreach: -10.18056%
- attention_is_all_you_need_pytorch, AdamW, cuda, (pt2) foreach, maximize, capturable: -14.42609%
- attention_is_all_you_need_pytorch, Adamax, cuda, default: -13.85727%
- attention_is_all_you_need_pytorch, Adamax, cuda, (pt2) foreach: -10.99541%
- basic_gnn_edgecnn, Adadelta, cuda, (pt2) default: -10.32324%
- basic_gnn_edgecnn, Adadelta, cuda, (pt2) foreach: -11.56253%
- basic_gnn_edgecnn, Adam, cuda, (pt2) foreach: +11.12482%
- basic_gnn_edgecnn, Adam, cuda, (pt2) foreach, maximize, capturable: -15.05659%
- basic_gnn_edgecnn, AdamW, cuda, foreach: -15.89943%
- basic_gnn_edgecnn, AdamW, cuda, (pt2) foreach, maximize, capturable: +13.45213%
- resnext50_32x4d, ASGD, cuda, (pt2) default: +19.55949%
- resnext50_32x4d, Rprop, cuda, foreach: +10.43589%
- timm_efficientnet, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +12.08320%
- timm_efficientnet, Rprop, cuda, (pt2) maximize: +18.90358%
- maml, Adadelta, cuda, (pt2) maximize: +10.84283%
- maml, AdamW, cuda, (pt2) fused, capturable, amsgrad: +10.71758%
- maml, Rprop, cuda, (pt2) default: +18.85075%
- hf_Bert, Rprop, cuda, foreach: +19.74067%
- hf_Longformer, Adadelta, cuda, (pt2) default: +11.03451%
- hf_Longformer, Adagrad, cuda, (pt2) default: +12.21079%
- maml_omniglot, Adam, cuda, foreach, maximize, capturable: +17.44950%
- maml_omniglot, AdamW, cuda, foreach: -11.20753%
- maml_omniglot, AdamW, cuda, foreach, maximize, capturable: +10.34224%
- maml_omniglot, AdamW, cuda, (pt2) fused, capturable: +10.60184%
- timm_efficientdet, ASGD, cuda, (pt2) foreach: +11.45630%
- detectron2_fasterrcnn_r_50_dc5, Adam, cuda, (pt2) default: +10.32434%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -19.00145%
- detectron2_maskrcnn_r_50_c4, Adagrad, cuda, (pt2) maximize: +11.59037%
- detectron2_maskrcnn_r_50_c4, Adamax, cuda, default: -11.16120%
- detectron2_maskrcnn_r_50_c4, Rprop, cuda, foreach: -10.73002%
- hf_T5_large, Rprop, cuda, no_foreach: +16.63699%
- hf_Albert, Rprop, cuda, maximize: -14.00619%
- drq, Adadelta, cuda, default: -16.33271%
- drq, Adagrad, cuda, (pt2) foreach: -14.37764%
- drq, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +16.67074%
- drq, Rprop, cuda, default: -30.89799%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5010378467

cc @janeyx99


In the process of determining a better threshold https://docs.google.com/spreadsheets/d/1r3a7UczDofej8p6KhAcbTf8moxnlbyQ1AWZKOo0kbh8/edit#gid=418753160