Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1b6d6e98fed5cdb0a0c2fedd5354dd8b83dd07f1'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: f38a04a5c5ce632ba8b0534e3689e89f98e89d73

Affected PyTorch commit: 1b6d6e98fed5cdb0a0c2fedd5354dd8b83dd07f1

Affected Tests:
- hf_GPT2, AdamW, cuda, (pt2) amsgrad, maximize: +0.07039%
- tts_angular, Adadelta, cpu, (pt2) foreach: +0.17375%
- tts_angular, Adadelta, cpu, foreach: +0.40308%
- tts_angular, Adagrad, cpu, (pt2) foreach: +0.84360%
- tts_angular, Adagrad, cpu, foreach: +0.67404%
- tts_angular, Adam, cpu, (pt2) foreach: +0.28131%
- tts_angular, Adam, cpu, foreach: +0.32040%
- tts_angular, AdamW, cpu, (pt2) foreach: +0.32037%
- tts_angular, NAdam, cpu, (pt2) foreach: +0.28554%
- vision_maskrcnn, ASGD, cuda, (pt2) foreach: +0.09838%
- vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.33821%
- vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: +0.66580%
- timm_regnet, Adam, cuda, (pt2) fused: +0.07160%
- mobilenet_v2, Adam, cuda, (pt2) fused: +0.17635%
- mobilenet_v2, SGD, cuda, (pt2) default: +0.07196%
- mnasnet1_0, SGD, cuda, (pt2) foreach, momentum=0.9: +0.07525%
- pytorch_CycleGAN_and_pix2pix, AdamW, cuda, (pt2) fused, capturable, amsgrad: +0.07018%
- detectron2_fasterrcnn_r_101_fpn, AdamW, cuda, (pt2) fused, capturable, amsgrad: +0.07484%
- detectron2_maskrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.67216%
- detectron2_maskrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9: +2.51042%
- hf_Albert, AdamW, cuda, (pt2) fused, capturable: +0.08247%
- hf_Albert, ASGD, cuda, (pt2) maximize: +0.08054%
- hf_T5_large, Adamax, cuda, default: +0.12739%
- hf_T5_large, RAdam, cuda, (pt2) foreach: +0.10356%
- hf_T5_large, RAdam, cuda, foreach: +0.09227%
- detectron2_fasterrcnn_r_50_c4, Adagrad, cpu, (pt2) foreach: +0.31063%
- detectron2_fasterrcnn_r_50_c4, Adagrad, cpu, foreach: +0.07358%
- detectron2_fasterrcnn_r_50_c4, Adamax, cpu, foreach: +0.08385%
- detectron2_fasterrcnn_r_50_c4, SGD, cpu, (pt2) foreach, momentum=0.9: +0.69527%
- detectron2_fasterrcnn_r_50_c4, RAdam, cpu, (pt2) foreach: +0.14180%
- detectron2_fasterrcnn_r_50_c4, RAdam, cpu, foreach: +0.10900%
- detectron2_fasterrcnn_r_50_c4, NAdam, cpu, (pt2) foreach: +0.08511%
- phlippe_densenet, ASGD, cuda, (pt2) maximize: +0.07766%
- phlippe_densenet, SGD, cuda, (pt2) default: +0.08626%
- drq, Adagrad, cpu, foreach: +0.07903%
- drq, Rprop, cpu, foreach: +0.11160%
- opacus_cifar10, Adadelta, cuda, (pt2) default: +0.07240%
- opacus_cifar10, Adam, cuda, (pt2) default: +0.07104%
- opacus_cifar10, AdamW, cuda, (pt2) fused: +0.08838%
- opacus_cifar10, ASGD, cuda, (pt2) default: +0.08068%
- opacus_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9: +0.07261%
- Background_Matting, SGD, cuda, (pt2) maximize: +0.08072%
- densenet121, RMSprop, cuda, (pt2) default: +0.07402%
- detectron2_fasterrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.11964%
- functorch_maml_omniglot, Adamax, cpu, (pt2) foreach: +0.07635%
- functorch_maml_omniglot, ASGD, cpu, (pt2) foreach: +0.12141%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cpu, (pt2) foreach: +0.31458%
- pytorch_CycleGAN_and_pix2pix, Adagrad, cpu, foreach: +0.67997%
- pytorch_CycleGAN_and_pix2pix, AdamW, cpu, (pt2) foreach: +0.15277%
- pytorch_CycleGAN_and_pix2pix, AdamW, cpu, foreach: +0.08650%
- pytorch_CycleGAN_and_pix2pix, RAdam, cpu, (pt2) foreach: +0.47224%
- pytorch_CycleGAN_and_pix2pix, Rprop, cpu, (pt2) foreach: +0.21103%
- pytorch_CycleGAN_and_pix2pix, Rprop, cpu, foreach: +0.19990%
- pytorch_CycleGAN_and_pix2pix, RMSprop, cpu, (pt2) foreach: +0.17672%
- pytorch_CycleGAN_and_pix2pix, RMSprop, cpu, foreach: +0.44534%
- pytorch_CycleGAN_and_pix2pix, NAdam, cpu, (pt2) foreach: +0.54719%
- pytorch_CycleGAN_and_pix2pix, NAdam, cpu, foreach: +0.21726%
- timm_resnest, Adadelta, cuda, (pt2) maximize: +0.10125%
- timm_resnest, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.20631%
- timm_resnest, SGD, cuda, (pt2) foreach, momentum=0.9: +0.09213%
- detectron2_maskrcnn_r_50_c4, Adagrad, cpu, foreach: +0.07340%
- squeezenet1_1, AdamW, cuda, (pt2) default: +0.07592%
- squeezenet1_1, AdamW, cuda, (pt2) fused, capturable: +0.08291%
- pytorch_stargan, RAdam, cpu, foreach: +0.61501%
- pytorch_stargan, Rprop, cpu, (pt2) foreach: +0.24492%
- BERT_pytorch, Adam, cuda, (pt2) differentiable: +0.08949%
- BERT_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +0.07432%
- BERT_pytorch, AdamW, cuda, (pt2) amsgrad, maximize: +0.09425%
- BERT_pytorch, SGD, cuda, (pt2) foreach: +0.07510%
- detectron2_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +5.40036%
- detectron2_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: +1.34544%
- timm_resnest, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: +0.87385%
- timm_resnest, SGD, cpu, (pt2) foreach, momentum=0.9: +0.44335%
- timm_resnest, Rprop, cpu, (pt2) foreach: +0.09115%
- timm_resnest, Rprop, cpu, foreach: +0.08273%
- functorch_dp_cifar10, RAdam, cpu, (pt2) foreach: +0.07421%
- functorch_dp_cifar10, RAdam, cpu, foreach: +0.12208%
- speech_transformer, Adadelta, cpu, (pt2) foreach: +0.39309%
- speech_transformer, Adagrad, cpu, foreach: +0.34836%
- detectron2_maskrcnn_r_101_c4, Adagrad, cuda, (pt2) maximize: +0.07065%
- hf_T5, SGD, cuda, (pt2) maximize: +0.07013%
- hf_T5, SGD, cuda, (pt2) foreach, momentum=0.9: +0.09545%
- doctr_reco_predictor, Adadelta, cuda, (pt2) foreach: +0.07847%
- doctr_reco_predictor, Adam, cuda, (pt2) foreach, maximize, capturable: +0.07510%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.13974%
- doctr_reco_predictor, SGD, cuda, (pt2) foreach, momentum=0.9: +0.18037%
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: +0.10151%
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.07629%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.08424%
- detectron2_maskrcnn_r_50_c4, SGD, cuda, (pt2) foreach, momentum=0.9: +0.10694%
- maml, SGD, cuda, (pt2) foreach, momentum=0.9: +0.07964%
- hf_Albert, RAdam, cpu, (pt2) foreach: +0.23018%
- hf_Albert, RAdam, cpu, foreach: +0.27274%
- hf_Albert, Rprop, cpu, (pt2) foreach: +0.22452%
- hf_Albert, Rprop, cpu, foreach: +0.21720%
- resnet50, Adam, cuda, (pt2) fused, capturable, amsgrad: +0.08309%
- hf_GPT2_large, RAdam, cuda, default: +0.07619%
- mobilenet_v2_quantized_qat, Adam, cuda, (pt2) fused, capturable, amsgrad: +0.13707%
- hf_Reformer, Adam, cuda, (pt2) fused, capturable: +0.12663%
- timm_vovnet, Adam, cuda, (pt2) foreach: +0.10027%
- mobilenet_v3_large, Adadelta, cuda, (pt2) default: +0.11779%
- tacotron2, Adam, cuda, (pt2) foreach, maximize, capturable: +0.08427%
- tacotron2, AdamW, cuda, (pt2) fused, capturable, amsgrad: +0.07859%
- functorch_maml_omniglot, AdamW, cuda, (pt2) fused, capturable: +0.07132%
- functorch_maml_omniglot, SGD, cuda, (pt2) default: +0.09651%
- timm_vision_transformer, SGD, cuda, maximize: +0.07808%
- timm_vision_transformer, SGD, cuda, differentiable: +0.07240%
- soft_actor_critic, Adam, cuda, (pt2) fused, amsgrad, maximize: +0.11978%
- soft_actor_critic, Adamax, cuda, (pt2) maximize: +0.07905%
- soft_actor_critic, RMSprop, cuda, (pt2) default: +0.08452%
- LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.07725%
- DALLE2_pytorch, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +0.11842%
- LearningToPaint, RAdam, cpu, foreach: +0.07223%
- LearningToPaint, Rprop, cpu, (pt2) foreach: +0.08189%
- LearningToPaint, Rprop, cpu, foreach: +0.10422%
- llama, SGD, cuda, (pt2) foreach, momentum=0.9: +0.99046%
- hf_T5_base, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.07491%
- dcgan, SGD, cuda, (pt2) foreach: +0.07092%
- dcgan, NAdam, cuda, (pt2) default: +0.10265%
- dcgan, Adam, cpu, (pt2) foreach: +0.10825%
- dcgan, Rprop, cpu, (pt2) foreach: +0.13102%
- nvidia_deeprecommender, SGD, cuda, (pt2) foreach, momentum=0.9: +0.20874%
- resnet18, Adagrad, cpu, (pt2) foreach: +0.30363%
- resnet18, Adam, cpu, (pt2) foreach: +0.16045%
- resnet18, RAdam, cpu, foreach: +0.09177%
- torchrec_dlrm, Adam, cpu, (pt2) foreach: +0.24640%
- torchrec_dlrm, NAdam, cpu, foreach: +0.54906%
- drq, AdamW, cuda, (pt2) foreach: +0.09247%
- phlippe_resnet, AdamW, cuda, (pt2) fused, capturable, amsgrad: +0.08873%
- hf_Longformer, ASGD, cuda, foreach: +0.07354%
- phlippe_resnet, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: +0.14449%
- lennard_jones, Adadelta, cuda, (pt2) maximize: +0.11378%
- lennard_jones, Adam, cuda, (pt2) fused, capturable: +0.07257%
- lennard_jones, Adam, cuda, (pt2) fused, capturable, amsgrad: +0.08554%
- lennard_jones, AdamW, cuda, (pt2) fused: +0.07196%
- detectron2_fasterrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9: +0.16062%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.09155%
- detectron2_fasterrcnn_r_50_dc5, SGD, cuda, (pt2) foreach, momentum=0.9: +0.35580%
- demucs, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +0.10858%
- demucs, SGD, cuda, (pt2) foreach, momentum=0.9: +0.10325%
- hf_Bert, SGD, cuda, (pt2) foreach, momentum=0.9: +0.08515%
- opacus_cifar10, Adam, cpu, (pt2) foreach: +0.08684%
- opacus_cifar10, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: +0.22992%
- opacus_cifar10, SGD, cpu, (pt2) foreach, momentum=0.9: +0.18048%
- opacus_cifar10, Rprop, cpu, foreach: +0.17317%
- pytorch_stargan, AdamW, cuda, (pt2) fused, capturable, amsgrad: +0.07901%
- pytorch_stargan, RMSprop, cuda, (pt2) foreach: +0.07242%


Tests that were no longer run on affected commit:
- yolov3, NAdam, cuda, no_foreach: 0.02333217109553516
- yolov3, NAdam, cuda, differentiable: 0.041708804410882296


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/4899620594

cc @janeyx99


This is just noise..I may need to up the threshold but would like to see some more data before doing that. Some observations:
Total: ~150
CUDA: 90  |  CPU: 60
foreach: 104 | fused: 18 | ...
SGD:  45 | (N/R)Adam(W): 59 | ...
pt2: 117

These were all slowdowns :/