Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '8a24a912a5f545d18059b59629aa3598f3783f25'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected Tests:
- pytorch_stargan, RAdam, cuda, no_foreach: +35.86739%
- basic_gnn_gin, Adadelta, cuda, no_foreach: +43.72250%
- basic_gnn_gin, Adadelta, cuda, differentiable: +47.32065%
- hf_Reformer, AdamW, cuda, no_foreach: +58.76754%
- resnet152, Rprop, cuda, default: +34.76140%
- hf_Bart, Adamax, cuda, no_foreach: +46.89345%
- yolov3, NAdam, cuda, no_foreach: +56.84873%
- phlippe_densenet, Adagrad, cuda, differentiable: +32.06170%
- phlippe_densenet, Rprop, cuda, foreach: +30.40127%
- basic_gnn_gcn, ASGD, cuda, foreach: +49.97573%
- basic_gnn_gcn, RMSprop, cuda, no_foreach: +42.70028%
- basic_gnn_gcn, NAdam, cuda, no_foreach: +38.58713%
- Background_Matting, Adamax, cuda, no_foreach: +42.57347%
- Background_Matting, RMSprop, cuda, differentiable: +54.26796%
- shufflenet_v2_x1_0, ASGD, cuda, differentiable: +55.44373%
- speech_transformer, Adagrad, cuda, differentiable: +32.92915%
- DALLE2_pytorch, Adam, cuda, amsgrad, maximize: +44.68240%
- DALLE2_pytorch, AdamW, cuda, amsgrad, maximize: +44.59333%
- DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +38.13315%
- DALLE2_pytorch, RMSprop, cuda, differentiable: +56.90665%
- DALLE2_pytorch, NAdam, cuda, foreach: +41.37757%
- timm_resnest, NAdam, cuda, no_foreach: +54.09296%
- hf_Albert, Adadelta, cuda, differentiable: +30.72750%
- clip, ASGD, cuda, differentiable: +50.69125%
- timm_vovnet, RAdam, cuda, no_foreach: +43.83019%
- tts_angular, SGD, cuda, no_foreach: +51.59787%
- hf_DistilBert, Rprop, cuda, foreach: +43.56024%
- nanogpt_generate, Adagrad, cuda, no_foreach: +51.35528%
- stable_diffusion, Rprop, cuda, maximize: +34.73407%
- mobilenet_v3_large, Rprop, cuda, no_foreach: +31.70700%
- fambench_xlmr, ASGD, cuda, no_foreach: +48.90229%
- tacotron2, Rprop, cuda, default: +37.89578%
- opacus_cifar10, Rprop, cuda, default: +50.36759%
- timm_regnet, ASGD, cuda, no_foreach: +54.59931%
- timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: +40.28716%
- timm_vision_transformer, Rprop, cuda, default: +34.69739%
- detectron2_maskrcnn_r_101_fpn, Adamax, cuda, maximize: +42.46166%
- detectron2_maskrcnn_r_101_fpn, ASGD, cuda, foreach: +36.04409%
- mobilenet_v2_quantized_qat, Adamax, cuda, maximize: +39.64085%


Tests that were no longer run on affected commit:
- llama_v2_7b_8h, Adadelta, cuda, no_foreach: 0.15159791599944583
- llama_v2_7b_8h, Adadelta, cuda, differentiable: 0.1632760105003399
- llama_v2_7b_8h, Adagrad, cuda, default: 0.07678349050001998
- llama_v2_7b_8h, Adagrad, cuda, maximize: 0.0892258906654509
- llama_v2_7b_8h, Adagrad, cuda, no_foreach: 0.061240769799769626
- llama_v2_7b_8h, Adagrad, cuda, differentiable: 0.06111675029969774
- llama_v2_7b_8h, Adagrad, cuda, foreach: 0.07675365340001009
- llama_v2_7b_8h, Adam, cuda, default: 0.10741205049998825
- llama_v2_7b_8h, Adam, cuda, no_foreach: 0.10063007299868332
- llama_v2_7b_8h, Adam, cuda, differentiable: 0.1006583465004951
- llama_v2_7b_8h, Adam, cuda, foreach: 0.10724916949948238
- llama_v2_7b_8h, Adam, cuda, foreach, maximize, capturable: 0.12956633800058626
- llama_v2_7b_8h, Adam, cuda, fused: 0.047593135700299175
- llama_v2_7b_8h, Adam, cuda, fused, amsgrad, maximize: 0.0548043669998151
- llama_v2_7b_8h, Adam, cuda, fused, capturable: 0.047569899300287946
- llama_v2_7b_8h, Adam, cuda, fused, capturable, amsgrad: 0.054856603799998994
- llama_v2_7b_8h, AdamW, cuda, default: 0.11951695999960066
- llama_v2_7b_8h, AdamW, cuda, no_foreach: 0.11190600550071395
- llama_v2_7b_8h, AdamW, cuda, differentiable: 0.11188161950303765
- llama_v2_7b_8h, AdamW, cuda, foreach: 0.11934497300171643
- llama_v2_7b_8h, AdamW, cuda, foreach, maximize, capturable: 0.14165637599944603
- llama_v2_7b_8h, AdamW, cuda, fused: 0.048842989200056766
- llama_v2_7b_8h, AdamW, cuda, fused, amsgrad, maximize: 0.05536436530019273
- llama_v2_7b_8h, AdamW, cuda, fused, capturable: 0.048859178600105226
- llama_v2_7b_8h, AdamW, cuda, fused, capturable, amsgrad: 0.05531093780009542
- llama_v2_7b_8h, Adamax, cuda, default: 0.13432470149928122
- llama_v2_7b_8h, Adamax, cuda, maximize: 0.14536954399954993
- llama_v2_7b_8h, Adamax, cuda, no_foreach: 0.11235202699936053
- llama_v2_7b_8h, Adamax, cuda, differentiable: 0.12396432849891426
- llama_v2_7b_8h, Adamax, cuda, foreach: 0.13361414199971478
- llama_v2_7b_8h, ASGD, cuda, default: 0.04140297779995308
- llama_v2_7b_8h, ASGD, cuda, maximize: 0.053347088399823404
- llama_v2_7b_8h, ASGD, cuda, no_foreach: 0.0397140010001749
- llama_v2_7b_8h, ASGD, cuda, differentiable: 0.039713525300248874
- llama_v2_7b_8h, ASGD, cuda, foreach: 0.041407850199902896
- llama_v2_7b_8h, SGD, cuda, default: 0.01765107960000023
- llama_v2_7b_8h, SGD, cuda, maximize: 0.029613750200223877
- llama_v2_7b_8h, SGD, cuda, no_foreach: 0.01687094874996546
- llama_v2_7b_8h, SGD, cuda, differentiable: 0.017645066749901163
- llama_v2_7b_8h, SGD, cuda, foreach: 0.01766391119981563
- llama_v2_7b_8h, SGD, cuda, foreach, momentum=0.9, nesterov: 0.06498112930021308
- llama_v2_7b_8h, SGD, cuda, foreach, momentum=0.9: 0.047357607899903086
- llama_v2_7b_8h, RAdam, cuda, default: 0.1321774629977881
- llama_v2_7b_8h, RAdam, cuda, no_foreach: 0.08425681666631135
- llama_v2_7b_8h, RAdam, cuda, differentiable: 0.08420716199907474
- llama_v2_7b_8h, RAdam, cuda, foreach: 0.13208772000143654
- llama_v2_7b_8h, Rprop, cuda, no_foreach: 0.19160295549954753
- llama_v2_7b_8h, Rprop, cuda, differentiable: 0.20319654000195442
- llama_v2_7b_8h, RMSprop, cuda, default: 0.07682939510013967
- llama_v2_7b_8h, RMSprop, cuda, maximize: 0.08914196633365161
- llama_v2_7b_8h, RMSprop, cuda, no_foreach: 0.07250457560003269
- llama_v2_7b_8h, RMSprop, cuda, differentiable: 0.072339521899994
- llama_v2_7b_8h, RMSprop, cuda, foreach: 0.07683562179990985
- llama_v2_7b_8h, NAdam, cuda, default: 0.1428483835006773
- llama_v2_7b_8h, NAdam, cuda, no_foreach: 0.13389658500091173
- llama_v2_7b_8h, NAdam, cuda, differentiable: 0.1560422375005146
- llama_v2_7b_8h, NAdam, cuda, foreach: 0.14284194200081402
- llama_v2_7b_16h, SGD, cuda, default: 0.03284975149981619
- llama_v2_7b_16h, SGD, cuda, no_foreach: 0.03140719880029792
- llama_v2_7b_16h, SGD, cuda, differentiable: 0.03282726500001445
- llama_v2_7b_16h, SGD, cuda, foreach: 0.032831028399959904


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5698427389

cc @janeyx99


This is fine--I may need to do more to denoise