Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '8a24a912a5f545d18059b59629aa3598f3783f25'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: b5222f140da05e40ac90ff42bd1db6564343daff

Affected PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- llama_v2_7b_8h, Adadelta, cuda, no_foreach: 0.15159791599944583
- llama_v2_7b_8h, Adadelta, cuda, differentiable: 0.1632760105003399
- llama_v2_7b_8h, Adagrad, cuda, default: 0.07678349050001998
- llama_v2_7b_8h, Adagrad, cuda, maximize: 0.0892258906654509
- llama_v2_7b_8h, Adagrad, cuda, no_foreach: 0.061240769799769626
- llama_v2_7b_8h, Adagrad, cuda, differentiable: 0.06111675029969774
- llama_v2_7b_8h, Adagrad, cuda, foreach: 0.07675365340001009
- llama_v2_7b_8h, Adam, cuda, default: 0.10741205049998825
- llama_v2_7b_8h, Adam, cuda, no_foreach: 0.10063007299868332
- llama_v2_7b_8h, Adam, cuda, differentiable: 0.1006583465004951
- llama_v2_7b_8h, Adam, cuda, foreach: 0.10724916949948238
- llama_v2_7b_8h, Adam, cuda, foreach, maximize, capturable: 0.12956633800058626
- llama_v2_7b_8h, Adam, cuda, fused: 0.047593135700299175
- llama_v2_7b_8h, Adam, cuda, fused, amsgrad, maximize: 0.0548043669998151
- llama_v2_7b_8h, Adam, cuda, fused, capturable: 0.047569899300287946
- llama_v2_7b_8h, Adam, cuda, fused, capturable, amsgrad: 0.054856603799998994
- llama_v2_7b_8h, AdamW, cuda, default: 0.11951695999960066
- llama_v2_7b_8h, AdamW, cuda, no_foreach: 0.11190600550071395
- llama_v2_7b_8h, AdamW, cuda, differentiable: 0.11188161950303765
- llama_v2_7b_8h, AdamW, cuda, foreach: 0.11934497300171643
- llama_v2_7b_8h, AdamW, cuda, foreach, maximize, capturable: 0.14165637599944603
- llama_v2_7b_8h, AdamW, cuda, fused: 0.048842989200056766
- llama_v2_7b_8h, AdamW, cuda, fused, amsgrad, maximize: 0.05536436530019273
- llama_v2_7b_8h, AdamW, cuda, fused, capturable: 0.048859178600105226
- llama_v2_7b_8h, AdamW, cuda, fused, capturable, amsgrad: 0.05531093780009542
- llama_v2_7b_8h, Adamax, cuda, default: 0.13432470149928122
- llama_v2_7b_8h, Adamax, cuda, maximize: 0.14536954399954993
- llama_v2_7b_8h, Adamax, cuda, no_foreach: 0.11235202699936053
- llama_v2_7b_8h, Adamax, cuda, differentiable: 0.12396432849891426
- llama_v2_7b_8h, Adamax, cuda, foreach: 0.13361414199971478
- llama_v2_7b_8h, ASGD, cuda, default: 0.04140297779995308
- llama_v2_7b_8h, ASGD, cuda, maximize: 0.053347088399823404
- llama_v2_7b_8h, ASGD, cuda, no_foreach: 0.0397140010001749
- llama_v2_7b_8h, ASGD, cuda, differentiable: 0.039713525300248874
- llama_v2_7b_8h, ASGD, cuda, foreach: 0.041407850199902896
- llama_v2_7b_8h, SGD, cuda, default: 0.01765107960000023
- llama_v2_7b_8h, SGD, cuda, maximize: 0.029613750200223877
- llama_v2_7b_8h, SGD, cuda, no_foreach: 0.01687094874996546
- llama_v2_7b_8h, SGD, cuda, differentiable: 0.017645066749901163
- llama_v2_7b_8h, SGD, cuda, foreach: 0.01766391119981563
- llama_v2_7b_8h, SGD, cuda, foreach, momentum=0.9, nesterov: 0.06498112930021308
- llama_v2_7b_8h, SGD, cuda, foreach, momentum=0.9: 0.047357607899903086
- llama_v2_7b_8h, RAdam, cuda, default: 0.1321774629977881
- llama_v2_7b_8h, RAdam, cuda, no_foreach: 0.08425681666631135
- llama_v2_7b_8h, RAdam, cuda, differentiable: 0.08420716199907474
- llama_v2_7b_8h, RAdam, cuda, foreach: 0.13208772000143654
- llama_v2_7b_8h, Rprop, cuda, no_foreach: 0.19160295549954753
- llama_v2_7b_8h, Rprop, cuda, differentiable: 0.20319654000195442
- llama_v2_7b_8h, RMSprop, cuda, default: 0.07682939510013967
- llama_v2_7b_8h, RMSprop, cuda, maximize: 0.08914196633365161
- llama_v2_7b_8h, RMSprop, cuda, no_foreach: 0.07250457560003269
- llama_v2_7b_8h, RMSprop, cuda, differentiable: 0.072339521899994
- llama_v2_7b_8h, RMSprop, cuda, foreach: 0.07683562179990985
- llama_v2_7b_8h, NAdam, cuda, default: 0.1428483835006773
- llama_v2_7b_8h, NAdam, cuda, no_foreach: 0.13389658500091173
- llama_v2_7b_8h, NAdam, cuda, differentiable: 0.1560422375005146
- llama_v2_7b_8h, NAdam, cuda, foreach: 0.14284194200081402
- stable_diffusion, Adadelta, cuda, default: 0.02912470689989277
- stable_diffusion, Adadelta, cuda, maximize: 0.031385445699561386
- stable_diffusion, Adadelta, cuda, no_foreach: 0.03207660729967756
- stable_diffusion, Adadelta, cuda, differentiable: 0.03809691930000554
- stable_diffusion, Adadelta, cuda, foreach: 0.029127642800449393
- stable_diffusion, Adagrad, cuda, default: 0.014190627450079773
- stable_diffusion, Adagrad, cuda, maximize: 0.016259470650038566
- stable_diffusion, Adagrad, cuda, no_foreach: 0.017587975149945125
- stable_diffusion, Adagrad, cuda, differentiable: 0.018914114300059737
- stable_diffusion, Adagrad, cuda, foreach: 0.014156745249783852
- stable_diffusion, Adam, cuda, default: 0.01961144439956115
- stable_diffusion, Adam, cuda, amsgrad, maximize: 0.02479445519929868
- stable_diffusion, Adam, cuda, no_foreach: 0.030453162099729526
- stable_diffusion, Adam, cuda, differentiable: 0.058713280499796384
- stable_diffusion, Adam, cuda, foreach: 0.019602170799771557
- stable_diffusion, Adam, cuda, foreach, maximize, capturable: 0.0264966574999562
- stable_diffusion, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.02966636720011593
- stable_diffusion, Adam, cuda, fused: 0.009037688000292594
- stable_diffusion, Adam, cuda, fused, amsgrad, maximize: 0.010580969299917342
- stable_diffusion, Adam, cuda, fused, capturable: 0.009033604200036885
- stable_diffusion, Adam, cuda, fused, capturable, amsgrad: 0.010574914650351274
- stable_diffusion, AdamW, cuda, default: 0.02176126580015989
- stable_diffusion, AdamW, cuda, amsgrad, maximize: 0.027017596299992876
- stable_diffusion, AdamW, cuda, no_foreach: 0.03364145330051542
- stable_diffusion, AdamW, cuda, differentiable: 0.06166438719956204
- stable_diffusion, AdamW, cuda, foreach: 0.021723915600159672
- stable_diffusion, AdamW, cuda, foreach, maximize, capturable: 0.028676636900490847
- stable_diffusion, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.03185097700043116
- stable_diffusion, AdamW, cuda, fused: 0.009307471466551457
- stable_diffusion, AdamW, cuda, fused, amsgrad, maximize: 0.010723312199479551
- stable_diffusion, AdamW, cuda, fused, capturable: 0.009291926533478545
- stable_diffusion, AdamW, cuda, fused, capturable, amsgrad: 0.010709186399617465
- stable_diffusion, Adamax, cuda, default: 0.02935145450028358
- stable_diffusion, Adamax, cuda, maximize: 0.030832736600132193
- stable_diffusion, Adamax, cuda, no_foreach: 0.03958210720011266
- stable_diffusion, Adamax, cuda, differentiable: 0.047172184100054436
- stable_diffusion, Adamax, cuda, foreach: 0.02894683580016135
- stable_diffusion, ASGD, cuda, default: 0.012040043350134512
- stable_diffusion, ASGD, cuda, maximize: 0.013093538049724886
- stable_diffusion, ASGD, cuda, no_foreach: 0.022523093999916456
- stable_diffusion, ASGD, cuda, differentiable: 0.022585181199974613
- stable_diffusion, ASGD, cuda, foreach: 0.011757589850094518
- stable_diffusion, SGD, cuda, default: 0.0031925819699972634
- stable_diffusion, SGD, cuda, maximize: 0.0053588985100213905
- stable_diffusion, SGD, cuda, no_foreach: 0.00420516125006543
- stable_diffusion, SGD, cuda, differentiable: 0.003190564929973334
- stable_diffusion, SGD, cuda, foreach: 0.00319391576005728
- stable_diffusion, SGD, cuda, foreach, momentum=0.9, nesterov: 0.01183656624998548
- stable_diffusion, SGD, cuda, foreach, momentum=0.9: 0.008652353500171255
- stable_diffusion, RAdam, cuda, default: 0.023989324899594065
- stable_diffusion, RAdam, cuda, no_foreach: 0.049642508499528046
- stable_diffusion, RAdam, cuda, differentiable: 0.050729216399486174
- stable_diffusion, RAdam, cuda, foreach: 0.023972665800101822
- stable_diffusion, Rprop, cuda, default: 0.049574793099600355
- stable_diffusion, Rprop, cuda, maximize: 0.05052526920044329
- stable_diffusion, Rprop, cuda, no_foreach: 0.06387991719966521
- stable_diffusion, Rprop, cuda, differentiable: 0.06791017959985765
- stable_diffusion, Rprop, cuda, foreach: 0.04925429810027708
- stable_diffusion, RMSprop, cuda, default: 0.014027737699871068
- stable_diffusion, RMSprop, cuda, maximize: 0.016216300300220608
- stable_diffusion, RMSprop, cuda, no_foreach: 0.016114069750256022
- stable_diffusion, RMSprop, cuda, differentiable: 0.01721924925004714
- stable_diffusion, RMSprop, cuda, foreach: 0.014014913600112778
- stable_diffusion, NAdam, cuda, default: 0.026117308699758722
- stable_diffusion, NAdam, cuda, no_foreach: 0.03929794840005343
- stable_diffusion, NAdam, cuda, differentiable: 0.06536746450001374
- stable_diffusion, NAdam, cuda, foreach: 0.026151609199587256
- llama_v2_7b_16h, SGD, cuda, default: 0.03284975149981619
- llama_v2_7b_16h, SGD, cuda, no_foreach: 0.03140719880029792
- llama_v2_7b_16h, SGD, cuda, differentiable: 0.03282726500001445
- llama_v2_7b_16h, SGD, cuda, foreach: 0.032831028399959904


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5693564120

cc @janeyx99


This is fixed