Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '8a24a912a5f545d18059b59629aa3598f3783f25'
Root cause: Resource Mishandling


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: b5222f140da05e40ac90ff42bd1db6564343daff

Affected PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected Tests:
- maml_omniglot, Adagrad, cuda, differentiable: +34.96047%
- DALLE2_pytorch, NAdam, cuda, foreach: +32.06328%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- stable_diffusion, Adadelta, cuda, default: 0.02913637429010123
- stable_diffusion, Adadelta, cuda, maximize: 0.03135403001215309
- stable_diffusion, Adadelta, cuda, no_foreach: 0.032581815612502396
- stable_diffusion, Adadelta, cuda, differentiable: 0.03898170131724328
- stable_diffusion, Adadelta, cuda, foreach: 0.02910422768909484
- stable_diffusion, Adagrad, cuda, default: 0.014199691300746053
- stable_diffusion, Adagrad, cuda, maximize: 0.01626062225550413
- stable_diffusion, Adagrad, cuda, no_foreach: 0.01823261013487354
- stable_diffusion, Adagrad, cuda, differentiable: 0.019371134345419705
- stable_diffusion, Adagrad, cuda, foreach: 0.014156167954206466
- stable_diffusion, Adam, cuda, default: 0.019608439749572427
- stable_diffusion, Adam, cuda, amsgrad, maximize: 0.0248642697930336
- stable_diffusion, Adam, cuda, no_foreach: 0.032092174305580555
- stable_diffusion, Adam, cuda, differentiable: 0.061248939903452995
- stable_diffusion, Adam, cuda, foreach: 0.01961243875557557
- stable_diffusion, Adam, cuda, foreach, maximize, capturable: 0.026531827985309064
- stable_diffusion, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.029736343212425707
- stable_diffusion, Adam, cuda, fused: 0.009052158131574591
- stable_diffusion, Adam, cuda, fused, amsgrad, maximize: 0.010601754498202354
- stable_diffusion, Adam, cuda, fused, capturable: 0.009067654702812433
- stable_diffusion, Adam, cuda, fused, capturable, amsgrad: 0.010624984255991878
- stable_diffusion, AdamW, cuda, default: 0.021796787995845078
- stable_diffusion, AdamW, cuda, amsgrad, maximize: 0.02699682880192995
- stable_diffusion, AdamW, cuda, no_foreach: 0.03481089789420366
- stable_diffusion, AdamW, cuda, differentiable: 0.06454430541489273
- stable_diffusion, AdamW, cuda, foreach: 0.021780900401063265
- stable_diffusion, AdamW, cuda, foreach, maximize, capturable: 0.02869984828867018
- stable_diffusion, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.0318968998035416
- stable_diffusion, AdamW, cuda, fused: 0.009326355702554186
- stable_diffusion, AdamW, cuda, fused, amsgrad, maximize: 0.010720548254903406
- stable_diffusion, AdamW, cuda, fused, capturable: 0.009304978131937484
- stable_diffusion, AdamW, cuda, fused, capturable, amsgrad: 0.010788114846218378
- stable_diffusion, Adamax, cuda, default: 0.03153444179333746
- stable_diffusion, Adamax, cuda, maximize: 0.03204364739358425
- stable_diffusion, Adamax, cuda, no_foreach: 0.042078401404432954
- stable_diffusion, Adamax, cuda, differentiable: 0.050771533884108065
- stable_diffusion, Adamax, cuda, foreach: 0.03027029219083488
- stable_diffusion, ASGD, cuda, default: 0.012297250702977179
- stable_diffusion, ASGD, cuda, maximize: 0.013347465742845089
- stable_diffusion, ASGD, cuda, no_foreach: 0.023148602899163963
- stable_diffusion, ASGD, cuda, differentiable: 0.023466631304472685
- stable_diffusion, ASGD, cuda, foreach: 0.012111318460665644
- stable_diffusion, SGD, cuda, default: 0.003198232650756836
- stable_diffusion, SGD, cuda, maximize: 0.005351354631129652
- stable_diffusion, SGD, cuda, no_foreach: 0.004293798601720482
- stable_diffusion, SGD, cuda, differentiable: 0.003198513351380825
- stable_diffusion, SGD, cuda, foreach: 0.0031965583912096918
- stable_diffusion, SGD, cuda, foreach, momentum=0.9, nesterov: 0.011855958600062878
- stable_diffusion, SGD, cuda, foreach, momentum=0.9: 0.008635990042239427
- stable_diffusion, RAdam, cuda, default: 0.02406868920661509
- stable_diffusion, RAdam, cuda, no_foreach: 0.05100888260640204
- stable_diffusion, RAdam, cuda, differentiable: 0.05291640439536423
- stable_diffusion, RAdam, cuda, foreach: 0.023989999294281007
- stable_diffusion, Rprop, cuda, default: 0.05101145298685879
- stable_diffusion, Rprop, cuda, maximize: 0.05159520939923823
- stable_diffusion, Rprop, cuda, no_foreach: 0.0670445242896676
- stable_diffusion, Rprop, cuda, differentiable: 0.07102200652007014
- stable_diffusion, Rprop, cuda, foreach: 0.05045220819301903
- stable_diffusion, RMSprop, cuda, default: 0.014063256653025746
- stable_diffusion, RMSprop, cuda, maximize: 0.016241732705384492
- stable_diffusion, RMSprop, cuda, no_foreach: 0.01631824605865404
- stable_diffusion, RMSprop, cuda, differentiable: 0.01744278445839882
- stable_diffusion, RMSprop, cuda, foreach: 0.014013096259441226
- stable_diffusion, NAdam, cuda, default: 0.026155458018183707
- stable_diffusion, NAdam, cuda, no_foreach: 0.040981810400262476
- stable_diffusion, NAdam, cuda, differentiable: 0.06792989155898492
- stable_diffusion, NAdam, cuda, foreach: 0.026160873100161554
- llama_v2_7b_16h, SGD, cuda, default: 0.03289474779739976
- llama_v2_7b_16h, SGD, cuda, no_foreach: 0.03143358461093158
- llama_v2_7b_16h, SGD, cuda, differentiable: 0.03286938138771802
- llama_v2_7b_16h, SGD, cuda, foreach: 0.03288652340415865


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5688293241

cc @janeyx99


These are reasonable. Engaging with the OOMs on llama2 now.