Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '753f5a11bf46f2c5f52c31bbb0bca5c1771fb509'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 73c794dadccfcc204b7db42dc0c915978fc61434

Affected PyTorch commit: 753f5a11bf46f2c5f52c31bbb0bca5c1771fb509

Affected Tests:
- resnet18, NAdam, cuda, (pt2) no_foreach: +812584.17652%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- doctr_det_predictor, Adadelta, cuda, default: 0.0026292953919619323
- doctr_det_predictor, Adadelta, cuda, maximize: 0.0032149860821664334
- doctr_det_predictor, Adadelta, cuda, no_foreach: 0.018369652424007656
- doctr_det_predictor, Adadelta, cuda, differentiable: 0.02209976240992546
- doctr_det_predictor, Adadelta, cuda, foreach: 0.002556222057901323
- doctr_det_predictor, Adagrad, cuda, default: 0.00362060260027647
- doctr_det_predictor, Adagrad, cuda, maximize: 0.0035475869104266166
- doctr_det_predictor, Adagrad, cuda, no_foreach: 0.009911318092296522
- doctr_det_predictor, Adagrad, cuda, differentiable: 0.010628464701585471
- doctr_det_predictor, Adagrad, cuda, foreach: 0.003371613728813827
- doctr_det_predictor, Adam, cuda, default: 0.003717809719964862
- doctr_det_predictor, Adam, cuda, amsgrad, maximize: 0.004432897898368537
- doctr_det_predictor, Adam, cuda, no_foreach: 0.017648058384656905
- doctr_det_predictor, Adam, cuda, differentiable: 0.03414801750332117
- doctr_det_predictor, Adam, cuda, foreach: 0.003585186209529638
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable: 0.006313692880794406
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.006239619390107691
- doctr_det_predictor, Adam, cuda, fused: 0.001308153700083494
- doctr_det_predictor, Adam, cuda, fused, amsgrad, maximize: 0.0013835008488968016
- doctr_det_predictor, Adam, cuda, fused, capturable: 0.0013164045568555592
- doctr_det_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0013857730943709612
- doctr_det_predictor, AdamW, cuda, default: 0.0038083583721891046
- doctr_det_predictor, AdamW, cuda, amsgrad, maximize: 0.004562685308046639
- doctr_det_predictor, AdamW, cuda, no_foreach: 0.019139475654810667
- doctr_det_predictor, AdamW, cuda, differentiable: 0.03625544081442058
- doctr_det_predictor, AdamW, cuda, foreach: 0.0035984329925850033
- doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable: 0.006134662479162216
- doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.006387698287144303
- doctr_det_predictor, AdamW, cuda, fused: 0.001354459135327488
- doctr_det_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0014223708398640156
- doctr_det_predictor, AdamW, cuda, fused, capturable: 0.001355045116506517
- doctr_det_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0014215037948451937
- doctr_det_predictor, Adamax, cuda, default: 0.01847565993666649
- doctr_det_predictor, Adamax, cuda, maximize: 0.018551887711510062
- doctr_det_predictor, Adamax, cuda, no_foreach: 0.024136744905263186
- doctr_det_predictor, Adamax, cuda, differentiable: 0.02819731030613184
- doctr_det_predictor, Adamax, cuda, foreach: 0.017862983187660574
- doctr_det_predictor, ASGD, cuda, default: 0.0075671265972778205
- doctr_det_predictor, ASGD, cuda, maximize: 0.00817235239315778
- doctr_det_predictor, ASGD, cuda, no_foreach: 0.013323935144580901
- doctr_det_predictor, ASGD, cuda, differentiable: 0.013237837702035904
- doctr_det_predictor, ASGD, cuda, foreach: 0.007336815898306668
- doctr_det_predictor, SGD, cuda, default: 0.0007898519220761955
- doctr_det_predictor, SGD, cuda, maximize: 0.0014158144732937215
- doctr_det_predictor, SGD, cuda, no_foreach: 0.0017102576256729663
- doctr_det_predictor, SGD, cuda, differentiable: 0.0007992154723033309
- doctr_det_predictor, SGD, cuda, foreach: 0.0006098133730702102
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0010085412906482814
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9: 0.0008339240271598101
- doctr_det_predictor, RAdam, cuda, default: 0.004220310193486512
- doctr_det_predictor, RAdam, cuda, no_foreach: 0.029050867026671767
- doctr_det_predictor, RAdam, cuda, differentiable: 0.02951332931406796
- doctr_det_predictor, RAdam, cuda, foreach: 0.004071950302459299
- doctr_det_predictor, Rprop, cuda, default: 0.0225975482724607
- doctr_det_predictor, Rprop, cuda, maximize: 0.021982805291190743
- doctr_det_predictor, Rprop, cuda, no_foreach: 0.036685329116880896
- doctr_det_predictor, Rprop, cuda, differentiable: 0.038977233320474626
- doctr_det_predictor, Rprop, cuda, foreach: 0.02208272321149707
- doctr_det_predictor, RMSprop, cuda, default: 0.0018999073887243866
- doctr_det_predictor, RMSprop, cuda, maximize: 0.002458558562211692
- doctr_det_predictor, RMSprop, cuda, no_foreach: 0.009237234372024734
- doctr_det_predictor, RMSprop, cuda, differentiable: 0.009788384583468238
- doctr_det_predictor, RMSprop, cuda, foreach: 0.0017003671242855488
- doctr_det_predictor, NAdam, cuda, default: 0.005071037118323147
- doctr_det_predictor, NAdam, cuda, no_foreach: 0.020660046162083744
- doctr_det_predictor, NAdam, cuda, differentiable: 0.03628514283336699
- doctr_det_predictor, NAdam, cuda, foreach: 0.0049220167333260175
- doctr_reco_predictor, Adadelta, cuda, default: 0.0015291775274090468
- doctr_reco_predictor, Adadelta, cuda, maximize: 0.0016330820601433513
- doctr_reco_predictor, Adadelta, cuda, no_foreach: 0.006301340921781957
- doctr_reco_predictor, Adadelta, cuda, differentiable: 0.007435454181395471
- doctr_reco_predictor, Adadelta, cuda, foreach: 0.001492248473223299
- doctr_reco_predictor, Adagrad, cuda, default: 0.0013776801014319063
- doctr_reco_predictor, Adagrad, cuda, maximize: 0.0014111423748545348
- doctr_reco_predictor, Adagrad, cuda, no_foreach: 0.0033829636219888924
- doctr_reco_predictor, Adagrad, cuda, differentiable: 0.0036344571225345136
- doctr_reco_predictor, Adagrad, cuda, foreach: 0.0013251316058449448
- doctr_reco_predictor, Adam, cuda, default: 0.0014454349619336428
- doctr_reco_predictor, Adam, cuda, amsgrad, maximize: 0.0017631584894843401
- doctr_reco_predictor, Adam, cuda, no_foreach: 0.006018105316907167
- doctr_reco_predictor, Adam, cuda, differentiable: 0.011723358067683876
- doctr_reco_predictor, Adam, cuda, foreach: 0.0014403933705762029
- doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable: 0.0023685230687260626
- doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0024250594805926085
- doctr_reco_predictor, Adam, cuda, fused: 0.0005269603771157563
- doctr_reco_predictor, Adam, cuda, fused, amsgrad, maximize: 0.0007342772209085524
- doctr_reco_predictor, Adam, cuda, fused, capturable: 0.000527110620867461
- doctr_reco_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0007346532992087304
- doctr_reco_predictor, AdamW, cuda, default: 0.0014990828069858253
- doctr_reco_predictor, AdamW, cuda, amsgrad, maximize: 0.0017757598101161422
- doctr_reco_predictor, AdamW, cuda, no_foreach: 0.006482547638006509
- doctr_reco_predictor, AdamW, cuda, differentiable: 0.012344431434758007
- doctr_reco_predictor, AdamW, cuda, foreach: 0.0014049155032262205
- doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable: 0.002360413633286953
- doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.0024067497812211513
- doctr_reco_predictor, AdamW, cuda, fused: 0.0005437816013582051
- doctr_reco_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0007552005341276526
- doctr_reco_predictor, AdamW, cuda, fused, capturable: 0.0005438589537516236
- doctr_reco_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0007580557661131024
- doctr_reco_predictor, Adamax, cuda, default: 0.006167916161939502
- doctr_reco_predictor, Adamax, cuda, maximize: 0.0063001333037391305
- doctr_reco_predictor, Adamax, cuda, no_foreach: 0.007950956067070365
- doctr_reco_predictor, Adamax, cuda, differentiable: 0.009237475056822101
- doctr_reco_predictor, Adamax, cuda, foreach: 0.006077779419720173
- doctr_reco_predictor, ASGD, cuda, default: 0.0026487851934507488
- doctr_reco_predictor, ASGD, cuda, maximize: 0.0028727091616019607
- doctr_reco_predictor, ASGD, cuda, no_foreach: 0.004422628283500671
- doctr_reco_predictor, ASGD, cuda, differentiable: 0.004424051321111619
- doctr_reco_predictor, ASGD, cuda, foreach: 0.0025616905512288214
- doctr_reco_predictor, SGD, cuda, default: 0.0003075455138459802
- doctr_reco_predictor, SGD, cuda, maximize: 0.000548856953624636
- doctr_reco_predictor, SGD, cuda, no_foreach: 0.0006119461338967085
- doctr_reco_predictor, SGD, cuda, differentiable: 0.0003084831018932164
- doctr_reco_predictor, SGD, cuda, foreach: 0.00023922385089099408
- doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0006103817708790302
- doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9: 0.00044087666412815453
- doctr_reco_predictor, RAdam, cuda, default: 0.0016939417459070683
- doctr_reco_predictor, RAdam, cuda, no_foreach: 0.009693776452913881
- doctr_reco_predictor, RAdam, cuda, differentiable: 0.009982583043165506
- doctr_reco_predictor, RAdam, cuda, foreach: 0.0015770491003058852
- doctr_reco_predictor, Rprop, cuda, default: 0.00789695258717984
- doctr_reco_predictor, Rprop, cuda, maximize: 0.007850581449456513
- doctr_reco_predictor, Rprop, cuda, no_foreach: 0.012482122448273004
- doctr_reco_predictor, Rprop, cuda, differentiable: 0.013338435138575733
- doctr_reco_predictor, Rprop, cuda, foreach: 0.007718318062834442
- doctr_reco_predictor, RMSprop, cuda, default: 0.0007522431863471866
- doctr_reco_predictor, RMSprop, cuda, maximize: 0.0009753403269375364
- doctr_reco_predictor, RMSprop, cuda, no_foreach: 0.0031346205715090036
- doctr_reco_predictor, RMSprop, cuda, differentiable: 0.0033559000212699175
- doctr_reco_predictor, RMSprop, cuda, foreach: 0.0006938977488316595
- doctr_reco_predictor, NAdam, cuda, default: 0.0019539591344073415
- doctr_reco_predictor, NAdam, cuda, no_foreach: 0.007027625893242657
- doctr_reco_predictor, NAdam, cuda, differentiable: 0.012337709241546691
- doctr_reco_predictor, NAdam, cuda, foreach: 0.0019001618539914488


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6062324224

cc @janeyx99


<img width="386" alt="image" src="https://github.com/pytorch/benchmark/assets/31798555/fdc8043a-4846-4a8c-bc3e-dbb01edfb821">
closing as it's very similar to the night before and the variance is being observed #1870