Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '73c794dadccfcc204b7db42dc0c915978fc61434'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: b5b99fe13b890232bb61155a46239922661f4695

Affected PyTorch commit: 73c794dadccfcc204b7db42dc0c915978fc61434

Affected Tests:
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: +415449.59809%
- timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: +408444.32952%


Tests that were no longer run on affected commit:
- doctr_det_predictor, Adadelta, cuda, default: 0.0027073046937584875
- doctr_det_predictor, Adadelta, cuda, maximize: 0.003044134653173387
- doctr_det_predictor, Adadelta, cuda, no_foreach: 0.017554870597086845
- doctr_det_predictor, Adadelta, cuda, differentiable: 0.020798875717446208
- doctr_det_predictor, Adadelta, cuda, foreach: 0.002557106697931886
- doctr_det_predictor, Adagrad, cuda, default: 0.003475225772708654
- doctr_det_predictor, Adagrad, cuda, maximize: 0.003432596339844167
- doctr_det_predictor, Adagrad, cuda, no_foreach: 0.009520975748697917
- doctr_det_predictor, Adagrad, cuda, differentiable: 0.010207952465862036
- doctr_det_predictor, Adagrad, cuda, foreach: 0.0034436467615887524
- doctr_det_predictor, Adam, cuda, default: 0.0036622950108721854
- doctr_det_predictor, Adam, cuda, amsgrad, maximize: 0.004268973777070642
- doctr_det_predictor, Adam, cuda, no_foreach: 0.016576920938678085
- doctr_det_predictor, Adam, cuda, differentiable: 0.03418538197875023
- doctr_det_predictor, Adam, cuda, foreach: 0.0034749558102339507
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable: 0.006178876259364188
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0061795333586633205
- doctr_det_predictor, Adam, cuda, fused: 0.0013073311396874488
- doctr_det_predictor, Adam, cuda, fused, amsgrad, maximize: 0.0013793339440599083
- doctr_det_predictor, Adam, cuda, fused, capturable: 0.0013068144186399878
- doctr_det_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0013806636724621058
- doctr_det_predictor, AdamW, cuda, default: 0.003633675989694893
- doctr_det_predictor, AdamW, cuda, amsgrad, maximize: 0.004379119598306716
- doctr_det_predictor, AdamW, cuda, no_foreach: 0.01796465297229588
- doctr_det_predictor, AdamW, cuda, differentiable: 0.03641272569075227
- doctr_det_predictor, AdamW, cuda, foreach: 0.003495411202311516
- doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable: 0.00614771522115916
- doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.006344107356853783
- doctr_det_predictor, AdamW, cuda, fused: 0.001352854110300541
- doctr_det_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0014202436991035939
- doctr_det_predictor, AdamW, cuda, fused, capturable: 0.0013530117482878268
- doctr_det_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0014192780340090395
- doctr_det_predictor, Adamax, cuda, default: 0.017358969803899525
- doctr_det_predictor, Adamax, cuda, maximize: 0.018484242423437536
- doctr_det_predictor, Adamax, cuda, no_foreach: 0.02334124930202961
- doctr_det_predictor, Adamax, cuda, differentiable: 0.026815916411578655
- doctr_det_predictor, Adamax, cuda, foreach: 0.017275632079690694
- doctr_det_predictor, ASGD, cuda, default: 0.007048043450340629
- doctr_det_predictor, ASGD, cuda, maximize: 0.007594040366820991
- doctr_det_predictor, ASGD, cuda, no_foreach: 0.012185968761332332
- doctr_det_predictor, ASGD, cuda, differentiable: 0.012251253332942726
- doctr_det_predictor, ASGD, cuda, foreach: 0.0069041242403909565
- doctr_det_predictor, SGD, cuda, default: 0.0007654977729544043
- doctr_det_predictor, SGD, cuda, maximize: 0.0013926966651342811
- doctr_det_predictor, SGD, cuda, no_foreach: 0.001658875134307891
- doctr_det_predictor, SGD, cuda, differentiable: 0.000779025873169303
- doctr_det_predictor, SGD, cuda, foreach: 0.0006166908228769899
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0010155974142253398
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9: 0.0008244425822049379
- doctr_det_predictor, RAdam, cuda, default: 0.0040710537601262335
- doctr_det_predictor, RAdam, cuda, no_foreach: 0.026744291465729476
- doctr_det_predictor, RAdam, cuda, differentiable: 0.027759813191369175
- doctr_det_predictor, RAdam, cuda, foreach: 0.004023751728236675
- doctr_det_predictor, Rprop, cuda, default: 0.021900387201458215
- doctr_det_predictor, Rprop, cuda, maximize: 0.022019408503547312
- doctr_det_predictor, Rprop, cuda, no_foreach: 0.03515805988572538
- doctr_det_predictor, Rprop, cuda, differentiable: 0.03796613998711109
- doctr_det_predictor, Rprop, cuda, foreach: 0.021781701827421784
- doctr_det_predictor, RMSprop, cuda, default: 0.0018626715266145766
- doctr_det_predictor, RMSprop, cuda, maximize: 0.002432669820263982
- doctr_det_predictor, RMSprop, cuda, no_foreach: 0.008774845069274307
- doctr_det_predictor, RMSprop, cuda, differentiable: 0.00953097357414663
- doctr_det_predictor, RMSprop, cuda, foreach: 0.0016923811379820108
- doctr_det_predictor, NAdam, cuda, default: 0.004904018267989158
- doctr_det_predictor, NAdam, cuda, no_foreach: 0.019678202155046166
- doctr_det_predictor, NAdam, cuda, differentiable: 0.03470979649573565
- doctr_det_predictor, NAdam, cuda, foreach: 0.004748878921382129
- doctr_reco_predictor, Adadelta, cuda, default: 0.0015111762052401902
- doctr_reco_predictor, Adadelta, cuda, maximize: 0.0016182090458460151
- doctr_reco_predictor, Adadelta, cuda, no_foreach: 0.006300687254406512
- doctr_reco_predictor, Adadelta, cuda, differentiable: 0.007525572529993951
- doctr_reco_predictor, Adadelta, cuda, foreach: 0.0014990748558193446
- doctr_reco_predictor, Adagrad, cuda, default: 0.00138232063036412
- doctr_reco_predictor, Adagrad, cuda, maximize: 0.001363828368484974
- doctr_reco_predictor, Adagrad, cuda, no_foreach: 0.003358970070257783
- doctr_reco_predictor, Adagrad, cuda, differentiable: 0.003691559312865138
- doctr_reco_predictor, Adagrad, cuda, foreach: 0.001262800255790353
- doctr_reco_predictor, Adam, cuda, default: 0.0014186731097288428
- doctr_reco_predictor, Adam, cuda, amsgrad, maximize: 0.0017028246191330256
- doctr_reco_predictor, Adam, cuda, no_foreach: 0.005939919068478048
- doctr_reco_predictor, Adam, cuda, differentiable: 0.012200380908325315
- doctr_reco_predictor, Adam, cuda, foreach: 0.0013791421893984078
- doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable: 0.00229101890232414
- doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0023586724093183874
- doctr_reco_predictor, Adam, cuda, fused: 0.000526725591160357
- doctr_reco_predictor, Adam, cuda, fused, amsgrad, maximize: 0.0007335765808820725
- doctr_reco_predictor, Adam, cuda, fused, capturable: 0.000526425426825881
- doctr_reco_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0007332733031362295
- doctr_reco_predictor, AdamW, cuda, default: 0.0015104900393635035
- doctr_reco_predictor, AdamW, cuda, amsgrad, maximize: 0.001709991169627756
- doctr_reco_predictor, AdamW, cuda, no_foreach: 0.006635033590719104
- doctr_reco_predictor, AdamW, cuda, differentiable: 0.012207796215079725
- doctr_reco_predictor, AdamW, cuda, foreach: 0.0013919418444857002
- doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable: 0.002326308689080179
- doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.002375237848609686
- doctr_reco_predictor, AdamW, cuda, fused: 0.000545838036108762
- doctr_reco_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0007518859459087253
- doctr_reco_predictor, AdamW, cuda, fused, capturable: 0.0005431951717473567
- doctr_reco_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0007545708888210356
- doctr_reco_predictor, Adamax, cuda, default: 0.00614970067050308
- doctr_reco_predictor, Adamax, cuda, maximize: 0.0062665791902691125
- doctr_reco_predictor, Adamax, cuda, no_foreach: 0.007987272930331529
- doctr_reco_predictor, Adamax, cuda, differentiable: 0.009259536598498623
- doctr_reco_predictor, Adamax, cuda, foreach: 0.006115082018077374
- doctr_reco_predictor, ASGD, cuda, default: 0.002603447032161057
- doctr_reco_predictor, ASGD, cuda, maximize: 0.002844027699902654
- doctr_reco_predictor, ASGD, cuda, no_foreach: 0.004519692133180797
- doctr_reco_predictor, ASGD, cuda, differentiable: 0.004565841248258948
- doctr_reco_predictor, ASGD, cuda, foreach: 0.0025234638294205068
- doctr_reco_predictor, SGD, cuda, default: 0.000316370100248605
- doctr_reco_predictor, SGD, cuda, maximize: 0.0005366304907947778
- doctr_reco_predictor, SGD, cuda, no_foreach: 0.0006179931699298322
- doctr_reco_predictor, SGD, cuda, differentiable: 0.00031235945085063575
- doctr_reco_predictor, SGD, cuda, foreach: 0.00024404445104300975
- doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0006132524800486862
- doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9: 0.00044030676083639265
- doctr_reco_predictor, RAdam, cuda, default: 0.001608842380810529
- doctr_reco_predictor, RAdam, cuda, no_foreach: 0.00962122555822134
- doctr_reco_predictor, RAdam, cuda, differentiable: 0.00987512654159218
- doctr_reco_predictor, RAdam, cuda, foreach: 0.0015390151273459197
- doctr_reco_predictor, Rprop, cuda, default: 0.008077386738732458
- doctr_reco_predictor, Rprop, cuda, maximize: 0.007936706417240202
- doctr_reco_predictor, Rprop, cuda, no_foreach: 0.012451964942738415
- doctr_reco_predictor, Rprop, cuda, differentiable: 0.013715455750934781
- doctr_reco_predictor, Rprop, cuda, foreach: 0.007821803311817348
- doctr_reco_predictor, RMSprop, cuda, default: 0.0007429753746837378
- doctr_reco_predictor, RMSprop, cuda, maximize: 0.0009531616078068812
- doctr_reco_predictor, RMSprop, cuda, no_foreach: 0.0031920567899942397
- doctr_reco_predictor, RMSprop, cuda, differentiable: 0.003331667738966644
- doctr_reco_predictor, RMSprop, cuda, foreach: 0.000701989240013063
- doctr_reco_predictor, NAdam, cuda, default: 0.0019406061386689545
- doctr_reco_predictor, NAdam, cuda, no_foreach: 0.007075377199798822
- doctr_reco_predictor, NAdam, cuda, differentiable: 0.012110543437302113
- doctr_reco_predictor, NAdam, cuda, foreach: 0.0018546592863276602


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6056226522

cc @janeyx99


415k% is quite a big percent hmmm

also the doctr_reco_predictor thing seems broken upstream…

Yes I can confirm that doctr_reco_predictor is an upstream model issue, looking into it.

Looking at my dashboard, the 400k% came back down:
<img width="386" alt="image" src="https://github.com/pytorch/benchmark/assets/31798555/f5763a5d-4bac-4c17-89ff-12fc86c0690d">