Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '3522f2a7b7f73e928a8366cb7bd62ab3883dbe75'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected PyTorch commit: 3522f2a7b7f73e928a8366cb7bd62ab3883dbe75

Affected Tests:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) default: -74.20654%
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: -54.46165%
- timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: -73.50638%
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: -56.98138%
- timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: -56.00882%
- timm_vision_transformer_large, Adam, cuda, (pt2) default: -60.92160%
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: -35.80434%
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: -56.35303%
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: -61.52401%
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: -35.62405%
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: -59.84570%
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: -74.02662%
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: -69.21701%
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: -74.06030%
- resnet18, NAdam, cuda, (pt2) default: +69.52585%
- resnet18, NAdam, cuda, (pt2) no_foreach: +43.28397%
- resnet18, NAdam, cuda, (pt2) foreach: +76.11730%
- timm_vision_transformer, NAdam, cuda, (pt2) default: +98.00990%
- timm_vision_transformer, NAdam, cuda, (pt2) foreach: +93.78024%


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.06946720439009368
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 2.7884387269150466
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.06552731158444658
- timm_vision_transformer_large, NAdam, cuda, (pt2) no_foreach: 0.08795584738254547
- resnet18, ASGD, cuda, (pt2) default: 2.738545122090727
- resnet18, ASGD, cuda, (pt2) foreach: 1.3846970193553716
- timm_vision_transformer, ASGD, cuda, (pt2) default: 0.9439682896869878
- timm_vision_transformer, ASGD, cuda, (pt2) foreach: 0.7286172377644107
- clip, Adadelta, cuda, default: 0.013201068702619523
- clip, Adadelta, cuda, maximize: 0.014266475301701576
- clip, Adadelta, cuda, no_foreach: 0.027475361595861614
- clip, Adadelta, cuda, differentiable: 0.031395936687476936
- clip, Adadelta, cuda, foreach: 0.01323636280139908
- clip, Adagrad, cuda, default: 0.006242717020213604
- clip, Adagrad, cuda, maximize: 0.00723681635921821
- clip, Adagrad, cuda, no_foreach: 0.014198726648464798
- clip, Adagrad, cuda, differentiable: 0.015473100647795945
- clip, Adagrad, cuda, foreach: 0.006267887121066451
- clip, Adam, cuda, default: 0.008960058966962002
- clip, Adam, cuda, amsgrad, maximize: 0.011395436199381947
- clip, Adam, cuda, no_foreach: 0.02523956010118127
- clip, Adam, cuda, differentiable: 0.05106047261506319
- clip, Adam, cuda, foreach: 0.008955305628478528
- clip, Adam, cuda, foreach, maximize, capturable: 0.013667864550370723
- clip, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.014539040392264724
- clip, Adam, cuda, fused: 0.004355747778899968
- clip, Adam, cuda, fused, amsgrad, maximize: 0.00520610025851056
- clip, Adam, cuda, fused, capturable: 0.0043514958210289474
- clip, Adam, cuda, fused, capturable, amsgrad: 0.00520092393970117
- clip, AdamW, cuda, default: 0.009922639233991504
- clip, AdamW, cuda, amsgrad, maximize: 0.012345757451839745
- clip, AdamW, cuda, no_foreach: 0.02821788629516959
- clip, AdamW, cuda, differentiable: 0.05146769001148641
- clip, AdamW, cuda, foreach: 0.009926098228121797
- clip, AdamW, cuda, foreach, maximize, capturable: 0.014088112104218453
- clip, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.015522209496703
- clip, AdamW, cuda, fused: 0.004489545160904527
- clip, AdamW, cuda, fused, amsgrad, maximize: 0.0052894597197882835
- clip, AdamW, cuda, fused, capturable: 0.004483225820586086
- clip, AdamW, cuda, fused, capturable, amsgrad: 0.005285731060430407
- clip, Adamax, cuda, default: 0.0249062845017761
- clip, Adamax, cuda, maximize: 0.02474480150267482
- clip, Adamax, cuda, no_foreach: 0.03191633310634643
- clip, Adamax, cuda, differentiable: 0.03847420650999993
- clip, Adamax, cuda, foreach: 0.02372765720356256
- clip, ASGD, cuda, default: 0.011097999557387083
- clip, ASGD, cuda, maximize: 0.011641559959389269
- clip, ASGD, cuda, no_foreach: 0.01847794944187626
- clip, ASGD, cuda, differentiable: 0.018602025648579
- clip, ASGD, cuda, foreach: 0.010482863313518465
- clip, SGD, cuda, default: 0.001457272438565269
- clip, SGD, cuda, maximize: 0.002440258280839771
- clip, SGD, cuda, no_foreach: 0.0025550051615573464
- clip, SGD, cuda, differentiable: 0.0014523062598891555
- clip, SGD, cuda, foreach: 0.0014472876209765673
- clip, SGD, cuda, foreach, momentum=0.9, nesterov: 0.005311355970334262
- clip, SGD, cuda, foreach, momentum=0.9: 0.003866169392131269
- clip, RAdam, cuda, default: 0.010977191757410763
- clip, RAdam, cuda, no_foreach: 0.04044272389728576
- clip, RAdam, cuda, differentiable: 0.041530937305651605
- clip, RAdam, cuda, foreach: 0.010951962543185801
- clip, Rprop, cuda, default: 0.03363393938634544
- clip, Rprop, cuda, maximize: 0.03777411391492933
- clip, Rprop, cuda, no_foreach: 0.052296759001910685
- clip, Rprop, cuda, differentiable: 0.05560901451390236
- clip, Rprop, cuda, foreach: 0.033326059393584725
- clip, RMSprop, cuda, default: 0.006237020930275321
- clip, RMSprop, cuda, maximize: 0.007238076031208038
- clip, RMSprop, cuda, no_foreach: 0.013629035942722113
- clip, RMSprop, cuda, differentiable: 0.014284973707981409
- clip, RMSprop, cuda, foreach: 0.0062284826091490685
- clip, NAdam, cuda, default: 0.011986838444136083
- clip, NAdam, cuda, no_foreach: 0.03197967128362507
- clip, NAdam, cuda, differentiable: 0.05420875509735197
- clip, NAdam, cuda, foreach: 0.011986217147205025


Tests that were newly added on affected commit:
- nvidia_deeprecommender, Adadelta, cuda, default: 0.017446905898395926
- nvidia_deeprecommender, Adadelta, cuda, maximize: 0.018748890107963233
- nvidia_deeprecommender, Adadelta, cuda, no_foreach: 0.016450105386320503
- nvidia_deeprecommender, Adadelta, cuda, differentiable: 0.0177056627930142
- nvidia_deeprecommender, Adadelta, cuda, foreach: 0.017435872415080666
- nvidia_deeprecommender, Adagrad, cuda, default: 0.00831607230938971
- nvidia_deeprecommender, Adagrad, cuda, maximize: 0.00964826566632837
- nvidia_deeprecommender, Adagrad, cuda, no_foreach: 0.006660659869667142
- nvidia_deeprecommender, Adagrad, cuda, differentiable: 0.0066388667817227544
- nvidia_deeprecommender, Adagrad, cuda, foreach: 0.008325473139993846
- nvidia_deeprecommender, Adam, cuda, default: 0.011598488665185869
- nvidia_deeprecommender, Adam, cuda, amsgrad, maximize: 0.014791397936642171
- nvidia_deeprecommender, Adam, cuda, no_foreach: 0.01092696749838069
- nvidia_deeprecommender, Adam, cuda, differentiable: 0.010918727901298553
- nvidia_deeprecommender, Adam, cuda, foreach: 0.011584434960968792
- nvidia_deeprecommender, Adam, cuda, foreach, maximize, capturable: 0.014089150750078262
- nvidia_deeprecommender, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.01598952104104683
- nvidia_deeprecommender, Adam, cuda, fused: 0.0052793167205527425
- nvidia_deeprecommender, Adam, cuda, fused, amsgrad, maximize: 0.006044540300499648
- nvidia_deeprecommender, Adam, cuda, fused, capturable: 0.00527084126835689
- nvidia_deeprecommender, Adam, cuda, fused, capturable, amsgrad: 0.006071205688640475
- nvidia_deeprecommender, AdamW, cuda, default: 0.012903886893764139
- nvidia_deeprecommender, AdamW, cuda, amsgrad, maximize: 0.016106620989739896
- nvidia_deeprecommender, AdamW, cuda, no_foreach: 0.012132873793598264
- nvidia_deeprecommender, AdamW, cuda, differentiable: 0.012142965849488974
- nvidia_deeprecommender, AdamW, cuda, foreach: 0.012907146604266017
- nvidia_deeprecommender, AdamW, cuda, foreach, maximize, capturable: 0.01540248425444588
- nvidia_deeprecommender, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.01730604284675792
- nvidia_deeprecommender, AdamW, cuda, fused: 0.0054086608299985526
- nvidia_deeprecommender, AdamW, cuda, fused, amsgrad, maximize: 0.0061228928016498685
- nvidia_deeprecommender, AdamW, cuda, fused, capturable: 0.005414998151827603
- nvidia_deeprecommender, AdamW, cuda, fused, capturable, amsgrad: 0.006108615880366415
- nvidia_deeprecommender, Adamax, cuda, default: 0.014486303098965436
- nvidia_deeprecommender, Adamax, cuda, maximize: 0.015792385942768307
- nvidia_deeprecommender, Adamax, cuda, no_foreach: 0.012202103191521018
- nvidia_deeprecommender, Adamax, cuda, differentiable: 0.01347301120404154
- nvidia_deeprecommender, Adamax, cuda, foreach: 0.014481523446738719
- nvidia_deeprecommender, ASGD, cuda, default: 0.004492332020308822
- nvidia_deeprecommender, ASGD, cuda, maximize: 0.005784110978711396
- nvidia_deeprecommender, ASGD, cuda, no_foreach: 0.004321837411262095
- nvidia_deeprecommender, ASGD, cuda, differentiable: 0.0043204523297026756
- nvidia_deeprecommender, ASGD, cuda, foreach: 0.004494141228497029
- nvidia_deeprecommender, SGD, cuda, default: 0.0019152774382382631
- nvidia_deeprecommender, SGD, cuda, maximize: 0.003204710530117154
- nvidia_deeprecommender, SGD, cuda, no_foreach: 0.0018399484606925398
- nvidia_deeprecommender, SGD, cuda, differentiable: 0.001909809830831364
- nvidia_deeprecommender, SGD, cuda, foreach: 0.001909827624913305
- nvidia_deeprecommender, SGD, cuda, foreach, momentum=0.9, nesterov: 0.007033650521188975
- nvidia_deeprecommender, SGD, cuda, foreach, momentum=0.9: 0.005119151291437447
- nvidia_deeprecommender, RAdam, cuda, default: 0.014233437145594507
- nvidia_deeprecommender, RAdam, cuda, no_foreach: 0.017013703542761504
- nvidia_deeprecommender, RAdam, cuda, differentiable: 0.016996120347175746
- nvidia_deeprecommender, RAdam, cuda, foreach: 0.014237003796733916
- nvidia_deeprecommender, Rprop, cuda, default: 0.019901570212095977
- nvidia_deeprecommender, Rprop, cuda, maximize: 0.02252480259630829
- nvidia_deeprecommender, Rprop, cuda, no_foreach: 0.020751465996727347
- nvidia_deeprecommender, Rprop, cuda, differentiable: 0.022010007011704146
- nvidia_deeprecommender, Rprop, cuda, foreach: 0.01990795275196433
- nvidia_deeprecommender, RMSprop, cuda, default: 0.00834360634131978
- nvidia_deeprecommender, RMSprop, cuda, maximize: 0.00963982252869755
- nvidia_deeprecommender, RMSprop, cuda, no_foreach: 0.007880905627583463
- nvidia_deeprecommender, RMSprop, cuda, differentiable: 0.007853274070657789
- nvidia_deeprecommender, RMSprop, cuda, foreach: 0.00834497583564371
- nvidia_deeprecommender, NAdam, cuda, default: 0.015415665542241185
- nvidia_deeprecommender, NAdam, cuda, no_foreach: 0.014516560651827604
- nvidia_deeprecommender, NAdam, cuda, differentiable: 0.01690723680658266
- nvidia_deeprecommender, NAdam, cuda, foreach: 0.015399227652233094


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5898649956

cc @janeyx99


@mlazos it looks like there was a change recently that greatly improved the pt2 perf of bigger models (timm_vision_transformer_large) but increased overhead on smaller models (resnet18)--does this have to do with your cudagraph stuffs?

Also, @mlazos, we get errors for ASGD: `scalar tensor is expected to be on cuda:0 but is on cpu` and OOMs for some runtime regressions. See the log in https://github.com/pytorch/benchmark/actions/runs/5898649956.

The clip and nvidia deep recommender getting removed makes sense. Rest of issues are getting resolved and tracked by #1839