Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '4a4af8fb1cc638ea249fb5afc2b2d1ade091d204'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 4a4af8fb1cc638ea249fb5afc2b2d1ade091d204

Affected PyTorch commit: 4a4af8fb1cc638ea249fb5afc2b2d1ade091d204

Affected Tests:
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: -99.97434%
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: -6354.08724%
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) fused: +289.62741%
- resnet18, Adam, cuda, (pt2) no_foreach: +613910.70889%
- resnet18, Adam, cuda, (pt2) foreach: +129.10820%
- compile_time, resnet18, ASGD, cuda, (pt2) foreach: +949.27977%
- compile_time, resnet18, NAdam, cuda, (pt2) foreach: +219.49703%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.0316724570335022
- timm_vision_transformer_large, Adam, cuda, foreach: 0.05831579123623669
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable: 0.07803720887750387
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0870506417316695
- timm_vision_transformer_large, Adam, cuda, fused, amsgrad, maximize: 0.03271116941635098
- timm_vision_transformer_large, Adam, cuda, fused, capturable, amsgrad: 0.03039571028202772
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03449522804779311
- timm_vision_transformer_large, AdamW, cuda, default: 0.0701791220344603
- timm_vision_transformer_large, AdamW, cuda, amsgrad, maximize: 0.08395658045386274
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.057252108003012836
- timm_vision_transformer_large, AdamW, cuda, no_foreach: 0.07071703284357984
- timm_vision_transformer_large, AdamW, cuda, differentiable: 0.08897854682678978
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.0351479589783897
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.06999102296928565
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable: 0.08452818139145772
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.09510688514759143
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.03193474368059209
- timm_vision_transformer_large, AdamW, cuda, fused: 0.02949583696733628
- timm_vision_transformer_large, AdamW, cuda, fused, amsgrad, maximize: 0.03311907021062715
- timm_vision_transformer_large, AdamW, cuda, fused, capturable: 0.027354372898116708
- timm_vision_transformer_large, AdamW, cuda, fused, capturable, amsgrad: 0.030674395011737944
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.08697450331722696
- timm_vision_transformer_large, Adamax, cuda, default: 0.08322860455761354
- timm_vision_transformer_large, Adamax, cuda, maximize: 0.08840999271099766
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.0758169274777174
- timm_vision_transformer_large, Adamax, cuda, no_foreach: 0.07255596419175465
- timm_vision_transformer_large, Adamax, cuda, differentiable: 0.08347715468456347
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08600766708453496
- timm_vision_transformer_large, Adamax, cuda, foreach: 0.08311301578457157
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.025268584489822388
- timm_vision_transformer_large, ASGD, cuda, default: 0.07161454778785507
- timm_vision_transformer_large, ASGD, cuda, maximize: 0.07757902750745416
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 0.03381943850157162
- timm_vision_transformer_large, ASGD, cuda, no_foreach: 0.08318280444170038
- timm_vision_transformer_large, ASGD, cuda, differentiable: 0.0831349768365423
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.024148027774774365
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.06779415595034759
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011584135858962933
- timm_vision_transformer_large, SGD, cuda, default: 0.009682776592671872
- timm_vision_transformer_large, SGD, cuda, maximize: 0.016261164820753037
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.013717115049560864
- timm_vision_transformer_large, SGD, cuda, no_foreach: 0.010717641096562147
- timm_vision_transformer_large, SGD, cuda, differentiable: 0.009671627590432764
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.01130968376269771
- timm_vision_transformer_large, SGD, cuda, foreach: 0.009647988248616457
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.035222459118813276
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9, nesterov: 0.035177787812426684
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.025784677639603616
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9: 0.02572958990931511
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07141732322052122
- timm_vision_transformer_large, RAdam, cuda, default: 0.07626173598691821
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09806881596644719
- timm_vision_transformer_large, RAdam, cuda, no_foreach: 0.09592344490811229
- timm_vision_transformer_large, RAdam, cuda, differentiable: 0.09649490001611412
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07139673698693513
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.07586906338110566
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.026664226839784533
- timm_vision_transformer_large, Rprop, cuda, default: 0.11215957975946367
- timm_vision_transformer_large, Rprop, cuda, maximize: 0.12487144721671939
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03513214007640878
- timm_vision_transformer_large, Rprop, cuda, no_foreach: 0.11868995800614357
- timm_vision_transformer_large, Rprop, cuda, differentiable: 0.12654822343029082
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.027897977794054896
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.11218832293525338
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.019017981576987288
- timm_vision_transformer_large, RMSprop, cuda, default: 0.041810289258137345
- timm_vision_transformer_large, RMSprop, cuda, maximize: 0.04842791757546365
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.029251408031476394
- timm_vision_transformer_large, RMSprop, cuda, no_foreach: 0.04349315916188061
- timm_vision_transformer_large, RMSprop, cuda, differentiable: 0.04433563151396811
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.018190853043713352
- timm_vision_transformer_large, RMSprop, cuda, foreach: 0.04184431047178805
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08355305700873335
- timm_vision_transformer_large, NAdam, cuda, default: 0.07443050915996234
- timm_vision_transformer_large, NAdam, cuda, no_foreach: 0.07537155769144495
- timm_vision_transformer_large, NAdam, cuda, differentiable: 0.09166955885787804
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.07767417250821988
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.07518236137305696
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 371.6090245083906
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) default: 384.27291971212253
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 306.34515024178353
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 382.76453707087785
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 7.025264319498092
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) default: 58.6315590312394
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 312.5973138500315
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 10.46647108082349
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) default: 449.71154894384864
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 294.82842412404716
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 405.06688643153757
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) default: 31.096511099642765
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 26.025238506806392
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 2.008613512540857
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 1.9843328158992033
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.03312441287562251
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.060495725056777395
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.06413208832964301
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.05861988632629315
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) default: 590.5638497977828
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 252.8236700684453
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 555.3013096260838
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 192.14534045166025
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 118.90560779689501
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 182.0893174825857
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) default: 75.84660780460884
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 35.15553080275034


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6218037368

cc @janeyx99


Closing, nothing new reported. Will be implicitly fixed by https://github.com/pytorch/benchmark/pull/1931 and https://github.com/pytorch/benchmark/pull/1932