Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'c0a66455c03fd83cdc4edef9d1e26953e393dc68'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 91bde862b20971c6c0176ae10759a3412751b304

Affected PyTorch commit: c0a66455c03fd83cdc4edef9d1e26953e393dc68

Affected Tests:
- DALLE2_pytorch, Adam, cuda, amsgrad, maximize: +32.02568%
- resnet18, AdamW, cuda, (pt2) no_foreach: -30.34275%
- resnet18, ASGD, cuda, (pt2) no_foreach: -38.31696%
- resnet18, NAdam, cuda, (pt2) no_foreach: -53.39803%
- timm_vision_transformer, Adagrad, cuda, (pt2) no_foreach: -35.90730%
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: -37.42656%


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 0.02517479582456872
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: 0.042113601602613925
- timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 0.025199367897585034
- timm_vision_transformer_large, Adadelta, cuda, foreach: 0.08877239399589598
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 0.023192190332338214
- timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 0.03683354895717154
- timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 0.023422410323595006
- timm_vision_transformer_large, Adam, cuda, (pt2) default: 0.030002495406993797
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 0.0704503448990484
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.03253361945306616
- timm_vision_transformer_large, Adam, cuda, foreach: 0.05818683230318129
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable: 0.07751660491339862
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.08688848706272741
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.030093037855944464
- timm_vision_transformer_large, Adam, cuda, fused: 0.026526149804703893
- timm_vision_transformer_large, Adam, cuda, fused, amsgrad, maximize: 0.030278270901180803
- timm_vision_transformer_large, Adam, cuda, fused, capturable: 0.02657948441337794
- timm_vision_transformer_large, Adam, cuda, fused, capturable, amsgrad: 0.03025384561624378
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03207084220567984
- timm_vision_transformer_large, AdamW, cuda, default: 0.06918989502203961
- timm_vision_transformer_large, AdamW, cuda, amsgrad, maximize: 0.08388727189352115
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.06934581696987152
- timm_vision_transformer_large, AdamW, cuda, no_foreach: 0.07012560940347612
- timm_vision_transformer_large, AdamW, cuda, differentiable: 0.08717114765507479
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.031968003271945884
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.06890591870372494
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable: 0.08355130488052964
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.09345587374021609
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.030986882613173554
- timm_vision_transformer_large, AdamW, cuda, fused: 0.027338458597660063
- timm_vision_transformer_large, AdamW, cuda, fused, amsgrad, maximize: 0.03057325470726937
- timm_vision_transformer_large, AdamW, cuda, fused, capturable: 0.02733217440545559
- timm_vision_transformer_large, AdamW, cuda, fused, capturable, amsgrad: 0.03061648760922253
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.08692726570491989
- timm_vision_transformer_large, Adamax, cuda, default: 0.08283174363896251
- timm_vision_transformer_large, Adamax, cuda, maximize: 0.08761599139931302
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.0847952920012176
- timm_vision_transformer_large, Adamax, cuda, no_foreach: 0.07164045454313357
- timm_vision_transformer_large, Adamax, cuda, differentiable: 0.07887981335322063
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08557407232001424
- timm_vision_transformer_large, Adamax, cuda, foreach: 0.08272632436516385
- timm_vision_transformer_large, ASGD, cuda, default: 0.024178724712692203
- timm_vision_transformer_large, ASGD, cuda, maximize: 0.03043388039804995
- timm_vision_transformer_large, ASGD, cuda, no_foreach: 0.03184148771688342
- timm_vision_transformer_large, ASGD, cuda, differentiable: 0.04225007810164243
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.024224729579873382
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011336652691372566
- timm_vision_transformer_large, SGD, cuda, default: 0.009684292095092436
- timm_vision_transformer_large, SGD, cuda, maximize: 0.016274779255036266
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.013498778423915307
- timm_vision_transformer_large, SGD, cuda, no_foreach: 0.010688607499469071
- timm_vision_transformer_large, SGD, cuda, differentiable: 0.009663545503281057
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.011435453560099833
- timm_vision_transformer_large, SGD, cuda, foreach: 0.009622758727831146
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 2.4633423609193414
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9, nesterov: 0.03526389990001917
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 2.4311089541297406
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9: 0.025745808985084294
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07688460770683984
- timm_vision_transformer_large, RAdam, cuda, default: 0.07657132165816923
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09566786061041058
- timm_vision_transformer_large, RAdam, cuda, no_foreach: 0.09568731330800802
- timm_vision_transformer_large, RAdam, cuda, differentiable: 0.09611548008397222
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07643889465058844
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.07661378189610939
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.02804035515873693
- timm_vision_transformer_large, Rprop, cuda, default: 0.11131335550453514
- timm_vision_transformer_large, Rprop, cuda, maximize: 0.12439714150968939
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03117914398067764
- timm_vision_transformer_large, Rprop, cuda, no_foreach: 0.11827560502570122
- timm_vision_transformer_large, Rprop, cuda, differentiable: 0.12571769405622035
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.028606365135471736
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.11158771545160562
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.01958702462302013
- timm_vision_transformer_large, RMSprop, cuda, default: 0.04184102329891175
- timm_vision_transformer_large, RMSprop, cuda, maximize: 0.04828611069824547
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.03211232401164515
- timm_vision_transformer_large, RMSprop, cuda, no_foreach: 0.04339892908465117
- timm_vision_transformer_large, RMSprop, cuda, differentiable: 0.04414248040411621
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.019085544817657632
- timm_vision_transformer_large, RMSprop, cuda, foreach: 0.041833868413232264
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08762203700219591
- timm_vision_transformer_large, NAdam, cuda, default: 0.07453014406686027
- timm_vision_transformer_large, NAdam, cuda, no_foreach: 0.07571519127426048
- timm_vision_transformer_large, NAdam, cuda, differentiable: 0.09347509771275024
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.08537063701078296
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.0745349876427402


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5988766790

cc @janeyx99


Seems related to https://github.com/pytorch/benchmark/issues/1839