Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'f54acf099180d680076065f7fdcc7c8be578a91e'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: c0a66455c03fd83cdc4edef9d1e26953e393dc68

Affected PyTorch commit: f54acf099180d680076065f7fdcc7c8be578a91e

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 0.024716824241396453
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: 0.0382675287934641
- timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 0.025371646275743842
- timm_vision_transformer_large, Adadelta, cuda, foreach: 0.08886035609369476
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 0.023466481102837458
- timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 0.03114198161555188
- timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 0.023240907686865993
- timm_vision_transformer_large, Adam, cuda, (pt2) default: 0.030208409364734377
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 0.057205872726626694
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.032477361017039845
- timm_vision_transformer_large, Adam, cuda, foreach: 0.05823490507900715
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable: 0.07724993551770847
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.08769935726498564
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.0297942173534206
- timm_vision_transformer_large, Adam, cuda, fused: 0.026534095406532288
- timm_vision_transformer_large, Adam, cuda, fused, amsgrad, maximize: 0.030286870198324323
- timm_vision_transformer_large, Adam, cuda, fused, capturable: 0.02650513080880046
- timm_vision_transformer_large, Adam, cuda, fused, capturable, amsgrad: 0.0301821727771312
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03143281536176801
- timm_vision_transformer_large, AdamW, cuda, default: 0.06976253508279721
- timm_vision_transformer_large, AdamW, cuda, amsgrad, maximize: 0.08423921031256516
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.05751750967465341
- timm_vision_transformer_large, AdamW, cuda, no_foreach: 0.07014404330402613
- timm_vision_transformer_large, AdamW, cuda, differentiable: 0.0844041380720834
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.031338553476546495
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.0689435916331907
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable: 0.0839275650990506
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.09426943212747574
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.031078948439764126
- timm_vision_transformer_large, AdamW, cuda, fused: 0.02731346068903804
- timm_vision_transformer_large, AdamW, cuda, fused, amsgrad, maximize: 0.030564517714083195
- timm_vision_transformer_large, AdamW, cuda, fused, capturable: 0.027261663926765323
- timm_vision_transformer_large, AdamW, cuda, fused, capturable, amsgrad: 0.030560680478811265
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.08779065714528163
- timm_vision_transformer_large, Adamax, cuda, default: 0.0835760614524285
- timm_vision_transformer_large, Adamax, cuda, maximize: 0.08878852427005768
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.0727145482475559
- timm_vision_transformer_large, Adamax, cuda, no_foreach: 0.07136374836166699
- timm_vision_transformer_large, Adamax, cuda, differentiable: 0.07892704615369439
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08562881189088027
- timm_vision_transformer_large, Adamax, cuda, foreach: 0.08282822541271646
- timm_vision_transformer_large, ASGD, cuda, default: 0.024184560077264904
- timm_vision_transformer_large, ASGD, cuda, maximize: 0.030476126028224824
- timm_vision_transformer_large, ASGD, cuda, no_foreach: 0.02997885369695723
- timm_vision_transformer_large, ASGD, cuda, differentiable: 0.030123180896043777
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.024193057138472796
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011630810124592649
- timm_vision_transformer_large, SGD, cuda, default: 0.009698009522010883
- timm_vision_transformer_large, SGD, cuda, maximize: 0.016332991467788816
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.013455380344142516
- timm_vision_transformer_large, SGD, cuda, no_foreach: 0.010720461490564048
- timm_vision_transformer_large, SGD, cuda, differentiable: 0.009668166780223449
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.011415067568628324
- timm_vision_transformer_large, SGD, cuda, foreach: 0.009640619127700726
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 2.5152681740000844
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9, nesterov: 0.03532830947078765
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 2.4496319331228733
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9: 0.02573967631906271
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07672884687781334
- timm_vision_transformer_large, RAdam, cuda, default: 0.0768322441726923
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09590499368496239
- timm_vision_transformer_large, RAdam, cuda, no_foreach: 0.09591501220129431
- timm_vision_transformer_large, RAdam, cuda, differentiable: 0.09642023569904268
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07141922800801695
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.07703303638845682
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.028798684810421298
- timm_vision_transformer_large, Rprop, cuda, default: 0.11165153025649488
- timm_vision_transformer_large, Rprop, cuda, maximize: 0.12469612760469317
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03141881798260978
- timm_vision_transformer_large, Rprop, cuda, no_foreach: 0.11928873788565397
- timm_vision_transformer_large, Rprop, cuda, differentiable: 0.126827378757298
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.0293106950287308
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.11217139894142747
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.019677193835377693
- timm_vision_transformer_large, RMSprop, cuda, default: 0.04185558510944247
- timm_vision_transformer_large, RMSprop, cuda, maximize: 0.048396945698186755
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.02886543163497533
- timm_vision_transformer_large, RMSprop, cuda, no_foreach: 0.04354904359206557
- timm_vision_transformer_large, RMSprop, cuda, differentiable: 0.04427406028844416
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.01926916583695195
- timm_vision_transformer_large, RMSprop, cuda, foreach: 0.04184862920083106
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08738328361262877
- timm_vision_transformer_large, NAdam, cuda, default: 0.07466190711905558
- timm_vision_transformer_large, NAdam, cuda, (pt2) no_foreach: 0.06147412641439587
- timm_vision_transformer_large, NAdam, cuda, no_foreach: 0.07630786625668406
- timm_vision_transformer_large, NAdam, cuda, differentiable: 0.094220703933388
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.0862225020925204
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.07493501715362072


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5995440501

cc @janeyx99


Seems related to https://github.com/pytorch/benchmark/issues/1839