Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '4d1773d0c5ca48575ea0c53e3a043a5bc434776b'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 4a4af8fb1cc638ea249fb5afc2b2d1ade091d204

Affected PyTorch commit: 4d1773d0c5ca48575ea0c53e3a043a5bc434776b

Affected Tests:
- pytorch_unet, Adamax, cuda, no_foreach: +30.15432%
- hf_GPT2, Adam, cuda, differentiable: +34.23901%
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: +497404.09605%
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: +396699.98930%
- phlippe_densenet, Rprop, cuda, default: +32.88321%
- opacus_cifar10, AdamW, cuda, differentiable: +41.14107%
- detectron2_maskrcnn_r_101_fpn, Rprop, cuda, differentiable: +42.92647%
- DALLE2_pytorch, Adam, cuda, fused, amsgrad, maximize: +39.84832%
- phlippe_resnet, SGD, cuda, no_foreach: +55.33853%
- resnet18, Adam, cuda, (pt2) no_foreach: +610217.85766%
- resnet18, Adam, cuda, (pt2) foreach: +130.75038%
- compile_time, resnet18, Adamax, cuda, (pt2) foreach: +203.88620%
- compile_time, resnet18, SGD, cuda, (pt2) foreach: +784.64279%
- nanogpt_generate, Adamax, cuda, maximize: +32.54935%
- nanogpt_generate, Adamax, cuda, no_foreach: +30.20326%
- nanogpt_generate, Adamax, cuda, foreach: +38.15474%
- pytorch_stargan, Adamax, cuda, differentiable: +34.97999%


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.03241741417774132
- timm_vision_transformer_large, Adam, cuda, foreach: 0.08258711205174525
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable: 0.07854787353426218
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.08903549922009309
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.03212496265769005
- timm_vision_transformer_large, Adam, cuda, fused: 0.02952958843005555
- timm_vision_transformer_large, Adam, cuda, fused, amsgrad, maximize: 0.03356229762236277
- timm_vision_transformer_large, Adam, cuda, fused, capturable: 0.026630297023802995
- timm_vision_transformer_large, Adam, cuda, fused, capturable, amsgrad: 0.030373827647417785
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03235120539154325
- timm_vision_transformer_large, AdamW, cuda, default: 0.06979801288495462
- timm_vision_transformer_large, AdamW, cuda, amsgrad, maximize: 0.08434985515971978
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.05749114672653377
- timm_vision_transformer_large, AdamW, cuda, no_foreach: 0.07546847003201644
- timm_vision_transformer_large, AdamW, cuda, differentiable: 0.09333528702457745
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.034004210494458675
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.07038140824685495
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable: 0.08543585830678542
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.09544541190067928
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.03266184896762882
- timm_vision_transformer_large, AdamW, cuda, fused: 0.029894836779151644
- timm_vision_transformer_large, AdamW, cuda, fused, amsgrad, maximize: 0.0336314191420873
- timm_vision_transformer_large, AdamW, cuda, fused, capturable: 0.027351888362318277
- timm_vision_transformer_large, AdamW, cuda, fused, capturable, amsgrad: 0.030669845454394818
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.08802222087979317
- timm_vision_transformer_large, Adamax, cuda, default: 0.0842357265452544
- timm_vision_transformer_large, Adamax, cuda, maximize: 0.0895050372928381
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 0.07432674399266641
- timm_vision_transformer_large, Adamax, cuda, no_foreach: 0.07208388143529494
- timm_vision_transformer_large, Adamax, cuda, differentiable: 0.07956786671032508
- timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 0.08643113635480404
- timm_vision_transformer_large, Adamax, cuda, foreach: 0.08348223846405745
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.02609830629080534
- timm_vision_transformer_large, ASGD, cuda, default: 0.07199316161374252
- timm_vision_transformer_large, ASGD, cuda, maximize: 0.07906431363274653
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 0.03675717317188779
- timm_vision_transformer_large, ASGD, cuda, no_foreach: 0.08623406570404768
- timm_vision_transformer_large, ASGD, cuda, differentiable: 0.08660063830514748
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.02783471206203103
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.07258840401967366
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.012087432284127264
- timm_vision_transformer_large, SGD, cuda, default: 0.00973003025477131
- timm_vision_transformer_large, SGD, cuda, maximize: 0.016385755129158498
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.013287102105095983
- timm_vision_transformer_large, SGD, cuda, no_foreach: 0.01074524987488985
- timm_vision_transformer_large, SGD, cuda, differentiable: 0.009724162115405005
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.0117855774348273
- timm_vision_transformer_large, SGD, cuda, foreach: 0.009681930113583802
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.03529031379148364
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9, nesterov: 0.03524060761556029
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.02579468283802271
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9: 0.025782120041549207
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07791733214010794
- timm_vision_transformer_large, RAdam, cuda, default: 0.0779942674562335
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09845532601078351
- timm_vision_transformer_large, RAdam, cuda, no_foreach: 0.09582167891785502
- timm_vision_transformer_large, RAdam, cuda, differentiable: 0.09626505728811026
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07140608299523592
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.07744687609374523
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.027678536833263934
- timm_vision_transformer_large, Rprop, cuda, default: 0.11253243125975132
- timm_vision_transformer_large, Rprop, cuda, maximize: 0.12570577347651124
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.0371221707512935
- timm_vision_transformer_large, Rprop, cuda, no_foreach: 0.11985317477956414
- timm_vision_transformer_large, Rprop, cuda, differentiable: 0.12760619958862662
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.027441877755336463
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.11209800327196717
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.019280216317962517
- timm_vision_transformer_large, RMSprop, cuda, default: 0.04178968472406268
- timm_vision_transformer_large, RMSprop, cuda, maximize: 0.04832894401624799
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.028449304401874542
- timm_vision_transformer_large, RMSprop, cuda, no_foreach: 0.04343226635828614
- timm_vision_transformer_large, RMSprop, cuda, differentiable: 0.044272441696375606
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.01752886618487537
- timm_vision_transformer_large, RMSprop, cuda, foreach: 0.04369750656187534
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08516804936031501
- timm_vision_transformer_large, NAdam, cuda, default: 0.07628178720672925
- timm_vision_transformer_large, NAdam, cuda, no_foreach: 0.0760028250515461
- timm_vision_transformer_large, NAdam, cuda, differentiable: 0.09386057748148839
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.07826318374524514
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.0754976983492573
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 373.59219496448833
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) fused: 7.091818775671224
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) default: 394.42767009853077
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 307.8799784130727
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 375.08891114623594
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 8.027503235886494
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) default: 61.68055537932863
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 315.9682987652098
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 24.01290361583233
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) default: 448.1469118790701
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 319.22081204534817
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 435.1769620428483
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) default: 32.35708398402979
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 27.667252119941015
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 2.166107357169191
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 2.017007595859468
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.03432587658365568
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.0650776814048489
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.06502424335728088
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.06402730538199344
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) default: 641.1447192846487
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 270.06824701776105
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 592.3404514590899
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 191.37320780692002
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 119.65533386667569
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 16.177491795271635
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) default: 96.16184002812952
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 23.28606203974535


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6243831443

cc @janeyx99


Closing, nothing new reported. Will be indirectly fixed by https://github.com/pytorch/benchmark/pull/1931 and https://github.com/pytorch/benchmark/pull/1932