Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '4a4af8fb1cc638ea249fb5afc2b2d1ade091d204'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 5c903fb12cd09ddb306364d06b2c8da581bf481b

Affected PyTorch commit: 4a4af8fb1cc638ea249fb5afc2b2d1ade091d204

Affected Tests:
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: +399152.13618%
- resnet18, Adagrad, cuda, (pt2) no_foreach: +639550.66409%
- resnet18, Adam, cuda, (pt2) no_foreach: +623618.78470%
- resnet18, Adam, cuda, (pt2) foreach: +133.90065%
- compile_time, resnet18, Adadelta, cuda, (pt2) foreach: +1662.83555%


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.03172703759212579
- timm_vision_transformer_large, Adam, cuda, foreach: 0.05825375760905445
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable: 0.07747577037662268
- timm_vision_transformer_large, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.08797353071471055
- timm_vision_transformer_large, Adam, cuda, (pt2) fused: 0.030198675646845783
- timm_vision_transformer_large, Adam, cuda, fused: 0.02851294633001089
- timm_vision_transformer_large, Adam, cuda, fused, amsgrad, maximize: 0.03239971958100796
- timm_vision_transformer_large, Adam, cuda, fused, capturable: 0.026526362309232353
- timm_vision_transformer_large, Adam, cuda, fused, capturable, amsgrad: 0.030268650874495505
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.03229453494506223
- timm_vision_transformer_large, AdamW, cuda, default: 0.06926546012982726
- timm_vision_transformer_large, AdamW, cuda, amsgrad, maximize: 0.08410635807861884
- timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 0.056811471353285015
- timm_vision_transformer_large, AdamW, cuda, no_foreach: 0.07133351918309927
- timm_vision_transformer_large, AdamW, cuda, differentiable: 0.09228928418209155
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.03104895246880395
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.06894068885594606
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable: 0.08388662834962209
- timm_vision_transformer_large, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.09356137070183952
- timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 0.03091035251106535
- timm_vision_transformer_large, AdamW, cuda, fused: 0.02904160586850984
- timm_vision_transformer_large, AdamW, cuda, fused, amsgrad, maximize: 0.03271023370325565
- timm_vision_transformer_large, AdamW, cuda, fused, capturable: 0.027374790422618388
- timm_vision_transformer_large, AdamW, cuda, fused, capturable, amsgrad: 0.030634437408298253
- timm_vision_transformer_large, Adamax, cuda, (pt2) default: 0.084533647634089
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.0254441742436029
- timm_vision_transformer_large, ASGD, cuda, default: 0.0681253089569509
- timm_vision_transformer_large, ASGD, cuda, maximize: 0.07530166441574693
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 0.03469913359731436
- timm_vision_transformer_large, ASGD, cuda, no_foreach: 0.08873575537775953
- timm_vision_transformer_large, ASGD, cuda, differentiable: 0.08537480064357321
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.024452018758489028
- timm_vision_transformer_large, ASGD, cuda, foreach: 0.06748595042154193
- timm_vision_transformer_large, SGD, cuda, (pt2) default: 0.011938756386585096
- timm_vision_transformer_large, SGD, cuda, default: 0.00968396095559001
- timm_vision_transformer_large, SGD, cuda, maximize: 0.016317470860667524
- timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 0.01272842608159408
- timm_vision_transformer_large, SGD, cuda, no_foreach: 0.010730910836718977
- timm_vision_transformer_large, SGD, cuda, differentiable: 0.009670840684945385
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 0.011248641595658328
- timm_vision_transformer_large, SGD, cuda, foreach: 0.009621488520254693
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.03531276248395443
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9, nesterov: 0.035273146675899626
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.025762591697275638
- timm_vision_transformer_large, SGD, cuda, foreach, momentum=0.9: 0.025791855482384562
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07834087265655398
- timm_vision_transformer_large, RAdam, cuda, default: 0.07643644325435162
- timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.09921934828162193
- timm_vision_transformer_large, RAdam, cuda, no_foreach: 0.09601866621524095
- timm_vision_transformer_large, RAdam, cuda, differentiable: 0.09646318377926946
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07595638403048117
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.0766671219219764
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: 0.027830363134853542
- timm_vision_transformer_large, Rprop, cuda, default: 0.11156303738243878
- timm_vision_transformer_large, Rprop, cuda, maximize: 0.12540722102858126
- timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 0.03610477402495841
- timm_vision_transformer_large, Rprop, cuda, no_foreach: 0.11929131392389536
- timm_vision_transformer_large, Rprop, cuda, differentiable: 0.1268004160374403
- timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 0.02728326979558915
- timm_vision_transformer_large, Rprop, cuda, foreach: 0.11148018413223326
- timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 0.018667407418516548
- timm_vision_transformer_large, RMSprop, cuda, default: 0.041753352619707584
- timm_vision_transformer_large, RMSprop, cuda, maximize: 0.048347185086458924
- timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 0.028424972493667156
- timm_vision_transformer_large, RMSprop, cuda, no_foreach: 0.04370795730501413
- timm_vision_transformer_large, RMSprop, cuda, differentiable: 0.044400359783321616
- timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 0.018297586293721742
- timm_vision_transformer_large, RMSprop, cuda, foreach: 0.041758734267205
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.08519644538561504
- timm_vision_transformer_large, NAdam, cuda, default: 0.07425836070130269
- timm_vision_transformer_large, NAdam, cuda, no_foreach: 0.0751144103705883
- timm_vision_transformer_large, NAdam, cuda, differentiable: 0.09496557945385575
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.0769409656835099
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.07384949441378315
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 370.21813965169713
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) fused: 6.877883844853689
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) default: 394.68320915584144
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 308.80047401677194
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 385.4122420880012
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 7.702689774334431
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) default: 46.037039051608495
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) default: 444.41376759205014
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 273.85431434819475
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 418.2794219877881
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) default: 31.896771671405684
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 26.06529251486063
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 2.0414253220272562
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 1.9593043229542673
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 0.03866999503225088
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07330969286461672
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.068772635422647
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07135765658070645
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) default: 597.4147356548347
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 249.36279953581592
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 566.8153645899147
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 189.71103541739285
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 118.36260319516684
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 170.09074211058518
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) default: 77.90669706913953
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 21.220644644616794


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6205165468

cc @janeyx99


Closing, nothing new reported. Will be implicitly fixed by https://github.com/pytorch/benchmark/pull/1931 and https://github.com/pytorch/benchmark/pull/1932