Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '0200b1106c4fe80ea0884181dc8d649ef6078ea3'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: a618118ba9ffc33097699c120e8760c0eb0f67fc

Affected PyTorch commit: 0200b1106c4fe80ea0884181dc8d649ef6078ea3

Affected Tests:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: +668561.29568%
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: -30.98947%
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: -33.05736%
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -98.55378%
- timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: -98.93612%
- resnet18, Adadelta, cuda, (pt2) no_foreach: +793810.29350%
- resnet18, Adagrad, cuda, (pt2) no_foreach: +722466.16303%
- resnet18, Adam, cuda, (pt2) foreach: +872892.07472%
- resnet18, Adam, cuda, (pt2) fused: -44.17049%
- resnet18, AdamW, cuda, (pt2) fused: -46.89192%
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -99.83606%
- resnet18, SGD, cuda, (pt2) foreach, momentum=0.9: -99.82933%
- resnet18, Rprop, cuda, (pt2) foreach: +1585891.20098%
- resnet18, NAdam, cuda, (pt2) no_foreach: -99.99137%
- resnet18, NAdam, cuda, (pt2) foreach: -44.55896%
- timm_vision_transformer, Adadelta, cuda, (pt2) foreach: +851550.25140%
- timm_vision_transformer, Adam, cuda, (pt2) fused: -44.96414%
- timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: +652175.88232%
- timm_vision_transformer, AdamW, cuda, (pt2) foreach: +215.36091%
- timm_vision_transformer, AdamW, cuda, (pt2) fused: -40.51586%
- timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: -99.97419%
- timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -99.85308%
- timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9: -99.87554%
- timm_vision_transformer, NAdam, cuda, (pt2) foreach: -58.19527%


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, NAdam, cuda, (pt2) no_foreach: 0.06051368359476328


Tests that were newly added on affected commit:
- compile_time, timm_vision_transformer, Adadelta, cuda, (pt2) default: 38.935655916885786
- compile_time, timm_vision_transformer, Adadelta, cuda, (pt2) no_foreach: 56.782091966830194
- compile_time, timm_vision_transformer, Adadelta, cuda, (pt2) foreach: 0.07670339460795006
- compile_time, timm_vision_transformer, Adagrad, cuda, (pt2) default: 63.78063060514008
- compile_time, timm_vision_transformer, Adagrad, cuda, (pt2) no_foreach: 57.454121825440474
- compile_time, timm_vision_transformer, Adagrad, cuda, (pt2) foreach: 52.09552256638805
- compile_time, timm_vision_transformer, Adam, cuda, (pt2) default: 56.346197420420744
- compile_time, timm_vision_transformer, Adam, cuda, (pt2) no_foreach: 76.87888590913887
- compile_time, timm_vision_transformer, Adam, cuda, (pt2) foreach: 51.52186017033333
- compile_time, timm_vision_transformer, Adam, cuda, (pt2) fused: 2.089428629570951
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) default: 55.80541159802427
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: 1.086830297640219
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) foreach: 110.7428739098832
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) fused: 1.316812920384109
- compile_time, timm_vision_transformer, Adamax, cuda, (pt2) default: 6.516600772583236
- compile_time, timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: 69.85640457707147
- compile_time, timm_vision_transformer, Adamax, cuda, (pt2) foreach: 1.9435654579040904
- compile_time, timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: 86.99899413250387
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) default: 2.645103436584274
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) no_foreach: 7.289385402575135
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) foreach: 0.5947710676118731
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.6002681896400949
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9: 0.59690948544691
- compile_time, timm_vision_transformer, RAdam, cuda, (pt2) default: 0.014555967102448145
- compile_time, timm_vision_transformer, RAdam, cuda, (pt2) no_foreach: 0.0072455595557888355
- compile_time, timm_vision_transformer, RAdam, cuda, (pt2) foreach: 0.007961430431654056
- compile_time, timm_vision_transformer, Rprop, cuda, (pt2) default: 76.8649921513473
- compile_time, timm_vision_transformer, Rprop, cuda, (pt2) no_foreach: 55.76493847804765
- compile_time, timm_vision_transformer, Rprop, cuda, (pt2) foreach: 71.49089510831982
- compile_time, timm_vision_transformer, RMSprop, cuda, (pt2) default: 23.718703124206513
- compile_time, timm_vision_transformer, RMSprop, cuda, (pt2) no_foreach: 32.43360289248327
- compile_time, timm_vision_transformer, RMSprop, cuda, (pt2) foreach: 20.15155774463589
- compile_time, timm_vision_transformer, NAdam, cuda, (pt2) default: 9.143445813407501
- compile_time, timm_vision_transformer, NAdam, cuda, (pt2) foreach: 2.9753084161008396
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 129.92230025539175
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: -3.3969904125357857
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 116.40689636828999
- compile_time, timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 453.0773232639767
- compile_time, timm_vision_transformer_large, Adagrad, cuda, (pt2) no_foreach: 310.5190115387862
- compile_time, timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 411.78039422398433
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) default: 409.11566757845384
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: 313.3868534544793
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 385.27263944394264
- compile_time, timm_vision_transformer_large, Adam, cuda, (pt2) fused: 15.407216392612705
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) default: 404.11791469017044
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) no_foreach: 309.4473067660195
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 387.94118906774867
- compile_time, timm_vision_transformer_large, AdamW, cuda, (pt2) fused: 6.739764794707298
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) default: 45.460240966950856
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: 317.7203460706708
- compile_time, timm_vision_transformer_large, Adamax, cuda, (pt2) foreach: 10.888362704931447
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) default: 17.954011583855998
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) no_foreach: 26.771879871220637
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach: 1.9174904068931937
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 1.9344412783781688
- compile_time, timm_vision_transformer_large, SGD, cuda, (pt2) foreach, momentum=0.9: 11.002499868161976
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07164892243842283
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) no_foreach: 0.06669626540193954
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.06621869125713906
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) default: 605.7390600306293
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) no_foreach: 261.21320029410225
- compile_time, timm_vision_transformer_large, Rprop, cuda, (pt2) foreach: 570.0868673661413
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) default: 181.84867570471638
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) no_foreach: 118.45135036033268
- compile_time, timm_vision_transformer_large, RMSprop, cuda, (pt2) foreach: 170.5041208866363
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) default: 89.09015817552184
- compile_time, timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 22.070445171402145
- compile_time, resnet18, Adadelta, cuda, (pt2) default: 20.77019461632396
- compile_time, resnet18, Adadelta, cuda, (pt2) no_foreach: 5.168537940829992
- compile_time, resnet18, Adadelta, cuda, (pt2) foreach: 12.34020707073311
- compile_time, resnet18, Adagrad, cuda, (pt2) default: 26.98714356450364
- compile_time, resnet18, Adagrad, cuda, (pt2) no_foreach: 4.576479084479313
- compile_time, resnet18, Adagrad, cuda, (pt2) foreach: 15.31481483594204
- compile_time, resnet18, Adam, cuda, (pt2) default: 18.80687205803891
- compile_time, resnet18, Adam, cuda, (pt2) no_foreach: 33.398844711016864
- compile_time, resnet18, Adam, cuda, (pt2) foreach: 0.04696532323335667
- compile_time, resnet18, Adam, cuda, (pt2) fused: 0.7683865070963899
- compile_time, resnet18, AdamW, cuda, (pt2) default: 19.111395025470603
- compile_time, resnet18, AdamW, cuda, (pt2) no_foreach: 31.03081497270614
- compile_time, resnet18, AdamW, cuda, (pt2) foreach: 16.47397548550119
- compile_time, resnet18, AdamW, cuda, (pt2) fused: 0.8650590009056032
- compile_time, resnet18, Adamax, cuda, (pt2) default: 1.7345998779249687
- compile_time, resnet18, Adamax, cuda, (pt2) no_foreach: 34.3765250146389
- compile_time, resnet18, Adamax, cuda, (pt2) foreach: 0.6902369779224197
- compile_time, resnet18, ASGD, cuda, (pt2) no_foreach: 31.932984553432714
- compile_time, resnet18, SGD, cuda, (pt2) default: 0.8305236844656368
- compile_time, resnet18, SGD, cuda, (pt2) no_foreach: 4.393999088400354
- compile_time, resnet18, SGD, cuda, (pt2) foreach: 0.24974165173868337
- compile_time, resnet18, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.2763675549067557
- compile_time, resnet18, SGD, cuda, (pt2) foreach, momentum=0.9: 0.25348077015951276
- compile_time, resnet18, RAdam, cuda, (pt2) default: 0.010621811728924513
- compile_time, resnet18, RAdam, cuda, (pt2) no_foreach: 0.00427041482180357
- compile_time, resnet18, RAdam, cuda, (pt2) foreach: 0.004245384906729062
- compile_time, resnet18, Rprop, cuda, (pt2) default: 23.563274746139843
- compile_time, resnet18, Rprop, cuda, (pt2) no_foreach: 23.03950174121807
- compile_time, resnet18, Rprop, cuda, (pt2) foreach: -0.4652612420419864
- compile_time, resnet18, RMSprop, cuda, (pt2) default: 11.434194212934623
- compile_time, resnet18, RMSprop, cuda, (pt2) no_foreach: 14.445895934011787
- compile_time, resnet18, RMSprop, cuda, (pt2) foreach: 6.376413371724387
- compile_time, resnet18, NAdam, cuda, (pt2) default: 2.383625883143395
- compile_time, resnet18, NAdam, cuda, (pt2) no_foreach: 55.139883765019476
- compile_time, resnet18, NAdam, cuda, (pt2) foreach: 0.9896444046559433


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6117527173

cc @janeyx99


Compile time additions are expected