Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'a618118ba9ffc33097699c120e8760c0eb0f67fc'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 9ea5b31c6fd45cd3139540d3da74a2a248f644d9

Affected PyTorch commit: a618118ba9ffc33097699c120e8760c0eb0f67fc

Affected Tests:
- DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +31.61725%
- DALLE2_pytorch, AdamW, cuda, fused, capturable, amsgrad: -31.68515%
- timm_vision_transformer_large, Adamax, cuda, (pt2) no_foreach: -99.97725%
- basic_gnn_gcn, RAdam, cuda, differentiable: +33.26560%
- resnet18, NAdam, cuda, (pt2) no_foreach: +833754.67070%
- timm_vision_transformer, Adamax, cuda, (pt2) no_foreach: +402066.69445%
- timm_vision_transformer, RMSprop, cuda, differentiable: +45.38723%
- pytorch_CycleGAN_and_pix2pix, RMSprop, cuda, no_foreach: +40.33315%


Tests that were no longer run on affected commit:
- doctr_reco_predictor, Adadelta, cuda, default: 0.0015118557959794999
- doctr_reco_predictor, Adadelta, cuda, maximize: 0.0016793950460851192
- doctr_reco_predictor, Adadelta, cuda, no_foreach: 0.006579549969173968
- doctr_reco_predictor, Adadelta, cuda, differentiable: 0.008036546600051224
- doctr_reco_predictor, Adadelta, cuda, foreach: 0.0014900172012858094
- doctr_reco_predictor, Adagrad, cuda, default: 0.0014324660878628492
- doctr_reco_predictor, Adagrad, cuda, maximize: 0.0014121264428831638
- doctr_reco_predictor, Adagrad, cuda, no_foreach: 0.0036044308170676232
- doctr_reco_predictor, Adagrad, cuda, differentiable: 0.003974611819721759
- doctr_reco_predictor, Adagrad, cuda, foreach: 0.0013038757699541748
- doctr_reco_predictor, Adam, cuda, default: 0.0015397390886209904
- doctr_reco_predictor, Adam, cuda, amsgrad, maximize: 0.001755037831608206
- doctr_reco_predictor, Adam, cuda, no_foreach: 0.006290759420953691
- doctr_reco_predictor, Adam, cuda, differentiable: 0.012470017769373953
- doctr_reco_predictor, Adam, cuda, foreach: 0.001421032773796469
- doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable: 0.0023185645323246717
- doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.002398142269812524
- doctr_reco_predictor, Adam, cuda, fused: 0.0005276607391424477
- doctr_reco_predictor, Adam, cuda, fused, amsgrad, maximize: 0.0007339574391953647
- doctr_reco_predictor, Adam, cuda, fused, capturable: 0.0005269635589793324
- doctr_reco_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0007343357410281897
- doctr_reco_predictor, AdamW, cuda, default: 0.001506884088739753
- doctr_reco_predictor, AdamW, cuda, amsgrad, maximize: 0.0018163063935935496
- doctr_reco_predictor, AdamW, cuda, no_foreach: 0.006860061129555106
- doctr_reco_predictor, AdamW, cuda, differentiable: 0.012888150755316019
- doctr_reco_predictor, AdamW, cuda, foreach: 0.0015202851686626673
- doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable: 0.002350028129294515
- doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.002535825059749186
- doctr_reco_predictor, AdamW, cuda, fused: 0.0005438692159950733
- doctr_reco_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0007546893688850104
- doctr_reco_predictor, AdamW, cuda, fused, capturable: 0.0005430263862945139
- doctr_reco_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0007546159597113728
- doctr_reco_predictor, Adamax, cuda, default: 0.006315865139476955
- doctr_reco_predictor, Adamax, cuda, maximize: 0.006538099371828139
- doctr_reco_predictor, Adamax, cuda, no_foreach: 0.008441852969117463
- doctr_reco_predictor, Adamax, cuda, differentiable: 0.009772434380526345
- doctr_reco_predictor, Adamax, cuda, foreach: 0.006287246271967888
- doctr_reco_predictor, ASGD, cuda, default: 0.0027269958471879364
- doctr_reco_predictor, ASGD, cuda, maximize: 0.0029377152724191545
- doctr_reco_predictor, ASGD, cuda, no_foreach: 0.004635180947370827
- doctr_reco_predictor, ASGD, cuda, differentiable: 0.004681808357127011
- doctr_reco_predictor, ASGD, cuda, foreach: 0.002630722629837692
- doctr_reco_predictor, SGD, cuda, default: 0.0003205367140471935
- doctr_reco_predictor, SGD, cuda, maximize: 0.000551183800213039
- doctr_reco_predictor, SGD, cuda, no_foreach: 0.0006517176679335534
- doctr_reco_predictor, SGD, cuda, differentiable: 0.00031665874505415557
- doctr_reco_predictor, SGD, cuda, foreach: 0.0002438088357448578
- doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0006140765412710607
- doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9: 0.00044726467784494164
- doctr_reco_predictor, RAdam, cuda, default: 0.0016615080484189094
- doctr_reco_predictor, RAdam, cuda, no_foreach: 0.010071739857085048
- doctr_reco_predictor, RAdam, cuda, differentiable: 0.010977339162491263
- doctr_reco_predictor, RAdam, cuda, foreach: 0.0015853021782822906
- doctr_reco_predictor, Rprop, cuda, default: 0.008004421242512763
- doctr_reco_predictor, Rprop, cuda, maximize: 0.007978567372386655
- doctr_reco_predictor, Rprop, cuda, no_foreach: 0.012825244688428938
- doctr_reco_predictor, Rprop, cuda, differentiable: 0.013566349004395305
- doctr_reco_predictor, Rprop, cuda, foreach: 0.007963557836289208
- doctr_reco_predictor, RMSprop, cuda, default: 0.0007615568409673869
- doctr_reco_predictor, RMSprop, cuda, maximize: 0.0009844619315117597
- doctr_reco_predictor, RMSprop, cuda, no_foreach: 0.003333115568384528
- doctr_reco_predictor, RMSprop, cuda, differentiable: 0.003594477642327547
- doctr_reco_predictor, RMSprop, cuda, foreach: 0.0007056185239925981
- doctr_reco_predictor, NAdam, cuda, default: 0.0020884454296901823
- doctr_reco_predictor, NAdam, cuda, no_foreach: 0.007555278213694692
- doctr_reco_predictor, NAdam, cuda, differentiable: 0.013055888051167131
- doctr_reco_predictor, NAdam, cuda, foreach: 0.0019310746691189706
- doctr_det_predictor, Adadelta, cuda, default: 0.002604083609767258
- doctr_det_predictor, Adadelta, cuda, maximize: 0.003143137530423701
- doctr_det_predictor, Adadelta, cuda, no_foreach: 0.019317729375325145
- doctr_det_predictor, Adadelta, cuda, differentiable: 0.022277792217209935
- doctr_det_predictor, Adadelta, cuda, foreach: 0.00261587169021368
- doctr_det_predictor, Adagrad, cuda, default: 0.003555735726840794
- doctr_det_predictor, Adagrad, cuda, maximize: 0.0035965336533263326
- doctr_det_predictor, Adagrad, cuda, no_foreach: 0.01044131419621408
- doctr_det_predictor, Adagrad, cuda, differentiable: 0.011176057183183728
- doctr_det_predictor, Adagrad, cuda, foreach: 0.0034117329958826305
- doctr_det_predictor, Adam, cuda, default: 0.0036952577298507094
- doctr_det_predictor, Adam, cuda, amsgrad, maximize: 0.004453310049138963
- doctr_det_predictor, Adam, cuda, no_foreach: 0.018412414309568706
- doctr_det_predictor, Adam, cuda, differentiable: 0.03568160030990839
- doctr_det_predictor, Adam, cuda, foreach: 0.003561389222741127
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable: 0.0062941066082566974
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0065161372488364575
- doctr_det_predictor, Adam, cuda, fused: 0.0013070670887827874
- doctr_det_predictor, Adam, cuda, fused, amsgrad, maximize: 0.001383151861373335
- doctr_det_predictor, Adam, cuda, fused, capturable: 0.0013099863333627582
- doctr_det_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0014083455759100615
- doctr_det_predictor, AdamW, cuda, default: 0.0037779048085212707
- doctr_det_predictor, AdamW, cuda, amsgrad, maximize: 0.004486463847570121
- doctr_det_predictor, AdamW, cuda, no_foreach: 0.019737993413582445
- doctr_det_predictor, AdamW, cuda, differentiable: 0.03752181213349104
- doctr_det_predictor, AdamW, cuda, foreach: 0.003758206460624933
- doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable: 0.006353599322028458
- doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.006436423887498677
- doctr_det_predictor, AdamW, cuda, fused: 0.0013563009188510477
- doctr_det_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0014242091937921942
- doctr_det_predictor, AdamW, cuda, fused, capturable: 0.0013557416456751524
- doctr_det_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0014278318407014014
- doctr_det_predictor, Adamax, cuda, default: 0.019074663217179478
- doctr_det_predictor, Adamax, cuda, maximize: 0.018931925785727798
- doctr_det_predictor, Adamax, cuda, no_foreach: 0.025513230077922344
- doctr_det_predictor, Adamax, cuda, differentiable: 0.02838239590637386
- doctr_det_predictor, Adamax, cuda, foreach: 0.018977378960698843
- doctr_det_predictor, ASGD, cuda, default: 0.007440333859995008
- doctr_det_predictor, ASGD, cuda, maximize: 0.008223370811901987
- doctr_det_predictor, ASGD, cuda, no_foreach: 0.013728383393026889
- doctr_det_predictor, ASGD, cuda, differentiable: 0.01344390450976789
- doctr_det_predictor, ASGD, cuda, foreach: 0.0072589776292443274
- doctr_det_predictor, SGD, cuda, default: 0.0007966032102704048
- doctr_det_predictor, SGD, cuda, maximize: 0.0014476164802908896
- doctr_det_predictor, SGD, cuda, no_foreach: 0.0017710660537704826
- doctr_det_predictor, SGD, cuda, differentiable: 0.0007982702236622572
- doctr_det_predictor, SGD, cuda, foreach: 0.0006099045579321682
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0010049556009471415
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9: 0.0008399597378447651
- doctr_det_predictor, RAdam, cuda, default: 0.00431082772091031
- doctr_det_predictor, RAdam, cuda, no_foreach: 0.028773560421541333
- doctr_det_predictor, RAdam, cuda, differentiable: 0.029869057005271316
- doctr_det_predictor, RAdam, cuda, foreach: 0.0041476887837052345
- doctr_det_predictor, Rprop, cuda, default: 0.02272886587306857
- doctr_det_predictor, Rprop, cuda, maximize: 0.02262192629277706
- doctr_det_predictor, Rprop, cuda, no_foreach: 0.03796205781400204
- doctr_det_predictor, Rprop, cuda, differentiable: 0.039274227991700175
- doctr_det_predictor, Rprop, cuda, foreach: 0.022463215002790093
- doctr_det_predictor, RMSprop, cuda, default: 0.0018944092257879674
- doctr_det_predictor, RMSprop, cuda, maximize: 0.0024818194471299647
- doctr_det_predictor, RMSprop, cuda, no_foreach: 0.009498082458352049
- doctr_det_predictor, RMSprop, cuda, differentiable: 0.010186154046095908
- doctr_det_predictor, RMSprop, cuda, foreach: 0.001734984207432717
- doctr_det_predictor, NAdam, cuda, default: 0.005185517421923577
- doctr_det_predictor, NAdam, cuda, no_foreach: 0.02161580789834261
- doctr_det_predictor, NAdam, cuda, differentiable: 0.03766677281819284
- doctr_det_predictor, NAdam, cuda, foreach: 0.005006744931451976


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6105252415

cc @janeyx99


@xuzhao9 doctr_det_predictor seems broken again :C

@janeyx99  Yeah, I pushed another fix yesterday and hopefully it will fix today.

@janeyx99 I can confirm that the doctr_det_predictor problem has been fixed.

Talking offline about the random huge spikes with @xuzhao9 closing this to declutter and to centralize that discussion elsewhere