Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1a3d3669efa55e3360060c9b81f87900ae0c906c'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 9a8c6559561431401ed10d063c9d28a87aaa5db2

Affected PyTorch commit: 1a3d3669efa55e3360060c9b81f87900ae0c906c

Affected Tests:
- hf_GPT2_large, NAdam, cuda, default: -10.50156%
- densenet121, Adagrad, cuda, (pt2) maximize: +13.87319%
- densenet121, Adam, cuda, (pt2) amsgrad, maximize: -12.43193%
- resnet50_quantized_qat, Adadelta, cuda, (pt2) default: +10.31553%
- resnet50_quantized_qat, Adagrad, cuda, foreach: +14.13085%
- timm_regnet, AdamW, cuda, (pt2) foreach, maximize, capturable: -10.31644%
- timm_vovnet, Adadelta, cuda, (pt2) maximize: +14.21347%
- timm_vovnet, Adadelta, cuda, (pt2) foreach: +12.78635%
- timm_vovnet, Adam, cuda, (pt2) fused, capturable: +14.09552%
- timm_vovnet, ASGD, cuda, (pt2) foreach: -11.53746%
- detectron2_fasterrcnn_r_50_c4, ASGD, cuda, default: -16.53812%
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable: -18.29802%
- doctr_det_predictor, Adamax, cuda, maximize: -21.65031%
- doctr_det_predictor, ASGD, cuda, (pt2) maximize: -28.85095%
- doctr_det_predictor, Rprop, cuda, maximize: -17.08337%
- attention_is_all_you_need_pytorch, Adamax, cuda, default: -27.85157%
- attention_is_all_you_need_pytorch, Rprop, cuda, (pt2) foreach: -28.48658%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +58.45479%
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9: +21.68006%
- pytorch_unet, RAdam, cuda, (pt2) foreach: -16.78603%
- detectron2_maskrcnn_r_50_fpn, Adagrad, cuda, default: -13.22465%
- detectron2_maskrcnn_r_50_fpn, Adamax, cuda, default: -12.55920%
- detectron2_maskrcnn_r_50_fpn, ASGD, cuda, (pt2) maximize: -10.70029%
- detectron2_maskrcnn_r_50_fpn, SGD, cuda, (pt2) foreach, momentum=0.9: +296.99264%
- timm_resnest, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +17.18252%
- timm_resnest, SGD, cuda, (pt2) foreach, momentum=0.9: +47.91006%
- functorch_maml_omniglot, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: -12.67624%
- functorch_maml_omniglot, AdamW, cuda, (pt2) fused, capturable, amsgrad: +10.17781%
- functorch_maml_omniglot, SGD, cuda, (pt2) foreach: +26.63780%
- resnet152, Adagrad, cuda, default: -14.60746%
- resnet152, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -15.48302%
- resnet152, ASGD, cuda, (pt2) maximize: -10.40751%
- resnet152, ASGD, cuda, foreach: -13.39901%
- resnet152, NAdam, cuda, default: -10.70756%
- timm_efficientnet, Adadelta, cuda, (pt2) default: +12.07135%
- timm_efficientnet, Adadelta, cuda, (pt2) maximize: +11.63967%
- mobilenet_v3_large, Adagrad, cuda, (pt2) foreach: -31.89140%
- mobilenet_v3_large, Adamax, cuda, (pt2) maximize: -10.72782%
- vision_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: -25.26360%
- hf_T5_generate, Adadelta, cuda, (pt2) maximize: +10.15551%
- functorch_dp_cifar10, Adam, cuda, (pt2) default: +14.74851%
- functorch_dp_cifar10, ASGD, cuda, (pt2) foreach: -11.76424%
- functorch_dp_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -18.41101%
- functorch_dp_cifar10, RAdam, cuda, (pt2) foreach: -12.81600%
- functorch_dp_cifar10, Rprop, cuda, (pt2) default: +11.76647%
- phlippe_resnet, SGD, cuda, (pt2) foreach, momentum=0.9: -19.24300%
- tacotron2, Rprop, cuda, (pt2) foreach: +13.02938%
- yolov3, Adadelta, cuda, differentiable: -16.32569%
- yolov3, Adagrad, cuda, no_foreach: -15.42707%
- yolov3, ASGD, cuda, (pt2) differentiable: -11.84121%
- yolov3, SGD, cuda, (pt2) maximize: -10.15146%
- hf_Albert, AdamW, cuda, (pt2) foreach, maximize, capturable: -14.52128%
- opacus_cifar10, Adamax, cuda, (pt2) maximize: -12.24441%
- opacus_cifar10, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -10.46764%
- detectron2_fasterrcnn_r_50_dc5, Adagrad, cuda, (pt2) default: +14.79271%
- speech_transformer, Adadelta, cuda, (pt2) maximize: -10.31720%
- speech_transformer, Adagrad, cuda, (pt2) foreach: +11.05617%
- Super_SloMo, Adam, cuda, (pt2) foreach, maximize, capturable: -16.94853%
- Super_SloMo, AdamW, cuda, foreach, maximize, capturable: -11.30959%
- timm_vision_transformer, Adadelta, cuda, (pt2) maximize: +13.01481%
- timm_vision_transformer, Adadelta, cuda, (pt2) no_foreach: +10.06707%
- timm_vision_transformer, Rprop, cuda, no_foreach: -12.22104%
- mnasnet1_0, Adagrad, cuda, (pt2) maximize: +10.23310%
- mnasnet1_0, AdamW, cuda, (pt2) default: +11.22363%
- maml_omniglot, Adadelta, cuda, foreach: -15.00820%
- soft_actor_critic, Adadelta, cuda, (pt2) maximize: +16.98765%
- soft_actor_critic, Adam, cuda, default: +10.27636%
- pytorch_CycleGAN_and_pix2pix, Rprop, cuda, (pt2) foreach: -21.99082%
- mobilenet_v2_quantized_qat, Adagrad, cuda, (pt2) foreach: +13.06490%
- mobilenet_v2_quantized_qat, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: -14.52408%
- Background_Matting, Rprop, cuda, (pt2) foreach: -21.23967%
- detectron2_maskrcnn_r_101_fpn, Adagrad, cuda, (pt2) maximize: +10.72313%
- detectron2_maskrcnn_r_101_fpn, Rprop, cuda, (pt2) default: -16.79109%
- pytorch_stargan, Adagrad, cuda, (pt2) maximize: +13.27448%
- fastNLP_Bert, Rprop, cuda, (pt2) default: -15.40954%
- lennard_jones, Adagrad, cuda, (pt2) maximize: +12.55916%
- llama, Adadelta, cuda, (pt2) default: +73.65619%
- llama, Adadelta, cuda, default: +113.74034%
- llama, Adadelta, cuda, (pt2) maximize: +74.61675%
- llama, Adadelta, cuda, maximize: +113.27824%
- llama, Adadelta, cuda, (pt2) foreach: +75.30304%
- llama, Adadelta, cuda, foreach: +114.19001%
- llama, Adagrad, cuda, (pt2) default: +17.68345%
- llama, Adagrad, cuda, (pt2) maximize: +29.26268%
- llama, Adagrad, cuda, (pt2) foreach: +19.88164%
- llama, Adam, cuda, (pt2) default: +34.72430%
- llama, Adam, cuda, default: +22.11722%
- llama, Adam, cuda, (pt2) amsgrad, maximize: +43.78006%
- llama, Adam, cuda, amsgrad, maximize: +23.44803%
- llama, Adam, cuda, (pt2) foreach: +34.68003%
- llama, Adam, cuda, foreach: +21.16193%
- llama, Adam, cuda, (pt2) foreach, maximize, capturable: +32.71888%
- llama, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: +46.15145%
- llama, Adam, cuda, foreach, maximize, capturable, amsgrad: +10.49920%
- llama, Adam, cuda, (pt2) fused: +42.00424%
- llama, Adam, cuda, fused: +38.14060%
- llama, Adam, cuda, (pt2) fused, amsgrad, maximize: +42.10753%
- llama, Adam, cuda, fused, amsgrad, maximize: +68.32599%
- llama, Adam, cuda, (pt2) fused, capturable: +41.50641%
- llama, Adam, cuda, fused, capturable: +39.58757%
- llama, Adam, cuda, (pt2) fused, capturable, amsgrad: +48.35379%
- llama, Adam, cuda, fused, capturable, amsgrad: +62.50733%
- llama, AdamW, cuda, (pt2) default: +38.22173%
- llama, AdamW, cuda, default: +22.80842%
- llama, AdamW, cuda, (pt2) amsgrad, maximize: +38.43311%
- llama, AdamW, cuda, amsgrad, maximize: +23.06755%
- llama, AdamW, cuda, (pt2) foreach: +26.65525%
- llama, AdamW, cuda, foreach: +21.46907%
- llama, AdamW, cuda, (pt2) foreach, maximize, capturable: +31.45402%
- llama, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: +43.57822%
- llama, AdamW, cuda, foreach, maximize, capturable, amsgrad: +18.58118%
- llama, AdamW, cuda, (pt2) fused: +36.40687%
- llama, AdamW, cuda, fused: +48.37679%
- llama, AdamW, cuda, (pt2) fused, amsgrad, maximize: +43.96425%
- llama, AdamW, cuda, fused, amsgrad, maximize: +70.22028%
- llama, AdamW, cuda, (pt2) fused, capturable: +41.98799%
- llama, AdamW, cuda, fused, capturable: +48.76515%
- llama, AdamW, cuda, (pt2) fused, capturable, amsgrad: +36.23447%
- llama, AdamW, cuda, fused, capturable, amsgrad: +77.13192%
- llama, Adamax, cuda, (pt2) default: +10.95139%
- llama, Adamax, cuda, (pt2) foreach: +10.71929%
- llama, SGD, cuda, (pt2) default: +22.60288%
- llama, SGD, cuda, (pt2) maximize: +28.28276%
- llama, SGD, cuda, (pt2) foreach: +26.06063%
- llama, SGD, cuda, foreach, momentum=0.9, nesterov: +57.25724%
- llama, SGD, cuda, (pt2) foreach, momentum=0.9: +104.69350%
- llama, SGD, cuda, foreach, momentum=0.9: +20.81236%
- llama, RAdam, cuda, (pt2) default: +18.55974%
- llama, RAdam, cuda, default: +19.36237%
- llama, RAdam, cuda, (pt2) foreach: +24.01680%
- llama, RAdam, cuda, foreach: +23.26122%
- llama, RMSprop, cuda, (pt2) default: +53.70426%
- llama, RMSprop, cuda, default: +27.59370%
- llama, RMSprop, cuda, (pt2) maximize: +47.18620%
- llama, RMSprop, cuda, maximize: +36.20358%
- llama, RMSprop, cuda, (pt2) foreach: +47.60305%
- llama, RMSprop, cuda, foreach: +41.53371%
- llama, NAdam, cuda, (pt2) default: +23.90706%
- llama, NAdam, cuda, default: +25.48260%
- llama, NAdam, cuda, (pt2) foreach: +30.06701%
- llama, NAdam, cuda, foreach: +17.14021%
- pytorch_struct, Adam, cuda, (pt2) foreach: +11.62245%
- pytorch_struct, Adamax, cuda, (pt2) foreach: -12.40986%
- maml, Adam, cuda, (pt2) fused, amsgrad, maximize: +11.80986%
- shufflenet_v2_x1_0, Adadelta, cuda, (pt2) default: +10.67219%
- shufflenet_v2_x1_0, Adagrad, cuda, (pt2) default: +15.86355%
- shufflenet_v2_x1_0, Adagrad, cuda, (pt2) maximize: +10.81899%
- shufflenet_v2_x1_0, Adamax, cuda, (pt2) foreach: -16.69830%
- detectron2_fasterrcnn_r_101_fpn, ASGD, cuda, maximize: -24.19125%
- detectron2_fasterrcnn_r_101_fpn, Rprop, cuda, maximize: -12.36504%
- detectron2_fasterrcnn_r_101_c4, Adadelta, cuda, (pt2) default: +14.70079%
- detectron2_fasterrcnn_r_101_c4, Adagrad, cuda, (pt2) default: +16.21996%
- detectron2_fasterrcnn_r_101_c4, Adam, cuda, (pt2) fused, capturable, amsgrad: -10.21513%
- detectron2_fasterrcnn_r_101_c4, Rprop, cuda, (pt2) foreach: -10.99550%
- hf_Reformer, Adam, cuda, (pt2) foreach: +10.09424%
- hf_Reformer, AdamW, cuda, (pt2) foreach, maximize, capturable: +10.03689%
- demucs, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +14.63436%
- demucs, SGD, cuda, (pt2) foreach, momentum=0.9: +12.57132%
- drq, Adadelta, cuda, (pt2) foreach: +11.75935%
- drq, Adagrad, cuda, (pt2) maximize: +19.45440%
- drq, Adam, cuda, (pt2) foreach: +13.17421%
- drq, Adam, cuda, foreach, maximize, capturable: +12.24741%
- drq, AdamW, cuda, (pt2) fused, capturable: +10.09508%
- drq, ASGD, cuda, default: -14.85153%
- drq, RAdam, cuda, default: -11.18165%
- squeezenet1_1, Adadelta, cuda, (pt2) default: +12.67992%
- hf_GPT2, Adagrad, cuda, foreach: -14.23607%
- hf_GPT2, Rprop, cuda, (pt2) default: -14.04938%
- detectron2_maskrcnn_r_50_c4, Adagrad, cuda, (pt2) foreach: +18.11888%
- detectron2_maskrcnn_r_50_c4, AdamW, cuda, foreach, maximize, capturable: -14.44583%
- detectron2_maskrcnn, Adagrad, cuda, (pt2) maximize: -10.96835%
- detectron2_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: -50.21545%
- detectron2_maskrcnn, SGD, cuda, (pt2) foreach, momentum=0.9: -50.63553%
- LearningToPaint, Adadelta, cuda, (pt2) foreach: +12.73347%
- LearningToPaint, Adagrad, cuda, (pt2) foreach: -24.41349%
- LearningToPaint, Adam, cuda, (pt2) foreach: -11.30443%
- LearningToPaint, Adam, cuda, (pt2) foreach, maximize, capturable: -17.52428%
- LearningToPaint, AdamW, cuda, foreach, maximize, capturable: -10.04327%
- LearningToPaint, SGD, cuda, (pt2) default: +11.48838%
- LearningToPaint, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +14.08821%
- LearningToPaint, NAdam, cuda, foreach: -14.58526%
- timm_nfnet, Adadelta, cuda, (pt2) default: +13.51450%
- resnet50, Adadelta, cuda, (pt2) maximize: -10.68057%
- resnet50, Adagrad, cuda, foreach: -20.50655%
- resnet50, Adam, cuda, (pt2) differentiable: +12.62099%
- resnet50, RAdam, cuda, differentiable: -14.84169%
- phlippe_densenet, Adagrad, cuda, (pt2) maximize: +10.20832%
- phlippe_densenet, ASGD, cuda, (pt2) foreach: +12.44487%
- mobilenet_v2, Adamax, cuda, (pt2) foreach: -16.20184%
- dcgan, Adamax, cuda, (pt2) foreach: -20.38143%
- dcgan, Adamax, cuda, foreach: -11.16808%
- dcgan, ASGD, cuda, (pt2) foreach: -10.11913%
- dcgan, RMSprop, cuda, foreach: -13.39322%
- tts_angular, Adadelta, cuda, maximize: +10.67733%
- tts_angular, Adam, cuda, foreach, maximize, capturable, amsgrad: +12.12118%
- tts_angular, SGD, cuda, (pt2) foreach: -38.03241%
- timm_efficientdet, Adadelta, cuda, (pt2) default: +13.44273%
- timm_efficientdet, Adam, cuda, foreach: +11.87630%
- timm_efficientdet, Rprop, cuda, (pt2) default: -14.41727%
- DALLE2_pytorch, AdamW, cuda, (pt2) foreach, maximize, capturable: -19.13970%
- DALLE2_pytorch, Adamax, cuda, (pt2) default: -11.37056%
- DALLE2_pytorch, RAdam, cuda, (pt2) default: -20.49427%
- DALLE2_pytorch, RAdam, cuda, foreach: -18.53282%
- BERT_pytorch, Adagrad, cuda, (pt2) differentiable: -24.51961%
- BERT_pytorch, AdamW, cuda, (pt2) no_foreach: -12.31095%
- BERT_pytorch, ASGD, cuda, (pt2) maximize: -30.50961%
- BERT_pytorch, ASGD, cuda, no_foreach: -21.54547%
- BERT_pytorch, ASGD, cuda, differentiable: -24.65288%
- BERT_pytorch, RMSprop, cuda, differentiable: -32.18022%
- doctr_reco_predictor, Adagrad, cuda, (pt2) default: +10.50063%
- doctr_reco_predictor, Rprop, cuda, (pt2) maximize: -22.61721%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:
- basic_gnn_sage, Adadelta, cpu, (pt2) foreach: 0.0006405677149494807
- basic_gnn_sage, Adadelta, cpu, foreach: 0.0003752437517978251
- basic_gnn_sage, Adagrad, cpu, (pt2) foreach: 0.0009123375110158867
- basic_gnn_sage, Adagrad, cpu, foreach: 0.0006358757968991995
- basic_gnn_sage, Adam, cpu, (pt2) foreach: 0.000756185742551988
- basic_gnn_sage, Adam, cpu, foreach: 0.000521464663092047
- basic_gnn_sage, AdamW, cpu, (pt2) foreach: 0.0008047284814622627
- basic_gnn_sage, AdamW, cpu, foreach: 0.0005463612210005522
- basic_gnn_sage, Adamax, cpu, (pt2) foreach: 0.001033465351582956
- basic_gnn_sage, Adamax, cpu, foreach: 0.0008148383690665165
- basic_gnn_sage, ASGD, cpu, (pt2) foreach: 0.0006171550379636196
- basic_gnn_sage, ASGD, cpu, foreach: 0.0003769544898532331
- basic_gnn_sage, SGD, cpu, (pt2) foreach: 0.00017205479997210206
- basic_gnn_sage, SGD, cpu, foreach: 6.405975499656051e-05
- basic_gnn_sage, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: 0.0002999692640732974
- basic_gnn_sage, SGD, cpu, foreach, momentum=0.9, nesterov: 0.0001442699444014579
- basic_gnn_sage, SGD, cpu, (pt2) foreach, momentum=0.9: 0.0002814249179791659
- basic_gnn_sage, SGD, cpu, foreach, momentum=0.9: 0.00013289032399188728
- basic_gnn_sage, RAdam, cpu, (pt2) foreach: 0.0007455652331312497
- basic_gnn_sage, RAdam, cpu, foreach: 0.0005688970650080591
- basic_gnn_sage, Rprop, cpu, (pt2) foreach: 0.0012086587593363352
- basic_gnn_sage, Rprop, cpu, foreach: 0.0009530113676252465
- basic_gnn_sage, RMSprop, cpu, (pt2) foreach: 0.0004751562603225051
- basic_gnn_sage, RMSprop, cpu, foreach: 0.00022458632895722984
- basic_gnn_sage, NAdam, cpu, (pt2) foreach: 0.0008203341200046974
- basic_gnn_sage, NAdam, cpu, foreach: 0.0005572800550144166
- basic_gnn_edgecnn, Adadelta, cpu, (pt2) foreach: 0.0007689445033326916
- basic_gnn_edgecnn, Adadelta, cpu, foreach: 0.0004964201899711043
- basic_gnn_edgecnn, Adagrad, cpu, (pt2) foreach: 0.0010442224011057988
- basic_gnn_edgecnn, Adagrad, cpu, foreach: 0.0008164505641907454
- basic_gnn_edgecnn, Adam, cpu, (pt2) foreach: 0.0009536295408560406
- basic_gnn_edgecnn, Adam, cpu, foreach: 0.0006297392379492521
- basic_gnn_edgecnn, AdamW, cpu, (pt2) foreach: 0.0009320844809503056
- basic_gnn_edgecnn, AdamW, cpu, foreach: 0.0007230293760076166
- basic_gnn_edgecnn, Adamax, cpu, (pt2) foreach: 0.001292807805622297
- basic_gnn_edgecnn, Adamax, cpu, foreach: 0.001139334454201162
- basic_gnn_edgecnn, ASGD, cpu, (pt2) foreach: 0.0007351111226643508
- basic_gnn_edgecnn, ASGD, cpu, foreach: 0.0004812644300982356
- basic_gnn_edgecnn, SGD, cpu, (pt2) foreach: 0.00023993493054437338
- basic_gnn_edgecnn, SGD, cpu, foreach: 7.647400870919228e-05
- basic_gnn_edgecnn, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: 0.0004583585493743965
- basic_gnn_edgecnn, SGD, cpu, foreach, momentum=0.9, nesterov: 0.00017959653190337122
- basic_gnn_edgecnn, SGD, cpu, (pt2) foreach, momentum=0.9: 0.00043791735440867603
- basic_gnn_edgecnn, SGD, cpu, foreach, momentum=0.9: 0.00016361260355915875
- basic_gnn_edgecnn, RAdam, cpu, (pt2) foreach: 0.0008857908938080072
- basic_gnn_edgecnn, RAdam, cpu, foreach: 0.0007328716665506363
- basic_gnn_edgecnn, Rprop, cpu, (pt2) foreach: 0.0015201066730007756
- basic_gnn_edgecnn, Rprop, cpu, foreach: 0.0012593807897064834
- basic_gnn_edgecnn, RMSprop, cpu, (pt2) foreach: 0.0005301605009537919
- basic_gnn_edgecnn, RMSprop, cpu, foreach: 0.000292899691965431
- basic_gnn_edgecnn, NAdam, cpu, (pt2) foreach: 0.0009384241029097934
- basic_gnn_edgecnn, NAdam, cpu, foreach: 0.0007615521340630948
- basic_gnn_gcn, Adadelta, cpu, (pt2) foreach: 0.0004901931391149978
- basic_gnn_gcn, Adadelta, cpu, foreach: 0.00026518264482729137
- basic_gnn_gcn, Adagrad, cpu, (pt2) foreach: 0.0006005945699289441
- basic_gnn_gcn, Adagrad, cpu, foreach: 0.0004118209998123348
- basic_gnn_gcn, Adam, cpu, (pt2) foreach: 0.000519608560949564
- basic_gnn_gcn, Adam, cpu, foreach: 0.0003452850319445133
- basic_gnn_gcn, AdamW, cpu, (pt2) foreach: 0.000558210277929902
- basic_gnn_gcn, AdamW, cpu, foreach: 0.00037306814803741875
- basic_gnn_gcn, Adamax, cpu, (pt2) foreach: 0.0007276156260631978
- basic_gnn_gcn, Adamax, cpu, foreach: 0.0005502463199663907
- basic_gnn_gcn, ASGD, cpu, (pt2) foreach: 0.00044425931689329445
- basic_gnn_gcn, ASGD, cpu, foreach: 0.00025931458198465406
- basic_gnn_gcn, SGD, cpu, (pt2) foreach: 0.00013857380801346155
- basic_gnn_gcn, SGD, cpu, foreach: 5.050193581264466e-05
- basic_gnn_gcn, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: 0.00023992877709679307
- basic_gnn_gcn, SGD, cpu, foreach, momentum=0.9, nesterov: 0.00010338383005000651
- basic_gnn_gcn, SGD, cpu, (pt2) foreach, momentum=0.9: 0.0002231567029375583
- basic_gnn_gcn, SGD, cpu, foreach, momentum=0.9: 9.662940298827987e-05
- basic_gnn_gcn, RAdam, cpu, (pt2) foreach: 0.0005239814359229058
- basic_gnn_gcn, RAdam, cpu, foreach: 0.0003672780059278011
- basic_gnn_gcn, Rprop, cpu, (pt2) foreach: 0.0008472538826287673
- basic_gnn_gcn, Rprop, cpu, foreach: 0.0005998253650031984
- basic_gnn_gcn, RMSprop, cpu, (pt2) foreach: 0.00038344548592322516
- basic_gnn_gcn, RMSprop, cpu, foreach: 0.00016201575694140046
- basic_gnn_gcn, NAdam, cpu, (pt2) foreach: 0.0005982268615556297
- basic_gnn_gcn, NAdam, cpu, foreach: 0.0003772873841226101
- nanogpt_generate, Adadelta, cuda, (pt2) default: 0.011490172900569936
- nanogpt_generate, Adadelta, cuda, default: 0.010774604103062302
- nanogpt_generate, Adadelta, cuda, (pt2) maximize: 0.01255459371895995
- nanogpt_generate, Adadelta, cuda, maximize: 0.011568945541512221
- nanogpt_generate, Adadelta, cuda, (pt2) foreach: 0.01147904604052504
- nanogpt_generate, Adadelta, cuda, foreach: 0.010749321698676794
- nanogpt_generate, Adagrad, cuda, (pt2) default: 0.011512577236216102
- nanogpt_generate, Adagrad, cuda, default: 0.007925174928580721
- nanogpt_generate, Adagrad, cuda, (pt2) maximize: 0.012279153253663988
- nanogpt_generate, Adagrad, cuda, maximize: 0.008359225474608442
- nanogpt_generate, Adagrad, cuda, (pt2) foreach: 0.011419176938943565
- nanogpt_generate, Adagrad, cuda, foreach: 0.007637330607976765
- nanogpt_generate, Adam, cuda, (pt2) default: 0.0094683027445254
- nanogpt_generate, Adam, cuda, default: 0.007908693118952214
- nanogpt_generate, Adam, cuda, (pt2) amsgrad, maximize: 0.010965993456346424
- nanogpt_generate, Adam, cuda, amsgrad, maximize: 0.009936090698465705
- nanogpt_generate, Adam, cuda, (pt2) foreach: 0.009620260912925005
- nanogpt_generate, Adam, cuda, foreach: 0.008044244240348537
- nanogpt_generate, Adam, cuda, (pt2) foreach, maximize, capturable: 0.010992237918176934
- nanogpt_generate, Adam, cuda, foreach, maximize, capturable: 0.009916994293841222
- nanogpt_generate, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.012705400004051626
- nanogpt_generate, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.011077199736610055
- nanogpt_generate, Adam, cuda, (pt2) fused: 0.004437685939078422
- nanogpt_generate, Adam, cuda, fused: 0.003379375820513815
- nanogpt_generate, Adam, cuda, (pt2) fused, amsgrad, maximize: 0.004894393388318216
- nanogpt_generate, Adam, cuda, fused, amsgrad, maximize: 0.003912932921666652
- nanogpt_generate, Adam, cuda, (pt2) fused, capturable: 0.004469919530674815
- nanogpt_generate, Adam, cuda, fused, capturable: 0.003377664799336344
- nanogpt_generate, Adam, cuda, (pt2) fused, capturable, amsgrad: 0.0049051415095696365
- nanogpt_generate, Adam, cuda, fused, capturable, amsgrad: 0.003914057570509613
- nanogpt_generate, AdamW, cuda, (pt2) default: 0.010286583541892469
- nanogpt_generate, AdamW, cuda, default: 0.008858683495782315
- nanogpt_generate, AdamW, cuda, (pt2) amsgrad, maximize: 0.011792979559258503
- nanogpt_generate, AdamW, cuda, amsgrad, maximize: 0.010733511100988835
- nanogpt_generate, AdamW, cuda, (pt2) foreach: 0.010271595907397568
- nanogpt_generate, AdamW, cuda, foreach: 0.008858763597284753
- nanogpt_generate, AdamW, cuda, (pt2) foreach, maximize, capturable: 0.011796111679252456
- nanogpt_generate, AdamW, cuda, foreach, maximize, capturable: 0.010742128058336676
- nanogpt_generate, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.013000171806197613
- nanogpt_generate, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.011885485949460418
- nanogpt_generate, AdamW, cuda, (pt2) fused: 0.004811345316868808
- nanogpt_generate, AdamW, cuda, fused: 0.0034704431099817157
- nanogpt_generate, AdamW, cuda, (pt2) fused, amsgrad, maximize: 0.0049641095953652774
- nanogpt_generate, AdamW, cuda, fused, amsgrad, maximize: 0.003974462400656194
- nanogpt_generate, AdamW, cuda, (pt2) fused, capturable: 0.004392976533500073
- nanogpt_generate, AdamW, cuda, fused, capturable: 0.003474585099611431
- nanogpt_generate, AdamW, cuda, (pt2) fused, capturable, amsgrad: 0.004967122367106196
- nanogpt_generate, AdamW, cuda, fused, capturable, amsgrad: 0.003979186799842865
- nanogpt_generate, Adamax, cuda, (pt2) default: 0.013624586677178741
- nanogpt_generate, Adamax, cuda, default: 0.011761346145067363
- nanogpt_generate, Adamax, cuda, (pt2) maximize: 0.013827400902907053
- nanogpt_generate, Adamax, cuda, maximize: 0.012539678649045528
- nanogpt_generate, Adamax, cuda, (pt2) foreach: 0.013089707819744945
- nanogpt_generate, Adamax, cuda, foreach: 0.011752180708572268
- nanogpt_generate, ASGD, cuda, (pt2) default: 0.005203338996626628
- nanogpt_generate, ASGD, cuda, default: 0.004888158829417079
- nanogpt_generate, ASGD, cuda, (pt2) maximize: 0.005681993426858551
- nanogpt_generate, ASGD, cuda, maximize: 0.005359220658428967
- nanogpt_generate, ASGD, cuda, (pt2) foreach: 0.005249104199883265
- nanogpt_generate, ASGD, cuda, foreach: 0.004940203810110688
- nanogpt_generate, SGD, cuda, (pt2) default: 0.0019096303043798322
- nanogpt_generate, SGD, cuda, default: 0.0011696557700634003
- nanogpt_generate, SGD, cuda, (pt2) maximize: 0.002846917357217048
- nanogpt_generate, SGD, cuda, maximize: 0.00196280367905274
- nanogpt_generate, SGD, cuda, (pt2) foreach: 0.001873601963482449
- nanogpt_generate, SGD, cuda, foreach: 0.0011609782604500652
- nanogpt_generate, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.30477244895882905
- nanogpt_generate, SGD, cuda, foreach, momentum=0.9, nesterov: 0.004311978730838746
- nanogpt_generate, SGD, cuda, (pt2) foreach, momentum=0.9: 0.3035622760653496
- nanogpt_generate, SGD, cuda, foreach, momentum=0.9: 0.0031300386716611684
- nanogpt_generate, RAdam, cuda, (pt2) default: 0.01027563309762627
- nanogpt_generate, RAdam, cuda, default: 0.010228819295298307
- nanogpt_generate, RAdam, cuda, (pt2) foreach: 0.010226836148649453
- nanogpt_generate, RAdam, cuda, foreach: 0.01023237155750394
- nanogpt_generate, Rprop, cuda, (pt2) default: 0.021461287792772055
- nanogpt_generate, Rprop, cuda, default: 0.019715141050983223
- nanogpt_generate, Rprop, cuda, (pt2) maximize: 0.021737602329812943
- nanogpt_generate, Rprop, cuda, maximize: 0.020675799902528526
- nanogpt_generate, Rprop, cuda, (pt2) foreach: 0.021451122895814478
- nanogpt_generate, Rprop, cuda, foreach: 0.019717842200770976
- nanogpt_generate, RMSprop, cuda, (pt2) default: 0.005996557431952918
- nanogpt_generate, RMSprop, cuda, default: 0.0051173225790262225
- nanogpt_generate, RMSprop, cuda, (pt2) maximize: 0.0068679005606099965
- nanogpt_generate, RMSprop, cuda, maximize: 0.005903875110670924
- nanogpt_generate, RMSprop, cuda, (pt2) foreach: 0.005970124860622865
- nanogpt_generate, RMSprop, cuda, foreach: 0.00511982954107225
- nanogpt_generate, NAdam, cuda, (pt2) default: 0.012071556671906044
- nanogpt_generate, NAdam, cuda, default: 0.009642772500713666
- nanogpt_generate, NAdam, cuda, (pt2) foreach: 0.01173683497796042
- nanogpt_generate, NAdam, cuda, foreach: 0.009643426033047338
- basic_gnn_sage, Adadelta, cuda, (pt2) default: 0.0004248266640235917
- basic_gnn_sage, Adadelta, cuda, default: 0.00019312072149477898
- basic_gnn_sage, Adadelta, cuda, (pt2) maximize: 0.0004552814109377902
- basic_gnn_sage, Adadelta, cuda, maximize: 0.00022578263096511363
- basic_gnn_sage, Adadelta, cuda, (pt2) foreach: 0.0004179532856472412
- basic_gnn_sage, Adadelta, cuda, foreach: 0.00019274666788987816
- basic_gnn_sage, Adagrad, cuda, (pt2) default: 0.0008199238094317987
- basic_gnn_sage, Adagrad, cuda, default: 0.0005924832348246128
- basic_gnn_sage, Adagrad, cuda, (pt2) maximize: 0.0008435237589253097
- basic_gnn_sage, Adagrad, cuda, maximize: 0.0006515523889102042
- basic_gnn_sage, Adagrad, cuda, (pt2) foreach: 0.0008148577289534657
- basic_gnn_sage, Adagrad, cuda, foreach: 0.0006173222200013697
- basic_gnn_sage, Adam, cuda, (pt2) default: 0.0005295937338834087
- basic_gnn_sage, Adam, cuda, default: 0.0003325471279677004
- basic_gnn_sage, Adam, cuda, (pt2) amsgrad, maximize: 0.0005808780921380157
- basic_gnn_sage, Adam, cuda, amsgrad, maximize: 0.0003925335039384663
- basic_gnn_sage, Adam, cuda, (pt2) foreach: 0.0005197220373076278
- basic_gnn_sage, Adam, cuda, foreach: 0.0003044100140687078
- basic_gnn_sage, Adam, cuda, (pt2) foreach, maximize, capturable: 0.0008432657884418213
- basic_gnn_sage, Adam, cuda, foreach, maximize, capturable: 0.0006278026709333062
- basic_gnn_sage, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0008443177555872418
- basic_gnn_sage, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.000615163791924715
- basic_gnn_sage, Adam, cuda, (pt2) fused: 0.0002557187068179764
- basic_gnn_sage, Adam, cuda, fused: 8.189124761459729e-05
- basic_gnn_sage, Adam, cuda, (pt2) fused, amsgrad, maximize: 0.000266517093963177
- basic_gnn_sage, Adam, cuda, fused, amsgrad, maximize: 8.49974340138336e-05
- basic_gnn_sage, Adam, cuda, (pt2) fused, capturable: 0.0002555217701551596
- basic_gnn_sage, Adam, cuda, fused, capturable: 8.443138763929407e-05
- basic_gnn_sage, Adam, cuda, (pt2) fused, capturable, amsgrad: 0.0002645625521566857
- basic_gnn_sage, Adam, cuda, fused, capturable, amsgrad: 8.518681574302415e-05
- basic_gnn_sage, AdamW, cuda, (pt2) default: 0.0005410801562656825
- basic_gnn_sage, AdamW, cuda, default: 0.0003228365469258279
- basic_gnn_sage, AdamW, cuda, (pt2) amsgrad, maximize: 0.0005935662935932861
- basic_gnn_sage, AdamW, cuda, amsgrad, maximize: 0.0003862810127902776
- basic_gnn_sage, AdamW, cuda, (pt2) foreach: 0.000532178499889461
- basic_gnn_sage, AdamW, cuda, foreach: 0.0003280962719582021
- basic_gnn_sage, AdamW, cuda, (pt2) foreach, maximize, capturable: 0.0008465582190498018
- basic_gnn_sage, AdamW, cuda, foreach, maximize, capturable: 0.0006155027959030122
- basic_gnn_sage, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0009226158909362307
- basic_gnn_sage, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.0006395201277919114
- basic_gnn_sage, AdamW, cuda, (pt2) fused: 0.0002559806971127153
- basic_gnn_sage, AdamW, cuda, fused: 8.156115794554353e-05
- basic_gnn_sage, AdamW, cuda, (pt2) fused, amsgrad, maximize: 0.0002684945028100238
- basic_gnn_sage, AdamW, cuda, fused, amsgrad, maximize: 8.967911133853097e-05
- basic_gnn_sage, AdamW, cuda, (pt2) fused, capturable: 0.0002610624990653067
- basic_gnn_sage, AdamW, cuda, fused, capturable: 8.064033168678484e-05
- basic_gnn_sage, AdamW, cuda, (pt2) fused, capturable, amsgrad: 0.00026446931547629237
- basic_gnn_sage, AdamW, cuda, fused, capturable, amsgrad: 8.392042301905653e-05
- basic_gnn_sage, Adamax, cuda, (pt2) default: 0.001099161137212682
- basic_gnn_sage, Adamax, cuda, default: 0.0008429965702816843
- basic_gnn_sage, Adamax, cuda, (pt2) maximize: 0.0011033436007142723
- basic_gnn_sage, Adamax, cuda, maximize: 0.0009154769623031219
- basic_gnn_sage, Adamax, cuda, (pt2) foreach: 0.0010512484876312198
- basic_gnn_sage, Adamax, cuda, foreach: 0.0008304329837361971
- basic_gnn_sage, ASGD, cuda, (pt2) default: 0.0005921681877225637
- basic_gnn_sage, ASGD, cuda, default: 0.00038641505711711944
- basic_gnn_sage, ASGD, cuda, (pt2) maximize: 0.0006519657034875903
- basic_gnn_sage, ASGD, cuda, maximize: 0.00042888417304493486
- basic_gnn_sage, ASGD, cuda, (pt2) foreach: 0.0005976206810672336
- basic_gnn_sage, ASGD, cuda, foreach: 0.00037676206906326113
- basic_gnn_sage, SGD, cuda, (pt2) default: 0.00020077323948323666
- basic_gnn_sage, SGD, cuda, default: 6.266338131390513e-05
- basic_gnn_sage, SGD, cuda, (pt2) maximize: 0.0002588208636478611
- basic_gnn_sage, SGD, cuda, maximize: 9.723971302931507e-05
- basic_gnn_sage, SGD, cuda, (pt2) foreach: 0.00015150163718499244
- basic_gnn_sage, SGD, cuda, foreach: 5.869374671019614e-05
- basic_gnn_sage, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.0002731212019656072
- basic_gnn_sage, SGD, cuda, foreach, momentum=0.9, nesterov: 9.026631790523727e-05
- basic_gnn_sage, SGD, cuda, (pt2) foreach, momentum=0.9: 0.00018656785401981326
- basic_gnn_sage, SGD, cuda, foreach, momentum=0.9: 8.026367907101909e-05
- basic_gnn_sage, RAdam, cuda, (pt2) default: 0.000568526532035321
- basic_gnn_sage, RAdam, cuda, default: 0.0004121369880158454
- basic_gnn_sage, RAdam, cuda, (pt2) foreach: 0.0005755315970163793
- basic_gnn_sage, RAdam, cuda, foreach: 0.00041482376214116814
- basic_gnn_sage, Rprop, cuda, (pt2) default: 0.0015962549137868106
- basic_gnn_sage, Rprop, cuda, default: 0.001336352169746533
- basic_gnn_sage, Rprop, cuda, (pt2) maximize: 0.001616930472664535
- basic_gnn_sage, Rprop, cuda, maximize: 0.0013654637592844666
- basic_gnn_sage, Rprop, cuda, (pt2) foreach: 0.0016207212578475236
- basic_gnn_sage, Rprop, cuda, foreach: 0.001323518045246601
- basic_gnn_sage, RMSprop, cuda, (pt2) default: 0.0003457582058684196
- basic_gnn_sage, RMSprop, cuda, default: 0.00012782261439133434
- basic_gnn_sage, RMSprop, cuda, (pt2) maximize: 0.00037684439893692677
- basic_gnn_sage, RMSprop, cuda, maximize: 0.00015621774794999512
- basic_gnn_sage, RMSprop, cuda, (pt2) foreach: 0.00033322990695309026
- basic_gnn_sage, RMSprop, cuda, foreach: 0.00012442408804781734
- basic_gnn_sage, NAdam, cuda, (pt2) default: 0.0005884458098615355
- basic_gnn_sage, NAdam, cuda, default: 0.00036561906803399323
- basic_gnn_sage, NAdam, cuda, (pt2) foreach: 0.0005831106806190212
- basic_gnn_sage, NAdam, cuda, foreach: 0.0003625699421390891
- basic_gnn_gin, Adadelta, cuda, (pt2) default: 0.00045161446005644165
- basic_gnn_gin, Adadelta, cuda, default: 0.00023854407807812094
- basic_gnn_gin, Adadelta, cuda, (pt2) maximize: 0.0005035240088481849
- basic_gnn_gin, Adadelta, cuda, maximize: 0.00027307411190122363
- basic_gnn_gin, Adadelta, cuda, (pt2) foreach: 0.00044406131539749863
- basic_gnn_gin, Adadelta, cuda, foreach: 0.00023175900289788842
- basic_gnn_gin, Adagrad, cuda, (pt2) default: 0.0009656820945495453
- basic_gnn_gin, Adagrad, cuda, default: 0.0007575211068615317
- basic_gnn_gin, Adagrad, cuda, (pt2) maximize: 0.0010155919126145125
- basic_gnn_gin, Adagrad, cuda, maximize: 0.0007885242098321517
- basic_gnn_gin, Adagrad, cuda, (pt2) foreach: 0.00096860411239491
- basic_gnn_gin, Adagrad, cuda, foreach: 0.0007354472901206463
- basic_gnn_gin, Adam, cuda, (pt2) default: 0.0006329938069374973
- basic_gnn_gin, Adam, cuda, default: 0.00036267372383736075
- basic_gnn_gin, Adam, cuda, (pt2) amsgrad, maximize: 0.0006561092383487792
- basic_gnn_gin, Adam, cuda, amsgrad, maximize: 0.0004412880539894104
- basic_gnn_gin, Adam, cuda, (pt2) foreach: 0.0005902191349318949
- basic_gnn_gin, Adam, cuda, foreach: 0.00036308386991731824
- basic_gnn_gin, Adam, cuda, (pt2) foreach, maximize, capturable: 0.0009545717033601943
- basic_gnn_gin, Adam, cuda, foreach, maximize, capturable: 0.0007179512151051313
- basic_gnn_gin, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.000963054241415543
- basic_gnn_gin, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.000728410149924457
- basic_gnn_gin, Adam, cuda, (pt2) fused: 0.00027406756887339974
- basic_gnn_gin, Adam, cuda, fused: 9.423301035227876e-05
- basic_gnn_gin, Adam, cuda, (pt2) fused, amsgrad, maximize: 0.00028614863787960256
- basic_gnn_gin, Adam, cuda, fused, amsgrad, maximize: 9.919167829987904e-05
- basic_gnn_gin, Adam, cuda, (pt2) fused, capturable: 0.00027897867771590356
- basic_gnn_gin, Adam, cuda, fused, capturable: 9.221142367459832e-05
- basic_gnn_gin, Adam, cuda, (pt2) fused, capturable, amsgrad: 0.0002856307979221436
- basic_gnn_gin, Adam, cuda, fused, capturable, amsgrad: 9.649128296102086e-05
- basic_gnn_gin, AdamW, cuda, (pt2) default: 0.0006346906759316408
- basic_gnn_gin, AdamW, cuda, default: 0.0003690334320999682
- basic_gnn_gin, AdamW, cuda, (pt2) amsgrad, maximize: 0.0006477358738943018
- basic_gnn_gin, AdamW, cuda, amsgrad, maximize: 0.00043758043297566473
- basic_gnn_gin, AdamW, cuda, (pt2) foreach: 0.0005937735508652045
- basic_gnn_gin, AdamW, cuda, foreach: 0.0003671652160119265
- basic_gnn_gin, AdamW, cuda, (pt2) foreach, maximize, capturable: 0.0009409238118678331
- basic_gnn_gin, AdamW, cuda, foreach, maximize, capturable: 0.0007265664599835872
- basic_gnn_gin, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0009482412166469753
- basic_gnn_gin, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.0007393635450862348
- basic_gnn_gin, AdamW, cuda, (pt2) fused: 0.0002807305683610441
- basic_gnn_gin, AdamW, cuda, fused: 9.196478400068978e-05
- basic_gnn_gin, AdamW, cuda, (pt2) fused, amsgrad, maximize: 0.00028788732471976346
- basic_gnn_gin, AdamW, cuda, fused, amsgrad, maximize: 9.663684541980425e-05
- basic_gnn_gin, AdamW, cuda, (pt2) fused, capturable: 0.0002781044853489018
- basic_gnn_gin, AdamW, cuda, fused, capturable: 9.168586134910584e-05
- basic_gnn_gin, AdamW, cuda, (pt2) fused, capturable, amsgrad: 0.0003063505245187244
- basic_gnn_gin, AdamW, cuda, fused, capturable, amsgrad: 9.62655427865684e-05
- basic_gnn_gin, Adamax, cuda, (pt2) default: 0.0013732683321792786
- basic_gnn_gin, Adamax, cuda, default: 0.001054862515302375
- basic_gnn_gin, Adamax, cuda, (pt2) maximize: 0.0013336599944159388
- basic_gnn_gin, Adamax, cuda, maximize: 0.0011109136859886349
- basic_gnn_gin, Adamax, cuda, (pt2) foreach: 0.0012866247755785782
- basic_gnn_gin, Adamax, cuda, foreach: 0.00104439678019844
- basic_gnn_gin, ASGD, cuda, (pt2) default: 0.000713687011428522
- basic_gnn_gin, ASGD, cuda, default: 0.0004996763069648295
- basic_gnn_gin, ASGD, cuda, (pt2) maximize: 0.0007699576394560819
- basic_gnn_gin, ASGD, cuda, maximize: 0.0005359130199067294
- basic_gnn_gin, ASGD, cuda, (pt2) foreach: 0.0006910752665636868
- basic_gnn_gin, ASGD, cuda, foreach: 0.0004792072249110788
- basic_gnn_gin, SGD, cuda, (pt2) default: 0.00020746258570006782
- basic_gnn_gin, SGD, cuda, default: 7.270552909467369e-05
- basic_gnn_gin, SGD, cuda, (pt2) maximize: 0.0002851963910191954
- basic_gnn_gin, SGD, cuda, maximize: 0.00011159335146658124
- basic_gnn_gin, SGD, cuda, (pt2) foreach: 0.00016608082549646495
- basic_gnn_gin, SGD, cuda, foreach: 6.863743879366666e-05
- basic_gnn_gin, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.00029429033576675197
- basic_gnn_gin, SGD, cuda, foreach, momentum=0.9, nesterov: 0.00010059461893979459
- basic_gnn_gin, SGD, cuda, (pt2) foreach, momentum=0.9: 0.00030264014211932755
- basic_gnn_gin, SGD, cuda, foreach, momentum=0.9: 9.001648332923651e-05
- basic_gnn_gin, RAdam, cuda, (pt2) default: 0.000662025913828984
- basic_gnn_gin, RAdam, cuda, default: 0.0004950828810688108
- basic_gnn_gin, RAdam, cuda, (pt2) foreach: 0.000649745955131948
- basic_gnn_gin, RAdam, cuda, foreach: 0.0004837590390816331
- basic_gnn_gin, Rprop, cuda, (pt2) default: 0.0019894127346722795
- basic_gnn_gin, Rprop, cuda, default: 0.0017283637239597738
- basic_gnn_gin, Rprop, cuda, (pt2) maximize: 0.002090397412151409
- basic_gnn_gin, Rprop, cuda, maximize: 0.0017566292209085076
- basic_gnn_gin, Rprop, cuda, (pt2) foreach: 0.0020535512546989688
- basic_gnn_gin, Rprop, cuda, foreach: 0.0017317258450202644
- basic_gnn_gin, RMSprop, cuda, (pt2) default: 0.00039110460375013645
- basic_gnn_gin, RMSprop, cuda, default: 0.0001488857704680413
- basic_gnn_gin, RMSprop, cuda, (pt2) maximize: 0.0004152318885882242
- basic_gnn_gin, RMSprop, cuda, maximize: 0.00018451213894877584
- basic_gnn_gin, RMSprop, cuda, (pt2) foreach: 0.0003611391102592545
- basic_gnn_gin, RMSprop, cuda, foreach: 0.00014464504655916244
- basic_gnn_gin, NAdam, cuda, (pt2) default: 0.0006567169477033322
- basic_gnn_gin, NAdam, cuda, default: 0.0004371419730596244
- basic_gnn_gin, NAdam, cuda, (pt2) foreach: 0.0006677614866445462
- basic_gnn_gin, NAdam, cuda, foreach: 0.0004466084709856659
- basic_gnn_gin, Adadelta, cpu, (pt2) foreach: 0.0007210781440983037
- basic_gnn_gin, Adadelta, cpu, foreach: 0.0004568669460713863
- basic_gnn_gin, Adagrad, cpu, (pt2) foreach: 0.001102341646021539
- basic_gnn_gin, Adagrad, cpu, foreach: 0.0007788275929633528
- basic_gnn_gin, Adam, cpu, (pt2) foreach: 0.0009246378190695278
- basic_gnn_gin, Adam, cpu, foreach: 0.0006079896569717675
- basic_gnn_gin, AdamW, cpu, (pt2) foreach: 0.000890014497563243
- basic_gnn_gin, AdamW, cpu, foreach: 0.000665024349000305
- basic_gnn_gin, Adamax, cpu, (pt2) foreach: 0.0012108507869114358
- basic_gnn_gin, Adamax, cpu, foreach: 0.0010187703471941252
- basic_gnn_gin, ASGD, cpu, (pt2) foreach: 0.0007008633102531304
- basic_gnn_gin, ASGD, cpu, foreach: 0.0004651579789351672
- basic_gnn_gin, SGD, cpu, (pt2) foreach: 0.00018660680809989572
- basic_gnn_gin, SGD, cpu, foreach: 7.415533810853958e-05
- basic_gnn_gin, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: 0.00048020182007686983
- basic_gnn_gin, SGD, cpu, foreach, momentum=0.9, nesterov: 0.00016873398644383998
- basic_gnn_gin, SGD, cpu, (pt2) foreach, momentum=0.9: 0.0004124773181398812
- basic_gnn_gin, SGD, cpu, foreach, momentum=0.9: 0.00015398918965365738
- basic_gnn_gin, RAdam, cpu, (pt2) foreach: 0.0008515106164850295
- basic_gnn_gin, RAdam, cpu, foreach: 0.0006765846610069275
- basic_gnn_gin, Rprop, cpu, (pt2) foreach: 0.0013710563932226537
- basic_gnn_gin, Rprop, cpu, foreach: 0.0011819268809631466
- basic_gnn_gin, RMSprop, cpu, (pt2) foreach: 0.0005172427114368795
- basic_gnn_gin, RMSprop, cpu, foreach: 0.00027851082594133913
- basic_gnn_gin, NAdam, cpu, (pt2) foreach: 0.0009145528645803259
- basic_gnn_gin, NAdam, cpu, foreach: 0.0006631825750228017
- basic_gnn_edgecnn, Adadelta, cuda, (pt2) default: 0.000460367433020267
- basic_gnn_edgecnn, Adadelta, cuda, default: 0.0002207255740649998
- basic_gnn_edgecnn, Adadelta, cuda, (pt2) maximize: 0.000503683226983339
- basic_gnn_edgecnn, Adadelta, cuda, maximize: 0.00027299932204186914
- basic_gnn_edgecnn, Adadelta, cuda, (pt2) foreach: 0.00044682888203949133
- basic_gnn_edgecnn, Adadelta, cuda, foreach: 0.00021800129697658121
- basic_gnn_edgecnn, Adagrad, cuda, (pt2) default: 0.000968712007464922
- basic_gnn_edgecnn, Adagrad, cuda, default: 0.0007385590968187898
- basic_gnn_edgecnn, Adagrad, cuda, (pt2) maximize: 0.001010665399577431
- basic_gnn_edgecnn, Adagrad, cuda, maximize: 0.0008111238370183856
- basic_gnn_edgecnn, Adagrad, cuda, (pt2) foreach: 0.0009679064831745941
- basic_gnn_edgecnn, Adagrad, cuda, foreach: 0.0007488631340675056
- basic_gnn_edgecnn, Adam, cuda, (pt2) default: 0.0005780878406814757
- basic_gnn_edgecnn, Adam, cuda, default: 0.00036643702792935073
- basic_gnn_edgecnn, Adam, cuda, (pt2) amsgrad, maximize: 0.0006471073570390863
- basic_gnn_edgecnn, Adam, cuda, amsgrad, maximize: 0.0004256141330115497
- basic_gnn_edgecnn, Adam, cuda, (pt2) foreach: 0.0005911157684878701
- basic_gnn_edgecnn, Adam, cuda, foreach: 0.0003597761539276689
- basic_gnn_edgecnn, Adam, cuda, (pt2) foreach, maximize, capturable: 0.000938761946701697
- basic_gnn_edgecnn, Adam, cuda, foreach, maximize, capturable: 0.0007217767499387264
- basic_gnn_edgecnn, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0009727744833177444
- basic_gnn_edgecnn, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0007397137496930857
- basic_gnn_edgecnn, Adam, cuda, (pt2) fused: 0.00028149364319657473
- basic_gnn_edgecnn, Adam, cuda, fused: 9.423033148050308e-05
- basic_gnn_edgecnn, Adam, cuda, (pt2) fused, amsgrad, maximize: 0.0002894244830371715
- basic_gnn_edgecnn, Adam, cuda, fused, amsgrad, maximize: 9.669006623638173e-05
- basic_gnn_edgecnn, Adam, cuda, (pt2) fused, capturable: 0.0002799074771208363
- basic_gnn_edgecnn, Adam, cuda, fused, capturable: 9.274170361459256e-05
- basic_gnn_edgecnn, Adam, cuda, (pt2) fused, capturable, amsgrad: 0.00031964569025265334
- basic_gnn_edgecnn, Adam, cuda, fused, capturable, amsgrad: 9.678524096185963e-05
- basic_gnn_edgecnn, AdamW, cuda, (pt2) default: 0.0006547915394665725
- basic_gnn_edgecnn, AdamW, cuda, default: 0.0003728845079895109
- basic_gnn_edgecnn, AdamW, cuda, (pt2) amsgrad, maximize: 0.000663208964879051
- basic_gnn_edgecnn, AdamW, cuda, amsgrad, maximize: 0.00044485308486036954
- basic_gnn_edgecnn, AdamW, cuda, (pt2) foreach: 0.0005944758544479793
- basic_gnn_edgecnn, AdamW, cuda, foreach: 0.00036614235397428274
- basic_gnn_edgecnn, AdamW, cuda, (pt2) foreach, maximize, capturable: 0.0009646832141487931
- basic_gnn_edgecnn, AdamW, cuda, foreach, maximize, capturable: 0.0007340433029457927
- basic_gnn_edgecnn, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0009643660383657194
- basic_gnn_edgecnn, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.0007422984910663217
- basic_gnn_edgecnn, AdamW, cuda, (pt2) fused: 0.00028645607492272263
- basic_gnn_edgecnn, AdamW, cuda, fused: 9.518004472677906e-05
- basic_gnn_edgecnn, AdamW, cuda, (pt2) fused, amsgrad, maximize: 0.000289893795169242
- basic_gnn_edgecnn, AdamW, cuda, fused, amsgrad, maximize: 9.590941543380418e-05
- basic_gnn_edgecnn, AdamW, cuda, (pt2) fused, capturable: 0.00028267957725283286
- basic_gnn_edgecnn, AdamW, cuda, fused, capturable: 9.2432898003608e-05
- basic_gnn_edgecnn, AdamW, cuda, (pt2) fused, capturable, amsgrad: 0.000307511716734864
- basic_gnn_edgecnn, AdamW, cuda, fused, capturable, amsgrad: 9.69659686864664e-05
- basic_gnn_edgecnn, Adamax, cuda, (pt2) default: 0.0013569401769043022
- basic_gnn_edgecnn, Adamax, cuda, default: 0.0010735747101716696
- basic_gnn_edgecnn, Adamax, cuda, (pt2) maximize: 0.001330030695010988
- basic_gnn_edgecnn, Adamax, cuda, maximize: 0.0010976667038630696
- basic_gnn_edgecnn, Adamax, cuda, (pt2) foreach: 0.001290420782301695
- basic_gnn_edgecnn, Adamax, cuda, foreach: 0.0010534380900207906
- basic_gnn_edgecnn, ASGD, cuda, (pt2) default: 0.000706615261766325
- basic_gnn_edgecnn, ASGD, cuda, default: 0.0004903525209520013
- basic_gnn_edgecnn, ASGD, cuda, (pt2) maximize: 0.0007744497236612336
- basic_gnn_edgecnn, ASGD, cuda, maximize: 0.0005357214307878166
- basic_gnn_edgecnn, ASGD, cuda, (pt2) foreach: 0.0007017623499635008
- basic_gnn_edgecnn, ASGD, cuda, foreach: 0.00048039371194317934
- basic_gnn_edgecnn, SGD, cuda, (pt2) default: 0.00021222937992385817
- basic_gnn_edgecnn, SGD, cuda, default: 7.198739519808441e-05
- basic_gnn_edgecnn, SGD, cuda, (pt2) maximize: 0.00028791414010975
- basic_gnn_edgecnn, SGD, cuda, maximize: 0.00011161546595394612
- basic_gnn_edgecnn, SGD, cuda, (pt2) foreach: 0.00016746152949053794
- basic_gnn_edgecnn, SGD, cuda, foreach: 6.737151490524411e-05
- basic_gnn_edgecnn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.00030126149242979484
- basic_gnn_edgecnn, SGD, cuda, foreach, momentum=0.9, nesterov: 0.00010046569292899222
- basic_gnn_edgecnn, SGD, cuda, (pt2) foreach, momentum=0.9: 0.00029309324498832403
- basic_gnn_edgecnn, SGD, cuda, foreach, momentum=0.9: 9.061213691408435e-05
- basic_gnn_edgecnn, RAdam, cuda, (pt2) default: 0.0006480076462030411
- basic_gnn_edgecnn, RAdam, cuda, default: 0.0004896881601307541
- basic_gnn_edgecnn, RAdam, cuda, (pt2) foreach: 0.0006740732539910823
- basic_gnn_edgecnn, RAdam, cuda, foreach: 0.00048574408097192646
- basic_gnn_edgecnn, Rprop, cuda, (pt2) default: 0.0019747057596367656
- basic_gnn_edgecnn, Rprop, cuda, default: 0.0017262674507219345
- basic_gnn_edgecnn, Rprop, cuda, (pt2) maximize: 0.0020301004359028255
- basic_gnn_edgecnn, Rprop, cuda, maximize: 0.001752198450267315
- basic_gnn_edgecnn, Rprop, cuda, (pt2) foreach: 0.002000255531165749
- basic_gnn_edgecnn, Rprop, cuda, foreach: 0.0017110374942421913
- basic_gnn_edgecnn, RMSprop, cuda, (pt2) default: 0.00038083572163752957
- basic_gnn_edgecnn, RMSprop, cuda, default: 0.00014923246600665152
- basic_gnn_edgecnn, RMSprop, cuda, (pt2) maximize: 0.0004158176708261709
- basic_gnn_edgecnn, RMSprop, cuda, maximize: 0.00018293860100675375
- basic_gnn_edgecnn, RMSprop, cuda, (pt2) foreach: 0.0003619456059968008
- basic_gnn_edgecnn, RMSprop, cuda, foreach: 0.00014613556698895992
- basic_gnn_edgecnn, NAdam, cuda, (pt2) default: 0.0006670964160002768
- basic_gnn_edgecnn, NAdam, cuda, default: 0.00043209474580362437
- basic_gnn_edgecnn, NAdam, cuda, (pt2) foreach: 0.0006538694260301056
- basic_gnn_edgecnn, NAdam, cuda, foreach: 0.0004398880482185632
- basic_gnn_gcn, Adadelta, cuda, (pt2) default: 0.00038471602825806115
- basic_gnn_gcn, Adadelta, cuda, default: 0.0001764075574465096
- basic_gnn_gcn, Adadelta, cuda, (pt2) maximize: 0.00041021655512149215
- basic_gnn_gcn, Adadelta, cuda, maximize: 0.00019889542157761753
- basic_gnn_gcn, Adadelta, cuda, (pt2) foreach: 0.00038280719125475984
- basic_gnn_gcn, Adadelta, cuda, foreach: 0.00017505810211878269
- basic_gnn_gcn, Adagrad, cuda, (pt2) default: 0.000664505651963262
- basic_gnn_gcn, Adagrad, cuda, default: 0.00046108169597573577
- basic_gnn_gcn, Adagrad, cuda, (pt2) maximize: 0.0006821221991942651
- basic_gnn_gcn, Adagrad, cuda, maximize: 0.0004723676280118525
- basic_gnn_gcn, Adagrad, cuda, (pt2) foreach: 0.0006200127438642085
- basic_gnn_gcn, Adagrad, cuda, foreach: 0.00046740797511301933
- basic_gnn_gcn, Adam, cuda, (pt2) default: 0.0004672655848773209
- basic_gnn_gcn, Adam, cuda, default: 0.0002783519069198519
- basic_gnn_gcn, Adam, cuda, (pt2) amsgrad, maximize: 0.000516912083500077
- basic_gnn_gcn, Adam, cuda, amsgrad, maximize: 0.0003168030658271164
- basic_gnn_gcn, Adam, cuda, (pt2) foreach: 0.00043694492895156144
- basic_gnn_gcn, Adam, cuda, foreach: 0.00026085729897022247
- basic_gnn_gcn, Adam, cuda, (pt2) foreach, maximize, capturable: 0.0006970961878541857
- basic_gnn_gcn, Adam, cuda, foreach, maximize, capturable: 0.0005088703678920865
- basic_gnn_gcn, Adam, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0007004545589443296
- basic_gnn_gcn, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.000524891977198422
- basic_gnn_gcn, Adam, cuda, (pt2) fused: 0.00023878923687319268
- basic_gnn_gcn, Adam, cuda, fused: 7.05544508062303e-05
- basic_gnn_gcn, Adam, cuda, (pt2) fused, amsgrad, maximize: 0.00024453746433845724
- basic_gnn_gcn, Adam, cuda, fused, amsgrad, maximize: 7.307244869880379e-05
- basic_gnn_gcn, Adam, cuda, (pt2) fused, capturable: 0.00023878126673018094
- basic_gnn_gcn, Adam, cuda, fused, capturable: 7.06519688013941e-05
- basic_gnn_gcn, Adam, cuda, (pt2) fused, capturable, amsgrad: 0.00025299985453900884
- basic_gnn_gcn, Adam, cuda, fused, capturable, amsgrad: 7.25177648011595e-05
- basic_gnn_gcn, AdamW, cuda, (pt2) default: 0.0005077974933440689
- basic_gnn_gcn, AdamW, cuda, default: 0.0002690971929114312
- basic_gnn_gcn, AdamW, cuda, (pt2) amsgrad, maximize: 0.0005365602163992443
- basic_gnn_gcn, AdamW, cuda, amsgrad, maximize: 0.0003168418281711638
- basic_gnn_gcn, AdamW, cuda, (pt2) foreach: 0.000451244181022048
- basic_gnn_gcn, AdamW, cuda, foreach: 0.00026741006318479777
- basic_gnn_gcn, AdamW, cuda, (pt2) foreach, maximize, capturable: 0.0006879513019230216
- basic_gnn_gcn, AdamW, cuda, foreach, maximize, capturable: 0.0005110270758159459
- basic_gnn_gcn, AdamW, cuda, (pt2) foreach, maximize, capturable, amsgrad: 0.0007159707529935986
- basic_gnn_gcn, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.0005273034949786961
- basic_gnn_gcn, AdamW, cuda, (pt2) fused: 0.0002598358212377537
- basic_gnn_gcn, AdamW, cuda, fused: 7.13235649978742e-05
- basic_gnn_gcn, AdamW, cuda, (pt2) fused, amsgrad, maximize: 0.00024239619580474086
- basic_gnn_gcn, AdamW, cuda, fused, amsgrad, maximize: 7.37658079015091e-05
- basic_gnn_gcn, AdamW, cuda, (pt2) fused, capturable: 0.0002414302788652363
- basic_gnn_gcn, AdamW, cuda, fused, capturable: 7.057968368753792e-05
- basic_gnn_gcn, AdamW, cuda, (pt2) fused, capturable, amsgrad: 0.00024195838980872224
- basic_gnn_gcn, AdamW, cuda, fused, capturable, amsgrad: 7.283758481498808e-05
- basic_gnn_gcn, Adamax, cuda, (pt2) default: 0.0008271124371720001
- basic_gnn_gcn, Adamax, cuda, default: 0.0006303306790068745
- basic_gnn_gcn, Adamax, cuda, (pt2) maximize: 0.0008716503340426994
- basic_gnn_gcn, Adamax, cuda, maximize: 0.0006355994630139321
- basic_gnn_gcn, Adamax, cuda, (pt2) foreach: 0.0007908618977914253
- basic_gnn_gcn, Adamax, cuda, foreach: 0.0006114679470192641
- basic_gnn_gcn, ASGD, cuda, (pt2) default: 0.0004980771232451966
- basic_gnn_gcn, ASGD, cuda, default: 0.0002807428662199527
- basic_gnn_gcn, ASGD, cuda, (pt2) maximize: 0.0005407047791507196
- basic_gnn_gcn, ASGD, cuda, maximize: 0.00031535445712506773
- basic_gnn_gcn, ASGD, cuda, (pt2) foreach: 0.0004496780929621309
- basic_gnn_gcn, ASGD, cuda, foreach: 0.0002911854370031506
- basic_gnn_gcn, SGD, cuda, (pt2) default: 0.00014180471701547503
- basic_gnn_gcn, SGD, cuda, default: 5.3235680004581806e-05
- basic_gnn_gcn, SGD, cuda, (pt2) maximize: 0.0001925961449742317
- basic_gnn_gcn, SGD, cuda, maximize: 8.118531736545266e-05
- basic_gnn_gcn, SGD, cuda, (pt2) foreach: 0.00013619967200793326
- basic_gnn_gcn, SGD, cuda, foreach: 5.005109161138535e-05
- basic_gnn_gcn, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: 0.00018256349256262183
- basic_gnn_gcn, SGD, cuda, foreach, momentum=0.9, nesterov: 7.84975024095426e-05
- basic_gnn_gcn, SGD, cuda, (pt2) foreach, momentum=0.9: 0.00017295303044375033
- basic_gnn_gcn, SGD, cuda, foreach, momentum=0.9: 6.990915439091623e-05
- basic_gnn_gcn, RAdam, cuda, (pt2) default: 0.0004846932750660926
- basic_gnn_gcn, RAdam, cuda, default: 0.0003287171800620854
- basic_gnn_gcn, RAdam, cuda, (pt2) foreach: 0.00048470705514773725
- basic_gnn_gcn, RAdam, cuda, foreach: 0.0003256590578239411
- basic_gnn_gcn, Rprop, cuda, (pt2) default: 0.0011941795057195815
- basic_gnn_gcn, Rprop, cuda, default: 0.0009550762494715551
- basic_gnn_gcn, Rprop, cuda, (pt2) maximize: 0.0012071397399574697
- basic_gnn_gcn, Rprop, cuda, maximize: 0.000998670479748398
- basic_gnn_gcn, Rprop, cuda, (pt2) foreach: 0.0011729428255985005
- basic_gnn_gcn, Rprop, cuda, foreach: 0.0009366219397634268
- basic_gnn_gcn, RMSprop, cuda, (pt2) default: 0.00031153545272464043
- basic_gnn_gcn, RMSprop, cuda, default: 0.00010978164547123016
- basic_gnn_gcn, RMSprop, cuda, (pt2) maximize: 0.00034307998702517913
- basic_gnn_gcn, RMSprop, cuda, maximize: 0.00013217158801853658
- basic_gnn_gcn, RMSprop, cuda, (pt2) foreach: 0.00030400957572514523
- basic_gnn_gcn, RMSprop, cuda, foreach: 0.00010432458447758109
- basic_gnn_gcn, NAdam, cuda, (pt2) default: 0.000511587798700232
- basic_gnn_gcn, NAdam, cuda, default: 0.00029777500894851984
- basic_gnn_gcn, NAdam, cuda, (pt2) foreach: 0.000514210677636275
- basic_gnn_gcn, NAdam, cuda, foreach: 0.00029593827691860495
- nanogpt_generate, Adadelta, cpu, (pt2) foreach: 1.2292320281267166
- nanogpt_generate, Adadelta, cpu, foreach: 1.2100508369039744
- nanogpt_generate, Adagrad, cpu, (pt2) foreach: 0.8427368488628417
- nanogpt_generate, Adagrad, cpu, foreach: 0.8060000510886312
- nanogpt_generate, Adam, cpu, (pt2) foreach: 0.6589399171061814
- nanogpt_generate, Adam, cpu, foreach: 0.5971417389810085
- nanogpt_generate, AdamW, cpu, (pt2) foreach: 0.6150765139609575
- nanogpt_generate, AdamW, cpu, foreach: 0.6282726731151342
- nanogpt_generate, Adamax, cpu, (pt2) foreach: 1.222835793858394
- nanogpt_generate, Adamax, cpu, foreach: 1.241614885861054
- nanogpt_generate, ASGD, cpu, (pt2) foreach: 0.2131362238433212
- nanogpt_generate, ASGD, cpu, foreach: 0.21020758710801601
- nanogpt_generate, SGD, cpu, (pt2) foreach: 0.08165178587660193
- nanogpt_generate, SGD, cpu, foreach: 0.07976425730157644
- nanogpt_generate, SGD, cpu, (pt2) foreach, momentum=0.9, nesterov: 0.57738480810076
- nanogpt_generate, SGD, cpu, foreach, momentum=0.9, nesterov: 0.28351306403055787
- nanogpt_generate, SGD, cpu, (pt2) foreach, momentum=0.9: 0.4952849270775914
- nanogpt_generate, SGD, cpu, foreach, momentum=0.9: 0.20528484811075032
- nanogpt_generate, RAdam, cpu, (pt2) foreach: 0.9709670641459525
- nanogpt_generate, RAdam, cpu, foreach: 0.9500419900286943
- nanogpt_generate, Rprop, cpu, (pt2) foreach: 1.7792327681090683
- nanogpt_generate, Rprop, cpu, foreach: 1.7742392979562283
- nanogpt_generate, RMSprop, cpu, (pt2) foreach: 0.36828134092502296
- nanogpt_generate, RMSprop, cpu, foreach: 0.3790253240149468
- nanogpt_generate, NAdam, cpu, (pt2) foreach: 0.708936874056235
- nanogpt_generate, NAdam, cpu, foreach: 0.6920408071018755


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/4966690518

cc @janeyx99


Intriguing that llama had massive slowdowns all across the board....

Everything else seems fine