Repository: pytorch/benchmark
Issue: ASGD perf regression
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 0200b1106c4fe80ea0884181dc8d649ef6078ea3

Affected PyTorch commit: 806d1a871ddfd2d38e1791489892009feaec8425

Affected Tests:
- resnet50, ASGD, cuda, default: +124.02993%
- resnet50, ASGD, cuda, maximize: +91.23220%
- resnet50, ASGD, cuda, no_foreach: +123.98829%
- resnet50, ASGD, cuda, differentiable: +119.85504%
- resnet50, ASGD, cuda, foreach: +105.76765%
- mnasnet1_0, ASGD, cuda, default: +115.15906%
- mnasnet1_0, ASGD, cuda, maximize: +93.84647%
- mnasnet1_0, ASGD, cuda, no_foreach: +138.73016%
- mnasnet1_0, ASGD, cuda, differentiable: +121.80861%
- mnasnet1_0, ASGD, cuda, foreach: +104.30969%
- squeezenet1_1, ASGD, cuda, default: +94.39419%
- squeezenet1_1, ASGD, cuda, maximize: +86.72449%
- squeezenet1_1, ASGD, cuda, no_foreach: +119.51013%
- squeezenet1_1, ASGD, cuda, differentiable: +113.43829%
- squeezenet1_1, ASGD, cuda, foreach: +102.96500%
- sam, ASGD, cuda, default: +195.67645%
- sam, ASGD, cuda, maximize: +181.55680%
- sam, ASGD, cuda, no_foreach: +140.95377%
- sam, ASGD, cuda, differentiable: +147.18748%
- sam, ASGD, cuda, foreach: +193.14937%
- vgg16, ASGD, cuda, default: +128.22068%
- vgg16, ASGD, cuda, maximize: +95.99951%
- vgg16, ASGD, cuda, no_foreach: +129.91319%
- vgg16, ASGD, cuda, differentiable: +129.00090%
- vgg16, ASGD, cuda, foreach: +126.44894%
- timm_vision_transformer, Adadelta, cuda, (pt2) foreach: -99.98788%
- timm_vision_transformer, Adam, cuda, (pt2) fused: +82.68205%
- timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: -99.98358%
- timm_vision_transformer, AdamW, cuda, (pt2) foreach: -67.46933%
- timm_vision_transformer, AdamW, cuda, (pt2) fused: +71.99635%
- timm_vision_transformer, ASGD, cuda, default: +106.71975%
- timm_vision_transformer, ASGD, cuda, maximize: +96.59117%
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: +143.52541%
- timm_vision_transformer, ASGD, cuda, no_foreach: +91.65669%
- timm_vision_transformer, ASGD, cuda, differentiable: +116.06366%
- timm_vision_transformer, ASGD, cuda, foreach: +115.47567%
- compile_time, timm_vision_transformer, Adadelta, cuda, (pt2) foreach: +37807.53782%
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: +7077.47102%
- compile_time, timm_vision_transformer, AdamW, cuda, (pt2) fused: +361.28020%
- compile_time, timm_vision_transformer, Adamax, cuda, (pt2) foreach: +217.33371%
- compile_time, timm_vision_transformer, SGD, cuda, (pt2) foreach: +813.99024%
- timm_vovnet, ASGD, cuda, default: +79.70207%
- timm_vovnet, ASGD, cuda, maximize: +81.93381%
- timm_vovnet, ASGD, cuda, no_foreach: +92.76432%
- timm_vovnet, ASGD, cuda, differentiable: +100.97167%
- timm_vovnet, ASGD, cuda, foreach: +79.48599%
- speech_transformer, ASGD, cuda, default: +98.57801%
- speech_transformer, ASGD, cuda, maximize: +92.68815%
- speech_transformer, ASGD, cuda, no_foreach: +113.97402%
- speech_transformer, ASGD, cuda, differentiable: +109.36097%
- speech_transformer, ASGD, cuda, foreach: +99.81050%
- basic_gnn_sage, ASGD, cuda, default: +86.73222%
- basic_gnn_sage, ASGD, cuda, maximize: +75.81285%
- basic_gnn_sage, ASGD, cuda, no_foreach: +116.76823%
- basic_gnn_sage, ASGD, cuda, differentiable: +106.38985%
- basic_gnn_sage, ASGD, cuda, foreach: +96.91066%
- hf_T5, ASGD, cuda, default: +127.93302%
- hf_T5, ASGD, cuda, maximize: +114.74867%
- hf_T5, ASGD, cuda, no_foreach: +131.01003%
- hf_T5, ASGD, cuda, differentiable: +127.07553%
- hf_T5, ASGD, cuda, foreach: +155.85461%
- mobilenet_v2_quantized_qat, ASGD, cuda, default: +95.82244%
- mobilenet_v2_quantized_qat, ASGD, cuda, maximize: +86.96235%
- mobilenet_v2_quantized_qat, ASGD, cuda, no_foreach: +107.97934%
- mobilenet_v2_quantized_qat, ASGD, cuda, differentiable: +116.05228%
- mobilenet_v2_quantized_qat, ASGD, cuda, foreach: +106.87671%
- maml_omniglot, ASGD, cuda, default: +87.10180%
- maml_omniglot, ASGD, cuda, maximize: +75.81007%
- maml_omniglot, ASGD, cuda, no_foreach: +109.66096%
- maml_omniglot, ASGD, cuda, differentiable: +116.56157%
- maml_omniglot, ASGD, cuda, foreach: +90.88092%
- alexnet, ASGD, cuda, default: +142.00344%
- alexnet, ASGD, cuda, maximize: +117.11555%
- alexnet, ASGD, cuda, no_foreach: +148.61658%
- alexnet, ASGD, cuda, differentiable: +147.32024%
- alexnet, ASGD, cuda, foreach: +140.08364%
- opacus_cifar10, ASGD, cuda, default: +88.46431%
- opacus_cifar10, ASGD, cuda, maximize: +77.82195%
- opacus_cifar10, ASGD, cuda, no_foreach: +114.27061%
- opacus_cifar10, ASGD, cuda, differentiable: +109.31896%
- opacus_cifar10, ASGD, cuda, foreach: +84.54530%
- hf_Bert, ASGD, cuda, default: +142.73106%
- hf_Bert, ASGD, cuda, maximize: +117.33846%
- hf_Bert, ASGD, cuda, no_foreach: +117.59002%
- hf_Bert, ASGD, cuda, differentiable: +109.75356%
- hf_Bert, ASGD, cuda, foreach: +137.88056%
- timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: -99.98482%
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: +31.17705%
- timm_vision_transformer_large, Adam, cuda, (pt2) default: +33.26611%
- timm_vision_transformer_large, Adam, cuda, (pt2) no_foreach: +41.87344%
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: +64.34802%
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: +37.09363%
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: +41.07935%
- timm_vision_transformer_large, ASGD, cuda, default: +123.28451%
- timm_vision_transformer_large, ASGD, cuda, maximize: +140.95934%
- timm_vision_transformer_large, ASGD, cuda, no_foreach: +149.50390%
- timm_vision_transformer_large, ASGD, cuda, differentiable: +218.22416%
- timm_vision_transformer_large, ASGD, cuda, foreach: +124.79011%
- timm_vision_transformer_large, Rprop, cuda, (pt2) default: +31.40632%
- compile_time, timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: -7716.53938%
- hf_T5_large, ASGD, cuda, default: +211.69880%
- hf_T5_large, ASGD, cuda, maximize: +169.96072%
- hf_T5_large, ASGD, cuda, no_foreach: +141.02743%
- hf_T5_large, ASGD, cuda, differentiable: +143.63304%
- hf_T5_large, ASGD, cuda, foreach: +208.19100%
- vision_maskrcnn, ASGD, cuda, default: +100.31787%
- vision_maskrcnn, ASGD, cuda, maximize: +89.12360%
- vision_maskrcnn, ASGD, cuda, no_foreach: +98.99280%
- vision_maskrcnn, ASGD, cuda, differentiable: +99.44593%
- vision_maskrcnn, ASGD, cuda, foreach: +101.92607%
- hf_GPT2_large, ASGD, cuda, default: +193.84464%
- hf_GPT2_large, ASGD, cuda, maximize: +153.89762%
- hf_GPT2_large, ASGD, cuda, no_foreach: +158.69141%
- hf_GPT2_large, ASGD, cuda, differentiable: +163.62273%
- hf_GPT2_large, ASGD, cuda, foreach: +192.48542%
- BERT_pytorch, ASGD, cuda, default: +135.65602%
- BERT_pytorch, ASGD, cuda, maximize: +119.74721%
- BERT_pytorch, ASGD, cuda, no_foreach: +125.09949%
- BERT_pytorch, ASGD, cuda, differentiable: +109.46087%
- BERT_pytorch, ASGD, cuda, foreach: +133.43466%
- detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, default: +122.49460%
- detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, maximize: +103.09817%
- detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, no_foreach: +118.01190%
- detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, differentiable: +120.14865%
- detectron2_fasterrcnn_r_50_fpn, ASGD, cuda, foreach: +121.01315%
- dlrm, ASGD, cuda, default: +78.09405%
- dlrm, ASGD, cuda, maximize: +59.68144%
- dlrm, ASGD, cuda, foreach: +78.35686%
- fastNLP_Bert, ASGD, cuda, default: +124.20659%
- fastNLP_Bert, ASGD, cuda, maximize: +129.74865%
- fastNLP_Bert, ASGD, cuda, no_foreach: +131.86236%
- fastNLP_Bert, ASGD, cuda, differentiable: +124.50400%
- fastNLP_Bert, ASGD, cuda, foreach: +129.15546%
- basic_gnn_gcn, ASGD, cuda, default: +95.32429%
- basic_gnn_gcn, ASGD, cuda, maximize: +87.14749%
- basic_gnn_gcn, ASGD, cuda, no_foreach: +110.78651%
- basic_gnn_gcn, ASGD, cuda, differentiable: +111.55591%
- basic_gnn_gcn, ASGD, cuda, foreach: +107.76394%
- phlippe_resnet, ASGD, cuda, default: +93.23522%
- phlippe_resnet, ASGD, cuda, maximize: +86.87177%
- phlippe_resnet, ASGD, cuda, no_foreach: +130.84100%
- phlippe_resnet, ASGD, cuda, differentiable: +115.97482%
- phlippe_resnet, ASGD, cuda, foreach: +98.55166%
- timm_resnest, ASGD, cuda, default: +103.57621%
- timm_resnest, ASGD, cuda, maximize: +107.16194%
- timm_resnest, ASGD, cuda, no_foreach: +119.73967%
- timm_resnest, ASGD, cuda, differentiable: +129.29690%
- timm_resnest, ASGD, cuda, foreach: +118.03992%
- basic_gnn_gin, ASGD, cuda, default: +115.13085%
- basic_gnn_gin, ASGD, cuda, maximize: +83.92329%
- basic_gnn_gin, ASGD, cuda, no_foreach: +107.37685%
- basic_gnn_gin, ASGD, cuda, differentiable: +111.10815%
- basic_gnn_gin, ASGD, cuda, foreach: +101.07616%
- resnet50_quantized_qat, ASGD, cuda, default: +81.96953%
- resnet50_quantized_qat, ASGD, cuda, maximize: +73.56566%
- resnet50_quantized_qat, ASGD, cuda, no_foreach: +126.73629%
- resnet50_quantized_qat, ASGD, cuda, differentiable: +108.65208%
- resnet50_quantized_qat, ASGD, cuda, foreach: +90.47366%
- Background_Matting, ASGD, cuda, default: +85.40617%
- Background_Matting, ASGD, cuda, maximize: +80.75659%
- Background_Matting, ASGD, cuda, no_foreach: +104.39196%
- Background_Matting, ASGD, cuda, differentiable: +131.27337%
- Background_Matting, ASGD, cuda, foreach: +95.64608%
- tacotron2, ASGD, cuda, default: +119.66504%
- tacotron2, ASGD, cuda, maximize: +107.00849%
- tacotron2, ASGD, cuda, no_foreach: +140.78569%
- tacotron2, ASGD, cuda, differentiable: +129.29204%
- tacotron2, ASGD, cuda, foreach: +121.62964%
- llama, ASGD, cuda, default: -37.89294%
- llama, ASGD, cuda, maximize: -37.69863%
- llama, ASGD, cuda, foreach: -39.76405%
- demucs, ASGD, cuda, default: +133.49940%
- demucs, ASGD, cuda, maximize: +100.55682%
- demucs, ASGD, cuda, no_foreach: +145.58503%
- demucs, ASGD, cuda, differentiable: +136.17159%
- demucs, ASGD, cuda, foreach: +129.51777%
- pytorch_unet, ASGD, cuda, default: +89.85632%
- pytorch_unet, ASGD, cuda, maximize: +88.41714%
- pytorch_unet, ASGD, cuda, no_foreach: +133.38601%
- pytorch_unet, ASGD, cuda, differentiable: +124.86886%
- pytorch_unet, ASGD, cuda, foreach: +95.00974%
- hf_Albert, ASGD, cuda, default: +105.88921%
- hf_Albert, ASGD, cuda, maximize: +95.25225%
- hf_Albert, ASGD, cuda, no_foreach: +113.10521%
- hf_Albert, ASGD, cuda, differentiable: +112.50182%
- hf_Albert, ASGD, cuda, foreach: +121.51764%
- tts_angular, ASGD, cuda, default: +111.68752%
- tts_angular, ASGD, cuda, maximize: +91.98089%
- tts_angular, ASGD, cuda, no_foreach: +106.44684%
- tts_angular, ASGD, cuda, differentiable: +109.35268%
- tts_angular, ASGD, cuda, foreach: +124.90487%
- timm_nfnet, ASGD, cuda, default: +109.12394%
- timm_nfnet, ASGD, cuda, maximize: +99.66265%
- timm_nfnet, ASGD, cuda, no_foreach: +112.02449%
- timm_nfnet, ASGD, cuda, differentiable: +120.72541%
- timm_nfnet, ASGD, cuda, foreach: +97.20528%
- dcgan, ASGD, cuda, default: +80.42478%
- dcgan, ASGD, cuda, maximize: +74.82730%
- dcgan, ASGD, cuda, no_foreach: +93.25956%
- dcgan, ASGD, cuda, differentiable: +106.39970%
- dcgan, ASGD, cuda, foreach: +84.90491%
- moco, ASGD, cuda, default: +85.82898%
- moco, ASGD, cuda, maximize: +79.96736%
- moco, ASGD, cuda, no_foreach: +122.59127%
- moco, ASGD, cuda, differentiable: +116.06429%
- moco, ASGD, cuda, foreach: +100.20179%
- detectron2_maskrcnn_r_101_fpn, ASGD, cuda, default: +121.34472%
- detectron2_maskrcnn_r_101_fpn, ASGD, cuda, maximize: +102.83182%
- detectron2_maskrcnn_r_101_fpn, ASGD, cuda, no_foreach: +119.60952%
- detectron2_maskrcnn_r_101_fpn, ASGD, cuda, differentiable: +106.19279%
- detectron2_maskrcnn_r_101_fpn, ASGD, cuda, foreach: +107.71527%
- detectron2_maskrcnn, ASGD, cuda, default: +117.57517%
- detectron2_maskrcnn, ASGD, cuda, maximize: +103.30191%
- detectron2_maskrcnn, ASGD, cuda, no_foreach: +116.90257%
- detectron2_maskrcnn, ASGD, cuda, differentiable: +117.56160%
- detectron2_maskrcnn, ASGD, cuda, foreach: +110.69690%
- mobilenet_v2, ASGD, cuda, default: +96.95437%
- mobilenet_v2, ASGD, cuda, maximize: +91.66412%
- mobilenet_v2, ASGD, cuda, no_foreach: +105.93441%
- mobilenet_v2, ASGD, cuda, differentiable: +106.24384%
- mobilenet_v2, ASGD, cuda, foreach: +101.98825%
- phlippe_densenet, ASGD, cuda, default: +90.63482%
- phlippe_densenet, ASGD, cuda, maximize: +82.98210%
- phlippe_densenet, ASGD, cuda, no_foreach: +109.94099%
- phlippe_densenet, ASGD, cuda, differentiable: +99.62891%
- phlippe_densenet, ASGD, cuda, foreach: +102.07940%
- stable_diffusion, ASGD, cuda, default: +173.16434%
- stable_diffusion, ASGD, cuda, maximize: +172.79215%
- stable_diffusion, ASGD, cuda, no_foreach: +133.62548%
- stable_diffusion, ASGD, cuda, differentiable: +150.74043%
- stable_diffusion, ASGD, cuda, foreach: +193.07518%
- detectron2_fasterrcnn_r_101_dc5, ASGD, cuda, default: +212.36047%
- detectron2_fasterrcnn_r_101_dc5, ASGD, cuda, maximize: +183.13374%
- detectron2_fasterrcnn_r_101_dc5, ASGD, cuda, no_foreach: +157.73100%
- detectron2_fasterrcnn_r_101_dc5, ASGD, cuda, differentiable: +158.22935%
- detectron2_fasterrcnn_r_101_dc5, ASGD, cuda, foreach: +214.21778%
- Super_SloMo, ASGD, cuda, default: +106.84278%
- Super_SloMo, ASGD, cuda, maximize: +91.77756%
- Super_SloMo, ASGD, cuda, no_foreach: +120.37297%
- Super_SloMo, ASGD, cuda, differentiable: +107.75837%
- Super_SloMo, ASGD, cuda, foreach: +105.20420%
- timm_efficientnet, ASGD, cuda, default: +100.11609%
- timm_efficientnet, ASGD, cuda, maximize: +96.02220%
- timm_efficientnet, ASGD, cuda, no_foreach: +117.61484%
- timm_efficientnet, ASGD, cuda, differentiable: +126.42353%
- timm_efficientnet, ASGD, cuda, foreach: +104.98403%
- shufflenet_v2_x1_0, ASGD, cuda, default: +113.97946%
- shufflenet_v2_x1_0, ASGD, cuda, maximize: +94.74763%
- shufflenet_v2_x1_0, ASGD, cuda, no_foreach: +126.35174%
- shufflenet_v2_x1_0, ASGD, cuda, differentiable: +110.48815%
- shufflenet_v2_x1_0, ASGD, cuda, foreach: +104.07755%
- yolov3, ASGD, cuda, default: +78.25093%
- yolov3, ASGD, cuda, maximize: +77.13986%
- yolov3, ASGD, cuda, no_foreach: +93.41386%
- yolov3, ASGD, cuda, differentiable: +90.89548%
- yolov3, ASGD, cuda, foreach: +80.57876%
- basic_gnn_edgecnn, ASGD, cuda, default: +99.06302%
- basic_gnn_edgecnn, ASGD, cuda, maximize: +83.71059%
- basic_gnn_edgecnn, ASGD, cuda, no_foreach: +113.92414%
- basic_gnn_edgecnn, ASGD, cuda, differentiable: +110.96076%
- basic_gnn_edgecnn, ASGD, cuda, foreach: +98.60346%
- hf_Reformer, ASGD, cuda, default: +91.67268%
- hf_Reformer, ASGD, cuda, maximize: +82.64084%
- hf_Reformer, ASGD, cuda, no_foreach: +125.85424%
- hf_Reformer, ASGD, cuda, differentiable: +105.94022%
- hf_Reformer, ASGD, cuda, foreach: +90.47308%
- fambench_xlmr, ASGD, cuda, default: +201.16636%
- fambench_xlmr, ASGD, cuda, maximize: +188.32381%
- fambench_xlmr, ASGD, cuda, no_foreach: +140.31486%
- fambench_xlmr, ASGD, cuda, differentiable: +148.74302%
- fambench_xlmr, ASGD, cuda, foreach: +203.57251%
- hf_Bert_large, ASGD, cuda, default: +165.08672%
- hf_Bert_large, ASGD, cuda, maximize: +151.30732%
- hf_Bert_large, ASGD, cuda, no_foreach: +128.21581%
- hf_Bert_large, ASGD, cuda, differentiable: +126.48303%
- hf_Bert_large, ASGD, cuda, foreach: +158.99745%
- hf_GPT2, ASGD, cuda, default: +160.72487%
- hf_GPT2, ASGD, cuda, maximize: +160.21525%
- hf_GPT2, ASGD, cuda, no_foreach: +138.23310%
- hf_GPT2, ASGD, cuda, differentiable: +133.31532%
- hf_GPT2, ASGD, cuda, foreach: +173.83932%
- pytorch_stargan, ASGD, cuda, default: +87.11727%
- pytorch_stargan, ASGD, cuda, maximize: +79.06760%
- pytorch_stargan, ASGD, cuda, no_foreach: +114.98173%
- pytorch_stargan, ASGD, cuda, differentiable: +109.09947%
- pytorch_stargan, ASGD, cuda, foreach: +95.94788%
- nanogpt_generate, ASGD, cuda, default: +159.68400%
- nanogpt_generate, ASGD, cuda, maximize: +150.49595%
- nanogpt_generate, ASGD, cuda, no_foreach: +138.45369%
- nanogpt_generate, ASGD, cuda, differentiable: +136.21444%
- nanogpt_generate, ASGD, cuda, foreach: +172.49098%
- resnet152, ASGD, cuda, default: +104.25649%
- resnet152, ASGD, cuda, maximize: +96.57891%
- resnet152, ASGD, cuda, no_foreach: +119.36834%
- resnet152, ASGD, cuda, differentiable: +121.75067%
- resnet152, ASGD, cuda, foreach: +107.04176%
- hf_Whisper, ASGD, cuda, default: +103.45888%
- hf_Whisper, ASGD, cuda, maximize: +89.34926%
- hf_Whisper, ASGD, cuda, no_foreach: +125.70960%
- hf_Whisper, ASGD, cuda, differentiable: +123.69277%
- hf_Whisper, ASGD, cuda, foreach: +109.05051%
- maml, ASGD, cuda, default: +92.78801%
- maml, ASGD, cuda, maximize: +82.35410%
- maml, ASGD, cuda, no_foreach: +100.27183%
- maml, ASGD, cuda, differentiable: +109.62979%
- maml, ASGD, cuda, foreach: +98.58460%
- detectron2_fasterrcnn_r_50_dc5, ASGD, cuda, default: +171.64929%
- detectron2_fasterrcnn_r_50_dc5, ASGD, cuda, maximize: +138.27427%
- detectron2_fasterrcnn_r_50_dc5, ASGD, cuda, no_foreach: +179.65960%
- detectron2_fasterrcnn_r_50_dc5, ASGD, cuda, differentiable: +187.33681%
- detectron2_fasterrcnn_r_50_dc5, ASGD, cuda, foreach: +167.31098%
- hf_Bart, ASGD, cuda, default: +138.82232%
- hf_Bart, ASGD, cuda, maximize: +130.23887%
- hf_Bart, ASGD, cuda, no_foreach: +123.55852%
- hf_Bart, ASGD, cuda, differentiable: +130.70510%
- hf_Bart, ASGD, cuda, foreach: +134.35986%
- cm3leon_generate, ASGD, cuda, default: +198.06865%
- cm3leon_generate, ASGD, cuda, maximize: +151.59751%
- cm3leon_generate, ASGD, cuda, no_foreach: +154.95043%
- cm3leon_generate, ASGD, cuda, differentiable: +165.66967%
- cm3leon_generate, ASGD, cuda, foreach: +197.54174%
- mobilenet_v3_large, ASGD, cuda, default: +104.23360%
- mobilenet_v3_large, ASGD, cuda, maximize: +93.58383%
- mobilenet_v3_large, ASGD, cuda, no_foreach: +119.33540%
- mobilenet_v3_large, ASGD, cuda, differentiable: +123.04459%
- mobilenet_v3_large, ASGD, cuda, foreach: +110.83179%
- hf_T5_base, ASGD, cuda, default: +173.34395%
- hf_T5_base, ASGD, cuda, maximize: +154.96389%
- hf_T5_base, ASGD, cuda, no_foreach: +131.31289%
- hf_T5_base, ASGD, cuda, differentiable: +121.65876%
- hf_T5_base, ASGD, cuda, foreach: +184.47838%
- hf_BigBird, ASGD, cuda, default: +150.36223%
- hf_BigBird, ASGD, cuda, maximize: +140.99390%
- hf_BigBird, ASGD, cuda, no_foreach: +137.36228%
- hf_BigBird, ASGD, cuda, differentiable: +130.69910%
- hf_BigBird, ASGD, cuda, foreach: +163.78429%
- nvidia_deeprecommender, ASGD, cuda, default: +84.22858%
- nvidia_deeprecommender, ASGD, cuda, maximize: +73.59427%
- nvidia_deeprecommender, ASGD, cuda, no_foreach: +38.45586%
- nvidia_deeprecommender, ASGD, cuda, differentiable: +38.19497%
- nvidia_deeprecommender, ASGD, cuda, foreach: +84.71843%
- DALLE2_pytorch, ASGD, cuda, default: +120.79583%
- DALLE2_pytorch, ASGD, cuda, maximize: +101.18644%
- DALLE2_pytorch, ASGD, cuda, no_foreach: +128.45109%
- DALLE2_pytorch, ASGD, cuda, differentiable: +119.58039%
- DALLE2_pytorch, ASGD, cuda, foreach: +104.22114%
- resnet18, Adadelta, cuda, (pt2) no_foreach: -99.98704%
- resnet18, Adagrad, cuda, (pt2) no_foreach: -99.98506%
- resnet18, Adam, cuda, (pt2) no_foreach: +646679.56820%
- resnet18, Adam, cuda, (pt2) foreach: -99.97264%
- resnet18, Adam, cuda, (pt2) fused: +86.64405%
- resnet18, AdamW, cuda, (pt2) fused: +99.35395%
- resnet18, ASGD, cuda, default: +97.33380%
- resnet18, ASGD, cuda, maximize: +96.01251%
- resnet18, ASGD, cuda, (pt2) no_foreach: +135.46526%
- resnet18, ASGD, cuda, no_foreach: +111.06213%
- resnet18, ASGD, cuda, differentiable: +123.23425%
- resnet18, ASGD, cuda, foreach: +97.16978%
- resnet18, SGD, cuda, (pt2) foreach: +32.97657%
- resnet18, Rprop, cuda, (pt2) foreach: -99.99361%
- compile_time, resnet18, Adadelta, cuda, (pt2) no_foreach: +400.19988%
- compile_time, resnet18, Adagrad, cuda, (pt2) no_foreach: +429.48024%
- compile_time, resnet18, Adam, cuda, (pt2) foreach: +66977.29421%
- compile_time, resnet18, Adam, cuda, (pt2) fused: +223.51315%
- compile_time, resnet18, Adamax, cuda, (pt2) foreach: +302.31806%
- compile_time, resnet18, Rprop, cuda, (pt2) foreach: -4850.60281%
- compile_time, resnet18, NAdam, cuda, (pt2) foreach: +317.34996%
- detectron2_maskrcnn_r_101_c4, ASGD, cuda, default: +117.24819%
- detectron2_maskrcnn_r_101_c4, ASGD, cuda, maximize: +105.55289%
- detectron2_maskrcnn_r_101_c4, ASGD, cuda, no_foreach: +124.03172%
- detectron2_maskrcnn_r_101_c4, ASGD, cuda, differentiable: +116.10379%
- detectron2_maskrcnn_r_101_c4, ASGD, cuda, foreach: +111.02508%
- hf_T5_generate, ASGD, cuda, default: +125.90060%
- hf_T5_generate, ASGD, cuda, maximize: +113.89290%
- hf_T5_generate, ASGD, cuda, no_foreach: +127.24467%
- hf_T5_generate, ASGD, cuda, differentiable: +124.64940%
- hf_T5_generate, ASGD, cuda, foreach: +124.15655%
- hf_Longformer, ASGD, cuda, default: +125.81518%
- hf_Longformer, ASGD, cuda, maximize: +113.85564%
- hf_Longformer, ASGD, cuda, no_foreach: +126.27509%
- hf_Longformer, ASGD, cuda, differentiable: +115.95017%
- hf_Longformer, ASGD, cuda, foreach: +131.63718%
- timm_regnet, ASGD, cuda, default: +125.73607%
- timm_regnet, ASGD, cuda, maximize: +100.80882%
- timm_regnet, ASGD, cuda, no_foreach: +120.90401%
- timm_regnet, ASGD, cuda, differentiable: +126.60772%
- timm_regnet, ASGD, cuda, foreach: +109.82537%
- hf_DistilBert, ASGD, cuda, default: +136.46659%
- hf_DistilBert, ASGD, cuda, maximize: +154.20666%
- hf_DistilBert, ASGD, cuda, no_foreach: +133.22829%
- hf_DistilBert, ASGD, cuda, differentiable: +132.68567%
- hf_DistilBert, ASGD, cuda, foreach: +141.10390%
- pytorch_CycleGAN_and_pix2pix, ASGD, cuda, default: +85.50919%
- pytorch_CycleGAN_and_pix2pix, ASGD, cuda, maximize: +79.91551%
- pytorch_CycleGAN_and_pix2pix, ASGD, cuda, no_foreach: +114.22815%
- pytorch_CycleGAN_and_pix2pix, ASGD, cuda, differentiable: +105.53023%
- pytorch_CycleGAN_and_pix2pix, ASGD, cuda, foreach: +99.25474%


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07160614579916
- timm_vision_transformer_large, RAdam, cuda, default: 0.08048462790126602
- timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.07142815878614783
- timm_vision_transformer_large, RAdam, cuda, foreach: 0.07780114312966664
- timm_vision_transformer_large, NAdam, cuda, default: 0.07558898767456412
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) default: 0.07164892243842283
- compile_time, timm_vision_transformer_large, RAdam, cuda, (pt2) foreach: 0.06621869125713906


Tests that were newly added on affected commit:
- timm_vision_transformer, ASGD, cuda, (pt2) default: 0.003701934844932773
- timm_vision_transformer, ASGD, cuda, (pt2) foreach: 0.0037623849197256343
- compile_time, timm_vision_transformer, ASGD, cuda, (pt2) default: 54.06454637941594
- compile_time, timm_vision_transformer, ASGD, cuda, (pt2) foreach: 51.20224057789892
- timm_vision_transformer_large, ASGD, cuda, (pt2) default: 0.023793959265781775
- timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 206.14174058521166
- timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 0.0820496737336119
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) default: 427.85859452995163
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) no_foreach: 13.715244447502016
- compile_time, timm_vision_transformer_large, ASGD, cuda, (pt2) foreach: 458.7825940608357
- doctr_det_predictor, Adadelta, cuda, default: 0.0028802707511931657
- doctr_det_predictor, Adadelta, cuda, maximize: 0.0031606352096423505
- doctr_det_predictor, Adadelta, cuda, no_foreach: 0.017995787295512856
- doctr_det_predictor, Adadelta, cuda, differentiable: 0.021498963190242647
- doctr_det_predictor, Adadelta, cuda, foreach: 0.0025630442099645735
- doctr_det_predictor, Adagrad, cuda, default: 0.003945592548698187
- doctr_det_predictor, Adagrad, cuda, maximize: 0.0039761970192193985
- doctr_det_predictor, Adagrad, cuda, no_foreach: 0.009838954374815028
- doctr_det_predictor, Adagrad, cuda, differentiable: 0.010537376883439718
- doctr_det_predictor, Adagrad, cuda, foreach: 0.0033657571813091635
- doctr_det_predictor, Adam, cuda, default: 0.003894355911761522
- doctr_det_predictor, Adam, cuda, amsgrad, maximize: 0.004776262207888067
- doctr_det_predictor, Adam, cuda, no_foreach: 0.01696098819375038
- doctr_det_predictor, Adam, cuda, differentiable: 0.0349869754165411
- doctr_det_predictor, Adam, cuda, foreach: 0.0039052480412647126
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable: 0.00634846021886915
- doctr_det_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.006354622212238609
- doctr_det_predictor, Adam, cuda, fused: 0.0013104215636849403
- doctr_det_predictor, Adam, cuda, fused, amsgrad, maximize: 0.0013829589146189393
- doctr_det_predictor, Adam, cuda, fused, capturable: 0.0013211383065208793
- doctr_det_predictor, Adam, cuda, fused, capturable, amsgrad: 0.0013856540946289896
- doctr_det_predictor, AdamW, cuda, default: 0.003750761039555073
- doctr_det_predictor, AdamW, cuda, amsgrad, maximize: 0.0046320953080430625
- doctr_det_predictor, AdamW, cuda, no_foreach: 0.01873129182495177
- doctr_det_predictor, AdamW, cuda, differentiable: 0.03530704751610756
- doctr_det_predictor, AdamW, cuda, foreach: 0.003599941679276526
- doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable: 0.006230019740760326
- doctr_det_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.006564915589988232
- doctr_det_predictor, AdamW, cuda, fused: 0.0013546907948330045
- doctr_det_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0014247736567631363
- doctr_det_predictor, AdamW, cuda, fused, capturable: 0.001353818892966956
- doctr_det_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0014276149054057896
- doctr_det_predictor, Adamax, cuda, default: 0.018037584936246277
- doctr_det_predictor, Adamax, cuda, maximize: 0.019045850331895053
- doctr_det_predictor, Adamax, cuda, no_foreach: 0.02449008859694004
- doctr_det_predictor, Adamax, cuda, differentiable: 0.029248994728550314
- doctr_det_predictor, Adamax, cuda, foreach: 0.01793181947432458
- doctr_det_predictor, ASGD, cuda, default: 0.014260424789972602
- doctr_det_predictor, ASGD, cuda, maximize: 0.014247538428753615
- doctr_det_predictor, ASGD, cuda, no_foreach: 0.028070646012201904
- doctr_det_predictor, ASGD, cuda, differentiable: 0.027313575381413102
- doctr_det_predictor, ASGD, cuda, foreach: 0.014710384048521518
- doctr_det_predictor, SGD, cuda, default: 0.0007995309079997241
- doctr_det_predictor, SGD, cuda, maximize: 0.0014174172608181833
- doctr_det_predictor, SGD, cuda, no_foreach: 0.0017253931146115065
- doctr_det_predictor, SGD, cuda, differentiable: 0.0008048358295733729
- doctr_det_predictor, SGD, cuda, foreach: 0.000608943731058389
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0009989061842982968
- doctr_det_predictor, SGD, cuda, foreach, momentum=0.9: 0.0008235378485793868
- doctr_det_predictor, RAdam, cuda, default: 0.004357524802908301
- doctr_det_predictor, RAdam, cuda, no_foreach: 0.027767344983294605
- doctr_det_predictor, RAdam, cuda, differentiable: 0.028750475216656923
- doctr_det_predictor, RAdam, cuda, foreach: 0.004138795551843941
- doctr_det_predictor, Rprop, cuda, default: 0.021949287690222263
- doctr_det_predictor, Rprop, cuda, maximize: 0.022139092488214374
- doctr_det_predictor, Rprop, cuda, no_foreach: 0.0354912742972374
- doctr_det_predictor, Rprop, cuda, differentiable: 0.03846609420143068
- doctr_det_predictor, Rprop, cuda, foreach: 0.022724414803087713
- doctr_det_predictor, RMSprop, cuda, default: 0.0019387207855470477
- doctr_det_predictor, RMSprop, cuda, maximize: 0.002601792109198868
- doctr_det_predictor, RMSprop, cuda, no_foreach: 0.00912425147059063
- doctr_det_predictor, RMSprop, cuda, differentiable: 0.009757765879233679
- doctr_det_predictor, RMSprop, cuda, foreach: 0.0017368051270022988
- doctr_det_predictor, NAdam, cuda, default: 0.005110027710907161
- doctr_det_predictor, NAdam, cuda, no_foreach: 0.021710659796372055
- doctr_det_predictor, NAdam, cuda, differentiable: 0.03594747381284833
- doctr_det_predictor, NAdam, cuda, foreach: 0.004948554779402912
- doctr_reco_predictor, Adadelta, cuda, default: 0.0015303045674227179
- doctr_reco_predictor, Adadelta, cuda, maximize: 0.0017265696777030827
- doctr_reco_predictor, Adadelta, cuda, no_foreach: 0.0067323700617998835
- doctr_reco_predictor, Adadelta, cuda, differentiable: 0.00797147342003882
- doctr_reco_predictor, Adadelta, cuda, foreach: 0.0014914630353450775
- doctr_reco_predictor, Adagrad, cuda, default: 0.0014977923524565995
- doctr_reco_predictor, Adagrad, cuda, maximize: 0.0014047086122445762
- doctr_reco_predictor, Adagrad, cuda, no_foreach: 0.0037795773101970552
- doctr_reco_predictor, Adagrad, cuda, differentiable: 0.0040812154300510885
- doctr_reco_predictor, Adagrad, cuda, foreach: 0.001298849075101316
- doctr_reco_predictor, Adam, cuda, default: 0.0014606264233589172
- doctr_reco_predictor, Adam, cuda, amsgrad, maximize: 0.0017359229386784136
- doctr_reco_predictor, Adam, cuda, no_foreach: 0.006150229680351913
- doctr_reco_predictor, Adam, cuda, differentiable: 0.013960111676715313
- doctr_reco_predictor, Adam, cuda, foreach: 0.0014205886446870862
- doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable: 0.002500549927353859
- doctr_reco_predictor, Adam, cuda, foreach, maximize, capturable, amsgrad: 0.0024232029216364028
- doctr_reco_predictor, Adam, cuda, fused: 0.0005264460835605859
- doctr_reco_predictor, Adam, cuda, fused, amsgrad, maximize: 0.0007339034359902144
- doctr_reco_predictor, Adam, cuda, fused, capturable: 0.000526783952023834
- doctr_reco_predictor, Adam, cuda, fused, capturable, amsgrad: 0.000733910609036684
- doctr_reco_predictor, AdamW, cuda, default: 0.0014952924964018166
- doctr_reco_predictor, AdamW, cuda, amsgrad, maximize: 0.0017838270077481866
- doctr_reco_predictor, AdamW, cuda, no_foreach: 0.006765100718475878
- doctr_reco_predictor, AdamW, cuda, differentiable: 0.012846773536875845
- doctr_reco_predictor, AdamW, cuda, foreach: 0.0015132253337651492
- doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable: 0.0023649467574432493
- doctr_reco_predictor, AdamW, cuda, foreach, maximize, capturable, amsgrad: 0.0025600428320467473
- doctr_reco_predictor, AdamW, cuda, fused: 0.0005436234101653099
- doctr_reco_predictor, AdamW, cuda, fused, amsgrad, maximize: 0.0007554181362502277
- doctr_reco_predictor, AdamW, cuda, fused, capturable: 0.0005439776740968227
- doctr_reco_predictor, AdamW, cuda, fused, capturable, amsgrad: 0.0007581162578426301
- doctr_reco_predictor, Adamax, cuda, default: 0.0063193910196423534
- doctr_reco_predictor, Adamax, cuda, maximize: 0.006546114273369312
- doctr_reco_predictor, Adamax, cuda, no_foreach: 0.008442916348576546
- doctr_reco_predictor, Adamax, cuda, differentiable: 0.010225607780739665
- doctr_reco_predictor, Adamax, cuda, foreach: 0.0063998927315697075
- doctr_reco_predictor, ASGD, cuda, default: 0.005111781670711935
- doctr_reco_predictor, ASGD, cuda, maximize: 0.005300870798528194
- doctr_reco_predictor, ASGD, cuda, no_foreach: 0.00970820013123254
- doctr_reco_predictor, ASGD, cuda, differentiable: 0.009756598388776183
- doctr_reco_predictor, ASGD, cuda, foreach: 0.005147161749191582
- doctr_reco_predictor, SGD, cuda, default: 0.00033027638401836155
- doctr_reco_predictor, SGD, cuda, maximize: 0.0005508650196716189
- doctr_reco_predictor, SGD, cuda, no_foreach: 0.000654176879208535
- doctr_reco_predictor, SGD, cuda, differentiable: 0.00033117017801851035
- doctr_reco_predictor, SGD, cuda, foreach: 0.00024379345728084444
- doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9, nesterov: 0.0006105483700521291
- doctr_reco_predictor, SGD, cuda, foreach, momentum=0.9: 0.0004410154549404979
- doctr_reco_predictor, RAdam, cuda, default: 0.0016630525072105229
- doctr_reco_predictor, RAdam, cuda, no_foreach: 0.009923019562847912
- doctr_reco_predictor, RAdam, cuda, differentiable: 0.010303066140040755
- doctr_reco_predictor, RAdam, cuda, foreach: 0.0016399111156351863
- doctr_reco_predictor, Rprop, cuda, default: 0.008071534298360349
- doctr_reco_predictor, Rprop, cuda, maximize: 0.008054946968331932
- doctr_reco_predictor, Rprop, cuda, no_foreach: 0.0125361071433872
- doctr_reco_predictor, Rprop, cuda, differentiable: 0.013898213882930577
- doctr_reco_predictor, Rprop, cuda, foreach: 0.007811435428448021
- doctr_reco_predictor, RMSprop, cuda, default: 0.0007729522828012705
- doctr_reco_predictor, RMSprop, cuda, maximize: 0.0009816498712946972
- doctr_reco_predictor, RMSprop, cuda, no_foreach: 0.0032276885583996775
- doctr_reco_predictor, RMSprop, cuda, differentiable: 0.0036167620681226255
- doctr_reco_predictor, RMSprop, cuda, foreach: 0.0006942034750245512
- doctr_reco_predictor, NAdam, cuda, default: 0.0020321781514212487
- doctr_reco_predictor, NAdam, cuda, no_foreach: 0.007569201388396323
- doctr_reco_predictor, NAdam, cuda, differentiable: 0.013424197630956768
- doctr_reco_predictor, NAdam, cuda, foreach: 0.0019183275150135158
- resnet18, ASGD, cuda, (pt2) default: 0.0017022616861356516
- resnet18, ASGD, cuda, (pt2) foreach: 0.0017011328324107295
- compile_time, resnet18, ASGD, cuda, (pt2) default: 17.65067977675547
- compile_time, resnet18, ASGD, cuda, (pt2) foreach: 16.845290329307318


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6128788174

cc @janeyx99


@mlazos moving the etas and mus and step to device doubles the runtime for eager on foreach and singletensor. I’ll probably open a PR to make it not the default