Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '61239df555df02e8c60a2ad63363878a2a57c161'
Root cause: Resource Mishandling

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 329bb2a33e40f4bc76b2e061b180d3234984c91b

Affected PyTorch commit: 61239df555df02e8c60a2ad63363878a2a57c161

Affected Tests:
- test_eval[BERT_pytorch-cuda-eager]_cmem: -79.07180%
- test_train[Background_Matting-cuda-eager]_cmem: -67.26876%
- test_train[DALLE2_pytorch-cuda-eager]_latency: -7.44915%
- test_train[DALLE2_pytorch-cuda-eager]_cmem: -56.89486%
- test_train[DALLE2_pytorch-cuda-eager]_gmem: -11.56869%
- test_eval[DALLE2_pytorch-cuda-eager]_latency: -13.95248%
- test_eval[DALLE2_pytorch-cuda-eager]_cmem: -58.15871%
- test_eval[DALLE2_pytorch-cuda-eager]_gmem: -11.62170%
- test_eval[LearningToPaint-cuda-eager]_cmem: -63.77610%
- test_eval[LearningToPaint-cuda-eager]_gmem: -11.03741%
- test_train[Super_SloMo-cuda-eager]_latency: +8.51985%
- test_train[Super_SloMo-cuda-eager]_cmem: -75.84077%
- test_eval[Super_SloMo-cuda-eager]_cmem: -78.67384%
- test_train[alexnet-cuda-eager]_cmem: -76.99482%
- test_train[alexnet-cuda-eager]_gmem: -9.80054%
- test_eval[alexnet-cuda-eager]_cmem: -78.62267%
- test_eval[alexnet-cuda-eager]_gmem: -8.34575%
- test_train[attention_is_all_you_need_pytorch-cuda-eager]_cmem: -63.81307%
- test_eval[attention_is_all_you_need_pytorch-cuda-eager]_cmem: -65.66542%
- test_train[dcgan-cuda-eager]_cmem: -77.31749%
- test_train[dcgan-cuda-eager]_gmem: -18.10467%
- test_eval[dcgan-cuda-eager]_latency: +181.87949%
- test_eval[dcgan-cuda-eager]_cmem: -79.86553%
- test_eval[dcgan-cuda-eager]_gmem: +31.45016%
- test_eval[demucs-cuda-eager]_cmem: -79.85989%
- test_train[densenet121-cuda-eager]_cmem: -75.79231%
- test_train[detectron2_fasterrcnn_r_101_c4-cuda-eager]_cmem: -65.92032%
- test_train[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_latency: -10.28579%
- test_train[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_cmem: -68.62023%
- test_eval[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_latency: +21.19994%
- test_eval[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_cmem: -66.60876%
- test_train[detectron2_fasterrcnn_r_101_fpn-cuda-eager]_latency: -14.97605%
- test_train[detectron2_fasterrcnn_r_101_fpn-cuda-eager]_cmem: -67.57576%
- test_train[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem: -70.38153%
- test_train[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem: -7.01524%
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: -11.15633%
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem: -66.76027%
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem: -10.75968%
- test_train[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_latency: -12.47477%
- test_train[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_cmem: -68.92484%
- test_train[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_gmem: -7.50503%
- test_eval[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_cmem: -66.66667%
- test_eval[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_gmem: -7.91105%
- test_train[detectron2_maskrcnn-cuda-eager]_latency: -14.67654%
- test_train[detectron2_maskrcnn-cuda-eager]_cmem: -62.29723%
- test_train[detectron2_maskrcnn-cuda-eager]_gmem: -9.68350%
- test_eval[detectron2_maskrcnn-cuda-eager]_cmem: -62.82095%
- test_train[detectron2_maskrcnn_r_101_c4-cuda-eager]_cmem: -66.39839%
- test_eval[detectron2_maskrcnn_r_101_c4-cuda-eager]_cmem: -64.04770%
- test_train[detectron2_maskrcnn_r_101_fpn-cuda-eager]_latency: -13.24208%
- test_train[detectron2_maskrcnn_r_101_fpn-cuda-eager]_cmem: -65.94886%
- test_eval[detectron2_maskrcnn_r_101_fpn-cuda-eager]_cmem: -66.15385%
- test_eval[detectron2_maskrcnn_r_101_fpn-cuda-eager]_gmem: -8.16754%
- test_train[detectron2_maskrcnn_r_50_c4-cuda-eager]_cmem: -68.80214%
- test_train[detectron2_maskrcnn_r_50_c4-cuda-eager]_gmem: +14.13978%
- test_eval[detectron2_maskrcnn_r_50_fpn-cuda-eager]_cmem: -66.09457%
- test_eval[detectron2_maskrcnn_r_50_fpn-cuda-eager]_gmem: -8.23306%
- test_train[dlrm-cuda-eager]_cmem: -74.72955%
- test_train[dlrm-cuda-eager]_gmem: -9.01917%
- test_eval[dlrm-cuda-eager]_cmem: -80.15700%
- test_eval[dlrm-cuda-eager]_gmem: -10.24853%
- test_eval[doctr_det_predictor-cuda-eager]_cmem: -76.02599%
- test_eval[doctr_det_predictor-cuda-eager]_gmem: -10.08361%
- test_eval[doctr_reco_predictor-cuda-eager]_cmem: -76.69676%
- test_eval[doctr_reco_predictor-cuda-eager]_gmem: -17.20154%
- test_train[drq-cuda-eager]_cmem: -68.11406%
- test_train[drq-cuda-eager]_gmem: -20.50663%
- test_train[fambench_xlmr-cuda-eager]_cmem: -52.80581%
- test_eval[fambench_xlmr-cuda-eager]_cmem: -53.77049%
- test_eval[fambench_xlmr-cuda-eager]_gmem: -10.69658%
- test_train[fastNLP_Bert-cuda-eager]_cmem: -73.56674%
- test_eval[fastNLP_Bert-cuda-eager]_cmem: -76.90957%
- test_eval[fastNLP_Bert-cuda-eager]_gmem: -11.95815%
- test_eval[functorch_dp_cifar10-cuda-eager]_cmem: -80.75117%
- test_eval[functorch_dp_cifar10-cuda-eager]_gmem: -17.33853%
- test_train[functorch_maml_omniglot-cuda-eager]_cmem: -74.93169%
- test_train[hf_Albert-cuda-eager]_cmem: -74.15438%
- test_eval[hf_Albert-cuda-eager]_cmem: -75.32580%
- test_eval[hf_Albert-cuda-eager]_gmem: -14.52267%
- test_train[hf_Bart-cuda-eager]_cmem: -70.77586%
- test_eval[hf_Bart-cuda-eager]_cmem: -70.38478%
- test_eval[hf_Bart-cuda-eager]_gmem: -11.87105%
- test_train[hf_Bert-cuda-eager]_cmem: -72.84970%
- test_eval[hf_Bert-cuda-eager]_cmem: -72.67593%
- test_eval[hf_Bert-cuda-eager]_gmem: -14.94122%
- test_train[hf_Bert_large-cuda-eager]_cmem: -75.94991%
- test_eval[hf_Bert_large-cuda-eager]_cmem: -75.56424%
- test_eval[hf_Bert_large-cuda-eager]_gmem: -12.85347%
- test_train[hf_BigBird-cuda-eager]_cmem: -71.05832%
- test_eval[hf_BigBird-cuda-eager]_cmem: -70.32538%
- test_eval[hf_BigBird-cuda-eager]_gmem: -7.32026%
- test_train[hf_DistilBert-cuda-eager]_cmem: -73.93755%
- test_eval[hf_DistilBert-cuda-eager]_cmem: -74.83759%
- test_eval[hf_DistilBert-cuda-eager]_gmem: -14.08056%
- test_train[hf_GPT2-cuda-eager]_cmem: -74.87047%
- test_eval[hf_GPT2-cuda-eager]_cmem: -74.82669%
- test_train[hf_GPT2_large-cuda-eager]_cmem: -73.65011%
- test_eval[hf_GPT2_large-cuda-eager]_cmem: -73.85082%
- test_eval[hf_GPT2_large-cuda-eager]_gmem: -9.90199%
- test_train[hf_Longformer-cuda-eager]_cmem: -60.92601%
- test_eval[hf_Longformer-cuda-eager]_cmem: -62.76550%
- test_train[hf_Reformer-cuda-eager]_cmem: -70.63903%
- test_train[hf_Reformer-cuda-eager]_gmem: -7.76850%
- test_eval[hf_Reformer-cuda-eager]_cmem: -72.38838%
- test_eval[hf_Reformer-cuda-eager]_gmem: -14.52390%
- test_train[hf_T5-cuda-eager]_cmem: -70.27610%
- test_eval[hf_T5-cuda-eager]_cmem: -72.41977%
- test_eval[hf_T5-cuda-eager]_gmem: -8.87587%
- test_eval[hf_T5_base-cuda-eager]_latency: -7.29486%
- test_eval[hf_T5_base-cuda-eager]_cmem: -61.04853%
- test_eval[hf_T5_base-cuda-eager]_gmem: -12.75603%
- test_train[hf_T5_large-cuda-eager]_cmem: -71.68028%
- test_eval[hf_T5_large-cuda-eager]_latency: -25.75441%
- test_eval[hf_T5_large-cuda-eager]_cmem: -73.26690%
- test_eval[hf_T5_large-cuda-eager]_gmem: -10.55950%
- test_train[lennard_jones-cuda-eager]_cmem: -77.28693%
- test_train[lennard_jones-cuda-eager]_gmem: -20.60793%
- test_eval[llama-cuda-eager]_latency: -15.70181%
- test_eval[llama-cuda-eager]_cmem: -79.03872%
- test_eval[llama-cuda-eager]_gmem: -13.03356%
- test_eval[maml-cuda-eager]_cmem: -78.31449%
- test_eval[maml-cuda-eager]_gmem: -13.00550%
- test_train[maml_omniglot-cuda-eager]_cmem: -77.41604%
- test_train[maml_omniglot-cuda-eager]_gmem: -19.31764%
- test_eval[maml_omniglot-cuda-eager]_cmem: -82.12983%
- test_eval[maml_omniglot-cuda-eager]_gmem: -20.56555%
- test_train[mnasnet1_0-cuda-eager]_cmem: -76.03250%
- test_eval[mnasnet1_0-cuda-eager]_cmem: -74.79413%
- test_train[mobilenet_v2-cuda-eager]_cmem: -76.15332%
- test_eval[mobilenet_v2-cuda-eager]_cmem: -74.62419%
- test_train[mobilenet_v2_quantized_qat-cuda-eager]_cmem: -75.33738%
- test_train[mobilenet_v3_large-cuda-eager]_latency: -14.70865%
- test_train[mobilenet_v3_large-cuda-eager]_cmem: -74.81381%
- test_train[mobilenet_v3_large-cuda-eager]_gmem: -7.74087%
- test_eval[mobilenet_v3_large-cuda-eager]_cmem: -73.37384%
- test_eval[mobilenet_v3_large-cuda-eager]_gmem: -10.25961%
- test_train[moco-cuda-eager]_cmem: -66.20186%
- test_train[moco-cuda-eager]_gmem: -8.00519%
- test_eval[moco-cuda-eager]_cmem: -70.67591%
- test_train[nvidia_deeprecommender-cuda-eager]_cmem: -78.27022%
- test_train[nvidia_deeprecommender-cuda-eager]_gmem: -7.02864%
- test_eval[nvidia_deeprecommender-cuda-eager]_cmem: -81.41248%
- test_eval[nvidia_deeprecommender-cuda-eager]_gmem: -10.30131%
- test_train[opacus_cifar10-cuda-eager]_latency: -7.69928%
- test_train[opacus_cifar10-cuda-eager]_cmem: -74.77906%
- test_train[opacus_cifar10-cuda-eager]_gmem: -7.69675%
- test_eval[opacus_cifar10-cuda-eager]_cmem: -79.46714%
- test_eval[opacus_cifar10-cuda-eager]_gmem: -17.33853%
- test_train[phlippe_densenet-cuda-eager]_latency: -13.33246%
- test_train[phlippe_densenet-cuda-eager]_cmem: -77.49484%
- test_eval[phlippe_densenet-cuda-eager]_latency: -15.19264%
- test_eval[phlippe_densenet-cuda-eager]_cmem: -80.84180%
- test_train[phlippe_resnet-cuda-eager]_cmem: -77.50770%
- test_train[phlippe_resnet-cuda-eager]_gmem: -18.45870%
- test_eval[phlippe_resnet-cuda-eager]_cmem: -81.53005%
- test_eval[phlippe_resnet-cuda-eager]_gmem: -17.95757%
- test_eval[pyhpc_equation_of_state-cuda-eager]_cmem: -69.39935%
- test_eval[pyhpc_isoneutral_mixing-cuda-eager]_cmem: -68.42105%
- test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]_cmem: -67.77434%
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_cmem: -74.31224%
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_gmem: -9.25154%
- test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_cmem: -78.79553%
- test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_gmem: -16.64586%
- test_train[pytorch_stargan-cuda-eager]_cmem: -73.43486%
- test_train[pytorch_stargan-cuda-eager]_gmem: -7.10317%
- test_eval[pytorch_stargan-cuda-eager]_cmem: -79.69142%
- test_eval[pytorch_stargan-cuda-eager]_gmem: -10.64712%
- test_train[pytorch_struct-cuda-eager]_cmem: -70.82105%
- test_train[pytorch_unet-cuda-eager]_cmem: -75.67291%
- test_train[pytorch_unet-cuda-eager]_gmem: -9.05466%
- test_eval[pytorch_unet-cuda-eager]_cmem: -78.65169%
- test_eval[pytorch_unet-cuda-eager]_gmem: -7.19424%
- test_train[resnet152-cuda-eager]_latency: -7.65806%
- test_train[resnet152-cuda-eager]_cmem: -74.62032%
- test_eval[resnet152-cuda-eager]_cmem: -76.59879%
- test_train[resnet18-cuda-eager]_cmem: -75.76271%
- test_train[resnet18-cuda-eager]_gmem: -15.23739%
- test_eval[resnet18-cuda-eager]_cmem: -77.84560%
- test_eval[resnet18-cuda-eager]_gmem: -7.96442%
- test_train[resnet50-cuda-eager]_cmem: -74.70409%
- test_eval[resnet50-cuda-eager]_cmem: -77.13262%
- test_eval[resnet50-cuda-eager]_gmem: -8.27052%
- test_train[resnet50_quantized_qat-cuda-eager]_cmem: -75.21136%
- test_eval[resnext50_32x4d-cuda-eager]_cmem: -75.99284%
- test_train[shufflenet_v2_x1_0-cuda-eager]_cmem: -75.04233%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_cmem: -75.75215%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_gmem: -11.84330%
- test_eval[soft_actor_critic-cuda-eager]_cmem: -80.93146%
- test_eval[soft_actor_critic-cuda-eager]_gmem: -20.48131%
- test_train[speech_transformer-cuda-eager]_latency: -10.35698%
- test_train[speech_transformer-cuda-eager]_cmem: -71.20674%
- test_train[speech_transformer-cuda-eager]_gmem: -11.83709%
- test_eval[speech_transformer-cuda-eager]_cmem: -72.98137%
- test_eval[speech_transformer-cuda-eager]_gmem: -18.77053%
- test_train[squeezenet1_1-cuda-eager]_cmem: -75.96286%
- test_train[squeezenet1_1-cuda-eager]_gmem: -9.81912%
- test_eval[squeezenet1_1-cuda-eager]_cmem: -77.74587%
- test_eval[squeezenet1_1-cuda-eager]_gmem: -7.81967%
- test_train[tacotron2-cuda-eager]_latency: -22.71893%
- test_train[tacotron2-cuda-eager]_cmem: -70.53301%
- test_eval[tacotron2-cuda-eager]_latency: -19.49046%
- test_eval[tacotron2-cuda-eager]_cmem: -71.71419%
- test_train[timm_efficientdet-cuda-eager]_cmem: -73.01587%
- test_train[timm_efficientnet-cuda-eager]_cmem: -73.34890%
- test_eval[timm_efficientnet-cuda-eager]_cmem: -73.48887%
- test_eval[timm_efficientnet-cuda-eager]_gmem: -12.56848%
- test_train[timm_nfnet-cuda-eager]_cmem: -72.90257%
- test_eval[timm_nfnet-cuda-eager]_cmem: -75.22936%
- test_eval[timm_nfnet-cuda-eager]_gmem: -11.93578%
- test_train[timm_regnet-cuda-eager]_cmem: -72.84812%
- test_eval[timm_regnet-cuda-eager]_cmem: -73.27465%
- test_eval[timm_regnet-cuda-eager]_gmem: -9.60059%
- test_train[timm_resnest-cuda-eager]_cmem: -74.37521%
- test_eval[timm_resnest-cuda-eager]_cmem: -73.94068%
- test_eval[timm_resnest-cuda-eager]_gmem: -20.32745%
- test_train[timm_vision_transformer-cuda-eager]_cmem: -77.78535%
- test_train[timm_vision_transformer-cuda-eager]_gmem: -8.42290%
- test_eval[timm_vision_transformer-cuda-eager]_cmem: -77.23894%
- test_eval[timm_vision_transformer-cuda-eager]_gmem: -16.03582%
- test_eval[timm_vision_transformer_large-cuda-eager]_cmem: -78.22552%
- test_eval[timm_vision_transformer_large-cuda-eager]_gmem: -8.19912%
- test_train[timm_vovnet-cuda-eager]_cmem: -73.90724%
- test_train[timm_vovnet-cuda-eager]_gmem: -11.99780%
- test_eval[timm_vovnet-cuda-eager]_cmem: -76.26939%
- test_eval[timm_vovnet-cuda-eager]_gmem: -10.22580%
- test_train[torchrec_dlrm-cuda-eager]_latency: -9.08342%
- test_train[torchrec_dlrm-cuda-eager]_cmem: -83.97155%
- test_train[torchrec_dlrm-cuda-eager]_gmem: -9.26657%
- test_eval[torchrec_dlrm-cuda-eager]_cmem: -86.31752%
- test_eval[torchrec_dlrm-cuda-eager]_gmem: -9.55428%
- test_train[tts_angular-cuda-eager]_latency: -12.98617%
- test_train[tts_angular-cuda-eager]_cmem: -78.47903%
- test_train[tts_angular-cuda-eager]_gmem: -16.48216%
- test_eval[tts_angular-cuda-eager]_cmem: -82.92282%
- test_eval[tts_angular-cuda-eager]_gmem: -13.14492%
- test_train[vgg16-cuda-eager]_cmem: -77.14385%
- test_eval[vgg16-cuda-eager]_cmem: -78.51772%
- test_eval[vgg16-cuda-eager]_gmem: -15.95376%
- test_train[vision_maskrcnn-cuda-eager]_cmem: -69.46229%
- test_eval[vision_maskrcnn-cuda-eager]_cmem: -71.50977%
- test_train[yolov3-cuda-eager]_cmem: -70.15415%
- test_eval[yolov3-cuda-eager]_cmem: -73.52735%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5026119700

cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/5212009491