Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.5.0.dev20240730+cu124'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 28fd2e905da8484de38faf4e04073e896f7a95f7
Control PyTorch version: 2.5.0.dev20240727+cu124

Treatment PyTorch commit: 05a8540041cea936a63355c2e38b7b3beb5ce168
Treatment PyTorch version: 2.5.0.dev20240730+cu124

Bisection result:

```
[
    {
        "commit1": "dfa18bf3f3",
        "commit1_time": "2024-07-29 07:40:42 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "dfa18bf3f39c5a90b48baf956e50fa7da4462d3d",
                "pytorch_version": "2.5.0a0+gitdfa18bf",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "dfa18bf3f39c5a90b48baf956e50fa7da4462d3d"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 51.054556,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.9765625,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        },
        "commit2": "d47c470f47",
        "commit2_time": "2024-07-29 08:29:59 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "d47c470f474b1b4cba1c4388261effc3da46e4a3",
                "pytorch_version": "2.5.0a0+gitd47c470",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "d47c470f474b1b4cba1c4388261effc3da46e4a3"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 59.237219,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.99609375,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        }
    },
    {
        "commit1": "d47c470f47",
        "commit1_time": "2024-07-29 08:29:59 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "d47c470f474b1b4cba1c4388261effc3da46e4a3",
                "pytorch_version": "2.5.0a0+gitd47c470",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "d47c470f474b1b4cba1c4388261effc3da46e4a3"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 59.237219,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.99609375,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        },
        "commit2": "b67811abda",
        "commit2_time": "2024-07-29 12:37:56 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "b67811abda6617b8416f172e0e033ff6efaa8506",
                "pytorch_version": "2.5.0a0+gitb67811a",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "b67811abda6617b8416f172e0e033ff6efaa8506"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 50.278016,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 1.052734375,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        }
    },
    {
        "commit1": "b67811abda",
        "commit1_time": "2024-07-29 12:37:56 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "b67811abda6617b8416f172e0e033ff6efaa8506",
                "pytorch_version": "2.5.0a0+gitb67811a",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "b67811abda6617b8416f172e0e033ff6efaa8506"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 50.278016,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 1.052734375,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        },
        "commit2": "9d497887b8",
        "commit2_time": "2024-07-29 12:38:23 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "9d497887b872a63f1cb84fc0d76ffe4091a78979",
                "pytorch_version": "2.5.0a0+git9d49788",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "9d497887b872a63f1cb84fc0d76ffe4091a78979"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 49.284458,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.9658203125,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        }
    },
    {
        "commit1": "9d497887b8",
        "commit1_time": "2024-07-29 12:38:23 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "9d497887b872a63f1cb84fc0d76ffe4091a78979",
                "pytorch_version": "2.5.0a0+git9d49788",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "9d497887b872a63f1cb84fc0d76ffe4091a78979"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 49.284458,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 0.9658203125,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        },
        "commit2": "eb9409511e",
        "commit2_time": "2024-07-29 12:40:00 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "eb9409511e1c6ae0c5bf37121f500d91a34d6102",
                "pytorch_version": "2.5.0a0+giteb94095",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "eb9409511e1c6ae0c5bf37121f500d91a34d6102"
            },
            "metrics": {
                "test_eval[doctr_det_predictor-cuda-eager]_latency": 59.254005,
                "test_eval[doctr_det_predictor-cuda-eager]_cmem": 1.005859375,
                "test_eval[doctr_det_predictor-cuda-eager]_gmem": 3.49334716796875
            }
        }
    }
]
```

cc @xuzhao9


This test is noisy, closed.