Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.4.0.dev20240509+cu121'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 48b6c8dbc376db4406a979b35cd6909bcb428931
Control PyTorch version: 2.4.0.dev20240507+cu121

Treatment PyTorch commit: ff090c693751f564f45951f1b13c11478892799d
Treatment PyTorch version: 2.4.0.dev20240509+cu121

Bisection result:

```
[
    {
        "commit1": "8b4d62009dd",
        "commit1_time": "2024-05-07 23:40:54 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "8b4d62009ddbc24a69dfcdbebc2cc84e4b2ee8f5",
                "pytorch_version": "2.4.0a0+git8b4d620",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "8b4d62009ddbc24a69dfcdbebc2cc84e4b2ee8f5"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 0.966324,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.4501953125,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.54608154296875
            }
        },
        "commit2": "244d93039d5",
        "commit2_time": "2024-05-07 23:48:20 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "244d93039d516ebd86c79027973ef5216051ddf8",
                "pytorch_version": "2.4.0a0+git244d930",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "244d93039d516ebd86c79027973ef5216051ddf8"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.151606,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.4501953125,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.54608154296875
            }
        }
    },
    {
        "commit1": "a3d97f6ce44",
        "commit1_time": "2024-05-08 01:10:05 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "a3d97f6ce4492d637c757fb13a122fe7470c0beb",
                "pytorch_version": "2.4.0a0+gita3d97f6",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "a3d97f6ce4492d637c757fb13a122fe7470c0beb"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.159181,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.4501953125,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.54608154296875
            }
        },
        "commit2": "21aaac47e76",
        "commit2_time": "2024-05-08 01:14:23 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "21aaac47e7668b4791c29d1bfe909f74272376a8",
                "pytorch_version": "2.4.0a0+git21aaac4",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "21aaac47e7668b4791c29d1bfe909f74272376a8"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 0.958181,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.4501953125,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.54608154296875
            }
        }
    },
    {
        "commit1": "9fedf41b601",
        "commit1_time": "2024-05-08 01:52:56 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "9fedf41b601e566ef0dcdd590c49bfaa4596a43b",
                "pytorch_version": "2.4.0a0+git9fedf41",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "9fedf41b601e566ef0dcdd590c49bfaa4596a43b"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 0.989529,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.4501953125,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.54608154296875
            }
        },
        "commit2": "2deea9e6e9f",
        "commit2_time": "2024-05-08 02:34:31 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "2deea9e6e9faf5eacebefa2336861d129c598c99",
                "pytorch_version": "2.4.0a0+git2deea9e",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "2deea9e6e9faf5eacebefa2336861d129c598c99"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.13423,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.5400390625,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.54608154296875
            }
        }
    },
    {
        "commit1": "2deea9e6e9f",
        "commit1_time": "2024-05-08 02:34:31 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "2deea9e6e9faf5eacebefa2336861d129c598c99",
                "pytorch_version": "2.4.0a0+git2deea9e",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "2deea9e6e9faf5eacebefa2336861d129c598c99"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.13423,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.5400390625,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.54608154296875
            }
        },
        "commit2": "38baa02a40c",
        "commit2_time": "2024-05-08 03:11:22 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "38baa02a40cf3638e4460be52f42ae909154b514",
                "pytorch_version": "2.4.0a0+git38baa02",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "38baa02a40cf3638e4460be52f42ae909154b514"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 0.987436,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.5400390625,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.54608154296875
            }
        }
    }
]
```

cc @xuzhao9


Noise, closed.