Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '88eb6bbb1ab58d7cdb49349b64c02a04911be8f2'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: f468e74875f6b7f95b7b01ccf3b05c3917e2865d

Affected PyTorch commit: 88eb6bbb1ab58d7cdb49349b64c02a04911be8f2

Affected Tests:
- test_train[dcgan-cuda-eager]_latency: +14.06185%
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: +27.42408%
- test_train[lennard_jones-cuda-eager]_latency: +24.20559%
- test_eval[opacus_cifar10-cuda-eager]_latency: +13.65104%
- test_eval[phlippe_densenet-cuda-eager]_latency: +10.80079%
- test_train[resnet18-cuda-eager]_latency: +10.74420%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: +12.33621%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6615910251

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/6728007090

```
{
  "target_repo": "pytorch",
  "start": "bf01a7b023c1ce4deb6316d25eadc1791807e88d",
  "end": "c65c0682b19ed479ea66653fc0457958a3e2c59c",
  "result": [
    {
      "commit1": "bf01a7b023c",
      "commit1_time": "2023-10-22 04:35:15 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "bf01a7b023c1ce4deb6316d25eadc1791807e88d",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "bf01a7b023c1ce4deb6316d25eadc1791807e88d"
        },
        "metrics": {
          "test_eval[phlippe_densenet-cuda-eager]_latency": 7.673357,
          "test_eval[phlippe_densenet-cuda-eager]_cmem": 0.5869140625,
          "test_eval[phlippe_densenet-cuda-eager]_gmem": 2.50213623046875,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_latency": 8.732462,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_cmem": 0.7216796875,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_gmem": 1.92987060546875,
          "test_train[dcgan-cuda-eager]_latency": 42.947154,
          "test_train[dcgan-cuda-eager]_cmem": 0.712890625,
          "test_train[dcgan-cuda-eager]_gmem": 1.72869873046875
        }
      },
      "commit2": "57c7aa12dbf",
      "commit2_time": "2023-10-22 06:10:09 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "57c7aa12dbf71617bd21fe7e076df8e823b5b7bb",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "57c7aa12dbf71617bd21fe7e076df8e823b5b7bb"
        },
        "metrics": {
          "test_eval[phlippe_densenet-cuda-eager]_latency": 8.713159,
          "test_eval[phlippe_densenet-cuda-eager]_cmem": 0.58984375,
          "test_eval[phlippe_densenet-cuda-eager]_gmem": 2.50213623046875,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_latency": 9.914182,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_cmem": 0.7294921875,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_gmem": 1.92987060546875,
          "test_train[dcgan-cuda-eager]_latency": 46.003801,
          "test_train[dcgan-cuda-eager]_cmem": 0.712890625,
          "test_train[dcgan-cuda-eager]_gmem": 1.72869873046875
        }
      }
    },
    {
      "commit1": "57c7aa12dbf",
      "commit1_time": "2023-10-22 06:10:09 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "57c7aa12dbf71617bd21fe7e076df8e823b5b7bb",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "57c7aa12dbf71617bd21fe7e076df8e823b5b7bb"
        },
        "metrics": {
          "test_eval[phlippe_densenet-cuda-eager]_latency": 8.713159,
          "test_eval[phlippe_densenet-cuda-eager]_cmem": 0.58984375,
          "test_eval[phlippe_densenet-cuda-eager]_gmem": 2.50213623046875,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_latency": 9.914182,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_cmem": 0.7294921875,
          "test_eval[shufflenet_v2_x1_0-cuda-eager]_gmem": 1.92987060546875,
          "test_train[dcgan-cuda-eager]_latency": 46.003801,
          "test_train[dcgan-cuda-eager]_cmem": 0.712890625,
          "test_train[dcgan-cuda-eager]_gmem": 1.72869873046875
        }
      },
      "commit2": "8e60d646b93",
      "commit2_time": "2023-10-22 13:22:58 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "8e60d646b93b96bad8f069905c4bdba67ff1317a",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "8e60d646b93b96bad8f069905c4bdba67ff1317a"
        },
        "metrics": {}
      }
    },
    {
      "commit1": "8e60d646b93",
      "commit1_time": "2023-10-22 13:22:58 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "8e60d646b93b96bad8f069905c4bdba67ff1317a",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "8e60d646b93b96bad8f069905c4bdba67ff1317a"
        },
        "metrics": {}
      },
      "commit2": "e2e1189f416",
      "commit2_time": "2023-10-22 15:16:21 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "e2e1189f416b235a17e1e053c13f8644606a43a9",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "e2e1189f416b235a17e1e053c13f8644606a43a9"
        },
        "metrics": {}
      }
    }
  ]
}
```

https://github.com/pytorch/pytorch/commit/57c7aa12dbf
https://github.com/pytorch/pytorch/commit/8e60d646b93
https://github.com/pytorch/pytorch/commit/e2e1189f416