Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'fecde478ac83edf78e7d0e9d11ab73cb1580f6cf'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: d7520d8668dc08f7bed27a64f006c909006e653a

Affected PyTorch commit: fecde478ac83edf78e7d0e9d11ab73cb1580f6cf

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: +24.59110%
- test_train[lennard_jones-cuda-eager]_latency: +11.11600%
- test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]_latency: -38.85690%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6341420670

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/6342417468

```
{
  "target_repo": "pytorch",
  "start": "869226bf947bad0c4ba740c7b56ffe38acdddf52",
  "end": "75462fd87024bfe9fb79f4c46ab689bec227caa7",
  "result": [
    {
      "commit1": "869226bf947",
      "commit1_time": "2023-09-27 02:52:48 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "869226bf947bad0c4ba740c7b56ffe38acdddf52",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "869226bf947bad0c4ba740c7b56ffe38acdddf52"
        },
        "metrics": {
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 26.733557,
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 0.9580078125,
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.98065185546875
        }
      },
      "commit2": "33d8f5f73e7",
      "commit2_time": "2023-09-27 03:32:04 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "33d8f5f73e7343598a7a1259f66cbe54785ee309",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "33d8f5f73e7343598a7a1259f66cbe54785ee309"
        },
        "metrics": {
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 23.198113,
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 0.8876953125,
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.98065185546875
        }
      }
    },
    {
      "commit1": "33d8f5f73e7",
      "commit1_time": "2023-09-27 03:32:04 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "33d8f5f73e7343598a7a1259f66cbe54785ee309",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "33d8f5f73e7343598a7a1259f66cbe54785ee309"
        },
        "metrics": {
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 23.198113,
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 0.8876953125,
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.98065185546875
        }
      },
      "commit2": "f7c9ef88f5a",
      "commit2_time": "2023-09-27 04:07:58 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "f7c9ef88f5a021e835412c7bf7674919b5eb59de",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "f7c9ef88f5a021e835412c7bf7674919b5eb59de"
        },
        "metrics": {
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency": 23.640759,
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem": 0.9599609375,
          "test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_gmem": 2.98065185546875
        }
      }
    }
  ]
}
```