Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'b2f25d6342ed483b461e831c6f970ae59a4fcca2'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: df0c1b3a939b5d9be09eb28016dfec63e51204a6

Affected PyTorch commit: b2f25d6342ed483b461e831c6f970ae59a4fcca2

Affected Tests:
- test_train[detectron2_maskrcnn-cuda-eager]_latency: -10.14322%
- test_train[lennard_jones-cuda-eager]_latency: +10.07153%
- test_eval[mobilenet_v3_large-cuda-eager]_latency: +10.58355%
- test_eval[phlippe_densenet-cuda-eager]_latency: +11.78843%
- test_train[phlippe_resnet-cuda-eager]_latency: +16.48857%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: +13.30183%
- test_train[tts_angular-cuda-eager]_latency: +15.94473%
- test_eval[yolov3-cuda-eager]_latency: +14.09519%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/7007627094

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/7010668491

```
{
  "target_repo": "pytorch",
  "start": "d37c4c69954ad7bdccca96854105c48e93d4587e",
  "end": "624f2025229312bffd725775760dc9b147cc32d8",
  "result": [
    {
      "commit1": "d37c4c69954",
      "commit1_time": "2023-11-25 23:15:47 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "d37c4c69954ad7bdccca96854105c48e93d4587e",
          "pytorch_version": "2.2.0a0+gitd37c4c6",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "d37c4c69954ad7bdccca96854105c48e93d4587e"
        },
        "metrics": {
          "test_eval[yolov3-cuda-eager]_latency": 20.786368,
          "test_eval[yolov3-cuda-eager]_cmem": 0.9599609375,
          "test_eval[yolov3-cuda-eager]_gmem": 4.27752685546875
        }
      },
      "commit2": "bbdd9b059fa",
      "commit2_time": "2023-11-26 03:50:54 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "bbdd9b059fa06404701f0d39abaae75ebab5d27f",
          "pytorch_version": "2.2.0a0+gitbbdd9b0",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "bbdd9b059fa06404701f0d39abaae75ebab5d27f"
        },
        "metrics": {
          "test_eval[yolov3-cuda-eager]_latency": 18.344583,
          "test_eval[yolov3-cuda-eager]_cmem": 4.81640625,
          "test_eval[yolov3-cuda-eager]_gmem": 5.39276123046875
        }
      }
    },
    {
      "commit1": "bbdd9b059fa",
      "commit1_time": "2023-11-26 03:50:54 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "bbdd9b059fa06404701f0d39abaae75ebab5d27f",
          "pytorch_version": "2.2.0a0+gitbbdd9b0",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "bbdd9b059fa06404701f0d39abaae75ebab5d27f"
        },
        "metrics": {
          "test_eval[yolov3-cuda-eager]_latency": 18.344583,
          "test_eval[yolov3-cuda-eager]_cmem": 4.81640625,
          "test_eval[yolov3-cuda-eager]_gmem": 5.39276123046875
        }
      },
      "commit2": "028071c4a16",
      "commit2_time": "2023-11-26 09:25:41 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "028071c4a1609b822c328b9b4a6a6befd3588042",
          "pytorch_version": "2.2.0a0+git028071c",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "028071c4a1609b822c328b9b4a6a6befd3588042"
        },
        "metrics": {
          "test_eval[yolov3-cuda-eager]_latency": 19.247673,
          "test_eval[yolov3-cuda-eager]_cmem": 4.810546875,
          "test_eval[yolov3-cuda-eager]_gmem": 5.39276123046875
        }
      }
    },
    {
      "commit1": "028071c4a16",
      "commit1_time": "2023-11-26 09:25:41 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "028071c4a1609b822c328b9b4a6a6befd3588042",
          "pytorch_version": "2.2.0a0+git028071c",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "028071c4a1609b822c328b9b4a6a6befd3588042"
        },
        "metrics": {
          "test_eval[yolov3-cuda-eager]_latency": 19.247673,
          "test_eval[yolov3-cuda-eager]_cmem": 4.810546875,
          "test_eval[yolov3-cuda-eager]_gmem": 5.39276123046875
        }
      },
      "commit2": "4fa1ff8404b",
      "commit2_time": "2023-11-26 13:44:30 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "4fa1ff8404b6c26c076288aa2a0aa77f0c24916a",
          "pytorch_version": "2.2.0a0+git4fa1ff8",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "4fa1ff8404b6c26c076288aa2a0aa77f0c24916a"
        },
        "metrics": {
          "test_eval[yolov3-cuda-eager]_latency": 21.483247,
          "test_eval[yolov3-cuda-eager]_cmem": 4.8173828125,
          "test_eval[yolov3-cuda-eager]_gmem": 5.39276123046875
        }
      }
    },
    {
      "commit1": "081c5b3adc3",
      "commit1_time": "2023-11-26 23:40:32 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "081c5b3adc342d0e201df18977ffa4d939c5edad",
          "pytorch_version": "2.2.0a0+git081c5b3",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "081c5b3adc342d0e201df18977ffa4d939c5edad"
        },
        "metrics": {
          "test_eval[yolov3-cuda-eager]_latency": 21.100987,
          "test_eval[yolov3-cuda-eager]_cmem": 4.8173828125,
          "test_eval[yolov3-cuda-eager]_gmem": 5.44940185546875
        }
      },
      "commit2": "624f2025229",
      "commit2_time": "2023-11-27 02:40:28 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "624f2025229312bffd725775760dc9b147cc32d8",
          "pytorch_version": "2.2.0a0+git624f202",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "624f2025229312bffd725775760dc9b147cc32d8"
        },
        "metrics": {
          "test_eval[yolov3-cuda-eager]_latency": 18.717937,
          "test_eval[yolov3-cuda-eager]_cmem": 0.958984375,
          "test_eval[yolov3-cuda-eager]_gmem": 4.27752685546875
        }
      }
    }
  ]
}
```

Noisy benchmark