Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.4.0.dev20240331+cu121'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240330+cu121

Base PyTorch commit: f4ff063c333f286d4384523bac67c047aca4d7b9

Affected PyTorch version: 2.4.0.dev20240331+cu121

Affected PyTorch commit: 35c493f2cf9b623bfdc7e6b34dc1cb39690a7919

Affected Tests:
- test_eval[pyhpc_equation_of_state-cpu-jit]: +23.40365%
- test_eval[pytorch_stargan-cpu-eager]: -7.31655%
- test_eval[pytorch_stargan-cpu-jit]: -17.09158%
- test_train[timm_regnet-cpu-eager]: -11.03407%


cc @xuzhao9

Result json: 
```
{
  "start": "f4ff063c333f286d4384523bac67c047aca4d7b9",
  "end": "35c493f2cf9b623bfdc7e6b34dc1cb39690a7919",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "07f0ff6ed73",
      "commit1_time": "2024-03-30 07:28:41 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.1003414630657062
      },
      "commit2": "2cd3ef47779",
      "commit2_time": "2024-03-30 07:32:32 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07728302914183587
      }
    },
    {
      "commit1": "99f8f77de99",
      "commit1_time": "2024-03-30 07:34:41 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07987629220588133
      },
      "commit2": "eb7adc3ae08",
      "commit2_time": "2024-03-30 13:04:38 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08645859599346295
      }
    },
    {
      "commit1": "eb7adc3ae08",
      "commit1_time": "2024-03-30 13:04:38 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08645859599346295
      },
      "commit2": "b8550f527f0",
      "commit2_time": "2024-03-30 13:07:53 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.06774573716102168
      }
    },
    {
      "commit1": "5e878be1012",
      "commit1_time": "2024-03-30 14:20:54 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.0706779852276668
      },
      "commit2": "ec58f1f74eb",
      "commit2_time": "2024-03-30 16:39:57 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08439142653951422
      }
    },
    {
      "commit1": "3a9eead4ab0",
      "commit1_time": "2024-03-30 05:46:09 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07884371672989801
      },
      "commit2": "ed457c7dbe2",
      "commit2_time": "2024-03-30 06:47:15 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.06899793337797747
      }
    },
    {
      "commit1": "ed457c7dbe2",
      "commit1_time": "2024-03-30 06:47:15 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.06899793337797747
      },
      "commit2": "07f0ff6ed73",
      "commit2_time": "2024-03-30 07:28:41 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.1003414630657062
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8499757153


We should disable this test as it seems to be very noisy.