Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.5.0.dev20240824+cu124'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240606+cu121

Base PyTorch commit: ffaea656b5d8ff6518669494cc8f664b94f8e8b1

Affected PyTorch version: 2.5.0.dev20240824+cu124

Affected PyTorch commit: 2553278bae5993bd94bae4f04bf4586fb3f30d57

Affected Tests:
- test_train[Super_SloMo-cuda-eager]: +14.62530%
- test_eval[Super_SloMo-cuda-eager]: +11.06929%
- test_train[Super_SloMo-cuda-jit]: +14.75419%
- test_eval[Super_SloMo-cuda-jit]: +11.21729%
- test_eval[alexnet-cuda-eager]: +10.58900%
- test_eval[alexnet-cuda-jit]: +13.94897%
- test_train[dcgan-cuda-eager]: +63.66254%
- test_train[dcgan-cuda-jit]: +63.55263%
- test_eval[hf_T5-cpu-eager]: +9.57526%
- test_eval[pyhpc_equation_of_state-cpu-jit]: +7.55344%
- test_eval[pytorch_stargan-cpu-jit]: -7.74671%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +13.86265%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -27.60880%
- test_eval[timm_regnet-cuda-eager]: +170.47170%
- test_eval[timm_regnet-cuda-jit]: +356.90428%
- test_train[yolov3-cuda-eager]: -9.81750%


cc @xuzhao9

Result json: 
```
{
  "start": "ffaea656b5d8ff6518669494cc8f664b94f8e8b1",
  "end": "2553278bae5993bd94bae4f04bf4586fb3f30d57",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "6767e38267f",
      "commit1_time": "2024-06-14 00:12:09 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09083288144320249
      },
      "commit2": "9ebec1f3455",
      "commit2_time": "2024-06-14 00:12:58 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08305119965225458
      }
    },
    {
      "commit1": "45f28769348",
      "commit1_time": "2024-06-25 20:20:03 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07507259100675583
      },
      "commit2": "ae0f84d89c6",
      "commit2_time": "2024-06-25 20:34:18 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09291335977613926
      }
    },
    {
      "commit1": "9ebec1f3455",
      "commit1_time": "2024-06-14 00:12:58 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08305119965225458
      },
      "commit2": "18f5357f4f8",
      "commit2_time": "2024-06-14 00:31:22 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.0768194893375039
      }
    },
    {
      "commit1": "7cf454ec52e",
      "commit1_time": "2024-06-25 19:30:07 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08632602961733937
      },
      "commit2": "b1f486aff9f",
      "commit2_time": "2024-06-25 19:30:07 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07549587078392506
      }
    },
    {
      "commit1": "ae0f84d89c6",
      "commit1_time": "2024-06-25 20:34:18 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09291335977613926
      },
      "commit2": "79959d707cc",
      "commit2_time": "2024-06-25 20:54:14 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08177612768486142
      }
    },
    {
      "commit1": "e58ef5b65f4",
      "commit1_time": "2024-06-25 21:04:53 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08085977099835873
      },
      "commit2": "e1499f63420",
      "commit2_time": "2024-06-25 21:09:34 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09405456278473139
      }
    },
    {
      "commit1": "e1499f63420",
      "commit1_time": "2024-06-25 21:09:34 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09405456278473139
      },
      "commit2": "fd414d61892",
      "commit2_time": "2024-06-25 21:25:50 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07339736185967922
      }
    },
    {
      "commit1": "fd414d61892",
      "commit1_time": "2024-06-25 21:25:50 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07339736185967922
      },
      "commit2": "1c5df9107d2",
      "commit2_time": "2024-06-25 22:10:31 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.06779037117958069
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/10545990908


Noisy test pyhpc_equation_of_state, we should disable it