zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.3.0.dev20240215+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.3.0.dev20240214+cu121

Base PyTorch commit: 16369816a2e1c298fe5d15a439e922d10eb1894b

Affected PyTorch version: 2.3.0.dev20240215+cu121

Affected PyTorch commit: cd08dc37f831f7b7deabdc92893758de84c4d3d9

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +9.70669%
- test_train[timm_nfnet-cpu-jit]: +8.59857%


cc @xuzhao9

Result json: 
```
{
  "start": "16369816a2e1c298fe5d15a439e922d10eb1894b",
  "end": "cd08dc37f831f7b7deabdc92893758de84c4d3d9",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "3f09c5ee660",
      "commit1_time": "2024-02-14 05:09:59 +0000",
      "commit1_digest": {
        "test_train[timm_nfnet-cpu-jit]": 13.287690505897626
      },
      "commit2": "5ffac768f69",
      "commit2_time": "2024-02-14 05:09:59 +0000",
      "commit2_digest": {
        "test_train[timm_nfnet-cpu-jit]": 10.769222106598317
      }
    },
    {
      "commit1": "99cb807e25f",
      "commit1_time": "2024-02-15 00:10:05 +0000",
      "commit1_digest": {
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.16090625864453614,
        "test_train[timm_nfnet-cpu-jit]": 10.735739358142018
      },
      "commit2": "daf1050ae56",
      "commit2_time": "2024-02-15 00:35:56 +0000",
      "commit2_digest": {
        "test_train[timm_nfnet-cpu-jit]": 12.197181356372312
      }
    },
    {
      "commit1": "9ff9798716f",
      "commit1_time": "2024-02-15 02:49:11 +0000",
      "commit1_digest": {
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.15107986186631023
      },
      "commit2": "5f9b4324944",
      "commit2_time": "2024-02-15 02:56:34 +0000",
      "commit2_digest": {
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.16202701050788165
      }
    },
    {
      "commit1": "5f9b4324944",
      "commit1_time": "2024-02-15 02:56:34 +0000",
      "commit1_digest": {
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.16202701050788165
      },
      "commit2": "cd08dc37f83",
      "commit2_time": "2024-02-15 03:33:49 +0000",
      "commit2_digest": {
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.13968734657391907,
        "test_train[timm_nfnet-cpu-jit]": 13.069052706472576
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/7919750544


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.3.0.dev20240216+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.3.0.dev20240215+cu121

Base PyTorch commit: cd08dc37f831f7b7deabdc92893758de84c4d3d9

Affected PyTorch version: 2.3.0.dev20240216+cu121

Affected PyTorch commit: 62e5840b36b6790a5008faa76a920c6d154897b9

Affected Tests:
- test_eval[pytorch_stargan-cpu-jit]: +9.41706%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: -10.98939%
- test_train[tts_angular-cpu-eager]: +11.18676%


cc @xuzhao9

Result json: 
```
{
  "start": "cd08dc37f831f7b7deabdc92893758de84c4d3d9",
  "end": "62e5840b36b6790a5008faa76a920c6d154897b9",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "cd08dc37f83",
      "commit1_time": "2024-02-15 03:33:49 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2492261623032391,
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.15827200869098307,
        "test_train[tts_angular-cpu-eager]": 0.2234866420738399
      },
      "commit2": "af346df6a0f",
      "commit2_time": "2024-02-15 03:59:44 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.29972179257310927
      }
    },
    {
      "commit1": "0898ead2d55",
      "commit1_time": "2024-02-15 04:48:40 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.3020823733415455
      },
      "commit2": "68328ad3940",
      "commit2_time": "2024-02-15 06:28:17 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.27401044252328577
      }
    },
    {
      "commit1": "c83af673bcc",
      "commit1_time": "2024-02-15 07:03:59 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.27152388421818613
      },
      "commit2": "f4d641ba2fb",
      "commit2_time": "2024-02-15 07:13:42 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.3001837338786572
      }
    },
    {
      "commit1": "4319735ace2",
      "commit1_time": "2024-02-16 00:23:23 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.3020825733430684
      },
      "commit2": "c802c501960",
      "commit2_time": "2024-02-16 00:33:18 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2735131626017392
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/7933961116


Noise, closed.
Repository: pytorch/benchmark
Issue: WSL machine_config.py trouble because of intel_pstate/no_burbo file
Root cause: Validation Gap

Ubuntu 22.04.3 LTS WSL Windows 11 Home 23H2
CUDA 12.3
13th Intel Core(TM) Core(TM) i9-13900H

I executed the following command to test benchmark:

pytest test_bench.py --benchmark-autosave

It occured the error:

Exit: [Errno 2] No such file or directory: '/sys/devices/system/cpu/intel_pstate/no_turbo'
See README.md for machine tuning script usage, or use --ignore_machine_config
! _pytest.outcomes.Exit: [Errno 2] No such file or directory: '/sys/devices/system/cpu/intel_pstate/no_turbo'
See README.md for machine tuning script usage, or use --ignore_machine_config !

I read README.md and try to configure my CPU informations by executing

sudo $(which python) torchbenchmark/util/machine_config.py --configure

but it occured error as following:

Write 1 to /sys/devices/system/cpu/intel_pstate/no_turbo
Traceback (most recent call last):
  File "/home/wglee/apps/benchmark/torchbenchmark/util/machine_config.py", line 315, in <module>
    set_intel_no_turbo_state(1)
  File "/home/wglee/apps/benchmark/torchbenchmark/util/machine_config.py", line 32, in set_intel_no_turbo_state
    write_sys_file(turbo_file, str(state))
  File "/home/wglee/apps/benchmark/torchbenchmark/util/machine_config.py", line 24, in write_sys_file
    with open(sysfile, 'w') as f:
         ^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/sys/devices/system/cpu/intel_pstate/no_turbo'

This error message means that the "sys/device/system/cpu/intel_pstate/no_turbo" file is nessesary, but my system doesn't have this. How can I solve this problem?

Our machine config only applies to our CI machine, which is an AWS g4dn.metal instance. It comes with Intel(R) Xeon(R) Platinum 8259CL CPU and NVIDIA T4 GPU. Therefore, if your CPU does not support no_turbo, our machine config does not work. Please add `--ignore_machine_config ` to your command.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.3.0.dev20240229+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.3.0.dev20240228+cu121

Base PyTorch commit: 9c597ff137ead9f7f7ec8fdcbf473de2d328e61b

Affected PyTorch version: 2.3.0.dev20240229+cu121

Affected PyTorch commit: f5b99976adcbb01fd71bd0a39ea15bdac6c9e48a

Affected Tests:
- test_eval[pyhpc_equation_of_state-cpu-jit]: -14.59373%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: -9.24551%


cc @xuzhao9

Result json: 
```
{
  "start": "9c597ff137ead9f7f7ec8fdcbf473de2d328e61b",
  "end": "f5b99976adcbb01fd71bd0a39ea15bdac6c9e48a",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "dbe0967a0a5",
      "commit1_time": "2024-02-28 22:26:59 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08920297771692276
      },
      "commit2": "86ff31c4a00",
      "commit2_time": "2024-02-28 22:30:18 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07762912157922983
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8103212641


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.3.0.dev20240305+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.3.0.dev20240301+cu121

Base PyTorch commit: 2e84d01d0576a31a6387e4ccc195f11449fb875a

Affected PyTorch version: 2.3.0.dev20240305+cu121

Affected PyTorch commit: 6566b3db677a6d3f2f90a3fcd4dd1dd2d9d0bfc4

Affected Tests:
- test_eval[pyhpc_equation_of_state-cpu-jit]: +46.42263%
- test_eval[resnext50_32x4d-cpu-jit]: -10.21749%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +16.63101%
- test_train[tts_angular-cpu-eager]: +18.15878%


cc @xuzhao9

Result json: 
```
{
  "start": "2e84d01d0576a31a6387e4ccc195f11449fb875a",
  "end": "6566b3db677a6d3f2f90a3fcd4dd1dd2d9d0bfc4",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "2e84d01d057",
      "commit1_time": "2024-03-01 04:02:09 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.10431044045835733,
        "test_eval[resnext50_32x4d-cpu-jit]": 0.042695724638178945,
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.15409916471689938,
        "test_train[tts_angular-cpu-eager]": 0.2375387055799365
      },
      "commit2": "518a23bb039",
      "commit2_time": "2024-03-01 04:20:15 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09588903468102217
      }
    },
    {
      "commit1": "518a23bb039",
      "commit1_time": "2024-03-01 04:20:15 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09588903468102217
      },
      "commit2": "02a410ee12e",
      "commit2_time": "2024-03-01 04:47:13 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07992306221276521
      }
    },
    {
      "commit1": "02a410ee12e",
      "commit1_time": "2024-03-01 04:47:13 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07992306221276521
      },
      "commit2": "c5472628ff9",
      "commit2_time": "2024-03-01 04:48:44 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08828765004873276
      }
    },
    {
      "commit1": "8c2e569928a",
      "commit1_time": "2024-03-01 08:35:22 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08602353353053331
      },
      "commit2": "82b356193da",
      "commit2_time": "2024-03-01 08:48:13 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07808832637965679
      }
    },
    {
      "commit1": "d053dcfa69a",
      "commit1_time": "2024-03-01 04:54:11 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08677876936271786
      },
      "commit2": "13a54ce2792",
      "commit2_time": "2024-03-01 05:05:28 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09509436106309295
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8160298189


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.3.0.dev20240307+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.3.0.dev20240306+cu121

Base PyTorch commit: dad1b765848c4f52501c4c60b1c3e6fbd3cc8837

Affected PyTorch version: 2.3.0.dev20240307+cu121

Affected PyTorch commit: a2854ae904c55ebf19031ca6e883e5dcbb6a4a9a

Affected Tests:
- test_eval[maml_omniglot-cuda-eager]: +7.71768%
- test_eval[pyhpc_equation_of_state-cpu-jit]: -15.84761%
- test_train[timm_regnet-cpu-eager]: -9.54262%


cc @xuzhao9

Result json: 
```
{
  "start": "dad1b765848c4f52501c4c60b1c3e6fbd3cc8837",
  "end": "a2854ae904c55ebf19031ca6e883e5dcbb6a4a9a",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "dad1b765848",
      "commit1_time": "2024-03-06 02:30:52 +0000",
      "commit1_digest": {
        "test_eval[maml_omniglot-cuda-eager]": 0.0010689303987538037,
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.12596094869077207,
        "test_train[timm_regnet-cpu-eager]": 0.8145502582192421
      },
      "commit2": "f72eb5ae4c5",
      "commit2_time": "2024-03-06 03:44:59 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08880764041095972
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8192159491


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.3.0.dev20240310+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.3.0.dev20240309+cu121

Base PyTorch commit: d482614fec5fb9bccb49bf4ee4ab561e872c0f50

Affected PyTorch version: 2.3.0.dev20240310+cu121

Affected PyTorch commit: 41286f1505ffb214d386d72e4b72ebd680a4a475

Affected Tests:
- test_eval[pyhpc_equation_of_state-cpu-jit]: -22.01826%
- test_eval[pytorch_stargan-cpu-jit]: -13.06567%
- test_train[tts_angular-cpu-eager]: +14.38111%


cc @xuzhao9

Result json: 
```
{
  "start": "d482614fec5fb9bccb49bf4ee4ab561e872c0f50",
  "end": "41286f1505ffb214d386d72e4b72ebd680a4a475",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "242e03ba866",
      "commit1_time": "2024-03-09 06:17:23 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09320541741326452
      },
      "commit2": "761783a4ff3",
      "commit2_time": "2024-03-09 06:54:51 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07978621888905764,
        "test_train[tts_angular-cpu-eager]": 0.2384247883222997
      }
    },
    {
      "commit1": "761783a4ff3",
      "commit1_time": "2024-03-09 06:54:51 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07978621888905764,
        "test_train[tts_angular-cpu-eager]": 0.2384247883222997
      },
      "commit2": "cf9742371ca",
      "commit2_time": "2024-03-09 07:20:53 +0000",
      "commit2_digest": {
        "test_train[tts_angular-cpu-eager]": 0.20655944561585785
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8223574233


Noise, closed.
Repository: pytorch/benchmark
Issue: Validate the bf16 precisions across models
Root cause: Validation Gap

A couple of models only run on the default precisions, and we would like to to enable bf16 precisions on them.

The reason models are failing because their inputs either can't be directly cast to bf16, or not completely being cast to bf16.

Per @drisspg:
> Using this script:
> 
> ```python
> from transformer_nuggets.utils.shape_trace import ShapeLog
> import torch
> from pathlib import Path
> from tqdm import tqdm
> import logging
> import json
> 
> logging.basicConfig(level=logging.INFO)
> 
> def main():
>     import torchbenchmark.models as models
>     models = []
>     success_count = 0
>     failure_count = 0
>     model_failures = {}
>     for file in Path("torchbenchmark/models/").iterdir():
>         if file.is_dir():
>             models.append(file.name)
>     for model_name in tqdm(models, desc="Logging models", aunit="model"):
>         try:
>             module = __import__(f"torchbenchmark.models.{model_name}", fromlist=[model_name])
>             model, example_inputs = module.Model(test="train", device="cuda", extra_args=["--precision=bf16",]).get_module()
>             model(*example_inputs)
>             success_count += 1
>         except Exception as e:
>             tqdm.write(f"Failed to log {module}: {e}")
>             failure_count += 1
>             model_failures[model_name] = str(e)
> 
>     tqdm.write(f"Successfully logged {success_count} models")
>     tqdm.write(f"Failed to log {failure_count} models")
>     with open("model_failures_bf16.txt", "w") as f:
>         json.dump(model_failures, f)
> 
> 
> 
> if __name__ == "__main__":
>     main()
> ```
> 
> returns the following model failures;
> 
> [internalfb.com/intern/paste/P1197341526](https://www.internalfb.com/intern/paste/P1197341526/)



We also want to validate amp_bf16 on CPU.

We are migrating to pt2 benchmark runner, so we do not plan to support bf16 in the legacy runner.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.4.0.dev20240328+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240326+cu121

Base PyTorch commit: 5e66bf5f42104bc60a8e77b51c84a4cce81efdd8

Affected PyTorch version: 2.4.0.dev20240328+cu121

Affected PyTorch commit: 12116aee6852df2b040255b8fcc7deb52b897792

Affected Tests:
- test_eval[pyhpc_equation_of_state-cpu-jit]: -16.56640%
- test_eval[resnext50_32x4d-cpu-jit]: -14.76194%
- test_train[timm_regnet-cpu-eager]: -13.03219%
- test_train[tts_angular-cpu-eager]: -10.99061%


cc @xuzhao9

Result json: 
```
{
  "start": "5e66bf5f42104bc60a8e77b51c84a4cce81efdd8",
  "end": "12116aee6852df2b040255b8fcc7deb52b897792",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "ccfc87b1993",
      "commit1_time": "2024-03-27 19:45:25 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08162907222867943
      },
      "commit2": "2564f6cf0ea",
      "commit2_time": "2024-03-27 20:39:39 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07224691243609413
      }
    },
    {
      "commit1": "cd51496f8bf",
      "commit1_time": "2024-03-27 17:24:43 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07855590819381178
      },
      "commit2": "37e3c8f33f1",
      "commit2_time": "2024-03-27 17:43:32 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09175305935204961
      }
    },
    {
      "commit1": "37e3c8f33f1",
      "commit1_time": "2024-03-27 17:43:32 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09175305935204961
      },
      "commit2": "1d6fc0d4de7",
      "commit2_time": "2024-03-27 18:38:42 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08459161429782398
      }
    },
    {
      "commit1": "2564f6cf0ea",
      "commit1_time": "2024-03-27 20:39:39 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07224691243609413
      },
      "commit2": "35382f05735",
      "commit2_time": "2024-03-27 20:39:39 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.0781458432902582
      }
    },
    {
      "commit1": "35382f05735",
      "commit1_time": "2024-03-27 20:39:39 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.0781458432902582
      },
      "commit2": "3a67c86f72c",
      "commit2_time": "2024-03-27 20:39:39 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.09286960659665056
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8472248079


Likely noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.4.0.dev20240329+cu121'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240328+cu121

Base PyTorch commit: 12116aee6852df2b040255b8fcc7deb52b897792

Affected PyTorch version: 2.4.0.dev20240329+cu121

Affected PyTorch commit: 3beb9d85a6501d8df5a89e2cc2d7cb0f8f94dbfc

Affected Tests:
- test_eval[pytorch_stargan-cpu-jit]: +18.94569%
- test_eval[resnext50_32x4d-cpu-jit]: +20.69475%
- test_train[timm_nfnet-cpu-jit]: -8.34629%
- test_train[timm_regnet-cpu-eager]: +13.33914%


cc @xuzhao9

Result json: 
```
{
  "start": "12116aee6852df2b040255b8fcc7deb52b897792",
  "end": "3beb9d85a6501d8df5a89e2cc2d7cb0f8f94dbfc",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "d4317becce1",
      "commit1_time": "2024-03-28 03:01:27 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2637837524584029
      },
      "commit2": "60f3c092d49",
      "commit2_time": "2024-03-28 03:01:27 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2914577202463988
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8483638718


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.4.0.dev20240330+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240329+cu121

Base PyTorch commit: 3beb9d85a6501d8df5a89e2cc2d7cb0f8f94dbfc

Affected PyTorch version: 2.4.0.dev20240330+cu121

Affected PyTorch commit: f4ff063c333f286d4384523bac67c047aca4d7b9

Affected Tests:
- test_eval[pyhpc_equation_of_state-cpu-jit]: -16.22548%
- test_eval[pytorch_stargan-cpu-eager]: +8.48205%
- test_eval[resnext50_32x4d-cpu-jit]: -17.59618%
- test_train[timm_nfnet-cpu-jit]: +10.09698%
- test_train[tts_angular-cpu-eager]: +13.13359%


cc @xuzhao9

Result json: 
```
{
  "start": "3beb9d85a6501d8df5a89e2cc2d7cb0f8f94dbfc",
  "end": "f4ff063c333f286d4384523bac67c047aca4d7b9",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "ea33adf6c26",
      "commit1_time": "2024-03-29 21:48:29 +0000",
      "commit1_digest": {
        "test_train[tts_angular-cpu-eager]": 0.20725607405765914
      },
      "commit2": "d7d77a152c5",
      "commit2_time": "2024-03-29 21:49:27 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08147674165666104,
        "test_train[tts_angular-cpu-eager]": 0.23855975429178217
      }
    },
    {
      "commit1": "b5bef9bbfd3",
      "commit1_time": "2024-03-29 22:41:45 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.0818151738087181
      },
      "commit2": "f4ff063c333",
      "commit2_time": "2024-03-30 00:25:39 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.06712286436813883,
        "test_eval[pytorch_stargan-cpu-eager]": 0.26271451319335026,
        "test_eval[resnext50_32x4d-cpu-jit]": 0.04234138745717549,
        "test_train[timm_nfnet-cpu-jit]": 11.997019111760892,
        "test_train[tts_angular-cpu-eager]": 0.2407134069595486
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8492261479


Clearly noise.