zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220917+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220916+cu116

Base PyTorch commit: 0ec19db7ac88e307135100ddcfc418ae3925844f

Affected PyTorch version: 1.13.0.dev20220917+cu116

Affected PyTorch commit: d6c2080eb49ccaaf43cff37b7f07a85906250b92

Affected Tests:
- test_eval[demucs-cpu-jit]: +9.54067%
- test_eval[demucs-cpu-eager]: +8.87435%
- test_eval[mnasnet1_0-cpu-jit]: +12.69526%
- test_train[pytorch_stargan-cpu-jit]: +9.74704%
- test_eval[pytorch_stargan-cpu-jit]: +13.76241%
- test_train[pytorch_stargan-cpu-eager]: +7.24911%
- test_eval[pytorch_stargan-cpu-eager]: +17.36230%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/issues/1191
Result:
```
{
  "start": "0ec19db7ac88e307135100ddcfc418ae3925844f",
  "end": "d6c2080eb49ccaaf43cff37b7f07a85906250b92",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "607eccb13ca",
      "commit1_time": "2022-09-16 20:20:04 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.41550340594549195
      },
      "commit2": "e5fac7f5dc4",
      "commit2_time": "2022-09-16 20:21:03 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.45193747275479834
      }
    }
  ]
}
```

Verify workflow: https://github.com/pytorch/pytorch/pull/85228
Result:
```
Benchmark                                 Run 607eccb13ca586f775fb09daeb728a4b4e30ebdd    Run e5fac7f5dc4f16070193bb7d06322e0faaa94099  Delta
--------------------------------------  ----------------------------------------------  ----------------------------------------------  -----------
test_train[pytorch_stargan-cpu-jit]                                          1.06894                                         1.05101    -1.705758%
test_eval[pytorch_stargan-cpu-jit]                                           0.472737                                        0.440328   -7.360039%*
test_train[pytorch_stargan-cpu-eager]                                        1.06109                                         1.03642    -2.379924%
test_eval[pytorch_stargan-cpu-eager]                                         0.450545                                        0.420559   -7.130166%*
test_train[pytorch_stargan-cuda-jit]                                         0.274033                                        0.26579    -3.101282%
test_eval[pytorch_stargan-cuda-jit]                                          0.0543139                                       0.0543111  -0.005221%
test_train[pytorch_stargan-cuda-eager]                                       0.26583                                         0.265763   -0.025091%
test_eval[pytorch_stargan-cuda-eager]                                        0.0580968                                       0.0581017  +0.008455%
```

Looks like noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220918+cu116'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220917+cu116

Base PyTorch commit: d6c2080eb49ccaaf43cff37b7f07a85906250b92

Affected PyTorch version: 1.13.0.dev20220918+cu116

Affected PyTorch commit: 12a19a4846c924e9d1e2d37fa0a706fb8eaef9a7

Affected Tests:
- test_eval[demucs-cpu-jit]: -9.30736%
- test_eval[demucs-cpu-eager]: -7.13191%
- test_eval[mnasnet1_0-cpu-jit]: +9.91555%
- test_eval[pytorch_stargan-cpu-jit]: -15.94250%
- test_eval[pytorch_stargan-cpu-eager]: -15.75860%
- test_train[timm_resnest-cpu-eager]: +7.12065%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3077942471
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220920+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220919+cu116

Base PyTorch commit: 9024015adf01d93fd2533c71fa1e7f06831c2ac7

Affected PyTorch version: 1.13.0.dev20220920+cu116

Affected PyTorch commit: e41d758e26bd2de00e9dd50e94e878f46f9f1b88

Affected Tests:
- test_eval[demucs-cpu-jit]: +14.38958%
- test_eval[demucs-cpu-eager]: +10.48203%
- test_train[pytorch_stargan-cpu-jit]: +7.55360%
- test_eval[pytorch_stargan-cpu-jit]: +19.50994%
- test_train[pytorch_stargan-cpu-eager]: +7.25950%
- test_eval[pytorch_stargan-cpu-eager]: +18.31690%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3092099407
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220920+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220919+cu116

Base PyTorch commit: 9024015adf01d93fd2533c71fa1e7f06831c2ac7

Affected PyTorch version: 1.13.0.dev20220920+cu116

Affected PyTorch commit: e41d758e26bd2de00e9dd50e94e878f46f9f1b88

Affected Tests:
- test_eval[maml_omniglot-cpu-eager]: +9.59486%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3093597674
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220921+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220920+cu116

Base PyTorch commit: e41d758e26bd2de00e9dd50e94e878f46f9f1b88

Affected PyTorch version: 1.13.0.dev20220921+cu116

Affected PyTorch commit: cf2f552cd8a41f4913c370c15804173a3b56a415

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -9.32047%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3099788213
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220921+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220920+cu116

Base PyTorch commit: e41d758e26bd2de00e9dd50e94e878f46f9f1b88

Affected PyTorch version: 1.13.0.dev20220921+cu116

Affected PyTorch commit: cf2f552cd8a41f4913c370c15804173a3b56a415

Affected Tests:
- test_eval[maml_omniglot-cpu-eager]: -8.34660%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3101267342
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220922+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220921+cu116

Base PyTorch commit: cf2f552cd8a41f4913c370c15804173a3b56a415

Affected PyTorch version: 1.13.0.dev20220922+cu116

Affected PyTorch commit: 99ad8a304898de8bf1e20a6fc12e335e9b7c5064

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +10.36084%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3107409043
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220922+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220921+cu116

Base PyTorch commit: cf2f552cd8a41f4913c370c15804173a3b56a415

Affected PyTorch version: 1.13.0.dev20220922+cu116

Affected PyTorch commit: 99ad8a304898de8bf1e20a6fc12e335e9b7c5064

Affected Tests:
- test_eval[yolov3-cpu-eager]: +9.71378%
- test_train[yolov3-cuda-eager]: -17.99603%


cc @xuzhao9
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220923+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220922+cu116

Base PyTorch commit: 99ad8a304898de8bf1e20a6fc12e335e9b7c5064

Affected PyTorch version: 1.13.0.dev20220923+cu116

Affected PyTorch commit: cf0de77c2cfb8843b8ae67e6a6f053e6bf6bb3d9

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +7.42927%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +7.94275%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/3114467300
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220923+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220922+cu116

Base PyTorch commit: 99ad8a304898de8bf1e20a6fc12e335e9b7c5064

Affected PyTorch version: 1.13.0.dev20220923+cu116

Affected PyTorch commit: cf0de77c2cfb8843b8ae67e6a6f053e6bf6bb3d9

Affected Tests:
- test_eval[densenet121-cpu-eager]: -7.03931%
- test_train[yolov3-cuda-eager]: +21.84121%


cc @xuzhao9


bisection-gh3115003106

Workflow: https://github.com/pytorch/benchmark/actions/runs/3278821077

Bisector failed because it failed to build
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220926+cu116'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220923+cu116

Base PyTorch commit: cf0de77c2cfb8843b8ae67e6a6f053e6bf6bb3d9

Affected PyTorch version: 1.13.0.dev20220926+cu116

Affected PyTorch commit: 9c036aa112b0a8fd9afb824d1fda058e2b66ba1d

Affected Tests:
- test_eval[demucs-cpu-jit]: -7.70194%
- test_eval[demucs-cpu-eager]: -7.51122%
- test_eval[hf_Longformer-cpu-eager]: +14.35433%
- test_eval[mnasnet1_0-cpu-jit]: -25.75383%
- test_train[opacus_cifar10-cpu-eager]: +27.77841%
- test_train[opacus_cifar10-cuda-eager]: +19.59345%
- test_train[pytorch_stargan-cpu-jit]: -8.43136%
- test_eval[pytorch_stargan-cpu-jit]: -16.73703%
- test_train[pytorch_stargan-cpu-eager]: -9.22075%
- test_eval[pytorch_stargan-cpu-eager]: -15.32210%
- test_train[timm_resnest-cpu-eager]: +7.84642%
- test_train[timm_vision_transformer-cpu-eager]: -11.17509%


cc @xuzhao9


bisection-gh3128458079

workflow: https://github.com/pytorch/benchmark/actions/runs/3269829660

/usr/lib/gcc/x86_64-linux-gnu/9/include/avx512dqintrin.h(1372): error: identifier "__builtin_ia32_fpclasssd" is undefined on commit d3be4245bb4
Repository: pytorch/benchmark
Issue: Improve bisection workflow
Root cause: Unimplemented Feature Gap

- [x] Only post GitHub issues when the bisection result is non-empty
- [x] Post the bisection result in the GitHub issues
- [x] Also post GitHub issues when the bisection workflow fails
- [x] Make bisection does not rely on the attached storage (use S3 instead)
- [x] Skip the PyTorch commits that are known to fail the build on Ubuntu 20.04
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220926+cu116'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220923+cu116

Base PyTorch commit: cf0de77c2cfb8843b8ae67e6a6f053e6bf6bb3d9

Affected PyTorch version: 1.13.0.dev20220926+cu116

Affected PyTorch commit: 9c036aa112b0a8fd9afb824d1fda058e2b66ba1d

Affected Tests:
- test_eval[densenet121-cuda-jit]: +25.39836%
- test_train[opacus_cifar10-cuda-eager]: +32.37094%
- test_eval[pytorch_struct-cuda-jit]: +8.72616%
- test_eval[timm_efficientnet-cuda-jit]: +8.84136%


cc @xuzhao9


Workflow id: bisection-gh3108035500

Bisection failed - build failure on commit cf2f552cd8a
Workflow: https://github.com/pytorch/benchmark/actions/runs/3278366844

Can't resolve. Closed.
Repository: pytorch/benchmark
Issue: Fail to install timm_efficientdet
Root cause: External Dependency Breakage

```
running setup for /root/workspace/benchmark/torchbenchmark/models/timm_efficientdet...FAIL
Error for /root/workspace/benchmark/torchbenchmark/models/timm_efficientdet:
---------------------------------------------------------------------------
  WARNING: Did not find branch or tag '79d26d8', assuming revision or ref.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Failed to patch pycocotools. Exit.



---------------------------------------------------------------------------

Traceback (most recent call last):
  File "install.py", line 134, in <module>
    raise RuntimeError("Failed to complete setup")
RuntimeError: Failed to complete setup
```

The `WARNING: Did not find branch or tag ` is expected.
Are you running the benchmark as the `root` user? I guess that is the root cause.

Thanks for you reply. Yes, I am running as Root uses as I am using docker env which uses root user by default.

We require to patch pycocotools because our internal customers don't want to see extra prints. It might because recent pycocotools upgrade breaks the patch, looking into it.

Can you try if https://github.com/pytorch/benchmark/pull/1224 fixes this problem?

Fixed in #1224.
Repository: pytorch/benchmark
Issue: Correctness test for train()
Root cause: Unimplemented Feature Gap

Can model it after https://github.com/pytorch/torchdynamo/blob/f11a33107407c51e93cba96e26c2f65e56edbc1d/torchdynamo/testing.py#L37

Fixed in https://github.com/pytorch/benchmark/commit/65e4f4492236769f5f853669888af9adc45e70bb
Repository: pytorch/benchmark
Issue: Run benchmarks periodically in PyTorch CI
Root cause: Unimplemented Feature Gap

- new workflow on pytorch side
  - install pytorch, torchvision, torchtext
     - can't use conda like benchmark CI becuase we want access to more fine grained versions of pytorch than nightly
     - @malfet can help with how to access build artifacts from any particular git version of pytorch repo in last X days
     - install those
  - run benchmarks
  - upload results

bisect job would build on top by
  - get previous and current periodic result
  - 
  - if regression suspected, then
     - do binary search over all commits between A, B
        - for each, get the corresponding pytorch artifacts, install them, run measurement
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20221004+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220930+cu116

Base PyTorch commit: 95681929e4c379c504d8a7761f8104118a5a16db

Affected PyTorch version: 1.13.0.dev20221004+cu116

Affected PyTorch commit: 82d9592f1baaf943b81bca13a51d655139f050aa

Affected Tests:
- test_eval[demucs-cpu-jit]: -10.35765%
- test_eval[pytorch_stargan-cpu-jit]: +11.90628%


cc @xuzhao9


```
{
  "start": "95681929e4c379c504d8a7761f8104118a5a16db",
  "end": "82d9592f1baaf943b81bca13a51d655139f050aa",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "94da90e41f1",
      "commit1_time": "2022-09-30 20:07:08 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.4778245795023395
      },
      "commit2": "4c04fa9587f",
      "commit2_time": "2022-09-30 20:32:05 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.5597359047998907
      }
    },
    {
      "commit1": "a4cc63991ad",
      "commit1_time": "2022-09-30 22:40:50 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.5621657722949749
      },
      "commit2": "d9421f81584",
      "commit2_time": "2022-09-30 22:51:59 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.452000054702512
      }
    },
    {
      "commit1": "fe89cd6c574",
      "commit1_time": "2022-10-01 17:21:31 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.47870248219696804
      },
      "commit2": "b562987c28b",
      "commit2_time": "2022-10-01 19:30:21 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.5643056452972814
      }
    }
  ]
} 
```

Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221011+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221009+cu116

Base PyTorch commit: 92ac84c98a19310885f3d818aba56b981940d615

Affected PyTorch version: 1.14.0.dev20221011+cu116

Affected PyTorch commit: ad2b04c39c41949d8869de743736bcaeec2dfa0d

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +7.04205%
- test_eval[mobilenet_v2_quantized_qat-cpu-jit]: +7.20524%
- test_eval[resnet50_quantized_qat-cpu-jit]: +16.45182%
- test_train[timm_regnet-cpu-eager]: +18.21067%


cc @xuzhao9


Workflow:
https://github.com/pytorch/benchmark/actions/runs/3228936917

Result:
```
{
  "start": "92ac84c98a19310885f3d818aba56b981940d615",
  "end": "ad2b04c39c41949d8869de743736bcaeec2dfa0d",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "224ae0da107",
      "commit1_time": "2022-10-10 23:52:28 +0000",
      "commit1_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-jit]": 0.0035105288513368063,
        "test_eval[resnet50_quantized_qat-cpu-jit]": 0.003983804583754636
      },
      "commit2": "8a47a49d5ee",
      "commit2_time": "2022-10-11 00:07:41 +0000",
      "commit2_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-jit]": 0.003876594351529474,
        "test_eval[resnet50_quantized_qat-cpu-jit]": 0.004674500253822423
      }
    },
    {
      "commit1": "bd381121b9e",
      "commit1_time": "2022-10-11 03:24:32 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.035327554262917615
      },
      "commit2": "ad2b04c39c4",
      "commit2_time": "2022-10-11 03:29:01 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.041959999362006783,
        "test_eval[mobilenet_v2_quantized_qat-cpu-jit]": 0.0038100034445199336,
        "test_eval[resnet50_quantized_qat-cpu-jit]": 0.004712501642573175,
        "test_train[timm_regnet-cpu-eager]": 0.6885808095510584
      }
    }
  ]
}
```

I can confirm that `ad2b04c39c4` is noise. However https://github.com/pytorch/pytorch/commit/8a47a49d5ee seems to be related. Notify PR author if this is expected.

cc @jerryzh168 @malfet
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221012+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221011+cu116

Base PyTorch commit: ad2b04c39c41949d8869de743736bcaeec2dfa0d

Affected PyTorch version: 1.14.0.dev20221012+cu116

Affected PyTorch commit: 25725fd62448165b91647304c26d676db22b6955

Affected Tests:
- test_eval[demucs-cpu-eager]: +9.15365%
- test_train[pytorch_stargan-cpu-jit]: +7.95926%
- test_eval[pytorch_stargan-cpu-jit]: +18.26027%
- test_train[pytorch_stargan-cpu-eager]: +10.32072%
- test_eval[pytorch_stargan-cpu-eager]: +22.05820%
- test_train[timm_vision_transformer-cpu-eager]: -10.91496%


cc @xuzhao9


Result:
```
{
  "start": "ad2b04c39c41949d8869de743736bcaeec2dfa0d",
  "end": "25725fd62448165b91647304c26d676db22b6955",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "3a2cfbb813e",
      "commit1_time": "2022-10-11 10:17:27 +0000",
      "commit1_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.21708421050570906
      },
      "commit2": "49c9b0a1541",
      "commit2_time": "2022-10-11 12:00:35 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.238304991601035
      }
    }
  ]
}
```

Should be irrelevant, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221013+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221012+cu116

Base PyTorch commit: 25725fd62448165b91647304c26d676db22b6955

Affected PyTorch version: 1.14.0.dev20221013+cu116

Affected PyTorch commit: 48c648d75df4a2d02ede71f34c11b7f48c80da0e

Affected Tests:
- test_eval[demucs-cpu-jit]: -9.28181%
- test_eval[pytorch_stargan-cpu-jit]: -20.84319%
- test_train[pytorch_stargan-cpu-eager]: -7.73809%
- test_eval[pytorch_stargan-cpu-eager]: -16.52717%
- test_train[timm_regnet-cpu-eager]: -12.28661%
- test_train[timm_vision_transformer-cpu-eager]: +18.79796%


cc @xuzhao9


```
{
  "start": "25725fd62448165b91647304c26d676db22b6955",
  "end": "48c648d75df4a2d02ede71f34c11b7f48c80da0e",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "cd7c86eaa46",
      "commit1_time": "2022-10-12 18:22:00 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.49192751239752397,
        "test_eval[pytorch_stargan-cpu-eager]": 0.44198681169655174
      },
      "commit2": "ce56ee11fdf",
      "commit2_time": "2022-10-12 18:37:50 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.5305446345126257,
        "test_eval[pytorch_stargan-cpu-eager]": 0.5130556826712563
      }
    },
    {
      "commit1": "67fbd940bae",
      "commit1_time": "2022-10-13 01:57:16 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-eager]": 0.5118519469629973
      },
      "commit2": "48c648d75df",
      "commit2_time": "2022-10-13 03:12:59 +0000",
      "commit2_digest": {
        "test_eval[demucs-cpu-jit]": 2.2618393443641254,
        "test_eval[pytorch_stargan-cpu-jit]": 0.5604625766165554,
        "test_train[pytorch_stargan-cpu-eager]": 1.093090152356308,
        "test_eval[pytorch_stargan-cpu-eager]": 0.4701112618204206,
        "test_train[timm_regnet-cpu-eager]": 0.6866150594316422,
        "test_train[timm_vision_transformer-cpu-eager]": 0.22907948535867034
      }
    }
  ]
}
```

Likely all of them are noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221016+cu116'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221015+cu116

Base PyTorch commit: 5210fab64d4322438ebfd8ec9c1170d5effab0a3

Affected PyTorch version: 1.14.0.dev20221016+cu116

Affected PyTorch commit: a0c2a7f2eda788a48f1d243940297f1467faf138

Affected Tests:
- test_eval[demucs-cpu-eager]: -7.49549%
- test_train[pytorch_stargan-cpu-jit]: -7.39722%
- test_eval[pytorch_stargan-cpu-jit]: -14.73966%
- test_train[pytorch_stargan-cpu-eager]: -7.06163%
- test_eval[pytorch_stargan-cpu-eager]: -17.60996%
- test_train[timm_regnet-cpu-eager]: +9.46323%


cc @xuzhao9


Result:
```
{
  "start": "5210fab64d4322438ebfd8ec9c1170d5effab0a3",
  "end": "a0c2a7f2eda788a48f1d243940297f1467faf138",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "5210fab64d4",
      "commit1_time": "2022-10-15 01:55:08 +0000",
      "commit1_digest": {
        "test_eval[demucs-cpu-eager]": 2.282899834541604,
        "test_train[pytorch_stargan-cpu-jit]": 1.106276608526241,
        "test_eval[pytorch_stargan-cpu-jit]": 0.5625019979081116,
        "test_train[pytorch_stargan-cpu-eager]": 1.0787336850655265,
        "test_eval[pytorch_stargan-cpu-eager]": 0.5045576174510643,
        "test_train[timm_regnet-cpu-eager]": 0.695530328527093
      },
      "commit2": "46aaae98c5b",
      "commit2_time": "2022-10-15 01:57:42 +0000",
      "commit2_digest": {
        "test_train[pytorch_stargan-cpu-jit]": 1.0261005135951564,
        "test_eval[pytorch_stargan-cpu-jit]": 0.49478353729937224,
        "test_eval[pytorch_stargan-cpu-eager]": 0.44964335130061955
      }
    }
  ]
}
```

Speedup, closed.