zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Models that don't support CPU tests or disabled
Root cause: Unimplemented Feature Gap

In this issue, we maintain a list heavy-weight models which we disable CPU tests, because they are too slow (>60s):

# Core Models

- [ ] [fambench_xlmr](https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models/fambench_xlmr)
- [x] [timm_efficientdet](https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models/timm_efficientdet), test slowness

These heavy-weight models are not supposed to run on CPU or even small memory GPUs.

The checked models means they are already disabled on CPU.

# Canary Models

- [x] [fambench_dlrm](https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models/fambench_dlrm)
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220707+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220630+cu113

Base PyTorch commit: 331c0c18033bf4138c9ea468a8c759865dd8ffff

Affected PyTorch version: 1.13.0.dev20220707+cu113

Affected PyTorch commit: 74877943b84cb6d3fc82993ebc8f0e2f8b3c5ce1

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +7.33722%
- test_eval[pytorch_struct-cpu-jit]: +9.16033%
- test_eval[yolov3-cpu-eager]: -8.57516%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2632944146
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220708+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220707+cu113

Base PyTorch commit: 74877943b84cb6d3fc82993ebc8f0e2f8b3c5ce1

Affected PyTorch version: 1.13.0.dev20220708+cu113

Affected PyTorch commit: 8389ccbcd809e2596bd4f8a20d069fc919beefdf

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -14.23991%
- test_train[timm_vision_transformer-cpu-eager]: -8.74397%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2638286995
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220708+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220707+cu113

Base PyTorch commit: 74877943b84cb6d3fc82993ebc8f0e2f8b3c5ce1

Affected PyTorch version: 1.13.0.dev20220708+cu113

Affected PyTorch commit: 8389ccbcd809e2596bd4f8a20d069fc919beefdf

Affected Tests:
- test_eval[pytorch_struct-cpu-jit]: -9.53345%
- test_eval[yolov3-cpu-eager]: +9.49284%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2639106171
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220711+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220710+cu113

Base PyTorch commit: f3008be9004418c3651bf89de764355580b994a2

Affected PyTorch version: 1.13.0.dev20220711+cu113

Affected PyTorch commit: 3c2199b159b6ec57af3f7ea22d61ace9ce5cf5bc

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: +10.69376%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2652856841
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220712+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220711+cu113

Base PyTorch commit: 3c2199b159b6ec57af3f7ea22d61ace9ce5cf5bc

Affected PyTorch version: 1.13.0.dev20220712+cu113

Affected PyTorch commit: 28776c45e393ec7d6731a5613f5d79d309036d3a

Affected Tests:
- test_eval[demucs-cpu-jit]: -11.28049%
- test_eval[demucs-cpu-eager]: -7.59035%
- test_eval[mnasnet1_0-cpu-jit]: -20.43911%
- test_eval[pytorch_stargan-cpu-jit]: -19.20106%
- test_train[pytorch_stargan-cpu-eager]: -9.20558%
- test_eval[pytorch_stargan-cpu-eager]: -19.21910%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2658204603
Root cause commit: b4e342928b8

```
{
  "start": "3c2199b159b6ec57af3f7ea22d61ace9ce5cf5bc",
  "end": "28776c45e393ec7d6731a5613f5d79d309036d3a",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "528ee0fa75a",
      "commit1_time": "2022-07-11 18:37:50 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.4770754603086971,
        "test_eval[pytorch_stargan-cpu-eager]": 0.4429763081483543
      },
      "commit2": "b4e342928b8",
      "commit2_time": "2022-07-11 19:13:06 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.42968716875184326,
        "test_eval[pytorch_stargan-cpu-eager]": 0.4093787627003621
      }
    }
  ]
}
```

Looks like noise commit, closing it.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220713+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220712+cu113

Base PyTorch commit: 28776c45e393ec7d6731a5613f5d79d309036d3a

Affected PyTorch version: 1.13.0.dev20220713+cu113

Affected PyTorch commit: 07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98

Affected Tests:
- test_eval[demucs-cpu-jit]: +7.55813%
- test_eval[demucs-cpu-eager]: +8.70668%
- test_eval[mnasnet1_0-cpu-jit]: +35.27310%
- test_train[opacus_cifar10-cpu-eager]: +9.78627%
- test_train[opacus_cifar10-cuda-eager]: +11.89006%
- test_eval[pytorch_stargan-cpu-jit]: +23.09278%
- test_train[pytorch_stargan-cpu-eager]: +7.06729%
- test_eval[pytorch_stargan-cpu-eager]: +21.33526%
- test_eval[resnet18-cpu-jit]: +27.64338%
- test_train[timm_vision_transformer-cpu-eager]: -9.69950%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2665116830

```
{
  "start": "28776c45e393ec7d6731a5613f5d79d309036d3a",
  "end": "07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "56d1c755185",
      "commit1_time": "2022-07-12 13:54:23 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.4307093706098385,
        "test_eval[pytorch_stargan-cpu-eager]": 0.41257073894375934
      },
      "commit2": "e505796a2c0",
      "commit2_time": "2022-07-12 14:28:54 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.46755424539442175,
        "test_eval[pytorch_stargan-cpu-eager]": 0.4445980873540975
      }
    }
  ]
}
```
https://github.com/pytorch/pytorch/commit/e505796a2c0

Unrelated. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220713+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220711+cu113

Base PyTorch commit: 3c2199b159b6ec57af3f7ea22d61ace9ce5cf5bc

Affected PyTorch version: 1.13.0.dev20220713+cu113

Affected PyTorch commit: 07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: -12.91311%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2666546617
Empty result, closed.
Repository: pytorch/benchmark
Issue: cyclegan is extremely slow
Root cause: Algorithmic Error

can it be sped up, are there overheads to eliminate, is it properly using cuda when it says it is?

if fixable, re-enable running on PR jobs


The runtime looks good to me now.

```
(py310) xzhao9@a100-test-runner:~/benchmark$ python run.py pytorch_CycleGAN_and_pix2pix -d cuda
Running eval method from pytorch_CycleGAN_and_pix2pix on cuda in eager mode with input batch size 1.
GPU Time:              4.902 milliseconds
CPU Total Wall Time:   4.926 milliseconds
GPU 0 Peak Memory:              1.9021 GB
CPU Peak Memory:                2.8125 GB
(py310) xzhao9@a100-test-runner:~/benchmark$ python run.py pytorch_CycleGAN_and_pix2pix -d cuda -t train
Running train method from pytorch_CycleGAN_and_pix2pix on cuda in eager mode with input batch size 1.
GPU Time:            392.328 milliseconds
CPU Total Wall Time: 392.416 milliseconds
GPU 0 Peak Memory:              4.3904 GB
CPU Peak Memory:                2.9326 GB
```
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220714+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220713+cu113

Base PyTorch commit: 07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98

Affected PyTorch version: 1.13.0.dev20220714+cu113

Affected PyTorch commit: 12c30a8250d1a00a0ac2e03d70ec909c76d922c3

Affected Tests:
- test_train[LearningToPaint-cpu-eager]: -7.04981%
- test_eval[demucs-cpu-eager]: -8.98488%
- test_eval[mnasnet1_0-cpu-jit]: -27.06814%
- test_train[pytorch_stargan-cpu-jit]: -7.87873%
- test_eval[pytorch_stargan-cpu-jit]: -12.60054%
- test_eval[pytorch_stargan-cpu-eager]: -16.11738%
- test_eval[resnet18-cpu-jit]: -21.63590%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2671959051
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220714+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220713+cu113

Base PyTorch commit: 07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98

Affected PyTorch version: 1.13.0.dev20220714+cu113

Affected PyTorch commit: 12c30a8250d1a00a0ac2e03d70ec909c76d922c3

Affected Tests:
- test_eval[mobilenet_v2-cuda-jit]: -13.13019%
- test_eval[mobilenet_v3_large-cpu-jit]: +10.60343%
- test_eval[mobilenet_v3_large-cuda-jit]: -9.17068%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -11.03332%


cc @xuzhao9


```
{
  "start": "07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98",
  "end": "12c30a8250d1a00a0ac2e03d70ec909c76d922c3",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v1.0",
  "result": [
    {
      "commit1": "667607b3abf",
      "commit1_time": "2022-07-13 05:54:32 +0000",
      "commit1_digest": {
        "test_eval[mobilenet_v2-cuda-jit]": 0.0045820662344340235,
        "test_eval[mobilenet_v3_large-cuda-jit]": 0.006103763426654041,
        "test_eval[shufflenet_v2_x1_0-cuda-jit]": 0.005173423272898402
      },
      "commit2": "d3acbc821e4",
      "commit2_time": "2022-07-13 06:08:38 +0000",
      "commit2_digest": {
        "test_eval[mobilenet_v2-cuda-jit]": 0.004011775953732731,
        "test_eval[mobilenet_v3_large-cuda-jit]": 0.005608914097728955,
        "test_eval[shufflenet_v2_x1_0-cuda-jit]": 0.004755015665956583
      }
    }
  ]
}
```
Workflow: https://github.com/pytorch/benchmark/actions/runs/2673265555
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220715+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220714+cu113

Base PyTorch commit: 12c30a8250d1a00a0ac2e03d70ec909c76d922c3

Affected PyTorch version: 1.13.0.dev20220715+cu113

Affected PyTorch commit: 75fdebde6290e2955129e665c2b0d99c391542a7

Affected Tests:
- test_eval[demucs-cpu-jit]: +12.38683%
- test_eval[demucs-cpu-eager]: +11.88383%
- test_eval[mnasnet1_0-cpu-jit]: +14.74810%
- test_train[pytorch_stargan-cpu-jit]: +9.02838%
- test_eval[pytorch_stargan-cpu-jit]: +14.49020%
- test_train[pytorch_stargan-cpu-eager]: +9.01640%
- test_eval[pytorch_stargan-cpu-eager]: +20.30786%
- test_train[timm_resnest-cpu-eager]: -7.43343%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2678416375
Empty result, close.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220715+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220714+cu113

Base PyTorch commit: 12c30a8250d1a00a0ac2e03d70ec909c76d922c3

Affected PyTorch version: 1.13.0.dev20220715+cu113

Affected PyTorch commit: 75fdebde6290e2955129e665c2b0d99c391542a7

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: -8.87684%
- test_eval[timm_vovnet-cpu-eager]: +9.81278%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2679549956
Empty result, close.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220716+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220715+cu113

Base PyTorch commit: 75fdebde6290e2955129e665c2b0d99c391542a7

Affected PyTorch version: 1.13.0.dev20220716+cu113

Affected PyTorch commit: 00359ff886e69f064360e5e8c6f2d5c306a105f1

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +35.92552%
- test_train[timm_vision_transformer-cpu-eager]: +7.76820%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2682600936
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220718+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220717+cu113

Base PyTorch commit: 0b5b10002af2dce031d4fad337728f81c58a8b10

Affected PyTorch version: 1.13.0.dev20220718+cu113

Affected PyTorch commit: 471397d0ee27b61ae1a088ece9f3ead1562152e0

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -35.33229%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2692254368
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220718+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220717+cu113

Base PyTorch commit: 0b5b10002af2dce031d4fad337728f81c58a8b10

Affected PyTorch version: 1.13.0.dev20220718+cu113

Affected PyTorch commit: 471397d0ee27b61ae1a088ece9f3ead1562152e0

Affected Tests:
- test_eval[timm_efficientnet-cpu-jit]: +457.65788%
- test_eval[timm_efficientnet-cpu-eager]: +387.80496%


cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/2693627667

Result:

```
{
  "start": "0b5b10002af2dce031d4fad337728f81c58a8b10",
  "end": "471397d0ee27b61ae1a088ece9f3ead1562152e0",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v1.0",
  "result": [
    {
      "commit1": "95c148e502b",
      "commit1_time": "2022-07-17 05:49:39 +0000",
      "commit1_digest": {
        "test_eval[timm_efficientnet-cpu-jit]": 0.032386388344457374,
        "test_eval[timm_efficientnet-cpu-eager]": 0.037657551842534706
      },
      "commit2": "4655c3bace4",
      "commit2_time": "2022-07-17 14:58:26 +0000",
      "commit2_digest": {
        "test_eval[timm_efficientnet-cpu-jit]": 0.17101764229009858,
        "test_eval[timm_efficientnet-cpu-eager]": 0.18220487465150653
      }
    }
  ]
}
```

Root cause commit: https://github.com/pytorch/pytorch/commit/4655c3bace4

@ngimel   @malfet @yanbing-j Can you help take a look?


@yanbing-j can conditions to disable oneDNN be relaxed to not regress existing perf?


Hi @ngimel @xuzhao9 @malfet  , current solution needs more check before running into oneDNN Conv, which causes the perf downgrade. And we are trying to mitigate the regression in other approaches.
You may revert this PR first, I will create a new PR to avoid the benchmark regression. Thank you!


> Hi @ngimel @xuzhao9 @malfet , current solution needs more check before running into oneDNN Conv, which causes the perf downgrade. And we are trying to mitigate the regression in other approaches. You may revert this PR first, I will create a new PR to avoid the benchmark regression. Thank you!

Reverted. Can you please run the TorchBench CI on the new PR to verify that it doesn't introduce regression (similar to https://github.com/pytorch/pytorch/pull/65839)? Thanks!

Closed as we have reverted the diff.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220719+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220718+cu113

Base PyTorch commit: 471397d0ee27b61ae1a088ece9f3ead1562152e0

Affected PyTorch version: 1.13.0.dev20220719+cu113

Affected PyTorch commit: a8f4011e90aeffac0757bfdb971b139e1d348967

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +13.83005%
- test_train[timm_resnest-cpu-eager]: +7.20028%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2699363978
Empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220719+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220718+cu113

Base PyTorch commit: 471397d0ee27b61ae1a088ece9f3ead1562152e0

Affected PyTorch version: 1.13.0.dev20220719+cu113

Affected PyTorch commit: a8f4011e90aeffac0757bfdb971b139e1d348967

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: +8.35210%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2700736902
Empty result, closed.
Repository: pytorch/benchmark
Issue: How to add an new backend?
Root cause: Unimplemented Feature Gap

Hello, I want to add an new backend to run benchmark **without** modify this repo's code. In torchdynamo repo, I use @create_backend decorator to finish this, but I can't find suitable interface in this repo. 

Can you specify what backend you want to add? We automatically support all backends available in `torchdynamo.list_backends()` (https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/backends/torchdynamo.py#L16)

If you want to support something not in torchdynamo, you need to write your own backend in https://github.com/pytorch/benchmark/tree/main/torchbenchmark/util/backends.

> Can you specify what backend you want to add?

we want to add our [BladeDISC](https://github.com/alibaba/BladeDISC) backend both for torchscript and torchdynamo frontend.

>  you need to write your own backend in https://github.com/pytorch/benchmark/tree/main/torchbenchmark/util/backends.

OK, I add script in this folder now, but do you have plans to add an interface to register backends, so that other backend could easily trace up this repo?

@zzpmiracle 
We have already supported the torchdynamo frontend, so if you have already added the BladeDISC backend for torchdynamo, you don't need to modify this repo. Instead, you can directly use something like `--torchdynamo BladeDISC`, if you have already registered it in torchdynamo (i.e., BladeDISC is available in `torchdynamo.list_backends()`). No modification is needed in this repo. So I believe we don't need interface to register backends when using torchdynamo.

For your request to "add an interface to register backends", I still need more context information. Do you mean you would like to run BladeDISC directly on eager mode, without torchscript or torchdynamo frontend, or do you mean you would like an option for torchscript to add backend? We can consider some torchdynamo-like interface here, e.g., use decorator to register new eager or torchscript backends. What do you think?


Yes, we'd like to add backends just for torchscript now. 
> We can consider some torchdynamo-like interface here, e.g., use decorator to register new eager or torchscript backends.

Yeah, this would be fascinating for the ecology of compiler. And I think it's necessary when more backends want to use this benchmark repo.

@zzpmiracle I created https://github.com/pytorch/benchmark/pull/1059 to make it easier to add more backends. Please let me know what you think.

@xuzhao9 Thanks! I think it's very convinient now to add more backends. And maybe I can change other backends in this repo in this way later.
Thanks again!

Closed by #1059
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220720+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220719+cu113

Base PyTorch commit: a8f4011e90aeffac0757bfdb971b139e1d348967

Affected PyTorch version: 1.13.0.dev20220720+cu113

Affected PyTorch commit: 589e8a1da5aefeb27c2187b8f0d11471edb10153

Affected Tests:
- test_eval[maml_omniglot-cpu-eager]: +7.17770%
- test_eval[mnasnet1_0-cpu-jit]: -11.87829%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2706451873
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220720+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220719+cu113

Base PyTorch commit: a8f4011e90aeffac0757bfdb971b139e1d348967

Affected PyTorch version: 1.13.0.dev20220720+cu113

Affected PyTorch commit: 589e8a1da5aefeb27c2187b8f0d11471edb10153

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: -9.46448%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +8.43156%
- test_eval[timm_efficientnet-cpu-jit]: -81.78407%
- test_eval[timm_efficientnet-cpu-eager]: -80.37847%


cc @xuzhao9


Bisection job failed: https://github.com/pytorch/benchmark/actions/runs/2707816859

Result:
```
{
  "start": "a8f4011e90aeffac0757bfdb971b139e1d348967",
  "end": "589e8a1da5aefeb27c2187b8f0d11471edb10153",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v1.0",
  "result": [
    {
      "commit1": "5bd7abf2814",
      "commit1_time": "2022-07-19 18:52:35 +0000",
      "commit1_digest": {
        "test_eval[timm_efficientnet-cpu-jit]": 0.17810393729014323,
        "test_eval[timm_efficientnet-cpu-eager]": 0.1781334939878434
      },
      "commit2": "f82b19f15be",
      "commit2_time": "2022-07-19 18:58:52 +0000",
      "commit2_digest": {
        "test_eval[timm_efficientnet-cpu-jit]": 0.03223941979231313,
        "test_eval[timm_efficientnet-cpu-eager]": 0.03740746547743954
      }
    }
  ]
}
```


Revert commit https://github.com/pytorch/pytorch/commit/f82b19f15be. Known issue. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220721+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220720+cu113

Base PyTorch commit: 589e8a1da5aefeb27c2187b8f0d11471edb10153

Affected PyTorch version: 1.13.0.dev20220721+cu113

Affected PyTorch commit: 2fb2740ef976ab6f139c5ccfa17c5c265bd2f78c

Affected Tests:
- test_eval[maml_omniglot-cpu-eager]: -7.69953%
- test_eval[mnasnet1_0-cpu-jit]: +50.82209%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2713496853
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220722+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220721+cu113

Base PyTorch commit: 2fb2740ef976ab6f139c5ccfa17c5c265bd2f78c

Affected PyTorch version: 1.13.0.dev20220722+cu113

Affected PyTorch commit: 12cb26509a30077f4bd11972dea9baa03c8d943f

Affected Tests:
- test_eval[demucs-cpu-jit]: -9.42417%
- test_eval[demucs-cpu-eager]: -10.30374%
- test_eval[mnasnet1_0-cpu-jit]: -34.82306%
- test_eval[pytorch_stargan-cpu-jit]: -8.09479%
- test_eval[pytorch_stargan-cpu-eager]: -17.46179%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2720104245

Empty result, closed.
Repository: pytorch/benchmark
Issue: Add  huggingface diffuser models
Root cause: Unimplemented Feature Gap

https://github.com/huggingface/diffusers

Hi @xuzhao9 is this still something you're working on, was interested in adding #1444 and it'd be trivial to add if diffusers support was included

I can work on an example to add instructpix2pix

That'd be super helpful thanks! I'll focus on LLAMA then

Fixed by https://github.com/pytorch/benchmark/pull/1451