zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.14.0.dev20221130+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221129+cu116

Base PyTorch commit: 2b522670d2840522e0eeb87afec7f1956eaa8748

Affected PyTorch version: 1.14.0.dev20221130+cu116

Affected PyTorch commit: 12f98f85bc7effd48f4728dcaba6362c8b2dd697

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +15.93172%


cc @xuzhao9

Result json: 
```
{
  "start": "2b522670d2840522e0eeb87afec7f1956eaa8748",
  "end": "12f98f85bc7effd48f4728dcaba6362c8b2dd697",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "2b522670d28",
      "commit1_time": "2022-11-29 04:29:42 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.0584651097189635
      },
      "commit2": "9048cf16fe4",
      "commit2_time": "2022-11-29 04:38:53 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.037169994586812596
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3634859846


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20221213+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.14.0.dev20221130+cu116

Base PyTorch commit: 12f98f85bc7effd48f4728dcaba6362c8b2dd697

Affected PyTorch version: 2.0.0.dev20221213+cu116

Affected PyTorch commit: 7498e23bd52f049071500d443a18e6e5382c87b1

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +15.45242%
- test_eval[pytorch_stargan-cpu-eager]: +11.04896%
- test_eval[speech_transformer-cuda-eager]: +7.62460%


cc @xuzhao9

Result json: 
```
{
  "start": "12f98f85bc7effd48f4728dcaba6362c8b2dd697",
  "end": "7498e23bd52f049071500d443a18e6e5382c87b1",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "454361435c5",
      "commit1_time": "2022-12-08 15:59:09 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03550548493262011
      },
      "commit2": "793a999ce03",
      "commit2_time": "2022-12-08 16:18:03 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04247148862486938
      }
    },
    {
      "commit1": "9bb16cd3ca3",
      "commit1_time": "2022-12-08 21:41:15 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04287325220927111
      },
      "commit2": "fec39f6310e",
      "commit2_time": "2022-12-08 22:03:24 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03497551082671572
      }
    },
    {
      "commit1": "f978a8b0269",
      "commit1_time": "2022-12-09 01:59:02 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.036377358998489116
      },
      "commit2": "797544f1c45",
      "commit2_time": "2022-12-09 02:00:18 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.05029919004955445
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3687793581


Noisy result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20221217+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20221216+cu116

Base PyTorch commit: 4372dbb89f457a0aa90f22d27f1468442a2a51bd

Affected PyTorch version: 2.0.0.dev20221217+cu116

Affected PyTorch commit: e689c509222a2dbc4796bcf32625193b9ea13c49

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -26.52636%
- test_train[timm_regnet-cpu-eager]: -9.19928%


cc @xuzhao9

Result json: 
```
{
  "start": "4372dbb89f457a0aa90f22d27f1468442a2a51bd",
  "end": "e689c509222a2dbc4796bcf32625193b9ea13c49",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "4372dbb89f4",
      "commit1_time": "2022-12-16 03:01:47 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04274958052206784,
        "test_train[timm_regnet-cpu-eager]": 0.7028522098466056
      },
      "commit2": "855f4b7d247",
      "commit2_time": "2022-12-16 04:03:52 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03579472458784886
      }
    },
    {
      "commit1": "2f6ada84b40",
      "commit1_time": "2022-12-16 04:29:12 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03529511510338313
      },
      "commit2": "82a191313ee",
      "commit2_time": "2022-12-16 05:07:49 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.046526840911775995
      }
    },
    {
      "commit1": "82a191313ee",
      "commit1_time": "2022-12-16 05:07:49 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.046526840911775995
      },
      "commit2": "eef019c14a1",
      "commit2_time": "2022-12-16 05:13:51 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03653280746116901
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3721042699


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20221222+cu116'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20221221+cu116

Base PyTorch commit: 5d37890b8eb952560f22b1b8ff661a024bf4b6d9

Affected PyTorch version: 2.0.0.dev20221222+cu116

Affected PyTorch commit: 6cea4f3d57927b30c3fc0a2f7103684fde0c75ea

Affected Tests:
- test_eval[maml_omniglot-cpu-eager]: -7.62340%
- test_eval[mnasnet1_0-cpu-jit]: -29.00994%
- test_train[timm_regnet-cpu-eager]: -7.83978%


cc @xuzhao9

Result json: 
```
{
  "start": "5d37890b8eb952560f22b1b8ff661a024bf4b6d9",
  "end": "6cea4f3d57927b30c3fc0a2f7103684fde0c75ea",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "68e9da68cbe",
      "commit1_time": "2022-12-22 01:13:35 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.05003480956934038
      },
      "commit2": "b50f379cece",
      "commit2_time": "2022-12-22 01:56:08 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03614221171652577
      }
    },
    {
      "commit1": "2a23dfe8ed1",
      "commit1_time": "2022-12-21 22:52:24 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.036394208777762414
      },
      "commit2": "300f7777969",
      "commit2_time": "2022-12-21 23:34:24 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.05002080377501746
      }
    },
    {
      "commit1": "71318742f99",
      "commit1_time": "2022-12-22 03:33:06 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03529444368202882
      },
      "commit2": "6cea4f3d579",
      "commit2_time": "2022-12-22 04:35:26 +0000",
      "commit2_digest": {
        "test_eval[maml_omniglot-cpu-eager]": 0.001255917840263031,
        "test_eval[mnasnet1_0-cpu-jit]": 0.042348919329621516,
        "test_train[timm_regnet-cpu-eager]": 0.7171015405911021
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3759806343


Noise, closed.
Repository: pytorch/benchmark
Issue: test resnet50 failed?
Root cause: External Dependency Breakage

Hi，I encountered a problem when executing the following test command :
```
 python run.py resnet50 -d cuda -t train --bs 32 --torchdynamo inductor
```
The result is:
```
...
[2022-12-28 06:50:30,069] torch._inductor.ir: [WARNING] Using FallbackKernel: torch.ops.aten.addcmul_.default
[2022-12-28 06:50:30,087] torch._inductor.ir: [WARNING] Using FallbackKernel: torch.ops.aten.addcmul_.default
[2022-12-28 06:50:42,387] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
grad of param conv1.weight after running with dynamo doesn't have gradient matching with eager mode
Running train method from resnet50 on cuda in eager mode with input batch size 32.
GPU Time:             38.122 milliseconds
CPU Total Wall Time:  38.148 milliseconds
Correctness:                        False
```
It shows that the correctness is false. Is it abnormal?

env: [pytorch2.0](https://pytorch.org/get-started/pytorch-2.0/#getting-started) , A100, cuda11.7


I can reproduce it on the latest PyTorch nightly and A100:

```
Running train method from resnet50 on cuda in eager mode with input batch size 32.
GPU Time:             37.284 milliseconds
CPU Total Wall Time:  37.334 milliseconds
Correctness:                        False
```
Version: torch-2.0.0.dev20221222+cu116

I have some problem installing the latest nightly (problem on torchtriton dependencies related to https://pytorch.org/blog/compromised-nightly-dependency/).

@desertfire Can you help look into this and see if you can reproduce?

I can reproduce it locally, but running resnet50 through the script in PyTorch core does pass (`python benchmarks/dynamo/torchbench.py --training --accuracy --device cuda --inductor  --only resnet50`). So I suspect there is something inconsistent between two scripts.

Fixed by https://github.com/pytorch/benchmark/pull/1700
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230104+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230102+cu116

Base PyTorch commit: f6136331244fbe680ccc03d3bd5c5d9a71933d9f

Affected PyTorch version: 2.0.0.dev20230104+cu116

Affected PyTorch commit: 80394bb73423bb849f46a7a9ffc55ebf478b8639

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -27.33321%
- test_eval[mobilenet_v2_quantized_qat-cpu-eager]: +1222.30368%
- test_eval[pytorch_stargan-cpu-eager]: -7.07683%
- test_eval[resnet50_quantized_qat-cpu-eager]: +1265.28386%
- test_train[timm_vision_transformer-cpu-eager]: +11.74280%


cc @xuzhao9

Result json: 
```
{
  "start": "f6136331244fbe680ccc03d3bd5c5d9a71933d9f",
  "end": "80394bb73423bb849f46a7a9ffc55ebf478b8639",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "2b52db9c953",
      "commit1_time": "2023-01-02 11:12:19 +0000",
      "commit1_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-eager]": 0.007353231086573728,
        "test_eval[resnet50_quantized_qat-cpu-eager]": 0.007460841989578833
      },
      "commit2": "34f2d3e6ae5",
      "commit2_time": "2023-01-02 15:53:36 +0000",
      "commit2_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-eager]": 0.09523613586788997,
        "test_eval[resnet50_quantized_qat-cpu-eager]": 0.1003648154437542
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3840153077


@malfet This looks like a valid signal (>10x slowdown on qat models), but I couldn't relate the root cause commit [34f2d3e6ae5](https://github.com/pytorch/pytorch/commit/34f2d3e6ae5)

Wow, this looks like a lot, though it sounds very suspicious, as diff in question should only affect performance on the error checking path. Is there a one-liner one can use to re-run the benchmark for a particular test (say resnet50_quantized_qat)?
cc: @vkuzo 

Yes, here is the command:
```
python run.py resnet50_quantized_qat -d cpu -t eval
```

I can reproduce it for model `resnet50_quantized_qat` on my side.  I roughly went through the code but didn't find suspicious functions.  The code lines leading to this issue are hidden from the pytorch profiler's trace, i.e. between python and aten. need more time to dig out the root cause.

And the following two figures show the difference of the nightly version between Jan 02 and Jan 04.



Jan 02's profiling result
![Jan02](https://user-images.githubusercontent.com/3823240/210912176-d63d044e-4323-41ab-894a-5b5d0e915be8.png)

Jan 04's reprofiling result. 
![Jan04](https://user-images.githubusercontent.com/3823240/210912173-2438f7f4-31d0-4da2-b26a-3e3ad8afd267.png)



The PR is now reverted by https://github.com/pytorch/pytorch/commit/b3603f81290ce137381933eff3cefc4c0380f1b2

Found the root cause (details in https://github.com/pytorch/pytorch/pull/87855) , closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230104+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230102+cu116

Base PyTorch commit: f6136331244fbe680ccc03d3bd5c5d9a71933d9f

Affected PyTorch version: 2.0.0.dev20230104+cu116

Affected PyTorch commit: 80394bb73423bb849f46a7a9ffc55ebf478b8639

Affected Tests:
- test_eval[mobilenet_v2_quantized_qat-cpu-eager]: +1228.11928%
- test_eval[pytorch_stargan-cpu-jit]: -8.02998%
- test_eval[pytorch_stargan-cpu-eager]: -7.51826%
- test_eval[resnet50_quantized_qat-cpu-eager]: +1266.74974%
- test_train[timm_vision_transformer-cpu-eager]: +12.05276%
- test_eval[timm_vovnet-cpu-eager]: -8.18224%


cc @xuzhao9

Result json: 
```
{
  "start": "f6136331244fbe680ccc03d3bd5c5d9a71933d9f",
  "end": "80394bb73423bb849f46a7a9ffc55ebf478b8639",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "fae821c2f16",
      "commit1_time": "2023-01-02 20:30:39 +0000",
      "commit1_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-eager]": 0.09535517445765436,
        "test_eval[resnet50_quantized_qat-cpu-eager]": 0.10035340173635632,
        "test_train[timm_vision_transformer-cpu-eager]": 0.1991692195413634
      },
      "commit2": "cce577b3915",
      "commit2_time": "2023-01-02 21:08:30 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.22858046093024315
      }
    },
    {
      "commit1": "6e236553f5b",
      "commit1_time": "2023-01-03 03:06:33 +0000",
      "commit1_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.23327672648010775
      },
      "commit2": "cb3204823e8",
      "commit2_time": "2023-01-03 04:21:39 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.2078154262737371
      }
    },
    {
      "commit1": "2b52db9c953",
      "commit1_time": "2023-01-02 11:12:19 +0000",
      "commit1_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-eager]": 0.007349947207223843,
        "test_eval[resnet50_quantized_qat-cpu-eager]": 0.007411844153992004
      },
      "commit2": "34f2d3e6ae5",
      "commit2_time": "2023-01-02 15:53:36 +0000",
      "commit2_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-eager]": 0.09527151132933795,
        "test_eval[resnet50_quantized_qat-cpu-eager]": 0.10046712551265954
      }
    },
    {
      "commit1": "dfb651452a8",
      "commit1_time": "2023-01-03 14:24:36 +0000",
      "commit1_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.20553749243263156
      },
      "commit2": "688e3519707",
      "commit2_time": "2023-01-03 16:01:19 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.23750303009292112
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3848923289


Duplicate of https://github.com/pytorch/benchmark/issues/1357. Closed.
Repository: pytorch/benchmark
Issue: Improvements for benchmark score
Root cause: Unimplemented Feature Gap

This is more like an epic to track feedback relative to the first iteration of the benchmark score.  We should file specific tasks if there are actionable chunks we intend to do.

**Top Level Metrics**
- what should be produced besides a single score?
- how can we compare e.g. cuda vs cpu, or jit vs eager?
  - can we highlight which specific tests or categories are responsible for most of the score delta (relative to norm, or to another dataset)?

**Weight Hierarchy**
- overly complicated, not easy to understand or modify

**Implementation**
- multiple scripts and input/output files are clunky. Can we wrap a score configuration up inside a named function (e.g. compute_score_v1(input_timings) -> value)?

**Numerics**
- we should make sure weights are positive
- and ensure all the (weight, norm, measurement) are in a numerically stable range, e.g. enforce a minimum epsilon

**Other**
- names used in data and score are 'abbreviated' ones
- some of the metadata (e.g. model type, category, etc.) could be moved into Model class (in model/__init__.py)

@zdevito @zheng-xq  - feel free to continue discussion here.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230107+cu116'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230106+cu116

Base PyTorch commit: 33547bb58745a5deedaa36ca4e9363413f803e12

Affected PyTorch version: 2.0.0.dev20230107+cu116

Affected PyTorch commit: faed4db4971af151e3dba7233ae49f9c0149dc18

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -14.84012%
- test_eval[mobilenet_v2_quantized_qat-cpu-eager]: -92.21094%
- test_eval[pytorch_stargan-cpu-jit]: +8.34615%
- test_eval[pytorch_stargan-cpu-eager]: +7.29022%
- test_eval[resnet50_quantized_qat-cpu-eager]: -92.55970%


cc @xuzhao9

Result json: 
```
{
  "start": "33547bb58745a5deedaa36ca4e9363413f803e12",
  "end": "faed4db4971af151e3dba7233ae49f9c0149dc18",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "f2199709908",
      "commit1_time": "2023-01-06 19:06:48 +0000",
      "commit1_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-eager]": 0.09521561793517322,
        "test_eval[resnet50_quantized_qat-cpu-eager]": 0.10040152366273106
      },
      "commit2": "b3603f81290",
      "commit2_time": "2023-01-06 19:56:35 +0000",
      "commit2_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-eager]": 0.007348496507662926,
        "test_eval[resnet50_quantized_qat-cpu-eager]": 0.007436530902645642
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3863142929


Known revert, closed.
This is revert of https://github.com/pytorch/benchmark/issues/1357
Repository: pytorch/benchmark
Issue: Is it possible to run with Cuda 11.8?
Root cause: Environment Incompatibility

pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cu118

We haven't tested it on CUDA 11.8, but it should be runnable. 

It's strange because if I install cuda 11.8 torch not detected by script but with cuda11.6 binaries yes

> 

Can I have the error messages?
Make sure you have installed the domain libraries, including torchvision, torchtext, and torchaudio.

```
conda create -n torchbenchmark python=3.8
conda activate torchbenchmark
conda install -y git-lfs
conda install -y -c pytorch magma-cuda118
conda install pytorch torchvision torchtext torchaudio pytorch-cuda=11.8 -c pytorch-nightly -c nvidia
```
solved
Repository: pytorch/benchmark
Issue: Recent PyTorch nightly builds breaks DALLE2_pytorch code
Root cause: External Dependency Breakage

Example run: https://github.com/pytorch/benchmark/actions/runs/3896093654/jobs/6671547864

This is reproducible on PT 20230112 CUDA 11.6:

```
$ python run.py DALLE2_pytorch -d cuda -t train
Running train method from DALLE2_pytorch on cuda in eager mode with input batch size 4.
Traceback (most recent call last):
  File "/fsx/users/xzhao9/benchmark/run.py", line 338, in <module>
    run_one_step(test, model=m, export_metrics_file=export_metrics_file,
  File "/fsx/users/xzhao9/benchmark/run.py", line 96, in run_one_step
    func()
  File "/fsx/users/xzhao9/benchmark/torchbenchmark/util/model.py", line 277, in invoke
    self.train()
  File "/fsx/users/xzhao9/benchmark/torchbenchmark/models/DALLE2_pytorch/__init__.py", line 104, in train
    loss.backward()
  File "/data/home/xzhao9/cluster/miniconda3/envs/py310/lib/python3.10/site-packages/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/data/home/xzhao9/cluster/miniconda3/envs/py310/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
```

This problem exists in 20230101, but doesn't exist in 20221222 (https://github.com/pytorch/benchmark/actions/runs/3760224486/jobs/6390700236).

Fixed by https://github.com/pytorch/pytorch/commit/97b7e4cdd54bfef0a8965ccdef832b933645fcfc
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230113+cu116'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230112+cu116

Base PyTorch commit: 5766764d6cf9b14380c1b6ef5c2fe5392722e37b

Affected PyTorch version: 2.0.0.dev20230113+cu116

Affected PyTorch commit: fbbb19599a1d162e5927542ed251fd2ba63d5163

Affected Tests:
- test_eval[resnet50_quantized_qat-cpu-jit]: -16.68465%
- test_train[timm_vision_transformer-cpu-eager]: -7.31114%


cc @xuzhao9

Result json: 
```
{
  "start": "5766764d6cf9b14380c1b6ef5c2fe5392722e37b",
  "end": "fbbb19599a1d162e5927542ed251fd2ba63d5163",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "5766764d6cf",
      "commit1_time": "2023-01-12 01:54:34 +0000",
      "commit1_digest": {
        "test_eval[resnet50_quantized_qat-cpu-jit]": 0.004590594053059538,
        "test_train[timm_vision_transformer-cpu-eager]": 0.20478388655465096
      },
      "commit2": "e1bcbbf18ca",
      "commit2_time": "2023-01-12 02:14:28 +0000",
      "commit2_digest": {
        "test_eval[resnet50_quantized_qat-cpu-jit]": 0.004188317870316298
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/3913486919


Root cause commit: e1bcbbf18ca . Looks like a valid speedup.

Confirmed valid, closed.
Repository: pytorch/benchmark
Issue: Add torchvision maskrcnn (enable jit, dynamic shapes) 
Root cause: Unimplemented Feature Gap

The 'maskrcnn-benchmark' model has already been added to the benchmark, but it doesn't easily JIT, and we want dynamic shape models to go through PE/TE.  

This model is known to jit compile, and should be added to the benchmark
https://github.com/pytorch/vision/blob/master/torchvision/models/detection/mask_rcnn.py
see this test for the exact configuration that is tested with JIT
https://github.com/pytorch/vision/blob/master/test/test_models.py#L68

ideally, reuse coco dataset artifacts that are checked in already from the other maskrcnn.

follow general instructions in[ ADDING_MODELS.md](https://github.com/pytorch/benchmark/blob/master/torchbenchmark/models/ADDING_MODELS.md)


Added in https://github.com/pytorch/benchmark/pull/547, but it still doesn't jit (see the ci failures for details https://app.circleci.com/pipelines/github/pytorch/benchmark/2731/workflows/4f549e88-efac-4d6d-861c-4a954f31f050/jobs/2814)

Closed