zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220410+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220409+cu113

Base PyTorch commit: 58a44523c1f74093ad9a05513a32c45a109b6c55

Affected PyTorch version: 1.12.0.dev20220410+cu113

Affected PyTorch commit: f42bdff0166ef503c844353c32b6725391e440ac

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +35.66740%


cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/2151987786

Empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220410+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220409+cu113

Base PyTorch commit: 58a44523c1f74093ad9a05513a32c45a109b6c55

Affected PyTorch version: 1.12.0.dev20220410+cu113

Affected PyTorch commit: f42bdff0166ef503c844353c32b6725391e440ac

Affected Tests:
- test_train[hf_Bert-cpu-eager]: -7.08611%


cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/2152259375

Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220411+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220410+cu113

Base PyTorch commit: f42bdff0166ef503c844353c32b6725391e440ac

Affected PyTorch version: 1.12.0.dev20220411+cu113

Affected PyTorch commit: caa28ff4959c7cf7165dabf1ffae7c233b8e4b61

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -25.13017%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: -15.32507%
- test_train[timm_vision_transformer-cpu-eager]: -9.88254%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2152270179

Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220411+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220410+cu113

Base PyTorch commit: f42bdff0166ef503c844353c32b6725391e440ac

Affected PyTorch version: 1.12.0.dev20220411+cu113

Affected PyTorch commit: caa28ff4959c7cf7165dabf1ffae7c233b8e4b61

Affected Tests:
- test_train[yolov3-cuda-eager]: -17.79878%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2152265870

Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220412+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220411+cu113

Base PyTorch commit: caa28ff4959c7cf7165dabf1ffae7c233b8e4b61

Affected PyTorch version: 1.12.0.dev20220412+cu113

Affected PyTorch commit: 25aa251f37b74ea8354a3e9366912180a6c84613

Affected Tests:
- test_eval[mobilenet_v2_quantized_qat-cpu-jit]: +7.24508%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +16.91360%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2163949112
```
{
  "start": "caa28ff4959c7cf7165dabf1ffae7c233b8e4b61",
  "end": "25aa251f37b74ea8354a3e9366912180a6c84613",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "caa28ff4959",
      "commit1_time": "2022-04-11 01:41:07 +0000",
      "commit1_digest": {
        "test_eval[mobilenet_v2_quantized_qat-cpu-jit]": 0.004451133939437568,
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.13092206344590523
      },
      "commit2": "9855f1271c3",
      "commit2_time": "2022-04-11 03:50:04 +0000",
      "commit2_digest": {
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.14878578553907573
      }
    }
  ]
}
```

Unrelated, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220412+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220411+cu113

Base PyTorch commit: caa28ff4959c7cf7165dabf1ffae7c233b8e4b61

Affected PyTorch version: 1.12.0.dev20220412+cu113

Affected PyTorch commit: 25aa251f37b74ea8354a3e9366912180a6c84613

Affected Tests:
- test_train[yolov3-cuda-eager]: +21.79654%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2164047734 
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220413+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220412+cu113

Base PyTorch commit: 25aa251f37b74ea8354a3e9366912180a6c84613

Affected PyTorch version: 1.12.0.dev20220413+cu113

Affected PyTorch commit: 4afe2db641c526800fb8b33532a920be625f3cf4

Affected Tests:
- test_train[yolov3-cuda-eager]: -17.85576%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2164048178
Result:
```
{
  "start": "25aa251f37b74ea8354a3e9366912180a6c84613",
  "end": "4afe2db641c526800fb8b33532a920be625f3cf4",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v1.0",
  "result": [
    {
      "commit1": "04db1b874ff",
      "commit1_time": "2022-04-12 18:26:33 +0000",
      "commit1_digest": {
        "test_train[yolov3-cuda-eager]": 9.13075909030158
      },
      "commit2": "f5f64526da6",
      "commit2_time": "2022-04-12 18:33:29 +0000",
      "commit2_digest": {
        "test_train[yolov3-cuda-eager]": 11.134152245509904
      }
    }
  ]
}
```

Unrelated, closed.
Repository: pytorch/benchmark
Issue: Remove tacotron2 model
Root cause: External Dependency Breakage

I propose we remove the [tacotron2](https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models/tacotron2) model because:
1. the last update time to this model is ~2 years ago, which seems it has stopped development;
2. it hard depends on [librosa](https://github.com/NVIDIA/tacotron2/blob/master/requirements.txt#L5), which depends on [numba](https://pypi.org/project/numba/) and [numpy < 1.22](https://pypi.org/project/numba/). This breaks the current pytorch numpy api compatibility.

I think we should pin numpy version instead of removing this model, similar to https://github.com/pytorch/builder/blob/main/wheel/build_wheel.sh#L157

How hard would it be to upgrade that dependency?  What breaks if we upgrade it?
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220413+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220412+cu113

Base PyTorch commit: 25aa251f37b74ea8354a3e9366912180a6c84613

Affected PyTorch version: 1.12.0.dev20220413+cu113

Affected PyTorch commit: 4afe2db641c526800fb8b33532a920be625f3cf4

Affected Tests:
- test_train[yolov3-cuda-eager]: -17.96453%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2169718375, empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220416+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220415+cu113

Base PyTorch commit: cc1902a5ed64e5e3b956c5cc6fcbdcd45630f7b8

Affected PyTorch version: 1.12.0.dev20220416+cu113

Affected PyTorch commit: 5dcbcc6de83b552f43415ef60bb2e771bdbc0026

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +7.09007%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2177997802 
Result empty. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220419+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220418+cu113

Base PyTorch commit: f4d89aa28e182bd7768a997a70b033748591ceac

Affected PyTorch version: 1.12.0.dev20220419+cu113

Affected PyTorch commit: 736f0d0e46ad70f2382a9035bc72f491970d43f5

Affected Tests:
- test_train[timm_vision_transformer-cpu-eager]: -9.29122%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2191311096
Empty result, closed.
Repository: pytorch/benchmark
Issue: Improve the coverage of `gen_inputs()` interface
Root cause: Unimplemented Feature Gap

We added the `gen_inputs()` interface and we would like to increase its coverage towards 100%.
`gen_inputs(num_batches) -> Tuple[Generator, Optional[int]]` returns a generator and an optional int value.
The int value indicates how many batches the generator can return. For example, if the model is using coco128 dataset, and batch size is 1, num_batches is 1, then the length of the generator is 128. It could also be `None`, if the model uses randomly generated data instead of a mini-dataset, which means the length of the generator is infinite.

The generator, once called, should return a list, whose length could `num_batches` or smaller (if the last batch). Each element in the list should be able to work as input to the model, similar to the `get_module()` interface.

- [ ] Bert_pytorch
- [ ] Background_Matting
- [ ] LearningToPaint
- [ ] Super_SloMo
- [x] alexnet
- [ ] attention_is_all_you_need_pytorch
- [ ] dcgan
- [ ] demucs
- [x] densenet121
- [ ] detectron2_maskrcnn
- [ ] dlrm
- [ ] drq
- [ ] fambench_dlrm
- [ ] fambench_xlmr
- [ ] fastNLP_Bert
- [ ] hf_Albert
- [ ] hf_Bart
- [ ] hf_Bert
- [ ] hf_BigBird
- [ ] hf_DistilBert
- [ ] hf_GPT2
- [ ] hf_Longformer
- [ ] hf_Reformer
- [ ] hf_T5
- [ ] maml
- [ ] maml_omniglot
- [x] mnasnet1_0
- [x] mobilenet_v2
- [x] mobilenet_v2_quantized_qat
- [x] mobilenet_v3_large
- [ ] moco
- [ ] nvidia_deeprecommender
- [ ] opacus_cifar10
- [x] pplbench_beanmachine
- [ ] pyhpc_equation_of_state
- [ ] pyhpc_isoneutral_mixing
- [ ] pyhpc_turbulent_kinetic_energy
- [ ] pytorch_CycleGAN_and_pix2pix
- [ ] pytorch_stargan
- [ ] pytorch_struct
- [ ] pytorch_unet
- [x] resnet18
- [x] resnet50
- [x] resnet50_quantized_qat
- [x] resnext50_32x4d
- [x] shufflenet_v2_x1_0
- [ ] soft_actor_critic
- [ ] speech_transformer
- [x] squeezenet1_1
- [ ] tacotron2
- [x] timm_efficientdet
- [x] timm_efficientnet
- [x] timm_nfnet
- [x] timm_regnet
- [x] timm_resnest
- [x] timm_vision_transformer
- [x] timm_vovnet
- [ ] tts_angular
- [x] vgg16
- [ ] vision_maskrcnn
- [ ] yolov3

Note that you don't need to finish all models for this task - e.g., adding coverage to 5-10 models is very much appreciated.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220420+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220419+cu113

Base PyTorch commit: 736f0d0e46ad70f2382a9035bc72f491970d43f5

Affected PyTorch version: 1.12.0.dev20220420+cu113

Affected PyTorch commit: d0af05f931e271131e1e207bcc47235059cfd723

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +7.54689%
- test_train[timm_vision_transformer-cpu-eager]: +8.72316%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2197410743 
Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220426+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220424+cu113

Base PyTorch commit: 77f23d64607142c0cba47f5148ebc1959c9de366

Affected PyTorch version: 1.12.0.dev20220426+cu113

Affected PyTorch commit: e48b29b1fb1510535efa2a00b6501d5455556aa2

Affected Tests:
- test_eval[mobilenet_v2_quantized_qat-cpu-jit]: -8.60954%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2228508226
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220427+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220426+cu113

Base PyTorch commit: e48b29b1fb1510535efa2a00b6501d5455556aa2

Affected PyTorch version: 1.12.0.dev20220427+cu113

Affected PyTorch commit: ec62901a2c38b63d12843e0f079bdeb7644d8714

Affected Tests:
- test_eval[mobilenet_v2_quantized_qat-cpu-jit]: +9.03923%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: -7.73751%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2234894147
Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220427+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220426+cu113

Base PyTorch commit: e48b29b1fb1510535efa2a00b6501d5455556aa2

Affected PyTorch version: 1.12.0.dev20220427+cu113

Affected PyTorch commit: ec62901a2c38b63d12843e0f079bdeb7644d8714

Affected Tests:
- test_train[dlrm-cpu-eager]: -9.56331%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2235847484
Root cause: 2291960d3f0


Unrelated PR. Closed.
Repository: pytorch/benchmark
Issue: Make sure we have prefetched the data in `__init__()`
Root cause: Architectural Constraint

Since the core model set only measures the iteration time, it assumes the data has been copied from DRAM to GPU memory.
However, some of our models still break this assumption and has `.to(self.device)` calls in their `get_module()` function. For example, https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/Background_Matting/__init__.py#L120

We need to remove these calls in `get_module()` and add them to `__init__()` to pass the device test.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220428+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220427+cu113

Base PyTorch commit: ec62901a2c38b63d12843e0f079bdeb7644d8714

Affected PyTorch version: 1.12.0.dev20220428+cu113

Affected PyTorch commit: 177ea46332fcde48df5ce0c0efe1552991a5548a

Affected Tests:
- test_eval[mobilenet_v2_quantized_qat-cpu-jit]: -7.55342%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +9.75391%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2241066945
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220428+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220427+cu113

Base PyTorch commit: ec62901a2c38b63d12843e0f079bdeb7644d8714

Affected PyTorch version: 1.12.0.dev20220428+cu113

Affected PyTorch commit: 177ea46332fcde48df5ce0c0efe1552991a5548a

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cpu-jit]: +7.05594%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2242047378
Empty result, closed.
Repository: pytorch/benchmark
Issue: Fix models that generate unstable inference results
Root cause: Algorithmic Error

We should also add a stableness test for models:
- pyhpc_turbulent_kinetic_energy
- pyhpc_isoneutral_mixing

```
$ python torchbench.py -k pyhpc_turbulent_kinetic_energy -d cuda
XU: model is not using torchscript
cuda eval  pyhpc_turbulent_kinetic_energy     INCORRECT - Variation in Eager runs itself
```

For tke: looks like the non-determinism is from this line:
https://github.com/dionhaefner/pyhpc-benchmarks/blob/master/benchmarks/turbulent_kinetic_energy/tke_pytorch.py#L264



Submitted upstream issue: https://github.com/dionhaefner/pyhpc-benchmarks/issues/15
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220429+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220428+cu113

Base PyTorch commit: 177ea46332fcde48df5ce0c0efe1552991a5548a

Affected PyTorch version: 1.12.0.dev20220429+cu113

Affected PyTorch commit: ac31e5d4a3ba20315b5bae26411252c72e0f5c0b

Affected Tests:
- test_train[timm_vision_transformer-cpu-eager]: +8.43254%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2246991603. Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220430+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220429+cu113

Base PyTorch commit: ac31e5d4a3ba20315b5bae26411252c72e0f5c0b

Affected PyTorch version: 1.12.0.dev20220430+cu113

Affected PyTorch commit: 2540f866ff1eff10dbed7ca47ea9c432e8583da2

Affected Tests:
- test_train[timm_vision_transformer-cpu-eager]: -7.12141%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2250772322,
empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220503+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220430+cu113

Base PyTorch commit: 2540f866ff1eff10dbed7ca47ea9c432e8583da2

Affected PyTorch version: 1.12.0.dev20220503+cu113

Affected PyTorch commit: 201ddafc22e22c387b4cd654f397e05354d73d09

Affected Tests:
- test_train[pytorch_stargan-cpu-eager]: +8.20977%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2265465818

Empty result. Closed (noise)
Repository: pytorch/benchmark
Issue: Slow CPU performance results in frustrating user experience
Root cause: Configuration Error

I've noticed that in the past few months the user experience for using TorchBench for correctness testing [TorchDynamo](https://github.com/facebookresearch/torchdynamo) has gotten much worse.

TorchBench used to run quickly, and I could check all models in minutes.  Now doing the same thing seems to take hours.

Doing some bisecting, it seems what has happened to make it so much slower is increasing batch sizes.  The new batch sizes seem tuned for GPU and reasonable for GPU, but result in a painful experience on CPU.

I'd propose we use different batch sizes on CPU and GPU, especially for inference where batch size is rather arbitrary.  We should target execution times <10 seconds, and ideally <1 second.

We should check if these smaller batch sizes result in different performance characteristics.  For example, turning a compute-bound model into an overhead bound one.  If not, giving TorchBench users faster time-to-signal would be a huge benefit.

Perhaps we could also have a non-default "large batch size" mode, for extra testing.

cc @xuzhao9 

Correct, I agree that we should use small batch sizes for CPU inference. In addition, we should have different scales of for GPU inference, because some users use small GPU, which default test batch size will be CUDA OOM. And for extra large testing, we should support multi-GPU test case.

However, for train usually we just use the batch size in the example repository or the original paper, so the batch size can't be arbitrarily adjusted.

I'm also noticing very low CPU utilization.  Usually only 1 thread active.  Did something change in how we setup threading?

> I'm also noticing very low CPU utilization. Usually only 1 thread active. Did something change in how we setup threading?

I don't think anything changed in setup threading. Can I get a command to reproduce this problem?


```
./torchbench.py --nothing -n100 -k alexnet
```

> ```
> ./torchbench.py --nothing -n100 -k alexnet
> ```
@jansel  For the low cpu utilization, I tried to stress testing alexnet using the newly added `--stress` option (https://github.com/pytorch/benchmark/pull/1002):
`$ python run.py alexnet --stress 200`

From htop, it looks like all cores are being used:
<img width="1883" alt="image" src="https://user-images.githubusercontent.com/502017/176758985-5a5c47d5-5310-4ee8-b9f1-7e42d717f8d7.png">

so I believe it is environment setup issues if you only observe 1 thread is being used. For example, if you are using `taskset`, you need to specify env `GOMP_CPU_AFFINITY` at the same time, to enable multi-threading.

Closed by https://github.com/pytorch/benchmark/pull/1003
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220504+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220430+cu113

Base PyTorch commit: 2540f866ff1eff10dbed7ca47ea9c432e8583da2

Affected PyTorch version: 1.12.0.dev20220504+cu113

Affected PyTorch commit: 201ddafc22e22c387b4cd654f397e05354d73d09

Affected Tests:
- test_train[pytorch_stargan-cpu-eager]: +7.12649%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2271458068
Empty result. Closed.
Repository: pytorch/benchmark
Issue: Add Stanza model to benchmark repo
Root cause: Unimplemented Feature Gap

We're building a suite of pytorch benchmarks by forking popular open-source models and modifying them to conform to a common API that facilitates a central tool installing, running, and collecting measurements.

The hub repo is no longer used, models are no longer added as git submodules, other parts remain largely the same.  Instructions are found in the README in the new repo

Benchmark Repo: https://github.com/pytorch/benchmark
Instructions: https://github.com/pytorch/benchmark/blob/master/torchbenchmark/models/ADDING_MODELS.md
Source model: https://github.com/ezyang/stanza

The Stanza model has already been forked and modified in a previous phase of this process, but the API (install.py, hubconf.py) had not yet been standardized.  Copy the repo into benchmark.git, follow the latest instructions in the ADDING_MODELS readme to add the new API.  Note, the 'install.sh' and 'run.sh' scripts added in the previous effort would serve as good hints on how to proceed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220506+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220505+cu113

Base PyTorch commit: 4baf7c0899a2fa9c3630613f37d5fc65971db21c

Affected PyTorch version: 1.12.0.dev20220506+cu113

Affected PyTorch commit: 040e2e04dd35c49e2b0faeeca964333f7cbc45b9

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -26.49873%
- test_train[timm_regnet-cpu-eager]: +15.75680%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2283268418
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220507+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220506+cu113

Base PyTorch commit: 040e2e04dd35c49e2b0faeeca964333f7cbc45b9

Affected PyTorch version: 1.12.0.dev20220507+cu113

Affected PyTorch commit: 901cb7c2e40123822c2d8e5cf4c5af782a4e22a1

Affected Tests:
- test_train[timm_regnet-cpu-eager]: -14.59824%
- test_train[timm_vision_transformer-cpu-eager]: -7.65783%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2287149131
Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220509+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220507+cu113

Base PyTorch commit: 901cb7c2e40123822c2d8e5cf4c5af782a4e22a1

Affected PyTorch version: 1.12.0.dev20220509+cu113

Affected PyTorch commit: e3dcd175f7aacb6aa2fb150b01335afd508c9d52

Affected Tests:
- test_train[timm_regnet-cpu-eager]: +17.54018%
- test_train[timm_vision_transformer-cpu-eager]: +13.23216%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2295965805
Empty result, closed.