zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Add PyTorch-UNet model to benchmark repo
Root cause: Unimplemented Feature Gap

We're building a suite of pytorch benchmarks by forking popular open-source models and modifying them to conform to a common API that facilitates a central tool installing, running, and collecting measurements.

Benchmark Repo: https://github.com/pytorch/benchmark
Instructions: https://github.com/pytorch/benchmark/blob/master/models/ADDING_MODELS.md
Source model: https://github.com/supriyar/Pytorch-UNet

The UNet model has already been forked and modified in a previous phase of this process, but the API (install.py, hubconf.py) had not yet been standardized.  Copy the repo into benchmark.git, follow the latest instructions in the ADDING_MODELS readme to add the new API. Note, the 'install.sh' and 'run.sh' scripts added in the previous effort would serve as good hints on how to proceed.


@wconstab I'll try to do this one! 

Hi @Krovatkin @wconstab are you still interested in doing this? I was looking for a bootcamp task to work on tomorrow and I've already worked with this specific Unet implementation in the past

@msaroufim, yes, sounds like a great bootcamp task! Unfortunately, I didn't get very far :(

Closing issue per merged PR above
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220511+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220510+cu113

Base PyTorch commit: 4ceac494259e6b7d57e721793fb0a9ad598a0e12

Affected PyTorch version: 1.12.0.dev20220511+cu113

Affected PyTorch commit: fc4f8b5edec69c0ddc63e5210d3f34724d4bd672

Affected Tests:
- test_eval[drq-cpu-eager]: +13.52097%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2309843806
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220512+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220511+cu113

Base PyTorch commit: fc4f8b5edec69c0ddc63e5210d3f34724d4bd672

Affected PyTorch version: 1.12.0.dev20220512+cu113

Affected PyTorch commit: 188854eeaf148b685e54a438fa7b9a20d3a28e33

Affected Tests:
- test_train[timm_regnet-cpu-eager]: -13.61988%
- test_train[timm_vision_transformer-cpu-eager]: +10.73988%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465311747

Empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220512+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220511+cu113

Base PyTorch commit: fc4f8b5edec69c0ddc63e5210d3f34724d4bd672

Affected PyTorch version: 1.12.0.dev20220512+cu113

Affected PyTorch commit: 188854eeaf148b685e54a438fa7b9a20d3a28e33

Affected Tests:
- test_eval[drq-cpu-eager]: -11.50314%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465314013

Empty result, closed.
Repository: pytorch/benchmark
Issue: Add PyText model to benchmark repo
Root cause: Unimplemented Feature Gap

We're building a suite of pytorch benchmarks by forking popular open-source models and modifying them to conform to a common API that facilitates a central tool installing, running, and collecting measurements.

Benchmark Repo: https://github.com/pytorch/benchmark
Instructions: https://github.com/pytorch/benchmark/blob/master/torchbenchmark/models/ADDING_MODELS.md
Source model: https://github.com/facebookresearch/pytext

After finishing the install.py and __init__.py, prep a github PR for review.

Closing the issue since there has been no active development in the pytest repo. We should add models from fairseq repo instead.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220522+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220521+cu113

Base PyTorch commit: 47834679ba2f869e66450a74e2add4c04f0006e9

Affected PyTorch version: 1.13.0.dev20220522+cu113

Affected PyTorch commit: 5f1b0a4f48bad11907031ff473b99558ae703593

Affected Tests:
- test_train[timm_regnet-cpu-eager]: +16.21107%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465316513

Empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220522+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220521+cu113

Base PyTorch commit: 47834679ba2f869e66450a74e2add4c04f0006e9

Affected PyTorch version: 1.13.0.dev20220522+cu113

Affected PyTorch commit: 5f1b0a4f48bad11907031ff473b99558ae703593

Affected Tests:
- test_eval[drq-cpu-eager]: -7.41738%
- test_eval[mobilenet_v3_large-cpu-jit]: -10.45153%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465320266

Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220523+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220522+cu113

Base PyTorch commit: 5f1b0a4f48bad11907031ff473b99558ae703593

Affected PyTorch version: 1.13.0.dev20220523+cu113

Affected PyTorch commit: d4345ed0a6c06b1e489e41c219f94d26d3014ce6

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +73.26583%
- test_train[timm_regnet-cpu-eager]: -12.09723%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465323184

Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220524+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220523+cu113

Base PyTorch commit: d4345ed0a6c06b1e489e41c219f94d26d3014ce6

Affected PyTorch version: 1.13.0.dev20220524+cu113

Affected PyTorch commit: c7ce4fcc619fab5c82071eac934b505b396ee015

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -43.43333%
- test_train[timm_regnet-cpu-eager]: +14.29439%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465728543

Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220524+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220523+cu113

Base PyTorch commit: d4345ed0a6c06b1e489e41c219f94d26d3014ce6

Affected PyTorch version: 1.13.0.dev20220524+cu113

Affected PyTorch commit: c7ce4fcc619fab5c82071eac934b505b396ee015

Affected Tests:
- test_eval[densenet121-cpu-eager]: +9.15684%
- test_eval[yolov3-cpu-eager]: +31.16148%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2472658512

Empty result, closed.
Repository: pytorch/benchmark
Issue: Add StarGAN model to benchmark repo
Root cause: Unimplemented Feature Gap

We're building a suite of pytorch benchmarks by forking popular open-source models and modifying them to conform to a common API that facilitates a central tool installing, running, and collecting measurements.

Benchmark Repo: https://github.com/pytorch/benchmark
Instructions: https://github.com/pytorch/benchmark/blob/master/models/ADDING_MODELS.md
Source model: https://github.com/zou3519/stargan

The StarGAN model has already been forked and modified in a previous phase of this process, but the API (install.py, hubconf.py) had not yet been standardized.  Copy the repo below and follow the latest instructions in the playbook to add the new API.  Note, the 'install.sh' and 'run.sh' scripts added in the previous effort would serve as good hints on how to proceed.

note in this case it looks like there are no steps to do for installation, but the install.py file should still be present as a no-op for the API sake.


@wconstab pls review PR
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220525+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220524+cu113

Base PyTorch commit: c7ce4fcc619fab5c82071eac934b505b396ee015

Affected PyTorch version: 1.13.0.dev20220525+cu113

Affected PyTorch commit: d450034f240681f5133bc61ddc9c0d195c527b59

Affected Tests:
- test_train[timm_regnet-cpu-eager]: -16.81248%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465727038

Empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220525+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220524+cu113

Base PyTorch commit: c7ce4fcc619fab5c82071eac934b505b396ee015

Affected PyTorch version: 1.13.0.dev20220525+cu113

Affected PyTorch commit: d450034f240681f5133bc61ddc9c0d195c527b59

Affected Tests:
- test_eval[yolov3-cpu-eager]: -25.52186%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2472656067

Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220526+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220525+cu113

Base PyTorch commit: d450034f240681f5133bc61ddc9c0d195c527b59

Affected PyTorch version: 1.13.0.dev20220526+cu113

Affected PyTorch commit: 032d1ace1de47e93edcc69b06da9d79636652991

Affected Tests:
- test_eval[timm_vovnet-cpu-eager]: +7.19286%
- test_eval[yolov3-cpu-eager]: +8.86020%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2472650885

Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220527+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220526+cu113

Base PyTorch commit: 032d1ace1de47e93edcc69b06da9d79636652991

Affected PyTorch version: 1.13.0.dev20220527+cu113

Affected PyTorch commit: d98a8148b66fb6f74ce1b437d6bde4f24208bb78

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +56.95523%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465725545

Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220527+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220526+cu113

Base PyTorch commit: 032d1ace1de47e93edcc69b06da9d79636652991

Affected PyTorch version: 1.13.0.dev20220527+cu113

Affected PyTorch commit: d98a8148b66fb6f74ce1b437d6bde4f24208bb78

Affected Tests:
- test_eval[timm_vovnet-cpu-eager]: -7.20525%
- test_train[yolov3-cuda-eager]: +21.77338%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2472649109

Empty result, closed.
Repository: pytorch/benchmark
Issue: AttributeError: 'tuple' object has no attribute 'keys' when loading mobilenet_v2_quantized_qat
Root cause: External Dependency Breakage

Hi, I was trying to load model `mobilenet_v2_quantized_qat` from torchbenchmark, my repro looks like the following:
```
    Model = torchbenchmark.load_model_by_name(name)
    torch_model = Model(test="eval", device="cpu")
```

But the second line throws the above error with the stacktrace below:
```
  File "test_tvm.py", line 117, in get_scripted
    torch_model = Model(test="eval", device="cpu")
  File "/home/yj/benchmark/torchbenchmark/util/model.py", line 14, in __call__
    obj = type.__call__(cls, *args, **kwargs)
  File "/home/yj/benchmark/torchbenchmark/models/mobilenet_v2_quantized_qat/__init__.py", line 28, in __init__
    self.prep_qat_train()  # config+prepare steps are required for both train and eval
  File "/home/yj/benchmark/torchbenchmark/models/mobilenet_v2_quantized_qat/__init__.py", line 35, in prep_qat_train
    self.model = quantize_fx.prepare_qat_fx(self.model, qconfig_dict, self.example_inputs)
  File "/home/yj/anaconda3/lib/python3.8/site-packages/torch/ao/quantization/quantize_fx.py", line 564, in prepare_qat_fx
    return _prepare_fx(
  File "/home/yj/anaconda3/lib/python3.8/site-packages/torch/ao/quantization/quantize_fx.py", line 203, in _prepare_fx
    check_is_valid_prepare_custom_config_dict(prepare_custom_config_dict)
  File "/home/yj/anaconda3/lib/python3.8/site-packages/torch/ao/quantization/fx/qconfig_utils.py", line 223, in check_is_valid_prepare_custom_config_dict
    check_is_valid_config_dict(prepare_custom_config_dict,
  File "/home/yj/anaconda3/lib/python3.8/site-packages/torch/ao/quantization/fx/qconfig_utils.py", line 180, in check_is_valid_config_dict
    for k in config_dict.keys():
AttributeError: 'tuple' object has no attribute 'keys'
```

i am using torch==1.11.0 stable with cuda 11.3. I wonder if this is known issue for quantized models in torchbenchmark. Thanks!

@shingjan This is likely because https://github.com/pytorch/pytorch/pull/77608, which includes a bc-breaking change of the interface `prepare_qat_fx()`. Can you retry with the latest nightly and see if the error is gone?

@xuzhao9 Thanks! Can confirm that using pytorch nightly will solve the above issue.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220604+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220603+cu113

Base PyTorch commit: 4bb8db85e924b1850d5cf1709a2a0bb7deca603d

Affected PyTorch version: 1.13.0.dev20220604+cu113

Affected PyTorch commit: 7860ce5b79f28d601f31aeade400d9873137fee4

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -34.42520%
- test_eval[pytorch_stargan-cpu-eager]: -12.15910%


cc @xuzhao9


Same as https://github.com/pytorch/benchmark/issues/943, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220605+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220603+cu113

Base PyTorch commit: 4bb8db85e924b1850d5cf1709a2a0bb7deca603d

Affected PyTorch version: 1.13.0.dev20220605+cu113

Affected PyTorch commit: 7860ce5b79f28d601f31aeade400d9873137fee4

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -14.43667%
- test_train[pytorch_stargan-cpu-eager]: -7.11145%
- test_eval[pytorch_stargan-cpu-eager]: -10.76033%
- test_train[timm_regnet-cpu-eager]: +22.73357%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465722443

Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220606+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220605+cu113

Base PyTorch commit: 7860ce5b79f28d601f31aeade400d9873137fee4

Affected PyTorch version: 1.13.0.dev20220606+cu113

Affected PyTorch commit: e6cc2e8d384d3e045b1529c89bcb0460883691e9

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +35.71439%
- test_eval[pytorch_stargan-cpu-jit]: +8.17160%
- test_eval[pytorch_stargan-cpu-eager]: +9.40047%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2465720215

Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220607+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220606+cu113

Base PyTorch commit: e6cc2e8d384d3e045b1529c89bcb0460883691e9

Affected PyTorch version: 1.13.0.dev20220607+cu113

Affected PyTorch commit: b40c751c33439bb8721fddb85c25b893ba73ada7

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -43.83936%
- test_train[timm_regnet-cpu-eager]: -17.25634%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2464113790

Empty result, closed
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220607+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220606+cu113

Base PyTorch commit: e6cc2e8d384d3e045b1529c89bcb0460883691e9

Affected PyTorch version: 1.13.0.dev20220607+cu113

Affected PyTorch commit: b40c751c33439bb8721fddb85c25b893ba73ada7

Affected Tests:
- test_eval[yolov3-cpu-eager]: +8.65471%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2472646847

Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220609+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220607+cu113

Base PyTorch commit: b40c751c33439bb8721fddb85c25b893ba73ada7

Affected PyTorch version: 1.13.0.dev20220609+cu113

Affected PyTorch commit: bfaa187fb083aea66804673de033b17506061de2

Affected Tests:
- test_train[timm_regnet-cpu-eager]: +22.16191%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2470028282

Empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220610+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220609+cu113

Base PyTorch commit: bfaa187fb083aea66804673de033b17506061de2

Affected PyTorch version: 1.13.0.dev20220610+cu113

Affected PyTorch commit: 70d6446a3def513b8082d243d7996ef86c2787a6

Affected Tests:
- test_train[timm_resnest-cpu-eager]: +8.54433%
- test_eval[tts_angular-cpu-eager]: -9.93846%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2505544692

Commit: eb856daf0f5
Repository: pytorch/benchmark
Issue: NVFuser is the default fuser for Nvidia GPU now
Root cause: Configuration Error

Codes for switching fuser need to be updated.

@LSC527 Thanks for raising this issue. By default we are not applying any fuser to the model, and users can specify their fusers using the option `--fuser` (https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/extra_args.py#L92). Could you clarify on how we should update the code for switching fuser?

```python run.py``` looks good and needs no update. But ```pytest test_bench.py``` config fuser option in ```confest.py``` and did nothing when fuser is the default "te". This seems to enable nvfuser by default.
https://github.com/pytorch/benchmark/blob/f30d537c95702b59515c6d65c99e1b0f41ecd994/conftest.py#L24

Please help review https://github.com/pytorch/benchmark/pull/963, thanks!
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220615+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220614+cu113

Base PyTorch commit: 18305e30a785a0336837a3e4406b84e3bf36e727

Affected PyTorch version: 1.13.0.dev20220615+cu113

Affected PyTorch commit: bad7720dde598d4300b6523e2c06e61923a2820a

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -43.13159%
- test_train[timm_regnet-cpu-eager]: -17.59932%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2503942763

unrelated commit: 48242224725. closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220616+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220615+cu113

Base PyTorch commit: bad7720dde598d4300b6523e2c06e61923a2820a

Affected PyTorch version: 1.13.0.dev20220616+cu113

Affected PyTorch commit: 86606fbe22c76d71851a38003462e9b6428b47d3

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +16.68671%
- test_train[timm_regnet-cpu-eager]: +20.78271%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2510613387
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220617+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220616+cu113

Base PyTorch commit: 86606fbe22c76d71851a38003462e9b6428b47d3

Affected PyTorch version: 1.13.0.dev20220617+cu113

Affected PyTorch commit: 9d419e81e2cf64504126652def4eec46b72b75c5

Affected Tests:
- test_train[LearningToPaint-cpu-eager]: -8.19571%
- test_eval[demucs-cpu-jit]: +7.95337%
- test_eval[mnasnet1_0-cpu-jit]: +12.47160%
- test_eval[resnet18-cpu-jit]: +30.00438%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2516945166
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220617+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220616+cu113

Base PyTorch commit: 86606fbe22c76d71851a38003462e9b6428b47d3

Affected PyTorch version: 1.13.0.dev20220617+cu113

Affected PyTorch commit: 9d419e81e2cf64504126652def4eec46b72b75c5

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: +10.91041%
- test_eval[yolov3-cpu-eager]: +8.05122%


cc @xuzhao9


Empty result, closed. Workflow: https://github.com/pytorch/benchmark/actions/runs/2518087536
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220618+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220617+cu113

Base PyTorch commit: 9d419e81e2cf64504126652def4eec46b72b75c5

Affected PyTorch version: 1.13.0.dev20220618+cu113

Affected PyTorch commit: 1432a3d6acc7ac58269e5ea7ef6b2373e9a3dbf6

Affected Tests:
- test_train[LearningToPaint-cpu-eager]: +12.14624%
- test_eval[demucs-cpu-jit]: -12.56802%
- test_eval[mnasnet1_0-cpu-jit]: -23.11026%
- test_eval[pytorch_stargan-cpu-jit]: -7.63632%
- test_train[timm_regnet-cpu-eager]: -20.24115%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2521158882
root cause commit:  c8b073c5c57 (test_eval[mnasnet1_0-cpu-jit]: -23.11026%)

Unrelated. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220618+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220617+cu113

Base PyTorch commit: 9d419e81e2cf64504126652def4eec46b72b75c5

Affected PyTorch version: 1.13.0.dev20220618+cu113

Affected PyTorch commit: 1432a3d6acc7ac58269e5ea7ef6b2373e9a3dbf6

Affected Tests:
- test_eval[mobilenet_v3_large-cpu-jit]: -8.74662%
- test_eval[yolov3-cpu-eager]: -9.03092%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2521828209
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220619+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220618+cu113

Base PyTorch commit: 1432a3d6acc7ac58269e5ea7ef6b2373e9a3dbf6

Affected PyTorch version: 1.13.0.dev20220619+cu113

Affected PyTorch commit: 0545c85f744cadc6f06616f15748052cd7cebe20

Affected Tests:
- test_eval[demucs-cpu-jit]: +8.20755%
- test_eval[pytorch_stargan-cpu-jit]: +13.95172%
- test_eval[resnet18-cpu-jit]: -21.19655%
- test_train[timm_regnet-cpu-eager]: +25.92665%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2524722375
empty result, closed.