zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Super_SloMo doesn't work when run from torchdynamo's benchmark runner
Root cause: Validation Gap

Failure:

```
(/scratch/ezyang/pytorch-scratch2-env) ezyang@devfair040:/scratch/ezyang/torchdynamo/benchmarks$ ./torchbench.py --no-skip  -n 1 -k Super_SloMo --devices cuda --nothing
cuda eval  Super_SloMo                        Traceback (most recent call last):
  File "./torchbench.py", line 363, in <module>
    main(TorchBenchmarkRunner(), original_dir)
  File "/raid/ezyang/torchdynamo/benchmarks/common.py", line 1304, in main
    runner.run_one_model(
  File "/raid/ezyang/torchdynamo/benchmarks/common.py", line 678, in run_one_model
    correct_result = model_iter_fn(
  File "./torchbench.py", line 345, in forward_pass
    return mod(*inputs)
  File "/raid/ezyang/pytorch-scratch2/torch/nn/modules/module.py", line 1186, in _call_impl
    return forward_call(*input, **kwargs)
  File "/raid/ezyang/benchmark/torchbenchmark/models/Super_SloMo/model_wrapper.py", line 34, in forward
    fCoeff = model.getFlowCoeff(trainFrameIndex, I0.device)
  File "/raid/ezyang/benchmark/torchbenchmark/models/Super_SloMo/slomo_model.py", line 324, in getFlowCoeff
    C11 = C00 = - (1 - (t[ind])) * (t[ind])
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)
```

taken from https://github.com/pytorch/torchdynamo/issues/519
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220723+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220722+cu113

Base PyTorch commit: 12cb26509a30077f4bd11972dea9baa03c8d943f

Affected PyTorch version: 1.13.0.dev20220723+cu113

Affected PyTorch commit: 35d97e21c8e2e28fae4a4744ce42f1544972ba1f

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +14.87600%
- test_eval[pytorch_stargan-cpu-jit]: -9.68235%
- test_train[timm_vision_transformer-cpu-eager]: -13.97631%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2724392776
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220724+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220723+cu113

Base PyTorch commit: 35d97e21c8e2e28fae4a4744ce42f1544972ba1f

Affected PyTorch version: 1.13.0.dev20220724+cu113

Affected PyTorch commit: 194255bb56a7bdcb5e6fb236d82b4897c2e5967c

Affected Tests:
- test_eval[demucs-cpu-jit]: +13.59394%
- test_eval[demucs-cpu-eager]: +9.84871%
- test_eval[mnasnet1_0-cpu-jit]: +11.48060%
- test_train[pytorch_stargan-cpu-jit]: +9.14749%
- test_eval[pytorch_stargan-cpu-jit]: +25.50260%
- test_train[pytorch_stargan-cpu-eager]: +7.58909%
- test_eval[pytorch_stargan-cpu-eager]: +21.73190%
- test_train[timm_resnest-cpu-eager]: +7.15929%
- test_train[timm_vision_transformer-cpu-eager]: +11.15969%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2727995282
Result:
```
{
  "start": "35d97e21c8e2e28fae4a4744ce42f1544972ba1f",
  "end": "194255bb56a7bdcb5e6fb236d82b4897c2e5967c",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "47b4ec8aa7d",
      "commit1_time": "2022-07-23 05:58:56 +0000",
      "commit1_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.1927311151055619
      },
      "commit2": "a3fc076f22a",
      "commit2_time": "2022-07-23 06:04:21 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.2200143493129872
      }
    }
  ]
}
```

Noise. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220725+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220724+cu113

Base PyTorch commit: 194255bb56a7bdcb5e6fb236d82b4897c2e5967c

Affected PyTorch version: 1.13.0.dev20220725+cu113

Affected PyTorch commit: 64094d81fe3cf512d3dfc6b2aabfd1754c86e87d

Affected Tests:
- test_train[speech_transformer-cuda-eager]: +11.28914%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2735455910
Empty result, closed.
Repository: pytorch/benchmark
Issue: Collect raw benchmark data in scuba rather than only summarized stats
Root cause: Unimplemented Feature Gap

- Possibly add new scuba table for the raw data
- may need to modify upload_scribe to handle new fields and new table/columns
- pytest_benchmark already has cmd line option to include the raw data in the .json file it outputs, but this needs to be enabled in the CI script that runs the benchmark.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220726+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220725+cu113

Base PyTorch commit: 64094d81fe3cf512d3dfc6b2aabfd1754c86e87d

Affected PyTorch version: 1.13.0.dev20220726+cu113

Affected PyTorch commit: 1223e94469682271c94ac3b6a0cbe31a3078e351

Affected Tests:
- test_eval[demucs-cpu-eager]: -9.59467%
- test_eval[mnasnet1_0-cpu-jit]: -8.02371%
- test_eval[pytorch_stargan-cpu-jit]: -16.60777%
- test_eval[pytorch_stargan-cpu-eager]: -16.61180%
- test_train[timm_resnest-cpu-eager]: +10.99967%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/runs/7525493477

Result:
```
{
  "start": "64094d81fe3cf512d3dfc6b2aabfd1754c86e87d",
  "end": "1223e94469682271c94ac3b6a0cbe31a3078e351",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "110cd724fce",
      "commit1_time": "2022-07-25 15:48:47 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.49167706762673336,
        "test_eval[pytorch_stargan-cpu-eager]": 0.4397951771854423
      },
      "commit2": "0e995f88eaf",
      "commit2_time": "2022-07-25 16:28:08 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.42888796848710625,
        "test_eval[pytorch_stargan-cpu-eager]": 0.40006959248567
      }
    }
  ]
}
```


Looks like to be noise. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220727+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220726+cu113

Base PyTorch commit: 1223e94469682271c94ac3b6a0cbe31a3078e351

Affected PyTorch version: 1.13.0.dev20220727+cu113

Affected PyTorch commit: 3cf9c3d8769fa3509c190a4f4c3dacd61ba154af

Affected Tests:
- test_eval[demucs-cpu-jit]: +9.72896%
- test_eval[demucs-cpu-eager]: +9.88960%
- test_eval[mnasnet1_0-cpu-jit]: +8.77273%
- test_train[pytorch_stargan-cpu-jit]: +7.50459%
- test_eval[pytorch_stargan-cpu-jit]: +22.09692%
- test_train[pytorch_stargan-cpu-eager]: +8.89600%
- test_eval[pytorch_stargan-cpu-eager]: +19.57506%
- test_train[timm_vision_transformer-cpu-eager]: -13.65048%


cc @xuzhao9


Result:
```
{
  "start": "1223e94469682271c94ac3b6a0cbe31a3078e351",
  "end": "3cf9c3d8769fa3509c190a4f4c3dacd61ba154af",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "2c2f1226744",
      "commit1_time": "2022-07-26 16:58:12 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.4502282046712935
      },
      "commit2": "7bdafed4f10",
      "commit2_time": "2022-07-26 16:58:42 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.4895474997814745
      }
    }
  ]
}
```

Root cause commit: https://github.com/pytorch/pytorch/commit/7bdafed4f10
@malfet @YifanShenSZ
From the PR itself I don't immediately recognize if this is a signal or noise. Any ideas?

@xuzhao9 this commit is already reverted, do you see the perf gain after revert?

@malfet We need to wait for tomorrow's nightly run to know. Will update by then.

I guess it's a noise since I was simply adding a nested tensor kernel for `matmul`... nothing advanced

Yeah looks like this is a noise. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220729+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220727+cu113

Base PyTorch commit: 3cf9c3d8769fa3509c190a4f4c3dacd61ba154af

Affected PyTorch version: 1.13.0.dev20220729+cu113

Affected PyTorch commit: 357b7d589c9c451f4850449b9313d8b857ae7140

Affected Tests:
- test_eval[demucs-cpu-jit]: -13.35844%
- test_eval[demucs-cpu-eager]: -8.15705%
- test_eval[mnasnet1_0-cpu-jit]: -20.04748%
- test_eval[pytorch_stargan-cpu-jit]: -17.44306%
- test_train[pytorch_stargan-cpu-eager]: -8.74479%
- test_eval[pytorch_stargan-cpu-eager]: -14.76180%
- test_eval[tts_angular-cuda-eager]: +9.71031%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2762172744

Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220730+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220729+cu113

Base PyTorch commit: 357b7d589c9c451f4850449b9313d8b857ae7140

Affected PyTorch version: 1.13.0.dev20220730+cu113

Affected PyTorch commit: dc53cb4536839d11ab9a32ec10a0ff0dd96a590f

Affected Tests:
- test_train[timm_resnest-cpu-eager]: +8.55009%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2766559835
Result: 
```
{
  "start": "357b7d589c9c451f4850449b9313d8b857ae7140",
  "end": "dc53cb4536839d11ab9a32ec10a0ff0dd96a590f",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "0b2566456f0",
      "commit1_time": "2022-07-29 23:28:58 +0000",
      "commit1_digest": {
        "test_train[timm_resnest-cpu-eager]": 1.0250291926437058
      },
      "commit2": "dc53cb45368",
      "commit2_time": "2022-07-30 00:21:29 +0000",
      "commit2_digest": {
        "test_train[timm_resnest-cpu-eager]": 1.1016900838236325
      }
    }
  ]
}
```

Unrelated. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220731+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220730+cu113

Base PyTorch commit: dc53cb4536839d11ab9a32ec10a0ff0dd96a590f

Affected PyTorch version: 1.13.0.dev20220731+cu113

Affected PyTorch commit: 32cf6c6fb0d98f747f576f52abe533449ae5da84

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +37.51750%
- test_eval[resnet18-cpu-jit]: +27.83286%
- test_train[timm_resnest-cpu-eager]: -10.30155%
- test_train[timm_vision_transformer-cpu-eager]: +10.58634%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2770270944
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220801+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220731+cu113

Base PyTorch commit: 32cf6c6fb0d98f747f576f52abe533449ae5da84

Affected PyTorch version: 1.13.0.dev20220801+cu113

Affected PyTorch commit: 53f56894ae0b747d2833ee168599afcfdd4be471

Affected Tests:
- test_eval[demucs-cpu-jit]: +7.69606%
- test_eval[mnasnet1_0-cpu-jit]: -16.59821%
- test_eval[pytorch_stargan-cpu-jit]: +16.65008%
- test_train[pytorch_stargan-cpu-eager]: +7.20868%
- test_eval[pytorch_stargan-cpu-eager]: +17.05075%
- test_eval[resnet18-cpu-jit]: -21.41248%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2776759031


Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220801+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220731+cu113

Base PyTorch commit: 32cf6c6fb0d98f747f576f52abe533449ae5da84

Affected PyTorch version: 1.13.0.dev20220801+cu113

Affected PyTorch commit: 53f56894ae0b747d2833ee168599afcfdd4be471

Affected Tests:
- test_eval[demucs-cpu-jit]: +7.28676%
- test_eval[demucs-cpu-eager]: +9.58443%
- test_eval[mnasnet1_0-cpu-jit]: -17.22872%
- test_eval[pytorch_stargan-cpu-jit]: +16.90464%
- test_train[pytorch_stargan-cpu-eager]: +10.97533%
- test_eval[pytorch_stargan-cpu-eager]: +15.79360%
- test_eval[resnet18-cpu-jit]: -21.80772%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2783935178
Empty result, closed.
Repository: pytorch/benchmark
Issue: Add DLRM to benchmark
Root cause: Unimplemented Feature Gap

Original model source
https://github.com/facebookresearch/dlrm

looks like this one contains most (all?) the model code inside one file https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py

the __main__ function should be a good indicator how to set up the model for training or eval, but it has a lot of extra baggage that can be removed when creating the benchmark version, such as profiling, onnx, save/load, etc. 

Also, note that pytorch benchmark repo was just reorganized.  @zdevito is updating ADDING_MODELS.md with some updated instructions asap.  Nothing changes for install.py but the hubconf.py is now named __init__.py, and there are more rules now about making no assumptions about current working directory for relative paths to load files.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220803+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220801+cu113

Base PyTorch commit: 53f56894ae0b747d2833ee168599afcfdd4be471

Affected PyTorch version: 1.13.0.dev20220803+cu113

Affected PyTorch commit: 9647bec0ec41176adfd0cd5b80ab72c5bf6b2141

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +11.35792%
- test_train[timm_vision_transformer-cpu-eager]: -8.49438%


cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/2791194577
Closed, empty result.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220803+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220801+cu113

Base PyTorch commit: 53f56894ae0b747d2833ee168599afcfdd4be471

Affected PyTorch version: 1.13.0.dev20220803+cu113

Affected PyTorch commit: 9647bec0ec41176adfd0cd5b80ab72c5bf6b2141

Affected Tests:
- test_eval[mnasnet1_0-cpu-eager]: +8.30705%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2795149950
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220804+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220803+cu113

Base PyTorch commit: 9647bec0ec41176adfd0cd5b80ab72c5bf6b2141

Affected PyTorch version: 1.13.0.dev20220804+cu113

Affected PyTorch commit: 1164c83c3c3070c82618350d57c7bf65ec56f0f1

Affected Tests:
- test_train[timm_vision_transformer-cpu-eager]: +10.52920%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2798387019
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220805+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220804+cu113

Base PyTorch commit: 1164c83c3c3070c82618350d57c7bf65ec56f0f1

Affected PyTorch version: 1.13.0.dev20220805+cu113

Affected PyTorch commit: 802a4fd28620f03d6537374953dbb4bb887a47b4

Affected Tests:
- test_train[timm_vision_transformer-cpu-eager]: -11.04156%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2805145179
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220806+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220805+cu113

Base PyTorch commit: 802a4fd28620f03d6537374953dbb4bb887a47b4

Affected PyTorch version: 1.13.0.dev20220806+cu113

Affected PyTorch commit: 86437b8631918aa4e766df869bca43853f0be6e9

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: -13.18003%
- test_train[timm_vision_transformer-cpu-eager]: +9.17313%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2809673709
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220807+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220806+cu113

Base PyTorch commit: 86437b8631918aa4e766df869bca43853f0be6e9

Affected PyTorch version: 1.13.0.dev20220807+cu113

Affected PyTorch commit: 4f255dbfb320e8b30f83b35dd4a40c8abd3164fc

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +10.02396%
- test_eval[pytorch_stargan-cpu-jit]: +8.57034%
- test_eval[pytorch_stargan-cpu-eager]: +7.69630%
- test_eval[resnet18-cpu-jit]: +26.60791%
- test_train[timm_resnest-cpu-eager]: -8.43092%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2813420789
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220808+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220807+cu113

Base PyTorch commit: 4f255dbfb320e8b30f83b35dd4a40c8abd3164fc

Affected PyTorch version: 1.13.0.dev20220808+cu113

Affected PyTorch commit: 1d56ea5e92b7bd041cf84ba9e0bb73f47a7eacf9

Affected Tests:
- test_eval[resnet18-cpu-jit]: -21.28249%
- test_train[timm_resnest-cpu-eager]: +12.59222%
- test_train[timm_vision_transformer-cpu-eager]: +9.39734%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2819689761
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220809+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220808+cu113

Base PyTorch commit: 1d56ea5e92b7bd041cf84ba9e0bb73f47a7eacf9

Affected PyTorch version: 1.13.0.dev20220809+cu113

Affected PyTorch commit: 9e65e93c39238ec05aa7913693d7c3e4523bf257

Affected Tests:
- test_eval[resnet18-cpu-jit]: +26.68598%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2827078084
Empty result, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220810+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220809+cu113

Base PyTorch commit: 9e65e93c39238ec05aa7913693d7c3e4523bf257

Affected PyTorch version: 1.13.0.dev20220810+cu113

Affected PyTorch commit: e1007950484aa1df4a2f87c9c14b514ffd7736a5

Affected Tests:
- test_eval[resnet18-cpu-jit]: -20.84177%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2834428864
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220810+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220809+cu113

Base PyTorch commit: 9e65e93c39238ec05aa7913693d7c3e4523bf257

Affected PyTorch version: 1.13.0.dev20220810+cu113

Affected PyTorch commit: e1007950484aa1df4a2f87c9c14b514ffd7736a5

Affected Tests:
- test_eval[fastNLP-cuda-eager]: +9.04498%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2835866477
Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220811+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220810+cu113

Base PyTorch commit: e1007950484aa1df4a2f87c9c14b514ffd7736a5

Affected PyTorch version: 1.13.0.dev20220811+cu113

Affected PyTorch commit: 3aeb5e4ff9d56ecd680401cfa3f23e97a279efbe

Affected Tests:
- test_eval[fastNLP-cuda-eager]: -8.62826%
- test_train[hf_BigBird-cpu-eager]: +9.70593%


cc @xuzhao9


```
{
  "start": "e1007950484aa1df4a2f87c9c14b514ffd7736a5",
  "end": "3aeb5e4ff9d56ecd680401cfa3f23e97a279efbe",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v1.0",
  "result": [
    {
      "commit1": "017ecb782d2",
      "commit1_time": "2022-08-10 21:50:13 +0000",
      "commit1_digest": {
        "test_train[hf_BigBird-cpu-eager]": 14.339042848977261
      },
      "commit2": "4e9b969baa6",
      "commit2_time": "2022-08-10 21:58:00 +0000",
      "commit2_digest": {
        "test_train[hf_BigBird-cpu-eager]": 15.746276016975752
      }
    }
  ]
}
```

This should be duplicate issue with https://github.com/pytorch/benchmark/issues/1099
Repository: pytorch/benchmark
Issue: Timm_efficientdet's profiling result contains dataloader
Root cause: Algorithmic Error

<img width="1158" alt="Screen Shot 2022-08-11 at 5 38 30 PM" src="https://user-images.githubusercontent.com/3823240/184264592-55b467b4-fd3d-421d-a3b1-c721f859e616.png">
The profiling of timm_efficientdet's inference mode contains data loader part.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220812+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220811+cu113

Base PyTorch commit: 3aeb5e4ff9d56ecd680401cfa3f23e97a279efbe

Affected PyTorch version: 1.13.0.dev20220812+cu113

Affected PyTorch commit: 2ca721cda54bcaee2202819bee92088ae2320ebf

Affected Tests:
- test_train[hf_BigBird-cpu-eager]: -9.41734%
- test_eval[mnasnet1_0-cpu-jit]: +34.47448%
- test_train[timm_regnet-cpu-eager]: +7.71413%
- test_train[timm_vision_transformer-cpu-eager]: +12.88793%


cc @xuzhao9


The speedup of hf_BigBird-cpu-eager is related to https://github.com/pytorch/benchmark/issues/1099, closed.

Result:
```
{
  "start": "3aeb5e4ff9d56ecd680401cfa3f23e97a279efbe",
  "end": "2ca721cda54bcaee2202819bee92088ae2320ebf",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "7e6da2fb104",
      "commit1_time": "2022-08-11 22:10:28 +0000",
      "commit1_digest": {
        "test_train[hf_BigBird-cpu-eager]": 15.622324948664755
      },
      "commit2": "799620178cf",
      "commit2_time": "2022-08-11 22:13:23 +0000",
      "commit2_digest": {
        "test_train[hf_BigBird-cpu-eager]": 14.288087010243908
      }
    }
  ]
}
```
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.13.0.dev20220812+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220811+cu113

Base PyTorch commit: 3aeb5e4ff9d56ecd680401cfa3f23e97a279efbe

Affected PyTorch version: 1.13.0.dev20220812+cu113

Affected PyTorch commit: 2ca721cda54bcaee2202819bee92088ae2320ebf

Affected Tests:
- test_train[hf_BigBird-cpu-eager]: -8.07099%


cc @xuzhao9


Known issue, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220813+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220812+cu113

Base PyTorch commit: 2ca721cda54bcaee2202819bee92088ae2320ebf

Affected PyTorch version: 1.13.0.dev20220813+cu113

Affected PyTorch commit: c2808571bf823a26e97231d83f5f3f2c5e3d3fe3

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +63.15069%
- test_train[timm_vision_transformer-cpu-eager]: -12.46419%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2852760008
Empty result, closed.