zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Audit existing models
Root cause: Unimplemented Feature Gap

First, create a checklist of important things for each model

Then go through each model and make sure they comply

I will be editing this list as I come up with more checks:

[ ] Make sure that device and jit are routed to model's c-tors so a model actually runs on the right device and in the right mode (i.e. jit). This could be verified by simply checking the model's inputs and outputs. This could be automated.
[ ] **Double check** that in `test_eval` a model is running under `no_grad` and `eval` is called. we could automatically check that all model paratemeters don't require gradients and inputs.
[ ] Consider running a model under `autograd.profiler` and look for gaps in between events as gaps signify that we are doing extra work (constructing a model from scratch in every eval run) that's not relevant to model execution. (Note, optimizers do show up as a series of events) 
[ ] If the model is jittable, consider enabling `check_results` for the model
[ ] Consider provided the frozen version of the model
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220123+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220122+cu113

Base PyTorch commit: f7e78364e4d388eea2dd91c4864a5706559a4c46

Affected PyTorch version: 1.11.0.dev20220123+cu113

Affected PyTorch commit: a031636e607420964db38c638b26bf8e23f0a8ec

Affected Tests:
- test_train[dcgan-cpu-jit]: +11.23495%
- test_train[dcgan-cpu-eager]: +11.42574%
- test_train[dlrm-cpu-eager]: -8.82318%


cc @xuzhao9


Duplicate of https://github.com/pytorch/benchmark/issues/708
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220124+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220122+cu113

Base PyTorch commit: f7e78364e4d388eea2dd91c4864a5706559a4c46

Affected PyTorch version: 1.11.0.dev20220124+cu113

Affected PyTorch commit: e5e237cdd278652b1253d98784faeb681c65878b

Affected Tests:
- test_train[dcgan-cpu-jit]: +11.35637%
- test_train[dcgan-cpu-eager]: +11.48580%
- test_train[dlrm-cpu-eager]: -8.11028%
- test_eval[mobilenet_v2-cpu-jit]: +7.81478%


cc @xuzhao9


Bisection job (https://github.com/pytorch/benchmark/actions/runs/1745806750) points out two tests are actually affected:
```
test_train[dcgan-cpu-jit]: +11.35637%
test_train[dcgan-cpu-eager]: +11.48580%
```
The root cause commit is [c9bd1c60ed](https://github.com/pytorch/pytorch/commit/c9bd1c60ed)

The commit seems to be only related with jit, not sure why it affects eager test performance

Can't reproduce, treat as noise.
Repository: pytorch/benchmark
Issue: scribe.py fails to upload result to rds
Root cause: Interface Contract Mismatch

In the V2 score, we updated the score schema to include by-domain subscores.
This change seems to break `scribe.py` which uploads the results to rds (https://github.com/pytorch/benchmark/blob/main/scripts/upload_scribe_v2.py#L50).
Example failed workflow: https://github.com/pytorch/benchmark/runs/4937888834?check_suite_focus=true

So currently, I have disabled the rds upload in v2.

@janeyx99 , do we still need to upload the metrics to the DB behind `https://metrics.pytorch.org`?

It is fine to not upload through RDS as long as it's backed somewhere. We are in the process of migrating from using RDS to Rockset so this isn't a big issue.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20220125+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220123+cu113

Base PyTorch commit: 9b3a56eecfab13877d32f0a5051e1584746668bc

Affected PyTorch version: 1.11.0.dev20220125+cu113

Affected PyTorch commit: b372be4211d40be92ffbb025cac0038ee3445141

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +8.18274%


cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/1747841058 empty result. Closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220126+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220124+cu113

Base PyTorch commit: e5e237cdd278652b1253d98784faeb681c65878b

Affected PyTorch version: 1.11.0.dev20220126+cu113

Affected PyTorch commit: a2f0cefe703f82b3701e57f74c97dabfb3955b89

Affected Tests:
- test_train[dcgan-cpu-jit]: -10.33408%
- test_train[dcgan-cpu-eager]: -9.56053%
- test_train[pytorch_unet-cpu-eager]: -7.28128%


cc @xuzhao9


Bisection root cause commit: [e33cd8f382](https://github.com/pytorch/pytorch/commit/e33cd8f382) looks like this is noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220127+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220124+cu113

Base PyTorch commit: 9b3a56eecfab13877d32f0a5051e1584746668bc

Affected PyTorch version: 1.11.0.dev20220127+cu113

Affected PyTorch commit: 12e01f78257479f91b0f23d8d93318bc92945c87

Affected Tests:
- test_train[dcgan-cpu-jit]: -11.03951%
- test_train[dcgan-cpu-eager]: -9.88998%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/runs/4984142691?check_suite_focus=true

Empty result, noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220128+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220127+cu113

Base PyTorch commit: 12e01f78257479f91b0f23d8d93318bc92945c87

Affected PyTorch version: 1.11.0.dev20220128+cu113

Affected PyTorch commit: 7aa4a1f63e14406807ff6cb45faabf0151a63db8

Affected Tests:
- test_train[dlrm-cpu-eager]: +10.84960%
- test_eval[mobilenet_v2-cpu-jit]: -7.51539%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1763008321 empty result, noise.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20220128+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220126+cu113

Base PyTorch commit: 12e01f78257479f91b0f23d8d93318bc92945c87

Affected PyTorch version: 1.11.0.dev20220128+cu113

Affected PyTorch commit: 7aa4a1f63e14406807ff6cb45faabf0151a63db8

Affected Tests:
- test_eval[densenet121-cpu-eager]: -11.64367%


cc @xuzhao9


Closed as noise signal.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220129+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220128+cu113

Base PyTorch commit: 7aa4a1f63e14406807ff6cb45faabf0151a63db8

Affected PyTorch version: 1.11.0.dev20220129+cu113

Affected PyTorch commit: 9413c0cd3e15c2b38e7df500613cff1e7951699e

Affected Tests:


cc @xuzhao9


False positive and should be fixed by https://github.com/pytorch/benchmark/pull/728
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220130+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220129+cu113

Base PyTorch commit: 9413c0cd3e15c2b38e7df500613cff1e7951699e

Affected PyTorch version: 1.11.0.dev20220130+cu113

Affected PyTorch commit: 8fa5cde3a9fab1f82bbb424312995035f2342eaa

Affected Tests:
- test_train[dcgan-cpu-jit]: +11.47381%
- test_train[dcgan-cpu-eager]: +10.39681%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +7.40582%


cc @xuzhao9


Action: https://github.com/pytorch/benchmark/runs/4998006175?check_suite_focus=true

Closed as this is a noise signal.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220131+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220125+cu113

Base PyTorch commit: b372be4211d40be92ffbb025cac0038ee3445141

Affected PyTorch version: 1.11.0.dev20220131+cu113

Affected PyTorch commit: 8fa5cde3a9fab1f82bbb424312995035f2342eaa

Affected Tests:
- test_train[dlrm-cpu-eager]: +11.96043%


cc @xuzhao9


Action: https://github.com/pytorch/benchmark/actions/runs/1774032803

New workflow: https://github.com/pytorch/benchmark/actions/runs/1996520683

Empty result, noise signal.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220201+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220129+cu113

Base PyTorch commit: 9413c0cd3e15c2b38e7df500613cff1e7951699e

Affected PyTorch version: 1.11.0.dev20220201+cu113

Affected PyTorch commit: 8fa5cde3a9fab1f82bbb424312995035f2342eaa

Affected Tests:
- test_train[dcgan-cpu-jit]: +11.64633%
- test_train[dcgan-cpu-eager]: +10.34394%
- test_eval[mobilenet_v2-cpu-jit]: +8.07136%


cc @xuzhao9


Action: https://github.com/pytorch/benchmark/actions/runs/1779799559

New workflow: https://github.com/pytorch/benchmark/actions/runs/1998691073

test_train[dcgan-cpu-jit] and test_train[dcgan-cpu-eager] slowdown: commit [f499ab9cef](https://github.com/pytorch/pytorch/commit/f499ab9cef)

PR: https://github.com/pytorch/pytorch/pull/74696

Escalated to pytorch issue: https://github.com/pytorch/pytorch/issues/74752

Can't reproduce it locally, and after manually check the dcgan model doesn't use GeLU which is changed in the PR. The delta is likely to be caused by environment noise. I am closing this issue.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220202+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220201+cu113

Base PyTorch commit: 8fa5cde3a9fab1f82bbb424312995035f2342eaa

Affected PyTorch version: 1.11.0.dev20220202+cu113

Affected PyTorch commit: f20fa66f704a61b4570ce372add59c7469f44a96

Affected Tests:
- test_train[dcgan-cpu-jit]: -10.17793%
- test_train[dcgan-cpu-eager]: -9.87002%
- test_eval[mobilenet_v2-cpu-jit]: -8.05041%


cc @xuzhao9


Action: https://github.com/pytorch/benchmark/actions/runs/1785339373

Empty result. Close.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.11.0.dev20220203+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220131+cu113

Base PyTorch commit: 8fa5cde3a9fab1f82bbb424312995035f2342eaa

Affected PyTorch version: 1.11.0.dev20220203+cu113

Affected PyTorch commit: 58dabebcd746aad95a37bdfc7e60e5d22f0f5641

Affected Tests:
- test_train[dcgan-cpu-jit]: -10.56872%
- test_train[dcgan-cpu-eager]: -10.05785%


cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/1790907719

Unrelated, closed as noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220206+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220205+cu113

Base PyTorch commit: f2f40ce870488b0cfc6e1d9c1cdea3a95f01a4c1

Affected PyTorch version: 1.12.0.dev20220206+cu113

Affected PyTorch commit: 5c6b897516ac59775d7d58692fad052c399c4ec3

Affected Tests:
- test_train[dcgan-cpu-jit]: -10.45391%
- test_train[dcgan-cpu-eager]: -10.42632%
- test_train[dlrm-cpu-eager]: +13.79582%
- test_eval[resnet18-cpu-eager]: -8.02115%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1803068572

Noise signal, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220207+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220205+cu113

Base PyTorch commit: f2f40ce870488b0cfc6e1d9c1cdea3a95f01a4c1

Affected PyTorch version: 1.12.0.dev20220207+cu113

Affected PyTorch commit: 5c6b897516ac59775d7d58692fad052c399c4ec3

Affected Tests:
- test_train[dcgan-cpu-jit]: -10.17392%
- test_train[dcgan-cpu-eager]: -10.58934%
- test_train[dlrm-cpu-eager]: +14.37638%
- test_eval[resnet18-cpu-eager]: -9.19650%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1807891296

Closed as noise.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220207+cu113'
Root cause: Validation Gap


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220206+cu113

Base PyTorch commit: 5c6b897516ac59775d7d58692fad052c399c4ec3

Affected PyTorch version: 1.12.0.dev20220207+cu113

Affected PyTorch commit: 5c6b897516ac59775d7d58692fad052c399c4ec3

Affected Tests:
- test_eval[nvidia_deeprecommender-cpu-eager]: +8.53783%


cc @xuzhao9


Noise because base and target commit is the same version.
https://github.com/pytorch/benchmark/runs/5100821524?check_suite_focus=true
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220208+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220207+cu113

Base PyTorch commit: 5c6b897516ac59775d7d58692fad052c399c4ec3

Affected PyTorch version: 1.12.0.dev20220208+cu113

Affected PyTorch commit: 224093db113fd333a7cb8e5e07308de5098e4a62

Affected Tests:
- test_train[dcgan-cpu-jit]: +11.20770%
- test_train[dcgan-cpu-eager]: +10.67257%
- test_eval[pyhpc_equation_of_state-cpu-eager]: -8.57558%
- test_eval[resnet18-cpu-eager]: +7.93957%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1813988416

Closed as noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220209+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220208+cu113

Base PyTorch commit: 224093db113fd333a7cb8e5e07308de5098e4a62

Affected PyTorch version: 1.12.0.dev20220209+cu113

Affected PyTorch commit: 4b69a2373f8f77548fba82886ab0dc8c06a38d87

Affected Tests:
- test_train[dcgan-cpu-jit]: -9.94482%
- test_train[dcgan-cpu-eager]: -9.40117%
- test_train[hf_Reformer-cpu-eager]: +7.01471%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1819723535

Empty result. Closed Issue.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220209+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220208+cu113

Base PyTorch commit: 224093db113fd333a7cb8e5e07308de5098e4a62

Affected PyTorch version: 1.12.0.dev20220209+cu113

Affected PyTorch commit: 4b69a2373f8f77548fba82886ab0dc8c06a38d87

Affected Tests:
- test_eval[nvidia_deeprecommender-cpu-eager]: -9.76056%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1822130917

Empty result. Closed issue.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220210+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220209+cu113

Base PyTorch commit: 4b69a2373f8f77548fba82886ab0dc8c06a38d87

Affected PyTorch version: 1.12.0.dev20220210+cu113

Affected PyTorch commit: 239531fb3165fbca13a18febbabff493ea4fd55d

Affected Tests:
- test_eval[drq-cpu-eager]: +16.76241%
- test_eval[timm_efficientnet-cuda-eager]: +11.53292%


cc @xuzhao9


Empty result, closed as noise
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220210+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220209+cu113

Base PyTorch commit: 4b69a2373f8f77548fba82886ab0dc8c06a38d87

Affected PyTorch version: 1.12.0.dev20220210+cu113

Affected PyTorch commit: 239531fb3165fbca13a18febbabff493ea4fd55d

Affected Tests:
- test_eval[densenet121-cpu-eager]: -7.66385%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1827076968. Empty result, closing.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220214+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220210+cu113

Base PyTorch commit: 239531fb3165fbca13a18febbabff493ea4fd55d

Affected PyTorch version: 1.12.0.dev20220214+cu113

Affected PyTorch commit: 1776caf361fc2c652f0c00b2fd42714447cefc29

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cpu-eager]: -9.85918%
- test_train[timm_nfnet-cpu-jit]: +7.32522%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1842649968
Empty result, closed as noise.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220214+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220210+cu113

Base PyTorch commit: 239531fb3165fbca13a18febbabff493ea4fd55d

Affected PyTorch version: 1.12.0.dev20220214+cu113

Affected PyTorch commit: 1776caf361fc2c652f0c00b2fd42714447cefc29

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cpu-jit]: +7.02499%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1843627853
Empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220216+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220215+cu113

Base PyTorch commit: 47c69933558040158c1533f1b500d15153236b3b

Affected PyTorch version: 1.12.0.dev20220216+cu113

Affected PyTorch commit: 47c69933558040158c1533f1b500d15153236b3b

Affected Tests:
- test_train[yolov3-cuda-eager]: +8.62145%


cc @xuzhao9


Same commit, closed as noise.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220216+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220214+cu113

Base PyTorch commit: 1776caf361fc2c652f0c00b2fd42714447cefc29

Affected PyTorch version: 1.12.0.dev20220216+cu113

Affected PyTorch commit: 47c69933558040158c1533f1b500d15153236b3b

Affected Tests:
- test_train[pytorch_unet-cpu-jit]: -7.06556%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +11.75836%


cc @xuzhao9


Empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220217+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220216+cu113

Base PyTorch commit: 47c69933558040158c1533f1b500d15153236b3b

Affected PyTorch version: 1.12.0.dev20220217+cu113

Affected PyTorch commit: bbac8c9c4816c87eae17a04b60d4103940c08f83

Affected Tests:
- test_train[yolov3-cuda-eager]: -8.06660%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1861199701 empty result. Close the issue.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220222+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220221+cu113

Base PyTorch commit: f41db99a5690bf0ae4d51d9cdb9aaff95703f6c6

Affected PyTorch version: 1.12.0.dev20220222+cu113

Affected PyTorch commit: f41db99a5690bf0ae4d51d9cdb9aaff95703f6c6

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +7.42669%


cc @xuzhao9


Noise because the base and affected commit hash is the same.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220223+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220222+cu113

Base PyTorch commit: f41db99a5690bf0ae4d51d9cdb9aaff95703f6c6

Affected PyTorch version: 1.12.0.dev20220223+cu113

Affected PyTorch commit: 7807a83f6e677987e8cb0bdbebd04ccdd157b68b

Affected Tests:
- test_train[nvidia_deeprecommender-cuda-eager]: +7.89634%


cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/1888869668
Offending commit: [3aecce7015](https://github.com/pytorch/pytorch/commit/3aecce7015)
Parent commit: 237574db19

The fix has landed. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220223+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220222+cu113

Base PyTorch commit: f41db99a5690bf0ae4d51d9cdb9aaff95703f6c6

Affected PyTorch version: 1.12.0.dev20220223+cu113

Affected PyTorch commit: 7807a83f6e677987e8cb0bdbebd04ccdd157b68b

Affected Tests:
- test_eval[nvidia_deeprecommender-cuda-eager]: -8.70383%


cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/1889826610

New workflow: https://github.com/pytorch/benchmark/runs/5586157443?check_suite_focus=true

test_eval[nvidia_deeprecommender-cuda-eager] speedup commit: [3aecce7015](https://github.com/pytorch/pytorch/commit/3aecce7015)

Closed because this is a known issue.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220226+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220225+cu113

Base PyTorch commit: 6688487f3ec22b7e89553b8acd6782264061f5cd

Affected PyTorch version: 1.12.0.dev20220226+cu113

Affected PyTorch commit: 6688487f3ec22b7e89553b8acd6782264061f5cd

Affected Tests:
- test_eval[tts_angular-cpu-eager]: +9.53339%


cc @xuzhao9


Noise because the base and affected commit hash is the same.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.12.0.dev20220227+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220226+cu113

Base PyTorch commit: 6688487f3ec22b7e89553b8acd6782264061f5cd

Affected PyTorch version: 1.12.0.dev20220227+cu113

Affected PyTorch commit: 6688487f3ec22b7e89553b8acd6782264061f5cd

Affected Tests:
- test_eval[tts_angular-cpu-eager]: -9.50673%


cc @xuzhao9


Closed as this is a noise (same pytorch commit hash)
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.12.0.dev20220228+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.12.0.dev20220223+cu113

Base PyTorch commit: 7807a83f6e677987e8cb0bdbebd04ccdd157b68b

Affected PyTorch version: 1.12.0.dev20220228+cu113

Affected PyTorch commit: 6688487f3ec22b7e89553b8acd6782264061f5cd

Affected Tests:
- test_eval[mnasnet1_0-cpu-jit]: +34.88560%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/1911792526
Empty result. Closed as noise.

Maybe worth looking into, should try running another bisection

Empty result. Closed.