zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Errors in running with latest version of pytorch
Root cause: External Dependency Breakage

I am getting errors in running some networks with latest stable version of PyTorch for CPU

## commands for reproducing the issue
`conda create -n stable python=3.7 -y`
`conda activate stable`
`conda install -y pytorch torchtext torchvision cpuonly -c pytorch`
`git clone  https://github.com/pytorch/benchmark.git`
`cd benchmark`
`python install.py`
`pytest test_bench.py::TestBenchNetwork::test_eval[resnet50-freeze-cpu-jit] --fuser old --benchmark-json old.json --ignore_machine_config`

I am getting following error. 
[error_log.txt](https://github.com/pytorch/benchmark/files/6653651/error_log.txt)

The same command is working if I use nightly builds though.

Please let me know if there is need for more information.





There is a JIT API change in PyTorch master on https://github.com/pytorch/pytorch/commit/510d2ece81c18ba3f9bbcbc509effe281a77206c. The main branch works only for PyTorch versions later than this. Please try using the "v1.0" branch of pytorch/benchmark instead.

Fixed by https://github.com/pytorch/benchmark/pull/522
Repository: pytorch/benchmark
Issue: demucs - gpu utilization may be too low
Root cause: Algorithmic Error

Demucs time per test goes down when multiple copy streams are executed at the same time.

This can indicate that benchmark has too much idle time and will not be responsive to tools changes.

Test should have its data sizes checked.

streamcount, gpu time in ms:
1, 19.9744644
2, 13.226672
3, 12.40435133
4, 12.21978378
5, 11.82586898
6, 11.64228817
7, 11.59060123
8, 11.63492393
9, 11.601938
10, 11.562163
 

Fixed in https://github.com/pytorch/benchmark/pull/639.
Repository: pytorch/benchmark
Issue: learningtopaint - gpu utilization may be too low
Root cause: Algorithmic Error

learningtopaint gpu time per test goes down when multiple copy streams are executed at the same time.

This can indicate that benchmark has too much idle time and will not be responsive to tools changes.

Test should have its data sizes checked.

streamcount, gpu time in ms:
1, 2.132096052
2, 1.758159995
3, 1.696522713
4, 1.836799979
5, 1.737471962
6, 1.777754625
7, 1.755195481
8, 1.718631983
9, 1.776046223
10, 1.69311676


The GPU utilization issue is because of bad model code. Therefore, we are not fixing it to be faithful to the original model performance. 

See: https://github.com/pytorch/benchmark/pull/581
Repository: pytorch/benchmark
Issue: mobilenet_v3_large - gpu utilization may be too low
Root cause: Algorithmic Error

mobilenet_v3_large time per test goes down when multiple copy streams are executed at the same time.

This can indicate that benchmark has too much idle time and will not be responsive to tools changes.

Test should have its data sizes checked.

streamcount, gpu time in ms:
1, 13.56198406
2, 12.5697279
3, 12.53688558
4, 12.53544044
5, 12.16520309
6, 12.30045827
7, 12.66985103
8, 12.4240036
9, 12.38273027
10, 12.00705948
Repository: pytorch/benchmark
Issue: timm_nfnet - gpu utilization may be too low
Root cause: Algorithmic Error

timm_nfnet time per test goes down when multiple copy streams are executed at the same time.

This can indicate that benchmark has too much idle time and will not be responsive to tools changes.

Test should have its data sizes checked.

streamcount, gpu time in ms:
1, 32.51295853
2, 28.50992012
3, 27.51598867
4, 27.27095222
5, 26.7710968
6, 26.86407979
7, 27.00113351
8, 26.35978889
9, 26.9556054
10, 26.45357666


Fixed in https://github.com/pytorch/benchmark/pull/502
Repository: pytorch/benchmark
Issue: TODO: benchmark models
Root cause: Unimplemented Feature Gap

- [x] https://github.com/jihunchoi/recurrent-batch-normalization-pytorch/blob/master/bnlstm.py
- [x] LSTMVariants https://github.com/seba-1511/lstms.pth/blob/master/lstms/lstm.py, https://github.com/seba-1511/lstms.pth/blob/master/test/test_speed.py
- [x] (@zdevito) QRNN or SRU https://github.com/taolei87/sru/blob/master/cuda_functional.py https://gist.github.com/jekbradbury/a3a5ae890328db49d8093c1a5bdc8a1e https://openreview.net/pdf?id=H1zJ-v5xl 
- [ ] Yuandong's code (maybe not in the cont. build, but runnable): https://github.com/facebookresearch/ELF/ (compile directory: `./rts/game_MC`, run script: cd ELF
`sh ./train_minirts.sh --gpu 0`)
- [ ] convolution MT (meeting with Michael, David, Sergey to see if it is a good candidate)
- [ ] (@apaszke) tree2tree
- [x] (@ezyang) multiplicative LSTM

Follow up tasks:
- [x] Add batching to  LSTMVariants (some issues in a few layers with mismatched sizes)
- [ ] Annotate all examples with tracer code, and verify correctness.


Cleaning up benchmark repo, closing old issues, please reopen if needed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211020+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211019+cu102

Base PyTorch commit: c186c995df31a9db42a02d2bc159f36a2aab1a9f

Affected PyTorch version: 1.11.0.dev20211020+cu102

Affected PyTorch commit: 03975588d7886130a120f10bf6ffe28f1cc4477d

Affected Tests:
- test_eval[BERT_pytorch-cpu-eager]: -24.87201%
- test_train[demucs-cpu-eager]: +7.09120%
- test_train[densenet121-cpu-jit]: +12.29960%
- test_train[fastNLP-cpu-eager]: -18.25896%
- test_train[hf_Albert-cpu-eager]: +7.36448%
- test_eval[hf_BigBird-cpu-eager]: +16.94245%
- test_eval[hf_GPT2-cpu-eager]: +9.52532%
- test_train[shufflenet_v2_x1_0-cpu-jit]: +9.80691%
- test_train[squeezenet1_1-cpu-jit]: -9.80972%
- test_eval[squeezenet1_1-cpu-jit]: +12.30091%
- test_eval[vgg16-cpu-eager]: +14.79130%


cc @xuzhao9


This is a test issue created by CI workflow, closing it.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211020+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211019+cu102

Base PyTorch commit: c186c995df31a9db42a02d2bc159f36a2aab1a9f

Affected PyTorch version: 1.11.0.dev20211020+cu102

Affected PyTorch commit: 03975588d7886130a120f10bf6ffe28f1cc4477d

Affected Tests:
- test_eval[BERT_pytorch-cpu-eager]: -24.87201%
- test_train[demucs-cpu-eager]: +7.09120%
- test_train[densenet121-cpu-jit]: +12.29960%
- test_train[fastNLP-cpu-eager]: -18.25896%
- test_train[hf_Albert-cpu-eager]: +7.36448%
- test_eval[hf_BigBird-cpu-eager]: +16.94245%
- test_eval[hf_GPT2-cpu-eager]: +9.52532%
- test_train[shufflenet_v2_x1_0-cpu-jit]: +9.80691%
- test_train[squeezenet1_1-cpu-jit]: -9.80972%
- test_eval[squeezenet1_1-cpu-jit]: +12.30091%
- test_eval[vgg16-cpu-eager]: +14.79130%


cc @xuzhao9
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211020+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211019+cu102

Base PyTorch commit: c186c995df31a9db42a02d2bc159f36a2aab1a9f

Affected PyTorch version: 1.11.0.dev20211020+cu102

Affected PyTorch commit: 03975588d7886130a120f10bf6ffe28f1cc4477d

Affected Tests:
- test_eval[BERT_pytorch-cpu-eager]: -24.87201%
- test_train[demucs-cpu-eager]: +7.09120%
- test_train[densenet121-cpu-jit]: +12.29960%
- test_train[fastNLP-cpu-eager]: -18.25896%
- test_train[hf_Albert-cpu-eager]: +7.36448%
- test_eval[hf_BigBird-cpu-eager]: +16.94245%
- test_eval[hf_GPT2-cpu-eager]: +9.52532%
- test_train[shufflenet_v2_x1_0-cpu-jit]: +9.80691%
- test_train[squeezenet1_1-cpu-jit]: -9.80972%
- test_eval[squeezenet1_1-cpu-jit]: +12.30091%
- test_eval[vgg16-cpu-eager]: +14.79130%


cc @xuzhao9


Closed as it is a test.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211020+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211019+cu102

Base PyTorch commit: c186c995df31a9db42a02d2bc159f36a2aab1a9f

Affected PyTorch version: 1.11.0.dev20211020+cu102

Affected PyTorch commit: 03975588d7886130a120f10bf6ffe28f1cc4477d

Affected Tests:
- test_eval[BERT_pytorch-cpu-eager]: -24.87201%
- test_train[demucs-cpu-eager]: +7.09120%
- test_train[densenet121-cpu-jit]: +12.29960%
- test_train[fastNLP-cpu-eager]: -18.25896%
- test_train[hf_Albert-cpu-eager]: +7.36448%
- test_eval[hf_BigBird-cpu-eager]: +16.94245%
- test_eval[hf_GPT2-cpu-eager]: +9.52532%
- test_train[shufflenet_v2_x1_0-cpu-jit]: +9.80691%
- test_train[squeezenet1_1-cpu-jit]: -9.80972%
- test_eval[squeezenet1_1-cpu-jit]: +12.30091%
- test_eval[vgg16-cpu-eager]: +14.79130%


cc @xuzhao9


Closed as it is a test
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211022+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211020+cu102

Base PyTorch commit: 03975588d7886130a120f10bf6ffe28f1cc4477d

Affected PyTorch version: 1.11.0.dev20211022+cu102

Affected PyTorch commit: 5c9ba2feff08b98febd574514d79e2c8b6ffd703

Affected Tests:
- test_eval[BERT_pytorch-cpu-eager]: +19.09281%
- test_eval[LearningToPaint-cpu-jit]: -8.65493%
- test_train[densenet121-cpu-jit]: -8.43146%
- test_train[fastNLP-cpu-jit]: +8.60633%
- test_eval[hf_GPT2-cpu-eager]: +9.00522%
- test_train[mnasnet1_0-cpu-jit]: -7.37425%
- test_eval[pytorch_stargan-cpu-eager]: -13.84970%
- test_train[soft_actor_critic-cpu-eager]: +23.78989%
- test_train[squeezenet1_1-cpu-jit]: +10.10565%
- test_eval[timm_nfnet-cpu-eager]: +9.93401%
- test_eval[timm_vision_transformer-cpu-jit]: -13.59184%
- test_eval[vgg16-cpu-eager]: -12.90634%


cc @xuzhao9


This issue was generated for test purpose. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211023+cu102'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211022+cu102

Base PyTorch commit: 5c9ba2feff08b98febd574514d79e2c8b6ffd703

Affected PyTorch version: 1.11.0.dev20211023+cu102

Affected PyTorch commit: db84da55b7c215761a009bc09a318270d33b0f7c

Affected Tests:
- test_train[fastNLP-cpu-jit]: -7.86499%
- test_eval[hf_Albert-cpu-eager]: -9.25840%
- test_eval[hf_Bart-cpu-eager]: +8.20313%
- test_eval[hf_GPT2-cpu-eager]: -12.19796%
- test_eval[mobilenet_v2-cpu-eager]: -11.28131%
- test_eval[pytorch_stargan-cpu-eager]: +13.90842%
- test_train[soft_actor_critic-cpu-eager]: -17.53313%
- test_eval[timm_vision_transformer-cpu-jit]: +20.28398%
- test_eval[vgg16-cpu-eager]: +14.86526%


cc @xuzhao9


Closed for test purpose
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211024+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211022+cu102

Base PyTorch commit: 5c9ba2feff08b98febd574514d79e2c8b6ffd703

Affected PyTorch version: 1.11.0.dev20211024+cu102

Affected PyTorch commit: ef0fca42493fd29b38930a0d1588d454cf17ea75

Affected Tests:
- test_eval[LearningToPaint-cpu-jit]: +7.09098%
- test_eval[LearningToPaint-cpu-eager]: -14.94206%
- test_eval[attention_is_all_you_need_pytorch-cpu-eager]: -13.79423%
- test_train[densenet121-cpu-jit]: +18.18194%
- test_train[fastNLP-cpu-jit]: +13.90255%
- test_train[fastNLP-cpu-eager]: +28.46891%
- test_eval[mobilenet_v2-cpu-eager]: -10.88408%
- test_eval[pytorch_stargan-cpu-eager]: +13.70510%
- test_train[soft_actor_critic-cpu-eager]: -19.91296%
- test_train[squeezenet1_1-cpu-jit]: -15.26402%
- test_eval[timm_nfnet-cpu-eager]: -9.39465%
- test_eval[vgg16-cpu-eager]: +15.03596%


cc @xuzhao9


Closed for test purpose
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211025+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211022+cu102

Base PyTorch commit: 5c9ba2feff08b98febd574514d79e2c8b6ffd703

Affected PyTorch version: 1.11.0.dev20211025+cu102

Affected PyTorch commit: 52fec21b2f8b1ed5e4db2b9e1dd1f2879d6100fe

Affected Tests:
- test_eval[pyhpc_isoneutral_mixing-cpu-jit]: +7.55802%
- test_eval[pytorch_stargan-cpu-eager]: +13.88420%


cc @xuzhao9


Bisection shows empty result. These two tests seems to be noisy.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211026+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211025+cu102

Base PyTorch commit: 52fec21b2f8b1ed5e4db2b9e1dd1f2879d6100fe

Affected PyTorch version: 1.11.0.dev20211026+cu102

Affected PyTorch commit: 85a7535c032a46c88b0d4d782dfdb4504a4eff6e

Affected Tests:
- test_train[squeezenet1_1-cpu-jit]: +7.12465%
- test_eval[vgg16-cpu-eager]: +13.29941%


cc @xuzhao9


Bisection job returns empty result. These two tests are added to flaky test list.
Repository: pytorch/benchmark
Issue: Summary Metadata Validation - no_grad checks
Root cause: Unimplemented Feature Gap

After PR #526 lands, we need to fix these:
```
# FIXME: Models will use context "with torch.no_grad():", so the lifetime of no_grad will end after the eval().
# FIXME: Must incorporate this "torch.is_grad_enabled()" inside of actual eval() func.
# self._details.metadata["eval_nograd"] = not self.worker.load_stmt("torch.is_grad_enabled()")
```
To enable metadata checks for no_grad, we need to check inside of no_grad() context. This can be added after PR #486, which will allow us to capture no_grad (using step_fn()) inside the context of with torch.no_grad().
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211027+cu102'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211026+cu102

Base PyTorch commit: 0e371e413dd12b93cf50ad2441652ee6e0eb7217

Affected PyTorch version: 1.11.0.dev20211027+cu102

Affected PyTorch commit: c88da701e2f94987ea58a0980cc19cfb956cd980

Affected Tests:
- test_eval[hf_DistilBert-cpu-eager]: +13.43664%


cc @xuzhao9


Bisection job: https://github.com/pytorch/benchmark/actions/runs/1391238551

Result returns 49bf24fc83, but it is irrelevant to performance, so this result is just noise.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211028+cu102'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211027+cu102

Base PyTorch commit: c88da701e2f94987ea58a0980cc19cfb956cd980

Affected PyTorch version: 1.11.0.dev20211028+cu102

Affected PyTorch commit: fae1c0a4340ef4698f43875f5ed84ba512bdcd95

Affected Tests:


cc @xuzhao9


Closed as empty signal.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211031+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211030+cu102

Base PyTorch commit: c00806beda43063a0978a81aa4e6ce42cb893fe0

Affected PyTorch version: 1.11.0.dev20211031+cu102

Affected PyTorch commit: 510e3026a9bdac6e347e8756c7750029201ecc82

Affected Tests:
- test_train[hf_Bert-cpu-eager]: -7.25920%


cc @xuzhao9


Bisection job returns empty result. Closing this issue.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211101+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211031+cu102

Base PyTorch commit: 510e3026a9bdac6e347e8756c7750029201ecc82

Affected PyTorch version: 1.11.0.dev20211101+cu102

Affected PyTorch commit: 97f29bda59deab8c063cf01f0a8ff4321b93c55e

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +9.01735%


cc @xuzhao9


Bisection returns empty result. Closing this issue.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211104+cu102'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211103+cu102

Base PyTorch commit: 54241a9cfa1d4c5ee2096cd9acec1ae7bf623c37

Affected PyTorch version: 1.11.0.dev20211104+cu102

Affected PyTorch commit: 2486061c72c12ac7eb53946448f0a29525bf8cd1

Affected Tests:
- test_eval[soft_actor_critic-cuda-eager]: -12.60492%


cc @xuzhao9


This is a known improvement due to https://github.com/pytorch/pytorch/pull/67741
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211106+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211105+cu102

Base PyTorch commit: f1754319e32f302a53520ac3608addb31402209b

Affected PyTorch version: 1.11.0.dev20211106+cu102

Affected PyTorch commit: ae501a9727c16b8bc83c2fd7b40d9d801133ac91

Affected Tests:
- test_eval[LearningToPaint-cuda-jit]: +15.35136%
- test_eval[mnasnet1_0-cuda-jit]: -96.00715%
- test_eval[mobilenet_v3_large-cuda-jit]: -50.97254%
- test_eval[resnet18-cuda-jit]: +10.27948%
- test_eval[resnet50-cuda-jit]: +13.03570%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: +42.71758%
- test_eval[squeezenet1_1-cuda-jit]: +57.48286%
- test_eval[timm_regnet-cuda-jit]: +16.62421%
- test_eval[timm_resnest-cuda-jit]: +14.95034%
- test_eval[timm_vovnet-cuda-jit]: +9.83278%
- test_eval[vgg16-cuda-jit]: +15.94762%


cc @xuzhao9


Bisection shows root cause commit: https://github.com/pytorch/pytorch/commit/f6a4c80a5a

Escalated to pytorch issue: https://github.com/pytorch/pytorch/issues/68086
Repository: pytorch/benchmark
Issue: benchmark/rnns/benchmarks/lstm_variants/lstm.py equation error

I think in line 192 in file benchmark/rnns/benchmarks/lstm_variants/lstm.py

o_t = gates[:, -self.hidden_size:]

should be modified to 

o_t = gates[:, 2 * self.hidden_size:3 * self.hidden_size]

cc: @zou3519 if you remember.

Sorry, I was mistaken, both solutions are correct.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211109+cu102'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211108+cu102

Base PyTorch commit: 191b48b12f33e1e9525882da0c62b68686d69e42

Affected PyTorch version: 1.11.0.dev20211109+cu102

Affected PyTorch commit: 4b1d044498f49ad5f770354174070a4277279c2d

Affected Tests:
- test_eval[densenet121-cpu-eager]: -7.56276%


cc @xuzhao9


Bisector returns empty result. This is a noise signal.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211110+cu102'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211109+cu102

Base PyTorch commit: 4b1d044498f49ad5f770354174070a4277279c2d

Affected PyTorch version: 1.11.0.dev20211110+cu102

Affected PyTorch commit: 6b44e75f6bccca7acc8ec31a635f1175c265ac54

Affected Tests:
- test_eval[densenet121-cpu-eager]: +10.91808%
- test_eval[hf_T5-cuda-eager]: +7.45416%


cc @xuzhao9


This is a noise since https://github.com/pytorch/benchmark/actions/runs/1445366183 returns empty result.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211115+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211114+cu102

Base PyTorch commit: 1adeeabdc0c8832420c091c5c668843768530d7f

Affected PyTorch version: 1.11.0.dev20211115+cu102

Affected PyTorch commit: 1adeeabdc0c8832420c091c5c668843768530d7f

Affected Tests:
- test_eval[fastNLP-cpu-eager]: -7.02257%
- test_train[hf_Bert-cpu-eager]: +7.52605%


cc @xuzhao9


Closing the noise signal.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211118+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211117+cu102

Base PyTorch commit: aa9ee8d02aecee1a01f7e4fe22fd0660851227e8

Affected PyTorch version: 1.11.0.dev20211118+cu102

Affected PyTorch commit: 980778713588be8a583557c4693634c3782b00c4

Affected Tests:
- test_eval[mnasnet1_0-cuda-jit]: -7.12313%
- test_eval[mobilenet_v2-cuda-jit]: -7.04192%
- test_eval[mobilenet_v3_large-cuda-jit]: -8.07279%
- test_eval[resnext50_32x4d-cuda-eager]: -7.36881%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -8.91571%
- test_eval[squeezenet1_1-cuda-jit]: -7.21233%
- test_eval[timm_regnet-cuda-jit]: -11.18967%
- test_eval[timm_regnet-cuda-eager]: -9.60096%


cc @xuzhao9


Duplicate, closing it.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211118+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211117+cu102

Base PyTorch commit: aa9ee8d02aecee1a01f7e4fe22fd0660851227e8

Affected PyTorch version: 1.11.0.dev20211118+cu102

Affected PyTorch commit: 980778713588be8a583557c4693634c3782b00c4

Affected Tests:
- test_eval[mnasnet1_0-cuda-jit]: -7.05981%
- test_eval[mobilenet_v3_large-cuda-jit]: -7.45515%
- test_eval[resnext50_32x4d-cuda-eager]: -7.54913%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -8.78407%
- test_eval[squeezenet1_1-cuda-jit]: -7.30330%
- test_eval[timm_regnet-cuda-jit]: -12.30104%
- test_eval[timm_regnet-cuda-eager]: -9.49766%


cc @xuzhao9


Bisection returns empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211121+cu102'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211120+cu102

Base PyTorch commit: 75955e4ef8a941d72db20c5098371325bd83ffd1

Affected PyTorch version: 1.11.0.dev20211121+cu102

Affected PyTorch commit: c7ecf1498d961415006c3710ac8d99166fe5d634

Affected Tests:
- test_train[hf_Bert-cpu-eager]: -8.19688%


cc @xuzhao9


Closed as this is nose.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211125+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211124+cu102

Base PyTorch commit: b5b62b340891f041b378681577c74922d21700a9

Affected PyTorch version: 1.11.0.dev20211125+cu102

Affected PyTorch commit: 5fdcc20d8d96a6b42387f57c2ce331516ad94228

Affected Tests:
- test_eval[densenet121-cpu-eager]: -7.56584%


cc @xuzhao9


Bisection returns empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211125+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211124+cu102

Base PyTorch commit: b5b62b340891f041b378681577c74922d21700a9

Affected PyTorch version: 1.11.0.dev20211125+cu102

Affected PyTorch commit: 5fdcc20d8d96a6b42387f57c2ce331516ad94228

Affected Tests:
- test_eval[densenet121-cpu-eager]: -7.85071%
- test_train[hf_Bert-cpu-eager]: -10.62344%


cc @xuzhao9


Bisection returns empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211127+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211125+cu102

Base PyTorch commit: 5fdcc20d8d96a6b42387f57c2ce331516ad94228

Affected PyTorch version: 1.11.0.dev20211127+cu102

Affected PyTorch commit: cffad597ea620600472c509b642a9ad03a56fdbe

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +10.33140%
- test_train[timm_vision_transformer-cuda-jit]: +10.04580%


cc @xuzhao9


Bisection returns empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211128+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211127+cu102

Base PyTorch commit: cffad597ea620600472c509b642a9ad03a56fdbe

Affected PyTorch version: 1.11.0.dev20211128+cu102

Affected PyTorch commit: b10929a14a32c90ac5943cc3f27a3a9105e6f53f

Affected Tests:
- test_train[hf_DistilBert-cpu-eager]: +7.07404%
- test_eval[nvidia_deeprecommender-cpu-eager]: +7.16154%
- test_train[timm_vision_transformer-cuda-jit]: -7.42993%


cc @xuzhao9


Bisection returns empty result. Closed.