zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '8d78e808f9373a89d94c7e475653173698e765ed'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: e6fa795672dd659d287b48c76c8b41f97c8fa7b1

Affected PyTorch commit: 8d78e808f9373a89d94c7e475653173698e765ed

Affected Tests:
- DALLE2_pytorch, AdamW, cuda, default: +30.06245%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6375819551

cc @janeyx99


seems fine
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '4dae8b49630d2784f6a5d8726db30923e2d1e077'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8d78e808f9373a89d94c7e475653173698e765ed

Affected PyTorch commit: 4dae8b49630d2784f6a5d8726db30923e2d1e077

Affected Tests:
- DALLE2_pytorch, Adam, cuda, foreach: +33.90456%
- DALLE2_pytorch, AdamW, cuda, foreach: +38.89779%
- DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +33.92788%
- DALLE2_pytorch, RAdam, cuda, foreach: +44.08831%
- shufflenet_v2_x1_0, Adagrad, cuda, no_foreach: +56.42979%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6388504263

cc @janeyx99


conglomerating noise in #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '4e30fa82315208dcd38fa16a0ed9851fa8e98bc9'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 4dae8b49630d2784f6a5d8726db30923e2d1e077

Affected PyTorch commit: 4e30fa82315208dcd38fa16a0ed9851fa8e98bc9

Affected Tests:
- mobilenet_v3_large, Rprop, cuda, default: +58.95230%
- detectron2_maskrcnn, Adam, cuda, differentiable: +44.64110%
- shufflenet_v2_x1_0, Adagrad, cuda, no_foreach: -34.88800%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6401389857

cc @janeyx99


conglomerating noise in #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 4e30fa82315208dcd38fa16a0ed9851fa8e98bc9

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- hf_BigBird, Rprop, cuda, default: +49.31141%
- resnet50, NAdam, cuda, no_foreach: +35.15587%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6414461584

cc @janeyx99


seems like flukes
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: fecde478ac83edf78e7d0e9d11ab73cb1580f6cf

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: -18.96556%
- test_eval[hf_Bart-cuda-eager]_latency: -10.03955%
- test_eval[hf_T5_base-cuda-eager]_latency: -14.93088%
- test_train[lennard_jones-cuda-eager]_latency: -11.38641%
- test_eval[timm_vision_transformer-cuda-eager]_latency: -40.21496%
- test_eval[timm_vision_transformer_large-cuda-eager]_latency: -25.54539%


Tests that were no longer run on affected commit:
- test_train[fambench_xlmr-cuda-eager]_latency: 693.131569
- test_train[fambench_xlmr-cuda-eager]_cmem: 1.1865234375
- test_train[fambench_xlmr-cuda-eager]_gmem: 16.03338623046875
- test_eval[fambench_xlmr-cuda-eager]_latency: 24.715116
- test_eval[fambench_xlmr-cuda-eager]_cmem: 1.1435546875
- test_eval[fambench_xlmr-cuda-eager]_gmem: 3.37518310546875


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6420819390

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/6423087751

Bisection: https://github.com/pytorch/benchmark/actions/runs/6425442814/job/17448019085

```
{
  "target_repo": "pytorch",
  "start": "75462fd87024bfe9fb79f4c46ab689bec227caa7",
  "end": "510ec7e3c539dfed49df587d09e8a0a87e187201",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: [Question] What benchmarks does pytorch use to test a release?

Hi, I am running benchmarks for pytorch 2.1.0 on AWS hardware and I want to confirm which benchmarks/models were used by upstream pytorch for the release.
Do you use [userbenchmark/torch-nightly](https://github.com/pytorch/benchmark/tree/v3.0/userbenchmark/torch-nightly) to test the nightlies, then [userbenchmark/release-test/run_release_test.sh](https://github.com/pytorch/benchmark/blob/main/userbenchmark/release-test/run_release_test.sh) before the official release? I see that `torch-nightly` has much more models than `run_release_test.sh` does; are the models in `run_release_test.sh` sufficient for a release?

For the release we only use the `release-test` userbenchmark, because it tests end-to-end models which involves multiple batches, data loading, etc.

At the same time, we use `torch-nightly` to run more models on nightly release to prevent regressions, but `torch-nightly` only runs single-batch, module-level tests, the data is prefetched to the device so the dataloading is not involved.

cc @atalman is there any plan to build broader release test?

@xuzhao9 , @sallyseok . The torch-nightly tests detect nightly regressions and run on nightly basis. For release-test we run them few times before the release and compare basic performance regression between the releases. I agree we should also probably include torch-nightly tests within release testing. We do have plan on improving release perf testing for H1 2024.
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- hf_BigBird, Rprop, cuda, default: -30.36015%
- speech_transformer, Adadelta, cuda, differentiable: +47.25762%
- mobilenet_v2, RAdam, cuda, no_foreach: +42.18837%
- timm_vision_transformer_large, RAdam, cuda, no_foreach: -44.11201%
- DALLE2_pytorch, AdamW, cuda, foreach: +32.51113%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6427317387

cc @janeyx99


superceded in #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- timm_vision_transformer_large, RAdam, cuda, no_foreach: +78.50101%
- speech_transformer, Adadelta, cuda, differentiable: -32.16910%
- mobilenet_v2, RAdam, cuda, no_foreach: -31.10531%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6438899118

cc @janeyx99


conglomerating noise in #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- DALLE2_pytorch, Adam, cuda, fused, amsgrad, maximize: +31.76727%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6445393204

cc @janeyx99


conglomerating noise in #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- stable_diffusion_unet, RAdam, cuda, no_foreach: +92.67768%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6452197529

cc @janeyx99


superceded by #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- timm_vision_transformer_large, RAdam, cuda, no_foreach: -43.40771%
- timm_vision_transformer_large, RAdam, cuda, differentiable: -44.11391%
- stable_diffusion_unet, RAdam, cuda, no_foreach: -47.72116%
- DALLE2_pytorch, Adam, cuda, fused, capturable, amsgrad: +37.85595%
- DALLE2_pytorch, AdamW, cuda, foreach: +37.25520%
- DALLE2_pytorch, AdamW, cuda, fused, capturable, amsgrad: +36.33095%
- DALLE2_pytorch, RAdam, cuda, foreach: +40.75257%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6464564973

cc @janeyx99


conglomerating noise in #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- doctr_det_predictor, Adam, cuda, no_foreach: +40.54418%
- timm_vision_transformer_large, RAdam, cuda, no_foreach: +76.79410%
- timm_vision_transformer_large, RAdam, cuda, differentiable: +79.21604%
- tts_angular, Rprop, cuda, maximize: +46.56646%
- shufflenet_v2_x1_0, NAdam, cuda, no_foreach: +48.30327%
- DALLE2_pytorch, Adam, cuda, default: +35.45820%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6477813689

cc @janeyx99


conglomerating noise in #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- DALLE2_pytorch, Adam, cuda, default: -30.66293%
- shufflenet_v2_x1_0, NAdam, cuda, no_foreach: -34.01592%
- tts_angular, Rprop, cuda, maximize: -31.97153%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6491088661

cc @janeyx99


conglomerating noise in #1988
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- DALLE2_pytorch, Adam, cuda, fused, amsgrad, maximize: +45.06670%
- DALLE2_pytorch, Adam, cuda, fused, capturable, amsgrad: +36.11691%
- DALLE2_pytorch, AdamW, cuda, foreach: +49.40952%
- DALLE2_pytorch, AdamW, cuda, fused, amsgrad, maximize: +38.32376%
- timm_vision_transformer_large, RAdam, cuda, no_foreach: -44.00085%
- timm_vision_transformer_large, RAdam, cuda, differentiable: -44.24940%
- hf_T5_large, Rprop, cuda, differentiable: +30.94747%
- Super_SloMo, RMSprop, cuda, differentiable: +52.56252%
- hf_Reformer, Adadelta, cuda, differentiable: +32.41173%
- timm_efficientnet, Adadelta, cuda, no_foreach: +36.42987%
- phlippe_densenet, Adamax, cuda, no_foreach: +37.26340%
- phlippe_densenet, NAdam, cuda, no_foreach: +45.02834%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6503858966

cc @janeyx99


Noise, conglomerating in https://github.com/pytorch/benchmark/issues/1988
Repository: pytorch/benchmark
Issue: Permission Issue with installation.
Root cause: Environment Incompatibility

(python310) C:\Users\prs\work_projects\pytorch_benchmark>python install.py models phi_1_5
checking packages torch, torchvision, torchaudio are installed...OK
running setup for C:\Users\prs\work_projects\pytorch_benchmark\torchbenchmark\models\phi_1_5...Traceback (most recent call last):
  File "C:\Users\prs\work_projects\pytorch_benchmark\install.py", line 60, in <module>
    success &= setup(models=args.models, verbose=args.verbose, continue_on_fail=args.continue_on_fail, test_mode=args.test_mode, allow_canary=args.canary)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\prs\work_projects\pytorch_benchmark\torchbenchmark\__init__.py", line 153, in setup
    success, errmsg, stdout_stderr = _install_deps(model_path, verbose=verbose)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\prs\work_projects\pytorch_benchmark\torchbenchmark\__init__.py", line 99, in _install_deps
    os.remove(stdout_fpath)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\prs\\AppData\\Local\\Temp\\tmpu2dkrtmn'

Sorry we don't have Windows environment to test the installation. The benchmark suite assumes Linux only, and support on Windows/MacOS is maintained by the community.

Yes, I was able to get past this on Linux. There's some issue on Windows that needs to be debugged. I'll keep this open for now.
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '56af607c0437ed7321da4b96a4dbccdbd8b5a98b'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected Tests:
- DALLE2_pytorch, AdamW, cuda, default: +34.76710%
- Super_SloMo, RMSprop, cuda, differentiable: -33.06181%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6515446904

cc @janeyx99


noise
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '2438ac174d2f3ac0c99a1270ab3d100e08ac6bfe'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 56af607c0437ed7321da4b96a4dbccdbd8b5a98b

Affected PyTorch commit: 2438ac174d2f3ac0c99a1270ab3d100e08ac6bfe

Affected Tests:
- timm_vision_transformer_large, RAdam, cuda, differentiable: +79.81755%
- DALLE2_pytorch, Adam, cuda, foreach: -31.41948%
- DALLE2_pytorch, AdamW, cuda, default: -34.05249%
- DALLE2_pytorch, AdamW, cuda, foreach: -32.74721%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6521848431

cc @janeyx99


Noise--will be making a big issue tracking all these benchmarks https://github.com/pytorch/benchmark/issues/1988
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'abcc3fadc88502dd64dc52a0a208dc0606401f33'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 2438ac174d2f3ac0c99a1270ab3d100e08ac6bfe

Affected PyTorch commit: abcc3fadc88502dd64dc52a0a208dc0606401f33

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: +18.16546%
- test_eval[maml_omniglot-cuda-eager]_latency: -12.32743%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6525100749

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/6568512313

```
{
  "target_repo": "pytorch",
  "start": "625a3b1a428a8fa844090a87161b346b6d59075c",
  "end": "48989bc820d4dfbc0d23ce6f4a3e3e7495f5b315",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'abcc3fadc88502dd64dc52a0a208dc0606401f33'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 2438ac174d2f3ac0c99a1270ab3d100e08ac6bfe

Affected PyTorch commit: abcc3fadc88502dd64dc52a0a208dc0606401f33

Affected Tests:
- timm_vision_transformer_large, RAdam, cuda, no_foreach: +79.79328%
- timm_vision_transformer_large, RAdam, cuda, differentiable: -44.58859%
- DALLE2_pytorch, Adam, cuda, default: +36.11032%
- DALLE2_pytorch, AdamW, cuda, default: +40.75935%
- DALLE2_pytorch, AdamW, cuda, foreach: +37.00025%
- DALLE2_pytorch, AdamW, cuda, fused, capturable, amsgrad: +30.24755%
- sam, RAdam, cuda, no_foreach: -35.32676%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6528837699

cc @janeyx99


noise
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '6c94bfc7da1b91f110929dfb7f8b5fa6d6baaa0a'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: abcc3fadc88502dd64dc52a0a208dc0606401f33

Affected PyTorch commit: 6c94bfc7da1b91f110929dfb7f8b5fa6d6baaa0a

Affected Tests:
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: +10.31651%
- test_eval[yolov3-cuda-eager]_latency: +12.24338%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6536243306

cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/6564886474

```
{
  "target_repo": "pytorch",
  "start": "48989bc820d4dfbc0d23ce6f4a3e3e7495f5b315",
  "end": "4a388e70f2c7fedcdef598fb2144cc593452f04d",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'd902c353e928c8dccc1833b999fb97dfdde7ab12'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 6c94bfc7da1b91f110929dfb7f8b5fa6d6baaa0a

Affected PyTorch commit: d902c353e928c8dccc1833b999fb97dfdde7ab12

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: -20.32668%
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: -10.66427%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: -10.29876%
- test_eval[yolov3-cuda-eager]_latency: -14.08622%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6549849555

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/6553686295

```
{
  "target_repo": "pytorch",
  "start": "4a388e70f2c7fedcdef598fb2144cc593452f04d",
  "end": "013b51f8cc6f973d85c5adb36d5c110b7d42c7a2",
  "result": []
}
```

Closed.
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'd902c353e928c8dccc1833b999fb97dfdde7ab12'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 6c94bfc7da1b91f110929dfb7f8b5fa6d6baaa0a

Affected PyTorch commit: d902c353e928c8dccc1833b999fb97dfdde7ab12

Affected Tests:
- timm_vision_transformer_large, RAdam, cuda, no_foreach: -44.18635%
- DALLE2_pytorch, Adam, cuda, default: +30.85072%
- DALLE2_pytorch, AdamW, cuda, default: +39.92870%
- DALLE2_pytorch, AdamW, cuda, foreach: +37.87544%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6556047681

cc @janeyx99


noise, will deal with when i’m back from PTO