zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '8aa785fa2baa3280581c5fe0777e958821d07402'
Root cause: Resource Mishandling


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: aef7b6b424561141675db06dd41cf97fd42116ee

Affected PyTorch commit: 8aa785fa2baa3280581c5fe0777e958821d07402

Affected Tests:


Tests that were no longer run on affected commit:
- timm_vision_transformer_large, Adadelta, cuda, (pt2) default: 0.09635526003936927
- timm_vision_transformer_large, Adadelta, cuda, default: 0.09635420034949978
- timm_vision_transformer_large, Adadelta, cuda, (pt2) foreach: 0.09557987268393238
- timm_vision_transformer_large, Adadelta, cuda, foreach: 0.09508466130743425
- timm_vision_transformer_large, Adagrad, cuda, (pt2) default: 0.058340876013971865
- timm_vision_transformer_large, Adagrad, cuda, default: 0.05859477580524981
- timm_vision_transformer_large, Adagrad, cuda, (pt2) foreach: 0.05751379120629281
- timm_vision_transformer_large, Adagrad, cuda, foreach: 0.057667716802097854
- timm_vision_transformer_large, Adam, cuda, (pt2) default: 0.08039659967956443
- timm_vision_transformer_large, Adam, cuda, default: 0.07980197800012927
- timm_vision_transformer_large, Adam, cuda, (pt2) foreach: 0.07844041660428047
- timm_vision_transformer_large, Adam, cuda, foreach: 0.0785726213362068
- timm_vision_transformer_large, AdamW, cuda, (pt2) default: 0.08643882892404993
- timm_vision_transformer_large, AdamW, cuda, default: 0.08647659490816295
- timm_vision_transformer_large, AdamW, cuda, (pt2) foreach: 0.08519405724170308
- timm_vision_transformer_large, AdamW, cuda, foreach: 0.08504047368963559
- timm_vision_transformer_large, NAdam, cuda, (pt2) default: 0.09566177598511179
- timm_vision_transformer_large, NAdam, cuda, default: 0.0958128940158834
- timm_vision_transformer_large, NAdam, cuda, (pt2) foreach: 0.09461543934109311
- timm_vision_transformer_large, NAdam, cuda, foreach: 0.0945403470347325


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/4770804314

cc @janeyx99


This is fixed by moving things to subprocesses and then forcing dynamo to clear cache with a gc.collect :)
Repository: pytorch/benchmark
Issue: Tests (LearningToPaint, timm_vision_transformer_large) currently timing out in CI
Root cause: Algorithmic Error

Example logs: https://github.com/pytorch/benchmark/actions/runs/4810529752/jobs/8563280755

It seems to occur consistently after #1547: https://github.com/pytorch/benchmark/actions/workflows/pr-gha-runner.yml and the PR test for that verifies that it is likely culprit.
Repository: pytorch/benchmark
Issue: Build nightly torchbench docker to save CI time
Root cause: Unimplemented Feature Gap

The nightly docker can be used for:
- A10G PR test
- A100 PR test
- A100 userbenchmark test 

 - [x] Add script to check the installed torch, torchvision, torchaudio, torchtext versions are consistent and is the same as the date today.
 - [x] Add the docker build nightly workflow
 - [x] Push docker to remote
 - [x] Deploy docker to CI
 - [x] Test that the docker works on both A100 and A10G PR CI (https://github.com/pytorch/benchmark/pull/1613)
 - [x] Clone the "torchbench" env before the run, and clean it up after the run to restore the environment
 - [ ] Set the A100 nightly CI to run after the nightly docker workflow
 - [ ] Rewrite the CI to install only when it is needed
Repository: pytorch/benchmark
Issue: what means AssertionError: Canary model  not found in the canary_models path
Root cause: Environment Incompatibility

When doing

`python run.py "$name" -d cuda -t train
`
there are exceptions as:
e.g.

AssertionError: Canary model pytorch_CycleGAN_and_pix2pix is not found in the canary_models path


Have you run `python install.py` to install all models to your python environment?

to be checked. However, on another system the issue does not occur. Apology for the confusion

This means a dependency is missing for model pytorch_CycleGAN_and_pix2pix. Can you help post the full error log?

```
python run.py pytorch_CycleGAN_and_pix2pix -d cuda -t train
Warning: Could not find dependent module dominate for Model pytorch_CycleGAN_and_pix2pix, skip it. 
 No module named 'dominate'
Traceback (most recent call last):
  File "/media/joepareti54/Elements/x/finance-2023/AI/pytorch-AWS_benchmarks/LOCAL_test5/benchmark/run.py", line 301, in <module>
    Model = load_canary_model_by_name(args.model)
  File "/media/joepareti54/Elements/x/finance-2023/AI/pytorch-AWS_benchmarks/LOCAL_test5/benchmark/torchbenchmark/__init__.py", line 604, in load_canary_model_by_name
    assert _is_canary_model(model), f"Canary model {model} is not found in the {canary_model_dir} path."
AssertionError: Canary model pytorch_CycleGAN_and_pix2pix is not found in the canary_models path.
```

So the actual error message is ` No module named 'dominate'`, it is a dependency of pytorch_CycleGAN_and_pix2pix.
Repository: pytorch/benchmark
Issue: is it possible to run these tests with PyTorch 1.8 ?
Root cause: Environment Incompatibility

it seems that ``` python install.py ``` depends on having installed the newest releases

No, the main purpose of this repository is to evaluate the performance and prevent regressions of PyTorch nightly, so we do not maintain the compatibility of these tests with old PyTorch versions.

In the future, it might be an option to do branch cuts together with PyTorch stable releases, and use those branches for running tests on PyTorch stable release.
Repository: pytorch/benchmark
Issue: Make proper_bs.py a userbenchmark
Root cause: Unimplemented Feature Gap

We would like to deploy it as a new model profiling CI

When ever user submit a new model in PR, run a new model CI that:

1. profile it and upload the profiling result to Amazon S3
2. post the profiling link to the GH Issue
3. give suggestions on the proper batch size
4. run latency stableness check, inference result stableness check
5. for inference test, output dump flops numbers increase with batch size
Repository: pytorch/benchmark
Issue: Add check for nightly pytorch packages
Root cause: Unimplemented Feature Gap

In the nightly Docker workflow, before downloading the torch* nightly packages:

1. check if the torch package of the day is downloadable from the URL
2. if downloadable, print the package hash and the dependency versions, check if the dependency exists and the version is correct. Use pkginfo to check the package hash.

If any of them is incorrect, print error message and error out.


TODO: Add torch_trt package check for nightly
Repository: pytorch/benchmark
Issue: Cpu userbenchmark crashed when the `--launcher-args` with extra whitespace
Root cause: Validation Gap

For example,
```shell
$ python run_benchmark.py cpu -m resnet50 --launcher --launcher-args=" --throughput-mode"

Running benchmark: /localdisk/chuanqiw/miniconda3/envs/torchdynamo/bin/python -m torch.backends.xeon.run_cpu  --throughput-mode /localdisk/chuanqiw/PT/benchmark/userbenchmark/cpu/run_config.py -m resnet50 --device cpu -t eval -o /localdisk/chuanqiw/PT/benchmark/.userbenchmark/cpu/cpu-20230514213153
Traceback (most recent call last):
  File "/localdisk/chuanqiw/miniconda3/envs/torchdynamo/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/localdisk/chuanqiw/miniconda3/envs/torchdynamo/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/localdisk/chuanqiw/PT/pytorch/torch/backends/xeon/run_cpu.py", line 699, in <module>
    main(args)
  File "/localdisk/chuanqiw/PT/pytorch/torch/backends/xeon/run_cpu.py", line 659, in main
    raise RuntimeError("For non Python script, you should use \"--no-python\" parameter.")
RuntimeError: For non Python script, you should use "--no-python" parameter.
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '871fc7bb76f05c3c487214404f687cf7a6a8e453'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 61239df555df02e8c60a2ad63363878a2a57c161

Affected PyTorch commit: 871fc7bb76f05c3c487214404f687cf7a6a8e453

Affected Tests:
- test_eval[DALLE2_pytorch-cuda-eager]_latency: +8.31993%
- test_eval[doctr_det_predictor-cuda-eager]_cmem: +10.12839%
- test_eval[timm_resnest-cuda-eager]_gmem: +15.90606%
- test_train[torchrec_dlrm-cuda-eager]_cmem: +32.42321%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5047970627

cc @xuzhao9


2023-05-22
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '981d4c2578d10d8a96d173471802fc2812541fb1'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 871fc7bb76f05c3c487214404f687cf7a6a8e453

Affected PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected Tests:
- test_eval[doctr_det_predictor-cuda-eager]_cmem: -8.67876%
- test_eval[timm_resnest-cuda-eager]_gmem: -13.72323%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5059559500

cc @xuzhao9


2023-05-23
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '41917f7d923ad478e326102363df4e85c09c1dfc'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected PyTorch commit: 41917f7d923ad478e326102363df4e85c09c1dfc

Affected Tests:
- test_eval[detectron2_maskrcnn-cuda-eager]_cmem: +7.69990%
- test_train[fastNLP_Bert-cuda-eager]_cmem: +13.34432%
- test_eval[fastNLP_Bert-cuda-eager]_cmem: +16.89060%
- test_eval[yolov3-cuda-eager]_latency: +7.43747%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5070921215

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '41917f7d923ad478e326102363df4e85c09c1dfc'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected PyTorch commit: 41917f7d923ad478e326102363df4e85c09c1dfc

Affected Tests:
- test_eval[detectron2_maskrcnn-cuda-eager]_cmem: +7.69990%
- test_eval[doctr_det_predictor-cuda-eager]_cmem: +9.64539%
- test_train[fastNLP_Bert-cuda-eager]_cmem: +13.17957%
- test_eval[fastNLP_Bert-cuda-eager]_cmem: +15.93090%
- test_eval[timm_resnest-cuda-eager]_gmem: +7.04564%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5073002084

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected Tests:
- test_eval[Super_SloMo-cuda-eager]_cmem: +7.89916%
- test_train[fastNLP_Bert-cuda-eager]_cmem: +13.50906%
- test_eval[fastNLP_Bert-cuda-eager]_cmem: +16.12284%
- test_train[moco-cuda-eager]_gmem: -7.52278%
- test_eval[timm_resnest-cuda-eager]_gmem: +7.04564%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5082116103

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem: +7.05347%
- test_train[fastNLP_Bert-cuda-eager]_cmem: +13.34432%
- test_eval[fastNLP_Bert-cuda-eager]_cmem: +16.12284%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5082777241

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem: +7.96359%
- test_eval[detectron2_maskrcnn_r_50_fpn-cuda-eager]_cmem: +7.07965%
- test_eval[dlrm-cuda-eager]_latency: +30.68969%
- test_eval[doctr_det_predictor-cuda-eager]_cmem: +9.92908%
- test_train[fastNLP_Bert-cuda-eager]_cmem: +12.02636%
- test_eval[fastNLP_Bert-cuda-eager]_cmem: +16.89060%
- test_eval[opacus_cifar10-cuda-eager]_latency: +10.11113%
- test_eval[phlippe_densenet-cuda-eager]_latency: +8.90887%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: +7.20866%
- test_train[tacotron2-cuda-eager]_latency: +7.34085%
- test_eval[timm_resnest-cuda-eager]_gmem: +15.90606%
- test_eval[yolov3-cuda-eager]_latency: +7.34978%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5083541687

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected Tests:
- test_eval[DALLE2_pytorch-cuda-eager]_latency: +7.41947%
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem: +7.73606%
- test_eval[detectron2_maskrcnn-cuda-eager]_cmem: +8.29220%
- test_eval[detectron2_maskrcnn_r_50_fpn-cuda-eager]_cmem: +8.40708%
- test_eval[dlrm-cuda-eager]_latency: +7.14913%
- test_train[fastNLP_Bert-cuda-eager]_cmem: +12.85008%
- test_eval[fastNLP_Bert-cuda-eager]_cmem: +15.93090%
- test_train[phlippe_densenet-cuda-eager]_latency: +7.73124%
- test_eval[phlippe_densenet-cuda-eager]_latency: +7.54562%
- test_train[tacotron2-cuda-eager]_latency: +8.44564%
- test_eval[timm_resnest-cuda-eager]_gmem: +15.58580%
- test_train[torchrec_dlrm-cuda-eager]_latency: +10.44391%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5084129142

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 981d4c2578d10d8a96d173471802fc2812541fb1

Affected PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_cmem: +7.05347%
- test_eval[detectron2_maskrcnn-cuda-eager]_cmem: +7.99605%
- test_train[fastNLP_Bert-cuda-eager]_cmem: +12.85008%
- test_eval[fastNLP_Bert-cuda-eager]_cmem: +15.93090%
- test_train[opacus_cifar10-cuda-eager]_latency: +8.52980%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5085522927

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_cmem: +7.83069%
- test_train[lennard_jones-cuda-eager]_latency: +10.18362%
- test_train[opacus_cifar10-cuda-eager]_latency: -8.53818%
- test_eval[timm_resnest-cuda-eager]_gmem: +7.04564%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5090222907

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_cmem: +7.93651%
- test_eval[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_latency: +10.19378%
- test_eval[doctr_det_predictor-cuda-eager]_cmem: +7.87966%
- test_eval[hf_Bart-cuda-eager]_latency: +119.62223%
- test_eval[timm_resnest-cuda-eager]_gmem: +15.85268%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5091607527

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '10b46f7c7f69f9bf705d2b6ea53efb9c59145685'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected PyTorch commit: 10b46f7c7f69f9bf705d2b6ea53efb9c59145685

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_cmem: +7.19577%
- test_eval[doctr_det_predictor-cuda-eager]_cmem: +10.60172%
- test_train[opacus_cifar10-cuda-eager]_latency: -8.16085%
- test_eval[timm_resnest-cuda-eager]_gmem: +15.90606%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5092271489

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '10b46f7c7f69f9bf705d2b6ea53efb9c59145685'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected PyTorch commit: 10b46f7c7f69f9bf705d2b6ea53efb9c59145685

Affected Tests:
- test_train[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_latency: +7.11229%
- test_eval[llama-cuda-eager]_latency: +9.13903%
- test_train[opacus_cifar10-cuda-eager]_latency: -8.51704%
- test_eval[phlippe_resnet-cuda-eager]_latency: +20.98240%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5093295633

cc @xuzhao9
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '10b46f7c7f69f9bf705d2b6ea53efb9c59145685'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: eb2ef134b4e834a9b8a8b6de86ddd7d2780ce0ac

Affected PyTorch commit: 10b46f7c7f69f9bf705d2b6ea53efb9c59145685

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_101_dc5-cuda-eager]_cmem: +7.08995%
- test_train[opacus_cifar10-cuda-eager]_latency: -8.11113%
- test_eval[timm_resnest-cuda-eager]_gmem: +7.04564%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5094810765

cc @xuzhao9


2023-05-26

Test workflow: https://github.com/pytorch/benchmark/actions/runs/5204716477

Result:
```
{
  "target_repo": "pytorch",
  "start": "3ca068bc44440e780298d250f295d2b68a8a9ae3",
  "end": "fb468b6792213e0d8e6221b3bb51e71fcadbed30",
  "result": []
}
```

Closed.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '4316d8495a74264d27ad441f61af845ebd999639'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 3dedcb31fce3b5e55fee44d15eb693530c6606f8

Affected PyTorch commit: 4316d8495a74264d27ad441f61af845ebd999639

Affected Tests:
- test_eval[doctr_det_predictor-cuda-eager]_cmem: -9.14948%
- test_train[resnet50-cuda-eager]_gmem: +7.00060%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5113559506

cc @xuzhao9


cmem and gmem only change, skipped.