zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '9148727f4526481111054583ec3ac66aa98fab2b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 4316d8495a74264d27ad441f61af845ebd999639

Affected PyTorch commit: 9148727f4526481111054583ec3ac66aa98fab2b

Affected Tests:
- test_eval[timm_resnest-cuda-eager]_gmem: -13.72323%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5123740191

cc @xuzhao9


gmem only speedup, closed.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '48552338649ccc467060f5f93dbe19e2acbc4d1a'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 9148727f4526481111054583ec3ac66aa98fab2b

Affected PyTorch commit: 48552338649ccc467060f5f93dbe19e2acbc4d1a

Affected Tests:
- test_train[shufflenet_v2_x1_0-cuda-eager]_gmem: -7.54340%
- test_eval[timm_resnest-cuda-eager]_gmem: +15.05204%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5135124676

cc @xuzhao9


2023-05-31 (regression file fixed)

Workflow: https://github.com/pytorch/benchmark/actions/runs/5181836501

Output:
```
{
  "target_repo": "pytorch",
  "start": "0e72ada9bba2693e55fd7177ab4d2af826d7d15f",
  "end": "e7cc41772d65010749beab7734d29e32d1a7b494",
  "result": [
    {
      "commit1": "999bae0f541",
      "commit1_time": "2023-05-30 05:07:59 +0000",
      "commit1_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "999bae0f54108ffc5b7cf2524a02a83901554b16",
          "git_commit_hash": "999bae0f54108ffc5b7cf2524a02a83901554b16"
        },
        "metrics": {
          "test_eval[timm_resnest-cuda-eager]_latency": 39.871648,
          "test_eval[timm_resnest-cuda-eager]_cmem": 0.7060546875,
          "test_eval[timm_resnest-cuda-eager]_gmem": 3.6591796875
        }
      },
      "commit2": "076f84c46f3",
      "commit2_time": "2023-05-30 08:44:13 +0000",
      "commit2_digest": {
        "name": "torch-nightly",
        "environ": {
          "pytorch_git_version": "076f84c46f39b7ce7bff7bd4130b0c4f75e48753",
          "git_commit_hash": "076f84c46f39b7ce7bff7bd4130b0c4f75e48753"
        },
        "metrics": {
          "test_eval[timm_resnest-cuda-eager]_latency": 39.725244,
          "test_eval[timm_resnest-cuda-eager]_cmem": 0.70703125,
          "test_eval[timm_resnest-cuda-eager]_gmem": 4.2412109375
        }
      }
    }
  ]
}
```
Repository: pytorch/benchmark
Issue: Add Segment Anything
Root cause: Documentation Deficiency

https://segment-anything.com/

It is already added: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/sam/__init__.py

> It is already added: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/sam/__init__.py

thanks, https://github.com/pytorch/benchmark/blob/main/torchbenchmark/score/configs/v1/config-v1.md the list is outdated
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'a14be7981bcef6186441a6c5780976e27e6246ea'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 48552338649ccc467060f5f93dbe19e2acbc4d1a

Affected PyTorch commit: a14be7981bcef6186441a6c5780976e27e6246ea

Affected Tests:
- test_eval[opacus_cifar10-cuda-eager]_latency: +7.06551%
- test_eval[phlippe_resnet-cuda-eager]_latency: +7.10496%
- test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: +7.21425%
- test_eval[timm_resnest-cuda-eager]_gmem: -13.08281%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5146624396

cc @xuzhao9


2023-06-01

Result: https://github.com/pytorch/benchmark/actions/runs/5172020374

Empty result, closed.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'a14be7981bcef6186441a6c5780976e27e6246ea'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 48552338649ccc467060f5f93dbe19e2acbc4d1a

Affected PyTorch commit: a14be7981bcef6186441a6c5780976e27e6246ea

Affected Tests:
- test_train[detectron2_fasterrcnn_r_101_fpn-cuda-eager]_latency: +12.50772%
- test_train[shufflenet_v2_x1_0-cuda-eager]_gmem: +8.15886%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: +7.15283%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5149726362

cc @xuzhao9
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'a14be7981bcef6186441a6c5780976e27e6246ea'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 48552338649ccc467060f5f93dbe19e2acbc4d1a

Affected PyTorch commit: a14be7981bcef6186441a6c5780976e27e6246ea

Affected Tests:
- pytorch_unet, SGD, cuda, (pt2) foreach, momentum=0.9: +57.20391%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +50.13526%
- hf_Reformer, SGD, cuda, (pt2) foreach, momentum=0.9: +310.00312%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5151727961

cc @janeyx99


My guess is that these benchmarks are too small already to be meaningful. Will close this issue once I skip these. 
![image](https://github.com/pytorch/benchmark/assets/31798555/2504a92e-ff24-475b-b248-3474f8d0e80c)


These have no longer been triggering regressions, closing
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '7726721661ea114acb81a860519d0a1501d88fca'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: a14be7981bcef6186441a6c5780976e27e6246ea

Affected PyTorch commit: 7726721661ea114acb81a860519d0a1501d88fca

Affected Tests:
- test_train[detectron2_maskrcnn-cuda-eager]_latency: +9.25119%
- test_eval[doctr_det_predictor-cuda-eager]_cmem: +9.67283%
- test_train[lennard_jones-cuda-eager]_latency: +9.45848%
- test_train[mnasnet1_0-cuda-eager]_latency: +8.21508%
- test_train[mobilenet_v3_large-cuda-eager]_latency: +8.42297%
- test_eval[opacus_cifar10-cuda-eager]_latency: +8.53870%
- test_train[phlippe_densenet-cuda-eager]_latency: +14.65026%
- test_eval[phlippe_densenet-cuda-eager]_latency: +7.71325%
- test_train[phlippe_resnet-cuda-eager]_latency: +12.35126%
- test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]_latency: +9.91995%
- test_eval[timm_resnest-cuda-eager]_gmem: -13.08281%
- test_train[torchrec_dlrm-cuda-eager]_latency: +17.33417%
- test_eval[torchrec_dlrm-cuda-eager]_latency: +7.35767%
- test_train[tts_angular-cuda-eager]_latency: +9.57081%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5164520717

cc @xuzhao9


2023-06-03

Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/5176866505

Bisection workflow 2: https://github.com/pytorch/benchmark/actions/runs/5179562964

Bisection workflow 3: https://github.com/pytorch/benchmark/actions/runs/5181786643

Result:
```
{
  "target_repo": "pytorch",
  "start": "bebb8b7c1e6f3baf35c00d2a9265a285b4e439a8",
  "end": "39b04370db85313efe23966d04499970a882415c",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '7a5da818220cc4c950128db5ea65ec98dece559e'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 7726721661ea114acb81a860519d0a1501d88fca

Affected PyTorch commit: 7a5da818220cc4c950128db5ea65ec98dece559e

Affected Tests:
- test_train[phlippe_densenet-cuda-eager]_latency: -10.27443%
- test_eval[phlippe_densenet-cuda-eager]_latency: -9.05751%
- test_train[phlippe_resnet-cuda-eager]_latency: -7.84421%
- test_eval[timm_resnest-cuda-eager]_gmem: +7.04564%
- test_train[torchrec_dlrm-cuda-eager]_latency: -12.48976%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5179339828

cc @xuzhao9


2023-06-05

Workflow: https://github.com/pytorch/benchmark/actions/runs/5181829104

```
{
  "target_repo": "pytorch",
  "start": "39b04370db85313efe23966d04499970a882415c",
  "end": "881307abcfd8fd229fa521098597f4becb2d6c86",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: install.py installs datasets for all models, even if model to install is specified
Root cause: Validation Gap

Steps to reproduce: Run `python install.py asdfjksdfd`

Some models are sharing the same input sets like coco2017, and if all models need to specify their input data, we need to think about how to avoid downloading the same input data files repeatedly.

Currently https://github.com/pytorch/benchmark/pull/1734 only checks the existence of the file and directories without checking their integrity, which can be future work.
Repository: pytorch/benchmark
Issue: Comparation between compile and not compile in pytorch 2.0?


The compiled metrics are available here: https://hud.pytorch.org/benchmark/compilers

> The compiled metrics are available here: https://hud.pytorch.org/benchmark/compilers

is it possible to have this in local? It's cool

This website requires different infra services, see: https://github.com/pytorch/test-infra/tree/main/torchci

It is possible but won't be straightforward to deploy locally.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'c1406a99df2df9c06e8c7029e2eac41d5b2240cf'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 4d648e450b8e1386c0079f22c38aebc14fb93872

Affected PyTorch commit: c1406a99df2df9c06e8c7029e2eac41d5b2240cf

Affected Tests:
- test_eval[hf_T5_large-cuda-eager]_latency: -7.01952%
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: -8.59355%
- test_eval[vision_maskrcnn-cuda-eager]_latency: -75.07926%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5213143889

cc @xuzhao9


{
  "target_repo": "pytorch",
  "start": "872fdb329b75c832c2cd1314b3613e28289d8fac",
  "end": "d5142c52d3592e4a5777834be80b276ed1073012",
  "result": []
}
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'c1406a99df2df9c06e8c7029e2eac41d5b2240cf'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 4d648e450b8e1386c0079f22c38aebc14fb93872

Affected PyTorch commit: c1406a99df2df9c06e8c7029e2eac41d5b2240cf

Affected Tests:
- test_train[phlippe_densenet-cuda-eager]_latency: +7.46135%
- test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: +18.45467%
- test_eval[vision_maskrcnn-cuda-eager]_latency: -73.90283%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5213333440

cc @xuzhao9


{
  "target_repo": "pytorch",
  "start": "872fdb329b75c832c2cd1314b3613e28289d8fac",
  "end": "d5142c52d3592e4a5777834be80b276ed1073012",
  "result": []
}
Repository: pytorch/benchmark
Issue: Error: sometimes all packages are not build on the same date
Root cause: Documentation Deficiency

Hi,
for example last build torch is compiled 20230610 while torchdata is compiled in 20230608. How to force it on the same date for example 20230608?

It is possible that torch compiles successfully on 20230610 but torchdata/torchvision/torchtext/torchaudio does not. There is no tansactional guarantee on all torch* packge nightly release.

In this case, you need to use the torchbench docker: https://github.com/orgs/pytorch/packages/container/package/torchbench. For example, the [dev20230608](https://github.com/orgs/pytorch/packages/container/torchbench/99931534?tag=dev20230608) tag guarantees that all torch* packages are on 20230608.

You may check https://github.com/pytorch/benchmark/actions/workflows/build-nightly-docker.yml to see if today's torchbench nightly docker build successfully, if not, click into the workflow logs to see why.
Repository: pytorch/benchmark
Issue: Add torch.utils.flop_counter
Root cause: Unimplemented Feature Gap

Use FlopCounterMode to count the flops of the models.

Link to the PyTorch Core FLOP counter: https://dev-discuss.pytorch.org/t/the-ideal-pytorch-flop-counter-with-torch-dispatch/505

Added in https://github.com/pytorch/benchmark/commit/1dada69d4e4963506e0f76c9fc0b6945567ff611. Closed.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '43ec335ff295c55bd5d44a9fd03cfc884839a283'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 176a02ed90b218ffbf6a7b290ac28d37f06708ff

Affected PyTorch commit: 43ec335ff295c55bd5d44a9fd03cfc884839a283

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: +23.63661%
- test_eval[maml_omniglot-cuda-eager]_latency: +14.31050%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5392074355

cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/5393037151

```
{
  "target_repo": "pytorch",
  "start": "58feefa4ede4779d339d76b78b0e0a3de7d47212",
  "end": "968b7b5e0f892f993c65f8a365fb2df99359f09c",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '7bc858743ed14019fde2f05ce8ab0dab150c6099'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 43ec335ff295c55bd5d44a9fd03cfc884839a283

Affected PyTorch commit: 7bc858743ed14019fde2f05ce8ab0dab150c6099

Affected Tests:
- test_eval[maml_omniglot-cuda-eager]_latency: -13.00184%
- test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]_latency: +8.13162%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5414372536

cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/5416387008

Workflow 2nd try: https://github.com/pytorch/benchmark/actions/runs/5746035057


https://github.com/pytorch/benchmark/actions/runs/5751991174

```
{
  "target_repo": "pytorch",
  "start": "968b7b5e0f892f993c65f8a365fb2df99359f09c",
  "end": "998c07799f22a0bb63a39c84d15eae29778631f4",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'dc72046b235ac803e3875c23a1784e93b3d4812c'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 7bc858743ed14019fde2f05ce8ab0dab150c6099

Affected PyTorch commit: dc72046b235ac803e3875c23a1784e93b3d4812c

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: -20.43540%
- test_eval[maml_omniglot-cuda-eager]_latency: +8.17408%
- test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]_latency: -7.21416%
- test_eval[yolov3-cuda-eager]_latency: -7.73045%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5424816150

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/5425433714

```
{
  "target_repo": "pytorch",
  "start": "998c07799f22a0bb63a39c84d15eae29778631f4",
  "end": "89decc3a10137838cf4f7400af034645d52684e0",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'bb3df0bb7c6bce70941199401f6b3550e10cba50'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: dc72046b235ac803e3875c23a1784e93b3d4812c

Affected PyTorch commit: bb3df0bb7c6bce70941199401f6b3550e10cba50

Affected Tests:
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: -7.19603%
- test_eval[yolov3-cuda-eager]_latency: +7.59710%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5431907316

cc @xuzhao9


bisection workflow: https://github.com/pytorch/benchmark/actions/runs/5432711979

Closed.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '72404877d3437598669f8a746a0add48f6abcb4a'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: bb3df0bb7c6bce70941199401f6b3550e10cba50

Affected PyTorch commit: 72404877d3437598669f8a746a0add48f6abcb4a

Affected Tests:
- test_train[phlippe_resnet-cuda-eager]_latency: +8.58554%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5477664891

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/5488610916

```
{
  "target_repo": "pytorch",
  "start": "1de1bea60d96c75403beaa13b1ce80d0ea6ccdbd",
  "end": "13763f58ad86fadf49ef7960d1836318e6480d36",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '760dafbb05853f5f57f1a6869179df2efbc2cf6b'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 72404877d3437598669f8a746a0add48f6abcb4a

Affected PyTorch commit: 760dafbb05853f5f57f1a6869179df2efbc2cf6b

Affected Tests:
- test_eval[phlippe_densenet-cuda-eager]_latency: +7.00783%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5488340864

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/5511717722

```
{
  "target_repo": "pytorch",
  "start": "13763f58ad86fadf49ef7960d1836318e6480d36",
  "end": "a358a9262e4b8c7280d1817ac73f24a1bf383da6",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '3a919e00b8237a76ad6faa6040c00b425a96f1f3'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 760dafbb05853f5f57f1a6869179df2efbc2cf6b

Affected PyTorch commit: 3a919e00b8237a76ad6faa6040c00b425a96f1f3

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: +17.05591%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5495279958

cc @xuzhao9


2023-07-08 nightly release 

Test workflow: https://github.com/pytorch/benchmark/actions/runs/5755031881

```
{
  "target_repo": "pytorch",
  "start": "a358a9262e4b8c7280d1817ac73f24a1bf383da6",
  "end": "dbc22168008711f68775e9d5b159295527a9f108",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '9b5a84f5443c8e3b9db5511a4f58d727b4fade40'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 3a919e00b8237a76ad6faa6040c00b425a96f1f3

Affected PyTorch commit: 9b5a84f5443c8e3b9db5511a4f58d727b4fade40

Affected Tests:
- test_eval[hf_Bart-cuda-eager]_latency: +104.82938%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: -9.20539%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5500761066

cc @xuzhao9


```
{
  "target_repo": "pytorch",
  "start": "dbc22168008711f68775e9d5b159295527a9f108",
  "end": "dd6c38cb596199d70bba76688c99ed919d77002d",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: [export][llama] AssertionError: Mutating module attribute freqs_cis during export
Root cause: Validation Gap

```
        Mutating module attribute freqs_cis during export.
        
        from user code:
           File "/bert_ort/bowbao/repos/bench/torchbenchmark/torchbenchmark/models/llama/model.py", line 227, in forward
            self.freqs_cis = self.freqs_cis.to(h.device)
        
        Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
        
        
        You can suppress this exception and fall back to eager by setting:
            import torch._dynamo
            torch._dynamo.config.suppress_errors = True       
```

@msaroufim What's the general recommendation for this kind of error? Feels like the model code should be altered?

For a small set of popular models we've been OK with making some small changes to them