zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '8aa40192b25fd2fa304a640c6d5850afbcd9a51d'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: d902c353e928c8dccc1833b999fb97dfdde7ab12

Affected PyTorch commit: 8aa40192b25fd2fa304a640c6d5850afbcd9a51d

Affected Tests:
- timm_vision_transformer_large, RAdam, cuda, no_foreach: +78.79532%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6569698775

cc @janeyx99


noise—will deal with when i’m back on PTO
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '707b7b8613c0e6e19dbce565944d7c410e6b2ad7'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8aa40192b25fd2fa304a640c6d5850afbcd9a51d

Affected PyTorch commit: 707b7b8613c0e6e19dbce565944d7c410e6b2ad7

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6583221571

cc @janeyx99


known
Repository: pytorch/benchmark
Issue: tree2tree doesn't run with PyTorch HEAD
Root cause: External Dependency Breakage

After merging persist_rnn (which does work) with master pytorch/pytorch@909f31764faf65c80f4d8eaeae8a12c67eb24c17 tree2tree now fails with:

```
Traceback (most recent call last):
  File "tree2tree.py", line 683, in <module>
    train_until_fit(model, optimizer, data_iter, 1)
  File "tree2tree.py", line 613, in train_until_fit
    loss, dec_loss, seq_len = model(batch)
  File "/data/users/ezyang/pytorch/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "tree2tree.py", line 26, in wrapper
    r = func(*args, **kwargs)
  File "tree2tree.py", line 133, in forward
    batch.trg, encoding)
  File "tree2tree.py", line 26, in wrapper
    r = func(*args, **kwargs)
  File "tree2tree.py", line 544, in __call__
    self.step(self.tensor_actions[-1], targets)
  File "tree2tree.py", line 26, in wrapper
    r = func(*args, **kwargs)
  File "tree2tree.py", line 593, in step
    self.attention_states, self.alphas, self.ixs)
  File "tree2tree.py", line 26, in wrapper
    r = func(*args, **kwargs)
  File "tree2tree.py", line 424, in step
    x, self.stack_rnn(x, top.hc), parent=top.first)
  File "/data/users/ezyang/pytorch/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/users/ezyang/pytorch/torch/nn/modules/rnn.py", line 513, in forward
    self.bias_ih, self.bias_hh,
  File "/data/users/ezyang/pytorch/torch/nn/_functions/rnn.py", line 24, in LSTMCell
    igates = F.linear(input, w_ih)
  File "/data/users/ezyang/pytorch/torch/nn/functional.py", line 526, in linear
    return _functions.linear.Linear.apply(input, weight)
  File "/data/users/ezyang/pytorch/torch/nn/_functions/linear.py", line 12, in forward
    output.addmm_(0, 1, input, weight.t())
RuntimeError: size mismatch at /home/ezyang/local/pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:243
```

Cleaning up benchmark repo, closing old issues, please reopen if needed.
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '6ffe31abcae7d580c451cea195bd52258c72ac81'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 707b7b8613c0e6e19dbce565944d7c410e6b2ad7

Affected PyTorch commit: 6ffe31abcae7d580c451cea195bd52258c72ac81

Affected Tests:
- DALLE2_pytorch, RAdam, cuda, foreach: -31.19404%
- timm_vision_transformer_large, RAdam, cuda, differentiable: -44.37555%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6595126219

cc @janeyx99


noise and errors

will be “fixed” by #2000
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '147ac6b312c4c71e89013be592dc519c81fcac4e'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 6ffe31abcae7d580c451cea195bd52258c72ac81

Affected PyTorch commit: 147ac6b312c4c71e89013be592dc519c81fcac4e

Affected Tests:
- timm_vision_transformer_large, RAdam, cuda, differentiable: +79.47731%
- resnet152, Adadelta, cuda, maximize: +30.76596%
- DALLE2_pytorch, Adagrad, cuda, default: +30.41673%
- DALLE2_pytorch, Adagrad, cuda, maximize: +43.77639%
- DALLE2_pytorch, Adagrad, cuda, foreach: +34.07074%
- DALLE2_pytorch, AdamW, cuda, default: +42.56468%
- DALLE2_pytorch, SGD, cuda, maximize: +47.49684%
- DALLE2_pytorch, RAdam, cuda, foreach: +49.50562%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6601565408

cc @janeyx99


noise should be handled after #2000
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'f468e74875f6b7f95b7b01ccf3b05c3917e2865d'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 147ac6b312c4c71e89013be592dc519c81fcac4e

Affected PyTorch commit: f468e74875f6b7f95b7b01ccf3b05c3917e2865d

Affected Tests:
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: +11.29385%
- test_eval[yolov3-cuda-eager]_latency: +14.22212%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6604834645

cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/6720604288

```
{
  "target_repo": "pytorch",
  "start": "c84c86f018aea9aa9065ba3d57be1fc9fec51f5a",
  "end": "bf01a7b023c1ce4deb6316d25eadc1791807e88d",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'f468e74875f6b7f95b7b01ccf3b05c3917e2865d'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 147ac6b312c4c71e89013be592dc519c81fcac4e

Affected PyTorch commit: f468e74875f6b7f95b7b01ccf3b05c3917e2865d

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6608557268

cc @janeyx99


need to skip simple_gpt

will be reconsidering benchmarks after i come back from PTO
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'a422515e75cad4e785bbdbdff4f4e1c7c406bb24'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 88eb6bbb1ab58d7cdb49349b64c02a04911be8f2

Affected PyTorch commit: a422515e75cad4e785bbdbdff4f4e1c7c406bb24

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: -17.87432%
- test_eval[phlippe_resnet-cuda-eager]_latency: +10.49294%
- test_eval[yolov3-cuda-eager]_latency: +16.48829%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6629516682

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/6728010873

```
{
  "target_repo": "pytorch",
  "start": "c65c0682b19ed479ea66653fc0457958a3e2c59c",
  "end": "0e0f6a248db4ff054655ea405211dddc8e7ab817",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'e6efc2915b4ac65c784fbcbbf42ad87b49214417'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: a422515e75cad4e785bbdbdff4f4e1c7c406bb24

Affected PyTorch commit: e6efc2915b4ac65c784fbcbbf42ad87b49214417

Affected Tests:
- test_train[dcgan-cuda-eager]_latency: -11.45288%
- test_train[lennard_jones-cuda-eager]_latency: -11.72746%
- test_eval[maml-cuda-eager]_latency: -11.00041%
- test_train[maml_omniglot-cuda-eager]_latency: -10.99987%
- test_eval[opacus_cifar10-cuda-eager]_latency: -10.30727%
- test_eval[phlippe_densenet-cuda-eager]_latency: -12.09524%
- test_train[phlippe_resnet-cuda-eager]_latency: -13.23640%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6643078672

cc @xuzhao9


Test: https://github.com/pytorch/benchmark/actions/runs/6733585175

```
{
  "target_repo": "pytorch",
  "start": "0e0f6a248db4ff054655ea405211dddc8e7ab817",
  "end": "6e2dfb360b310a8676e96caa6550be083d72bd96",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: Reuse the code from dynamo bench for accuracy checking
Root cause: Architectural Constraint

Currently we are using our own code for accuracy checking:  https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/env_check.py#L426

We should avoid duplicating the code and directly call the code from the dynamobench for accuracy checking.
The accuracy checking code for dynamo bench is at: https://github.com/pytorch/benchmark/blob/main/userbenchmark/dynamo/dynamobench/common.py#L2090
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'bf9fd89f63f213a5c449dd589ac96cf9c9218885'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: e6efc2915b4ac65c784fbcbbf42ad87b49214417

Affected PyTorch commit: bf9fd89f63f213a5c449dd589ac96cf9c9218885

Affected Tests:
- test_train[lennard_jones-cuda-eager]_latency: +13.13242%
- test_eval[opacus_cifar10-cuda-eager]_latency: +11.17634%
- test_eval[phlippe_densenet-cuda-eager]_latency: +15.62673%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6656768120

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/6733592376

```
{
  "target_repo": "pytorch",
  "start": "6e2dfb360b310a8676e96caa6550be083d72bd96",
  "end": "2aaa7e542cb3b217e19284d10d0c0ae1e2df202b",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '525a926aaa127e59c71ed66915055ac027dba507'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: bf9fd89f63f213a5c449dd589ac96cf9c9218885

Affected PyTorch commit: 525a926aaa127e59c71ed66915055ac027dba507

Affected Tests:
- test_train[dcgan-cuda-eager]_latency: -10.02533%
- test_train[lennard_jones-cuda-eager]_latency: -18.16294%
- test_eval[opacus_cifar10-cuda-eager]_latency: -10.84995%
- test_eval[phlippe_densenet-cuda-eager]_latency: -13.99583%
- test_eval[yolov3-cuda-eager]_latency: -11.98668%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6669557382

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/6740805534

```
{
  "target_repo": "pytorch",
  "start": "2aaa7e542cb3b217e19284d10d0c0ae1e2df202b",
  "end": "c84dbd2c0393fed3e3d98030c85c30fbbfdc7c3e",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'c0e43d4625b81b7fffcd27334f791192505e3f55'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 525a926aaa127e59c71ed66915055ac027dba507

Affected PyTorch commit: c0e43d4625b81b7fffcd27334f791192505e3f55

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: +24.03732%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6677908793

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/6740810104

```
{
  "target_repo": "pytorch",
  "start": "c84dbd2c0393fed3e3d98030c85c30fbbfdc7c3e",
  "end": "31f605344f255faaef0aede0f98483f4cb09f072",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '07cc1132a95a98c64f7cc76bbd8940f981dd7642'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: c0e43d4625b81b7fffcd27334f791192505e3f55

Affected PyTorch commit: 07cc1132a95a98c64f7cc76bbd8940f981dd7642

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: -11.54413%
- test_train[mobilenet_v3_large-cuda-eager]_latency: +10.23197%
- test_train[phlippe_densenet-cuda-eager]_latency: +10.74853%
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: +10.68015%
- test_train[resnet18-cuda-eager]_latency: +11.87681%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6695512534

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/6740815200

```
{
  "target_repo": "pytorch",
  "start": "31f605344f255faaef0aede0f98483f4cb09f072",
  "end": "919c9b713e65aa1489123259a0ce7e79322d4d9d",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: Installing benchmarks fails with libtorchaudio.so underfined symbol error
Root cause: External Dependency Breakage

HI all, 

I have tried to install the benchmarks in a python environment where I have the following packages:

 - torch 2.1.0
 - torchxla 2.1.0
 - torchvision 0.16.0
 - torchdata 0.7.0
 - torchtext 0.16.0
 - torchaudio 2.1.0

and when installing the benchmarks I get the following error:

(python3-venv) aarch64_pytorch benchmark> python install.py 
checking packages torch, torchvision, torchaudio are installed...Traceback (most recent call last):
  File "/home/ubuntu/benchmark/install.py", line 39, in <module>
    versions = get_pkg_versions(TORCH_DEPS)
  File "/home/ubuntu/benchmark/utils/__init__.py", line 28, in get_pkg_versions
    module = importlib.import_module(module)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/ubuntu/python3-venv/lib/python3.10/site-packages/torchaudio/__init__.py", line 1, in <module>
    from . import (  # noqa: F401
  File "/home/ubuntu/python3-venv/lib/python3.10/site-packages/torchaudio/_extension/__init__.py", line 45, in <module>
    _load_lib("libtorchaudio")
  File "/home/ubuntu/python3-venv/lib/python3.10/site-packages/torchaudio/_extension/utils.py", line 64, in _load_lib
    torch.ops.load_library(path)
  File "/home/ubuntu/python3-venv/lib/python3.10/site-packages/torch/_ops.py", line 852, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/ubuntu/python3-venv/lib/python3.10/site-packages/torchaudio/lib/libtorchaudio.so: undefined symbol: _ZNK5torch8autograd4Node4nameEv


Any suggestion on what could be wrong? 
Thanks

I have been able to pass the error above by installing torchaudio from sources in my docker image. The install.py script has passed the checks for torch, torchvision, torchaudio and it's setting up the benchmarks.

This error is because torchaudio version you use is not ABI compatible with the torch version.

We only test the compatibility of pytorch/torchvision/torchaudio nightly releases. Please try it on the nightly releases.
Currently, we do not check if torchbenchmark runs on stable releases.
Repository: pytorch/benchmark
Issue: Speech_transformer issue
Root cause: Documentation Deficiency

I am trying to run the speech transformer model, but if I run it is displaying as "No hypothesis, Finish decoding"
1. Is there any way to fix it?
2. Can I pass my own wav audio file as input and predict it?

@davidberard98 can you help me please?

can you provide more details about how to reproduce the issue

> can you provide more details about how to reproduce the issue

1. I cloned the repo
2. python install.py model=speech transformer
3. python run.py speech transformer -d cpu -t eval

are you missing the underscore in the commands?

2. `python install.py speech_transformer`
3. `python run.py speech_transformer -d cpu -t eval`

> are you missing the underscore in the commands?
> 
> 2. `python install.py speech_transformer`
> 3. `python run.py speech_transformer -d cpu -t eval`

Thats not an issue, model working fine but prediction is not working
<img width="960" alt="Screenshot 2023-11-01 180338" src="https://github.com/pytorch/benchmark/assets/61832161/2ea5b088-05d3-4328-a3da-a3f3c280c244">



@Aswin-coder The goal of TorchBench core models is to run fast a single batch with one iteration test for performance testing, it does not run end-2-end model, therefore it can't produce e2e metrics such as prediction.

Therefore, you can't pass your own wav audio file and predict it.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'fd56aa0daeec624bcff822fef04fe9339770dc33'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: d2c3aa1fded6d1a0165eafe6e08816a85a1190d8

Affected PyTorch commit: fd56aa0daeec624bcff822fef04fe9339770dc33

Affected Tests:
- test_eval[DALLE2_pytorch-cuda-eager]_latency: +12.54825%
- test_train[dcgan-cuda-eager]_latency: +32.01213%
- test_train[detectron2_fasterrcnn_r_101_fpn-cuda-eager]_latency: +15.62042%
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: +13.46880%
- test_train[detectron2_fasterrcnn_r_50_fpn-cuda-eager]_latency: +12.28584%
- test_eval[hf_Bert-cuda-eager]_latency: +10.44053%
- test_train[lennard_jones-cuda-eager]_latency: +29.34321%
- test_eval[maml-cuda-eager]_latency: +24.93444%
- test_train[maml_omniglot-cuda-eager]_latency: +19.75669%
- test_train[mobilenet_v3_large-cuda-eager]_latency: +15.09031%
- test_eval[mobilenet_v3_large-cuda-eager]_latency: +10.93895%
- test_train[opacus_cifar10-cuda-eager]_latency: +15.84414%
- test_eval[opacus_cifar10-cuda-eager]_latency: +15.64927%
- test_train[phlippe_densenet-cuda-eager]_latency: +14.62465%
- test_eval[phlippe_densenet-cuda-eager]_latency: +15.91092%
- test_train[phlippe_resnet-cuda-eager]_latency: +19.91591%
- test_eval[phlippe_resnet-cuda-eager]_latency: +19.88433%
- test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]_latency: +12.15181%
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: +20.07919%
- test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: +14.88968%
- test_train[resnet18-cuda-eager]_latency: +21.09336%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: +10.72718%
- test_train[speech_transformer-cuda-eager]_latency: +10.47227%
- test_train[timm_efficientnet-cuda-eager]_latency: +10.42307%
- test_eval[timm_regnet-cuda-eager]_latency: +11.28450%
- test_train[tts_angular-cuda-eager]_latency: +14.82836%
- test_eval[vision_maskrcnn-cuda-eager]_latency: +10.08274%
- test_eval[yolov3-cuda-eager]_latency: +17.42123%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6747563114

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/6748552362

```{
  "target_repo": "pytorch",
  "start": "59869903b3a54acb3ac3a93470fc595d1de3fa7a",
  "end": "4a17693d193a8b23109ac3129e5cde92babfe191",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'a89fef71845d0dbc2c4c4a4c7878f51f4968ab90'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: fd56aa0daeec624bcff822fef04fe9339770dc33

Affected PyTorch commit: a89fef71845d0dbc2c4c4a4c7878f51f4968ab90

Affected Tests:
- test_train[dcgan-cuda-eager]_latency: -19.46518%
- test_train[lennard_jones-cuda-eager]_latency: -21.00054%
- test_eval[maml-cuda-eager]_latency: -14.48544%
- test_train[maml_omniglot-cuda-eager]_latency: -12.98050%
- test_train[mobilenet_v3_large-cuda-eager]_latency: -10.46858%
- test_eval[phlippe_densenet-cuda-eager]_latency: -11.16151%
- test_train[phlippe_resnet-cuda-eager]_latency: -14.43780%
- test_eval[phlippe_resnet-cuda-eager]_latency: -12.33820%
- test_eval[pyhpc_turbulent_kinetic_energy-cuda-eager]_latency: -10.86018%
- test_train[resnet18-cuda-eager]_latency: -13.69182%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: -11.43694%
- test_eval[timm_regnet-cuda-eager]_latency: -10.05124%
- test_train[tts_angular-cuda-eager]_latency: -11.65589%
- test_eval[yolov3-cuda-eager]_latency: -10.36281%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6755850399

cc @xuzhao9


2023-11-04:
https://github.com/pytorch/benchmark/actions/runs/6791122382

```
{
  "target_repo": "pytorch",
  "start": "4a17693d193a8b23109ac3129e5cde92babfe191",
  "end": "fb044e2b1783854da105c7166becb8cc37ac96c4",
  "result": []
}
```