zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: [Question][Background_Matting] Unused intermediate result `multi_feat`.

Observe `multi_feat` is unused and not returned. Should it be removed or is it a bug in the implementation?

https://github.com/pytorch/benchmark/blob/2ea018e9b6680e19d5ebb3267d6ceb6d14271dbc/torchbenchmark/models/Background_Matting/networks.py#L97-L111



This issue is from the upstream model code: https://github.com/senguptaumd/Background-Matting/blob/master/networks.py#L97

We prefer to keep the same code as the upstream code, to make sure we are improving together with the PyTorch ecosystem.
That being said, we are open to fix it, if you create a PR in the upstream repo, and reference to the upstream PR in the pytorch/benchmark PR.
Repository: pytorch/benchmark
Issue: install.py fails because of pyworld
Root cause: External Dependency Breakage

python install.py fails because of pyworld


```
running setup for /home/mat/github/benchmark/torchbenchmark/models/timm_resnest...OK
running setup for /home/mat/github/benchmark/torchbenchmark/models/timm_vision_transformer...OK
running setup for /home/mat/github/benchmark/torchbenchmark/models/timm_vision_transformer_large...OK
running setup for /home/mat/github/benchmark/torchbenchmark/models/timm_vovnet...OK
running setup for /home/mat/github/benchmark/torchbenchmark/models/tts_angular...FAIL
```

```
File "/home/mat/miniconda3/envs/torchbenchmark/lib/python3.10/json/encoder.py", line 179, in default
          raise TypeError(f'Object of type {o.__class__.__name__} '
      TypeError: Object of type PosixPath is not JSON serializable
```

Note:
When manually installing pyworld "pip install pyworld" I get the same error:

This problem seems specific to pyworld but anybody met this issue as well?


Specs:
Ubuntu 22.04 - WSL2
Python 3.10.12
benckmark commit: [7f71d89](https://github.com/angular/angular/commit/7f71d89c9ac0e3f66ff86a0bbbca452fb0d495cc)


Solved it using:
```
pip install pyworld --no-build-isolation
```

It might be best to fix this using `Pathlib` instead, if you wanna make a PR I can take a look

After searching the code base, pyworld is not really being used by this model.
Maybe just try removing pyworld from requirements.txt file: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/tts_angular/requirements.txt#L11

Then try running this model:

```
python test.py -k tts_angular
```

If it passes, we should remove this dependency.
Repository: pytorch/benchmark
Issue: model size info

Is there a way to get the number of parameters of each models for each tests?

for example:
```
python run.py llama -d cuda -t eval
```

How to know which llama model variant it is?  (7B, 13B, 33B, or 65B parameters)

cc @msaroufim Do you think we can add this information to the `metadata.yaml` ?

Sorry just saw this, I think for llama it might be best to rename the model 

For example for llama 2 its `llama_v2_7b_16h` to be very specific that its the version with 16 heads and 7b parameters
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '8a24a912a5f545d18059b59629aa3598f3783f25'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected Tests:
- shufflenet_v2_x1_0, ASGD, cuda, differentiable: -30.53309%
- yolov3, Adagrad, cuda, no_foreach: +36.72490%
- yolov3, NAdam, cuda, no_foreach: -32.89761%
- timm_resnest, NAdam, cuda, no_foreach: -32.32856%
- hf_Reformer, AdamW, cuda, no_foreach: -33.13247%
- nanogpt_generate, Adagrad, cuda, no_foreach: -30.80454%
- phlippe_densenet, AdamW, cuda, no_foreach: +47.23605%
- opacus_cifar10, AdamW, cuda, foreach, maximize, capturable, amsgrad: +31.05885%
- opacus_cifar10, Rprop, cuda, default: -33.20345%
- Background_Matting, RMSprop, cuda, differentiable: -32.00179%
- basic_gnn_gcn, ASGD, cuda, foreach: -32.20389%
- basic_gnn_gcn, RMSprop, cuda, no_foreach: -30.57904%
- speech_transformer, ASGD, cuda, differentiable: +50.44313%
- DALLE2_pytorch, RMSprop, cuda, differentiable: -34.96556%
- detectron2_maskrcnn_r_101_c4, Adam, cuda, no_foreach: +43.13601%
- basic_gnn_gin, Adadelta, cuda, no_foreach: -31.83760%
- basic_gnn_gin, Adadelta, cuda, differentiable: -31.86158%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5704173902

cc @janeyx99


Gotta do more to denoise
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '8a24a912a5f545d18059b59629aa3598f3783f25'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected Tests:
- phlippe_densenet, AdamW, cuda, no_foreach: -35.24609%
- hf_T5, Rprop, cuda, no_foreach: +40.75302%
- hf_Bert_large, RMSprop, cuda, no_foreach: +47.79120%
- hf_Bart, Rprop, cuda, no_foreach: +35.60743%
- timm_nfnet, Adadelta, cuda, no_foreach: +40.40037%
- detectron2_maskrcnn_r_101_c4, Adam, cuda, no_foreach: -34.86554%
- pytorch_struct, ASGD, cuda, foreach: +38.18946%
- speech_transformer, ASGD, cuda, differentiable: -37.29633%
- hf_T5_large, Rprop, cuda, no_foreach: +39.98589%
- DALLE2_pytorch, AdamW, cuda, amsgrad, maximize: -31.51808%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5710453078

cc @janeyx99


No actions needed
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '8a24a912a5f545d18059b59629aa3598f3783f25'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected Tests:
- hf_T5, Rprop, cuda, no_foreach: -32.30573%
- hf_T5_large, Rprop, cuda, no_foreach: -30.58711%
- hf_Bert_large, RMSprop, cuda, no_foreach: -35.80221%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5722438461

cc @janeyx99


The least noise in a while!
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '1d1670186abb4f2b8fd3a8024a8cf6c703d6d280'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected PyTorch commit: 1d1670186abb4f2b8fd3a8024a8cf6c703d6d280

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: +7.50122%
- test_eval[yolov3-cuda-eager]_latency: -8.57652%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5729150560

cc @xuzhao9


Bisection workflow: https://github.com/pytorch/benchmark/actions/runs/5740093854

Bisection output:
```
{
  "target_repo": "pytorch",
  "start": "707aadeedd081714060e51cee279abaea6305a98",
  "end": "ceea08a986805030014fac203d5b2411e23ff091",
  "result": []
}
```

Empty, closed.
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '1d1670186abb4f2b8fd3a8024a8cf6c703d6d280'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 8a24a912a5f545d18059b59629aa3598f3783f25

Affected PyTorch commit: 1d1670186abb4f2b8fd3a8024a8cf6c703d6d280

Affected Tests:
- DALLE2_pytorch, Adam, cuda, amsgrad, maximize: +43.49448%
- timm_vision_transformer, Adadelta, cuda, (pt2) default: -56.99585%
- timm_vision_transformer, Adadelta, cuda, (pt2) foreach: -53.90352%
- timm_vision_transformer, ASGD, cuda, (pt2) no_foreach: +30.78658%
- timm_vision_transformer, Rprop, cuda, (pt2) default: -65.58128%
- timm_vision_transformer, Rprop, cuda, (pt2) foreach: -66.67338%
- hf_DistilBert, Adadelta, cuda, no_foreach: +30.60847%
- resnet18, Adadelta, cuda, (pt2) default: -47.36183%
- resnet18, Adadelta, cuda, (pt2) foreach: -46.70130%
- resnet18, ASGD, cuda, (pt2) no_foreach: +36.94815%
- resnet18, Rprop, cuda, (pt2) default: -61.31989%
- resnet18, Rprop, cuda, (pt2) foreach: -60.53010%
- resnet18, NAdam, cuda, (pt2) no_foreach: +68.56828%


Tests that were no longer run on affected commit:
- timm_vision_transformer, NAdam, cuda, (pt2) no_foreach: 0.011040600889828056


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5734586409

cc @janeyx99


This is fixed by #1816
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '1d1670186abb4f2b8fd3a8024a8cf6c703d6d280'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 1d1670186abb4f2b8fd3a8024a8cf6c703d6d280

Affected PyTorch commit: 1d1670186abb4f2b8fd3a8024a8cf6c703d6d280

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5746465335

cc @janeyx99


"fixed" by #1816

The issue is the same that's already been reported in pytorch/pytorch "too many arguments (1520), maximum is 1024"
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'f89c73be3a3e8274d025ac46a33a780853841c9e'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 1d1670186abb4f2b8fd3a8024a8cf6c703d6d280

Affected PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected Tests:
- test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: -11.94105%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5753063401

cc @xuzhao9


```
{
  "target_repo": "pytorch",
  "start": "ceea08a986805030014fac203d5b2411e23ff091",
  "end": "a6f7dd4707ac116c0f5fb5f44f42429f38d23ab4",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'f89c73be3a3e8274d025ac46a33a780853841c9e'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 1d1670186abb4f2b8fd3a8024a8cf6c703d6d280

Affected PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected Tests:
- resnet50, NAdam, cuda, differentiable: +37.48276%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5792600919

cc @janeyx99


This is fixed
Repository: pytorch/benchmark
Issue: [PROBLEM] lstm script module has no attribute 'forward'
Root cause: External Dependency Breakage

I am currently using `torch==1.11.0`. I come up with a problem:

```python
x = torch.zeros([10])
layer = torch.nn.LSTM(10,64,num_layers=2)
script_layer = torch.jit.script(layer)
script_layer.forward(x)
```

and the error shows:
```
AttributeError: 'RecursiveScriptModule' object has no attribute 'forward'
```

Is there a way I can use scripted_lstm? 


same with https://github.com/pytorch/pytorch/issues/32976
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'f89c73be3a3e8274d025ac46a33a780853841c9e'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected PyTorch commit: f89c73be3a3e8274d025ac46a33a780853841c9e

Affected Tests:
- resnet50, NAdam, cuda, differentiable: -31.28031%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5804728865

cc @janeyx99
Repository: pytorch/benchmark
Issue: Monitor the CI status of the torch_trt userbenchmark
Root cause: Others

Test run: https://github.com/pytorch/benchmark/actions/runs/5833882739

Testing the torch_trt userbenchmark workflow: https://github.com/pytorch/benchmark/actions/runs/6341149525

Looking good: https://github.com/pytorch/benchmark/actions/runs/6409029498
Closing this PR.
Repository: pytorch/benchmark
Issue: how to run torchbenchmark in dynamo mode
Root cause: Documentation Deficiency

Hi,
 1. I want to test benchmark in dynamo mode, how can I run test_bench.py script?
 2. When I add code:
 `self.model = torch.compile(self.model)`
   in BERT_pytorch __init__.py, then run:
`pytest test_bench.py -k "test_train[BERT_pytorch-cuda-eager]" --ignore_machine_config --benchmark-autosave`, it raises below errors:
![image](https://github.com/pytorch/benchmark/assets/68674291/e92121ae-aa89-4558-bff3-17ee3ec10213)
how can I fix it? Thank you for you help~ @ezyang @orionr @romovpa @kostmo @zdevito 

`test_bench.py` does not support running in dynamo mode.
To run a model in dynamo mode, try:
```
python run.py BERT_pytorch -d cuda -t train --torchdynamo inductor
```




> `test_bench.py` does not support running in dynamo mode. To run a model in dynamo mode, try:
> 
> ```
> python run.py BERT_pytorch -d cuda -t train --torchdynamo inductor
> ```

OK, I see. Thank you for your help. @xuzhao9
Repository: pytorch/benchmark
Issue: Design the interface to support dynamic shapes and multiple batches.
Root cause: Unimplemented Feature Gap

By default, we should keep the consistent behaviour where each train or eval test runs only a single static batch.

Optionally, we should think about how to support multiple batches and let user set their preferred input streams for benchmarking.

My brief thoughts on how to design the interface:

`iter = self.get_input_iter() -> InputIterator`: return an iterator of input data, if max_batch is set, then len(iter) = max_batch, otherwise, return an iterator that will infinitely return some data batch (e.g. repeatly return the same batch).

cc @ezyang @janeyx99 @msaroufim @FindHao  for feedbacks

One thing I want to mention is that we also have to carefully review the input generation mechanism. torchrec_dlrm, in particular, doesn't actually generate inputs with varying sizes!

Also for my purposes, I don't need a fancy input distribution, I just need to be able to hit my model with a bunch of different unique shapes.

Can you help elaborate on the torchrec_dlrm case more? I thought the current assumption is that by default we are always generating the same inputs, and we should only provide dynamic inputs via the APIs suggested above.

I just mean upstream bugs like https://github.com/pytorch/torchrec/pull/1338

Can't take the dataset generators at their word, gotta make sure they actually work

The interface `get_input_iter()` has been added, need to test the input data generator is indeed working by unit tests.
Repository: pytorch/benchmark
Issue: RuntimeError: Triton Error [CUDA]: device kernel image is invalid
Root cause: Environment Incompatibility

Hi,
 When I run BERT_pytorch in torchdynamo mode, I get the output of the time and memory , but it raises below warning errors:
```
[2023-08-16 19:29:22,779] torch._dynamo.convert_frame: [WARNING] converting frame raised error, suppressing error
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING] WON'T CONVERT forward /test/workspaces/benchmark/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/language_model.py line 60 
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING] due to: 
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING] Traceback (most recent call last):
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING]   File "/test/anaconda3/envs/torchbenchmark/lib/python3.10/site-packages/triton/compiler/compiler.py", line 589, in _init_handles
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING]     mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device)
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING] torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING] RuntimeError: Triton Error [CUDA]: device kernel image is invalid
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING] 
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING] 
[2023-08-16 19:29:22,940] torch._dynamo.convert_frame: [WARNING] converting frame raised error, suppressing error
/test/anaconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/cuda/graphs.py:79: UserWarning: The CUDA Graph is empty. This usually means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at /opt/conda/conda-bld/pytorch_1690184485344/work/aten/src/ATen/cuda/CUDAGraph.cpp:190.)
  super().capture_end()
GPU Time:            104.788 milliseconds
CPU Total Wall Time: 104.821 milliseconds
GPU 3 Peak Memory:              5.6816 GB
CPU Peak Memory:                1.7363 GB
```

Does it matters?, Thank you for your help! @xuzhao9
```[tasklist]
### Tasks
```


Looks like a device driver and CUDA version compatiblity issue. What is the NVIDIA hardware, driver, and CUDA version you are using?

![image](https://github.com/pytorch/benchmark/assets/68674291/0b61c92c-3c30-402a-83c5-3dca7a4adada)


> Looks like a device driver and CUDA version compatiblity issue. What is the NVIDIA hardware, driver, and CUDA version you are using?
A100，40G memory, 1 node, 8 cards, cuda version: 11.0

@xuzhao9 


@Godlovecui 
We only support CUDA 11.8 or later: https://github.com/pytorch/benchmark/blob/main/utils/cuda_utils.py#L11

Could you please upgrade your CUDA version and try again?

> @Godlovecui We only support CUDA 11.8 or later: https://github.com/pytorch/benchmark/blob/main/utils/cuda_utils.py#L11
> 
> Could you please upgrade your CUDA version and try again?

OK，I will upgrade and try it again. Thank you~
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on 'bac60d88964c494932209f59e87236b479730e12'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 3522f2a7b7f73e928a8366cb7bd62ab3883dbe75

Affected PyTorch commit: bac60d88964c494932209f59e87236b479730e12

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5909336830

cc @janeyx99


The fix for this should be when we land https://github.com/pytorch/pytorch/pull/107857

The errors are indeed gone, but the performance regressed. Will close this and investigate further in https://github.com/pytorch/benchmark/issues/1886
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '668af075012c0857053a7cdf7ca764bb3569c6f1'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: bac60d88964c494932209f59e87236b479730e12

Affected PyTorch commit: 668af075012c0857053a7cdf7ca764bb3569c6f1

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5915274127

cc @janeyx99


duplicate of yesterdays
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '668af075012c0857053a7cdf7ca764bb3569c6f1'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 668af075012c0857053a7cdf7ca764bb3569c6f1

Affected PyTorch commit: 668af075012c0857053a7cdf7ca764bb3569c6f1

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5934294759

cc @janeyx99


same as #1839
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '1d3bf3da2129a882434dc8a44c14f21e826aac7b'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 668af075012c0857053a7cdf7ca764bb3569c6f1

Affected PyTorch commit: 1d3bf3da2129a882434dc8a44c14f21e826aac7b

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5946863436

cc @janeyx99


sorry, same as #1839
Repository: pytorch/benchmark
Issue: Add examples in algorithm_efficiency

Related post: https://discuss.pytorch.org/t/struggling-to-get-pytorch-fast-enough-to-use-in-public-competition/186015

Example link:
- FastMRI: https://github.com/mlcommons/algorithmic-efficiency/tree/main/algorithmic_efficiency/workloads/fastmri
- Criteo DLRMsmall: https://github.com/mlcommons/algorithmic-efficiency/tree/main/algorithmic_efficiency/workloads/criteo1tb
- WMT Transformer: https://github.com/mlcommons/algorithmic-efficiency/tree/main/algorithmic_efficiency/workloads/wmt
Repository: pytorch/benchmark
Issue: Optim Perf Signal Detected by TorchBench CI on '1d3bf3da2129a882434dc8a44c14f21e826aac7b'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 1d3bf3da2129a882434dc8a44c14f21e826aac7b

Affected PyTorch commit: 1d3bf3da2129a882434dc8a44c14f21e826aac7b

Affected Tests:


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/5959312095

cc @janeyx99


Seems related to https://github.com/pytorch/benchmark/issues/1839