zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.4.0.dev20240401+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240331+cu121

Base PyTorch commit: 35c493f2cf9b623bfdc7e6b34dc1cb39690a7919

Affected PyTorch version: 2.4.0.dev20240401+cu121

Affected PyTorch commit: 781e8d2201c1e2aaeccbbc7b7b13f9322b481bc9

Affected Tests:
- test_eval[pyhpc_equation_of_state-cpu-jit]: -12.93069%
- test_eval[pytorch_stargan-cpu-eager]: +9.73020%
- test_eval[pytorch_stargan-cpu-jit]: +8.94334%
- test_train[timm_regnet-cpu-eager]: +16.28495%


cc @xuzhao9

Result json: 
```
{
  "start": "35c493f2cf9b623bfdc7e6b34dc1cb39690a7919",
  "end": "781e8d2201c1e2aaeccbbc7b7b13f9322b481bc9",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "2b1ba0ceae2",
      "commit1_time": "2024-03-30 23:56:55 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07795581192476675
      },
      "commit2": "56451cd49d9",
      "commit2_time": "2024-03-31 03:07:32 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.08528426183620468
      }
    },
    {
      "commit1": "5fc0f52bf07",
      "commit1_time": "2024-03-31 16:07:38 +0000",
      "commit1_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.07554091756464913
      },
      "commit2": "781e8d2201c",
      "commit2_time": "2024-03-31 19:00:03 +0000",
      "commit2_digest": {
        "test_eval[pyhpc_equation_of_state-cpu-jit]": 0.0674160870257765,
        "test_eval[pytorch_stargan-cpu-eager]": 0.26128535440657286,
        "test_eval[pytorch_stargan-cpu-jit]": 0.29358342713676394,
        "test_train[timm_regnet-cpu-eager]": 0.8068878411781043
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8526822455


Noise, closing
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.4.0.dev20240401+cu121'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 35c493f2cf9b623bfdc7e6b34dc1cb39690a7919
Control PyTorch version: 2.4.0.dev20240331+cu121

Treatment PyTorch commit: 781e8d2201c1e2aaeccbbc7b7b13f9322b481bc9
Treatment PyTorch version: 2.4.0.dev20240401+cu121

Bisection result:

```
[
    {
        "commit1": "6d8bb0e9841",
        "commit1_time": "2024-03-31 09:06:35 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "6d8bb0e9841b3affb31a67ac26fc305bbccf1a97",
                "pytorch_version": "2.4.0a0+git6d8bb0e",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "6d8bb0e9841b3affb31a67ac26fc305bbccf1a97"
            },
            "metrics": {
                "test_eval[dlrm-cuda-eager]_latency": 1.429256,
                "test_eval[dlrm-cuda-eager]_cmem": 0.4921875,
                "test_eval[dlrm-cuda-eager]_gmem": 3.46673583984375
            }
        },
        "commit2": "6c2f36c9845",
        "commit2_time": "2024-03-31 11:29:40 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "6c2f36c9845f310db8ece23c0d2e4ad6f702bc57",
                "pytorch_version": "2.4.0a0+git6c2f36c",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "6c2f36c9845f310db8ece23c0d2e4ad6f702bc57"
            },
            "metrics": {
                "test_eval[dlrm-cuda-eager]_latency": 1.593378,
                "test_eval[dlrm-cuda-eager]_cmem": 0.4921875,
                "test_eval[dlrm-cuda-eager]_gmem": 3.46673583984375
            }
        }
    }
]
```

cc @xuzhao9


Noise, closed.
Repository: pytorch/benchmark
Issue: Reduce the size of the docker image
Root cause: Architectural Constraint

I noticed that the Docker image is significantly large.

The base image `ghcr.io/actions/actions-runner:latest` is about 1GB and the benchmark folder is about 9GB, but the whole torchbench image is over 40GB. 

Each `RUN` command in the [gcp-a100-runner-dind.dockerfile](https://github.com/pytorch/benchmark/blob/main/docker/gcp-a100-runner-dind.dockerfile) creates a layer during the Docker image building process. This could be a contributing factor to the image's size. Perhaps we could merge all commands into a single script or use && to chain them together, effectively reducing the number of layers.



The docker image contains both CUDA 11.8 and 12.1 runtime for release testing (needs to compare the performance of 2 CUDA versions). I suspect that is the main source of image size bloating.


```
du -h /usr/local/cuda-11.8
9.8G	cuda-11.8
```

```
du -h /usr/local/cuda-12.1
9.5G	cuda-12.1
```

```
du -h $HOME/miniconda3
8.8G	miniconda3/
```
I am wondering how much does reducing the number of layers help in this case?

I see. It looks like only several GBs, maybe 1 or 2,  we can reduce via reducing extra laybers.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.4.0.dev20240405+cu121'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240404+cu121

Base PyTorch commit: 669462817071d1badd88a832a7c38b9bafde2233

Affected PyTorch version: 2.4.0.dev20240405+cu121

Affected PyTorch commit: d8717c2d68bc0a8bda7e8ad823a0d6c2d8aae934

Affected Tests:
- test_eval[pytorch_stargan-cpu-eager]: -8.91440%
- test_eval[pytorch_stargan-cpu-jit]: +8.27639%
- test_train[tts_angular-cpu-eager]: -17.67907%


cc @xuzhao9

Result json: 
```
{
  "start": "669462817071d1badd88a832a7c38b9bafde2233",
  "end": "d8717c2d68bc0a8bda7e8ad823a0d6c2d8aae934",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "595613d746d",
      "commit1_time": "2024-04-04 14:20:51 +0000",
      "commit1_digest": {
        "test_train[tts_angular-cpu-eager]": 0.21390078791882844
      },
      "commit2": "e0c97646600",
      "commit2_time": "2024-04-04 16:05:10 +0000",
      "commit2_digest": {
        "test_train[tts_angular-cpu-eager]": 0.23464588292408733
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8573396710


Noise, closed
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '2.4.0.dev20240405+cu121'
Root cause: Others


TorchBench CI has detected a performance signal or runtime regression, and bisected its result.

Control PyTorch commit: 669462817071d1badd88a832a7c38b9bafde2233
Control PyTorch version: 2.4.0.dev20240404+cu121

Treatment PyTorch commit: d8717c2d68bc0a8bda7e8ad823a0d6c2d8aae934
Treatment PyTorch version: 2.4.0.dev20240405+cu121

Bisection result:

```
[
    {
        "commit1": "6890333e3de",
        "commit1_time": "2024-04-04 08:55:00 +0000",
        "commit1_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "6890333e3dec55499d87d2193d03109f04b8779a",
                "pytorch_version": "2.4.0a0+git6890333",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "6890333e3dec55499d87d2193d03109f04b8779a"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.003924,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.490234375,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.55462646484375
            }
        },
        "commit2": "54801e6fd6d",
        "commit2_time": "2024-04-04 13:22:22 +0000",
        "commit2_digest": {
            "name": "torch-nightly",
            "environ": {
                "pytorch_git_version": "54801e6fd6d8c94e6e18433fe45d4cc36411ab9c",
                "pytorch_version": "2.4.0a0+git54801e6",
                "device": "NVIDIA A100-SXM4-40GB",
                "git_commit_hash": "54801e6fd6d8c94e6e18433fe45d4cc36411ab9c"
            },
            "metrics": {
                "test_eval[maml_omniglot-cuda-eager]_latency": 1.224879,
                "test_eval[maml_omniglot-cuda-eager]_cmem": 0.4892578125,
                "test_eval[maml_omniglot-cuda-eager]_gmem": 1.55462646484375
            }
        }
    }
]
```

cc @xuzhao9


Noise, closed
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.4.0.dev20240408+cu121'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240407+cu121

Base PyTorch commit: e3ea31662334770bbbb6da4abd881abc875e04c3

Affected PyTorch version: 2.4.0.dev20240408+cu121

Affected PyTorch commit: bb04f3f66a5b92f0bed3712689f57774f00db349

Affected Tests:
- test_eval[pytorch_stargan-cpu-jit]: +8.85221%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +7.53370%
- test_train[timm_regnet-cpu-eager]: +15.58225%
- test_train[tts_angular-cpu-eager]: +12.16892%


cc @xuzhao9

Result json: 
```
{
  "start": "e3ea31662334770bbbb6da4abd881abc875e04c3",
  "end": "bb04f3f66a5b92f0bed3712689f57774f00db349",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "07cecf41685",
      "commit1_time": "2024-04-07 21:09:05 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.28941417232854294
      },
      "commit2": "bb04f3f66a5",
      "commit2_time": "2024-04-08 01:50:04 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2629191126557998,
        "test_eval[shufflenet_v2_x1_0-cpu-eager]": 0.15328924008645117,
        "test_train[timm_regnet-cpu-eager]": 0.8281278012320399,
        "test_train[tts_angular-cpu-eager]": 0.20588519893353804
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8604111670
Repository: pytorch/benchmark
Issue: CI Failure because of sam_fast CUDA mem leak
Root cause: Resource Mishandling

Failed workflow: https://github.com/pytorch/benchmark/actions/runs/8662210535/job/23753705235

Running bisection: https://github.com/pytorch/benchmark/actions/runs/8662953920


```
======================================================================
FAIL: test_sam_fast_eval_cuda (__main__.TestBenchmark.test_sam_fast_eval_cuda)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/_work/benchmark/benchmark/test.py", line 115, in eval_fn
    with task.watch_cuda_memory(
  File "/home/runner/miniconda3/envs/pr-ci-a100/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/home/runner/_work/benchmark/benchmark/torchbenchmark/__init__.py", line 625, in watch_cuda_memory
    assert_equal(
AssertionError: 0 != 12288

----------------------------------------------------------------------
Ran 800 tests in 6516.228s

FAILED (failures=1, skipped=205)
```

```
{
  "target_repo": "pytorch",
  "start": "d8e0c26e642bf8e44cc6e8d46822d54957b1c660",
  "start_version": "2.4.0.dev20240411+cu121",
  "end": "65710d95c9adcc166e4d4a19ac16b8d7516fc87b",
  "end_version": "2.4.0.dev20240411+cu121",
  "result": [
    {
      "commit1": "f772ea5493a",
      "commit1_time": "2024-04-09 19:39:31 +0000",
      "commit1_digest": {
        "name": "test_bench",
        "environ": {
          "pytorch_git_version": "f772ea5493a829cdd7f8c0b53f2a8f15a18a06ff",
          "pytorch_version": "2.4.0a0+gitf772ea5",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "f772ea5493a829cdd7f8c0b53f2a8f15a18a06ff"
        },
        "metrics": {
          "model=sam_fast, test=eval, device=cuda, bs=None, extra_args=['--memleak'], metric=memleak": "False"
        }
      },
      "commit2": "4044e93a51e",
      "commit2_time": "2024-04-09 19:42:19 +0000",
      "commit2_digest": {
        "name": "test_bench",
        "environ": {
          "pytorch_git_version": "4044e93a51e9238de3049a39e203688809df9bc5",
          "pytorch_version": "2.4.0a0+git4044e93",
          "device": "NVIDIA A100-SXM4-40GB",
          "git_commit_hash": "4044e93a51e9238de3049a39e203688809df9bc5"
        },
        "metrics": {
          "model=sam_fast, test=eval, device=cuda, bs=None, extra_args=['--memleak'], metric=memleak": "True"
        }
      }
    }
  ]
}
```

Fixed by https://github.com/pytorch/pytorch/pull/124345
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.4.0.dev20240420+cu121'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.4.0.dev20240419+cu121

Base PyTorch commit: 4946638f06e5916ea9bd0f790ff620bdb78a92a3

Affected PyTorch version: 2.4.0.dev20240420+cu121

Affected PyTorch commit: bad8d25881d850eaf0b326f6ce5c78305e38c001

Affected Tests:
- test_eval[pytorch_stargan-cpu-jit]: -11.31654%


cc @xuzhao9

Result json: 
```
{
  "start": "4946638f06e5916ea9bd0f790ff620bdb78a92a3",
  "end": "bad8d25881d850eaf0b326f6ce5c78305e38c001",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "4946638f06e",
      "commit1_time": "2024-04-19 00:51:24 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.24706086833029986
      },
      "commit2": "ddd0ed1b430",
      "commit2_time": "2024-04-19 00:57:08 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2903036438394338
      }
    },
    {
      "commit1": "0bde4efa842",
      "commit1_time": "2024-04-19 01:54:30 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2935471278615296
      },
      "commit2": "e620c3e814e",
      "commit2_time": "2024-04-19 01:58:19 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2664948482764885
      }
    },
    {
      "commit1": "87f44d70b19",
      "commit1_time": "2024-04-19 04:07:00 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2654835177352652
      },
      "commit2": "1ba85b34dd5",
      "commit2_time": "2024-04-19 04:47:27 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.24438257701694965
      }
    },
    {
      "commit1": "1ba85b34dd5",
      "commit1_time": "2024-04-19 04:47:27 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.24438257701694965
      },
      "commit2": "96a067b190b",
      "commit2_time": "2024-04-19 04:49:07 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2961612632498145
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/8766598360


Noise, closed.
Repository: pytorch/benchmark
Issue: Reason for chosing `data point: i = 20, size = 1048576` for pyhpc benchmarks
Root cause: Documentation Deficiency

For pyhpc_equation_of_state:
https://github.com/pytorch/benchmark/blob/3f146bbe7fdd65c6f4490993a8e0fa78c4465f37/torchbenchmark/models/pyhpc_equation_of_state/__init__.py#L31C5-L35C33
```python
class Model(BenchmarkModel):
    task = OTHER.OTHER_TASKS
    # Original size: [2 ** i for i in range(12, 23, 2)
    # Source: https://github.com/dionhaefner/pyhpc-benchmarks/blob/650ecc650e394df829944ffcf09e9d646ec69691/run.py#L25
    # Pick data point: i = 20, size = 1048576
    DEFAULT_EVAL_BSIZE = 1048576
    CANNOT_SET_CUSTOM_OPTIMIZER = True
```
---

For  pyhpc_turbulent_kinetic_energy:
https://github.com/pytorch/benchmark/blob/3f146bbe7fdd65c6f4490993a8e0fa78c4465f37/torchbenchmark/models/pyhpc_turbulent_kinetic_energy/__init__.py#L124C5-L129C39
```python
class Model(BenchmarkModel):
    task = OTHER.OTHER_TASKS

    # Original input size: [2 ** i for i in range(12, 23, 2)]
    # Source: https://github.com/dionhaefner/pyhpc-benchmarks/blob/650ecc650e394df829944ffcf09e9d646ec69691/run.py#L25
    # Pick data-point when i = 20, size = 1048576
    DEFAULT_EVAL_BSIZE = 1048576
    ALLOW_CUSTOMIZE_BSIZE = False
    CANNOT_SET_CUSTOM_OPTIMIZER = True
```

I was trying to test these with [TorchDynamo Benchmark](https://github.com/pytorch/pytorch/tree/main/benchmarks/dynamo)

I was wondering, is there any specific reason for chosing that specific `batch_size` of $2^{20}$? Something to do with dynamo performance? 
In the `pyhpc_equation_of_state` we are allowed to make use of custom batch size, but for `pyhpc_turbulent_kinectic_energy` and cannot make use of custom batch size. Why is this difference?

It is not dynamo performance related. We did a sweep benchmarking across the original input sizes on PyTorch eager mode, and on A100 it reaches the highest tflops when i=20 with the eager mode.

I think we had some issues with `pyhpc_turbulent_kinectic_energy ` when checking the numeric parity when changing the batch size. By default, if the model is able to change the batch size, the accuracy checking code will set the batch size to 4 to minimize the time needed to check the accuracy.
Repository: pytorch/benchmark
Issue: Unable to install with the latest `pip`
Root cause: External Dependency Breakage

This is a dependency issue. See https://github.com/facebookresearch/detectron2/issues/4921. Either `pip<=22` or explicit `--no-build-isolation` in scripts in this repo (after explicitly installing some build-time dependencies) is required.

Can you please pull the latest Docker image and check what is the pip version? 
https://github.com/pytorch/benchmark/pkgs/container/torchbench

> Can you please pull the latest Docker image and check what is the pip version? https://github.com/pytorch/benchmark/pkgs/container/torchbench

It is pip 23.3.1 from miniconda. I was using pure Python. I think the behavior of pip in miniconda and in pure Python may be different.

Since we assume python/pip/torch/torch* packages is already installed in the environment before running `python install.py`, we have no control of which pip version is installed on user's environment, and the only CI we have is the nightly docker. In the best effort we can add a warning about user's pip version, and I believe this is an issue that should be handled by detectron2 developers.

If I understand correctly, detectron2 detects installed torch when installing, which cannot be done with build isolation. I think detectron2 developers may not be able to solve the problem unless by installing torch by themselves, but that will make the torch version inconsistent with ours.

Maybe explicitly disabling build isolation should be considered. I have successfully installed this repo with `PIP_NO_BUILD_ISOLATION=1` environment variable, but this is applied to all packaged this repo depends on. A better solution may be only disable build isolation for detectron2.

@roastduck Can you please add the `PIP_NO_BUILD_ISOLATION=1` at https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/framework/detectron2/__init__.py#L46 and test? If it works, we can disable build isolation only for detectron2.

It works and please find it in PR #2254.
Repository: pytorch/benchmark
Issue: RuntimeError When Enabling Accuracy Checks in DALLE2_pytorch Training on GPU.
Root cause: Validation Gap

**Issue Description**
I encounter a RuntimeError related to gradient computation when enabling accuracy checks during the training of DALLE2_pytorch in a GPU docker environment. The training runs without issues when the `--accuracy` flag is not used.

**Steps to Reproduce**
`python install.py DALLE2_pytorch`
`python run.py DALLE2_pytorch -d cuda -t train --accuracy`

**Expected Behavior**
The training process should run without errors and perform accuracy checks without causing runtime errors.

**Actual Behavior**
The script executes successfully without the --accuracy flag. 
However, when the accuracy check is enabled, it fails with the following error message:

```
fp64 golden ref were not generated for DALLE2_pytorch. Setting accuracy check to cosine
element 0 of tensors does not require grad and does not have a grad_fn
Traceback (most recent call last):
  File "/benchmark/torchbenchmark/util/env_check.py", line 635, in check_accuracy
    correct_result = run_n_iterations(
  File "/benchmark/torchbenchmark/util/env_check.py", line 504, in run_n_iterations
    _model_iter_fn(mod, inputs, contexts, optimizer, collect_outputs=False)
  File "/benchmark/torchbenchmark/util/env_check.py", line 497, in _model_iter_fn
    return forward_and_backward_pass(
  File "/benchmark/torchbenchmark/util/env_check.py", line 480, in forward_and_backward_pass
    DummyGradScaler().scale(loss).backward(retain_graph=True)
  File "/venv_cuda/pytorch/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward
    torch.autograd.backward(
  File "/venv_cuda/pytorch/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
```

**Additional Context**
PyTorch version: 2.2.2
CUDA version: 12.4.0.041

I can confirm that this can be reproduced in the docker environment.
@FindHao Can you help take a look at this issue?

@xuzhao9 The problem also occurs on the previous version of TorchBench(ghcr.io/pytorch/torchbench:dev20230619). It looks like it is from the first time DALLE2 was included in TorchBench. I'm not sure if we can fix it on our side or from the upstream repo since we have limited control over the model's [__init__.py](https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/DALLE2_pytorch/__init__.py). I'll have a try.

We are dropping DALLE2_pytorch because it does not support numpy 2.0: https://github.com/pytorch/benchmark/pull/2311