zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: redundant memory allocation maybe the root cause of OOMs
Root cause: Resource Mishandling

Hi @xuzhao9 , 

during the investigation of LLAMA_7b OOM issue, we found that there are many redundant memory allocation. maybe it's not necessary for test.
1,  there is deepcopy for maybe_cast() and deepcopy_and_maybe_cast().  which would duplicate the memory on GPU allocated for this model. 
https://github.com/pytorch/benchmark/blob/main/userbenchmark/dynamo/dynamobench/common.py#L2400
https://github.com/pytorch/benchmark/blob/main/userbenchmark/dynamo/dynamobench/common.py#L2403

looks we need to check more strictly on deepcopy.

2,  there is deepcopy in validate_model() too.
https://github.com/pytorch/benchmark/blob/main/userbenchmark/dynamo/dynamobench/common.py#L1918

we can run the LLAMA_7b model(which has OOM issue previously https://github.com/pytorch/benchmark/issues/2051 ) with one A100 40G after commenting out the unnecessary deepcopy().

hope this information can help on fixing the OOM issues in this repo.

Thanks




dynamobench is owned by the PT2 team. In my understanding, it is used for accuracy check because some models are stateful.
cc @desertfire is there a way to turn off deepcopy in dynamobench?
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '5ae273f939d7a98b83d6717254d54c008933f6a8'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: b2f25d6342ed483b461e831c6f970ae59a4fcca2

Affected PyTorch commit: 5ae273f939d7a98b83d6717254d54c008933f6a8

Affected Tests:
- test_train[dcgan-cuda-eager]_latency: -10.44704%
- test_train[lennard_jones-cuda-eager]_latency: -10.54675%
- test_train[maml_omniglot-cuda-eager]_latency: -13.26192%
- test_train[mobilenet_v3_large-cuda-eager]_latency: -12.25009%
- test_train[opacus_cifar10-cuda-eager]_latency: -10.56186%
- test_eval[phlippe_densenet-cuda-eager]_latency: -11.04776%
- test_train[phlippe_resnet-cuda-eager]_latency: -15.18761%
- test_eval[shufflenet_v2_x1_0-cuda-eager]_latency: -11.67582%
- test_train[tts_angular-cuda-eager]_latency: -16.09062%
- test_eval[yolov3-cuda-eager]_latency: -17.31685%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/7021447400

cc @xuzhao9


bisection: https://github.com/pytorch/benchmark/actions/runs/7102622653

https://github.com/pytorch/benchmark/actions/runs/7116566298

```
{
  "target_repo": "pytorch",
  "start": "624f2025229312bffd725775760dc9b147cc32d8",
  "end": "11f11e95df9c205d427fe4dd7e63c9adb91ea03f",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: ModuleNotFoundError: No module named 'caffe2'
Root cause: Environment Incompatibility

To reproduce:

`pip install --pre torch torchvision torchaudio -i https://download.pytorch.org/whl/nightly/cu118`

`git clone https://github.com/pytorch/benchmark
cd benchmark
python install.py`

As per README

and then
`python run_benchmark.py dynamo`

Platform: Ubuntu 20.04.6 LTS
Python: 3.8.10
Torch: 2.1.1

Full error:

Failed to import user benchmark module dynamo, error: No module named 'caffe2'
Traceback (most recent call last):
  File "/data/agoldf6/benchmark/userbenchmark/dynamo/dynamobench/torchbench.py", line 14, in <module>
    from .common import BenchmarkRunner, main
ImportError: attempted relative import with no known parent package

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/agoldf6/benchmark/userbenchmark/dynamo/run.py", line 11, in <module>
    from torchbench import setup_torchbench_cwd, TorchBenchmarkRunner
  File "/data/agoldf6/benchmark/userbenchmark/dynamo/dynamobench/torchbench.py", line 16, in <module>
    from common import BenchmarkRunner, main
  File "/data/agoldf6/benchmark/userbenchmark/dynamo/dynamobench/common.py", line 58, in <module>
    from torch._dynamo.testing import (
ImportError: cannot import name 'reset_rng_state' from 'torch._dynamo.testing' (/data/agoldf6/mypy/lib/python3.8/site-packages/torch/_dynamo/testing.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_benchmark.py", line 29, in run
    benchmark = importlib.import_module(f"userbenchmark.{available_benchmarks[args.bm_name]}.run")
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/data/agoldf6/benchmark/userbenchmark/dynamo/run.py", line 15, in <module>
    from caffe2.benchmarks.dynamo.torchbench import setup_torchbench_cwd, TorchBenchmarkRunner
ModuleNotFoundError: No module named 'caffe2'

Hi @AidanGoldfarb , could you please try on the torch nightly release? Torchbench relies on the pytorch nightly build, and we do not test on the stable release.

The reason is because the symbol 'reset_rng_state' does not exist in the 'torch._dynamo.testing' package in the torch stable release, but it should exist in the nightly release.

Ah this was it. I thought I was running on nightly (from install command from readme), but I reinstalled and it worked, thanks!
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '95cc39e7d64f3597c9e723d6e8a567abef1744c9'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 5ae273f939d7a98b83d6717254d54c008933f6a8

Affected PyTorch commit: 95cc39e7d64f3597c9e723d6e8a567abef1744c9

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: +15.56352%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/7035265184

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/7137479166
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'ef73b5c9a88b47c93bcc88c90c5de8666f4eb22a'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 95cc39e7d64f3597c9e723d6e8a567abef1744c9

Affected PyTorch commit: ef73b5c9a88b47c93bcc88c90c5de8666f4eb22a

Affected Tests:
- test_eval[detectron2_fasterrcnn_r_50_dc5-cuda-eager]_latency: -13.53971%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/7049088709

cc @xuzhao9


https://github.com/pytorch/benchmark/actions/runs/7137484307
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.2.0.dev20231207+cu118'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.2.0.dev20231205+cu118

Base PyTorch commit: 7843df60e41f856edb148bbcbb5b9aee8292db74

Affected PyTorch version: 2.2.0.dev20231207+cu118

Affected PyTorch commit: 7ec145bfed9813599a0a9dcacbc7aa881626af79

Affected Tests:
- test_train[alexnet-cuda-eager]: +8.60516%
- test_eval[pytorch_stargan-cpu-jit]: -10.95130%
- test_train[pytorch_stargan-cpu-eager]: +7.30202%


cc @xuzhao9

Result json: 
```
{
  "start": "7843df60e41f856edb148bbcbb5b9aee8292db74",
  "end": "7ec145bfed9813599a0a9dcacbc7aa881626af79",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "3d8c1740691",
      "commit1_time": "2023-12-05 14:53:32 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2649913724977523
      },
      "commit2": "e06bff8bbea",
      "commit2_time": "2023-12-05 15:02:17 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2852564869564958
      }
    },
    {
      "commit1": "4c5fe668800",
      "commit1_time": "2023-12-05 18:28:40 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.271150191151537
      },
      "commit2": "a987ad3d897",
      "commit2_time": "2023-12-05 18:50:11 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.24330184361897408
      }
    },
    {
      "commit1": "cfa4370c079",
      "commit1_time": "2023-12-05 14:53:08 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2883812512853183
      },
      "commit2": "3d8c1740691",
      "commit2_time": "2023-12-05 14:53:32 +0000",
      "commit2_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2649913724977523
      }
    },
    {
      "commit1": "6c0a4ced530",
      "commit1_time": "2023-12-07 02:47:04 +0000",
      "commit1_digest": {
        "test_eval[pytorch_stargan-cpu-jit]": 0.2696367406984791
      },
      "commit2": "7ec145bfed9",
      "commit2_time": "2023-12-07 03:08:20 +0000",
      "commit2_digest": {
        "test_train[alexnet-cuda-eager]": 0.31006770200328904,
        "test_eval[pytorch_stargan-cpu-jit]": 0.231245014059823,
        "test_train[pytorch_stargan-cpu-eager]": 0.9320993624511175
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/7131728347
Repository: pytorch/benchmark
Issue: Use the pytorch/builder script to install CUDA packages in nightly docker.
Root cause: Architectural Constraint

Currently, we are building nightly dockers by copying the code from pytorch/builder. 

In order to make sure the CUDA and cudnn versions are consistent with the upstream, we should use pytorch/builder scripts to install CUDA packages in the docker build.

The source of the pytorch/builder code to install cuda: https://github.com/pytorch/builder/blob/main/common/install_cuda.sh

The source of docker script installing cuda: https://github.com/pytorch/benchmark/blob/main/docker/gcp-a100-runner-dind.dockerfile#L34

In the dockerfile, we should checkout the `pytorch/builder` git repo and use `install_cuda.sh` to install the CUDA libraries. 

We also want to keep using the latest CUDA version in `install_cuda.sh` by default, but that could be the next step.

Fixed by https://github.com/pytorch/benchmark/pull/2138
Repository: pytorch/benchmark
Issue: Remove  score.yml
Root cause: Architectural Constraint

The current torchbenchmark framework uses score.yml as the spec. Instead of maintaining a static file, we can generate the spec and thus eliminate the usage of score.yml. Below are the set of sub - tasks needed to accomplish this:

- [x] Update the Benchmark Model class , to have domain and task class attributes
- [x] Remove score.yml and generate the spec by reading the data from the Model class
- [x] Maintain an explicit list of enabled (required) models which are used for the score, rather than using whatever models are found in 'list_models'

`score.yml` is now replaced by score config yaml and model metadata yaml. Closing this issue.
Repository: pytorch/benchmark
Issue: how to get the memory test job
Root cause: Documentation Deficiency

https://arxiv.org/pdf/2304.14226.pdf  the paper says torchbench can do memory test， but I can‘t find any test jobs for memory test

https://github.com/pytorch/benchmark/actions

https://github.com/pytorch/pytorch/pull/85447   the paper says this issue was found by torchbench, can you please help me find the test job ?

@GuWei007 The test job was on CircleCI: https://app.circleci.com/pipelines/github/pytorch/benchmark/5273/workflows/8792c508-526d-4b42-af90-0e76ec818d64/jobs/5423. Now it has been replaced by GitHub Actions.

We could test the following memory metrics:
1. Watch for CUDA memory leak. https://github.com/pytorch/benchmark/blob/main/torchbenchmark/__init__.py#L512
2. Peak CPU and GPU memory usage: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/experiment/metrics.py#L52

https://github.com/pytorch/pytorch/pull/85447 was detected by the CUDA memory leak detector.

I want to understand the testing mechanism of memory metrics， can you show me the test job code,   maybe lilke this https://github.com/pytorch/benchmark/tree/main/.github/workflows ?  

As mentioned in the paper, I would like to understand which models were tested, what the thresholds were, how the testing cycles and frequencies were arranged, and how the test cases were designed. I work as an engineer at an AI chip manufacturer, and I'd like to learn from your testing mechanism. Thank you very much.

@GuWei007 
I can only briefly answer your questions here.
The test is done everyday on the latest PyTorch nightly build, the models tested are in https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models, the threshold is set here: https://github.com/pytorch/benchmark/blob/main/userbenchmark/torch-nightly/regression_detector.py#L4.
The test job code is at https://github.com/pytorch/benchmark/blob/main/userbenchmark/torch-nightly/run.py and https://github.com/pytorch/benchmark/blob/main/.github/workflows/v3-nightly.yml

You could also find example benchmark result here: https://github.com/pytorch/benchmark/actions/runs/7049088709

Thanks very much
Repository: pytorch/benchmark
Issue: Wrap up the distributed benchmark and get them running on conda_mast
Root cause: Unimplemented Feature Gap

We would like to introduce basic distributed benchmarking support on synthetic data.

The idea is to wrap up single-GPU model on DDP/FSDP, then get them running on conda_mast.
The initial OSS `distributed` userbenchmark could be used as the starting point:
https://github.com/pytorch/benchmark/tree/main/userbenchmark/distributed
Repository: pytorch/benchmark
Issue: Update score and plot to compute subscore for gpu/cpu/train/eval
Root cause: Unimplemented Feature Gap

The current torchbenchmark framework  reports one single benchmark score and doesn’t give a way to 
calculate the subscore for GPU/CPU/train/eval/eager/JIT. So we need a way to get the benchmark subscore by calculating the subscore for each of the following 8 configurations:

    * train-cuda-eager
    * train-cpu-eager
    * eval-cuda-eager
    * eval-cpu-eager
    * train-cuda-jit
    * train-cpu-jit
    * eval-cuda-jit
    * eval-cpu-jit

Here are the two subtasks required to accomplish this:

- [ ] Update the subscore depending on the configuration
- [ ] Generate 9 plots . One for each of the above configuration(8) and then one with the whole score
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.3.0.dev20231222+cu118'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.3.0.dev20231221+cu118

Base PyTorch commit: f2c1fb3ee40012ab17a0548dcbff509d6b1dbf39

Affected PyTorch version: 2.3.0.dev20231222+cu118

Affected PyTorch commit: b47aa69685aa6e561d3ba0edaff84b7833b98a82

Affected Tests:
- test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]: -7.84927%
- test_train[timm_resnest-cpu-eager]: +7.06478%


cc @xuzhao9

Result json: 
```
{
  "start": "f2c1fb3ee40012ab17a0548dcbff509d6b1dbf39",
  "end": "b47aa69685aa6e561d3ba0edaff84b7833b98a82",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "ff4aac109a9",
      "commit1_time": "2023-12-21 15:09:10 +0000",
      "commit1_digest": {
        "test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]": 55.269965320127085
      },
      "commit2": "a357a0f3151",
      "commit2_time": "2023-12-21 16:32:19 +0000",
      "commit2_digest": {
        "test_train[pytorch_CycleGAN_and_pix2pix-cuda-eager]": 51.21763134845533
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/7302266171


Speedup, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.3.0.dev20231223+cu118'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.3.0.dev20231222+cu118

Base PyTorch commit: b47aa69685aa6e561d3ba0edaff84b7833b98a82

Affected PyTorch version: 2.3.0.dev20231223+cu118

Affected PyTorch commit: 84b2a323594bc7c4b47d61223b3f6466fe054416

Affected Tests:
- test_eval[resnext50_32x4d-cpu-jit]: +12.15859%
- test_train[tts_angular-cpu-eager]: +7.07610%


cc @xuzhao9

Result json: 
```
{
  "start": "b47aa69685aa6e561d3ba0edaff84b7833b98a82",
  "end": "84b2a323594bc7c4b47d61223b3f6466fe054416",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "764b4cd44e5",
      "commit1_time": "2023-12-22 04:31:56 +0000",
      "commit1_digest": {
        "test_train[tts_angular-cpu-eager]": 0.1993940736632794
      },
      "commit2": "ad3c0b2c00b",
      "commit2_time": "2023-12-22 04:46:25 +0000",
      "commit2_digest": {
        "test_train[tts_angular-cpu-eager]": 0.22041498066391796
      }
    },
    {
      "commit1": "ad3c0b2c00b",
      "commit1_time": "2023-12-22 04:46:25 +0000",
      "commit1_digest": {
        "test_train[tts_angular-cpu-eager]": 0.22041498066391796
      },
      "commit2": "b342286646d",
      "commit2_time": "2023-12-22 05:22:39 +0000",
      "commit2_digest": {
        "test_train[tts_angular-cpu-eager]": 0.19716433212161064
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/7309496994


Likely noise, closed.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '1ed8e26791ba21e84011782d55f4904788976685'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: c70de4e370709bb1b2b541c57236217be7149718

Affected PyTorch commit: 1ed8e26791ba21e84011782d55f4904788976685

Affected Tests:
- test_eval[hf_Bert-cuda-eager]_latency: +10.74548%
- test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: +12.93765%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/7315585667

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/7413537570

Empty result, closed.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'b68f93b094cb6e922359e12adf4c9e1bc7ec650d'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 1ed8e26791ba21e84011782d55f4904788976685

Affected PyTorch commit: b68f93b094cb6e922359e12adf4c9e1bc7ec650d

Affected Tests:
- test_train[lennard_jones-cuda-eager]_latency: +16.12998%
- test_eval[pytorch_CycleGAN_and_pix2pix-cuda-eager]_latency: -11.28580%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/7322923310

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/7413595043

Stableness issue, closed.
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on 'eb8dedaa361f5c61ea08044e4ae513145b5ac250'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: b68f93b094cb6e922359e12adf4c9e1bc7ec650d

Affected PyTorch commit: eb8dedaa361f5c61ea08044e4ae513145b5ac250

Affected Tests:
- test_train[lennard_jones-cuda-eager]_latency: -14.91195%
- test_train[tts_angular-cuda-eager]_latency: +17.81982%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/7330920231

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/7423585722

```
{
  "target_repo": "pytorch",
  "start": "36dccc2aba61a2637aa5d42f38b6fd1fe10dcbdc",
  "end": "f6dfbffb3bb46ada6fe66b5da4f989f9d4d69b3c",
  "result": []
}
```
Repository: pytorch/benchmark
Issue: V3 Performance Signal Detected by TorchBench Userbenchmark "torch-nightly" on '50f7a03676c9c35f4d16168113febb42dabc4c51'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: eb8dedaa361f5c61ea08044e4ae513145b5ac250

Affected PyTorch commit: 50f7a03676c9c35f4d16168113febb42dabc4c51

Affected Tests:
- test_train[tts_angular-cuda-eager]_latency: -12.39332%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/7339935632

cc @xuzhao9


Bisection: https://github.com/pytorch/benchmark/actions/runs/7423592178

```
{
  "target_repo": "pytorch",
  "start": "f6dfbffb3bb46ada6fe66b5da4f989f9d4d69b3c",
  "end": "9c3ae37fc453505f5e437d1edadefdb278c2c39c",
  "result": []
}
```