zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Add support to AMD ROCm
Root cause: Unimplemented Feature Gap

PyTorch could be easily installed with AMD ROCm. But torchbench has some limitations on ROCm envs.
- [x] model installation
- [x] model execution
- [ ] GPU memory measurement

Models `detectron2_*` and `opacus_cifar10` have installation errors. If we only want to measure execution time, torchbench works well on other models. Verified on MI210 and ROCm 5.2.0. 

> Models `detectron2_*` and `opacus_cifar10` have installation errors. If we only want to measure execution time, torchbench works well on other models. Verified on MI210 and ROCm 5.2.0.

For those models' installation, need to make sure rocm has its llvm installed. 

https://github.com/pytorch/benchmark/pull/1752 adds a userbenchmark for testing on ROCm.

how about running test_bency.py on rocm+amd?


> how about running test_bency.py on rocm+amd?

Are you asking about the current support status of rocm?
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230215+cu117'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230214+cu117

Base PyTorch commit: d567df9f36094e1efa89d11599067773f45032c8

Affected PyTorch version: 2.0.0.dev20230215+cu117

Affected PyTorch commit: c10acb834d73f399589ce08a9fea1e4520a8c02c

Affected Tests:
- test_eval[hf_T5-cpu-eager]: +8.26482%
- test_eval[mnasnet1_0-cpu-jit]: -38.51033%
- test_train[timm_vision_transformer-cpu-eager]: +17.39668%
- test_eval[tts_angular-cpu-eager]: -9.96386%


cc @xuzhao9

Result json: 
```
{
  "start": "d567df9f36094e1efa89d11599067773f45032c8",
  "end": "c10acb834d73f399589ce08a9fea1e4520a8c02c",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "39511697d44",
      "commit1_time": "2023-02-14 08:13:15 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03657377125013487
      },
      "commit2": "b005ec62b93",
      "commit2_time": "2023-02-14 09:14:14 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04223469310575941
      }
    },
    {
      "commit1": "18d93cdc5db",
      "commit1_time": "2023-02-14 15:51:23 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.047131446946877986
      },
      "commit2": "3fc4bc115f0",
      "commit2_time": "2023-02-14 16:13:37 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03608971956218111
      }
    },
    {
      "commit1": "b7e1477e9b6",
      "commit1_time": "2023-02-14 17:58:51 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.0421930217416957
      },
      "commit2": "63bf7674fad",
      "commit2_time": "2023-02-14 18:42:06 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03596523766614625
      }
    },
    {
      "commit1": "3d5f4dcc4df",
      "commit1_time": "2023-02-15 03:27:48 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.03775070583547598
      },
      "commit2": "afadc3697a0",
      "commit2_time": "2023-02-15 04:09:59 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.048219396457584066
      }
    },
    {
      "commit1": "b005ec62b93",
      "commit1_time": "2023-02-14 09:14:14 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04223469310575941
      },
      "commit2": "1dbaa5c290b",
      "commit2_time": "2023-02-14 09:31:30 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.04732635628368536
      }
    },
    {
      "commit1": "33f13fc959a",
      "commit1_time": "2023-02-14 17:17:39 +0000",
      "commit1_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.036735671505864174
      },
      "commit2": "b7e1477e9b6",
      "commit2_time": "2023-02-14 17:58:51 +0000",
      "commit2_digest": {
        "test_eval[mnasnet1_0-cpu-jit]": 0.0421930217416957
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4186399756


Noise, closed.
Repository: pytorch/benchmark
Issue: Parallelize the installation script
Root cause: Unimplemented Feature Gap

Installing package dependencies should be independent, and we should install them in parallel.

I suggest making it configurable. Maybe no need to expose an API to the upper install.py but leave a variable or argument to disable it. Because I found an error named device busy caused by model installation in some clusters using NFS file system. I guess it is because the installation has too many write operations, and the cluster has a too harsh policy on write operations. After manually inserting a `time.sleep(1)` for each model, the installation becomes stable without any error. It happens several times when I use that cluster.

We could limit the parallelism with a configurable process pool, for example, by default we allow maximum 5 processes running for installation. If we set it to 1, the behaviour should be the same as the current approach.
Repository: pytorch/benchmark
Issue: Modify fastNLP to work with latest and older torchtext releases
Root cause: External Dependency Breakage

torchtext 0.8.0 release moved AG_NEWS from datasets.experimental. to datasets.  fastNLP should be able to run under new and old torchtext for the sake of running historical builds, so we can check the torchtext version in the fastNLP __init__.py and conditionally import the right name.

Cherry-picked [the fix](https://github.com/pytorch/benchmark/commit/ac7a531235509dea0cf7579d60aa62f7ba980172) to 0.1 branch
Repository: pytorch/benchmark
Issue: How to enable jit with nvfuser testing
Root cause: Configuration Error

I want to benchmark models in torchbenchmark with jit and nvfuser. I want to also dump the graph fused.
I tried following command, but nothing is printed.
PYTORCH_JIT_LOG_LEVEL=">>graph_fuser" python3 ../../run.py resnet50 -d cuda -m jit -t train

Sorry that I didn't realize that nvfuser is composable with jit. Can you try remove this line https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/extra_args.py#L144, then run the command with `-m jit --fuser fuser2` ?

Also cc @davidberard98 for visibility

I believe https://github.com/pytorch/benchmark/pull/1449 fixes this issue. @fxing-GitHub  If you still have problems, feel free to re-open this.
Repository: pytorch/benchmark
Issue: OOM for FP32 tests with PyTorch 20230102 cu118
Root cause: External Dependency Breakage

The following models have OOM for `--precision fp32` when running with PyTorch nightly  20230102 cuda 11.8. 

inference:
alexnet, densenet121, mnasnet1_0, mobilenet_v2, mobilenet_v3_large, resnet152, resnet18, resnet50, resnext50_32x4d, squeezenet1_1

training:
hf_T5_base 

What is the hardware platform?
We probably just aim to cover the default precision and don't aim to cover fp32 on A100 40GB across all models.

It might be interesting if you can prove the memory regression, for example, the memory usage cost increases because of PyTorch version or CUDA version change, then we have some place to start with.

The GPU I tesetd is A100 40GB.

Yes. the OOM happens on cuda 11.8 with cudnn 8.7.0, but cuda 11.6  with cudnn 8.3.2 works well. The experiments are a little bit tricky. I'll reproduce the results and show more details later.

What is the peak memory usage of CUDA 11.6+cudnn 8.3.2?
I can test on a CUDA 11.8+cudnn 8.7.0 on A100 80GB and check the peak GPU memory usage.
Repository: pytorch/benchmark
Issue: When the 4-card 4090 is running, the machine will be down in a few seconds
Root cause: Environment Incompatibility

When the 4-card 4090 is running, the machine will be down in a few seconds。
for the sample：python test.py -k "test_BERT_pytorch_train_cpu"
i use 
driver 525.89.02
cuda 12.1
pytorch 1.13

Thanks for your feedback. Could you describe more details? What do you mean the machine is down? 
According to your command, you test the train mode of BERT_pytorch on CPU. So it will not touch GPU. Have you overclocked your CPU or memory?  It is more like a hardware issue rather than a software issue.

Sounds like a hardware issue.

Personally I had a similar issue in a server before, the fix was to disable turbo boost on Intel p-state driver: https://askubuntu.com/questions/619875/disabling-intel-turbo-boost-in-ubuntu

So maybe you can try the same and see if it can fix your problem as well.

Closed as no response from the issue reporter.
Repository: pytorch/benchmark
Issue: Detectron2 models loading issues with cpu-only torch builds
Root cause: Configuration Error

The current model factory of `detectron2` will fetch the default cfg settings, which uses cuda as cfg.MODEL.DEVICE, and convert to whatever `device` user specified while loading the models. One caveat is that `detectrons` models cannot be loaded properly if the underlying torch is cpu only. Therefore making the `detectron2` models unusable for torchbenchmark with cpu-only torch.

Repro:
```
# the benchmarks script is from pytorch repo, make sure to run this on cpu-only torch
python benchmarks/dynamo/torchbench.py --only detectron2_fasterrcnn_r_101_c4 -dcpu --performance 
```

The error message while loading `detectron2` models:
```
Torch not compiled with CUDA enabled
```

cc: @xuzhao9
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230227+cu117'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230226+cu117

Base PyTorch commit: a88bfc60c75d22a047f03787aaa43130c6a8d6d9

Affected PyTorch version: 2.0.0.dev20230227+cu117

Affected PyTorch commit: 6624a73837bee5b59f01bdd4d2f0d5619bd3ab1f

Affected Tests:
- test_eval[attention_is_all_you_need_pytorch-cpu-jit]: +8.93107%
- test_train[timm_regnet-cpu-eager]: -7.57659%


cc @xuzhao9

Result json: 
```
{
  "start": "a88bfc60c75d22a047f03787aaa43130c6a8d6d9",
  "end": "6624a73837bee5b59f01bdd4d2f0d5619bd3ab1f",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "a88bfc60c75",
      "commit1_time": "2023-02-26 03:03:37 +0000",
      "commit1_digest": {
        "test_eval[attention_is_all_you_need_pytorch-cpu-jit]": 0.06848827373469249,
        "test_train[timm_regnet-cpu-eager]": 0.7061844899086281
      },
      "commit2": "fa7f17799a8",
      "commit2_time": "2023-02-26 06:18:31 +0000",
      "commit2_digest": {
        "test_eval[attention_is_all_you_need_pytorch-cpu-jit]": 0.0636323293321766
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4285014812


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230227+cu117'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230226+cu117

Base PyTorch commit: a88bfc60c75d22a047f03787aaa43130c6a8d6d9

Affected PyTorch version: 2.0.0.dev20230227+cu117

Affected PyTorch commit: 6624a73837bee5b59f01bdd4d2f0d5619bd3ab1f

Affected Tests:
- test_train[timm_nfnet-cpu-jit]: +16.36015%


cc @xuzhao9

Result json: 
```
{
  "start": "a88bfc60c75d22a047f03787aaa43130c6a8d6d9",
  "end": "6624a73837bee5b59f01bdd4d2f0d5619bd3ab1f",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "261b019a645",
      "commit1_time": "2023-02-26 20:21:40 +0000",
      "commit1_digest": {
        "test_train[timm_nfnet-cpu-jit]": 12.0108121272875
      },
      "commit2": "d3e1f165b36",
      "commit2_time": "2023-02-26 20:49:36 +0000",
      "commit2_digest": {
        "test_train[timm_nfnet-cpu-jit]": 11.045707068557386
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4295611285


Noise, closed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230301+cu117'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230227+cu117

Base PyTorch commit: 6624a73837bee5b59f01bdd4d2f0d5619bd3ab1f

Affected PyTorch version: 2.0.0.dev20230301+cu117

Affected PyTorch commit: 21b1134be690bdf338689ed0da287e772e8fd859

Affected Tests:
- test_train[timm_nfnet-cpu-jit]: -16.33697%
- test_eval[tts_angular-cpu-eager]: +7.75738%


cc @xuzhao9

Result json: 
```
{
  "start": "6624a73837bee5b59f01bdd4d2f0d5619bd3ab1f",
  "end": "21b1134be690bdf338689ed0da287e772e8fd859",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "6624a73837b",
      "commit1_time": "2023-02-27 02:50:58 +0000",
      "commit1_digest": {
        "test_train[timm_nfnet-cpu-jit]": 10.998455684341025,
        "test_eval[tts_angular-cpu-eager]": 0.23060131589882077
      },
      "commit2": "52651700296",
      "commit2_time": "2023-02-27 03:17:33 +0000",
      "commit2_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.20563803476979955
      }
    },
    {
      "commit1": "46385b3e482",
      "commit1_time": "2023-02-28 03:44:24 +0000",
      "commit1_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.21717428184347226
      },
      "commit2": "3944e7c3e8c",
      "commit2_time": "2023-02-28 03:44:40 +0000",
      "commit2_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.2361386345815845
      }
    },
    {
      "commit1": "b7c2a65139f",
      "commit1_time": "2023-02-28 05:24:31 +0000",
      "commit1_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.2275406932923943
      },
      "commit2": "9e16f1281ff",
      "commit2_time": "2023-02-28 06:49:46 +0000",
      "commit2_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.25262375610182064
      }
    },
    {
      "commit1": "4e926db1f82",
      "commit1_time": "2023-02-27 23:22:36 +0000",
      "commit1_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.23508838899433612
      },
      "commit2": "eff5ae87462",
      "commit2_time": "2023-02-28 00:02:22 +0000",
      "commit2_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.2532051463844255
      }
    },
    {
      "commit1": "38c32e19c8d",
      "commit1_time": "2023-02-28 03:43:57 +0000",
      "commit1_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.24040269735269248
      },
      "commit2": "46385b3e482",
      "commit2_time": "2023-02-28 03:44:24 +0000",
      "commit2_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.21717428184347226
      }
    },
    {
      "commit1": "9e16f1281ff",
      "commit1_time": "2023-02-28 06:49:46 +0000",
      "commit1_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.25262375610182064
      },
      "commit2": "05943712a44",
      "commit2_time": "2023-02-28 07:14:32 +0000",
      "commit2_digest": {
        "test_eval[tts_angular-cpu-eager]": 0.21040280702291056
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4306177550


Noise, closed.
Repository: pytorch/benchmark
Issue: Add LLAMA model to benchmarks
Root cause: Unimplemented Feature Gap

https://github.com/facebookresearch/llama

Looking at the repo it looks like it needs to download both the tokenizer and weights from a signed URL which I'd get after filling out an application https://github.com/facebookresearch/llama/blob/main/download.sh

The application is more geared towards researchers so I might be able to work fine with random weights but not sure how the random tokenizer will work, let's see
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.1.0.dev20230303+cu117'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230301+cu117

Base PyTorch commit: 21b1134be690bdf338689ed0da287e772e8fd859

Affected PyTorch version: 2.1.0.dev20230303+cu117

Affected PyTorch commit: 53c9866ffa8baa7b7070901ce74704e0771bb05b

Affected Tests:
- test_eval[pytorch_stargan-cpu-eager]: +9.95659%
- test_train[soft_actor_critic-cuda-eager]: -8.95655%
- test_train[timm_nfnet-cpu-jit]: -8.03770%
- test_eval[tts_angular-cpu-eager]: +7.65274%


cc @xuzhao9

Result json: 
```
{
  "start": "21b1134be690bdf338689ed0da287e772e8fd859",
  "end": "53c9866ffa8baa7b7070901ce74704e0771bb05b",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "2bcf863fada",
      "commit1_time": "2023-03-02 04:15:33 +0000",
      "commit1_digest": {
        "test_train[soft_actor_critic-cuda-eager]": 0.014656348423897356
      },
      "commit2": "75cb99e5497",
      "commit2_time": "2023-03-02 04:15:33 +0000",
      "commit2_digest": {
        "test_train[soft_actor_critic-cuda-eager]": 0.013118841860733636
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4325749501


cc @janeyx99 Looks like this is another speedup from optimizer? Or do you think this is noise?

The optimizer of tts_angular model should be `RAdam`: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/tts_angular/angular_tts_main.py#L29

This seems likely related to my foreach as default change on 3/1, though it is interesting the bisector is bisecting 2 days later.  https://github.com/pytorch/pytorch/pull/95820

Yes, I guess this is because the workflow on Mar 02 failed due to timeout.
Repository: pytorch/benchmark
Issue: aarch64 runs for torchbench benchmarks
Root cause: Unimplemented Feature Gap

Hi, I would like to check if there are any nightly runs for these benchmarks.  If yes, I would like to enable few environment variables for aarch64 platforms. Please let me know where to contribute. Thank you!

Yes, we have nightly runs on an Amazon g4dn.metal instance, on CPU and NVIDIA T4 GPU: https://github.com/pytorch/benchmark/actions/workflows/v2-nightly.yml.

We encourage people can develop their own userbenchmarks to create benchmark scripts for their own purpose: https://github.com/pytorch/benchmark/blob/main/userbenchmark/ADDING_USERBENCHMARKS.md

My question is, currently, we don't have any aarch64 GitHub runner in our infra. I am not sure if you are planning to add new runners on the AWS side, or do you want to reuse the runners in the pytorch/pytorch infra to run the nightly workflow?

thanks for the prompt response, @xuzhao9 . I wanted to understand if they are already running. Probably good option would be just use pytorch/pytorch infra as an extended CI.

@snadampal 
So my answer is: we only have x86_64 nightly benchmarks but no infra or benchmark on the aarch64 platform. Currently, we don't have plans to build aarch64 nightly runs, but I can support you if you would like to do that under the userbenchmark framework.

cc @malfet @seemethere , do you think we can use the aarch64 runners in pytorch/pytorch to build the aarch64 workflows?
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.1.0.dev20230311+cu117'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.1.0.dev20230310+cu117

Base PyTorch commit: 065de4301200117c65607b213322cbde800b50f8

Affected PyTorch version: 2.1.0.dev20230311+cu117

Affected PyTorch commit: f3fc4d035d8fc1ed560ade0c8b8eff6be1c95972

Affected Tests:
- test_train[hf_Albert-cpu-eager]: +13.53803%
- test_train[timm_nfnet-cpu-eager]: +7.65575%
- test_train[timm_regnet-cpu-eager]: +8.06683%
- test_train[timm_resnest-cpu-eager]: +9.58247%
- test_train[tts_angular-cpu-eager]: -8.24563%
- test_train[vgg16-cpu-eager]: -8.34980%


cc @xuzhao9

Result json: 
```
{
  "start": "065de4301200117c65607b213322cbde800b50f8",
  "end": "f3fc4d035d8fc1ed560ade0c8b8eff6be1c95972",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "987eade3f38",
      "commit1_time": "2023-03-10 06:46:30 +0000",
      "commit1_digest": {
        "test_train[hf_Albert-cpu-eager]": 7.75660874734167,
        "test_train[timm_regnet-cpu-eager]": 0.7053629019996152
      },
      "commit2": "1b59c3feb55",
      "commit2_time": "2023-03-10 10:40:01 +0000",
      "commit2_digest": {
        "test_train[hf_Albert-cpu-eager]": 8.79867735193111
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4393357911
Repository: pytorch/benchmark
Issue: Add detectron2 models
Root cause: Unimplemented Feature Gap

Currently the benchmark includes maskrcnn-benchmark models: https://github.com/pytorch/benchmark/tree/master/torchbenchmark/models/maskrcnn_benchmark

However as the link above says, maskrcnn-benchmark is now deprecated in favor of detectron2. Detectron2 models are used widely in facebook production and research so we should add or switch to benchmark detectron2 models instead. The two projects are similar in their core models, so the switch shouldn't be hard.

I can follow up with a few most important models in detectron2 that should be benchmarked.

https://github.com/pytorch/benchmark/blob/master/torchbenchmark/models/ADDING_MODELS.md

Two most representative models:

* https://github.com/facebookresearch/detectron2/blob/main/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.py
*https://github.com/facebookresearch/detectron2/blob/main/configs/COCO-Detection/retinanet_R_50_FPN_1x.py

To run training it should use 2 images per GPU. The configs are made for 8-GPU training and `SOLVER.IMS_PER_BATCH` should be changed to 2 for single-GPU case.
Inference benchmark should use the pre-trained models. Random weights don't do the same computation. The [model zoo API](https://detectron2.readthedocs.io/modules/model_zoo.html) can help get them easily


https://github.com/facebookresearch/detectron2/blob/master/tools/benchmark.py is an example code that shows how to benchmark detectron2 training with less variance