zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: cpu pstate freq checking not working and setting not enabled
Root cause: Validation Gap

Currently this check is failing on an ec2 machine that appears to have pstate freq pinned correctly on cores 0-47, but it complains about core 88

machine_config.py#L277

Suggest changing set/get pstate freq to use the `parse_lscpu_cpu_core_list` helper in the same file, so it can check the 'online' flag for each cpu and only get/set the min/max/curr freq for the online cpus.

and, make sure to set/get cpu freq _after_ disablling hyperthread, so that online list is accurate.

also, would be good to set pstate cpu freq when running with --configure flag (unless for some reason this isn't stable, then maybe add a comment about why)

Address this issue in [PR#250](https://github.com/pytorch/benchmark/pull/250).
If we first disable HT, then pin the frequency, then the offline CPUs will remain in the old frequency. I don't think this will affect the online CPU cores' frequencies, but I don't find any driver documentation to support it either.

Closed by PR#250.
Repository: pytorch/benchmark
Issue: Failed CPU workflow
Root cause: External Dependency Breakage

Start hash: 4c73f163cd9c313b0ac2ea6ef71fe06d62bb4fba
End hash: 64141411e0de61b61857e216ae7a8766f4f5969b
Userbenchmark: test_bench
Userbenchmark options: -d cpu -m detectron2_fasterrcnn_r_101_c4 -t train

Bisection: https://github.com/pytorch/benchmark/actions/runs/11116416639
Repository: pytorch/benchmark
Issue: How is TorchBench applied to testing new versions of PyTorch?

Hello, may I ask what tasks will be used for end-to-end testing before the release of the new version of PyTorch?
Will the test focus on the consistency of metrics between the previous and subsequent versions, such as the loss of training tasks, iteration speed, etc

You can install nightly pytorch for testing, but the default benchmark only focus on sinlge iteration's computensive part(forward, backward, and optimizer phases). 

you may check this paper for more details. https://arxiv.org/abs/2304.14226 

> You can install nightly pytorch for testing, but the default benchmark only focus on sinlge iteration's computensive part(forward, backward, and optimizer phases).
> 
> you may check this paper for more details. https://arxiv.org/abs/2304.14226

非常感谢您的回复，还想问一下，当pytorch的训练功能有组件更新或者添加了新功能，是如何进行回归测试的？

> > You can install nightly pytorch for testing, but the default benchmark only focus on sinlge iteration's computensive part(forward, backward, and optimizer phases).
> > you may check this paper for more details. https://arxiv.org/abs/2304.14226
> 
> 非常感谢您的回复，还想问一下，当pytorch的训练功能有组件更新或者添加了新功能，是如何进行回归测试的？

If the new features are enabled by default and used in models included in torchbench, the ci will do the testing naturally. Since we use the nightly pytorch, if there are some updates on already applied testings, they will be tested automatically. 

> > > You can install nightly pytorch for testing, but the default benchmark only focus on sinlge iteration's computensive part(forward, backward, and optimizer phases).
> > > you may check this paper for more details. https://arxiv.org/abs/2304.14226
> > 
> > 
> > 非常感谢您的回复，还想问一下，当pytorch的训练功能有组件更新或者添加了新功能，是如何进行回归测试的？
> 
> If the new features are enabled by default and used in models included in torchbench, the ci will do the testing naturally. Since we use the nightly pytorch, if there are some updates on already applied testings, they will be tested automatically.

For other new features, the PR authors are usually required to write unit tests in pytorch repo.
Repository: pytorch/benchmark
Issue: Add more metrics
Root cause: Unimplemented Feature Gap

Since [ncu analyzer](https://github.com/pytorch/benchmark/blob/main/torchbenchmark/_components/ncu/analyzer.py) has been integrated, we can measure the actual memory_traffic and arithmetic_intensity now. We are able to add more metrics. not sure what kind of metrics we should add, but I think we should add more metrics.
Repository: pytorch/benchmark
Issue: Models missing 'requirements.txt' files
Root cause: Validation Gap

Models `lit_llama`, `lit_llama_generate` and `lit_llama_lora` are failing with `ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'` during the installation process as they are missing their requirements.txt files.

Full error with command:

```
$ python3 benchmark/install.py lit_llama_lora --canary -v
checking packages numpy, torch, torchvision, torchaudio are installed, generating constaints...OK
running setup for /home/user/benchmark/torchbenchmark/canary_models/lit_llama_lora...ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
Traceback (most recent call last):
  File "/home/user/benchmark/torchbenchmark/canary_models/lit_llama_lora/install.py", line 4, in <module>
    install_lit_llama()
  File "/home/user/benchmark/torchbenchmark/util/framework/lit_llama.py", line 57, in install_lit_llama
    pip_install_requirements()
  File "/home/user/benchmark/utils/python_utils.py", line 47, in pip_install_requirements
    subprocess.check_call(
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bin/python3', '-m', 'pip', 'install', '-r', 'requirements.txt', '-c', '/home/user/benchmark/build/constraints.txt']' returned non-zero exit status 1.
FAIL
Error for /home/user/benchmark/torchbenchmark/canary_models/lit_llama_lora:
---------------------------------------------------------------------------
None
---------------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/user/benchmark/install.py", line 120, in <module>
    raise RuntimeError("Failed to complete setup")
RuntimeError: Failed to complete setup
```
Repository: pytorch/benchmark
Issue: pyg_lib module installation not available
Root cause: Configuration Error

Canary models `sage`, `gcn` and `gat` attempt to install the `pyg_lib` module during their installation process but their requirements files do not include the link for the custom wheel which is needed as noted in the [pyg-lib module’s README](https://github.com/pyg-team/pyg-lib?tab=readme-ov-file#installation), or hard code to `torch-2.1.0+cpu`, which makes them fail for other versions.

This causes the following error:



```
$ python benchmark/install.py sage --canary -v
checking packages numpy, torch, torchvision, torchaudio are installed, generating constaints...OK
running setup for /home/user/benchmark/torchbenchmark/canary_models/sage...decompressing input tarball: /home/user/benchmark/torchbenchmark/data/Reddit_minimal.tar.gz...OK
Looking in indexes: ...api/pypi/pypi-virtual/simple, https://pypi.python.org/simple/ 
ERROR: Could not find a version that satisfies the requirement pyg_lib (from versions: none)
ERROR: No matching distribution found for pyg_lib
```
Repository: pytorch/benchmark
Issue: Is there a way to change batch size from calling test_bench.py
Root cause: Unimplemented Feature Gap

I'm running various benchmarks by calling 

`pytest test_bench.py testname `

but I was wondering if there was an argument/way to adjust the batchsize/specific model args when calling the test from this interface. I couldn't find anything in the --help/documentation. Ideally something simple like:

`pytest test_bench.py testname batch_size = 12 model_arg1 = xxx`

Thanks!

Actually we should deprecate `test_bench.py` and use [userbenchmark/dynamo](https://github.com/pytorch/benchmark/blob/main/userbenchmark/dynamo/run.py) as the suggested benchmarking script by default. cc @kit1980
Repository: pytorch/benchmark
Issue: install_diffusers issue with stable_diffusion_xl
Root cause: Configuration Error

The call to `install_diffusers()` in [benchmark/torchbenchmark/canary_models/stable_diffusion_xl/install.py at main · pytorch/benchmark](https://github.com/pytorch/benchmark/blob/main/torchbenchmark/canary_models/stable_diffusion_xl/install.py) should be made before error checking for the token, similarly to how it's done for other models such as [stable_diffusion_unet](https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/stable_diffusion_unet/install.py) and [stable_diffusion_text_encoder](https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/stable_diffusion_text_encoder/install.py). If not, then when it warns about the token and exits, it has not installed `diffusers` yet.

The following is an example of the error when we try to run the model after installation and the token is not set.


```
$ python3 run.py stable_diffusion_xl -t eval
Warning: The model stable_diffusion_xl cannot be found at core set.
Traceback (most recent call last):
  File "/home/user/benchmark/run.py", line 599, in <module>
    main()  # pragma: no cover
  File "/home/user/benchmark/run.py", line 511, in main
    model_flops = get_model_flops(config) if "model_flops" in metrics_needed else None
  File "/home/user/benchmark/torchbenchmark/util/experiment/metrics.py", line 129, in get_model_flops
    model = load_model(eager_model_config)
  File "/home/user/benchmark/torchbenchmark/util/experiment/instantiator.py", line 90, in load_model
    Model = load_canary_model_by_name(config.name)
  File "/home/user/benchmark/torchbenchmark/__init__.py", line 686, in load_canary_model_by_name
    module = importlib.import_module(f".canary_models.{model}", package=__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/benchmark/torchbenchmark/canary_models/stable_diffusion_xl/__init__.py", line 11, in <module>
    from diffusers import DiffusionPipeline
ModuleNotFoundError: No module named 'diffusers'
```
 

Note that this is a low priority issue and a bit of an edge case as it doesn’t affect performance if the token is set.
Repository: pytorch/benchmark
Issue: Benchmark reliability of torchbenchmarks
Root cause: Algorithmic Error

Recently I found that for the same model, the native benchmark code in torchbenchmarks does not give expected time, i.e. one is consistently slower than the other one, or one could be slower by up to 20%, I'm relying on torchao.utils.benchmark_model for now, please help take a look to see what might be the problem.

For details please see: https://github.com/pytorch/benchmark/pull/2519

This seems like this is an issue with model code, our expectation is that repo owners should own model code while our team owns infrastructure.

I think the time variability from run to run is expected when running on a devgpu.
TorchBench servers have some special settings to reduce the variability.

> I think the time variability from run to run is expected when running on a devgpu. TorchBench servers have some special settings to reduce the variability.

Oh so is this more of an infrastructure thing?

I feel this might be related to benchmarking code, since with the exact same setup, machine etc. `torchao.utils.benchmark_model` gives stable results
Repository: pytorch/benchmark
Issue: Mutating module attribute cache_k during export.
Root cause: Validation Gap

[rank0]:   File "/content/llama.py", line 353, in forward
[rank0]:     h = x + self.attention(self.attention_norm(x), start_pos, freqs_cis, mask)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/content/llama.py", line 269, in forward
[rank0]:     self.cache_k = self.cache_k.to(xq)

[rank0]: AssertionError: Mutating module attribute cache_k during export.


Facing this issue, inside the forward method of Attention class. Linking a similar solved issue for reference : [https://github.com/pytorch/benchmark/issues/1767](Issue1767).
Repository: pytorch/benchmark
Issue: How to get benchmark statistics?
Root cause: Unimplemented Feature Gap

I'm building a CI to test some models on certain types of devices. I want get benchmark statistics like which model cases failed? which tests were skipped and why? These statistics will be used to generate a table like this:

<table>
	<tr>
	    <th rowspan="2">Devices</th>
	    <th colspan="2">BERT_pytorch</th>
	    <th colspan="2">hf_GPT2</th>
	</tr>
	<tr>
	    <th>train</th>
	    <th>eval</th>
	    <th>train</th>
	    <th>eval</th>
	</tr>
	<tr>
	    <th>CPU</th>
	    <th>✅</th>
	    <th>✅</th>
	    <th>✅</th>
	    <th>✅</th>
	</tr>
	<tr>
	    <th>CUDA</th>
	    <th>✅</th>
	    <th>✅</th>
	    <th>✅</th>
	    <th>✅</th>
	</tr>
	<tr>
	    <th>Foo</th>
	    <th>❌ (failed)</th>
	    <th>✅</th>
	    <th>⚠️ (skipped)</th>
	    <th>✅</th>
	</tr>
</table>

So how can I get benchmark statistics? Is there a recommended way to do this? Can anyone give suggestions? Thanks so much!



I suggest to write your own userbenchmark for this use case: https://github.com/pytorch/benchmark/blob/main/userbenchmark/ADDING_USERBENCHMARKS.md

@xuzhao9 Thanks for your guidance! Actually, I'm building a CI to test some out-of-tree devices, such as Intel HPU, Ascend NPU, so I want to run some official model cases on them to prove the quality of these backends. I want to ask, is there any userbenchmarks that can be used directly? Or if I write my own benchmark, will it be merged?

ah I think `test_bench` might be a good choice.

But it only supports cpu and cuda. Will send a pull request to allow it to run on other devices.
Repository: pytorch/benchmark
Issue: pynvml 12.0.0 cause test failure
Root cause: External Dependency Breakage

Pynvml release a new version on Dec 2 which removes the pynvml module and __version__ attribute https://pypi.org/project/pynvml/.

That change makes this line fail. https://github.com/pytorch/benchmark/blob/main/torchbenchmark/_components/model_analyzer/dcgm/nvml_monitor.py#L61
Repository: pytorch/benchmark
Issue: Inefficient usage of torch.randn() in vision/model_factory.py
Root cause: Algorithmic Error

torch.randn() used [here](https://github.com/pytorch/benchmark/blob/675fb8f537d302a4fef3ed2a67349209e65046ac/torchbenchmark/util/framework/vision/model_factory.py#L36C13-L36C24) first allocates space for self.batch_size in the host memory and then copies it to the device memory, wasting host-device bandwidth. 

As self.batch_size contains random values, we can directly create self.batch_size on the device to bypass this non-sense host-device data copy. The proposed patch is as follows.

-torch.randn((self.batch_size, 3, 224, 224)).to(self.device)
+torch.randn((self.batch_size, 3, 224, 224), device=self.device)
Repository: pytorch/benchmark
Issue: Add SAM2 to TorchBench model set
Root cause: Unimplemented Feature Gap

We want to add SAM2 from https://github.com/facebookresearch/sam2?tab=readme-ov-file#download-checkpoints to TorchBench so that it can be included as part of PT2 OSS benchmark run. 

cc @kit1980  Let's take this opportunity to follow @xuzhao9 documentation in https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/ADDING_MODELS.md to add the model.

cc @cpuhrsch 

cc @josephgreer

In particular we'd need the [backbone models](https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/modeling/backbones/hieradet.py) with the [various configurations](https://github.com/facebookresearch/sam2?tab=readme-ov-file#download-) checkpoints loaded via

https://github.com/pytorch/ao/blob/522f5b854a278ee9e68e80bf8213e19c9da4e547/examples/sam2_amg_server/server.py#L472-L480

It's sufficient to use the resulting model for a particular task to exercise this. For example https://github.com/pytorch/ao/blob/522f5b854a278ee9e68e80bf8213e19c9da4e547/examples/sam2_amg_server/server.py#L294 on some example shape such as https://github.com/pytorch/ao/blob/522f5b854a278ee9e68e80bf8213e19c9da4e547/examples/sam2_amg_server/server.py#L63 after instantiating an instance via
https://github.com/pytorch/ao/blob/522f5b854a278ee9e68e80bf8213e19c9da4e547/examples/sam2_amg_server/server.py#L478-L480


Should we try to add it somehow directly to PyTorch instead? As CI pin is pretty much behind repo head.

[can be deleted][just to myself]
what is PT2 OSS benchmark run
Repository: pytorch/benchmark
Issue: [eager_fail_to_run ] mobilenet_v2_quantized_qat, eager run failed on dtype mismatch
Root cause: Interface Contract Mismatch

Running from pytorch:
```
python benchmarks/dynamo/torchbench.py  --only mobilenet_v2_quantized_qat --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --output out.csv
```
Error:
```
cuda train mobilenet_v2_quantized_qat
Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 453, in forward_and_backward_pass
    pred = mod(*cloned_inputs)
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 824, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 400, in __call__
    raise e
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 387, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1749, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1760, in _call_impl
    return forward_call(*args, **kwargs)
  File "<eval_with_key>.3", line 207, in forward
    activation_post_process_101 = self.activation_post_process_101(classifier_1);  classifier_1 = None
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1749, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1760, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/ao/quantization/fake_quantize.py", line 408, in forward
    return torch.fused_moving_avg_obs_fake_quant(
RuntimeError: expected scalar type Float but found Half

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
    ) = runner.load_model(
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
    self.validate_model(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
    raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed
```
Repository: pytorch/benchmark
Issue: [eager_fail_to_run] opacus_cifar10
Root cause: Interface Contract Mismatch

Running from pytorch main
```
python benchmarks/dynamo/torchbench.py --only opacus_cifar10 --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --output out.csv
```

```
cuda train opacus_cifar10
Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
    self.grad_scaler.scale(loss).backward()
  File "/data/users/ivankobzarev/a/pytorch/torch/_tensor.py", line 648, in backward
    torch.autograd.backward(
  File "/data/users/ivankobzarev/a/pytorch/torch/autograd/__init__.py", line 353, in backward
    _engine_run_backward(
  File "/data/users/ivankobzarev/a/pytorch/torch/autograd/graph.py", line 815, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 86, in __call__
    return self.hook(module, *args, **kwargs)
  File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook
    activations, backprops = self.rearrange_grad_samples(
  File "/data/users/ivankobzarev/a/opacus/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples
    raise ValueError(
ValueError: No activations detected for <class 'torch.nn.modules.linear.Linear'>, run forward after add_hooks(model)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
    ) = runner.load_model(
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
    self.validate_model(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
    raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed

eager_fail_to_run
```