zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: [eager_fail_to_run] resnet50_quantized_qat
Root cause: Interface Contract Mismatch

Running from pytorch main
```
python benchmarks/dynamo/torchbench.py --only resnet50_quantized_qat --accuracy --no-translation-validation --training --amp --backend inductor --disable-cudagraphs --device cuda --output out.csv
```
```
cuda train resnet50_quantized_qat
Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 453, in forward_and_backward_pass
    pred = mod(*cloned_inputs)
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 824, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 400, in __call__
    raise e
  File "/data/users/ivankobzarev/a/pytorch/torch/fx/graph_module.py", line 387, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1749, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1760, in _call_impl
    return forward_call(*args, **kwargs)
  File "<eval_with_key>.3", line 167, in forward
    activation_post_process_73 = self.activation_post_process_73(fc);  fc = None
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1749, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/nn/modules/module.py", line 1760, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/users/ivankobzarev/a/pytorch/torch/ao/quantization/fake_quantize.py", line 408, in forward
    return torch.fused_moving_avg_obs_fake_quant(
RuntimeError: expected scalar type Float but found Half

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
    ) = runner.load_model(
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
    self.validate_model(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
    raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed

eager_fail_to_run
```
Repository: pytorch/benchmark
Issue: [eager_fail_to_run] cuda train sam
Root cause: Validation Gap

```
python benchmarks/dynamo/torchbench.py --only sam --accuracy --no-translation-validation --training --amp --backend inductor --disable-cudagraphs --device cuda
```
```
cuda train sam
Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2744, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
    self.grad_scaler.scale(loss).backward()
  File "/home/ivankobzarev/local/a/pytorch-env/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/home/ivankobzarev/local/a/pytorch-env/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/home/ivankobzarev/local/a/pytorch-env/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 4857, in run
    ) = runner.load_model(
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/torchbench.py", line 372, in load_model
    self.validate_model(model, example_inputs)
  File "/data/users/ivankobzarev/a/pytorch/benchmarks/dynamo/common.py", line 2746, in validate_model
    raise RuntimeError("Eager run failed") from e
RuntimeError: Eager run failed

eager_fail_to_run
```
Repository: pytorch/benchmark
Issue: [release-test] A100 3803mnist_hogwild-latency increase 10x on linux.aws.a100 vs [a100-runner]
Root cause: Environment Incompatibility

Looking at the result of the run for 2.6.0 vs 2.5.1
https://github.com/pytorch/benchmark/actions/runs/12878326305/job/35904096937
Benchmark,pytorch-2.5.1-cuda-12.4,pytorch-2.6.0-cuda-12.4
mnist-cpu_memory,1118.67,1146.76
mnist-gpu_memory,0.0,0.0
mnist-latency,42.46,40.00
mnist_hogwild-cpu_memory,556.57,601.289
mnist_hogwild-gpu_memory,0.0,0.0
mnist_hogwild-latency,671.28,586.02
wlm_cpu_lstm-cpu_memory,885.141,907.066
wlm_cpu_lstm-gpu_memory,0.0,0.0
wlm_cpu_lstm-latency,1266.83,1079.37
wlm_cpu_trans-cpu_memory,852.113,899.531
wlm_cpu_trans-gpu_memory,0.0,0.0
wlm_cpu_trans-latency,1081.98,1078.99
wlm_gpu_lstm-cpu_memory,995.402,954.391
wlm_gpu_lstm-gpu_memory,0.0,0.0
wlm_gpu_lstm-latency,54.78,52.76
wlm_gpu_trans-cpu_memory,1007.86,993.949
wlm_gpu_trans-gpu_memory,0.0,0.0
wlm_gpu_trans-latency,56.41,55.54

Run 2.4.1 vs 2.5.0 (mnist_hogwild only):
https://github.com/pytorch/benchmark/actions/runs/12895573722
Benchmark,pytorch-2.4.1-cuda-12.4,pytorch-2.5.0-cuda-12.4
mnist_hogwild-cpu_memory,561.797,556.758
mnist_hogwild-gpu_memory,0.0,0.0
mnist_hogwild-latency,613.91,610.53


Run 2.5.1 vs 2.6.0 (mnist_hogwild only):
https://github.com/pytorch/benchmark/actions/runs/12894636482
Benchmark,pytorch-2.5.1-cuda-12.4,pytorch-2.6.0-cuda-12.4
mnist_hogwild-cpu_memory,561.73,579.324
mnist_hogwild-gpu_memory,0.0,0.0
mnist_hogwild-latency,592.67,599.23


Comparing mnist_hogwild-latency number with run on A100 hosted on GCP I see 10x difference:
Run 2.4.1 vs 2.5.0:
3803mnist_hogwild-latency ,61.42,62.19

mnist_hogwild is a CPU-only benchmark.

CPU hardware spec of GCP A100 ("a2-highgpu-1g"):

Intel(R) Xeon(R) Platinum 8481C CPU 12 threads @2.70 GHz  with 85 GB DRAM

Yes, those AWS instances are way weaker in terms of CPU compared to the GCP A100. They have only 10 or 11 cores available.

Maybe we should consider moving away those simple CPU benchmark to cheaper instances. Given the very limited pool we have at hands and constrained budget for A100 instances.
Repository: pytorch/benchmark
Issue: [release-test] GPU memory numbers are always 0 on A100 runner
Root cause: Environment Incompatibility

The command `nvidia-smi pmon -s m -c 1 -o T` does not give GPU memory numbers correctly.

https://github.com/pytorch/benchmark/blob/main/userbenchmark/release-test/monitor_proc.sh#L24

Error workflow:
https://github.com/pytorch/benchmark/actions/runs/12878326305/job/35904096937

Running on the GCP H100 machine or devGPU, the GPU memory data is generated successfully:

```
$ bash ~/benchmark/userbenchmark/release-test/monitor_proc.sh python main.py --epochs 4
Running  python main.py --epochs 4 > "/tmp/tmp.fVD4eQHgYE_monitor_proc" 2>&1 &
Watching PID: 2582
Searching for spawned processes with
Tail log file with: ps --forest -o pid --ppid 2582 --pid 2582 | awk 'NR>1 {print}'
    tail -f /tmp/tmp.fVD4eQHgYE_monitor_proc

Max GPU Mem.   Max RSS Mem.   Max PSS Mem.
904            1149.47        1438.21
1438           1209.12        1438.21
1438           1209.15        1438.21
1438           1209.59        1845.58
```

Debug reproduce: on the CI machine, the output is: https://github.com/pytorch/benchmark/actions/runs/12982872402/job/36203285897

```
#Time         gpu         pid   type     fb   ccpm    command 
#HH:MM:SS     Idx           #    C/G     MB     MB    name 
 05:53:45       0          -     -      -      -    -              

#Time         gpu         pid   type     fb   ccpm    command 
#HH:MM:SS     Idx           #    C/G     MB     MB    name 
 05:53:47       0        698     C      0      0    python  
```

cc @atalman @jeanschmidt 

I can't pinpoint yet the reason you get those specific responses. Having said that, the command:

```
$ nvidia-smi dmon -s m -c 1 -o T -i 0
#Time         gpu     fb   bar1   ccpm
#HH:MM:SS     Idx     MB     MB     MB
 15:54:14       0    429      4      0
```

works, so maybe you can rely on it?
Repository: pytorch/benchmark
Issue: dlrm_s_pytorch.py profile api arguments mismatch
Root cause: Interface Contract Mismatch

```
$ python -m torchbenchmark.models.dlrm.dlrm_s_pytorch --mini-batch-size=2 --data-size=6 --debug-mode
```
```
/opt/py3.10/lib/python3.10/runpy.py:126: RuntimeWarning: 'torchbenchmark.models.dlrm.dlrm_s_pytorch' found in sys.modules after import of package 'torchbenchmark.models.dlrm', but prior to execution of 'torchbenchmark.models.dlrm.dlrm_s_pytorch'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
Using CPU...
model arch:
mlp top arch 3 layers, with input to output dimensions:
[8 4 2 1]
# of interactions
8
mlp bot arch 2 layers, with input to output dimensions:
[4 3 2]
# of features (sparse and dense)
4
dense feature size
4
sparse feature size
2
# of embeddings (= # of sparse features) 3, with dimensions 2x:
[4 3 2]
data (inputs and targets):
mini-batch: 0
[[0.69647 0.28614 0.22685 0.55131]
 [0.71947 0.42311 0.98076 0.68483]]
[[1, 2], [1, 1], [1, 1]]
[[1, 0, 1], [0, 1], [1, 0]]
[[0.36179]
 [0.22826]]
mini-batch: 1
[[0.29371 0.63098 0.0921  0.4337 ]
 [0.43086 0.49369 0.42583 0.31226]]
[[1, 2], [1, 2], [1, 1]]
[[3, 0, 2], [1, 1, 2], [1, 1]]
[[0.60306]
 [0.54507]]
mini-batch: 2
[[0.34276 0.30412 0.41702 0.6813 ]
 [0.87546 0.51042 0.66931 0.58594]]
[[2, 1], [1, 2], [1, 1]]
[[2, 3, 2], [0, 0, 2], [1, 1]]
[[0.55679]
 [0.15896]]
initial parameters (weights and bias):
[[-0.34693  0.19553]
 [-0.18123  0.19197]
 [ 0.05438 -0.11105]
 [ 0.42513  0.34167]]
[[-0.16466 -0.52702]
 [-0.22543 -0.11757]
 [ 0.23667  0.57199]]
[[-0.20377  0.3713 ]
 [ 0.13177  0.27111]]
[[-0.16825 -0.58044 -0.39152 -0.64812]
 [ 1.11561  0.0879   0.61481 -0.67743]
 [ 0.09677  0.62959 -0.17907  0.55115]]
[0. 0. 0.]
[[-0.68594 -0.86234  0.23995]
 [-0.23981  0.40607 -1.25093]]
[0. 0.]
[[ 0.29078  1.06075 -0.01005  0.01394  0.0733  -0.76015  0.17397 -0.65541]
 [-0.1746   0.5074  -0.30015  0.20463  0.41345  0.1138  -0.55969 -0.13573]
 [ 0.79993 -0.82672 -0.11259 -0.2254   0.04929  0.30546  0.65675 -0.11032]
 [ 0.33164  0.20402  0.19365 -0.23022 -0.40715 -0.44909 -0.30881  0.13133]]
[0. 0. 0. 0.]
[[ 0.43933  0.18675 -0.31694  1.04268]
 [ 0.87692 -0.20438 -0.47541  0.07518]]
[0. 0.]
[[1.03474 0.2717 ]]
[0.]
time/loss/accuracy (if enabled):
Traceback (most recent call last):
  File "/opt/py3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/py3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/torch/benchmark/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 970, in <module>
    with torch.autograd.profiler.profile(args.enable_profiling, use_gpu) as prof:
TypeError: profile.__init__() takes from 1 to 2 positional arguments but 3 were given
```

```
with torch.autograd.profiler.profile(args.enable_profiling, use_gpu) as prof:
```
should be changed to
```
with torch.autograd.profiler.profile(args.enable_profiling, use_cuda=use_gpu) as prof:
```
https://pytorch.org/docs/stable/autograd.html#profiler
Repository: pytorch/benchmark
Issue: v0 torchbench (and master) broken by BC change to torchtext
Root cause: External Dependency Breakage

fix both v0 and master branches so that both old and new torchtext works.  
 - try to import the old way, if it fails, try to import the new way

https://github.com/pytorch/text/commits/master

see this error 
ImportError: cannot import name 'TranslationDataset' from 'torchtext.datasets' (/home/ec2-user/anaconda3/envs/gh578224670/lib/python3.7/site-packages/torchtext/datasets/__init__.py)

from nightly CI
https://github.com/pytorch/benchmark/runs/1927202422?check_suite_focus=true

but look for any related imports of the other renamed packages, since the CI stopped at first failure.

Now fixed in both master and v0. PR's for the fixes - #269, #266.  Closing this bug.
Repository: pytorch/benchmark
Issue: Missing `test_bench` configuration support
Root cause: Configuration Error

Seems `--config` is defined but not used actually in the `test_bench` userbenchmark. I would like to add support for it if this change makes sense.

https://github.com/pytorch/benchmark/blob/43851fb1208fad5708af8488789754b90fe94ec6/userbenchmark/test_bench/run.py#L307-L309

cc: @xuzhao9 


Actually, I'm working on a PyTorch out-of-tree test infra project, which is using `torchbenchmark` to test out-of-tree devices and can be found at [here][1].
And we want to let each device vendor define their own configuration, might like this:

```yaml
devices:
  - "foo"
models:
  - model: BERT_pytorch
    batch_size: 1

  - model: yolov3
    skip: true
extra_args:
  - "--accuracy"
```

[1]: https://github.com/cosdt/pytorch-integration-tests
[2]: https://github.com/pytorch/pytorch/pull/142114

<img width="935" alt="Image" src="https://github.com/user-attachments/assets/2cce2ce5-b5f3-4159-a10b-f1e9f7367be2" />
Repository: pytorch/benchmark
Issue: AttributeError: 'ModelTask' object has no attribute 'replace_invoke'
Root cause: Unimplemented Feature Gap

When I run `python3 run_benchmark.py test-user-invoke --model alexnet`, I get an AttributeError error:
```bash
Running TorchBenchModelConfig(name='alexnet', test='eval', device='cuda', batch_size=1, extra_args=[], extra_env=None, output_dir=None, skip=False) ...Traceback (most recent call last):
  File "/home/chensichao/benchmark-mm/run_benchmark.py", line 48, in <module>
    run()
  File "/home/chensichao/benchmark-mm/run_benchmark.py", line 41, in run
    benchmark.run(bm_args)
  File "/home/chensichao/benchmark-mm/userbenchmark/test-user-invoke/run.py", line 94, in run
    result = run_config(config)
  File "/home/chensichao/benchmark-mm/userbenchmark/test-user-invoke/run.py", line 72, in run_config
    inject_model_invoke(model, user_defined_invoke)
  File "/home/chensichao/benchmark-mm/torchbenchmark/util/experiment/instantiator.py", line 45, in inject_model_invoke
    model_task.replace_invoke(inject_function.__module__, inject_function.__name__)
AttributeError: 'ModelTask' object has no attribute 'replace_invoke'
```
Then I search the source code, and I find that `replace_invoke` is not implemented in the code.
Repository: pytorch/benchmark
Issue: [Bug] ModuleNotFoundError: No module named 'torch._C._nativert' with recent PyTorch nightly
Root cause: External Dependency Breakage

I am encountering a ModuleNotFoundError when trying to run the dynamobench/torchbench.py benchmark with a recent PyTorch nightly build. It seems the internal API torch._C._nativert has been removed or changed in the latest PyTorch versions, causing the import in common.py to fail.

* I also meet same error in `pytorch 2.8.0`. `ModuleNotFoundError: No module named 'torch._C._nativert'; 'torch._C' is not a package`
Repository: pytorch/benchmark
Issue: ModuleNotFoundError: No module named 'torchbenchmark.util.framework.fb'
Root cause: Environment Incompatibility

when running `python benchmarks/dynamo/torchbench.py   --accuracy --inference  --bfloat16 --device cpu    --inductor      --only Background_Matting`，the following error occurs:

<img width="1237" height="89" alt="Image" src="https://github.com/user-attachments/assets/00826d6d-cfd5-48b8-a947-6c5fa7533d6d" />

If I run `python install.py Background_Matting` to get model data, the following error occurs:

<img width="859" height="498" alt="Image" src="https://github.com/user-attachments/assets/2ec63e54-0635-4065-a8c3-f7c7d5208cb1" />

How can I successfully obtain the model data?


my torch version is 2.6.0
Repository: pytorch/benchmark
Issue: (auto)Bump diffusers version pin
Root cause: External Dependency Breakage

Right now it's ~1 year behind:
https://github.com/pytorch/benchmark/blob/f0b2a09591de5cf276bbfa7ce06a3d35f508da84/torchbenchmark/util/framework/diffusers/requirements.txt#L1

Bumping it would induce breakages like example inputs dimensions in https://github.com/pytorch/benchmark/pull/2654, and maybe more for the old stable diffusion models.

Maybe we can auto update this like https://github.com/pytorch/pytorch/pull/160635?

cc @BoyuanFeng, @huydhn 

This makes sense.  I would be able to pick this up next week if there is no owner by that time.
Repository: pytorch/benchmark
Issue: Several models do not honor the device flag
Root cause: Validation Gap

These models silently use CPU in 'cuda' mode.  
![image](https://user-images.githubusercontent.com/4984825/108788830-be342a80-752d-11eb-901c-b670de16beb6.png)


@Krovatkin filed to track

Fixed by Nick in https://github.com/pytorch/benchmark/pull/274
Repository: pytorch/benchmark
Issue: Address dynamic shapes
Root cause: Unimplemented Feature Gap

come up with a plan for benchmarking dynamic shapes.

Two issues we'll have to address to make this work
- identify a set of models we can use
- come up with a way to run enough steps that we are approaching the real dynamic-shape distribution in the full dataset, but without training to completion

@eellison Any thoughts on how to approach this?

Added by https://github.com/pytorch/benchmark/pull/1914
Repository: pytorch/benchmark
Issue: Vision models use N=32 for inference
Root cause: Documentation Deficiency

The example inputs for AlexNet and DenseNet (and maybe some others that I haven't checked yet) are size (32, 3, 224, 224).  I doubt this is representative for inference; N=1 seems more likely.

I am told that batched inference is not an uncommon usecase.  I rescind my bug report :-p
Repository: pytorch/benchmark
Issue: Add Speech Transformer
Root cause: External Dependency Breakage

https://github.com/kaituoxu/Speech-Transformer transformer-variants are popping up throughout machine learning and it would be nice to capture more of them. This is an audio-variant.

Current blocked by installing the kaldi dependency. Kaldi requires python 2.7: https://github.com/kaldi-asr/kaldi/blob/5de039aeaacb90241d0362f113eeb88c5c46943d/tools/extras/check_dependencies.sh#L96

Added in: https://github.com/pytorch/benchmark/pull/374

Fixed
Repository: pytorch/benchmark
Issue: [JIT not enabled] MOCO
Root cause: Unimplemented Feature Gap

See https://github.com/pytorch/benchmark/blob/master/torchbenchmark/models/moco/__init__.py#L73

@wconstab do you happen to know by any chance why jit mode for MOCO was disabled?

No, I don't know. if git blame shows it as part of my initial copy to this repo, you could check blame on the old repo which was pytorch/hub.  There could be some insight in the commit message.
Repository: pytorch/benchmark
Issue: Add Freezing
Root cause: Validation Gap


@Krovatkin Updates on this issue:
1. We have enabled freezing (i.e. `optimize_for_inference()`) by default in https://github.com/pytorch/benchmark/pull/761
2. 24 models out of 50 models don't support freeze ( or they don't jit but works in eager)
3. 2 models (BERT_pytorch and timm_efficientnet) don't work because their output are incorrect (cosine similarity too low comparing with eager)
Command to reproduce:
```
$ python run.py BERT_pytorch -d cuda -t eval -m jit
Running eval method from BERT_pytorch on cuda in jit mode.
Building Vocab
GPU Time:             14.134 milliseconds
CPU Dispatch Time:    12.545 milliseconds
CPU Total Wall Time:  14.159 milliseconds
Correctness:            0.384378939867020
```
```
$ python run.py timm_efficientnet -d cuda -t eval -m jit
Running eval method from timm_efficientnet on cuda in jit mode.
GPU Time:             13.549 milliseconds
CPU Dispatch Time:     5.530 milliseconds
CPU Total Wall Time:  13.576 milliseconds
Correctness:            0.009885539300740
```

Resolved by https://github.com/pytorch/benchmark/pull/761
Repository: pytorch/benchmark
Issue: FastNLP model misconfigured
Root cause: Configuration Error

This benchmark is only running the classifier portion of text-classification, the fine tuning task on BERT or so the code suggests. Therefore, the entire network consists of a Linear, Dropout, and LogSoftmax. It is also a very small portion of work as it only consumes one Block on the entire GPU and 1 step iteration runs in 1ms.

https://dev-discuss.pytorch.org/t/pytorch-benchmarks-issues-with-general-usability-and-issues-with-individual-benchmarks/169


I think this load is fatally unsuitable to run on gpu. Cost of gpu transaction is much higher than the work on each element.

The fastnlp codebase has code for several other models, perhaps we could instead adopt a more complex one?
Repository: pytorch/benchmark
Issue: BERT model misconfigured for training
Root cause: Configuration Error

Batch Size and Sequence Length: The batch size (number of sequences) of 1 seems inappropriate for training along with a sequence length of 20. These seem like inference parameters as the benchmark is overloaded by allreduce communication, the optimizer, and CPU latency. I would suggest a minimum batch size of 32 with a sequence length of 128 (4096 total tokens). Bigger sequence lengths also expose improvements to multihead attention.

https://dev-discuss.pytorch.org/t/pytorch-benchmarks-issues-with-general-usability-and-issues-with-individual-benchmarks/169


Fixed with

https://github.com/pytorch/benchmark/pull/434
Repository: pytorch/benchmark
Issue: Add NFNet Model

People observe that NFNet model performance regresses for pytorch 1.8.0: https://twitter.com/wightmanr/status/1368360389704966144.

NFNet(Normalization Free Nets) is a resnet improvement on image recognization: https://arxiv.org/abs/2102.06171.

Closed this because timm models are taken care of in https://github.com/pytorch/benchmark/issues/342.
Repository: pytorch/benchmark
Issue: Add a bunch of timm models
Root cause: Unimplemented Feature Gap

https://github.com/rwightman/pytorch-image-models

pip install a specific version
then pick one or more models from the library based on popularity and interest, and wrap them in __init__ 

Created https://github.com/pytorch/benchmark/pull/350 to add NFNet-F0 (DeepMind compatible) model to TorchBench. It should support other variants of the nfnet models as well.

PR that adds a bunch of timm models: https://github.com/pytorch/benchmark/pull/352

Closed by merging https://github.com/pytorch/benchmark/pull/352. Feel free to reopen it if people need more timm models.
Repository: pytorch/benchmark
Issue: Compare some PyTorch models against corresponding JAX models in the suite
Root cause: Unimplemented Feature Gap

This was brought up by @bertmaher offline, I am just writing an issue for it. It would be nice if we could have a set of models where JAX is more competitive than PyTorch and be able to run comparisons on both in torchbench.

Some prior work: https://github.com/TheSalon/fast-dpsgd benchmarked various differential private models between PT, JAX, and some other frameworks.