zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.1.0.dev20230311+cu117'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.1.0.dev20230310+cu117

Base PyTorch commit: 065de4301200117c65607b213322cbde800b50f8

Affected PyTorch version: 2.1.0.dev20230311+cu117

Affected PyTorch commit: f3fc4d035d8fc1ed560ade0c8b8eff6be1c95972

Affected Tests:
- test_train[hf_Albert-cpu-eager]: +15.39479%
- test_train[timm_regnet-cpu-eager]: +8.40256%
- test_train[timm_resnest-cpu-eager]: +19.38015%
- test_train[tts_angular-cpu-eager]: -8.19111%


cc @xuzhao9

Result json: 
```
{
  "start": "065de4301200117c65607b213322cbde800b50f8",
  "end": "f3fc4d035d8fc1ed560ade0c8b8eff6be1c95972",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "987eade3f38",
      "commit1_time": "2023-03-10 06:46:30 +0000",
      "commit1_digest": {
        "test_train[hf_Albert-cpu-eager]": 7.794565519620664
      },
      "commit2": "1b59c3feb55",
      "commit2_time": "2023-03-10 10:40:01 +0000",
      "commit2_digest": {
        "test_train[hf_Albert-cpu-eager]": 8.494273057905957
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4398581040


cc @malfet, this looks like a valid signal. Root cause commit: https://github.com/pytorch/pytorch/commit/1b59c3feb55, what do you think?

(bisect_nightly_20230310_cu117) weiwangmeta@hostname : ~/repos/benchmark $ pip list |grep torch
pytorch-triton           2.0.0+b8b470bc59
torch                    2.1.0.dev20230310+cu117
torchaudio               2.0.0.dev20230313+cu117
torchdata                0.6.0
torchtext                0.15.1
torchvision              0.15.0.dev20230314+cu117
(bisect_nightly_20230310_cu117) weiwangmeta@hostname : ~/repos/benchmark $ python test.py -k test_hf_Albert_train_cpu
.
----------------------------------------------------------------------
Ran 1 test in 10.555s


(bisect_nightly_20230310_cu117) weiwangmeta@hostname :  ~/repos/benchmark $ pip list |grep torch
pytorch-triton           2.1.0+2c32f43999
torch                    2.1.0.dev20230315+cu117
torchaudio               2.0.0.dev20230313+cu117
torchdata                0.6.0
torchtext                0.15.1
torchvision              0.15.0.dev20230314+cu117
(bisect_nightly_20230310_cu117) weiwangmeta@hostname : ~/repos/benchmark $ python test.py -k test_hf_Albert_train_cpu
.
----------------------------------------------------------------------
Ran 1 test in 10.279s

I'm comparing 03/10 nightly vs. 03/15 nightly, the performance does not seem to indicate regression. 

cc @malfet 


Thanks @weiwangmeta for helping verify. I will close and mark this as noise.
Repository: pytorch/benchmark
Issue: issues with userbenchmark/ddp_experiments/__init__.py
Root cause: Architectural Constraint

Hi everyone, 

the --timeout flag doesn't seem to work due to the hard coding here: https://github.com/pytorch/benchmark/blob/355dc8002ea89f2899cc048553d5975c2ea02954/userbenchmark/ddp_experiments/__init__.py#L220

The command I'm using is `python userbenchmark/ddp_experiments/__init__.py --ngpus 4 --distributed ddp --nodes 1 --cluster local --filter_models resnet50 --timeout 2`

Additionally, would it be better if the script leaves an option flag for the network interface name: https://github.com/pytorch/benchmark/blob/355dc8002ea89f2899cc048553d5975c2ea02954/userbenchmark/ddp_experiments/__init__.py#L374 ?

please correct if I'm not using the ddp benchmark correctly. 

Looks the timeout is for slurm job, is there any plan to support timeout for local?


@wconstab Can you help answer this question?

No plans per se, but happy to accept a PR to improve the script.  also cc @davidberard98 

I can open a PR

thanks everyone!
Repository: pytorch/benchmark
Issue: Some models require larger GPU memory size when running in isolated mode
Root cause: External Dependency Breakage

- alexnet: in-process bs=1024, isolated bs=512
- densenet121: in-process bs=64m isolated bs=16


See:  https://github.com/pytorch/benchmark/pull/1487

The reason is the recent code change in torchvision, fixed by updating the model code.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.0.0.dev20230209+cu117'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.0.0.dev20230208+cu117

Base PyTorch commit: 61ecaf1dd40ba93dde11249bd8ee64274098d3d7

Affected PyTorch version: 2.0.0.dev20230209+cu117

Affected PyTorch commit: cb715c26e270759e3955968fbd3d763d51b0b83d

Affected Tests:
- test_train[Background_Matting-cuda-jit]: -8.83328%
- test_eval[Super_SloMo-cuda-jit]: -16.15896%
- test_eval[Super_SloMo-cuda-eager]: -16.13090%
- test_eval[alexnet-cuda-jit]: -11.90016%
- test_eval[alexnet-cuda-eager]: -9.51970%
- test_train[dcgan-cuda-jit]: -22.74619%
- test_train[dcgan-cuda-eager]: -22.84246%
- test_eval[mnasnet1_0-cuda-jit]: -11.40067%
- test_eval[mobilenet_v3_large-cuda-jit]: -14.85722%
- test_eval[pyhpc_equation_of_state-cpu-jit]: +10.81298%
- test_eval[shufflenet_v2_x1_0-cuda-jit]: -19.10218%
- test_eval[shufflenet_v2_x1_0-cuda-eager]: +16.67777%
- test_eval[squeezenet1_1-cuda-jit]: -8.74146%
- test_eval[squeezenet1_1-cuda-eager]: -11.16717%
- test_train[timm_regnet-cpu-eager]: +10.39259%
- test_train[timm_regnet-cuda-eager]: +8.12088%
- test_train[timm_resnest-cuda-jit]: -7.17150%
- test_train[timm_resnest-cuda-eager]: -7.38399%
- test_train[timm_vision_transformer-cpu-eager]: -11.37185%


cc @xuzhao9

Result json: 
```
{
  "start": "61ecaf1dd40ba93dde11249bd8ee64274098d3d7",
  "end": "cb715c26e270759e3955968fbd3d763d51b0b83d",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "bbe33532aec",
      "commit1_time": "2023-02-08 10:41:10 +0000",
      "commit1_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.20370699933264405
      },
      "commit2": "566eb49ed2e",
      "commit2_time": "2023-02-08 11:22:55 +0000",
      "commit2_digest": {
        "test_train[timm_vision_transformer-cpu-eager]": 0.22996871413197367
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4458597621
Repository: pytorch/benchmark
Issue: Add SSD to benchmark model repo
Root cause: Unimplemented Feature Gap

This is in pytorch/hub: https://pytorch.org/hub/nvidia_deeplearningexamples_ssd/.

There's some evidence that this can be made much faster (up to 3x) with some graph optimizations: https://paulbridger.com/posts/video-analytics-deepstream-pipeline/


https://pytorch.org/hub/nvidia_deeplearningexamples_ssd/ is highly coupled with Nvidia's DeepStream and TensorRT. I decide to pick a more general implementation from https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection.git .
Repository: pytorch/benchmark
Issue: `doctr_det_predictor` failed installation without conda
Root cause: Configuration Error

with #1496, now torchbenchmark will fail the installation of `doctr_det_predictor` if `conda` is not installed. This looks like a breaking change as `conda` is explicitly mentioned as a optional, instead of required, dependency of `torchbench`.

cc: @xuzhao9 

Thanks for the feedback. I am leaning to change `conda` to a hard requirement because `doctr_det_predictor` requires the `libglib` version provided in the `conda-forge` repo. Although users can install `libglib` provided by their OS, the version might not be compatible to the version accepted by `doctr_det_predictor` and will complicate the user's installation process. What do you think? 

Also cc the PR author @janeyx99 

I feel like making `conda` a hard requirement just for one model is likely an overkill if there could be a workaround. If not then probably `conda` is the only way to go. `conda` also ships with some outdated runtimes so I will recommend to steer away from `conda` if possible.

BTW what about not failing the whole `python install.py` if some requirements of a model is not met? Like a graceful exit and warning about a specific model not available due to unmet dependencies. 

I think adding a warning here will be sufficient, as I understand that others like @shingjan may want to avoid conda for the reasons mentioned.
Repository: pytorch/benchmark
Issue: [Roadmap] Torchbench V3 release

We are working towards releasing TorchBench V3 by Apr 10, 2023.

The V3 nightly CI will be A100 GPU-only. This is because:
1. running all models CPU tests is too slow and takes over 24 hours to finish.
2. the Intel team is working on a new 'CPU' userbenchmark to track CPU performance
3. Before 2) is done, we will keep running V2 CI to track both CPU and GPU performance.

To release the V3 nightly CI, we need to do the following:

# Milestone 1
- [x] Run the model stableness test (Done, see https://github.com/pytorch/benchmark/actions/runs/4585968352)
- [x] Run the V3 workflow without score (https://github.com/pytorch/benchmark/pull/1523)
- [x] Add the V3 score metric to the workflow (https://github.com/pytorch/benchmark/pull/1538)
- [x] Build the internal dashboard to track the score and regression

# Milestone 2
- [x] Use userbenchmark regression detector to auto-detect regressions
- [x] Use userbenchmark bisector to bisect regressions


The model stableness workflow is done: https://github.com/pytorch/benchmark/actions/runs/4585968352.
Run each model test for 15 iterations. 16 tests are unstable, 141 tests are stable. The stableness threshold is 7%.

List of unstable tests:
- [ ] LearningToPaint train
- [ ] timm_efficientdet eval
- [ ] soft_actor_critic train
- [ ] resnext50_32x4d train
- [ ] lennard_jones eval
- [ ] functorch_maml_omniglot eval
- [ ] functorch_dp_cifar10 train
- [ ] drq eval
- [ ] detectron2_maskrcnn_r_50_fpn train
- [ ] detectron2_maskrcnn_r_50_c4 eval
- [ ] detectron2_fcos_r_50_fpn eval
- [ ] detectron2_fasterrcnn_r_50_c4 eval
- [ ] detectron2_fasterrcnn_r_50_c4 train
- [ ] detectron2_fasterrcnn_r_101_fpn eval
- [ ] detectron2_fasterrcnn_r_101_c4 eval
- [ ]  densenet121 eval
Repository: pytorch/benchmark
Issue: Add model with inner optimization loop to model repo
Root cause: Unimplemented Feature Gap

An inner optimization loop is when the functional autograd API is used in the forward pass and then the user differentiates through it. Uses include [MAML](https://github.com/dragen1860/MAML-Pytorch), [Energy Based Models](https://github.com/swyoon/pytorch-energy-based-model), and [(shameless self plug) set generation](https://github.com/CUAI/BetterSetRepresentations).

Would be nice as it requires autograd in the forward pass, and requires higher order gradients in the backwards pass. Definitely more "researchy" code.
Repository: pytorch/benchmark
Issue: Instruction unclear that conda is absolutely required 
Root cause: Documentation Deficiency

Error encountered while installing from python venv. It seems conda is required absolutely.

```
running setup for /home/user/try/benchmark/torchbenchmark/models/doctr_det_predictor...FAIL
Error for /home/user/try/benchmark/torchbenchmark/models/doctr_det_predictor:
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/try/benchmark/torchbenchmark/models/doctr_det_predictor/install.py", line 12, in <module>
    pip_install_requirements()
  File "/home/user/try/benchmark/torchbenchmark/models/doctr_det_predictor/install.py", line 8, in pip_install_requirements
    subprocess.check_call(["conda", "install", "-y", "expecttest", "libglib", "pango", "-c", "conda-forge"])
  File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 345, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.10/subprocess.py", line 969, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1845, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'conda'



---------------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/user/try/benchmark/install.py", line 114, in <module>
    raise RuntimeError("Failed to complete setup")
```

@dany-nonstop We recently fixed this issue in https://github.com/pytorch/benchmark/pull/1528, can you update and try again?

> @dany-nonstop We recently fixed this issue in #1528, can you update and try again?

yes it's gone!
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '2.1.0.dev20230401+cu117'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 2.1.0.dev20230331+cu117

Base PyTorch commit: 3dc440527817b065fa5cb652467921c9514cdd1c

Affected PyTorch version: 2.1.0.dev20230401+cu117

Affected PyTorch commit: 57c6f3fe90bccfed878b301c2d61bf42270c5b52

Affected Tests:
- test_eval[alexnet-cpu-eager]: -8.30050%
- test_eval[attention_is_all_you_need_pytorch-cpu-eager]: +8.68291%
- test_eval[pytorch_stargan-cpu-jit]: -7.04653%
- test_train[timm_nfnet-cpu-jit]: -7.10217%
- test_train[timm_nfnet-cpu-eager]: +7.59075%
- test_train[timm_regnet-cpu-eager]: -8.14479%


cc @xuzhao9

Result json: 
```
{
  "start": "3dc440527817b065fa5cb652467921c9514cdd1c",
  "end": "57c6f3fe90bccfed878b301c2d61bf42270c5b52",
  "threshold": 7,
  "timeout": 120,
  "torchbench_branch": "v2.0",
  "result": [
    {
      "commit1": "cb4bc8e0f5f",
      "commit1_time": "2023-03-31 18:16:17 +0000",
      "commit1_digest": {
        "test_train[timm_regnet-cpu-eager]": 0.6619071679015178
      },
      "commit2": "14ef91cea64",
      "commit2_time": "2023-03-31 18:18:12 +0000",
      "commit2_digest": {
        "test_train[timm_regnet-cpu-eager]": 0.7628113852988463
      }
    }
  ]
}
```
Bisection workflow link: https://github.com/pytorch/benchmark/actions/runs/4648633190


Noise, closed.
Repository: pytorch/benchmark
Issue: Understand the bisection job failures
Root cause: Environment Incompatibility

1. bisection-gh4589099003: torchaudio build failure, need to add env "BUILD_SOX=0"
2. bisection-gh4589099003: torch_cpu.so build failure, 2023-04-05T22:26:00.2705440Z /usr/bin/ld: /data/nvme/srcs/pytorch/build/lib/libtorch_cpu.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30' https://github.com/pytorch/benchmark/actions/runs/4622635280/jobs/8176824439

bisection-gh4589099003 on xz9/v2.0-test: 
[failed] https://github.com/pytorch/benchmark/actions/runs/4651511576
[failed] https://github.com/pytorch/benchmark/actions/runs/4654554091 
[failed] https://github.com/pytorch/benchmark/actions/runs/4657703908
[succeeded] https://github.com/pytorch/benchmark/actions/runs/4658057707


The reason is because the anaconda channel `ncurses` package was not packaged correctly. It doesn't include the `-ltinfo` dependency. The conda-forge version doesn't have this problem, so we need to use the ncurses package from conda-forge.

This should solve the "BUILD_SOX" problem.

Using system level libstdcxx:
https://github.com/pytorch/benchmark/actions/runs/4659539943

We have decided to use the conda libstdcxx,  not the system one, so closing this issue.
Repository: pytorch/benchmark
Issue: Enhance TorchBench coverage for large distributed workloads with CI support for IBM Cloud
Root cause: Unimplemented Feature Gap

The proposed work tasks are as below:
- [ ] Enable CI support for IBM Cloud to enhance the testing infrastructure for FSDP
- [ ] Benchmark new model(s) for FSDP training - e.g. add new hf_T5 with 3B parameters, etc.

Looping @mrshenli into the discussion, since the PyTorch Distributed team is also interested in building benchmarks in distributed setup.

We tested the workflow. Closing it for now and will reopen when we decide to move further.
Repository: pytorch/benchmark
Issue: NotImplementedError & AssertionError raised when launch LLAMA model eval 
Root cause: Validation Gap

Reproduce
```
python run.py llama -d cpu
```
Error message

```python
Traceback (most recent call last):
  File "run.py", line 298, in <module>
    m = Model(device=args.device, test=args.test, jit=(args.mode == "jit"), batch_size=args.bs, extra_args=extra_args)
  File "/workspace/benchmark/torchbenchmark/util/model.py", line 20, in __call__
    obj = type.__call__(cls, *args, **kwargs)
  File "/workspace/benchmark/torchbenchmark/models/llama/__init__.py", line 16, in __init__
    super().__init__(test=test, device=device, jit=jit, batch_size=batch_size, extra_args=extra_args)
  File "/workspace/benchmark/torchbenchmark/util/model.py", line 84, in __init__
    self.determine_batch_size(batch_size)
  File "/workspace/benchmark/torchbenchmark/util/model.py", line 216, in determine_batch_size
    raise NotImplementedError(f"Test {self.test} is not implemented.")
NotImplementedError: Test eval is not implemented.


```
The model have implemented eval test, seems like `DEFAULT_EVAL_BSIZE` was not provided , also, we need to modify `.cuda()` in `model.py `to avoid `AssertionError: Torch not compiled with CUDA enabled`  when cuda not available.


Also failed in [TorchInductor CPU Performance Dashboard](https://github.com/pytorch/pytorch/issues/93531#issuecomment-1502578929)
@chuanqi129
Repository: pytorch/benchmark
Issue: GAT model is broken
Root cause: External Dependency Breakage

From https://github.com/pytorch/benchmark/actions/runs/4684546103/jobs/8308885461?pr=1549 we found the gat model is broken with error message `ImportError: 'NeighborSampler' requires either 'pyg-lib' or 'torch-sparse'`.

Can you help take a look, @yanbing-j ?

Sure. I'm investigating.

May I ask if the error comes from recent days in CI check? If so,the error probably is the incompatibility between pytorch nightly and dependency libraries from https://data.pyg.org/whl/torch-2.0.0+cu117.html.

> May I ask if the error comes from recent days in CI check? If so,the error probably is the incompatibility between pytorch nightly and dependency libraries from https://data.pyg.org/whl/torch-2.0.0+cu117.html.

Yes, the error comes from recent days in CI check. If the dependency library is unstable, we consider move the model from `models/` directory to the [canary_models/](https://github.com/pytorch/benchmark/tree/main/torchbenchmark/canary_models) directory, so that CI won't check their status. What do you think?

> > May I ask if the error comes from recent days in CI check? If so,the error probably is the incompatibility between pytorch nightly and dependency libraries from https://data.pyg.org/whl/torch-2.0.0+cu117.html.
> 
> Yes, the error comes from recent days in CI check. If the dependency library is unstable, we consider move the model from `models/` directory to the [canary_models/](https://github.com/pytorch/benchmark/tree/main/torchbenchmark/canary_models) directory, so that CI won't check their status. What do you think?

Yes, I think moving the models to `canary_models` is a better choice now. I have connected with PyG maintainer, he confirmed that the wheel is compiled based on PyTorch 2.0 and could not be adaptive to pytorch nightly.

> > > May I ask if the error comes from recent days in CI check? If so,the error probably is the incompatibility between pytorch nightly and dependency libraries from https://data.pyg.org/whl/torch-2.0.0+cu117.html.
> > 
> > 
> > Yes, the error comes from recent days in CI check. If the dependency library is unstable, we consider move the model from `models/` directory to the [canary_models/](https://github.com/pytorch/benchmark/tree/main/torchbenchmark/canary_models) directory, so that CI won't check their status. What do you think?
> 
> Yes, I think moving the models to `canary_models` is a better choice now. I have connected with PyG maintainer, he confirmed that the wheel is compiled based on PyTorch 2.0 and could not be adaptive to pytorch nightly.

Let's move it to the `canary_models` then, can you help create a PR to do that? @yanbing-j 

Sure. @xuzhao9
Repository: pytorch/benchmark
Issue: speech_transformer does not respect batch_size
Root cause: Algorithmic Error

Even if you pass in train_bs to vary batch size (as is done in this backtrace

```
  /data/users/ezyang/c/pytorch/benchmarks/dynamo/torchbench.py(402)<module>()
-> torchbench_main()
  /data/users/ezyang/c/pytorch/benchmarks/dynamo/torchbench.py(398)torchbench_main()
-> main(TorchBenchmarkRunner(), original_dir)
  /data/users/ezyang/c/pytorch/benchmarks/dynamo/common.py(2000)main()
-> return maybe_fresh_cache(
  /data/users/ezyang/c/pytorch/benchmarks/dynamo/common.py(1017)inner()
-> return fn(*args, **kwargs)
  /data/users/ezyang/c/pytorch/benchmarks/dynamo/common.py(2377)run()
-> ) = runner.load_model(device, model_name, batch_size=batch_size)
  /data/users/ezyang/c/pytorch/benchmarks/dynamo/torchbench.py(291)load_model()
-> benchmark = benchmark_cls(
  /data/users/ezyang/c/torchbenchmark/torchbenchmark/util/model.py(19)__call__()
-> obj = type.__call__(cls, *args, **kwargs)
  /data/users/ezyang/c/torchbenchmark/torchbenchmark/models/speech_transformer/__init__.py(32)__init__()
-> self.traincfg = SpeechTransformerTrainConfig(prefetch=True, train_bs=self.batch_size, num_train_batch=NUM_TRAIN_BATCH, device=self.device)
  /data/users/ezyang/c/torchbenchmark/torchbenchmark/models/speech_transformer/config.py(73)__init__()
-> self.tr_dataset = AudioDataset(self.train_json, train_bs,
> /data/users/ezyang/c/torchbenchmark/torchbenchmark/models/speech_transformer/speech_transformer/data/data.py(84)__init__()
-> self.minibatch = minibatch
```

when you construct the dataset:

```
        self.cv_dataset = AudioDataset(self.valid_json, train_bs,
                                       self.maxlen_in, self.maxlen_out,
                                       batch_frames=self.batch_frames)
```

batch_frames is always specified, which means train_bs is ignored:

```
        # Method 2: Generate minibatch based on batch_frames
        # i.e. each batch contains approximately #batch_frames frames
        else:  # batch_frames > 0
            start = 0
            while True:
                total_frames = 0
                end = start
                while total_frames < batch_frames and end < len(sorted_data):
                    ilen = int(sorted_data[end][1]['input'][0]['shape'][0])
                    total_frames += ilen
                    end += 1
                # print(total_frames, end-start)
                minibatch.append(sorted_data[start:end])
                if end == len(sorted_data):
                    break
                start = end
```

Yeah we are using the configs from the upstream training script, and they specify both batch size and batch_frames:

https://github.com/kaituoxu/Speech-Transformer/blob/master/egs/aishell/run.sh#L167-L168

I believe this is an upstream problem. I will mark this model as `ALLOW_CUSTOMIZE_BSIZE = False`

Fixed by https://github.com/pytorch/benchmark/pull/1569
Repository: pytorch/benchmark
Issue: maml_omniglot does not respect batch size
Root cause: Configuration Error

Found the same way as https://github.com/pytorch/benchmark/issues/1560

Not all models support customizing batch size. In maml_omniglot, we specify the variable `ALLOW_CUSTOMIZE_BSIZE = False`: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/maml_omniglot/__init__.py#L50

This will throw error if user specifies anything different than the default batch size: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/model.py#L224

Ah! In that case I think the default batch size is wrong here, it didn't match the tensor I got out