Repository: pytorch/benchmark
Issue: ask help: fail to run benchmark with model dlrm on cpu device and torchscript.
Root cause: Validation Gap

(torchbenchmark) root@ip-172-31-0-218:~/benchmark# python3 run_benchmark.py cpu --model dlrm --test eval --metrics=latencies,throughputs --backend=torchscript

Running benchmark: /root/miniconda3/envs/torchbenchmark/bin/python3 /root/benchmark/userbenchmark/cpu/run_config.py -m dlrm -d cpu -t eval --backend=torchscript --metrics latencies,throughputs --nwarmup 20 --niter 50 -o /root/benchmark/.userbenchmark/cpu/cpu-20231124033252
Running TorchBenchModelConfig(name='dlrm', test='eval', device='cpu', batch_size=None, extra_args=['--backend=torchscript'], extra_env=None) ...Traceback (most recent call last):
  File "/root/benchmark/userbenchmark/cpu/run_config.py", line 99, in <module>
    run(args, extra_args)
  File "/root/benchmark/userbenchmark/cpu/run_config.py", line 77, in run
    metrics_res = run_config(config, metrics, nwarmup=int(args.nwarmup), niter=int(args.niter), dryrun=args.dryrun)
  File "/root/benchmark/userbenchmark/cpu/run_config.py", line 55, in run_config
    model = load_model(config)
  File "/root/benchmark/torchbenchmark/util/experiment/instantiator.py", line 50, in load_model
    model_instance = Model(test=config.test, device=config.device, batch_size=config.batch_size, extra_args=config.extra_args)
  File "/root/benchmark/torchbenchmark/util/model.py", line 25, in __call__
    obj.__post__init__()
  File "/root/benchmark/torchbenchmark/util/model.py", line 136, in __post__init__
    apply_opt_args(self, self.opt_args)
  File "/root/benchmark/torchbenchmark/util/extra_args.py", line 139, in apply_opt_args
    model._enable_backend()
  File "/root/benchmark/torchbenchmark/util/backends/jit.py", line 36, in _torchscript
    module = torch.jit.script(module, example_inputs=[example_inputs, ])
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_script.py", line 1324, in script
    return torch.jit._recursive.create_script_module(
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 556, in create_script_module
    concrete_type = get_module_concrete_type(nn_module, share_types)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 505, in get_module_concrete_type
    concrete_type = concrete_type_store.get_or_create_concrete_type(nn_module)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 438, in get_or_create_concrete_type
    concrete_type_builder = infer_concrete_type_builder(nn_module)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 284, in infer_concrete_type_builder
    sub_concrete_type = get_module_concrete_type(item, share_types)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 505, in get_module_concrete_type
    concrete_type = concrete_type_store.get_or_create_concrete_type(nn_module)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 438, in get_or_create_concrete_type
    concrete_type_builder = infer_concrete_type_builder(nn_module)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 284, in infer_concrete_type_builder
    sub_concrete_type = get_module_concrete_type(item, share_types)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 505, in get_module_concrete_type
    concrete_type = concrete_type_store.get_or_create_concrete_type(nn_module)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 438, in get_or_create_concrete_type
    concrete_type_builder = infer_concrete_type_builder(nn_module)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 328, in infer_concrete_type_builder
    name, _get_valid_constant(name, value, type(nn_module).__name__)
  File "/root/miniconda3/envs/torchbenchmark/lib/python3.10/site-packages/torch/jit/_recursive.py", line 142, in _get_valid_constant
    raise TypeError(

TypeError: 
'numpy.int64' object in attribute 'EmbeddingBag.num_embeddings' is not a valid constant.
Valid constants are:
1. a nn.ModuleList
2. a value of type {bool, float, int, str, NoneType, torch.device, torch.layout, torch.dtype}
3. a list or tuple of (2)


By the way I can use same method to run benchmark on resnet50, vgg16 and BERT_pytoch

torchscript is now in maintenance mode and we do not aim to cover all models with torchscript. cc @davidberard98 

> torchscript is now in maintenance mode and we do not aim to cover all models with torchscript. cc @davidberard98

ok, understand。2 quick questions：
(1) is there a list to show supported models by torchscript ?
(2) generally speaking,  is there other backends I can use on pytorch for better performance than torchscript backends ?

thanks ~

For 1): you could run `python run.py <model-name> -d cuda -t eval --backend torchscript` to test a model with torchscript.
2): The torchinductor backend is recommended, but I am not sure if that is supported by the cpu userbenchmark. With `run.py`, you could run it with the `--torchdynamo inductor` argument, for example:

```
python run.py resnet50 -d cuda -t train --torchdynamo inductor
```

To run all the models with torchscript with a single command, you could try:

```
python run_benchmark.py test_bench -d cuda -t train,eval --backend torchscript
```

Closing it, feel free to re-open if you still have questions.