Repository: mlcommons/inference
Issue: [SDXL] v4.1 nvidia automated benchmark issue
Root cause: Validation Gap

Hi, I am currently facing an issue with the sdxl benchmark. It seems to be an engine generation issue, and I successfully entered the docker container. How do I fix this? Is this an error with the cloned script?

run command:
```
mlcr run-mlperf,inference,_find-performance,_full,_r4.1-dev \
   --model=sdxl \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=datacenter \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=50
```

related error message
```
[2025-03-11 03:48:14,035 module.py:5095 DEBUG] -       - Running native script "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/run.sh" from temporal script "tmp-run.sh" in "/home/mlcuser/MLC/repos
/local/cache/app-mlperf-inference-nvidia_fcb3a0f4" ...
[2025-03-11 03:48:14,035 module.py:5102 INFO] -            ! cd /home/mlcuser/MLC/repos/local/cache/app-mlperf-inference-nvidia_fcb3a0f4
[2025-03-11 03:48:14,035 module.py:5103 INFO] -            ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/run.sh from tmp-run.sh
make generate_engines RUN_ARGS=' --benchmarks=stable-diffusion-xl --scenarios=offline  --test_mode=PerformanceOnly  --offline_expected_qps=1 --use_deque_limit --no_audit_verify  --model_path /home/mlcuser/MLC/repos/local/cache/get-mlperf
-inference-nvidia-scratch-space_addd1660/models/SDXL/ '
[2025-03-11 03:48:15,380 main.py:229 INFO] Detected system ID: KnownSystem.Nvidia_063d8e3ffd92
[2025-03-11 03:48:15,928 generate_engines.py:173 INFO] Building engines for stable-diffusion-xl benchmark in Offline scenario...
/home/mlcuser/.local/lib/python3.8/site-packages/torchvision/datapoints/__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some
APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn mor
e about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
/home/mlcuser/.local/lib/python3.8/site-packages/torchvision/transforms/v2/__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, so
me APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn
more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
[03/11/2025-03:48:18] [TRT] [I] [MemUsageChange] Init CUDA: CPU +1, GPU +0, now: CPU 140, GPU 4249 (MiB)
[03/11/2025-03:48:20] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1807, GPU +312, now: CPU 2083, GPU 4561 (MiB)
[03/11/2025-03:48:20] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 2083, GPU 4561 (MiB)
[03/11/2025-03:48:20] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 2083, GPU 4561 (MiB)
[03/11/2025-03:48:20] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 2083, GPU 4561 (MiB)
[2025-03-11 03:48:20,879 builder.py:379 INFO] Building CLIP1 from /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_addd1660/models/SDXL/onnx_models/clip1/model.onnx
[2025-03-11 03:48:21,431 sdxl_graphsurgeon.py:61 INFO] CLIP: original .. 1608 nodes, 1805 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:21,448 sdxl_graphsurgeon.py:61 INFO] CLIP: cleanup .. 1608 nodes, 1805 tensors, 1 inputs, 2 outputs
2025-03-11 03:48:21.698098919 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/Unsqueeze_2
2025-03-11 03:48:21.698126333 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/Unsqueeze_1
2025-03-11 03:48:21.698146931 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/Unsqueeze
2025-03-11 03:48:21.698151693 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/Unsqueeze_7
2025-03-11 03:48:22.740724937 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/embeddings/Unsqueeze
2025-03-11 03:48:23.644944623 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_17
2025-03-11 03:48:23.644967446 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_16
2025-03-11 03:48:23.644972711 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_14
2025-03-11 03:48:23.644976855 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_11
2025-03-11 03:48:23.644981138 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_8
2025-03-11 03:48:23.644985091 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_3
[2025-03-11 03:48:25,525 sdxl_graphsurgeon.py:61 INFO] CLIP: fold constants .. 1014 nodes, 1729 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:26,191 sdxl_graphsurgeon.py:61 INFO] CLIP: shape inference .. 1014 nodes, 1729 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:26,255 sdxl_graphsurgeon.py:61 INFO] CLIP: added hidden_states .. 1014 nodes, 1729 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:26,257 sdxl_graphsurgeon.py:61 INFO] CLIP: GS finished .. 1014 nodes, 1729 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:26,571 builder.py:123 INFO] Updating network outputs to ['text_embeddings', 'hidden_states']
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/base.py", line 189, in subprocess_target
    return self.action_handler.handle()
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 176, in handle
    total_engine_build_time += self.build_engine(job)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 167, in build_engine
    builder.build_engines()
  File "/home/mlcuser/.local/lib/python3.8/site-packages/nvmitten/nvidia/builder.py", line 513, in build_engines
    self.mitten_builder.run(self.legacy_scratch, None)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/builder.py", line 386, in run
    builder_config=self.clip_builder.create_builder_config(),
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/builder.py", line 205, in create_builder_config
    self.create_profiles(network=None, builder_config=None, batch_size=self.batch_size)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/builder.py", line 156, in gpu_profiles
    input_profile = self.model.get_input_profile(batch_size)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/network.py", line 92, in inner
    self._actual_check_dims(batch_size, )
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/network.py", line 61, in _actual_check_dims
    assert batch_size >= self.min_batch and batch_size <= self.max_batch, f"batch_size out of range of ({self.min_batch}, {self.max_batch})"
AssertionError: batch_size out of range of (1, 0)
[2025-03-11 03:48:28,158 generate_engines.py:173 INFO] Building engines for stable-diffusion-xl benchmark in Offline scenario...
/home/mlcuser/.local/lib/python3.8/site-packages/torchvision/datapoints/__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some
APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn mor
e about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
/home/mlcuser/.local/lib/python3.8/site-packages/torchvision/transforms/v2/__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, so
me APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn
more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
[03/11/2025-03:48:30] [TRT] [I] [MemUsageChange] Init CUDA: CPU +1, GPU +0, now: CPU 140, GPU 4249 (MiB)                                                                                                                   18:48:38 [69/1818]
[03/11/2025-03:48:32] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1807, GPU +312, now: CPU 2083, GPU 4561 (MiB)
[03/11/2025-03:48:32] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 2083, GPU 4561 (MiB)
[03/11/2025-03:48:32] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 2083, GPU 4561 (MiB)
[03/11/2025-03:48:32] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 2083, GPU 4561 (MiB)
[2025-03-11 03:48:32,294 builder.py:379 INFO] Building CLIP1 from /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_addd1660/models/SDXL/onnx_models/clip1/model.onnx
[2025-03-11 03:48:32,723 sdxl_graphsurgeon.py:61 INFO] CLIP: original .. 1608 nodes, 1805 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:32,740 sdxl_graphsurgeon.py:61 INFO] CLIP: cleanup .. 1608 nodes, 1805 tensors, 1 inputs, 2 outputs
2025-03-11 03:48:32.973030796 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/Unsqueeze_2
2025-03-11 03:48:32.973066217 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/Unsqueeze_1
2025-03-11 03:48:32.973096089 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/Unsqueeze
2025-03-11 03:48:32.973102153 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/Unsqueeze_7
2025-03-11 03:48:34.002360560 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/embeddings/Unsqueeze
2025-03-11 03:48:34.886182340 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_17
2025-03-11 03:48:34.886205198 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_16
2025-03-11 03:48:34.886210671 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_14
2025-03-11 03:48:34.886215222 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_11
2025-03-11 03:48:34.886219790 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_8
2025-03-11 03:48:34.886224026 [W:onnxruntime:, unsqueeze_elimination.cc:20 Apply] UnsqueezeElimination cannot remove node /text_model/encoder/layers.0/self_attn/Unsqueeze_3
[2025-03-11 03:48:36,738 sdxl_graphsurgeon.py:61 INFO] CLIP: fold constants .. 1014 nodes, 1729 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:37,384 sdxl_graphsurgeon.py:61 INFO] CLIP: shape inference .. 1014 nodes, 1729 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:37,450 sdxl_graphsurgeon.py:61 INFO] CLIP: added hidden_states .. 1014 nodes, 1729 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:37,451 sdxl_graphsurgeon.py:61 INFO] CLIP: GS finished .. 1014 nodes, 1729 tensors, 1 inputs, 2 outputs
[2025-03-11 03:48:37,749 builder.py:123 INFO] Updating network outputs to ['text_embeddings', 'hidden_states']
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/base.py", line 189, in subprocess_target
    return self.action_handler.handle()
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 176, in handle
    total_engine_build_time += self.build_engine(job)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 167, in build_engine
    builder.build_engines()
  File "/home/mlcuser/.local/lib/python3.8/site-packages/nvmitten/nvidia/builder.py", line 513, in build_engines
    self.mitten_builder.run(self.legacy_scratch, None)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/builder.py", line 386, in run
    builder_config=self.clip_builder.create_builder_config(),
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/builder.py", line 205, in create_builder_config
    self.create_profiles(network=None, builder_config=None, batch_size=self.batch_size)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/builder.py", line 156, in gpu_profiles
    input_profile = self.model.get_input_profile(batch_size)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/network.py", line 92, in inner
    self._actual_check_dims(batch_size, )
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/network.py", line 61, in _actual_check_dims
    assert batch_size >= self.min_batch and batch_size <= self.max_batch, f"batch_size out of range of ({self.min_batch}, {self.max_batch})"
AssertionError: batch_size out of range of (1, 0)
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/main.py", line 231, in <module>
    main(main_args, DETECTED_SYSTEM)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/main.py", line 144, in main
    dispatch_action(main_args, config_dict, workload_setting)
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/main.py", line 202, in dispatch_action
    handler.run()
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/base.py", line 82, in run
    self.handle_failure()
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/base.py", line 186, in handle_failure
    self.action_handler.handle_failure()
  File "/home/mlcuser/MLC/repos/local/cache/get-git-repo_19d644bb/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 184, in handle_failure
    raise RuntimeError("Building engines failed!")
RuntimeError: Building engines failed!
make: *** [Makefile:37: generate_engines] Error 1
Traceback (most recent call last):
  File "/home/mlcuser/.local/bin/mlcr", line 8, in <module>
    sys.exit(mlcr())
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 86, in mlcr
    main()
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 173, in main
    res = method(run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 141, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 121, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
    r = self._run(i)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1839, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3289, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3459, in _run_deps
    r = self.action_object.access(ii)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 141, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 121, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
    r = self._run(i)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1627, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3289, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3459, in _run_deps
    r = self.action_object.access(ii)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 141, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 131, in call_script_module_function
    raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = app-mlperf-inference-nvidia, return code = 256)
```

Can you try `--batch_size=2` option? Which GPU is this?