Repository: mlcommons/inference
Issue: nvidia build issues
Root cause: Environment Incompatibility

o, I have now 4 solid test scenarios thanks to everyone's help here.  The have all been tested in cpu mode.  I am now switching to nvidia and the docker container doesn't seem to build.

I will be testing resnet, retina, bert and gpt
command to set up container:
```
cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1 \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=1000
```

error:
```
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
INFO:root:ENV[CM_CUDA_VERSION]: 11.8
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu118
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /home/fuseadm/CM/repos/local/cache/af058f4059c24b16/install/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /home/fuseadm/CM/repos/local/cache/af058f4059c24b16/install
INFO:root:           ! cd /home/fuseadm
INFO:root:           ! call /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/run.sh from tmp-run.sh
rm: cannot remove 'a.out': No such file or directory

Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

Compiling program ...


Running program ...

INFO:root:========================================================
INFO:root:Print file tmp-run.out:
INFO:root:
INFO:root:Error: problem obtaining number of CUDA devices: 35

INFO:root:

CM error: Portable CM script failed (name = get-cuda-devices, return code = 256)
```

Also is there a simple way to reset / clear back to step 1?  

@howudodat is `nvidia-smi` working on the host machine? This error can happen if CUDA driver is missing on host or if it's not working inside the container. For the later, you can quit the container. Get the container id using `docker ps -a` and then do `docker start ID` and `docker attach ID` and it should work again from the previous working state. 

ok, I feel pretty dumb, I didn't have CUDA driver...we had to switch hardware and the previous board had the driver already when I started on it.  The resnet50 prep stage now results in **Specified optimization profiles must satisfy MIN<=OPT<=MAX.**

```
[2024-08-21 06:25:07,983 generate_engines.py:173 INFO] Building engines for resnet50 benchmark in Offline scenario...
[08/21/2024-06:25:08] [TRT] [I] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 33, GPU 194 (MiB)
[08/21/2024-06:25:12] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1799, GPU +312, now: CPU 1968, GPU 506 (MiB)
[2024-08-21 06:25:13,478 rn50_graphsurgeon.py:531 INFO] Adding Squeeze
[2024-08-21 06:25:13,479 rn50_graphsurgeon.py:566 INFO] Adding Conv layer, instead of FC
[2024-08-21 06:25:13,497 rn50_graphsurgeon.py:587 INFO] Adding TopK layer
[2024-08-21 06:25:13,498 rn50_graphsurgeon.py:604 INFO] Adding Cast Layer
[2024-08-21 06:25:13,499 rn50_graphsurgeon.py:623 INFO] Removing obsolete layers
[2024-08-21 06:25:13,502 rn50_graphsurgeon.py:413 INFO] Fusing ops in res2_mega
[2024-08-21 06:25:13,508 rn50_graphsurgeon.py:525 INFO] Plugin RES2_FULL_FUSION successful
[2024-08-21 06:25:13,509 rn50_graphsurgeon.py:333 INFO] Replacing all branch2c beta=1 conv with smallk kernel.
[2024-08-21 06:25:13,509 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res3a_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,510 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res3b_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,511 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res3c_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,512 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res3d_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,512 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res4a_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,513 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res4b_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,514 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res4c_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,514 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res4d_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,515 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res4e_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,516 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res4f_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,517 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res5a_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,518 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res5b_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,518 rn50_graphsurgeon.py:399 INFO] Fusing SmallTileGEMM_TRT_res5c_branch2c_conv_residual_relu with smallk...
[2024-08-21 06:25:13,522 rn50_graphsurgeon.py:406 INFO] Plugin SmallTileGEMM_TRT fused successful for res3/4/5 branch2c
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: RnRes2FullFusion_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: RnRes2FullFusion_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: RnRes2FullFusion_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: RnRes2FullFusion_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[08/21/2024-06:25:13] [TRT] [I] No checker registered for op: SmallTileGEMM_TRT. Attempting to check as plugin.
[08/21/2024-06:25:13] [TRT] [I] No importer registered for op: SmallTileGEMM_TRT. Attempting to import as plugin.
[08/21/2024-06:25:13] [TRT] [I] Searching for plugin: SmallTileGEMM_TRT, plugin_version: 1, plugin_namespace:
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBias not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasRelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasGelu not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [W] builtin_op_importers.cpp:5677: Attribute epilogueScaleBiasBeta not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[08/21/2024-06:25:13] [TRT] [I] Successfully created plugin: SmallTileGEMM_TRT
[2024-08-21 06:25:13,980 ResNet50.py:259 INFO] Unmarking output: topk_layer_output_value
[2024-08-21 06:25:13,981 builder.py:329 INFO] Building engine to build/engines/SN00050/resnet50/Offline/resnet50-Offline-gpu-b0-int8.lwis_k_99_MaxP.plan
[2024-08-21 06:25:13,981 builder.py:335 INFO] Building optimization profiles.
[08/21/2024-06:25:13] [TRT] [E] 4: Specified optimization profiles must satisfy MIN<=OPT<=MAX.
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/actionhandler/base.py", line 189, in subprocess_target
    return self.action_handler.handle()
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 176, in handle
    total_engine_build_time += self.build_engine(job)
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 167, in build_engine
    builder.build_engines()
  File "/home/cmuser/.local/lib/python3.8/site-packages/nvmitten/nvidia/builder.py", line 512, in build_engines
    self.mitten_builder.run(self.legacy_scratch, None)
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/resnet50/tensorrt/ResNet50.py", line 324, in run
    self.build_engine(network, builder_config, batch_size, engine_fpath)
  File "/home/cmuser/.local/lib/python3.8/site-packages/nvmitten/nvidia/builder.py", line 336, in build_engine
    self.create_profiles(network, builder_config, batch_size)
  File "/home/cmuser/.local/lib/python3.8/site-packages/nvmitten/nvidia/builder.py", line 375, in gpu_profiles
    raise RuntimeError("Invalid optimization profile!")
RuntimeError: Invalid optimization profile!
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/main.py", line 231, in <module>
    main(main_args, DETECTED_SYSTEM)
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/main.py", line 144, in main
    dispatch_action(main_args, config_dict, workload_setting)
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/main.py", line 202, in dispatch_action
    handler.run()
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/actionhandler/base.py", line 82, in run
    self.handle_failure()
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/actionhandler/base.py", line 186, in handle_failure
    self.action_handler.handle_failure()
  File "/home/cmuser/CM/repos/local/cache/9d77f3bbd29e43eb/repo/closed/NVIDIA/code/actionhandler/generate_engines.py", line 184, in handle_failure
    raise RuntimeError("Building engines failed!")
RuntimeError: Building engines failed!
make: *** [Makefile:37: generate_engines] Error 1

CM error: Portable CM script failed (name = app-mlperf-inference-nvidia, return code = 256)
```

Oh. This happens if the gpu memory is different from the already handled ones by CM. Due you know the gpu memory available? Or the gpu name? Using `--batch_size=64` is an alternate way to get around this issue for resnet50. 

```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A2000 Embedde...    Off | 00000000:02:00.0 Off |                  Off |
| N/A   42C    P8               8W /  26W |     18MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1062      G   /usr/lib/xorg/Xorg                            9MiB |
|    0   N/A  N/A      1285      G   /usr/bin/gnome-shell                          2MiB |
+---------------------------------------------------------------------------------------+
```

It's 8GB which is supposed to be handled - let me check and fix it. Can you please try the batch_size option? Resnet50 should work on 8GB but most other benchmarks will need more GPU memory. 

is there a reason why this needs to re-clone all the repose (ie pytorch) every time you run the command?  cant it just do it once and check for updates?  Everytime this fails I end up wasting 6-8 hours waiting for it to re-clone the repos and rebuild.  If I run it in screen I cant scroll back far enough to get the error, if I dont, well...vpn goes down, wifi drops, you name it and I'm starting over at 0 

```
cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1 \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=1000 \
   --batch_size=64
```

last time it ran it was compiling took 5+ hours to compile and then died

The problem is not in cloning but in building pytorch from src which is part of the Nvidia workflow. And this build is slow - on a 128GB, 32 cores, Zen4 system it takes about 90 minutes. But once this is done and you are inside the container, there's no need to do it again. 

the problem is that the above "setup" command is failing so I have to rerun that command, right?  (PS any chance there is a gitter channel for this project?)

When the set up command fails are you returning to the host or staying inside the container? If its host, yes need to redo it. But if pytorch build failed a rerun is unlikely to help.

How many cores are there in the system and what's the system memory size? 

Currently there are no chat channels for CM as we are finalising the developments. Github issues are the best way to communicate as of now. 

I have 2 boards I am testing with, sadly the other board had a thermal issue so I am awaiting its replacement.  It had 32G of memory.  This one only has 16G.  CPU is Intel Core i7-8650U 8-Core.

Would it be better to do this as a non-docker build?

I dont know if it's building pytorch...it's currently on XXNPACK at about 56%

Oh okay. I believe it'll be extremely tough to get it working on 16GB. Which all benchmarks are you targeting?

Yes, xnnpack is part of pytorch build from src. If you are not planning to run LLM benchmarks we can try a way to do without pytorch build from src. 

we are trying as many benchmarks as possible.  We need to profile our edge devices with a standardized test in order to quantify improved performance with a GPU

well this just keeps failing no matter what I try, but I dont think it's memory related.  It seems the git clone is timing out?

```
 5969.3 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/cutlass'...
6135.8 error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)
6135.8 fatal: the remote end hung up unexpectedly
6135.8 fatal: early EOF
6135.8 fatal: index-pack failed
6135.9 fatal: clone of 'https://github.com/NVIDIA/cutlass.git' into submodule path '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/cutlass' failed
6135.9 Failed to clone 'third_party/cutlass'. Retry scheduled
6135.9 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/eigen'...
6148.7 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/fbgemm'...
6154.0 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/flatbuffers'...
6158.8 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/fmt'...
6164.8 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/foxi'...
6165.4 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/gemmlowp/gemmlowp'...
6237.0 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/gloo'...
6537.1 fatal: unable to access 'https://github.com/facebookincubator/gloo/': Operation timed out after 300039 milliseconds with 0 out of 0 bytes received
6537.1 fatal: clone of 'https://github.com/facebookincubator/gloo' into submodule path '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/gloo' failed
6537.1 Failed to clone 'third_party/gloo'. Retry scheduled
6537.1 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/googletest'...
6697.5 fatal: the remote end hung up unexpectedly
6697.5 fatal: early EOF
6697.5 fatal: index-pack failed
6697.5 fatal: clone of 'https://github.com/google/googletest.git' into submodule path '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/googletest' failed
6697.5 Failed to clone 'third_party/googletest'. Retry scheduled
6697.5 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/ideep'...
6699.1 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/ios-cmake'...
6699.7 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/ittapi'...
6700.5 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/kineto'...
6709.2 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/mimalloc'...
6711.6 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/nccl/nccl'...
6713.3 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/neon2sse'...
6714.1 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/nlohmann'...
6740.8 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/onnx'...
6749.8 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/onnx-tensorrt'...
6782.8 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/pocketfft'...
6784.4 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/protobuf'...
6970.4 fatal: the remote end hung up unexpectedly
6970.4 fatal: early EOF
6970.4 fatal: index-pack failed
6970.4 fatal: clone of 'https://github.com/protocolbuffers/protobuf.git' into submodule path '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/protobuf' failed
6970.4 Failed to clone 'third_party/protobuf'. Retry scheduled
6970.4 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/psimd'...
6971.0 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/pthreadpool'...
6971.7 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/pybind11'...
6974.9 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/python-enum'...
6975.5 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/python-peachpy'...
6976.5 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/python-six'...
6977.5 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/sleef'...
6979.3 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/tbb'...
6989.0 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/tensorpipe'...
6991.5 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/zstd'...
6999.9 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/XNNPACK'...
7037.0 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/cudnn_frontend'...
7043.2 Cloning into '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/cutlass'...
7204.3 fatal: the remote end hung up unexpectedly
7204.3 fatal: early EOF
7204.3 fatal: index-pack failed
7204.3 fatal: clone of 'https://github.com/NVIDIA/cutlass.git' into submodule path '/home/cmuser/CM/repos/local/cache/be59865dbe4e4789/pytorch/third_party/cutlass' failed
7204.3 Failed to clone 'third_party/cutlass' a second time, aborting
7205.9 Requirement already satisfied: astunparse in /home/cmuser/.local/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (1.6.3)
7206.1 Collecting expecttest
7206.1   Downloading expecttest-0.2.1-py3-none-any.whl (7.4 kB)
7208.8 Collecting hypothesis
7208.8   Downloading hypothesis-6.111.1-py3-none-any.whl (467 kB)
7209.0 Requirement already satisfied: numpy in /home/cmuser/.local/lib/python3.8/site-packages (from -r requirements.txt (line 5)) (1.22.4)
7209.5 Collecting psutil
7209.5   Downloading psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (290 kB)
7209.6 Requirement already satisfied: pyyaml in /home/cmuser/.local/lib/python3.8/site-packages (from -r requirements.txt (line 7)) (6.0.2)
7209.6 Requirement already satisfied: requests in /home/cmuser/.local/lib/python3.8/site-packages (from -r requirements.txt (line 8)) (2.32.3)
7209.6 Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from -r requirements.txt (line 9)) (45.2.0)
7209.8 Collecting types-dataclasses



(cm) fuseadm@fuseadm-ADDC-KBL-DDR4:~$ cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1 \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=1000 \
   --batch_size=64
INFO:root:* cm run script "run-mlperf inference _find-performance _full _r4.1"
INFO:root:  * cm run script "get mlcommons inference src"
INFO:root:       ! load /home/fuseadm/CM/repos/local/cache/7a44084e40774cc5/cm-cached-state.json
INFO:root:  * cm run script "get sut description"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /home/fuseadm
INFO:root:           ! call /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "detect cpu"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/fuseadm
INFO:root:             ! call /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /home/fuseadm
INFO:root:           ! call /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:    * cm run script "get python3"
INFO:root:         ! load /home/fuseadm/CM/repos/local/cache/6f4f5b90a8dd432f/cm-cached-state.json
INFO:root:Path to Python: /home/fuseadm/cm/bin/python3
INFO:root:Python version: 3.10.12
INFO:root:    * cm run script "get compiler"
INFO:root:         ! load /home/fuseadm/CM/repos/local/cache/6e7260ef555940af/cm-cached-state.json
INFO:root:    * cm run script "get cuda-devices"
INFO:root:      * cm run script "get cuda _toolkit"
INFO:root:           ! load /home/fuseadm/CM/repos/local/cache/f1c9408fe8bc4fa8/cm-cached-state.json
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
INFO:root:ENV[CM_CUDA_VERSION]: 11.8
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu118
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /home/fuseadm/CM/repos/local/cache/af058f4059c24b16/install/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /home/fuseadm/CM/repos/local/cache/af058f4059c24b16/install
INFO:root:           ! cd /home/fuseadm
INFO:root:           ! call /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/run.sh from tmp-run.sh
rm: cannot remove 'a.out': No such file or directory

Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

Compiling program ...


Running program ...

/home/fuseadm
INFO:root:           ! call "postprocess" from /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/customize.py
GPU Device ID: 0
GPU Name: NVIDIA RTX A2000 Embedded GPU
GPU compute capability: 8.6
CUDA driver version: 12.2
CUDA runtime version: 11.8
Global memory: 8361017344
Max clock rate: 1815.000000 MHz
Total amount of shared memory per block: 49152
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor:  1536
Maximum number of threads per block: 1024
Max dimension size of a thread block X: 1024
Max dimension size of a thread block Y: 1024
Max dimension size of a thread block Z: 64
Max dimension size of a grid size X: 2147483647
Max dimension size of a grid size Y: 65535
Max dimension size of a grid size Z: 65535


INFO:root:    * cm run script "get generic-python-lib _package.dmiparser"
INFO:root:         ! load /home/fuseadm/CM/repos/local/cache/b50994c9e5934f1c/cm-cached-state.json
INFO:root:    * cm run script "get cache dir _name.mlperf-inference-sut-descriptions"
INFO:root:         ! load /home/fuseadm/CM/repos/local/cache/0c4b51e1cbc84566/cm-cached-state.json
Generating SUT description file for fuseadm_ADDC_KBL_DDR4-tensorrt
INFO:root:         ! call "postprocess" from /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-sut-description/customize.py
INFO:root:  * cm run script "install pip-package for-cmind-python _package.tabulate"
INFO:root:       ! load /home/fuseadm/CM/repos/local/cache/6c459a3250cf42b4/cm-cached-state.json
INFO:root:  * cm run script "get mlperf inference utils"
INFO:root:    * cm run script "get mlperf inference src"
INFO:root:         ! load /home/fuseadm/CM/repos/local/cache/7a44084e40774cc5/cm-cached-state.json
INFO:root:         ! call "postprocess" from /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /home/fuseadm/CM/repos/local/cache/bdc1cb1047e7440e/inference

Running loadgen scenario: Offline and mode: performance
INFO:root:* cm run script "build dockerfile"

Dockerfile generated at /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference/dockerfiles/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public.Dockerfile
INFO:root:* cm run script "get docker"
INFO:root:     ! load /home/fuseadm/CM/repos/local/cache/d7fd254631474769/cm-cached-state.json
INFO:root:* cm run script "get mlperf inference results dir local"
INFO:root:     ! load /home/fuseadm/CM/repos/local/cache/ac37f504c62f496b/cm-cached-state.json
INFO:root:* cm run script "get mlperf inference submission dir local"
INFO:root:     ! load /home/fuseadm/CM/repos/local/cache/d13a9436c2df4e5c/cm-cached-state.json
INFO:root:* cm run script "get dataset imagenet validation original _full"
INFO:root:     ! load /home/fuseadm/CM/repos/local/cache/c7ba5ae996334777/cm-cached-state.json
INFO:root:* cm run script "get nvidia-docker"
INFO:root:     ! load /home/fuseadm/CM/repos/local/cache/d38e0ab441a44d71/cm-cached-state.json
INFO:root:* cm run script "get mlperf inference nvidia scratch space"
INFO:root:     ! load /home/fuseadm/CM/repos/local/cache/d4f6158f8d5a4ebe/cm-cached-state.json
INFO:root:* cm run script "get nvidia-docker"
INFO:root:     ! load /home/fuseadm/CM/repos/local/cache/d38e0ab441a44d71/cm-cached-state.json

CM command line regenerated to be used inside Docker:

cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_TMP_CURRENT_PATH=/home/fuseadm --env.CM_TMP_PIP_VERSION_STRING= --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True  --env.CM_DATASET_IMAGENET_PATH=/home/cmuser/CM/repos/local/cache/c7ba5ae996334777/imagenet-2012-val  --env.CM_MLPERF_INFERENCE_RESULTS_DIR=/home/cmuser/CM/repos/local/cache/ac37f504c62f496b  --env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/home/cmuser/CM/repos/local/cache/d13a9436c2df4e5c/mlperf-inference-submission  --env.MLPERF_SCRATCH_PATH=/home/cmuser/CM/repos/local/cache/d4f6158f8d5a4ebe  --docker_run_deps


INFO:root:* cm run script "run docker container"

Checking Docker images:

  docker images -q local/cm-script-app-mlperf-inference:ubuntu-20.04-latest 2> /dev/null

INFO:root:  * cm run script "build docker image"
================================================
CM generated the following Docker build command:

docker build  --build-arg GID=\" $(id -g $USER) \" --build-arg UID=\" $(id -u $USER) \" -f "/home/fuseadm/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference/dockerfiles/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public.Dockerfile" -t "local/cm-script-app-mlperf-inference:ubuntu-20.04-latest" .

INFO:root:         ! cd /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference/dockerfiles
INFO:root:         ! call /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/build-docker-image/run.sh from tmp-run.sh
[+] Building 90.9s (17/17) FINISHED                                                                                                          docker:default
 => [internal] load build definition from mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public.Dockerfile                          0.0s
 => => transferring dockerfile: 3.19kB                                                                                                                 0.0s
 => [internal] load metadata for nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public                        0.5s
 => [internal] load .dockerignore                                                                                                                      0.0s
 => => transferring context: 45B                                                                                                                       0.0s
 => [ 1/14] FROM nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public@sha256:816bd0cc61b061c9b9e2ebecabd67f  0.0s
 => CACHED [ 2/14] RUN apt-get update -y                                                                                                               0.0s
 => CACHED [ 3/14] RUN apt-get install -y python3 python3-pip git sudo wget python3-venv                                                               0.0s
 => CACHED [ 4/14] RUN ln -snf /usr/share/zoneinfo/US/Pacific /etc/localtime && echo US/Pacific >/etc/timezone                                         0.0s
 => CACHED [ 5/14] RUN groupadd -g  1000  -o cm                                                                                                        0.0s
 => CACHED [ 6/14] RUN useradd -m -u  1000  -g  1000  -o --create-home --shell /bin/bash cmuser                                                        0.0s
 => CACHED [ 7/14] RUN echo "cmuser ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers                                                                           0.0s
 => CACHED [ 8/14] WORKDIR /home/cmuser                                                                                                                0.0s
 => CACHED [ 9/14] RUN python3 -m venv cm-venv                                                                                                         0.0s
 => CACHED [10/14] RUN . cm-venv/bin/activate                                                                                                          0.0s
 => CACHED [11/14] RUN python3 -m pip install --user cmind requests giturlparse tabulate                                                               0.0s
 => CACHED [12/14] RUN cm pull repo mlcommons@cm4mlops  --branch=mlperf-inference                                                                      0.0s
 => CACHED [13/14] RUN cm run script --tags=get,sys-utils-cm --quiet                                                                                   0.0s
 => ERROR [14/14] RUN cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true   90.2s
------
 > [14/14] RUN cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.CM_RUN_STATE_DOCKER=True:
0.554 INFO:root:* cm run script "app mlperf inference generic _nvidia _resnet50 _tensorrt _cuda _test _r4.1_default _offline"
0.554 DEBUG:root:  - Number of scripts found: 1
0.554 DEBUG:root:  - Found script::app-mlperf-inference,d775cac873ee4231 in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference
0.556 DEBUG:root:    Prepared variations: _nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline,_nvidia-original,_float32
0.557 INFO:root:  - Doing fake run for script::app-mlperf-inference,d775cac873ee4231 as we are inside docker
0.558 DEBUG:root:  - Checking dependencies on other CM scripts:
0.572 INFO:root:  * cm run script "detect os"
0.572 DEBUG:root:    - Number of scripts found: 1
0.572 DEBUG:root:    - Found script::detect-os,863735b7db8c44fc in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os
0.573 DEBUG:root:    - Running preprocess ...
0.593 DEBUG:root:    - Running native script "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh" from temporal script "tmp-run.sh" in "/home/cmuser" ...
0.593 INFO:root:         ! cd /home/cmuser
0.593 INFO:root:         ! call /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
0.625 INFO:root:         ! call "postprocess" from /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
0.625 DEBUG:root:    - Running postprocess ...
0.657 INFO:root:    - running time of script "detect-os,detect,os,info": 0.10 sec.
0.675 INFO:root:  * cm run script "get sys-utils-cm"
0.675 DEBUG:root:    - Number of scripts found: 1
0.675 DEBUG:root:    - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-cm
0.677 DEBUG:root:      - Number of cached script outputs found: 1
0.677 DEBUG:root:    - Found script::get-sys-utils-cm,bc90993277e84b8e in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-sys-utils-cm
0.677 DEBUG:root:    - Checking if script execution is already cached ...
0.677 DEBUG:root:      - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-cm
0.678 DEBUG:root:      - Found cached script output: /home/cmuser/CM/repos/local/cache/2c09663703e24afa
0.678 DEBUG:root:    - Checking dynamic dependencies on other CM scripts:
0.679 DEBUG:root:    - Processing env after dependencies ...
0.679 DEBUG:root:      - Checking prehook dependencies on other CM scripts:
0.679 DEBUG:root:        - Loading state from cached entry ...
0.679 INFO:root:       ! load /home/cmuser/CM/repos/local/cache/2c09663703e24afa/cm-cached-state.json
0.680 DEBUG:root:      - Checking posthook dependencies on other CM scripts:
0.680 DEBUG:root:      - Checking post dependencies on other CM scripts:
0.680 INFO:root:    - running time of script "get,sys-utils-cm": 0.02 sec.
0.693 INFO:root:  * cm run script "get python"
0.693 DEBUG:root:    - Number of scripts found: 1
0.693 DEBUG:root:    - Searching for cached script outputs with the following tags: -tmp,get,python
0.697 DEBUG:root:      - Number of cached script outputs found: 0
0.697 DEBUG:root:    - Found script::get-python3,d0b5dd74373f4a62 in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-python3
0.698 DEBUG:root:    - Checking if script execution is already cached ...
0.698 DEBUG:root:      - Searching for cached script outputs with the following tags: -tmp,get,python,python3,get-python,get-python3
0.702 DEBUG:root:    - Creating new "cache" script artifact in the CM local repository ...
0.702 DEBUG:root:      - Tags: tmp,get,python,python3,get-python,get-python3,script-artifact-d0b5dd74373f4a62
0.767 DEBUG:root:    - Changing to /home/cmuser/CM/repos/local/cache/f1ca0c9587ff4c57
0.773 DEBUG:root:    - Running preprocess ...
0.829 INFO:root:      * /usr/bin/python3
0.830 DEBUG:root:        - Running native script "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-python3/run.sh" from temporal script "tmp-run.sh" in "/home/cmuser/CM/repos/local/cache/f1ca0c9587ff4c57" ...
0.830 INFO:root:             ! cd /home/cmuser/CM/repos/local/cache/f1ca0c9587ff4c57
0.830 INFO:root:             ! call /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-python3/run.sh from tmp-run.sh
0.841 INFO:root:             ! call "detect_version" from /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-python3/customize.py
0.841 DEBUG:root:        - Running detect_version ...
0.862 INFO:root:      # Found artifact in /usr/bin/python3
0.862 DEBUG:root:    - Checking prehook dependencies on other CM scripts:
0.863 DEBUG:root:    - Running native script "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-python3/run.sh" from temporal script "tmp-run.sh" in "/home/cmuser/CM/repos/local/cache/f1ca0c9587ff4c57" ...
0.863 INFO:root:         ! cd /home/cmuser/CM/repos/local/cache/f1ca0c9587ff4c57
0.863 INFO:root:         ! call /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-python3/run.sh from tmp-run.sh
0.874 INFO:root:         ! call "postprocess" from /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-python3/customize.py
0.874 DEBUG:root:    - Running postprocess ...
0.901 DEBUG:root:    - Removing tmp tag in the script cached output f1ca0c9587ff4c57 ...
0.925 INFO:root:    - cache UID: f1ca0c9587ff4c57
0.925 INFO:root:    - running time of script "get,python,python3,get-python,get-python3": 0.24 sec.
0.925 INFO:root:Path to Python: /usr/bin/python3
0.926 INFO:root:Python version: 3.8.10
0.939 INFO:root:  * cm run script "get mlcommons inference src"
0.940 DEBUG:root:    - Number of scripts found: 1
0.940 DEBUG:root:    - Searching for cached script outputs with the following tags: -tmp,get,mlcommons,inference,src
0.945 DEBUG:root:      - Number of cached script outputs found: 0
0.945 DEBUG:root:    - Found script::get-mlperf-inference-src,4b57186581024797 in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-src
0.945 DEBUG:root:      Prepared variations: _short-history
0.945 DEBUG:root:    - Checking if script execution is already cached ...
0.945 DEBUG:root:      - Searching for cached script outputs with the following tags: -tmp,get,mlcommons,inference,src,source,inference-src,inference-source,mlperf
0.951 DEBUG:root:    - Creating new "cache" script artifact in the CM local repository ...
0.951 DEBUG:root:      - Tags: tmp,get,mlcommons,inference,src,source,inference-src,inference-source,mlperf,script-artifact-4b57186581024797
0.980 DEBUG:root:    - Changing to /home/cmuser/CM/repos/local/cache/d3b3109edbfb4720
0.980 DEBUG:root:    - Checking dependencies on other CM scripts:
0.992 INFO:root:    * cm run script "detect os"
0.992 DEBUG:root:      - Number of scripts found: 1
0.992 DEBUG:root:      - Found script::detect-os,863735b7db8c44fc in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os
0.993 DEBUG:root:      - Running preprocess ...
1.014 DEBUG:root:      - Running native script "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh" from temporal script "tmp-run.sh" in "/home/cmuser/CM/repos/local/cache/d3b3109edbfb4720" ...
1.014 INFO:root:           ! cd /home/cmuser/CM/repos/local/cache/d3b3109edbfb4720
1.014 INFO:root:           ! call /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
1.046 INFO:root:           ! call "postprocess" from /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
1.046 DEBUG:root:      - Running postprocess ...
1.079 INFO:root:      - running time of script "detect-os,detect,os,info": 0.10 sec.
1.094 INFO:root:    * cm run script "get python3"
1.095 DEBUG:root:      - Number of scripts found: 1
1.095 DEBUG:root:      - Searching for cached script outputs with the following tags: -tmp,get,python3
1.096 DEBUG:root:        - Number of cached script outputs found: 1
1.096 DEBUG:root:      - Found script::get-python3,d0b5dd74373f4a62 in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-python3
1.096 DEBUG:root:      - Checking if script execution is already cached ...
1.097 DEBUG:root:        - Searching for cached script outputs with the following tags: -tmp,get,python3,python,get-python,get-python3
1.098 DEBUG:root:        - Found cached script output: /home/cmuser/CM/repos/local/cache/f1ca0c9587ff4c57
1.098 DEBUG:root:        - Checking prehook dependencies on other CM scripts:
1.098 DEBUG:root:          - Loading state from cached entry ...
1.098 INFO:root:         ! load /home/cmuser/CM/repos/local/cache/f1ca0c9587ff4c57/cm-cached-state.json
1.099 DEBUG:root:        - Checking posthook dependencies on other CM scripts:
1.099 DEBUG:root:        - Checking post dependencies on other CM scripts:
1.099 INFO:root:      - running time of script "get,python,python3,get-python,get-python3": 0.02 sec.
1.099 INFO:root:Path to Python: /usr/bin/python3
1.099 INFO:root:Python version: 3.8.10
1.100 DEBUG:root:    - Processing env after dependencies ...
1.103 DEBUG:root:    - Running preprocess ...
1.129 DEBUG:root:    - Checking prehook dependencies on other CM scripts:
1.142 INFO:root:    * cm run script "get git repo _branch.master _repo.https://github.com/mlcommons/inference"
1.142 DEBUG:root:      - Number of scripts found: 1
1.142 DEBUG:root:      - Searching for cached script outputs with the following tags: -tmp,get,git,repo,_branch.master,_repo.https://github.com/mlcommons/inference
1.147 DEBUG:root:        - Number of cached script outputs found: 0
1.147 DEBUG:root:      - Found script::get-git-repo,ed603e7292974f10 in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-git-repo
1.147 DEBUG:root:        Prepared variations: _branch.master,_repo.https://github.com/mlcommons/inference,_short-history
1.148 DEBUG:root:      - Checking if script execution is already cached ...
1.148 DEBUG:root:        - Prepared explicit variations: _branch.master,_repo.https://github.com/mlcommons/inference
1.148 DEBUG:root:        - Prepared variations: _branch.master,_repo.https://github.com/mlcommons/inference
1.148 DEBUG:root:        - Searching for cached script outputs with the following tags: -tmp,get,git,repo,repository,clone,_branch.master,_repo.https://github.com/mlcommons/inference,inference,src
1.153 DEBUG:root:      - Creating new "cache" script artifact in the CM local repository ...
1.153 DEBUG:root:        - Tags: tmp,get,git,repo,repository,clone,_branch.master,_repo.https://github.com/mlcommons/inference,inference,src,script-artifact-ed603e7292974f10
1.183 DEBUG:root:      - Changing to /home/cmuser/CM/repos/local/cache/a164a48590d34909
1.183 DEBUG:root:      - Checking dependencies on other CM scripts:
1.194 INFO:root:      * cm run script "detect os"
1.194 DEBUG:root:        - Number of scripts found: 1
1.194 DEBUG:root:        - Found script::detect-os,863735b7db8c44fc in /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os
1.195 DEBUG:root:        - Running preprocess ...
1.220 DEBUG:root:        - Running native script "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh" from temporal script "tmp-run.sh" in "/home/cmuser/CM/repos/local/cache/a164a48590d34909" ...
1.220 INFO:root:             ! cd /home/cmuser/CM/repos/local/cache/a164a48590d34909
1.220 INFO:root:             ! call /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
1.254 INFO:root:             ! call "postprocess" from /home/cmuser/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
1.254 DEBUG:root:        - Running postprocess ...
1.282 INFO:root:        - running time of script "detect-os,detect,os,info": 0.10 sec.
1.283 DEBUG:root:      - Processing env after dependencies ...
1.285 DEBUG:root:      - Running preprocess ...
1.325 DEBUG:root:      - Running native script "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-git-repo/run.sh" from temporal script "tmp-run.sh" in "/home/cmuser/CM/repos/local/cache/a164a48590d34909" ...
1.325 INFO:root:           ! cd /home/cmuser/CM/repos/local/cache/a164a48590d34909
1.325 INFO:root:           ! call /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-git-repo/run.sh from tmp-run.sh
1.333 /home/cmuser/CM/repos/local/cache/a164a48590d34909
1.335 ******************************************************
1.335 Current directory: /home/cmuser/CM/repos/local/cache/a164a48590d34909
1.335
1.335 Cloning inference from https://github.com/mlcommons/inference
1.335
1.335 git clone  -b master https://github.com/mlcommons/inference --depth 5 inference
1.335
1.338 Cloning into 'inference'...
90.12 fatal: the remote end hung up unexpectedly
90.12 fatal: early EOF
90.13 fatal: index-pack failed
90.14
90.14 CM error: Portable CM script failed (name = get-git-repo, return code = 256)
90.14
90.14
90.14 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
90.14 Note that it is often a portability issue of a third-party tool or a native script
90.14 wrapped and unified by this CM script (automation recipe). Please re-run
90.14 this script with --repro flag and report this issue with the original
90.14 command line, cm-repro directory and full log here:
90.14
90.14 https://github.com/mlcommons/cm4mlops/issues
90.14
90.14 The CM concept is to collaboratively fix such issues inside portable CM scripts
90.14 to make existing tools and native scripts more portable, interoperable
90.14 and deterministic. Thank you!
90.14         Detected version: 3.8.10
90.14         Detected version: 3.8.10
------

 1 warning found (use docker --debug to expand):
 - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "CM_GH_TOKEN") (line 14)
mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public.Dockerfile:47
--------------------
  45 |
  46 |     # Run commands
  47 | >>> RUN cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.CM_RUN_STATE_DOCKER=True
  48 |
--------------------
ERROR: failed to solve: process "/bin/bash -c cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.CM_RUN_STATE_DOCKER=True" did not complete successfully: exit code: 2

CM error: Portable CM script failed (name = build-docker-image, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note that it is often a portability issue of a third-party tool or a native script
wrapped and unified by this CM script (automation recipe). Please re-run
this script with --repro flag and report this issue with the original
command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts
to make existing tools and native scripts more portable, interoperable
and deterministic. Thank you!
(cm) fuseadm@fuseadm-ADDC-KBL-DDR4:~$
```



yes, git clone is failing here. Pytorch clone has recursive git clones for multiple submodules. Is this issue happening repeatedly? It has nothing to do with the workflow, We have also once faced this issue and it was alright after a few hours. 

well I seem to have made it further...after almost 8hours to build pytorch, this is where it died...and of course it didn;t leave in a docker shell

```
27350.0 adding 'torchgen/static_runtime/gen_static_runtime_ops.py'
27350.0 adding 'torchgen/static_runtime/generator.py'
27350.0 adding 'torch-2.1.0a0+git32f93b1.dist-info/LICENSE'
27350.0 adding 'torch-2.1.0a0+git32f93b1.dist-info/METADATA'
27350.0 adding 'torch-2.1.0a0+git32f93b1.dist-info/NOTICE'
27350.0 adding 'torch-2.1.0a0+git32f93b1.dist-info/WHEEL'
27350.0 adding 'torch-2.1.0a0+git32f93b1.dist-info/entry_points.txt'
27350.0 adding 'torch-2.1.0a0+git32f93b1.dist-info/top_level.txt'
27350.0 adding 'torch-2.1.0a0+git32f93b1.dist-info/RECORD'
27350.0 removing build/bdist.linux-x86_64/wheel
27350.0 Processing ./torch-2.1.0a0+git32f93b1-cp38-cp38-linux_x86_64.whl
27350.0 Requirement already satisfied: networkx in /home/cmuser/.local/lib/python3.8/site-packages (from torch==2.1.0a0+git32f93b1) (3.1)
27350.0 Requirement already satisfied: fsspec in /home/cmuser/.local/lib/python3.8/site-packages (from torch==2.1.0a0+git32f93b1) (2024.6.1)
27350.0 Requirement already satisfied: filelock in /home/cmuser/.local/lib/python3.8/site-packages (from torch==2.1.0a0+git32f93b1) (3.15.4)
27350.0 Requirement already satisfied: jinja2 in /home/cmuser/.local/lib/python3.8/site-packages (from torch==2.1.0a0+git32f93b1) (3.1.4)
27350.0 Requirement already satisfied: sympy in /home/cmuser/.local/lib/python3.8/site-packages (from torch==2.1.0a0+git32f93b1) (1.13.2)
27350.0 Requirement already satisfied: typing-extensions in /home/cmuser/.local/lib/python3.8/site-packages (from torch==2.1.0a0+git32f93b1) (4.12.2)
27350.0 Requirement already satisfied: MarkupSafe>=2.0 in /home/cmuser/.local/lib/python3.8/site-packages (from jinja2->torch==2.1.0a0+git32f93b1) (2.1.5)
27350.0 Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/cmuser/.local/lib/python3.8/site-packages (from sympy->torch==2.1.0a0+git32f93b1) (1.3.0)
27350.0 Installing collected packages: torch
27350.0 Successfully installed torch-2.1.0a0+git32f93b1
27350.0 /home/cmuser/CM/repos/local/cache/72b1835f2b114d10
27350.0 ******************************************************
27350.0 Current directory: /home/cmuser/CM/repos/local/cache/72b1835f2b114d10
27350.0
27350.0 Cloning vision from https://github.com/pytorch/vision
27350.0
27350.0 git clone  --recurse-submodules https://github.com/pytorch/vision  repo
27350.0
27350.0
27350.0           Detected version: 3.25.1
27350.0             Detected version: 2.13.4
27350.0             Detected version: 4.10.0.84
27350.0
27350.0             Extra PIP CMD:  --upgrade --default-timeout=900 --extra-index-url https://developer.download.nvidia.com/compute/redist
27350.0
27350.0             Detected version: 1.40.0
27350.0               Detected version: 1.23.5
27350.0
27350.0               Extra PIP CMD:
27350.0
27350.0               Detected version: 1.22.4
27350.0
27350.0               Extra PIP CMD:
27350.0
27350.0               Detected version: 3.1
------

 1 warning found (use docker --debug to expand):
 - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "CM_GH_TOKEN") (line 14)
mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public.Dockerfile:47
--------------------
  45 |
  46 |     # Run commands
  47 | >>> RUN cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.CM_RUN_STATE_DOCKER=True
  48 |
--------------------
ERROR: failed to solve: process "/bin/bash -c cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.CM_RUN_STATE_DOCKER=True" did not complete successfully: exit code: 2

CM error: Portable CM script failed (name = build-docker-image, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note that it is often a portability issue of a third-party tool or a native script
wrapped and unified by this CM script (automation recipe). Please re-run
this script with --repro flag and report this issue with the original
command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts
to make existing tools and native scripts more portable, interoperable
and deterministic. Thank you!
(cm) fuseadm@fuseadm-ADDC-KBL-DDR4:~$
```



it might actually be running...I started the command again last night and this morning here are the last few lines of output:

```
[08/23/2024-06:30:31] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +7, GPU +8, now: CPU 2477, GPU 516 (MiB)
[08/23/2024-06:30:31] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 2478, GPU 526 (MiB)
[08/23/2024-06:30:31] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
```

nvidia-smi on the host is returning an error:  ```Unable to determine the device handle for GPU0000:02:00.0: Unknown Error```
I assume this is because the GPU has been given to the container?

ps ax on the host shows:
```
fuseadm@fuseadm-ADDC-KBL-DDR4:~$ ps ax | grep pts/0
 591521 pts/0    Ss+    0:00 bash -c (cm pull repo && cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_TMP_CURRENT_PATH=/home/fuseadm --env.CM_TMP_PIP_VERSION_STRING= --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True  --env.CM_DATASET_IMAGENET_PATH=/home/cmuser/CM/repos/local/cache/c7ba5ae996334777/imagenet-2012-val  --env.CM_MLPERF_INFERENCE_RESULTS_DIR=/home/cmuser/CM/repos/local/cache/ac37f504c62f496b  --env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/home/cmuser/CM/repos/local/cache/d13a9436c2df4e5c/mlperf-inference-submission  --env.MLPERF_SCRATCH_PATH=/home/cmuser/CM/repos/local/cache/d4f6158f8d5a4ebe     && bash ) || bash
 591567 pts/0    S+     0:00 bash -c (cm pull repo && cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_TMP_CURRENT_PATH=/home/fuseadm --env.CM_TMP_PIP_VERSION_STRING= --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True  --env.CM_DATASET_IMAGENET_PATH=/home/cmuser/CM/repos/local/cache/c7ba5ae996334777/imagenet-2012-val  --env.CM_MLPERF_INFERENCE_RESULTS_DIR=/home/cmuser/CM/repos/local/cache/ac37f504c62f496b  --env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/home/cmuser/CM/repos/local/cache/d13a9436c2df4e5c/mlperf-inference-submission  --env.MLPERF_SCRATCH_PATH=/home/cmuser/CM/repos/local/cache/d4f6158f8d5a4ebe     && bash ) || bash
 591578 pts/0    S+     0:07 /usr/bin/python3 /home/cmuser/.local/bin/cm run script --tags=app,mlperf,inference,generic,_nvidia,_resnet50,_tensorrt,_cuda,_test,_r4.1_default,_offline --quiet=true --env.CM_QUIET=yes --env.CM_MLPERF_IMPLEMENTATION=nvidia --env.CM_MLPERF_MODEL=resnet50 --env.CM_MLPERF_RUN_STYLE=test --env.CM_MLPERF_LOADGEN_MAX_BATCHSIZE=64 --env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.CM_MLPERF_DEVICE=cuda --env.CM_MLPERF_USE_DOCKER=True --env.CM_MLPERF_BACKEND=tensorrt --env.CM_MLPERF_LOADGEN_SCENARIO=Offline --env.CM_TEST_QUERY_COUNT=1000 --env.CM_MLPERF_FIND_PERFORMANCE_MODE=yes --env.CM_MLPERF_LOADGEN_ALL_MODES=no --env.CM_MLPERF_LOADGEN_MODE=performance --env.CM_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.CM_MLPERF_SKIP_SUBMISSION_GENERATION=yes --env.CM_MLPERF_INFERENCE_VERSION=4.1 --env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1_default --env.CM_MLPERF_LAST_RELEASE=v4.1 --env.CM_SUT_DESC_CACHE=no --env.CM_TMP_CURRENT_PATH=/home/fuseadm --env.CM_TMP_PIP_VERSION_STRING= --env.CM_SUT_META_EXISTS=yes --env.CM_MODEL=resnet50 --env.CM_MLPERF_LOADGEN_COMPLIANCE=no --env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline --env.CM_MLPERF_LOADGEN_MODES,=performance --env.CM_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.mlperf-inference-implementation.tags=_batch_size.64 --v=False --print_env=False --print_deps=False --dump_version_info=True --env.CM_DATASET_IMAGENET_PATH=/home/cmuser/CM/repos/local/cache/c7ba5ae996334777/imagenet-2012-val --env.CM_MLPERF_INFERENCE_RESULTS_DIR=/home/cmuser/CM/repos/local/cache/ac37f504c62f496b --env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/home/cmuser/CM/repos/local/cache/d13a9436c2df4e5c/mlperf-inference-submission --env.MLPERF_SCRATCH_PATH=/home/cmuser/CM/repos/local/cache/d4f6158f8d5a4ebe
 596638 ?        S      0:04 sshd: fuseadm@pts/0
 596639 pts/0    Ss     0:00 -bash
 596646 pts/0    S+     0:00 screen -r
 597203 pts/0    S+     0:00 sh -c bash -c ". ./tmp-run.sh"
 597204 pts/0    S+     0:00 bash -c . ./tmp-run.sh
 597205 pts/0    S+     0:00 make generate_engines RUN_ARGS= --benchmarks=resnet50 --scenarios=offline  --test_mode=PerformanceOnly  --offline_expected_qps=1 --gpu_batch_size=64 --no_audit_verify
 597412 pts/0    Sl+    0:02 python3.8 -m code.main --benchmarks=resnet50 --scenarios=offline --test_mode=PerformanceOnly --offline_expected_qps=1 --gpu_batch_size=64 --no_audit_verify --action=generate_engines
 597440 pts/0    S+     0:00 /usr/bin/python3.8 -c from multiprocessing.resource_tracker import main;main(28)
 597441 pts/0    Sl+    0:09 /usr/bin/python3.8 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=29, pipe_handle=31) --multiprocessing-fork
```

Is there anyway to attach to the running container and run nvidia-smi to watch gpu usage?

and just to add to one question from above...the git clone has failed repeatedly.  It didn't fail yesterday during the day, but did during the evening, and many times over the course of the previous 2 weeks 

I think the container is finally running, but I think either the test is crashing the driver, or I'm blowing past memory, or I have a hardware problem

```
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.896209] usb 1-2: USB disconnect, device number 2
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.896217] usb 1-2.1: USB disconnect, device number 3
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.896314] pcieport 0000:00:1c.0: AER: Multiple Correctable error message received from 0000:00:1c.0
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946350] NVRM: GPU at PCI:0000:02:00: GPU-f8229d43-6bf4-cff1-fda6-97c1a946d97a
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946358] NVRM: Xid (PCI:0000:02:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946362] pcieport 0000:00:1c.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946365] NVRM: GPU 0000:02:00.0: GPU has fallen off the bus.
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946368] pcieport 0000:00:1c.0:   device [8086:9d10] error status/mask=00002001/00002000
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946374] pcieport 0000:00:1c.0:    [ 0] RxErr                  (First)
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946381] NVRM: A GPU crash dump has been created. If possible, please run
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946381] NVRM: nvidia-bug-report.sh as root to collect this data before
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946381] NVRM: the NVIDIA kernel module is unloaded.
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946384] pcieport 0000:00:1c.0: AER: Multiple Correctable error message received from 0000:00:1c.0
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946405] pcieport 0000:00:1c.0: AER: found no error details for 0000:00:1c.0
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946410] pcieport 0000:00:1c.0: AER: Multiple Correctable error message received from 0000:00:1c.0
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946431] pcieport 0000:00:1c.0: AER: found no error details for 0000:00:1c.0
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946435] pcieport 0000:00:1c.0: AER: Multiple Correctable error message received from 0000:00:1c.0
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946457] pcieport 0000:00:1c.0: AER: found no error details for 0000:00:1c.0
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946462] pcieport 0000:00:1c.0: AER: Multiple Correctable error message received from 0000:00:1c.0
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946461] pcieport 0000:00:1c.1: AER: Correctable error message received from 0000:00:1c.1
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946475] pcieport 0000:00:1c.1: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946480] pcieport 0000:00:1c.1:   device [8086:9d11] error status/mask=00000001/00002000
Aug 23 11:18:28 fuseadm-ADDC-KBL-DDR4 kernel: [ 1013.946486] pcieport 0000:00:1c.1:    [ 0] RxErr                  (First)

```

Yes, that looks like a hardware issue.

For attaching the docker you can do

```docker ps -a```

get the container id. And then do
```docker attach ID ```