Repository: mlcommons/inference
Issue: CM script failed to run harness after docker done
Root cause: Environment Incompatibility

Hi @arjunsuresh 

**I am running the Resnet50 benchmark with the command:**

cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=5000

**failed to run harness as below and the docker was created successfully. How to resolve it?**
[log with dock done.txt](https://github.com/user-attachments/files/18239125/log.with.dock.done.txt)


<pre>make: *** [Makefile:45: run_harness] Error 1

CM error: Portable CM script failed (name = benchmark-program, return code = 512)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note that it is often a portability issue of a third-party tool or a native script
wrapped and unified by this CM script (automation recipe). Please re-run
this script with --repro flag and report this issue with the original
command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts
to make existing tools and native scripts more portable, interoperable
and deterministic. Thank you!
cmuser@9951fc73ce5b:~$ pwd
/home/cmuser
</pre>

Looks like you are having GPUs from different generation on the system. AFAIK this is not supported by Nvidia implementation. 


```
{GPU(name='NVIDIA RTX A6000', accelerator_type=<AcceleratorType.Discrete: AliasedName(name='Discrete', aliases=(), patterns=())>, vram=Memory(quantity=47.98828125, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=51527024640), max_power_limit=300.0, pci_id='0x223010DE', compute_sm=86): 1, GPU(name='NVIDIA T1000 8GB', accelerator_type=<AcceleratorType.Discrete: AliasedName(name='Discrete', aliases=(), patterns=())>, vram=Memory(quantity=8.0, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=8589934592), max_power_limit=50.0, pci_id='0x1FF010DE', compute_sm=75): 1})), numa_conf=None, system_id='Nvidia_9951fc73ce5b')
```

Thank you @arjunsuresh.

No same error prompt out after I removed T1000 and only kept A6000 in the system.

But when I run the below commands in the docker that is the same docker created by CM before system reboot:
 
cm run script --tags=run-mlperf,inference,_r4.1-dev \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --quiet

Another error occurred:
[log_in_docker_resnet50_1.txt](https://github.com/user-attachments/files/18244016/log_in_docker_resnet50_1.txt)


<pre>Traceback (most recent call last):
  File &quot;/home/cmuser/CM/repos/local/cache/1406981516ca4974/inference/vision/classification_and_detection/tools/accuracy-imagenet.py&quot;, line 89, in &lt;module&gt;
    main()
  File &quot;/home/cmuser/CM/repos/local/cache/1406981516ca4974/inference/vision/classification_and_detection/tools/accuracy-imagenet.py&quot;, line 54, in main
    with open(args.mlperf_accuracy_file, &quot;r&quot;) as f:
FileNotFoundError: [Errno 2] No such file or directory: &apos;/home/cmuser/CM/repos/local/cache/d54d1a74ced144d0/valid_results/9951fc73ce5b-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/offline/accuracy/mlperf_log_accuracy.json&apos;

CM error: Portable CM script failed (name = process-mlperf-accuracy, return code = 256)
</pre>