Repository: mlcommons/inference
Issue: ResNet50 v5.0dev error. Multiple variation tags selected

Hello everyone, I recently encountered an error.


When trying to run ResNet50 after some time the message below
( https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_49_1 )
``` 
Traceback (most recent call last):
  File "/home/mlcuser/.local/bin/mlcr", line 8, in <module>
    sys.exit(mlcr())
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 86, in mlcr
    main()
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 273, in main
    res = method(run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 309, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
    r = self._run(i)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1784, in _run
    r = customize_code.preprocess(ii)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 285, in preprocess
    r = mlc.access(ii)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 309, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
    r = self._run(i)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1858, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3323, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3496, in _run_deps
    r = self.action_object.access(ii)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 309, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
    r = self._run(i)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1643, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3323, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3496, in _run_deps
    r = self.action_object.access(ii)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 309, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
    r = self._run(i)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1902, in _run
    r = self._run_deps(post_deps, clean_env_keys_post_deps, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3496, in _run_deps
    r = self.action_object.access(ii)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 309, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
    r = self._run(i)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1643, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3323, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3496, in _run_deps
    r = self.action_object.access(ii)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 309, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
    r = self._run(i)
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1643, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3323, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3496, in _run_deps
    r = self.action_object.access(ii)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 309, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 244, in call_script_module_function
    raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : Multiple variation tags selected for the variation group "repo-branch": {'code-only-for-v5.0', 'code-only'} 
```

@arjunsuresh please help

Hi @Agalakdak , could you also share some log messages from MLCFlow before this error message?

Hi @anandhu-eng! Certainly !

[log_resnet50.txt](https://github.com/user-attachments/files/21287791/log_resnet50.txt)

Hi @Agalakdak , Im testing the fix. I noted that you are running v5.0-dev. I wanted to inform you that we now support 5.1-dev for resnet50, retinanet and 3d-unet which takes code from 5.0 submission round. I shall update the inference documentation ASAP.

Could you try the following command:

```
mlcr run-mlperf,inference,_find-performance,_full,_r5.1-dev    --model=resnet50    --implementation=nvidia    --framework=tensorrt    --category=edge    --scenario=Offline    --execution_mode=test    --device=cuda     --docker --quiet    --test_query_count=5000 --rerun
```

@anandhu-eng thanks for the quick reply! I'll try but it will take some time. I'll try to write back with the results within a couple of hours.

@anandhu-eng sorry, but I got an error.
Most likely, my 730 GB was not enough to deploy the container.
Are there any specific requirements or recommendations for the amount of storage I can add to the server?

[log_resnet51_v5.1.txt](https://github.com/user-attachments/files/21293027/log_resnet51_v5.1.txt)



I believe 730 GB should be sufficient for running ResNet50. However, previously built containers and images might be consuming a significant amount of disk space.
Assuming there are no critical containers that need to remain active, you can stop all running containers and perform a cleanup using Docker's prune commands to free up space.

```
docker stop $(docker ps -q)
```

```
docker system prune -a
```

@anandhu-eng I continue working on running tests.
I previously created a bug report here, but it seems that the problem was the connection between docker and the video card. At the moment, I was able to solve it. But now it seems to me that the problem is some incompatibility of H100 with mlcr.
Were mlcr tests performed on H100?
Does the test depend on the version of CUDA, nvidia-drivers?

Hi @Agalakdak , it has been some time since we tested on H100. It would be really helpful if you could share some logs regarding the issue you are facing currently.

Meanwhile, let me try to get a H100 instance to test.

Adding @arjunsuresh to the loop for awareness

@anandhu-eng 

Ok, I'll try to run other networks and post their logs later. By the way, when running the command
```
mlcr run-mlperf,inference,_find-performance,_full,_r5.1-dev    
  --model=resnet50    
  --implementation=nvidia    
  --framework=tensorrt    
  --category=edge    
  --scenario=Offline    
  --execution_mode=test    
  --device=cuda     
  --docker 
  --quiet    
  --test_query_count=5000 
  --rerun
```
I was rapidly running out of space (so much so that 730 GB was not enough =) ). Having looked at where it was "leaking", it turned out that 99.8% was occupied by files /var/lib/docker/vfs/dir
I do not exclude that this may be an individual problem of the system being investigated.

## Version Tag Issue Clarification

The error occurs because `_r5.0dev` is not a valid version tag. Based on the MLCommons README, here are the correct version options:

### Valid Version Tags:
- **`_r5.0`** - Stable v5.0 (current, submission deadline Feb 28, 2025)
- **`_r5.1-dev`** - Development v5.1 (upcoming, submission deadline July 25, 2025)
- **`_r4.1`** - Previous stable release

### The Issue:
`_r5.0dev` causes conflict between variation tags:
- `code-only-for-v5.0` (v5.0 specific)
- `code-only` (generic)

### Solution:
```bash
# Use stable v5.0
mlcr run-mlperf,inference,_find-performance,_full,_r5.0 --model=resnet50 ...

# OR use development v5.1 (as suggested by @anandhu-eng)
mlcr run-mlperf,inference,_find-performance,_full,_r5.1-dev --model=resnet50 ...
```

### ResNet50 Support:
✅ ResNet50 is supported in both v5.0 and v5.1-dev for edge/datacenter categories
✅ Your H100 GPU is fully capable of running this benchmark

### Recommendation:
Use `_r5.1-dev` as suggested, since it's the latest development version with all current improvements.

**Note**: Always refer to the [official README](https://github.com/mlcommons/inference#readme) for valid version tags to avoid variation conflicts.

**Background**: Platform Technology Strategy | Repository Version Management

Hi @anandhu-eng 
Sorry for the long answer. There were some system problems with the server and it was sent to the bulkhead.

1) The problem with the huge consumption of space turned out to be local. It was resolved after completely removing the software and installing everything from 0.
2) Unfortunately, I had time to run only 2 of the tests. I am attaching the log below. Currently, 2x H100 are installed in the server.

[21_07_resnet50_v5.1.txt](https://github.com/user-attachments/files/21347295/21_07_resnet50_v5.1.txt)

[bert99_v5.1.txt](https://github.com/user-attachments/files/21347712/bert99_v5.1.txt)



Hi @Agalakdak , unfortunately the error is not reproducible on my end. Good news is, I have fixed the env variable where the backup url is being assigned in [this](https://github.com/mlcommons/mlperf-automations/pull/526) PR. Once merged, please try rebuild the docker with the flags: `--docker_rebuild --docker_cache=no`. This will try to download from the alternate download link.


Also to save some disk space, you could add an additional tag: `--env.MLC_USE_DATASET_FROM_HOST=yes`. This will download the dataset in host and mount it to the docker container which prevents repeated download while rebuilding.

Regarding Bert, we are yet to support Bert in v5.1-dev(NVIDIA code from. v5.0). We currently support `retinanet`,  `resnet50` and `3d-unet`.

@anandhu-eng 
Thank you for your help! It seems that the problem was in the server itself, because after replacing the RAM modules, the problem disappeared and I was able to get results for resnet50 v5.1 with 2x H100 
Thanks again!

```
+----------+----------+----------+------------+-----------------+
|  Model   | Scenario | Accuracy | Throughput | Latency (in ms) |
+----------+----------+----------+------------+-----------------+
| resnet50 | Offline  |  76.078  | X 114039.0 |        -        |
+----------+----------+----------+------------+-----------------+
```



Glad to see this is resolved @Agalakdak! Those ResNet50 results look solid - 114k samples/sec on 2x H100 is great performance.

Really interesting that it turned out to be faulty RAM causing the space consumption issues and errors. Hardware problems can definitely manifest in unexpected ways - good catch on identifying and replacing those modules!

Thanks for sharing the resolution - this will help others who might encounter similar hard-to-reproduce issues to consider checking their hardware too.