Repository: mlcommons/inference
Issue: Resnet50 MLPerf 4.1  Attribute Error: ResNet50EngineBuilder object has no attribute calibrator
Root cause: Validation Gap

We are a test team her trying to run MLPerf on a server with Nvidia GPUs / Cuda for a certification run. We are using the following script on Rocky Linux:

# Into container
cd ~/NVIDIA

make prebuild


# Enviroment setting in container
export USER=$(whoami)
sudo usermod -aG root $(whoami)
sudo chown -R $(whoami):$(whoami) /work
sudo chown -R $(whoami):$(whoami) ~/scratch


# Install Python packages in container
pip install opencv-python-headless mlperf_logging simplejson


# Download model
make download_model BENCHMARKS="resnet50"


# Download Dataset
make download_data BENCHMARKS="resnet50"
make preprocess_data BENCHMARKS="resnet50"


# Run Benchmark
make run RUN_ARGS="--benchmarks=resnet50 --scenarios=offline,server"

The run gives the error mentioned in the headline.

Resnet50 MLPerf 4.1  Attribute Error: ResNet50EngineBuilder object has no attribute calibrator

Any help appreciated, thank you.


---------------- added: From the terminal: ----------------

(mlperf) s370@mlperf-inference-s370-x86-64-21773:/work$ make run RUN_ARGS="--benchmarks=resnet50 --scenarios=offline,server"
make[1]: Entering directory '/work'
[2025-02-21 12:33:17,741 main.py:229 INFO] Detected system ID: KnownSystem.S370
[2025-02-21 12:33:25,598 generate_engines.py:171 INFO] Building engines for resnet50 benchmark in Offline scenario...
[2025-02-21 12:33:25,602 builder.py:460 WARNING] No batch_size dict provided for ResNet50EngineBuilderOp. Setting to default value {<ResNet50Component.ResNet50: AliasedName(name='resnet50', aliases=('resnet',), patterns=())>: 1}
[02/21/2025-12:33:25] [TRT] [I] [MemUsageChange] Init CUDA: CPU +22, GPU +0, now: CPU 40, GPU 446 (MiB)
[02/21/2025-12:33:27] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +2170, GPU +414, now: CPU 2363, GPU 860 (MiB)
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/work/code/actionhandler/base.py", line 189, in subprocess_target
    return self.action_handler.handle()
  File "/work/code/actionhandler/generate_engines.py", line 174, in handle
    total_engine_build_time += self.build_engine(job)
  File "/work/code/actionhandler/generate_engines.py", line 165, in build_engine
    builder.build_engines()
  File "/usr/local/lib/python3.10/dist-packages/nvmitten/nvidia/builder.py", line 570, in build_engines
    self.mitten_builder.run(self.legacy_scratch, None)
  File "/usr/local/lib/python3.10/dist-packages/nvmitten/debug/debug_manager.py", line 256, in _wrapper
    raise exc_info[1]
  File "/usr/local/lib/python3.10/dist-packages/nvmitten/debug/debug_manager.py", line 243, in _wrapper
    retval = obj(*args, **kwargs)
  File "/work/code/resnet50/tensorrt/builder.py", line 472, in run
    builder_config = builder.create_builder_config()
  File "/work/code/resnet50/tensorrt/builder.py", line 258, in create_builder_config
    builder_config.int8_calibrator = self.calibrator
AttributeError: 'ResNet50EngineBuilder' object has no attribute 'calibrator'
[2025-02-21 12:33:31,286 generate_engines.py:171 INFO] Building engines for resnet50 benchmark in Offline scenario...
[2025-02-21 12:33:31,287 builder.py:460 WARNING] No batch_size dict provided for ResNet50EngineBuilderOp. Setting to default value {<ResNet50Component.ResNet50: AliasedName(name='resnet50', aliases=('resnet',), patterns=())>: 1}
[02/21/2025-12:33:31] [TRT] [I] [MemUsageChange] Init CUDA: CPU +21, GPU +0, now: CPU 40, GPU 446 (MiB)
[02/21/2025-12:33:33] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +2170, GPU +414, now: CPU 2363, GPU 860 (MiB)
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/work/code/actionhandler/base.py", line 189, in subprocess_target
    return self.action_handler.handle()
  File "/work/code/actionhandler/generate_engines.py", line 174, in handle
    total_engine_build_time += self.build_engine(job)
  File "/work/code/actionhandler/generate_engines.py", line 165, in build_engine
    builder.build_engines()
  File "/usr/local/lib/python3.10/dist-packages/nvmitten/nvidia/builder.py", line 570, in build_engines
    self.mitten_builder.run(self.legacy_scratch, None)
  File "/usr/local/lib/python3.10/dist-packages/nvmitten/debug/debug_manager.py", line 256, in _wrapper
    raise exc_info[1]
  File "/usr/local/lib/python3.10/dist-packages/nvmitten/debug/debug_manager.py", line 243, in _wrapper
    retval = obj(*args, **kwargs)
  File "/work/code/resnet50/tensorrt/builder.py", line 472, in run
    builder_config = builder.create_builder_config()
  File "/work/code/resnet50/tensorrt/builder.py", line 258, in create_builder_config
    builder_config.int8_calibrator = self.calibrator
AttributeError: 'ResNet50EngineBuilder' object has no attribute 'calibrator'
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/work/code/main.py", line 231, in <module>
    main(main_args, DETECTED_SYSTEM)
  File "/work/code/main.py", line 144, in main
    dispatch_action(main_args, config_dict, workload_setting)
  File "/work/code/main.py", line 202, in dispatch_action
    handler.run()
  File "/work/code/actionhandler/base.py", line 82, in run
    self.handle_failure()
  File "/work/code/actionhandler/base.py", line 186, in handle_failure
    self.action_handler.handle_failure()
  File "/work/code/actionhandler/generate_engines.py", line 182, in handle_failure
    raise RuntimeError("Building engines failed!")
RuntimeError: Building engines failed!
make[1]: *** [Makefile:37: generate_engines] Error 1
make[1]: Leaving directory '/work'
make: *** [Makefile:31: run] Error 2