Repository: mlcommons/inference
Issue: [Vision - 3DUnet with Kits19] Accuracy reported in Readme
Root cause: Algorithmic Error

We observed that there is a switch in the order of accuracy values reported in the Readme here: https://github.com/mlcommons/inference/tree/master/vision/medical_imaging/3d-unet-kits19 

This is what is reported (Ref: https://github.com/mlcommons/inference/tree/master/vision/medical_imaging/3d-unet-kits19#supported-frameworks) :
ONNX	mean = 0.86330 kidney = 0.9342 tumor = 0.7924

What we observe with the ONNX_SUT is:
ONNX Accuracy: mean = 0.93418, kidney = 0.7924, tumor = 0.8633

This needs to be fixed to provide clarity on what are the right accuracy values to compare against.
























@nv-jinhosuh 

Hi @karkadad 
Would you please let me know how you obtained the accuracy of your ONNX_SUT? 
Is that what's printed out from `make run_onnxruntime_accuracy`? The command will also make the CSV file:  `build/postprocessed_data/summary.csv` that shows each case's accuracy on Kidney/Tumor segmentation and the mean(composite) of the two. I would like to double check, if you see the match between what's printed out and the CSV file.

FYI, this is what I got after I just ran the ONNX runtime accuracy:
```
...
Checking accuracy...
Loading necessary metadata...
Loading loadgen accuracy log...
Running postprocessing...
Saving predictions...
Running evaluation...
Processing evaluation summary...
Accuracy: mean = 0.86330, kidney = 0.9342, tumor = 0.7924
...
```

I used the run script: ` python run.py --backend=onnxruntime --scenario=Offline --accuracy --model=build/model/3dunet_kits19_128x128x128.onnx ` 
This is the result: 
`
Constructing SUT...
Finished constructing SUT.
Constructing QSL...
Found 42 preprocessed files
Using performance count = 16
Finished constructing QSL.
Loading ONNX model...
Running Loadgen test...
Loading file case_00066
Loading file case_00169
Loading file case_00171
Loading file case_00111
Loading file case_00125
Loading file case_00162
Loading file case_00000
Loading file case_00024
Loading file case_00203
Loading file case_00087
Loading file case_00049
Loading file case_00084
Loading file case_00160
Loading file case_00198
Loading file case_00086
Loading file case_00161
...
Loading file case_00185
Loading file case_00006
Loading file case_00187
Loading file case_00138
Loading file case_00056
Loading file case_00207
Loading file case_00061
Loading file case_00092
Loading file case_00044
Loading file case_00078
Loading file case_00034
Loading file case_00003
Loading file case_00128
Loading file case_00080
Loading file case_00112
Loading file case_00076
...
Loading file case_00157
Loading file case_00189
Loading file case_00176
Loading file case_00012
Loading file case_00041
Loading file case_00052
Loading file case_00206
Loading file case_00065
Loading file case_00070
Loading file case_00005
...
No errors encountered during test.
Checking accuracy...
Loading necessary metadata...
Loading loadgen accuracy log...
Running postprocessing...
Saving predictions...
Running evaluation...
Processing evaluation summary...
Accuracy: mean = 0.93418, kidney = 0.7924, tumor = 0.8633
Done!
Done!
Destroying SUT...
Destroying QSL...
`
This is after setup and preprocessing of the data.

@karkadad  I think I now understand what has happened. Looks like the PANDAS dataframe order might not be deterministic (maybe as per version, as functionality wise it's still valid; in my case it's always does alphanumeric sorting, making an order of composite, kidney, tumor) as the columns added. I believe your CSV column order would look different than mine. 

Let me add a fix to make this deterministic. 

@karkadad Would you please check the above PR to see if you now have consistent printouts?

Thank you @nv-jinhosuh ! I verified that the patch fixes the issue, closing this ticket.