Repository: mlcommons/inference
Issue: DLRM remaining development tasks
Root cause: Documentation Deficiency

1. Need to update Readme on the calibration dataset specifics.
   --use first 128k pairs in the second half of day 23. Test set is first half of day 23. 
2. Need a standalone script to calculate accuracy from the loadgen output. 
    -- example: https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#validate-accuracy-for-resnet50-and-mobilenet-benchmarks
@mnaumovfb 

Keeping @nvpohanh  in the loop.

I can update the README with instructions about calibration.

Why can't we just get the accuracy from the [run](https://github.com/mlperf/inference/blob/master/v0.5/recommendation/python/criteo.py#L339) and need to reconstruct it from the logfile?

I think the reason is that people prefer to have separate performance runs and accuracy runs. During the performance runs, the logging is turned off so that we can get the maximum throughput etc. We do not time the accuracy checking steps. Adding @nvpohanh if I miss out any other reason. 

The main reason is that some submitters are not running in these frameworks. It would be great to have a script which takes in the LoadGen accuracy JSON log (which is common across all submitters) and compute the accuracy from that.

Here are some of the examples for other benchmarks:
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-coco.py
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py
https://github.com/mlperf/inference/blob/master/v0.7/medical_imaging/3d-unet/brats_eval.py
https://github.com/mlperf/inference/blob/master/v0.7/language/bert/squad_eval.py
Please let me know if you need help with this. Thanks!

Specifically, we need a script which computes accuracy from `mlperf_log_accuracy.json` generated by LoadGen in AccuracyOnly mode

I have been thinking of how to implement an equivalent to [accuracy-imagenet.py](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py) script for DLRM.

This script relies on the log file `mlperf_log_accuracy.json`, which stores the indices of samples issued by loadgen and some hexadecimal data associated with them. I have a few questions about this log file:
1. What is the format of the hexadecimal data? Can you point me to the lines of code that encodes it? 
2. It seems that the prediction (floating point value) associated with the sample is stored as this hexadecimal data https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py#L60
However, in DLRM this is only the starting position of the query which involves several samples. Where can I find the prediction values of the remaining samples in the query? 

This is based on array.array: https://docs.python.org/3/library/array.html, basically give it an array of type T and it will give you a byte buffer. You pass that byte buffer back to loadgen which will serialize it into the hex format.
The validation script does the reverse - use array.fromhex() and numpy to decode the hex format and you get the array of T back.
Kind of a contract with yourself.
The code that calls array.array after you get the inference results this is here:
https://github.com/mlperf/inference/blob/master/v0.5/recommendation/python/main.py#L253

If you have 1 result per queryid - that is easy. If you have multiple results for a queryid the easiest would be to use a rank 2 numpy array and pass that to array.array.

Thank you @guschmue, this is very helpful. 

So, my understanding that in principle I could write both the labels and the targets into this hexadecimal string, and then compare between them, without having to touch the data set again. Is that correct?

I have implemented the script in the following PR https://github.com/mlperf/inference/pull/649

Please take a look and merge when you have a chance.

yeap, looks good to me.

@nvpohanh still need readme update:
Need to update Readme on the calibration dataset specifics.
--use first 128k pairs in the second half of day 23. Test set is first half of day 23.

See #654 for calibration set specification

I have checked the PR. It looks good to me.

done