Repository: mlcommons/inference
Issue: DLRM: why good_item = 0 && QPS is only 19?
Root cause: Validation Gap

I ran DLRM on Terabyte with a subsampling rate of 0.875. I downloaded the right weight, and dataset, and checked the md5sum.
However, I found we `results.json` weird:
```json
{
    "TestScenario.Server": {
        "count": 5880,
        "good_items": 0,
        "mean": 155.64582431972076,
        "percentiles": {
            "50.0": 155.21273523399998,
            "80.0": 248.65285731739996,
            "90.0": 277.3749345456,
            "95.0": 293.0874136949,
            "99.0": 303.59466438779003,
            "99.9": 304.681520337456
        },
        "qps": 19.07188202477713,
        "took": 308.3072762489319,
        "total_items": 5880
    },
    "cmdline": "Namespace(accuracy=False, backend='pytorch-native', cache=0, count_queries=None, count_samples=None, data_sub_sample_rate=0.875, dataset='terabyte', dataset_path='/research/dept8/gds/wrhuang/input', duration=300000, find_peak_performance=False, inputs=['continuous and categorical features'], max_batchsize=2048, max_ind_range=10000000, max_latency=None, mlperf_bin_loader=False, mlperf_conf='../../../mlperf.conf', model='dlrm', model_path='/research/dept8/gds/wrhuang/new_proj/dlrm_terabyte.pytorch', numpy_rand_seed=123, output='/research/dept8/gds/wrhuang/mlperf/inference/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm', outputs=['probability'], profile='dlrm-terabyte-pytorch', samples_per_query_multistream=None, samples_per_query_offline=2048, samples_to_aggregate_fix=None, samples_to_aggregate_max=None, samples_to_aggregate_min=None, samples_to_aggregate_quantile_file='./tools/dist_quantile.txt', samples_to_aggregate_trace_file='dlrm_trace_of_aggregated_samples.txt', scenario='Server', target_qps=None, test_num_workers=0, threads=192, use_gpu=False, user_conf='user.conf')",
    "runtime": "pytorch-native-dlrm",
    "time": 1622734942,
    "version": "1.8.1+cu102"
```
As you can see the `good_item` is 0, which means the accuracy of my model is 0. Also, while I set the `target_qps` to 10000.0, the qps achieved on my 192 core machine is only 19. I wonder do you have any idea why these things happen?



I carefully read the code and found that you actually never updated the variable `good_item`. Also, the `total_item` is set as `len(result_list)`, which is the number of queries instead of user-item pairs. In your code, both `count` and `total_item` are set `len(result_list)`, so I assume at least one of them is not supposed to hold that value. From the context, I guess the `good_item` and `total_item` should be the `good` and `total` in `DlrmProcess`. So I would insert two lines in line 600 of `main.py`:

```python
result_dict["good"] = runner.post_process.good
result_dict["total"] = runner.post_process.total
```

I wonder if my understanding is correct? @mnaumovfb 
This entry should not hurt the performance benchmarking but really made me confused during sample runs. 

@EtoDemerzel0427 This issue you raised will be looked at in few weeks by @mnaumovfb and his team.

> @EtoDemerzel0427 This issue you raised will be looked at in few weeks by @mnaumovfb and his team.

Thanks. But maybe I can close this issue now. I found that the reference implementation directly uses python's multi-threading which provides no concurrency because of the GIL. It is a naive implementation to show how a successful run looks like instead of measuring the performance. 

@EtoDemerzel0427, thank you for your suggestion, we added these lines to new PR

https://github.com/mlcommons/inference/pull/965