Repository: mlcommons/inference
Issue: can't run dlrm
Root cause: Resource Mishandling

**Since the dataset is too large, I used the make fake tool script. However, during runtime, an error occurred, and it seems that there is a problem with the data set. The specific information is as follows. Looking for a specific error, thank you!**

```+ python python/main_dlrm.py --profile dlrm-terabyte-pytorch --mlperf_conf /root/mlcommons/mlperf.conf --model dlrm --model-path /root/model/dlrm_terabyte.pytorch --dataset terabyte --dataset-path /root/data --output /root/mlcommons/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm --max-ind-range=10000000
Unable to import mlperf_logging,  No module named 'mlperf_logging'
INFO:main:Namespace(accuracy=False, backend='pytorch-native', cache=0, count_queries=None, count_samples=None, data_sub_sample_rate=0.0, dataset='terabyte', dataset_path='/root/data', duration=None, find_peak_performance=False, inputs=['continuous and categorical features'], max_batchsize=2048, max_ind_range=10000000, max_latency=None, mlperf_bin_loader=False, mlperf_conf='/root/mlcommons/mlperf.conf', model='dlrm', model_path='/root/model/dlrm_terabyte.pytorch', numpy_rand_seed=123, output='/root/mlcommons/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm', outputs=['probability'], profile='dlrm-terabyte-pytorch', samples_per_query_multistream=8, samples_per_query_offline=2048, samples_to_aggregate_fix=None, samples_to_aggregate_max=None, samples_to_aggregate_min=None, samples_to_aggregate_quantile_file=None, samples_to_aggregate_trace_file='dlrm_trace_of_aggregated_samples.txt', scenario='SingleStream', target_qps=None, test_num_workers=0, threads=8, use_gpu=False, user_conf='user.conf')
Using CPU...
Reading raw data=/root/data/day
Reading data from path=/root/data/day_0
Reading data from path=/root/data/day_1
Reading data from path=/root/data/day_2
Reading data from path=/root/data/day_3
Reading data from path=/root/data/day_4
Reading data from path=/root/data/day_5
Reading data from path=/root/data/day_6
Reading data from path=/root/data/day_7
Reading data from path=/root/data/day_8
Reading data from path=/root/data/day_9
Reading data from path=/root/data/day_10
Reading data from path=/root/data/day_11
Reading data from path=/root/data/day_12
Reading data from path=/root/data/day_13
Reading data from path=/root/data/day_14
Reading data from path=/root/data/day_15
Reading data from path=/root/data/day_16
Reading data from path=/root/data/day_17
Reading data from path=/root/data/day_18
Reading data from path=/root/data/day_19
Reading data from path=/root/data/day_20
Reading data from path=/root/data/day_21
Reading data from path=/root/data/day_22
Reading data from path=/root/data/day_23
Load 4095/4096  Split: 0  Label True: 0  Stored: 0
Saved /root/data/day_0.npz!
Load 4095/4096  Split: 1  Label True: 1  Stored: 1
Saved /root/data/day_1.npz!
Load 4095/4096  Split: 2  Label True: 1  Stored: 1
Saved /root/data/day_2.npz!
Load 4095/4096  Split: 3  Label True: 0  Stored: 0
Saved /root/data/day_3.npz!
Load 4095/4096  Split: 4  Label True: 1  Stored: 1
Saved /root/data/day_4.npz!
Load 4095/4096  Split: 5  Label True: 0  Stored: 0
Saved /root/data/day_5.npz!
Load 4095/4096  Split: 6  Label True: 0  Stored: 0
Saved /root/data/day_6.npz!
Load 4095/4096  Split: 7  Label True: 1  Stored: 1
Saved /root/data/day_7.npz!
Load 4095/4096  Split: 8  Label True: 0  Stored: 0
Saved /root/data/day_8.npz!
Load 4095/4096  Split: 9  Label True: 0  Stored: 0
Saved /root/data/day_9.npz!
Load 4095/4096  Split: 10  Label True: 0  Stored: 0
Saved /root/data/day_10.npz!
Load 4095/4096  Split: 11  Label True: 0  Stored: 0
Saved /root/data/day_11.npz!
Load 4095/4096  Split: 12  Label True: 0  Stored: 0
Saved /root/data/day_12.npz!
Load 4095/4096  Split: 13  Label True: 1  Stored: 1
Saved /root/data/day_13.npz!
Load 4095/4096  Split: 14  Label True: 0  Stored: 0
Saved /root/data/day_14.npz!
Load 4095/4096  Split: 15  Label True: 1  Stored: 1
Saved /root/data/day_15.npz!
Load 4095/4096  Split: 16  Label True: 0  Stored: 0
Saved /root/data/day_16.npz!
Load 4095/4096  Split: 17  Label True: 1  Stored: 1
Saved /root/data/day_17.npz!
Load 4095/4096  Split: 18  Label True: 0  Stored: 0
Saved /root/data/day_18.npz!
Load 4095/4096  Split: 19  Label True: 0  Stored: 0
Saved /root/data/day_19.npz!
Load 4095/4096  Split: 20  Label True: 1  Stored: 1
Saved /root/data/day_20.npz!
Load 4095/4096  Split: 21  Label True: 0  Stored: 0
Saved /root/data/day_21.npz!
Load 4095/4096  Split: 22  Label True: 0  Stored: 0
Saved /root/data/day_22.npz!
Load 4095/4096  Split: 23  Label True: 1  Stored: 1
Saved /root/data/day_23.npz!
Total number of samples: 98304
Divided into days/splits:
 [4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096]
Not existing /root/data/day_0_processed.npz
Processed /root/data/day_0_processed.npz
Not existing /root/data/day_1_processed.npz
Processed /root/data/day_1_processed.npz
Not existing /root/data/day_2_processed.npz
Processed /root/data/day_2_processed.npz
Not existing /root/data/day_3_processed.npz
Processed /root/data/day_3_processed.npz
Not existing /root/data/day_4_processed.npz
Processed /root/data/day_4_processed.npz
Not existing /root/data/day_5_processed.npz
Processed /root/data/day_5_processed.npz
Not existing /root/data/day_6_processed.npz
Processed /root/data/day_6_processed.npz
Not existing /root/data/day_7_processed.npz
Processed /root/data/day_7_processed.npz
Not existing /root/data/day_8_processed.npz
Processed /root/data/day_8_processed.npz
Not existing /root/data/day_9_processed.npz
Processed /root/data/day_9_processed.npz
Not existing /root/data/day_10_processed.npz
Processed /root/data/day_10_processed.npz
Not existing /root/data/day_11_processed.npz
Processed /root/data/day_11_processed.npz
Not existing /root/data/day_12_processed.npz
Processed /root/data/day_12_processed.npz
Not existing /root/data/day_13_processed.npz
Processed /root/data/day_13_processed.npz
Not existing /root/data/day_14_processed.npz
Processed /root/data/day_14_processed.npz
Not existing /root/data/day_15_processed.npz
Processed /root/data/day_15_processed.npz
Not existing /root/data/day_16_processed.npz
Processed /root/data/day_16_processed.npz
Not existing /root/data/day_17_processed.npz
Processed /root/data/day_17_processed.npz
Not existing /root/data/day_18_processed.npz
Processed /root/data/day_18_processed.npz
Not existing /root/data/day_19_processed.npz
Processed /root/data/day_19_processed.npz
Not existing /root/data/day_20_processed.npz
Processed /root/data/day_20_processed.npz
Not existing /root/data/day_21_processed.npz
Processed /root/data/day_21_processed.npz
Not existing /root/data/day_22_processed.npz
Processed /root/data/day_22_processed.npz
Not existing /root/data/day_23_processed.npz
Processed /root/data/day_23_processed.npz
Reordering (1st pass) /root/data/day_0_processed.npz
Reordering (1st pass) /root/data/day_1_processed.npz
Reordering (1st pass) /root/data/day_2_processed.npz
Reordering (1st pass) /root/data/day_3_processed.npz
Reordering (1st pass) /root/data/day_4_processed.npz
Reordering (1st pass) /root/data/day_5_processed.npz
Reordering (1st pass) /root/data/day_6_processed.npz
Reordering (1st pass) /root/data/day_7_processed.npz
Reordering (1st pass) /root/data/day_8_processed.npz
Reordering (1st pass) /root/data/day_9_processed.npz
Reordering (1st pass) /root/data/day_10_processed.npz
Reordering (1st pass) /root/data/day_11_processed.npz
Reordering (1st pass) /root/data/day_12_processed.npz
Reordering (1st pass) /root/data/day_13_processed.npz
Reordering (1st pass) /root/data/day_14_processed.npz
Reordering (1st pass) /root/data/day_15_processed.npz
Reordering (1st pass) /root/data/day_16_processed.npz
Reordering (1st pass) /root/data/day_17_processed.npz
Reordering (1st pass) /root/data/day_18_processed.npz
Reordering (1st pass) /root/data/day_19_processed.npz
Reordering (1st pass) /root/data/day_20_processed.npz
Reordering (1st pass) /root/data/day_21_processed.npz
Reordering (1st pass) /root/data/day_22_processed.npz
Reordering (1st pass) /root/data/day_23_processed.npz
Reordering (2nd pass) /root/data/day_0_reordered.npz
Reordering (2nd pass) /root/data/day_1_reordered.npz
Reordering (2nd pass) /root/data/day_2_reordered.npz
Reordering (2nd pass) /root/data/day_3_reordered.npz
Reordering (2nd pass) /root/data/day_4_reordered.npz
Reordering (2nd pass) /root/data/day_5_reordered.npz
Reordering (2nd pass) /root/data/day_6_reordered.npz
Reordering (2nd pass) /root/data/day_7_reordered.npz
Reordering (2nd pass) /root/data/day_8_reordered.npz
Reordering (2nd pass) /root/data/day_9_reordered.npz
Reordering (2nd pass) /root/data/day_10_reordered.npz
Reordering (2nd pass) /root/data/day_11_reordered.npz
Reordering (2nd pass) /root/data/day_12_reordered.npz
Reordering (2nd pass) /root/data/day_13_reordered.npz
Reordering (2nd pass) /root/data/day_14_reordered.npz
Reordering (2nd pass) /root/data/day_15_reordered.npz
Reordering (2nd pass) /root/data/day_16_reordered.npz
Reordering (2nd pass) /root/data/day_17_reordered.npz
Reordering (2nd pass) /root/data/day_18_reordered.npz
Reordering (2nd pass) /root/data/day_19_reordered.npz
Reordering (2nd pass) /root/data/day_20_reordered.npz
Reordering (2nd pass) /root/data/day_21_reordered.npz
Reordering (2nd pass) /root/data/day_22_reordered.npz
Reordering (2nd pass) /root/data/day_23_reordered.npz
Sparse features= 26, Dense features= 13
Using fixed query size: 1
./run_local.sh: line 20:    25 Killed                  python python/main_dlrm.py --profile $profile --mlperf_conf /root/mlcommons/mlperf.conf --model $model --model-path $model_path --dataset $dataset --dataset-path $DATA_DIR --output $OUTPUT_DIR $EXTRA_OPS $@


This error happens when the system run out of memory. DLRM needs ~200GB of RAM and by default many OS allows only 40-50% of RAM to a given process. So, either we should change the OS limit or put 512GB RAM to run DLRM.