Repository: mlcommons/inference
Issue: DLRM reference implementation is failing for both fake and real criteo dataset
Root cause: External Dependency Breakage

I tried to run DLRM using the full criteo dataset and the fake dataset. Both are giving similar errors as follows. Tried with torch 1.13, 1.12 and 1.11

To reproduce
1. [Install CM](https://github.com/mlcommons/ck/blob/master/docs/installation.md)
2. Run the below commands
```bash
cm pull repo octoml@ck
cm run script --tags=mlperf,inference,reference,_dlrm-99.9,_pytorch \
--adr.criteo-preprocessed.tags=_fake --adr.dlrm-model.tags=_debug
```

```
Run Directory: /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo
CMD: cd '/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/recommendation/dlrm/pytorch' && OUTPUT_DIR='/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/classification_and_detection/test_results/pytorch-cpu/dlrm-99.9/offline/accuracy' ./run_local.sh pytorch dlrm terabyte cpu --scenario Offline  --mlperf_conf '/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/mlperf.conf' --max-ind-range=10000000 --data-sub-sample-rate=0.875 --max-batchsize=2048 --test-num-workers=1 --count-samples=10 --samples-to-aggregate-quantile-file=./tools/dist_quantile.txt  --accuracy
+ python python/main.py --profile dlrm-terabyte-pytorch --mlperf_conf ../../../mlperf.conf --model dlrm --model-path /home/arjun/CM/repos/local/cache/d9743e6d850c4e52/dlrm_terabyte.pytorch --dataset terabyte --dataset-path /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo --output /home/arjun/CM/repos/local/cache/a127f06d39034768/inference/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm --scenario Offline --mlperf_conf /home/arjun/CM/repos/local/cache/a127f06d39034768/inference/mlperf.conf --max-ind-range=10000000 --data-sub-sample-rate=0.875 --max-batchsize=2048 --test-num-workers=1 --count-samples=10 --samples-to-aggregate-quantile-file=./tools/dist_quantile.txt --accuracy
INFO:main:Namespace(model='dlrm', model_path='/home/arjun/CM/repos/local/cache/d9743e6d850c4e52/dlrm_terabyte.pytorch', dataset='terabyte', dataset_path='/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo', profile='dlrm-terabyte-pytorch', scenario='Offline', test_num_workers=1, max_ind_range=10000000, data_sub_sample_rate=0.875, mlperf_bin_loader=False, max_batchsize=2048, output='/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm', inputs=['continuous and categorical features'], outputs=['probability'], backend='pytorch-native', use_gpu=False, threads=32, cache=0, accuracy=True, find_peak_performance=False, mlperf_conf='/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/mlperf.conf', user_conf='user.conf', duration=None, target_qps=None, max_latency=None, count_samples=10, count_queries=None, samples_per_query_multistream=8, samples_per_query_offline=2048, samples_to_aggregate_fix=None, samples_to_aggregate_min=None, samples_to_aggregate_max=None, samples_to_aggregate_quantile_file='./tools/dist_quantile.txt', samples_to_aggregate_trace_file='dlrm_trace_of_aggregated_samples.txt', numpy_rand_seed=123)
Using CPU...
Reading raw data=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_0
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_1
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_2
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_3
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_4
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_5
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_6
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_7
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_8
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_9
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_10
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_11
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_12
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_13
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_14
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_15
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_16
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_17
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_18
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_19
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_20
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_21
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_22
Reading data from path=/home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_23
Load 2299/4096  Split: 0  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_0.npz!
Load 2268/4096  Split: 1  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_1.npz!
Load 2276/4096  Split: 2  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_2.npz!
Load 2303/4096  Split: 3  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_3.npz!
Load 2273/4096  Split: 4  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_4.npz!
Load 2265/4096  Split: 5  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_5.npz!
Load 2271/4096  Split: 6  Label True: 0  Stored: 0
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_6.npz!
Load 2314/4096  Split: 7  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_7.npz!
Load 2311/4096  Split: 8  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_8.npz!
Load 2331/4096  Split: 9  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_9.npz!
Load 2296/4096  Split: 10  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_10.npz!
Load 2364/4096  Split: 11  Label True: 0  Stored: 0
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_11.npz!
Load 2291/4096  Split: 12  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_12.npz!
Load 2299/4096  Split: 13  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_13.npz!
Load 2270/4096  Split: 14  Label True: 0  Stored: 0
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_14.npz!
Load 2295/4096  Split: 15  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_15.npz!
Load 2298/4096  Split: 16  Label True: 0  Stored: 0
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_16.npz!
Load 2317/4096  Split: 17  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_17.npz!
Load 2349/4096  Split: 18  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_18.npz!
Load 2278/4096  Split: 19  Label True: 0  Stored: 0
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_19.npz!
Load 2303/4096  Split: 20  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_20.npz!
Load 2330/4096  Split: 21  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_21.npz!
Load 2318/4096  Split: 22  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_22.npz!
Load 2348/4096  Split: 23  Label True: 1  Stored: 1
Saved /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_23.npz!
Total number of samples: 55291
Divided into days/splits:
 [2300, 2269, 2277, 2304, 2274, 2266, 2272, 2315, 2312, 2332, 2297, 2365, 2292, 2300, 2271, 2296, 2299, 2318, 2350, 2279, 2304, 2331, 2319, 2349]
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_0_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_0_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_1_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_1_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_2_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_2_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_3_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_3_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_4_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_4_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_5_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_5_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_6_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_6_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_7_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_7_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_8_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_8_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_9_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_9_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_10_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_10_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_11_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_11_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_12_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_12_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_13_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_13_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_14_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_14_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_15_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_15_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_16_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_16_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_17_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_17_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_18_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_18_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_19_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_19_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_20_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_20_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_21_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_21_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_22_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_22_processed.npz
Not existing /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_23_processed.npz
Processed /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_23_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_0_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_1_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_2_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_3_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_4_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_5_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_6_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_7_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_8_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_9_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_10_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_11_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_12_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_13_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_14_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_15_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_16_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_17_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_18_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_19_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_20_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_21_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_22_processed.npz
Reordering (1st pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_23_processed.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_0_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_1_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_2_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_3_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_4_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_5_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_6_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_7_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_8_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_9_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_10_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_11_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_12_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_13_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_14_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_15_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_16_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_17_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_18_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_19_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_20_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_21_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_22_reordered.npz
Reordering (2nd pass) /home/arjun/CM/repos/local/cache/50e2c257986046fd/inference/recommendation/dlrm/pytorch/tools/fake_criteo/day_23_reordered.npz
Sparse features= 26, Dense features= 13
Using variable query size: custom distribution (file ./tools/dist_quantile.txt)
/home/arjun/CM/repos/local/cache/66d204212cf24049/dlrm/dlrm_data_pytorch.py:331: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:230.)
  X_int = torch.log(torch.tensor(transposed_data[0], dtype=torch.float) + 1)
Traceback (most recent call last):
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/recommendation/dlrm/pytorch/python/main.py", line 619, in <module>
    main()
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/recommendation/dlrm/pytorch/python/main.py", line 535, in main
    _ = backend.predict(batch_dense_X, batch_lS_o, batch_lS_i)
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/recommendation/dlrm/pytorch/python/backend_pytorch_native.py", line 125, in predict
    output = self.model(dense_x=batch_dense_X, lS_o=batch_lS_o, lS_i=batch_lS_i)
  File "/home/arjun/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/arjun/CM/repos/local/cache/66d204212cf24049/dlrm/dlrm_s_pytorch.py", line 514, in forward
    return self.sequential_forward(dense_x, lS_o, lS_i)
  File "/home/arjun/CM/repos/local/cache/66d204212cf24049/dlrm/dlrm_s_pytorch.py", line 586, in sequential_forward
    ly = self.apply_emb(lS_o, lS_i, self.emb_l, self.v_W_l)
  File "/home/arjun/CM/repos/local/cache/66d204212cf24049/dlrm/dlrm_s_pytorch.py", line 442, in apply_emb
    V = E(
  File "/home/arjun/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/arjun/.local/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 387, in forward
    return F.embedding_bag(input, self.weight, offsets,
  File "/home/arjun/.local/lib/python3.10/site-packages/torch/nn/functional.py", line 2392, in embedding_bag
    ret, _, _, _ = torch.embedding_bag(
RuntimeError: Expected i < index_size to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
```

@arjunsuresh This error seems to be caused by the pytorch version.

When I run the benchmark using cm in a VM without pytorch installed, I get the same error. Here, cm installs `pytorch==1.13`. However, I managed to run the benchmark using cm in VM with `torch==1.10` preinstalled. I think this should be fixed in the ck repo. I will also open a PR to specify this version in the dlrm readme.


Thank you @pgmpablo157321 for fixing the issue. In addition to `torch==1.10` I also needed to downgrade numpy to `1.19` to get it to work. I guess the errors are coming because of `dtype` missing for numpy arrays. Since we are going to replace this benchmark after v3.0 I guess there is no point making it compatible with latest version. I have fixed these versions in ck repo.