Repository: mlcommons/inference
Issue: DLRM: provide hashes and sizes for the Criteo dataset
Root cause: Documentation Deficiency

The [Criteo dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) consists of 24 files, each of up to 16 GB in total (over 350 GB in total?)

I've downloaded most of them with a simple bash script:
```
$ for i in {0..23}; do \
  url=http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_$i.gz; \
  echo "Downloading from '$url' ..."; \
  curl -O $url --progress-bar
done
```
Then, the log would show potential problems:
```
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_0.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_1.gz' ...
##############################################                            64.3%
curl: (56) Recv failure: Connection reset by peer
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_2.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_3.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_4.gz' ...
###################                                                       26.9%^[
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_5.gz' ...
################################################################          88.9%
curl: (56) Recv failure: Connection reset by peer
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_6.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_7.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_8.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_9.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_10.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_11.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_12.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_13.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_14.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_15.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_16.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_17.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_18.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_19.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_20.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_21.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_22.gz' ...
######################################################################## 100.0%
Downloading from 'http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_23.gz' ...
######################################################################## 100.0%
```
Even for the ones that are shown as 100.0% completed, I can't be 100.0% sure that they are not corrupted. I suggest to provide hashes (e.g. [MD5](https://en.wikipedia.org/wiki/MD5)) and sizes in README.

Here's what I've got:

File | Size (`du *`) | MD5 (`md5sum *`)
-|-|-
day_0.gz  | 15927304 | 6cef23542552c3195e9e6e2bdbc4c235
day_1.gz  | 16292832 | 94b73908ee8f912c175420376c8952db
day_2.gz  | 16022296 | c3c0272c26cfaa03d932b2856a751ff5
day_3.gz  | 14779644 | b727ecfaaf482507bb998248833aa4c2
day_4.gz  | 12514396 | b99eaa6e324e49d9df4e6f840e76b6d9
day_5.gz  | 14169436 | 1294d0a56a90294aebf80133078d9879
day_6.gz  | 16753584 | 68586521483313e13aefb29e7b767fdb
day_7.gz  | 16465452 | a2c1c4bfec20fc88b0b504175a16b644
day_8.gz  | 15845316 | faabf247fd56140a76effa6a3da63253
day_9.gz  | 16195668 | ee3347a28c1dd2fb2c92094e643c132b
day_10.gz | 15201768 | d043c2ec0346eb4c71aaae935416688e
day_11.gz | 12698432 | 8d4ba32f0c4f654a3860b6f2ae1a8ea7
day_12.gz | 14029428 | 908480917ed39be2a2ad2e1c339c40b4
day_13.gz | 15953380 | 567d6bfa672dd10a0cf76feaec0cf92b
day_14.gz | 15882872 | ed377357aecaccc5f93c754c4819fd8d
day_15.gz | 15382724 | 8e91f2a8d3d95202dfc3b22b88064c12
day_16.gz | 14606256 | 387269870bf8ec7d285cf0e8ce82e92e
day_17.gz | 13483480 | 48d3538fcf04807e0be4d72072dbda0b
day_18.gz | 11796984 | f26e23b6ef242f40b0e3fd92c986170c
day_19.gz | 13059032 | 3f6f36657b0ff1258428356451eea6c8
day_20.gz | 16022492 | db7ff2b830817d3b10960f02bfb68547
day_21.gz | 15707044 | f1a4ba7f7a555cb4a7e724a082479f4a
day_22.gz | 15463520 | 848ae20c4eab730ae487acc8ddaf52ba
day_23.gz | 14591792 | a2748bdbc67dd544b3ac470c4f1a52df

@mnaumovfb I can do a PR?

For the extracted files:

File | Size (`du *`) | MD5 (`md5sum *`)
-|-|-
day_0  |  48603344  |  331703bf14b9a699324d589efd676962
day_1  |  49518248  |  1df068493bb19edce48c09a1ce1a7fca
day_2  |  48820460  |  a5dec1724865504895e03508d6308046
day_3  |  44863556  |  c6de260d9eb835a2a1866b9931f4c474
day_4  |  37583076  |  9a41d00282b6b87db890518a86ab001f
day_5  |  42691840  |  170cba5a53f2b7cea8b75e19f10e152e
day_6  |  50857184  |  a47640af3fdffdec69c7a61f74e8b4f6
day_7  |  49874324  |  b0ff0428ce74fcb3f690f5330b7803d7
day_8  |  48082108  |  89113bd1eed24775ff1e757feb80447a
day_9  |  49252856  |  730903c294a98261beefc5d8c1634fc9
day_10 |  46029928  |  994fb3a9a43c0ebef0e28357a3b34068
day_11 |  37983540  |  a62384802da42c8e0a82aa0a26e5f99e
day_12 |  41832600  |  3f4dc3fca55231e7e9cff5a696325a28
day_13 |  48109928  |  2ad1cddf9a1b93a6315ba29ceb31124c
day_14 |  48059080  |  969bf0ec8cc212fe1816c153161df367
day_15 |  46383644  |  fb44b44d68237d694b56f85f603f52cb
day_16 |  44094432  |  866d4b3ef3381fec2c77afee58dcb987
day_17 |  40374712  |  89e82426b5bc49e93b40d9dbd1bbf7a4
day_18 |  35039044  |  294f2d46c8653d36c12c163d9f91b5ac
day_19 |  38662560  |  74bc0f7b0b6dd324ecc029445c875ea3
day_20 |  47981304  |  5b02f47b002cd374613d8e27f2daa7ce
day_21 |  47650792  |  70dfaf349be746e3ea1a7affe97f40b8
day_22 |  47037532  |  c9abe6cbae0b93f6702af27cda53d229
day_23 |  44152268  |  08e251af4f3d1e8771ea15e405f39600

The compression ratio is ~3x, so the total uncompressed size is ~1.1 TB.

I would suggest making a PR to add this information in point 2. in section "More information about the datasets" in [DLRM README in MLPerf inference](https://github.com/mlperf/inference/blob/master/v0.5/recommendation/README.md).