Repository: mlcommons/inference
Issue: MobileNet TF models at zenodo.org and tensorflow.org differ
Root cause: Configuration Error

Links for MobileNet TF models from the [v0.5 README](https://github.com/dividiti/inference/tree/update-ck/v0.5/classification_and_detection) point to different files:
```
anton@diviniti:/tmp$ wget https://zenodo.org/record/2271498/files/mobilenet_v1_1.0_224.tgz
--2019-08-02 10:26:28--  https://zenodo.org/record/2271498/files/mobilenet_v1_1.0_224.tgz
Resolving zenodo.org (zenodo.org)... 188.184.65.20
Connecting to zenodo.org (zenodo.org)|188.184.65.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 94321541 (90M) [application/octet-stream]
Saving to: ‘mobilenet_v1_1.0_224.tgz’

mobilenet_v1_1.0_224.tgz                           100%[================================================================================================================>]  89.95M  16.9MB/s    in 5.5s

2019-08-02 10:26:35 (16.4 MB/s) - ‘mobilenet_v1_1.0_224.tgz’ saved [94321541/94321541]

anton@diviniti:/tmp$ wget http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz
--2019-08-02 10:26:52--  http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz
Resolving download.tensorflow.org (download.tensorflow.org)... 216.58.213.16, 2a00:1450:4009:818::2010
Connecting to download.tensorflow.org (download.tensorflow.org)|216.58.213.16|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 94321559 (90M) [application/x-compressed-tar]
Saving to: ‘mobilenet_v1_1.0_224.tgz.1’

mobilenet_v1_1.0_224.tgz.1                         100%[================================================================================================================>]  89.95M  13.6MB/s    in 7.1s

2019-08-02 10:26:59 (12.7 MB/s) - ‘mobilenet_v1_1.0_224.tgz.1’ saved [94321559/94321559]

anton@diviniti:/tmp$ md5sum mobilenet_v1_1.0_224.tgz
d5f69cef81ad8afb335d9727a17c462a  mobilenet_v1_1.0_224.tgz
anton@diviniti:/tmp$ md5sum mobilenet_v1_1.0_224.tgz.1
bc1d3a910cc3f70d94e5d608fc5e9590  mobilenet_v1_1.0_224.tgz.1
```

The same issue for the MobileNet TF quantized models (not in the README yet):
```
anton@diviniti:/tmp$ wget https://zenodo.org/record/2271498/files/mobilenet_v1_1.0_224_quant.tgz
--2019-08-02 10:32:40--  https://zenodo.org/record/2271498/files/mobilenet_v1_1.0_224_quant.tgz
Resolving zenodo.org (zenodo.org)... 188.184.65.20
Connecting to zenodo.org (zenodo.org)|188.184.65.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35071054 (33M) [application/octet-stream]
Saving to: ‘mobilenet_v1_1.0_224_quant.tgz’

mobilenet_v1_1.0_224_quant.tgz                     100%[================================================================================================================>]  33.45M  12.2MB/s    in 2.8s

2019-08-02 10:32:44 (12.2 MB/s) - ‘mobilenet_v1_1.0_224_quant.tgz’ saved [35071054/35071054]

anton@diviniti:/tmp$ wget http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz
--2019-08-02 10:33:01--  http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz
Resolving download.tensorflow.org (download.tensorflow.org)... 216.58.210.208, 2a00:1450:4009:81b::2010
Connecting to download.tensorflow.org (download.tensorflow.org)|216.58.210.208|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35069912 (33M) [application/x-compressed-tar]
Saving to: ‘mobilenet_v1_1.0_224_quant.tgz.1’

mobilenet_v1_1.0_224_quant.tgz.1                   100%[================================================================================================================>]  33.44M  12.2MB/s    in 2.7s

2019-08-02 10:33:04 (12.2 MB/s) - ‘mobilenet_v1_1.0_224_quant.tgz.1’ saved [35069912/35069912]

anton@diviniti:/tmp$ md5sum mobilenet_v1_1.0_224_quant.tgz
21b204450bc1ffc81771275d59b6e787  mobilenet_v1_1.0_224_quant.tgz
anton@diviniti:/tmp$ md5sum mobilenet_v1_1.0_224_quant.tgz.1
36af340c00e60291931cb30ce32d4e86  mobilenet_v1_1.0_224_quant.tgz.1
```

I see: Zenodo contains files from the previous release (2018_02_22):
```
anton@diviniti:/tmp$ wget http://download.tensorflow.org/models/mobilenet_v1_2018_02_22/mobilenet_v1_1.0_224.tgz
--2019-08-02 10:39:40--  http://download.tensorflow.org/models/mobilenet_v1_2018_02_22/mobilenet_v1_1.0_224.tgz
Resolving download.tensorflow.org (download.tensorflow.org)... 216.58.204.48, 2a00:1450:4009:814::2010
Connecting to download.tensorflow.org (download.tensorflow.org)|216.58.204.48|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 94321541 (90M) [application/x-compressed-tar]
Saving to: ‘mobilenet_v1_1.0_224.tgz’

mobilenet_v1_1.0_224.tgz                           100%[================================================================================================================>]  89.95M  15.2MB/s    in 5.9s    

2019-08-02 10:39:47 (15.3 MB/s) - ‘mobilenet_v1_1.0_224.tgz’ saved [94321541/94321541]

anton@diviniti:/tmp$ wget http://download.tensorflow.org/models/mobilenet_v1_2018_02_22/mobilenet_v1_1.0_224_quant.tgz
--2019-08-02 10:39:59--  http://download.tensorflow.org/models/mobilenet_v1_2018_02_22/mobilenet_v1_1.0_224_quant.tgz
Resolving download.tensorflow.org (download.tensorflow.org)... 216.58.204.48, 2a00:1450:4009:814::2010
Connecting to download.tensorflow.org (download.tensorflow.org)|216.58.204.48|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35071054 (33M) [application/x-compressed-tar]
Saving to: ‘mobilenet_v1_1.0_224_quant.tgz’

mobilenet_v1_1.0_224_quant.tgz                     100%[================================================================================================================>]  33.45M  10.9MB/s    in 3.1s    

2019-08-02 10:40:02 (10.9 MB/s) - ‘mobilenet_v1_1.0_224_quant.tgz’ saved [35071054/35071054]

anton@diviniti:/tmp$ md5sum mobilenet_v1_1.0_224.tgz 
d5f69cef81ad8afb335d9727a17c462a  mobilenet_v1_1.0_224.tgz
anton@diviniti:/tmp$ md5sum mobilenet_v1_1.0_224_quant.tgz
21b204450bc1ffc81771275d59b6e787  mobilenet_v1_1.0_224_quant.tgz
```




The correct Zenodo record is `2269307` not `2271498` (despite the latter appearing to be more fresh):
```
anton@diviniti:/tmp$ wget https://zenodo.org/record/2269307/files/mobilenet_v1_1.0_224.tgz
--2019-08-02 10:43:28--  https://zenodo.org/record/2269307/files/mobilenet_v1_1.0_224.tgz
Resolving zenodo.org (zenodo.org)... 188.184.65.20
Connecting to zenodo.org (zenodo.org)|188.184.65.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 94321559 (90M) [application/octet-stream]
Saving to: ‘mobilenet_v1_1.0_224.tgz’

mobilenet_v1_1.0_224.tgz                           100%[================================================================================================================>]  89.95M  11.3MB/s    in 7.3s

2019-08-02 10:43:36 (12.3 MB/s) - ‘mobilenet_v1_1.0_224.tgz’ saved [94321559/94321559]

anton@diviniti:/tmp$ wget https://zenodo.org/record/2269307/files/mobilenet_v1_1.0_224_quant.tgz
--2019-08-02 10:43:20--  https://zenodo.org/record/2269307/files/mobilenet_v1_1.0_224_quant.tgz
Resolving zenodo.org (zenodo.org)... 188.184.65.20
Connecting to zenodo.org (zenodo.org)|188.184.65.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35069912 (33M) [application/octet-stream]
Saving to: ‘mobilenet_v1_1.0_224_quant.tgz’

mobilenet_v1_1.0_224_quant.tgz                     100%[================================================================================================================>]  33.44M  15.3MB/s    in 2.2s

2019-08-02 10:43:23 (15.3 MB/s) - ‘mobilenet_v1_1.0_224_quant.tgz’ saved [35069912/35069912]

anton@diviniti:/tmp$ md5sum mobilenet_v1_1.0_224.tgz
bc1d3a910cc3f70d94e5d608fc5e9590  mobilenet_v1_1.0_224.tgz
anton@diviniti:/tmp$ md5sum mobilenet_v1_1.0_224_quant.tgz
36af340c00e60291931cb30ce32d4e86  mobilenet_v1_1.0_224_quant.tgz
```

I blame @jveejay :). He asked @gfursin to rename https://zenodo.org/record/2271498 from:
> "MobileNets-v1-20180222 for MLPerf Inference (Edge)"

to:

> mobilenet-v1-.tensorflow model

but should have asked to rename https://zenodo.org/record/2269307 from:
> "MobileNets-v1-20180802 for MLPerf Inference (Edge)"

I would suggest to reinstate the name for https://zenodo.org/record/2271498 to:
> "MobileNets-v1-20180222 for MLPerf Inference (Edge)"

and rename https://zenodo.org/record/2269307 to:
> mobilenet-v1-tensorflow model

instead (no dot before tensorflow).

@jveejay @gfursin How does this sound?

By the way, with CK one can install the same models from alternative URLs e.g.:
```
$ ck install package --tags=model,tf,mlperf,mobilenet,quantized,from-google
```
or:
```
$ ck install package --tags=model,tf,mlperf,mobilenet,quantized,from-zenodo
```
when one of the model sources is unavailable. (This has happened on a couple of occasions.)

The Zenodo variation is the default, so one can simply run:
```
$ ck install package --tags=model,tf,mlperf,mobilenet,quantized
```
to get the quantized TF MobileNet model from Zenodo.

I guess @jveejay has to decide... I am fine with whatever is more convenient for MLPerf and can update the naming on Zenodo if needed...

Phew, sorry about that folks. Yea, keeping these links is being a pain. Good thing we are at least using Zenodo where we have unique IDs for these records. Imagine trying to figure this out if we were using Google drive and we were constantly updating the model there with no record IDs. 

So does this mean that #304 is resolved? 

@psyhtest is it possible to add https://zenodo.org/record/2269307#.XUhBiS2ZPUI to the MLPerf community also? It is in the collective knowledge but would be good to have it associated with MLPerf too cause then people know the model source is good for mlperf as well. 



I suggest to rename as follows:
- https://zenodo.org/record/2269307: "TensorFlow MobileNets-v1-20180802 (used in MLPerf Inference)"
- https://zenodo.org/record/2271498: "TensorFlow MobileNets-v1-20180222"
- https://zenodo.org/record/2280136: "TensorFlow MobileNets-v1-20170614"

I don't know if it's possible to associate the first one with the MLPerf community.

> So does this mean that #304 is resolved?

No, the quantized model has more than 1% degradation in accuracy, no matter how to squint at it.