Repository: mlcommons/inference
Issue: [classification_and_detection] Where can we find details on data pre-processing for inference for imagenet data with resnet50v1.5 model?
Root cause: Documentation Deficiency

Where can we find code and details on data pre-processing for inference for imagenet data with resnet50v1.5 model?

Hi @ahsabali, the official script for ImageNet is [here](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/imagenet.py). The preprocessed dataset takes about 29 GB in floats.

However, I'd strongly recommend to use [Collective Knowledge](http://cknowledge.org) for preprocessing. Here's how it works on a small dataset:
```
$ python3 -m pip install ck --user
$ ck pull repo:ck-env
$ ck install package --tags=dataset,imagenet,val,min --no_tags=resized
$ ck install package --tags=dataset,imagenet,preprocessed,for-resnet
$ ck locate env --tags=dataset,imagenet,preprocessed,for-resnet
/home/anton/CK_TOOLS/dataset-imagenet-preprocessed-using-opencv-crop.875-first.20-for-resnet-side.224-unmutilated
```

Unfortunately, the link for downloading the full ImageNet validation data set is [no longer available](https://groups.google.com/forum/m/#!topic/mlperf/3I0bh20uCNA).

However, if you already have this dataset in some location, you can register it with CK by providing the full path to `ILSVRC2012_val_00000001.JPEG` e.g.:
```
$ ck detect soft --tags=dataset,imagenet,val --full_path=/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00000001.JPEG
```
and then repeating the same:
```
$ ck install package --tags=dataset,imagenet,preprocessed,for-resnet
```
You can read more about preproprocessing options using CK [here](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification#preprocess-datasets) (including a "universal" method which only requires 7 GB of disk space and can be reused for MobileNet too, and using TensorFlow which results in the highest accuracy for ResNet).

Shout if you need any help.

@psyhtest Thanks for the answers, this is helpful. Any ideas how much is the difference in accuracy if we skip pre-processing for validation?

Do you mean, how much the accuracy can drop if you run prediction on raw JPEG? It can easily be a couple percent, if not a dozen. 

Yes. Any ballpark number for the minimum accuracy on raw data without pre-processing?

Well, I'm debugging a case right now where the ResNet accuracy drops to
under 50%, and the lack of preprocessing is the main suspect...

On Mon, 2 Dec 2019, 21:38 ahsabali, <notifications@github.com> wrote:

> Yes. Any ballpark number for the minimum accuracy on raw data without
> pre-processing?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/541?email_source=notifications&email_token=ABSKZOTBK5RLKHO2CYKF43TQWV533A5CNFSM4JR6B5H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFVGALY#issuecomment-560619567>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABSKZOW7GYGBG4ULYIZVK6LQWV533ANCNFSM4JR6B5HQ>
> .
>


@psyhtest Thanks for sharing the info. Can it drop to zero? E.g., for the fake imagenet data set in the example workflow, with pre-processing the accuracy for this data is 85%.

Everything is possible. In once instance, a model required normalized data, and the accuracy dropped to under 30% when we forgot to normalize the inputs.

> Hi @ahsabali, the official script for ImageNet is [here](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/imagenet.py). The preprocessed dataset takes about 29 GB in floats.
> 
> However, I'd strongly recommend to use [Collective Knowledge](http://cknowledge.org) for preprocessing. Here's how it works on a small dataset:
> 
> ```
> $ python3 -m pip install ck --user
> $ ck pull repo:ck-env
> $ ck install package --tags=dataset,imagenet,val,min --no_tags=resized
> $ ck install package --tags=dataset,imagenet,preprocessed,for-resnet
> $ ck locate env --tags=dataset,imagenet,preprocessed,for-resnet
> /home/anton/CK_TOOLS/dataset-imagenet-preprocessed-using-opencv-crop.875-first.20-for-resnet-side.224-unmutilated
> ```
> 
> Unfortunately, the link for downloading the full ImageNet validation data set is [no longer available](https://groups.google.com/forum/m/#!topic/mlperf/3I0bh20uCNA).
> 
> However, if you already have this dataset in some location, you can register it with CK by providing the full path to `ILSVRC2012_val_00000001.JPEG` e.g.:
> 
> ```
> $ ck detect soft --tags=dataset,imagenet,val --full_path=/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00000001.JPEG
> ```
> 
> and then repeating the same:
> 
> ```
> $ ck install package --tags=dataset,imagenet,preprocessed,for-resnet
> ```
> 
> You can read more about preproprocessing options using CK [here](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification#preprocess-datasets) (including a "universal" method which only requires 7 GB of disk space and can be reused for MobileNet too, and using TensorFlow which results in the highest accuracy for ResNet).
> 
> Shout if you need any help.

Hi Anton, what are the CK instructions to download and pre-process the complete  imagenet2012 validation dataset for all models that use it?

Hi Vilmara,

The instructions are still exactly the same as you and other submitters used for v0.5:
```bash
$ ck install package:imagenet-2012-val
```

Unfortunately, the underlying link is now [broken](#542), so this command does not work any more. It's not a CK issue. It's not even an MLPerf issue, if you ask me. But we are somewhat stuck. Resorting to torrents to download ImageNet would be akin to sourcing rare goods from a black market (similar to this ["Looking for Transistors"](https://www.youtube.com/watch?v=c4pCeEBNiUA) scene from a popular Soviet movie).

However, if you still have a copy of this dataset  in some location, you can register it with CK by providing the full path to `ILSVRC2012_val_00000001.JPEG` e.g.:
```bash
$ ck detect soft --tags=dataset,imagenet,val \
--full_path=/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00000001.JPEG
```

The best preprocessing method for ResNet (with the Top1 accuracy of 76.522%) uses TensorFlow:
```
$ ck install package --tags=dataset,imagenet,preprocessed,using-tensorflow --ask
```
Other methods are documented [here](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification#preprocess-datasets).