Repository: mlcommons/inference
Issue: ImageNet accuracy 73.87% (should be 76.47%)
Root cause: Algorithmic Error

For our image classification models, we are not hitting the reported ImageNet accuracy of 76%. We are at around 73.87% in our image classification and object detection [README](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection).

Possible reasons are pre-processing, or ... (something else). 

@itayhubara and @guschmue @parvizp

I ran validation on mlperf's resnet50 frozen model (tensorflow) from [here](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#supported-models), and the accuracy seems alright (~76.5%). [This](https://github.com/tensorflow/models/blob/r1.12.0/research/slim/preprocessing/vgg_preprocessing.py#L319) is the pre-processing I used.
```
Val:    [   100/   782] Time   0.084 (  0.113)  Prec@1  73.438 ( 76.795)        Prec@5  93.750 ( 93.007)
Val:    [   200/   782] Time   0.202 (  0.109)  Prec@1  81.250 ( 77.029)        Prec@5  95.312 ( 93.377)
Val:    [   300/   782] Time   0.198 (  0.136)  Prec@1  79.688 ( 76.905)        Prec@5  93.750 ( 93.304)
Val:    [   400/   782] Time   0.181 (  0.152)  Prec@1  71.875 ( 76.870)        Prec@5  92.188 ( 93.173)
Val:    [   500/   782] Time   0.151 (  0.160)  Prec@1  78.125 ( 76.771)        Prec@5  93.750 ( 93.083)
Val:    [   600/   782] Time   0.198 (  0.166)  Prec@1  75.000 ( 76.583)        Prec@5  92.188 ( 93.051)
Val:    [   700/   782] Time   0.339 (  0.168)  Prec@1  82.812 ( 76.516)        Prec@5  92.188 ( 93.039)
Val:    [   781/   782] Time   0.265 (  0.168)  Prec@1  68.750 ( 76.522)        Prec@5  87.500 ( 93.066)
 model_dir=/data/resnet50_v1.pb prec@1=76.522 prec@5=93.066              
```

Wow, thanks for sharing! The only problem is that the resize code itself uses TF. If we want to use it on platforms where TF is not well supported (e.g. Arm-based dev boards), we should convert it to a script using standard Python image processing libraries. 

According to Sambhav, the model is o.k so you probably need to change the
preprocess and in specific the way you resize the image. If I am not
mistaken all you should do is the following change
img.resize((w,h))  -> img.resize((w,h), PIL.Image.BILINEAR )

As PIL default resizing method is "nearest". It seems to work for me at
least on the default 500 images (from 72.6->74). Please let me know if this
works for you as well.

Best,
Itay

On Wed, Jun 19, 2019 at 5:45 PM Anton Lokhmotov <notifications@github.com>
wrote:

> Wow, thanks for sharing! The only problem is that the resize code itself
> uses TF. If we want to use it on platforms where TF is not well supported
> (e.g. Arm-based dev boards), we should convert it to a script using
> standard Python image processing libraries.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmlperf%2Finference%2Fissues%2F201%3Femail_source%3Dnotifications%26email_token%3DABTRU3ETKC6QK3OUJK56SKLP3LHIVA5CNFSM4HZL3UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYDVKOY%23issuecomment-503797051&data=02%7C01%7Citayh%40campus.technion.ac.il%7Ce4bdb3815d4b4535497508d6f51887e5%7Cf1502c4cee2e411c9715c855f6753b84%7C1%7C0%7C636965883049117343&sdata=F%2BNslrAmtiUlg3AE2zOswO70CAzFb2kyME8VO2gA3mQ%3D&reserved=0>,
> or mute the thread
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABTRU3HZALJG2A4IU2VXNOTP3LHIVANCNFSM4HZL3UNA&data=02%7C01%7Citayh%40campus.technion.ac.il%7Ce4bdb3815d4b4535497508d6f51887e5%7Cf1502c4cee2e411c9715c855f6753b84%7C1%7C0%7C636965883049117343&sdata=sf2pqYBs6klvEeckKYoF6Re3zMaUX8qPbjOllcGacSU%3D&reserved=0>
> .
>


@guschmue 

its the PIL.Image.BILINEAR , just fixing it.

Great!! I knew we can fix it :)

On Thu, Jun 20, 2019 at 10:29 AM Guenther Schmuelling <
notifications@github.com> wrote:

> its the PIL.Image.BILINEAR , just fixing it.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmlperf%2Finference%2Fissues%2F201%3Femail_source%3Dnotifications%26email_token%3DABTRU3APFMQDS4ITWQV5ESLP3O475A5CNFSM4HZL3UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYGCWDY%23issuecomment-504113935&data=02%7C01%7Citayh%40campus.technion.ac.il%7C6eb1319327a646e6f38e08d6f5a4dda2%7Cf1502c4cee2e411c9715c855f6753b84%7C1%7C0%7C636966485774037985&sdata=S5cHU7vJ6wALCq3vC%2FT2NgtXqA0qDXT6%2FxxEHMwGbnU%3D&reserved=0>,
> or mute the thread
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABTRU3HZALJZ2RM2DU5T6JTP3O475ANCNFSM4HZL3UNA&data=02%7C01%7Citayh%40campus.technion.ac.il%7C6eb1319327a646e6f38e08d6f5a4dda2%7Cf1502c4cee2e411c9715c855f6753b84%7C1%7C0%7C636966485774037985&sdata=ExiuWAvdsLbl154oD17hj3r5dWENOQ2Q%2B8%2BP7gjW0qc%3D&reserved=0>
> .
>


74.97 for resnet50, should be 76.4 ... one more fix to find.

Sadly I think this is the best we can do if we want to work with PIL.
Apparently, TF is not consistent with pytorch, PIL, and skimage when it
comes to aligning corners (as described for example here
<https://github.com/pytorch/pytorch/issues/10604>
Thus the best idea is to rewrite it in TF...

Itay

On Thu, Jun 20, 2019 at 12:29 PM Guenther Schmuelling <
notifications@github.com> wrote:

> 74.97 for resnet50, should be 76.4 ... one more fix to find.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmlperf%2Finference%2Fissues%2F201%3Femail_source%3Dnotifications%26email_token%3DABTRU3EBXT2SSX6FHUMHHULP3PLA5A5CNFSM4HZL3UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYGMGCA%23issuecomment-504152840&data=02%7C01%7Citayh%40campus.technion.ac.il%7C477bc38a52d145c73dcb08d6f5b59798%7Cf1502c4cee2e411c9715c855f6753b84%7C1%7C0%7C636966557638901890&sdata=c9tMAvTft89zfHbKH%2BV6ZzosPJtrZESx%2B0HiIwbz7hE%3D&reserved=0>,
> or mute the thread
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABTRU3ANFGMJ5YET6ROTHRLP3PLA5ANCNFSM4HZL3UNA&data=02%7C01%7Citayh%40campus.technion.ac.il%7C477bc38a52d145c73dcb08d6f5b59798%7Cf1502c4cee2e411c9715c855f6753b84%7C1%7C0%7C636966557638911884&sdata=8kwz694RBVfhFtu8Hc2ki91SL9xGDUJw9vUz2PwnEUQ%3D&reserved=0>
> .
>


https://github.com/mlperf/inference/pull/206. I'll look some more.


Dear @sjain-stanford, could you please share the code snippet of how you are using the vgg_preprocessing ? Exactly the call to the preprocessor ( preprocess_image() or preprocess_for_eval() ) with the parameters and any other manipulations before or after that call, if you do them (normalizations, flips, channel swapping, etc).

I am asking because even with your confirmation we are still struggling to get the 76.47% accuracy, so would like to reproduce your solution as close as possible.

Also, where does the magic number 782 come from? Did you choose to test on that many examples? If so, why?

Thank you very much in advance!

Hey @ens-lg4
> Dear @sjain-stanford, could you please share the code snippet of how you are using the vgg_preprocessing ? Exactly the call to the preprocessor ( preprocess_image() or preprocess_for_eval() ) with the parameters and any other manipulations before or after that call, if you do them (normalizations, flips, channel swapping, etc).
> 
> I am asking because even with your confirmation we are still struggling to get the 76.47% accuracy, so would like to reproduce your solution as close as possible.
> 
Sure. I use this [validation script](https://github.com/Xilinx/graffitist/blob/master/utils/validate_imagenet_tf.py) and this imagenet preprocessing [utils script](https://github.com/Xilinx/graffitist/blob/master/utils/imagenet_utils.py).

Here is the list of modifications I did to make it work with the frozen resnet50 from mlperf.
1. Rename `resnet50_v1.pb` to `resnet50_v1_frozen.pb` (tells the script it's a frozen graph, so don't load a ckpt)
2. Add the input and output tensor names to the [validation script](https://github.com/Xilinx/graffitist/blob/dcf7f4e8a36e1efdef217b3ad160af24d22d57df/utils/validate_imagenet_tf.py#L179) after line 179, comment out `ValueError`.
```
    input = g.get_tensor_by_name("input_tensor:0")
    output = g.get_tensor_by_name("softmax_tensor:0")
```
3. Add the preprocessing fn to the [utils script](https://github.com/Xilinx/graffitist/blob/dcf7f4e8a36e1efdef217b3ad160af24d22d57df/utils/imagenet_utils.py#L576) after line 576, comment out `ValueError`.
```
    image = vgg_preprocess_input_fn(image, image_size, image_size, is_training)
```

4. Run validation (assuming you have imagenet tf-records somewhere)
```
python utils/validate_imagenet_tf.py --data_dir ../imagenet/tf-records/ --model_dir /data/resnet50_v1_frozen.pb
```
#### Output:
```
Val:    [     0/   782] Time   3.588 (  3.588)  Prec@1  81.250 ( 81.250)        Prec@5  95.312 ( 95.312)
Val:    [   100/   782] Time   0.079 (  0.115)  Prec@1  73.438 ( 76.795)        Prec@5  93.750 ( 93.007)
Val:    [   200/   782] Time   0.080 (  0.098)  Prec@1  81.250 ( 77.029)        Prec@5  95.312 ( 93.377)
Val:    [   300/   782] Time   0.079 (  0.092)  Prec@1  79.688 ( 76.905)        Prec@5  93.750 ( 93.304)
Val:    [   400/   782] Time   0.080 (  0.089)  Prec@1  71.875 ( 76.870)        Prec@5  92.188 ( 93.173)
Val:    [   500/   782] Time   0.080 (  0.087)  Prec@1  78.125 ( 76.771)        Prec@5  93.750 ( 93.083)
Val:    [   600/   782] Time   0.080 (  0.086)  Prec@1  75.000 ( 76.583)        Prec@5  92.188 ( 93.051)
Val:    [   700/   782] Time   0.080 (  0.085)  Prec@1  82.812 ( 76.516)        Prec@5  92.188 ( 93.039)
Val:    [   781/   782] Time   0.270 (  0.085)  Prec@1  68.750 ( 76.522)        Prec@5  87.500 ( 93.066)
 model_dir=/data/resnet50_v1_frozen.pb prec@1=76.522 prec@5=93.066
```

> Also, where does the magic number 782 come from? Did you choose to test on that many examples? If so, why?
> 
I used a batch of 64, so there would be ~782 batches to go over 50k validation images.
> Thank you very much in advance!

@sjain-stanford , thank you very much for such a detailed answer!

Thanks @sjain-stanford! With that preprocessing, @ens-lg4 has now measured:
```
Accuracy top 1: 0.76522 (38261 of 50000)
Accuracy top 5: 0.93066 (46533 of 50000)
```
or exactly the same as you!

This is great news. However, the next big question is which preprocessing we should use for MobileNet?

An even more fundamental question is: If we can't use the same preprocessing even for the two image classification models in the Closed Division, how are we going to cope with Open Division submissions? If anybody can bring their own model and preprocessing, they may be claiming a superior accuracy simply by overfitting e.g. resizing ImageNet images to 267x272 pixels and then cropping to 232x232.

Thought I chime in and post my findings regarding the discrepancy seen in MLPerf's reported accuracy. My experiments are available at [my MLPerf/inference fork](https://github.com/ajarthurs/inference/tree/experiment).

In my first experiment, I replace MLPerf's entire PIL-based preprocessor with [a TF-based preprocessor](https://github.com/ajarthurs/inference/blob/experiment/v0.5/classification_and_detection/python/imagenet_preprocessing.py), which is mostly based on [an official ResNet-ImageNet preprocessor](https://github.com/tensorflow/models/blob/master/official/resnet/imagenet_preprocessing.py). The result is 76.52% accuracy ([see related commit](https://github.com/ajarthurs/inference/commit/d6acd67f64e8b389759fc2c4491c3b970de49072)).
```
[TestScenario.SingleStream qps=5317.70, mean=0.0092, time=9.40, acc=76.52, queries=50000, tiles=50.0:0.0089,80.0:0.0099,90.0:0.0103,95.0:0.0106,99.0:0.0112,99.9:0.0130
```

In my second experiment and in light of the difference in corner-alignment (or lack thereof) between PIL and TF, pointed out by @itayhubara, I substitute in TF's image resizer (`tf.compat.v1.image.resize`; [see related commit](https://github.com/ajarthurs/inference/commit/86d6e653672688857c4956104ad2187b5ca8f266)). The result is 75.28% accuracy, roughly 1% less than expected. I noticed that TF's resize output is in floating-point (`float32`) and that PIL typecasts NumPy arrays to `np.uint8`. When PIL (`Image.fromarray()`) does this typecast, it truncates the floating-point channel values. I suspect that truncation is creating the said 1% discrepancy.
```
TestScenario.SingleStream qps=5251.05, mean=0.0093, time=9.52, acc=75.28, queries=50000, tiles=50.0:0.0091,80.0:0.0101,90.0:0.0105,95.0:0.0108,99.0:0.0113,99.9:0.0127
```

cool, good to confirm. Now how we we wind yourself out of this ? I could replace PIL with tensorflow or take a look what opencv does or take another look at PIL if there is no way of getting the same as tensorflow.

@guschmue I wouldn't replace PIL with TF. As @psyhtest mentioned, some platforms don't support TF. Another reason is that TF's ResNet-ImageNet preprocessor resizes images in a non-standard fashion. PIL's resize method produces the same output as most other image processing libraries, making it easier to substitute preprocessors and keep results consistent.

Mobile/ResNet's checkpoints were trained with TF's preprocessor, so using it would yield consistently better accuracy. One consideration would be fine-tuning the model with the PIL-based preprocessor, document its accuracy in addition to the 74.97% seen with TF's checkpoint. (BTW, I get 74.99% running ef82b4be38f2ce94f2432c2c4093ba567ebb34ec).

I've not tried OpenCV but my understanding is that it will give the same results as PIL (https://stackoverflow.com/a/54497253/9730945).

agree with all you said. Another reason why I don't like the tf route is that eventually we want to have the pre-processing part of the timed path so it reflects real user workload. While this was out of scope in 0.5 it might bubble up again in the next version. That is also the reason why pre-processing is built in the app instead of using a simple script. I'll spend a little time on this today.

This https://github.com/mlperf/inference/pull/243  gets us pretty close:


rt | resnet50 | mobilenet
-- | -- | --
tf-gpu | 76.12 | 71.228
tf-cpu | 76.12 | 71.228
onnxruntime-cpu | 76.12 | 71.228


FYI, I came across this issue looking searching for something inference related. I have spent time looking at this concern in the past so I have a few cents.

All of PIL, TF preprocessing, and OpenCV2 preprocessing will have different results with the same set of model weights with all other parameters of the preprocessing being equal, the jpeg decoding + interpolation impelementation specifics will have an impact. 

The impact varies from model to model, some are much more sensitive, a training or even validation scheme that took multiple preprocessing pipelines into consideration would likely end up with a model more robust to all.

I have some measurements recorded for reference. The models are all ResNet-50, model compatible, you can load the weights with the same PyTorch def, should also be compatible with the NVIDIA ResNet50-v1.5. The overall trend you can see in all is that the best validation score is usually associated with the interpolation method and specific library used in training.

My ResNet-50 model, trained with PyTorch PIL pipeline with a 50/50 blend of bilinear and bicubic interpolation.
PIL bicubic - 78.47
PIL bilinear - 78.36
CV2 bicubic - 77.51
CV2 bilinear - 78.02
Tensorflow bicubic - 77.52
Tensorflow bilinear -  77.95

PyTorch torchvision ResNet-50 trained with PIL pipeline with bilinear interpolation 
PIL bicubic - 75.858
PIL bilinear - 76.13
CV2 bicubic - 74.096
CV2 bilinear - 75.308
TF bicubic - 74.046
TF bilinear - 75.332

MXNet Gluon Resnet-50 V1B trained in MxNet by Amazon, ported to PyTorch by me. MxNet gluon models look like they're trained with bilinear interpolation using OpenCV2 underneath
PIL bicubic - 77.578
PIL bilinear - 77.394
CV2 bicubic - 77.412
CV2 bilinear - 77.676
Tensorflow bicubic - 77.412
Tensorflow bilinear -  77.646

I don't have weights written down for a TF trained model running with PIL validation, but I did run some tests when porting EfficientNet models and recall there being 0.4-0.5ish differences in top-1 for the b0 that I was looking at initially.

So, they're all different, but OpenCV and TF do seem to be pretty close most of the time, I don't have enough data to say it's a rule, but seems that PIL is the odd duck out in this trio. PIL is also the slowest, with OpenCV being much faster at runtime.



```trained with PyTorch PIL pipeline with a 50/50 blend of bilinear and bicubic``` ... very clever, love it.

For the accuracy - its pretty close now, let me check with some folks if we need to get closer.

we agreed to switch to OpenCV.