Repository: mlcommons/inference
Issue: Unable to install the coco dataset from ck
Root cause: Documentation Deficiency

The step to install the coco dataset from ck in the steps listed out in the [edge/object_detection](https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_resnet50) readme doesn't work. 

Specifically, `ck install package:dataset-coco-2014`.

Dear @crearo,

Please see my comment [here](https://github.com/mlperf/inference/issues/69#issuecomment-484715951). Package names [may change](https://github.com/mlperf/inference/issues/69#issuecomment-486437386), which is exactly what happened in this case. Having said that, I will prepare a PR to exclude [SSD-ResNet50](https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_resnet50), as you are not the first to stumble over this model.

For [SSD-MobileNet](https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_mobilenet), you can install COCO 2017 as follows:
```
$ ck install package --tags=dataset,coco,2017
```

Here we go: https://github.com/mlperf/inference/pull/89/

I suggest you try to play with [SSD-MobileNet](https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_mobilenet) (specifically, [tf-py](https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_mobilenet/tf-py)).

Thanks a lot! Yes, I ended up trying SSD-MobileNet. If I may ask, why is SSD-ResNet not a part of MLPerf?

Also, I'll let the PR close the issue.

That's great @crearo! Please do not hesitate to open issues if you encounter any problems. We would like to make benchmarking experience as smooth as possible for a broad user base, so feedback from users like yourself is very helpful to iron out bugs.

The MLPerf Inference working group has decided to replace SSD-ResNet50 with SSD-ResNet34 to align with Training. So we are expecting SSD-ResNet34 to come back with a vengeance in a couple of weeks. There's already a PyTorch model, but we are waiting for a TensorFlow one. Stay tuned!

Thanks for your kind words! I intend to use MLPerf for benchmarking several hardware devices that I have - the Google Coral, Nvidia Xavier, and Qualcomm RedDragon.

And ah okay, as long as one of ResNet34/50 is still there, sounds good!

Hi @crearo, That's quite a collection you've got there! Please be aware that reference implementations are not required to be particularly performant. I mean, the TFLite version will probably be quite fast on the CPU of Google's Coral's, but I don't know how to get it TPU-accelerated. Similarly, you probably want to use TensorRT on Nvidia's Xavier and SNPE on Qualcomm's RedDragon.

Unofficially (taking my MLPerf hat off!), this is what we are aiming for with the [Collective Knowledge](http://cknowledge.org) initiative - to create CK wrappers for many vendor-specific tools to allow platforms to be benchmarked in a uniform and reproducible way across diverse hardware, workloads, datasets, etc. For example, you can see results from the 1st ACM [ReQuEST](http://cknowledge.org/request) competition on an interactive dashboard [here](http://cknowledge.org/dashboard/request.asplos18) or an (unofficial) MobileNets dashboard  [here](http://cknowledge.org/dashboard/mlperf.mobilenets).

We have made good start. For example, on your platforms you can also benchmark CPU-optimized [ArmNN](https://developer.arm.com/ip-products/processors/machine-learning/arm-nn) implementations in a [very similar way](https://github.com/ARM-software/armnn-mlperf) to the reference once. Instead of running:
```
$ ck benchmark program:image-classification-tflite ...
```
you can use:
```
$ ck benchmark program:image-classification-armnn-tflite --env.USE_NEON ...
```
(ArmNN parses TFLite models and accelerates them with NEON vector instructions.)

Imagine being able to run:
```
$ ck benchmark program:image-classification-tflite-tpu
$ ck benchmark program:image-classification-tensorrt
$ ck benchmark program:image-classification-snpe-cpu
$ ck benchmark program:image-classification-snpe-gpu
$ ck benchmark program:image-classification-snpe-dsp
$ ck benchmark program:image-classification-snpe-npu
...
```
and visualize all the results on the same dashboard!

Feel free to [ping me](mailto:anton@dividiti.com) if you are interested to join this initiative!

I read up on `ck` and it sounds like something we really need with all the different hardware vendors and their various SDKs. Thanks, I'll get in touch!

@crearo Thank _you_!