Repository: mlcommons/inference
Issue: Need help with converting SSD-ResNet34 in TF to TFLite
Root cause: External Dependency Breakage

As many embedded and mobile systems may not be suitable for running full TF models, we would like to port the SSD-ResNet34 model to TFLite. Unfortunately, the generic SSD architecture has control flow, which TFLite currently [does not support](https://github.com/tensorflow/tensorflow/issues/28485). Even when this support is introduced, it will take some time for vendors to catch up.

We have [successfully converted SSD-MobileNet](https://github.com/ctuning/ck-mlperf/tree/master/package/model-tflite-mlperf-ssd-mobilenet) by adapting instructions from [Google's blog](https://medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193). The trick was to cut off the NMS operator, and perform it outside of the graph as a custom operation.

We were hoping that the process for SSD-ResNet would be quite similar. Unfortunately, the model contains more operations that are not supported by the TF-to-TFLite converter. Here's a brief summar of what we've tried so far.

```
$ bazel run -c opt tensorflow/lite/toco:toco -- \
--input_file=/home/ivan/Downloads/tflite_ssd_resnet-34/ssd_resnet34_mAP_20.2.pb \
--output_file=/home/ivan/Downloads/tflite_ssd_resnet-34/detect.tflite \
--input_arrays='image' \
--output_arrays='detection_scores','detection_classes','detection_bboxes' \
--inference_type=FLOAT \
--change_concat_input_ranges=false \
--mean_values=128 \
--std_values=128
```

> Some of the operators in the model are not supported by the standard TensorFlow Lite runtime. If those are native TensorFlow operators, you might be able to use the extended runtime by passing --enable_select_tf_ops, or by setting target_ops=TFLITE_BUILTINS,SELECT_TF_OPS when calling tf.lite.TFLiteConverter(). Otherwise, if you have a custom implementation for them you can disable this error with --allow_custom_ops, or by setting allow_custom_ops=True when calling tf.lite.TFLiteConverter(). Here is a list of builtin operators you are using: ADD, CAST, CONCATENATION, CONV_2D, DIV, EXP, EXPAND_DIMS, GATHER, GREATER, LESS, LOGICAL_AND, MAXIMUM, MAX_POOL_2D, MINIMUM, MUL, PACK, PAD, RANGE, RESHAPE, SHAPE, SOFTMAX, SPLIT_V, STRIDED_SLICE, SUB, TOPK_V2, UNPACK. Here is a list of operators for which you will need custom implementations: **Enter, Exit, LoopCond, Merge, NonMaxSuppressionV2, Switch, TensorArrayGatherV3, TensorArrayReadV3, TensorArrayScatterV3, TensorArraySizeV3, TensorArrayV3, TensorArrayWriteV3**.

When adding the `--enable_select_tf_ops` key:
```
bazel run -c opt tensorflow/lite/toco:toco -- \
--input_file=/home/ivan/Downloads/tflite_ssd_resnet-34/ssd_resnet34_mAP_20.2.pb \
--output_file=/home/ivan/Downloads/tflite_ssd_resnet-34/detect.tflite \
--input_arrays='image' \
--output_arrays='detection_scores','detection_classes','detection_bboxes' \
--inference_type=FLOAT \
--change_concat_input_ranges=false \
--mean_values=128 \
--std_values=128 \
--enable_select_tf_ops
```
the list of required custom ops is reduced:
> Some of the operators in the model are not supported by the standard TensorFlow Lite runtime and are not recognized by TensorFlow. If you have a custom implementation for them you can disable this error with --allow_custom_ops, or by setting allow_custom_ops=True when calling tf.lite.TFLiteConverter(). Here is a list of builtin operators you are using: ADD, CAST, CONCATENATION, CONV_2D, DIV, EXP, EXPAND_DIMS, GATHER, GREATER, LESS, LOGICAL_AND, MAXIMUM, MAX_POOL_2D, MINIMUM, MUL, PACK, PAD, RANGE, RESHAPE, SHAPE, SOFTMAX, SPLIT_V, STRIDED_SLICE, SUB, TOPK_V2, UNPACK. Here is a list of operators for which you will need custom implementations: **Merge**, **Switch**.

We can enjoy error-free conversion without enabling `--enable_select_tf_ops` and `allow_custom_ops`, by cutting off rather large parts of the graph (blocks "Reshape", "map" and "map_1" ) via:
```
$ bazel run -c opt tensorflow/lite/toco:toco -- \
--input_file=/home/ivan/Downloads/tflite_ssd_resnet-34/ssd_resnet34_mAP_20.2.pb \
--output_file=/home/ivan/Downloads/tflite_ssd_resnet-34/detect.tflite \
--input_arrays='image' \
--output_arrays='ssd1200/concat','map/concat_1' \
--inference_type=FLOAT \
--change_concat_input_ranges=false \
--mean_values=128 \
--std_values=128
```
Unfortunately, we are not sure what to do with the partially converted graph, as reimplementing the missing parts would be far from trivial. (At the very least, we would need to understand in detail their semantics.)

From the last meeting, I understood that the current variant of SSD-ResNet may not be final. So if the route of converting the current variants does not look particular promising, the question is whether this model could be made more TFLite-friendly like.

From email correspondence between Jared Duke and Anton Lokhmotov on 28/Jun/2019:

#### Jared
> Support for SSD-ResNet34 in TFLite is on our radar, but hasn't been prioritized as there's not a ton of demand for executing such a model on mobile/embedded devices. That said, a priority for the TFLite team in Q3 is supporting control flow, which should help solve some of the conversion issues, but it's unlikely we'll have a functional .tflite variant for this model for use in 0.5 submissions.

#### Anton
> Understood. I'm wondering if we can try to produce an SSD-ResNet model that's more TFLite-friendly. I mean, we can run ResNet and SSD-MobileNet - why not just replace MobileNet layers with ResNet ones?

#### Jared
> Something like that could certainly work, there's some non-trivial plumbing that went into supporting the custom fast NMS operator in .tflite for SSD-MobileNet, but it's possible the same logic would map cleanly to SSD-ResNet.

#### Anton
> This is what I was hoping for all along, since we decided to include SSD-ResNet into v0.5. But this may make the TF version diverge from the PyTorch one - a possible equivalence nightmare...

TF lite:  we can get this resolved in v0.6.

> TF lite: we can get this resolved in v0.6.

Not sure about that if the model stays the same.

@psyhtest  I update channel_last pb model on https://zenodo.org/record/3708755#.XmszXqgzY2w. At the same time, I share pb2tflite tools at: https://github.com/lji72/inference/tree/master/others/cloud/single_stage_detector/tensorflow and have given pull request. pb2tflite process has sucessed using tf.1.12 and NHWC format pb model