Repository: mlcommons/inference
Issue: [VISION] Input 0 of node ToFloat was passed float from image_tensor:0 incompatible with expected uint8
Root cause: Algorithmic Error

Encountered the following error when trying to build and run the docker container in the `mlcommons
/inference/vision/classification_and_detection/run_and_time.sh` `r1.1` branch. The error log:

```ruby
Instructions for updating:
Use tf.gfile.GFile.
WARNING:tensorflow:From /opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/tools/strip_unused_lib.py:86: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
WARNING:tensorflow:From /opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/tools/optimize_for_inference_lib.py:113: remove_training_nodes (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.remove_training_nodes`
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 427, in import_graph_def
    graph._c_graph, serialized, options)  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input 0 of node ToFloat was passed float from image_tensor:0 incompatible with expected uint8.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "python/main.py", line 565, in <module>
    main()
  File "python/main.py", line 443, in main
    model = backend.load(args.model, inputs=args.inputs, outputs=args.outputs)
  File "/mlperf/python/backend_tf.py", line 58, in load
    g = tf.compat.v1.import_graph_def(graph_def, name='')
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 431, in import_graph_def
    raise ValueError(str(e))
ValueError: Input 0 of node ToFloat was passed float from image_tensor:0 incompatible with expected uint8.
```

The error occurs from TensorFlow 1.14 to TensorFlow 2.6.0. It happens when running 
```
./run_and_time.sh tf ssd-mobilenet gpu
```
More specifically, entering the docker built with `./run_and_time.sh` and run the following command will also reproduce the error.
```
python python/main.py \
  --dataset='coco-300' \
  --dataset-path=$DATA_DIR \
  --inputs="image_tensor:0" --max-batchsize=32 --mlperf_conf='./mlperf.conf' \
  --model=$MODEL_DIR \
  --model-name='ssd-mobilenet' --output "/output" \
  --outputs="num_detections:0,detection_boxes:0,detection_classes:0,detection_scores:0" \
  --profile='ssd-mobilenet-tf' --scenario='SingleStream' --threads=12 --user_conf='user.conf'
```

---
Proposed fix is to change a few lines in `inference/vision/classification_and_detection/python/backend_tf.py`. From
```python
for as_datatype_enum in [dtypes.float32.as_datatype_enum, dtypes.uint8.as_datatype_enum]:
    try:
        optimized_graph_def = optimize_for_inference(graph_def, [item.split(':')[0] for item in inputs],
                [item.split(':')[0] for item in outputs], as_datatype_enum, False)
        graph_def = optimized_graph_def
        break
    except ValueError:
        pass
g = tf.compat.v1.import_graph_def(graph_def, name='')
```
to
```python
for as_datatype_enum in [dtypes.float32.as_datatype_enum, dtypes.uint8.as_datatype_enum]:
    try:
        optimized_graph_def = optimize_for_inference(graph_def, [item.split(':')[0] for item in inputs],
                [item.split(':')[0] for item in outputs], as_datatype_enum, False)
        graph_def = optimized_graph_def
        g = tf.compat.v1.import_graph_def(graph_def, name='')
        break
    except ValueError:
        pass
```


The proposed fix was tested against the following TensorFlow versions:
- `2.6.0`: fixed this specific error but some other error occurs
- `2.5.0`: fixed this specific error but some other error occurs
- `2.0.0`: works
- `1.14.0`: works

Error log for the other error that occurs after fixing `Input 0 of node ToFloat was passed float from image_tensor:0 incompatible with expected uint8` :
```ruby
2021-09-20 15:37:41.436870: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2021-09-20 15:37:41.438997: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
    return fn(*args)
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1360, in _run_fn
    target_list, run_metadata)
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1453, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/BatchNorm/batchnorm/mul_1}}]]
	 [[Postprocessor/BatchMultiClassNonMaxSuppression/map/while/Exit_5/_67]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/BatchNorm/batchnorm/mul_1}}]]
0 successful operations.
0 derived errors ignored.
```

Adding @DZFromArm 
Related to https://github.com/mlcommons/inference/commit/cd105ff3a5c7f918ee0a153ef325a3610f59751f 

Thank you @cfRod.

@DZFromArm It appears that your commit [cd105ff](https://github.com/mlcommons/inference/commit/cd105ff3a5c7f918ee0a153ef325a3610f59751f) that went into r0.7 was "optimized" as [b62030a](https://github.com/mlcommons/inference/commit/b62030adda3293904e3d0e756a2218faf166fc45) to get into r1.1?

Thanks @cfRod, @psyhtest

Just checked that https://github.com/mlcommons/inference/commit/cd105ff3a5c7f918ee0a153ef325a3610f59751f works with the latest TF 2.6 and it went into the `r0.7` branch, but was broken by https://github.com/mlcommons/inference/commit/b62030adda3293904e3d0e756a2218faf166fc45 when it gets to `master` and `r1.1` branch.

Thanks for confirming @elimkwan ! Would you mind making a PR against master to fix this issue _&agrave; la_ [`cd105ff`](https://github.com/mlcommons/inference/commit/cd105ff3a5c7f918ee0a153ef325a3610f59751f)?