Repository: mlcommons/inference
Issue: SSD-Resnet34 Inference- training script in the model source directory is broken
Root cause: Validation Gap

SSD-Resnet34 Inference
Model source directory- https://github.com/lji72/inference/tree/tf_ssd_resent34_align_onnx/others/cloud/single_stage_detector/tensorflow/train

Problem statement- I need a NHWC model for SSD-Resnet34. The model at provided by MLPerf is in NCHW format, which is not supported by TF backend for CPU.  Since there is no good tool to do that, training the model in NHWC seems like a possible solution. However, when trying to run the training script following is the error I am encountering. 

Script- train_ssd_large.py
this gives following error
Traceback (most recent call last):
  File "eval_ssd_large.py", line 383, in <module>
    tf.app.run()
  File "/username/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/username/anaconda3/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/username/anaconda3/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "eval_ssd_large.py", line 379, in main
    evaluate(args)
  File "eval_ssd_large.py", line 349, in evaluate
    det_results = list(pred_results)
  File "/nfs/username/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 619, in predict
    features, None, ModeKeys.PREDICT, self.config)
  File "/nfs/username/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1146, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "eval_ssd_large.py", line 286, in ssd_model_fn
    pred_results = tf.map_fn(parse_bboxes_fn, (cls_pred, bboxes_pred), dtype=(tf.float32, tf.float32, tf.float32), back_prop=False)
  File "/username/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/map_fn.py", line 268, in map_fn
    maximum_iterations=n)
  File "/username/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3501, in while_loop
    return_same_structure)
  File "/username/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3012, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/username/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2937, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/username/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3456, in <lambda>
    body = lambda i, lv: (i + 1, orig_body(*lv))
  File "/username/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/map_fn.py", line 257, in compute
    packed_fn_values = fn(packed_values)
  File "eval_ssd_large.py", line 285, in <lambda>
    parse_bboxes_fn = lambda x: parse_by_class_fixed_bboxes(x[0], x[1], params)
  File "eval_ssd_large.py", line 232, in parse_by_class_fixed_bboxes
    detection_bboxes = tf.cond(cur_num < keep_max_boxes, lambda: tf.concat([detection_bboxes, tf.zeros(shape=(params['keep_max_boxes'] - cur_num, 4), dtype=tf.float32)], axis=0), lambda: detection_bboxes)
NameError: name 'cur_num' is not defined


@lji72


@ christ1ne  @Shama022  
I have fixed it at:
https://github.com/lji72/inference/tree/tf_ssd_resent34_align_onnx/others/cloud/single_stage_detector/tensorflow/train