Repository: tensorflow/model-analysis
Issue: Support for multi-dimensional labels
Root cause: Unimplemented Feature Gap

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: stock configuration with custom model
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS 10.15.5
-   **TensorFlow Model Analysis installed from (source or binary)**: binary
-   **TensorFlow Model Analysis version (use command below)**: 0.22.2 and 0.24.0
-   **Python version**: 3.7
-   **Jupyter Notebook version**: 2.1.5
-   **Exact command to reproduce**:

### Describe the problem

I encounter the following error when running the evaluator on a model where the output shape is (-1, -1, 5). The label is a variable length sparse categorical feature. Are multi-dimensional labels currently supported? I am unsure whether this qualifies as a bug or a feature request.

```
The given SavedModel SignatureDef contains the following input(s):
  inputs['examples'] tensor_info:
      dtype: DT_STRING
      shape: (-1)
      name: serving_default_examples:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['output_0'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, -1, 5)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
```

### Source code / logs

```
WARNING:absl:Tensorflow version (2.3.1) found. Note that TFMA support for TF 2.0 is currently in beta
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-18-1ba4457d108a> in <module>
     41     file_format='tfrecords',
     42     output_path='test',
---> 43     extractors=None)

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow_model_analysis/api/model_eval_lib.py in run_model_analysis(eval_shared_model, eval_config, data_location, file_format, output_path, extractors, evaluators, writers, pipeline_options, slice_spec, write_config, compute_confidence_intervals, min_slice_size, random_seed_for_testing, schema)
   1124             random_seed_for_testing=random_seed_for_testing,
   1125             tensor_adapter_config=tensor_adapter_config,
-> 1126             schema=schema))
   1127     # pylint: enable=no-value-for-parameter
   1128 

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/pipeline.py in __exit__(self, exc_type, exc_val, exc_tb)
    553     try:
    554       if not exc_type:
--> 555         self.run().wait_until_finish()
    556     finally:
    557       self._extra_context.__exit__(exc_type, exc_val, exc_tb)

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/pipeline.py in run(self, test_runner_api)
    532         finally:
    533           shutil.rmtree(tmpdir)
--> 534       return self.runner.run_pipeline(self, self._options)
    535     finally:
    536       shutil.rmtree(self.local_tempdir, ignore_errors=True)

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/direct/direct_runner.py in run_pipeline(self, pipeline, options)
    117       runner = BundleBasedDirectRunner()
    118 
--> 119     return runner.run_pipeline(pipeline, options)
    120 
    121 

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_pipeline(self, pipeline, options)
    171 
    172     self._latest_run_result = self.run_via_runner_api(
--> 173         pipeline.to_runner_api(default_environment=self._default_environment))
    174     return self._latest_run_result
    175 

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_via_runner_api(self, pipeline_proto)
    181     # TODO(pabloem, BEAM-7514): Create a watermark manager (that has access to
    182     #   the teststream (if any), and all the stages).
--> 183     return self.run_stages(stage_context, stages)
    184 
    185   @contextlib.contextmanager

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_stages(self, stage_context, stages)
    338           stage_results = self._run_stage(
    339               runner_execution_context,
--> 340               bundle_context_manager,
    341           )
    342           monitoring_infos_by_stage[stage.name] = (

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in _run_stage(self, runner_execution_context, bundle_context_manager)
    517               input_timers,
    518               expected_timer_output,
--> 519               bundle_manager)
    520 
    521       final_result = merge_results(last_result)

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in _run_bundle(self, runner_execution_context, bundle_context_manager, data_input, data_output, input_timers, expected_timer_output, bundle_manager)
    555 
    556     result, splits = bundle_manager.process_bundle(
--> 557         data_input, data_output, input_timers, expected_timer_output)
    558     # Now we collect all the deferred inputs remaining from bundle execution.
    559     # Deferred inputs can be:

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in process_bundle(self, inputs, expected_outputs, fired_timers, expected_output_timers, dry_run)
    939     with thread_pool_executor.shared_unbounded_instance() as executor:
    940       for result, split_result in executor.map(execute, zip(part_inputs,  # pylint: disable=zip-builtin-not-iterating
--> 941                                                             timer_inputs)):
    942         split_result_list += split_result
    943         if merged_result is None:

~/.pyenv/versions/3.7.7/lib/python3.7/concurrent/futures/_base.py in result_iterator()
    596                     # Careful not to keep a reference to the popped future
    597                     if timeout is None:
--> 598                         yield fs.pop().result()
    599                     else:
    600                         yield fs.pop().result(end_time - time.monotonic())

~/.pyenv/versions/3.7.7/lib/python3.7/concurrent/futures/_base.py in result(self, timeout)
    433                 raise CancelledError()
    434             elif self._state == FINISHED:
--> 435                 return self.__get_result()
    436             else:
    437                 raise TimeoutError()

~/.pyenv/versions/3.7.7/lib/python3.7/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/utils/thread_pool_executor.py in run(self)
     42       # If the future wasn't cancelled, then attempt to execute it.
     43       try:
---> 44         self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))
     45       except BaseException as exc:
     46         # Even though Python 2 futures library has #set_exection(),

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in execute(part_map_input_timers)
    935           input_timers,
    936           expected_output_timers,
--> 937           dry_run)
    938 
    939     with thread_pool_executor.shared_unbounded_instance() as executor:

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in process_bundle(self, inputs, expected_outputs, fired_timers, expected_output_timers, dry_run)
    835             process_bundle_descriptor.id,
    836             cache_tokens=[next(self._cache_token_generator)]))
--> 837     result_future = self._worker_handler.control_conn.push(process_bundle_req)
    838 
    839     split_results = []  # type: List[beam_fn_api_pb2.ProcessBundleSplitResponse]

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py in push(self, request)
    350       self._uid_counter += 1
    351       request.instruction_id = 'control_%s' % self._uid_counter
--> 352     response = self.worker.do_instruction(request)
    353     return ControlFuture(request.instruction_id, response)
    354 

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py in do_instruction(self, request)
    478       # E.g. if register is set, this will call self.register(request.register))
    479       return getattr(self, request_type)(
--> 480           getattr(request, request_type), request.instruction_id)
    481     else:
    482       raise NotImplementedError

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py in process_bundle(self, request, instruction_id)
    513         with self.maybe_profile(instruction_id):
    514           delayed_applications, requests_finalization = (
--> 515               bundle_processor.process_bundle(instruction_id))
    516           monitoring_infos = bundle_processor.monitoring_infos()
    517           monitoring_infos.extend(self.state_cache_metrics_fn())

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py in process_bundle(self, instruction_id)
    981       for op in self.ops.values():
    982         _LOGGER.debug('finish %s', op)
--> 983         op.finish()
    984 
    985       # Close every timer output stream

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/worker/operations.cpython-37m-darwin.so in apache_beam.runners.worker.operations.PGBKCVOperation.finish()

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/worker/operations.cpython-37m-darwin.so in apache_beam.runners.worker.operations.PGBKCVOperation.finish()

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/runners/worker/operations.cpython-37m-darwin.so in apache_beam.runners.worker.operations.PGBKCVOperation.output_key()

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py in compact(self, accumulator)
    362   def compact(self, accumulator: Any) -> Any:
    363     self._num_compacts.inc(1)
--> 364     return super(_ComputationsCombineFn, self).compact(accumulator)
    365 
    366   def extract_output(self, accumulator: Any) -> metric_types.MetricsDict:

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/transforms/combiners.py in compact(self, accumulator)
    717 
    718   def compact(self, accumulator):
--> 719     return [c.compact(a) for c, a in zip(self._combiners, accumulator)]
    720 
    721   def extract_output(self, accumulator):

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/apache_beam/transforms/combiners.py in <listcomp>(.0)
    717 
    718   def compact(self, accumulator):
--> 719     return [c.compact(a) for c, a in zip(self._combiners, accumulator)]
    720 
    721   def extract_output(self, accumulator):

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py in compact(self, accumulator)
    558       self, accumulator: _CompilableMetricsAccumulator
    559   ) -> _CompilableMetricsAccumulator:
--> 560     self._process_batch(accumulator)
    561     return accumulator
    562 

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py in _process_batch(self, accumulator)
    524       for metric_index, metric in enumerate(self._metrics[output_name]):
    525         metric.reset_states()
--> 526         metric.update_state(*inputs)
    527         accumulator.add_weights(output_index, metric_index,
    528                                 metric.get_weights())

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/keras/utils/metrics_utils.py in decorated(metric_obj, *args, **kwargs)
     88 
     89     with tf_utils.graph_context_for_symbolic_tensors(*args, **kwargs):
---> 90       update_op = update_state_fn(*args, **kwargs)
     91     if update_op is not None:  # update_op will be None in eager execution.
     92       metric_obj.add_update(update_op)

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/keras/metrics.py in update_state_fn(*args, **kwargs)
    174         control_status = ag_ctx.control_status_ctx()
    175         ag_update_state = autograph.tf_convert(obj_update_state, control_status)
--> 176         return ag_update_state(*args, **kwargs)
    177     else:
    178       if isinstance(obj.update_state, def_function.Function):

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    253       try:
    254         with conversion_ctx:
--> 255           return converted_call(f, args, kwargs, options=options)
    256       except Exception as e:  # pylint:disable=broad-except
    257         if hasattr(e, 'ag_error_metadata'):

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py in converted_call(f, args, kwargs, caller_fn_scope, options)
    455   if conversion.is_in_whitelist_cache(f, options):
    456     logging.log(2, 'Whitelisted %s: from cache', f)
--> 457     return _call_unconverted(f, args, kwargs, options, False)
    458 
    459   if ag_ctx.control_status_ctx().status == ag_ctx.Status.DISABLED:

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py in _call_unconverted(f, args, kwargs, options, update_cache)
    337 
    338   if kwargs is not None:
--> 339     return f(*args, **kwargs)
    340   return f(*args)
    341 

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/keras/metrics.py in update_state(self, y_true, y_pred, sample_weight)
    601       Update op.
    602     """
--> 603     y_true = math_ops.cast(y_true, self._dtype)
    604     y_pred = math_ops.cast(y_pred, self._dtype)
    605     [y_true, y_pred], sample_weight = \

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
    199     """Call target, and fall back on dispatchers if there is a TypeError."""
    200     try:
--> 201       return target(*args, **kwargs)
    202     except (TypeError, ValueError):
    203       # Note: convert_to_eager_tensor currently raises a ValueError, not a

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py in cast(x, dtype, name)
    918       # allows some conversions that cast() can't do, e.g. casting numbers to
    919       # strings.
--> 920       x = ops.convert_to_tensor(x, name="x")
    921       if x.dtype.base_dtype != base_type:
    922         x = gen_math_ops.cast(x, base_type, name=name)

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types)
   1497 
   1498     if ret is None:
-> 1499       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
   1500 
   1501     if ret is NotImplemented:

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/framework/tensor_conversion_registry.py in _default_conversion_function(***failed resolving arguments***)
     50 def _default_conversion_function(value, dtype, name, as_ref):
     51   del as_ref  # Unused.
---> 52   return constant_op.constant(value, dtype, name=name)
     53 
     54 

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name)
    262   """
    263   return _constant_impl(value, dtype, shape, name, verify_shape=False,
--> 264                         allow_broadcast=True)
    265 
    266 

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py in _constant_impl(value, dtype, shape, name, verify_shape, allow_broadcast)
    273       with trace.Trace("tf.constant"):
    274         return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
--> 275     return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
    276 
    277   g = ops.get_default_graph()

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py in _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
    298 def _constant_eager_impl(ctx, value, dtype, shape, verify_shape):
    299   """Implementation of eager constant."""
--> 300   t = convert_to_eager_tensor(value, ctx, dtype)
    301   if shape is None:
    302     return t

~/.pyenv/versions/3.7.7/envs/beam-3.7/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py in convert_to_eager_tensor(value, ctx, dtype)
     96       dtype = dtypes.as_dtype(dtype).as_datatype_enum
     97   ctx.ensure_initialized()
---> 98   return ops.EagerTensor(value, ctx.device_name, dtype)
     99 
    100 

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type numpy.ndarray).
​
```

I'm not sure which metrics you are using, but I don't think tensorflow supports this. You can try to create an instance of your tf.keras.metrics.xxx metric and then call update_state(y_true, y_pred) to see if it works for your metric.

Neither the Keras metrics nor `convert_to_eager_tensor() ` have any issues with `numpy.ndarray` when valid shapes are provided. Without more debugging I can't say what values TFMA is feeding into `y_true`, but it looks as if it is not processing these features correctly.

```python
import tensorflow as tf
import numpy

m = tf.keras.metrics.Accuracy()
m.update_state(numpy.array([[[0, 1, 0, 0, 0], [0, 0, 0, 0, 0]], [[1, 0, 0, 0, 0], [0, 0, 0, 0, 0]]]), 
               numpy.array([[[0, 1, 0, 0, 0], [0, 0, 0, 0, 0]], [[1, 0, 0, 0, 0], [0, 0, 0, 0, 0]]]))
m.result().numpy()
# 1.0
```

```python
from tensorflow.python.framework.constant_op import convert_to_eager_tensor
from tensorflow.python.eager import context

ctx = context.context()
convert_to_eager_tensor(numpy.array([[[0, 1, 0, 0, 0], [0, 0, 0, 0, 0]], [[1, 0, 0, 0, 0], [0, 0, 0, 0, 0]]]), ctx, tf.int32)
# <tf.Tensor: shape=(2, 2, 5), dtype=int32, numpy=
# array([[[0, 1, 0, 0, 0],
#         [0, 0, 0, 0, 0]],
# 
#        [[1, 0, 0, 0, 0],
#         [0, 0, 0, 0, 0]]], dtype=int32)>
```

Sorry for the late response. I believe the issue is caused because TFMA batches the requests that are sent to the TF calls. I think that your labels are not the same same dimensions (i.e. (-1, -1, 5) implies the second dimension is not fixed). If you update your example to have different sizes for different metrics then you will get an error:

```python
import tensorflow as tf
import numpy

m = tf.keras.metrics.Accuracy()
m.update_state(numpy.array([[[0, 1, 0, 0, 0]], [[1, 0, 0, 0, 0], [0, 0, 0, 0, 0]]]), 
               numpy.array([[[0, 1, 0, 0, 0]], [[1, 0, 0, 0, 0], [0, 0, 0, 0, 0]]]))
m.result().numpy()
# 1.0
```

Note that I used lists as input, but TFMA also builds numpy from lists of numpy arrays so the actual error would be unable to convert np.ndarray to tensor instead of list.

In short, we support multi-dimension arrays if they are the same size. There is upcoming support for tf-ranking to pad arrays that are not the same size, but this will add -1 labels which isn't what you want.



Closing this as the comment trace has not been updated for some time.Please feel free to reopen based on above comments.