Repository: tensorflow/model-analysis
Issue: Multilabel Metrics TFX
Root cause: Validation Gap

#14 # System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**:  No
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: CentOS Linux 7 (Core)
-   **TensorFlow Model Analysis installed from (source or binary)**: Binary
-   **TensorFlow Model Analysis version (use command below)**:0.24.3
-   **Python version**:3.6.9
-   **Jupyter Notebook version**:6.0.1
-   **Exact command to reproduce**:

### Describe the problem

I am following the tfx tutorial [https://www.tensorflow.org/tfx/tutorials/tfx/components_keras](url), but with multilabel data.(number of categories is 5). Here is the output from example_gen
```
{
 'Text': array([b'Football fans looking forward to seeing the renewal of the rivalry between Cristiano Ronaldo and Lionel Messi were made to wait a while longer after the Portuguese forward was forced to miss Juventus' Champions League tie against Barcelona on Wednesday.'],dtype=object),
 'Headline': array([b"Lionel Messi scores as Cristiano Ronaldo misses Barcelona's victory over Juventus."], dtype=object),
 'categories': array([b'Sports'], dtype=object), 
}

{
 'Text': array([b'COVID-19 has changed fan behavior and accelerated three to five years of technology adoption into six months'],dtype=object),
 'Headline': array([b"How Technology Is Improving Fan Transactions at Sports Venues"], dtype=object),
 'categories': array([b'Sports', b'Science and Technology'], dtype=object), 
}
```
Output from tf transform:
```
{
 'Text': array([b'Football fans looking forward to seeing the renewal of the rivalry between Cristiano Ronaldo and Lionel Messi were made to wait a while longer after the Portuguese forward was forced to miss Juventus' Champions League tie against Barcelona on Wednesday.'],dtype=object),
 'Headline': array([b"Lionel Messi scores as Cristiano Ronaldo misses Barcelona's victory over Juventus."], dtype=object),
 'categories': array([1., 0., 0., 0., 0.], dtype=object), 
}

{
 'Text_xf': array([b'COVID-19 has changed fan behavior and accelerated three to five years of technology adoption into six months'],dtype=object),
 'Headline_xf': array([b"How Technology Is Improving Fan Transactions at Sports Venues"], dtype=object),
 'categories_xf': array([1., 1., 0., 0., 0.'], dtype=object), 
}
```
I have trained the model using Trainer and then I want to use TFMA 
``` python
metrics = [
    tf.keras.metrics.Recall(name='recall', top_k=3),
]
metrics_specs = tfma.metrics.specs_from_metrics(metrics)

eval_config=tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key="categories")],
    slicing_specs=[tfma.SlicingSpec()],
    metrics_specs=metrics_specs,
    
)

evaluator = Evaluator(
    examples=example_gen.outputs['examples'],
    model=trainer.outputs['model'],
    baseline_model=model_resolver.outputs['model'],
    eval_config=eval_config
)
context.run(evaluator)
```



### logs
```
WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7f5a27adc9d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7f5a27abc6a8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:7 out of the last 7 calls to <function recreate_function.<locals>.restored_function_body at 0x7f5a27abc0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:8 out of the last 8 calls to <function recreate_function.<locals>.restored_function_body at 0x7f59386e9c80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:9 out of the last 9 calls to <function recreate_function.<locals>.restored_function_body at 0x7f59386d39d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.PGBKCVOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.PGBKCVOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py in add_input(self, accumulator, element)
    340     for i, (c, a) in enumerate(zip(self._combiners, accumulator)):
--> 341       result = c.add_input(a, get_combiner_input(elements[0], i))
    342       for e in elements[1:]:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py in add_input(self, accumulator, element)
    576         if self._is_top_k() and label.shape != prediction.shape:
--> 577           label = metric_util.one_hot(label, prediction)
    578         accumulator.add_input(i, label, prediction, example_weight)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_util.py in one_hot(tensor, target)
    703   # indexing the -1 and then removing it after.
--> 704   tensor = np.delete(np.eye(target.shape[-1] + 1)[tensor], -1, axis=-1)
    705   return tensor.reshape(target.shape)

IndexError: arrays used as indices must be of integer (or boolean) type

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
<ipython-input-31-952eda92fce9> in <module>
      5     eval_config=eval_config
      6 )
----> 7 context.run(evaluator)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/orchestration/experimental/interactive/interactive_context.py in run_if_ipython(*args, **kwargs)
     65       # __IPYTHON__ variable is set by IPython, see
     66       # https://ipython.org/ipython-doc/rel-0.10.2/html/interactive/reference.html#embedding-ipython.
---> 67       return fn(*args, **kwargs)
     68     else:
     69       absl.logging.warning(

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/orchestration/experimental/interactive/interactive_context.py in run(self, component, enable_cache, beam_pipeline_args)
    180         telemetry_utils.LABEL_TFX_RUNNER: runner_label,
    181     }):
--> 182       execution_id = launcher.launch().execution_id
    183 
    184     return execution_result.ExecutionResult(

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/orchestration/launcher/base_component_launcher.py in launch(self)
    203                          execution_decision.input_dict,
    204                          execution_decision.output_dict,
--> 205                          execution_decision.exec_properties)
    206 
    207     absl.logging.info('Running publisher for %s',

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/orchestration/launcher/in_process_component_launcher.py in _run_executor(self, execution_id, input_dict, output_dict, exec_properties)
     65         executor_context)  # type: ignore
     66 
---> 67     executor.Do(input_dict, output_dict, exec_properties)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/components/evaluator/executor.py in Do(self, input_dict, output_dict, exec_properties)
    258            output_path=output_uri,
    259            slice_spec=slice_spec,
--> 260            tensor_adapter_config=tensor_adapter_config))
    261     logging.info('Evaluation complete. Results written to %s.', output_uri)
    262 

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/pipeline.py in __exit__(self, exc_type, exc_val, exc_tb)
    553     try:
    554       if not exc_type:
--> 555         self.result = self.run()
    556         self.result.wait_until_finish()
    557     finally:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/pipeline.py in run(self, test_runner_api)
    532         finally:
    533           shutil.rmtree(tmpdir)
--> 534       return self.runner.run_pipeline(self, self._options)
    535     finally:
    536       shutil.rmtree(self.local_tempdir, ignore_errors=True)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_pipeline(self, pipeline, options)
    174 
    175     self._latest_run_result = self.run_via_runner_api(
--> 176         pipeline.to_runner_api(default_environment=self._default_environment))
    177     return self._latest_run_result
    178 

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_via_runner_api(self, pipeline_proto)
    184     # TODO(pabloem, BEAM-7514): Create a watermark manager (that has access to
    185     #   the teststream (if any), and all the stages).
--> 186     return self.run_stages(stage_context, stages)
    187 
    188   @contextlib.contextmanager

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_stages(self, stage_context, stages)
    342           stage_results = self._run_stage(
    343               runner_execution_context,
--> 344               bundle_context_manager,
    345           )
    346           monitoring_infos_by_stage[stage.name] = (

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in _run_stage(self, runner_execution_context, bundle_context_manager)
    521               input_timers,
    522               expected_timer_output,
--> 523               bundle_manager)
    524 
    525       final_result = merge_results(last_result)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in _run_bundle(self, runner_execution_context, bundle_context_manager, data_input, data_output, input_timers, expected_timer_output, bundle_manager)
    559 
    560     result, splits = bundle_manager.process_bundle(
--> 561         data_input, data_output, input_timers, expected_timer_output)
    562     # Now we collect all the deferred inputs remaining from bundle execution.
    563     # Deferred inputs can be:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in process_bundle(self, inputs, expected_outputs, fired_timers, expected_output_timers, dry_run)
    943     with thread_pool_executor.shared_unbounded_instance() as executor:
    944       for result, split_result in executor.map(execute, zip(part_inputs,  # pylint: disable=zip-builtin-not-iterating
--> 945                                                             timer_inputs)):
    946         split_result_list += split_result
    947         if merged_result is None:

~/anaconda3/envs/tf2/lib/python3.6/concurrent/futures/_base.py in result_iterator()
    584                     # Careful not to keep a reference to the popped future
    585                     if timeout is None:
--> 586                         yield fs.pop().result()
    587                     else:
    588                         yield fs.pop().result(end_time - time.monotonic())

~/anaconda3/envs/tf2/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

~/anaconda3/envs/tf2/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/utils/thread_pool_executor.py in run(self)
     42       # If the future wasn't cancelled, then attempt to execute it.
     43       try:
---> 44         self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))
     45       except BaseException as exc:
     46         # Even though Python 2 futures library has #set_exection(),

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in execute(part_map_input_timers)
    939           input_timers,
    940           expected_output_timers,
--> 941           dry_run)
    942 
    943     with thread_pool_executor.shared_unbounded_instance() as executor:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in process_bundle(self, inputs, expected_outputs, fired_timers, expected_output_timers, dry_run)
    839             process_bundle_descriptor.id,
    840             cache_tokens=[next(self._cache_token_generator)]))
--> 841     result_future = self._worker_handler.control_conn.push(process_bundle_req)
    842 
    843     split_results = []  # type: List[beam_fn_api_pb2.ProcessBundleSplitResponse]

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py in push(self, request)
    351       self._uid_counter += 1
    352       request.instruction_id = 'control_%s' % self._uid_counter
--> 353     response = self.worker.do_instruction(request)
    354     return ControlFuture(request.instruction_id, response)
    355 

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py in do_instruction(self, request)
    481       # E.g. if register is set, this will call self.register(request.register))
    482       return getattr(self, request_type)(
--> 483           getattr(request, request_type), request.instruction_id)
    484     else:
    485       raise NotImplementedError

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py in process_bundle(self, request, instruction_id)
    516         with self.maybe_profile(instruction_id):
    517           delayed_applications, requests_finalization = (
--> 518               bundle_processor.process_bundle(instruction_id))
    519           monitoring_infos = bundle_processor.monitoring_infos()
    520           monitoring_infos.extend(self.state_cache_metrics_fn())

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py in process_bundle(self, instruction_id)
    981           elif isinstance(element, beam_fn_api_pb2.Elements.Data):
    982             input_op_by_transform_id[element.transform_id].process_encoded(
--> 983                 element.data)
    984 
    985       # Finish all operations.

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py in process_encoded(self, encoded_windowed_values)
    217       decoded_value = self.windowed_coder_impl.decode_from_stream(
    218           input_stream, True)
--> 219       self.output(decoded_value)
    220 
    221   def monitoring_infos(self, transform_id, tag_to_pcollection_id):

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SdfProcessSizedElements.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SdfProcessSizedElements.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process_with_sized_restriction()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.ConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/future/utils/__init__.py in raise_with_traceback(exc, traceback)
    444         if traceback == Ellipsis:
    445             _, _, traceback = sys.exc_info()
--> 446         raise exc.with_traceback(traceback)
    447 
    448 else:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.PGBKCVOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.PGBKCVOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py in add_input(self, accumulator, element)
    339     results = []
    340     for i, (c, a) in enumerate(zip(self._combiners, accumulator)):
--> 341       result = c.add_input(a, get_combiner_input(elements[0], i))
    342       for e in elements[1:]:
    343         result = c.add_input(result, get_combiner_input(e, i))

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py in add_input(self, accumulator, element)
    575         # Keras requires non-sparse keys for top_k calcuations.
    576         if self._is_top_k() and label.shape != prediction.shape:
--> 577           label = metric_util.one_hot(label, prediction)
    578         accumulator.add_input(i, label, prediction, example_weight)
    579     if (accumulator.len_inputs() >= self._batch_size or

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_util.py in one_hot(tensor, target)
    702   # the row. The following handles -1 values by adding an additional column for
    703   # indexing the -1 and then removing it after.
--> 704   tensor = np.delete(np.eye(target.shape[-1] + 1)[tensor], -1, axis=-1)
    705   return tensor.reshape(target.shape)
    706 

IndexError: arrays used as indices must be of integer (or boolean) type [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots()/ComputePerSlice/ComputeUnsampledMetrics/CombinePerSliceKey/WindowIntoDiscarding']
```




Can you provide example outputs for the labels and predictions.

``` python
model = tf.saved_model.load(export_dir='/tmp/tfx-interactive-model/6/serving_model_dir/')
predict_fn = model.signatures["serving_default"]
text = "some random text"
example = tf.train.Example(features = tf.train.Features(feature ={
    "Headline":_string_feature('headline'),
    "InternalId":_int64_feature(1111),
    "Text":_string_feature(text),
    "content_id":_string_feature("2222"),
}))
serialized_example = example.SerializeToString()
predict_fn(tf.constant([serialized_example]))

{'outputs': <tf.Tensor: shape=(1, 10), dtype=float32, numpy=
 array([[0.6034294 , 0.40879828, 0.4462878 , 0.3196895 , 0.31108657,
         0.30687535, 0.27371922, 0.29507416, 0.2872882 , 0.26999897]],
       dtype=float32)>}
```
This is because I have 10 classes, so probability for all the 10 classes.



The categories labels you mentioned above only has 5 classes. These need to align.

@mdreves No, that's was an earlier iteration, I have 10 classes. The format remains the same, but now I have 10 classes.

``` python
{
 'Text': array([b'Football fans looking forward to seeing the renewal of the rivalry between Cristiano Ronaldo and Lionel Messi were made to wait a while longer after the Portuguese forward was forced to miss Juventus' Champions League tie against Barcelona on Wednesday.'],dtype=object),
 'Headline': array([b"Lionel Messi scores as Cristiano Ronaldo misses Barcelona's victory over Juventus."], dtype=object),
 'categories': array([1., 0., 0., 0., 0., 0., 0., 0., 0.,0.], dtype=object), 
}

{
 'Text_xf': array([b'COVID-19 has changed fan behavior and accelerated three to five years of technology adoption into six months'],dtype=object),
 'Headline_xf': array([b"How Technology Is Improving Fan Transactions at Sports Venues"], dtype=object),
 'categories_xf': array([1., 1., 0., 0., 0., 0., 0., 0., 0.,0.'], dtype=object), 
}

Sorry for late reply. This error is caused when the label output after processing is empty. I'm having trouble reproducing this issue with the information provided. Can you try with the latest release and let me know if you still see this.

@albertnanda , please reply on the above comment.Thanks.

Closing as this issue is in "awaiting response" status for more than 1 month. Please take a look into the answers provided above and post your comments(if you still have queries on this). Thank you!