Repository: tensorflow/model-analysis
Issue: "no value provided for label" error with TFX Keras + Evaluator component
Root cause: Validation Gap

### Issue

I'm part of the team supporting [TFX/Kubeflow Pipelines at Spotify](https://engineering.atspotify.com/2019/12/13/the-winding-road-to-better-machine-learning-infrastructure-through-tensorflow-extended-and-kubeflow/), we are currently upgrading our internal stack to `tfx==0.22.1` and `tensorflow-model-analysis==0.22.2`.

We can successfully evaluate an Estimator-based model with the [open-source evaluator component](https://github.com/tensorflow/tfx/blob/v0.22.1/tfx/components/evaluator/executor.py). Unfortunately, when using a Keras-based model, the Beam evaluation pipeline running on Dataflow fails with the following error:

```bash
Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 961, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 726, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 812, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "apache_beam/runners/common.py", line 1122, in apache_beam.runners.common._OutputProcessor.process_outputs
  File "apache_beam/runners/worker/operations.py", line 195, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 949, in apache_beam.runners.worker.operations.PGBKCVOperation.process
  File "apache_beam/runners/worker/operations.py", line 978, in apache_beam.runners.worker.operations.PGBKCVOperation.process
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py", line 356, in add_input
    result = c.add_input(a, get_combiner_input(elements[0], i))
  File "/usr/local/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py", line 551, in add_input
    flatten=self._class_weights is not None)):
  File "/usr/local/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_util.py", line 264, in to_label_prediction_example_weight
    sub_key, inputs))
ValueError: no value provided for label: model_name=, output_name=, sub_key=None, StandardMetricInputs=StandardMetricInputs(label=None, prediction=0.025542974, example_weight=None, features=None)
This may be caused by a configuration error (i.e. label, and/or prediction keys were not specified) or an error in the pipeline.
```

Have you ever faced this issue?

### Additional context

* We run a version of the Chicago taxi example pipeline
* Keras model was trained using a copy of the [GenericExecutor](https://www.tensorflow.org/tfx/api_docs/python/tfx/components/trainer/executor/GenericExecutor)
* The model's code was copied from [TFX's tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/components_keras#trainer)
* The following eval config is passed to the Evaluator component:
```python
import tensorflow_model_analysis as tfma
from google.protobuf.wrappers_pb2 import BoolValue
eval_config = tfma.EvalConfig(
    model_specs=[
      tfma.ModelSpec(model_type=tfma.constants.TF_KERAS, label_key="tips")
    ],
    slicing_specs=[
        tfma.SlicingSpec(),
    ],
    options=tfma.Options(include_default_metrics=BoolValue(value=True)),
)
```

### System information

-   **Have I written custom code**: Yes
-   **TensorFlow Model Analysis installed from**: binary via pip
-   **TensorFlow Model Analysis version (use command below)**: 0.22.2
-   **Python version**: 3.6.9



To add a bit of additional context, in case its important, I based the training for the Keras model based on the code from this ipynb example: https://github.com/tensorflow/tfx/blob/master/tfx/examples/chicago_taxi_pipeline/taxi_pipeline_native_keras.py

which is to say, this module file: https://github.com/tensorflow/tfx/blob/master/tfx/examples/chicago_taxi_pipeline/taxi_utils_native_keras.py

we have some slight modifications in terms of removing some of the transform features (specifically the bucket features), but its pretty much otherwise the same. If the problem may be in the construction of the model I could provide the exact code we have.

I can't be certain without more logs, but it looks like the 'tips' feature is not present in all of the input examples (or was removed later on in the pipeline). Note that because keras doesn't have support for label transformations yet, TFMA requires the label that is used to be present in the raw inputs. The taxi example was updated to add a 'big_tipper' label to account for this:

https://github.com/tensorflow/tfx/blob/1985b30b39d8833b6fe724828ce97441813cdb5a/tfx/examples/chicago_taxi_pipeline/taxi_utils_native_keras.py#L65

Thanks a lot @mdreves, we confirm it is now working for us! Do you have a timeline on when support for label transformations could be added to Keras?

It depends on KPL (keras preprocessing layer) support, but we are actively working on this quarter. If you are using TFT (tensorflow transform) with keras then that will likely land first.

We are using TFT, we'll keep an eye on the upcoming releases then. Thanks for your help!

@sngahane @mdreves which TFX version allows for Keras label transformations? I did not see it mentioned on the [release notes](https://github.com/tensorflow/tfx/blob/master/RELEASE.md). I am currently on 0.27.0, and I am hesitant to upgrade as I know there will be lots of things to change.

I see that @mdreves [committed changes](https://github.com/tensorflow/tfx/commit/4aa1a75219851eadc7245546166d8f555045a244) in the taxi example for Keras label transformations. A strong argument for me to upgrade to 1.0.0.

Hi @axeltidemann, those committed changes above have made it into the 0.28 release, so any version >= 0.28 should have this support.