zhiminy/EvalEng huggingface dataset:
Repository: tensorflow/model-analysis
Issue: Wrong batch size pass to predictor
Root cause: Validation Gap

### System information
0.13.*

### Describe the problem

When predictor input with batch size, say 256, we need to specify batch size (by setting desired_batch_size) when calling run_model_analysis (as in PR: https://github.com/tensorflow/model-analysis/pull/48/), or ExtractEvaluateAndWriteResults,  the problem is that if my input has 300 examples, first batch will be 256 will runs ok, but the second batch will be with 44 example feeding to predictor which will crash predictor (since batch size does not match).

One easy fix is to ignore not mismatched size of batch by adding a filter here: 
https://github.com/tensorflow/model-analysis/blob/master/tensorflow_model_analysis/extractors/predict_extractor.py#L81
if (self._desired_batch_size and self._desired_batch_size != batch_size):
        print("Skip process for {} element since batch size does not match".format(batch_size))
        return

### Source code / logs


Can you post a trace for the crash that you are seeing. The model should accept varying batch sizes and we certainly don't want to skip running predictions for some of the examples.

In my case, the reason the model does not accept varying batch size is I that I explicitly set batch size for my eval_input_receiver_fn (I cannot leave it as None for tf graph parsing issue because of my model_fn definition). I would be great if we can have option to skip some tail of the example to fit the batch size.

def eval_input_receiver_fn(self, batch_size):
        serialized_tf_example = tf.placeholder(
            dtype=tf.string, shape=[batch_size], name='input_example_tensor')
        # The key name MUST be 'examples'.
        receiver_tensors = {'examples': serialized_tf_example}
        raw_features = tf.parse_example(serialized_tf_example,
                                        self._feature_specs)
        return tfma.export.EvalInputReceiver(
            features=raw_features,
            receiver_tensors=receiver_tensors,
            labels=None)

I think you either need to pad your examples, truncate them at the source, or fix the model_fn.

That is a fare argument, if tmfa require batch size of eval_fn to be None.
But this will push the hard work to tfma user given adding a filter in tfma is so easy (tf dataset also support this kind of filter) :)


@zzhang2019 ,
Closing the issue as the resolution has been provided. Feel free to Open a Feature Request if you feel some functionality needs to be added. Thanks!
Repository: tensorflow/model-analysis
Issue: minimal example for launching/confirming tfma integration in a notebook?
Root cause: Documentation Deficiency

any chance someone from this project could provide a minimal code example for launching the notebook-based visual TFMA widget to ensure that it's properly integrated/installed in the notebook environment?

ideally this would just render the visual widget using dummy data that could be generated on a single machine w/o some complex training process behind it, primarily for cursory integration testing in the notebook environment.

@kwlzn Please refer to the following [link](https://github.com/tensorflow/tfx/blob/master/tfx/examples/chicago_taxi/README.md) which might give you the information that you are looking for. 

Also please post support related questions to Stack Overflow using the [tensorflow-model-analysis](https://stackoverflow.com/questions/tagged/tensorflow-model-analysis) tag

@gowtham-kp while this works, it is also a very long winded way to test an otherwise simple nbextension integration case. the goal here is to simply confirm whether or not TFMA is correctly installed in a notebook environment, ideally via a one-liner.

any chance you guys could add a simple mode of launching the analysis viewer functionality without actually requiring a full model be trained and analyzed just to ensure that's working?

or alternatively, could you freeze the binary products of the model training pre-requisite for easier drop-in analysis?

We have such testdata artifacts in [TFX repo ](https://github.com/tensorflow/tfx). This [test data](https://github.com/tensorflow/tfx/tree/master/tfx/components/testdata/trainer/current) represents a pre-saved exported model, which I believe can be used with the [example data](https://github.com/tensorflow/tfx/tree/master/tfx/components/testdata/csv_example_gen).

@gowthamkpr, maybe we can duplicate this issue in TFX's issue list and I can cc a couple of engineers to help out that way?



sure @zhitaoli We can do that

ah, great we'll give that a shot - thanks for the pointer @zhitaoli

@kwlzn 

Could you please refer this [link](https://github.com/tensorflow/tfx/tree/master/tfx/examples/chicago_taxi_pipeline) and Closing this issue,Feel free to reopen the issue if this still exists.Thanks
Repository: tensorflow/model-analysis
Issue: v0.14.0 tag points to the 0.15.0 release commit
GitHub labels: stat:awaiting tensorflower

type:docs

Hi, I was trying to look at the code for the 0.14.0 release, and I noticed that the `v0.14.0` tag actually points to https://github.com/tensorflow/model-analysis/commit/35ab9fc9f8f5e137baff07f208bf51120135884f. Is someone able to fix this?

Try now, should be fixed.
Repository: tensorflow/model-analysis
Issue: Saw the demo at the conference,, but
Root cause: Architectural Constraint

I was so hopeful after seeing the demo. When going to use the analysis was very disappointing to note the Py27 dependency,, which seems to be due to Beam,, why not just take the normal TF input flow?? eg among other things taking plain old csv files as input (ex Beam). I can think of no reason that Beam should be in the analysis package,, if it is needed/desired it should be structured as part of the TF IO package facilities. As it sits now, it seems like the package special purpose code that is not ready for users doing normal development on local machines for a variety of use cases.. Even Google Doc's generally advise working out flows, and this would include initial analysis, on local machines before deploying to cloud usage.. It seems pretty apparent that this is not getting much usage, if only due to the lack of SO activity (none) and the lack of GH issues,, just these two... Hopefully there is a plan to make this more usable as I feel it would be a huge help to model development.  We really need tools like this though!

https://github.com/tensorflow/model-analysis/issues/8 tracks Python 3 support. Wrt usability - fair point, currently the project only contains the core library, more info on future plans here: https://youtu.be/vdG7uKQ2eKk.
Repository: tensorflow/model-analysis
Issue: ExampleCount metric always returns 1 for multi-output models
Root cause: Algorithmic Error

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: No
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Mac OS X
-   **TensorFlow Model Analysis installed from (source or binary)**: binary
-   **TensorFlow Model Analysis version (use command below)**: 0.21.6
-   **Python version**: Python 3.6.9
-   **Jupyter Notebook version**: doesn't apply
-   **Exact command to reproduce**:

### Describe the problem

`tfma.metrics.ExampleCount` always returns 1 for multi-output models. I believe this happens because when the metric key is created, the output name is not considered (see https://github.com/tensorflow/model-analysis/blob/master/tensorflow_model_analysis/metrics/example_count.py#L49). This is not a problem in the similar `tfma.metrics.WeightedExampleCount` (https://github.com/tensorflow/model-analysis/blob/master/tensorflow_model_analysis/metrics/weighted_example_count.py#L60).

### Source code / logs

```
eval_config = tfma.EvalConfig(
    model_specs=[
        tfma.ModelSpec(
            prediction_key='softmax',
            signature_name="eval",
            label_key="candidate_app_id_integerized"
        )
    ],
    metrics_specs=tfma.metrics.specs_from_metrics(
        metrics=[
            tfma.metrics.ExampleCount(), 
            tfma.metrics.WeightedExampleCount(),
            tf.keras.metrics.SparseCategoricalCrossentropy()
        ],
        output_names=['softmax'],
        include_example_count=False,
        include_weighted_example_count=False
    ),
    slicing_specs=[
        tfma.SlicingSpec(),
        tfma.SlicingSpec(
            feature_keys=["language"]
        ),
        tfma.SlicingSpec(
            feature_keys=["country_code"]
        ),
        tfma.SlicingSpec(
            feature_keys=["is_ios"]
        )
    ]
)

eval_result = tfma.run_model_analysis(
    eval_shared_model=eval_shared_model,
    eval_config=eval_config,
    data_location=examples_dir,
    output_path=output_path
)
```

This is a display issue. The example count is the same for all outputs, so it should be filtered out of the choices for the drop down.

Right. This makes total sense.

@rafaelbarreto87 

Closing this issue as it is resolved,feel free to reopen if this still exists.Thanks
Repository: tensorflow/model-analysis
Issue: TimeSeriesView and PlotViewer broken due to incorrect model_module in HTML export
Root cause: Validation Gap

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: no
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Debian Buster
-   **TensorFlow Model Analysis installed from (source or binary)**: binary
-   **TensorFlow Model Analysis version (use command below)**: 0.22.1
-   **Python version**: 3.7.3
-   **Jupyter Notebook version**: 6.0.3

### Describe the problem

Module loading fails via unpck.com due to incorrect naming.  

https://github.com/tensorflow/model-analysis/blob/b5060f014ec76876327f9960433cf83aca4a9ddf/tensorflow_model_analysis/notebook/jupyter/tfma_widget/widget.py#L55 and https://github.com/tensorflow/model-analysis/blob/b5060f014ec76876327f9960433cf83aca4a9ddf/tensorflow_model_analysis/notebook/jupyter/tfma_widget/widget.py#L68 should be `tensorflow_model_analysis` instead of `tensorflow_model-analysis`



Thanks for debugging. Submitted a fix, should be in next release.
Repository: tensorflow/model-analysis
Issue: Documentation needed?
GitHub labels: stat:awaiting tensorflower

type:docs

Hi developers,

I don't know if this is the right place or channel to communicate this. But the [tfma website](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma) could use a bit more documentation.

Even though it is possible to dive in to the source code to find out the parameters and usage, the library usability can be greatly improved had there been a higher level explanation of the API available in tfma.

For example, the [tfma.genericvaluethreshold](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/GenericValueThreshold) class conditions the pusher to only push models that are within the two bounds configured in this class.

I think this endeavor should slightly prioritized as it's the gateway for people to start using the library.

Unfortunately our gendocs script doesn't currently pickup comments on protobufs. Looking into what we can do.
Repository: tensorflow/model-analysis
Issue: Cannot read property 'SlicingMetricsModel' of undefined
Root cause: Resource Mishandling

System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: Yes
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu
-   **TensorFlow Model Analysis installed from (source or binary)**: source
-   **TensorFlow Model Analysis version (use command below)**: 0.22.2
-   **Python version**: 3.6.9
-   **Jupyter Notebook version**: 6.0.3
-   **Exact command to reproduce**: `tfma.view.render_slicing_metrics(result)`

### Describe the problem

I'm running tfma inside a docker container to get around any issues with the tfma extension not being available in my conda environment. Everything looks good, I just can't display the interactive charts. When I call `tfma.view.render_slicing_metrics(result)`, I get the following error in the browser console. Chrome version 83.0.4103.106.

### Source code / logs

<img width="562" alt="Screen Shot 2020-06-28 at 7 01 57 PM" src="https://user-images.githubusercontent.com/13136287/85961695-da1d0780-b971-11ea-9151-6ebaa05cd159.png">


I get the same error, running locally. Occasionally if I hit some combination of page refreshes and runs of this command, the visualization will show up, but it is not reliable.

Chrome version 83.0.4103.116

- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.15.5
- TensorFlow Model Analysis installed from (source or binary): source
- TensorFlow Model Analysis version (use command below): 0.22.2
- Python version: 3.7.2
- Jupyter Notebook version: 6.0.3
- Exact command to reproduce: tfma.view.render_slicing_metrics(result)


I get the same error when running locally. 
Chrome version 85.0.4183.83

- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.14.6
- TensorFlow Model Analysis installed from (source or binary): binary from pypi
- TensorFlow Model Analysis version (use command below): 0.23
- Python version: 3.6.5
- Jupyter Notebook version: 6.1.3
- Exact command to reproduce: tfma.view.render_slicing_metrics(result)

I also experience the same issue. However, for me, opening the same notebook (without rerunning any cells or restarting the kernel) in a different tab or browser renders the plot.

Same problem for me. I am using Chrome (MAC) Version 85.0.4183.102 (Official Build) (64-bit)
And the extension status (running `jupyter nbextension list`) looks fine:
> notebook section
      jupyter-js-widgets/extension  enabled
      - Validating: OK
      tensorflow_model_analysis/extension  enabled
      - Validating: OK

I also tired open the same notebook in a different tab or browser, nothing works.

Same issue over here.

``jupyter nbextension list`` looks ok:
```
    notebook section
      jupyter-js-widgets/extension  enabled 
      - Validating: OK
      tensorflow_model_analysis/extension  enabled 
      - Validating: OK
```
Repository: tensorflow/model-analysis
Issue: Binarize options in default_multi_class_classification_specs  is broken
Root cause: Validation Gap

# System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: Yes
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu
-   **TensorFlow Model Analysis installed from (source or binary)**: binary (PyPI)
-   **TensorFlow Model Analysis version (use command below)**: 0.22.2
-   **Python version**: 3.6.8
-   **Jupyter Notebook version**: 6.0.3
-   **Exact command to reproduce**:
```python
tfma.metrics.default_multi_class_classification_specs(
    model_names=['model'],
    binarize=tfma.BinarizationOptions(class_ids={'values' : list(range(len(classes)))}),
    sparse=True
)
```
This gives the following error:
```
<ipython-input-17-1d79e900e9de> in <module>
      2     model_names=['model'],
      3     binarize=tfma.BinarizationOptions(class_ids={'values' : list(range(len(classes)))}),
----> 4     sparse=True
      5 )

/usr/local/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_specs.py in default_multi_class_classification_specs(model_names, output_names, binarize, aggregate, sparse)
    325       ])
    326     binarize = config.BinarizationOptions().CopyFrom(binarize)
--> 327     binarize.ClearField('top_k_list')  # pytype: disable=attribute-error
    328   multi_class_metrics = specs_from_metrics(
    329       metrics, model_names=model_names, output_names=output_names)

AttributeError: 'NoneType' object has no attribute 'ClearField'
```


Thanks for raising the issue. Fixed at head.

@mdreves sorry but I don't think your commit quite fixes the issue. It still is overwriting the user's BinarizationOptions. Would you please consider looking at my PR? #74 
I have approval from my employer and am just waiting for one thing to clear before I sign the CLA. This would be my first contribution to an open source project!

Sorry about that. I've committed a revised version of your fix. Thanks.
Repository: tensorflow/model-analysis
Issue: Confusion matrics and calibration plots don't visualize in kubeflow pipeline
Root cause: Documentation Deficiency

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: Yes
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu
-   **TensorFlow Model Analysis installed from (source or binary)**: binary (PyPI)
-   **TensorFlow Model Analysis version (use command below)**: 0.22.2
-   **Python version**: 3.6.9
-   **Jupyter Notebook version**: 6.0.3
-   **Exact command to reproduce**:
```python
from tensorflow_model_analysis import EvalConfig
from tensorflow_model_analysis.metrics import default_multi_class_classification_specs
from google.protobuf.json_format import ParseDict

class = ['class_1', 'class_2', ...]

eval_config = {
    'model_specs': [
        {
            'name': 'rig_state',
            'model_type': 'tf_keras',
            'signature_name': 'serve_raw',
            'label_key': ...,
            'example_weight_key': 'sample_weight'
        }
    ],
    'metrics_specs': [
        {
            'metrics': [
                {
                    'class_name': 'MultiClassConfusionMatrixPlot',
                    'config': '"thresholds": [0.5]'
                },
                {'class_name': 'ExampleCount'},
                {'class_name': 'WeightedExampleCount'},
                {'class_name': 'SparseCategoricalAccuracy'},
            ],
        },
        {
            'binarize': {'class_ids': {'values': list(range(len(classes)))}},
            'metrics': [
                {'class_name': 'AUC'},
                {'class_name': 'CalibrationPlot'},
                {'class_name': 'BinaryAccuracy'},
                {'class_name': 'MeanPrediction'}
            ]
        }
    ],
    'slicing_specs': [...]
}
eval_config: EvalConfig = ParseDict(eval_config, EvalConfig())
```

### Describe the problem
When using this EvalConfig in `tfma.run_model_analysis` everything runs successfully. Inspecting the result, there is plot data for the confusion matrix, yet when trying to plot the result using `tfma.view.render_slicing_metrics`, the confusion matrix won't show.

This happens in a kubeflow pipeline.

Can you provide the exact call you are making. The plots are visualized using tfma.view.render_plot.

"The plots are visualized using tfma.view.render_plot."

Ahh, I didn't know this. It seems to work now. However, the multiclass confusion matrix is virtually useless when it just displays integer class IDs (#78). Is there a way to add a list of class name strings in my `EvalConfig` or something?

Added comment in #78. The short answer is class labels are not yet supported. We are trying to avoid having to have this as part of the config and make it automatic.
Repository: tensorflow/model-analysis
Issue: TFMA fails when trying to use a custom metric from a module that ends with `keras.metrics`
Root cause: Validation Gap

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: Yes
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: `tensorflow/tensorflow:2.2.0-gpu-jupyter` docker image
-   **TensorFlow Model Analysis installed from (source or binary)**: binary
-   **TensorFlow Model Analysis version (use command below)**: 0.22.2
-   **Python version**: 3.6.9
-   **Jupyter Notebook version**: 6.0.2
-   **Exact command to reproduce**: NA

### Describe the problem

TFMA fails when trying to use a custom metric coming from a module that ends with `keras.metrics`. I have a contrib module whose name is `{PROJECT_NAME}.ml_utils.tf.keras.metrics` in which I have lots of custom metrics. When I try to use those in TFMA, it fails saying that the metrics can't be found. I _don't_ have this issue if I rename my module to something like `{PROJECT_NAME}.ml_utils.tf.keras.custom_metrics`. After tracking the issue down, I noticed [this snippet](https://github.com/tensorflow/model-analysis/blob/0b10d4175235cf68d5364282a862ed02151c3b1e/tensorflow_model_analysis/metrics/tf_metric_wrapper.py#L266) to be the issue:

```
def _custom_objects(
    metrics: Dict[Text, List[tf.keras.metrics.Metric]]) -> Dict[Text, Any]:
  custom_objects = {}
  for metric_list in metrics.values():
    for metric in metric_list:
      if (not metric.__class__.__module__.endswith('keras.metrics') and
          not metric.__class__.__module__.endswith('keras.losses')):
        custom_objects[metric.__class__.__module__] = metric.__class__.__name__
  return custom_objects
```

The if statement is problematic because my module indeed ends with `keras.metrics` and, yet, is a custom module.  It seems the more correct and robust approach here would be to replace it with:

```
def _custom_objects(
    metrics: Dict[Text, List[tf.keras.metrics.Metric]]) -> Dict[Text, Any]:
  custom_objects = {}
  for metric_list in metrics.values():
    for metric in metric_list:
      if (not metric.__class__.__module__ == tf.keras.metrics.__name__ and
          not metric.__class__.__module__ == tf.keras.losses.__name__):
        custom_objects[metric.__class__.__module__] = metric.__class__.__name__
  return custom_objects
```

Thanks for raising the issue. Should be fixed at head now.

That was quick! Thanks :D.