zhiminy/EvalEng huggingface dataset:
Repository: tensorflow/model-analysis
Issue: Is line #96 of examples/chicago_taxi/model.py example a bug ?
GitHub labels: type:bug

stat:awaiting tensorflower

https://github.com/tensorflow/model-analysis/blob/2fb97d4d6bac1b800c340ca9a6f2393bd5169cf2/examples/chicago_taxi/trainer/model.py#L96

The tf.estimator.export.ServingInputReceiver has only a constructor with 2 parameters which are features and receiver_tensors.
But in line#96 ServingInputReceiver is created with 2 parameters : transformed_features and serving_input_receiver.features, no receiver_tensors.

You are right, this is indeed a bug. It should pass serving_input_receiver.receiver_tensors as the second argument instead.

@chuanyu  -  Is this fixed now ? 

Yes, this has been fixed.
Repository: tensorflow/model-analysis
Issue: Ready for general use?

Hi,

I'd love to try this to test the accuracy of trained models.

Would you say this is alpha or beta quality at this point? How much bandwidth is being dedicated to it by the Tensorflow team? In other words, is this part of the the framework that we can already use or in early stages of development? 

Thanks,

Thanks for the interest!

On the question of quality: This is the same codebase we (Google) use for internal production pipelines. 

However, I do want to point out that we follow semantic versioning (https://www.tensorflow.org/guide/version_compat) and this is currently a 0.x release. Until we hit the first major version (1.0) we may make breaking changes to the APIs. 

As you may imagine, the fact that we use this code in production means that we will be careful with changes. 

Thank you!! Excited to adopt this in my workflow.
Repository: tensorflow/model-analysis
Issue: Validation bug when using feature values as slicing specs
Root cause: Algorithmic Error

TFMA version: 0.33

Hi team,

I found a bug in TFMA validation. It happens when I use multiple feature values to specify slicing like so:

```python
eval_config = text_format.Parse("""
  model_specs {
    signature_name: "serving_default"
    preprocessing_function_names: ["transform_features"]
    label_key: "income_bracket"
  }
  metrics_specs {
    metrics {
      class_name: "ExampleCount"
    }
    metrics {
      class_name: "BinaryAccuracy"
      config: '{"name": "accuracy"}'
      per_slice_thresholds {
        slicing_specs {
          feature_values: {key: "race", value: "White"}
          feature_values: {key: "sex", value: "Male"}
        }
        threshold {
          value_threshold {
            lower_bound { value: 0.7 }
          }
        }
      }
    }
  }
  slicing_specs {
    feature_keys: ["race", "sex"]
  }
""", tfma.EvalConfig())
```

In this example, I specify that I just want to validate accuracy on a particular group: white males. The evaluation will run, but when I load validation results, I get:

```
tfma.load_validation_result(evaluator.outputs['evaluation'].get()[0].uri)

missing_slices {
  feature_values {
    key: "race"
    value: "White"
  }
  feature_values {
    key: "sex"
    value: "Male"
  }
}
```
I've tracked down the cause of this bug, and I believe it's the following two lines:

https://github.com/tensorflow/model-analysis/blob/65bb4036a2a296bca228c196a576a355f3c29b1c/tensorflow_model_analysis/evaluators/metrics_validator.py#L213
https://github.com/tensorflow/model-analysis/blob/65bb4036a2a296bca228c196a576a355f3c29b1c/tensorflow_model_analysis/evaluators/metrics_validator.py#L262

Here, you're relying on serialization of slicing_specs for equality comparison. The problem is that feature_values is a map field, and the entry order is random in the serialized string. Sometimes you get race serialized before sex, and sometimes the reverse happens, which is why I get the missing slices message.

I think custom serialization using sorted map fields would fix this bug. 

Thanks a lot for reporting the bug. We are actively working on a fix.

This issue should be fixed at head. Let us know if you are still seeing the problem after https://github.com/tensorflow/model-analysis/commit/d31e583a7a8765b6d89264d24a18b3d133b33b89
Repository: tensorflow/model-analysis
Issue: Analytics 
GitHub labels: stat:awaiting response

type:support


In order to expedite the trouble-shooting process, please provide more information on the issue reported here. For more information, please refer to [TFMA](https://www.tensorflow.org/tfx/model_analysis/get_started) and [TensorFlow Model Analysis Visualizations](https://www.tensorflow.org/tfx/model_analysis/visualizations).

Thank you!
Repository: tensorflow/model-analysis
Issue: Breaking changes: tfma.metrics.MetricComputation `preprocessor` argument changed from accepting beam DoFn to `preprocessors` accepting a list of `Preprocessor`
Root cause: Documentation Deficiency

Recent release of TFMA (0.42 - 0.43 - master) changed the DoFn argument from `preprocessor` 
https://github.com/tensorflow/model-analysis/blob/a5c4c709e733bffe10038ee43b07704883d843b1/tensorflow_model_analysis/metrics/metric_types.py#L396 to `preprocessors` 
https://github.com/tensorflow/model-analysis/blob/e1c34a8b434440efa32d6617bc922a412c77b924/tensorflow_model_analysis/metrics/metric_types.py#L440

This breaking change is not reflected in the example for custom metrics [here](https://github.com/tensorflow/model-analysis/blob/r0.43.0/g3doc/metrics.md#custom-tfma-metrics).

@EdwardCuiPeacock,

Thank you for pointing this out.

@mdreves, Can you please take a look into custom TFMA metrics [example](https://github.com/tensorflow/model-analysis/blob/r0.43.0/g3doc/metrics.md#custom-tfma-metrics). The example is not updated as per the change in preprocessors attribute in MetricComputation. Thank you!
Repository: tensorflow/model-analysis
Issue: Move to numpy >=1.20 because <1.20 is difficult to build on Apple Silicon
Root cause: External Dependency Breakage

https://github.com/tensorflow/model-analysis/blob/b4f2e9fe2733a8ee4d6a129facf23efdc4b162ba/setup.py#L297

Numpy `>=1.20` is easy to install on Apple silicon, because it's precompiled, but `1.19` fails unless openblas is set up on the system (which it will not be for most users). (See [this NumPy](https://github.com/numpy/numpy/issues/17807) issue.)

See also this issue in scipy where they moved away from 1.19 because it fails to build on Apple Silicon. https://github.com/scipy/oldest-supported-numpy/issues/19

Hi, @sradc 

Apologies for the delay and as we mentioned in TFMA setup file `numpy>=1.16,<2` is not specific to particular OS so it seems like `numpy==1.19` needs `openblas` set up to install `numpy==1.19` on Apple Silicon so as you mentioned `numpy==1.20` working fine so I would suggest you to go with `numpy>=1.20` and as of now we haven't noticed more issues specific to `numpy==1.19` w.r.t Apple Silicon

Please let me know if I'm missing something here or you need any further assistance ?

Thank you!

Hey @gaikwadrahul8,

Thank you for the response. 

Does TFMA support Apple Silicon? I'm trying to install it alongside [tensorflow-metal](https://developer.apple.com/metal/tensorflow-plugin/). 

- When I run `pip install tensorflow-model-analysis`, it's been trying to install `numpy=1.19.*`, even though `numpy=1.24.*` is installed. (I checked, and it is trying to install `tensorflow-model-analysis==0.43.0`). 
- When I run `pip install 'tensorflow-model-analysis==0.43.0'`, it doesn't try to downgrade numpy, but the install fails with `tfx-bsl`, saying `ERROR: Could not find a version that satisfies the requirement tfx-bsl (from versions: none)`.

(I'm a bit confused about what's going on.) I'm thinking this could be user error on my part, so will keep trying things, but letting you know on the off chance you have any information that might help.

Hi, @sradc 

You can install Tensorflow with the help of `tensorflow-metal plugin` and use it but at the moment it seems like we do not support TFX and supported libraries like TFMA, TFDV and TFT for Apple Silicon(M1) you can check [similar issue ](https://github.com/tensorflow/data-validation/issues/205)and same discussion going on our [Tensorlflow Forum ](https://discuss.tensorflow.org/t/i-want-to-use-the-tfx-1-6-1-on-mac-m1-max/8110), In the meantime, some users have reported success with Rosetta. Other options include using a VM. We understand that neither of those is ideal.

If I'm missing something here, please let me know ? Thank you!

Thanks @gaikwadrahul8, appreciate the info. 👍
Repository: tensorflow/model-analysis
Issue: TFMA on Flink does not seems to be parallelizing work.
Root cause: Architectural Constraint

Hello I am running into an issue running the evaluator component of a tfx pipeline. I use the FlinkRunner for beam and the evaluator component is super slow as the size of data scales. It seems it is because the work is done only by a single Task Manager.

### System information

I am running a TFX pipeline using python 3.7. TFX version 1.8.1 which comes with TFMA version `tensorflow-model-analysis==0.39.0`.
I don't have a small example to reproduce, I can work on one if you think it will help.

### Describe the problem

I use the evaluator TFX component as such

```
evaluator = Evaluator(
        examples=example_gen.outputs[standard_component_specs.EXAMPLES_KEY],
        model=trainer.outputs[standard_component_specs.MODEL_KEY],
        eval_config=eval_config,
    )
```

With a simple eval_config without any splits. So we only have the `eval_split` which is used for evaluation.

To run the TFX pipeline we use the FlinkRunner for beam. The sidecar image is built from tensorflow/tfx:1.8.1.

We run flink with a parellism of 10. So 10 files of tf_records are in input of the evaluator component.

From what we could gather, beam tells flink to build a single task for the 3 p_transforms:

"ReadFromTFRecordToArrow" | "FlattenExamples" | "ExtractEvaluateAndWriteResults"

Our issue is that this ends up creating a single subtask for Flink, so a single task manager is doing all the work as you can see in the attached screenshot. So the issue seems to be with the beam workflow which does not parallelized.

I have two main questions:

- Is this behavior normal ? 
- Is it possible to better dispatch the workload between the different taskmanagers ?

![Screenshot from 2023-01-26 15-18-44](https://user-images.githubusercontent.com/23744708/215800317-58aca649-7732-4a5b-a2ed-cd8ca45a71e5.png)


@jccarles,

Can you please share the eval_config passed in evaluator component to analyse the root cause of the issue? Thank you!

Hello ! Thank you for your answer, here is the eval_config used. We used fake very low bounds for testing.

```
{
  "model_specs": [
    {
      "signature_name": "serving_default",
      "label_key": "label",
      "preprocessing_function_names": [
        "transform_features"
      ]
    }
  ],
  "metrics_specs": [
    {
      "thresholds": {
        "precision": {
          "value_threshold": {
            "lower_bound": 1e-03
          },
          "change_threshold": {
            "absolute": -1e-10,
            "direction": "HIGHER_IS_BETTER"
          }
        }
      }
    }
  ]
}
```

@mdreves, Can we dispatch the evaluator between different task managers. Thanks!

Hello, thank you for checking this issue, did you have time to take a look ? Have you identified anything so far, can I help somehow ?

This issue is currently preventing us from using the Evaluator  component in production, since it makes the memory requirements on a single Flink TaskManager rather huge.
Repository: tensorflow/model-analysis
Issue: Render_plot is failing
Root cause: Environment Incompatibility

### System information

-   **I have used stock example script provided in TensorFlow Model Analysis**: https://github.com/tensorflow/tfx/blob/master/docs/tutorials/model_analysis/tfma_basic.ipynb
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10
-   **TensorFlow Model Analysis installed from (source or binary)**: binary using pip install
-   **TensorFlow Model Analysis version (use command below)**: 0.41.0, 0.42.0, 0.43.0, 0.44.0 (tried all versions 1-by-1)
-   **Python version**: 3.9
-   **Jupyter Notebook version**: tried with jupyterlab==3.5.3 as well as 4.0.3
-   **Exact command to reproduce**: 
     tfma.view.render_plot(
                   eval_result,
                   tfma.SlicingSpec(feature_values={'trip_start_hour': '1'}))


### Describe the problem

No plot is being rendered. It fails with below error:
<img width="681" alt="TFMA render_plot JS error" src="https://github.com/tensorflow/model-analysis/assets/29692082/ece622ad-344f-4eee-b487-06b23c46dc78">

I tried to run the tutorial notebook on colab as well, but very first step of TFMA installation itself failed, so I couldn't proceed to generate this error in colab.

Please help. Thanks!!!


@siddharthroy,

I am able to run the `render_plot()` successfully in Colab Environment. Ref: screenshot below
TFMA installation in colab failed in your case because of Python version mismatch between Python 3.10 installed in colab and python supported version by TFMA. There is a workaround to install TFMA in colab. You can refer to attached [gist](https://colab.sandbox.google.com/gist/singhniraj08/168a9c158eb87290eb3f53cbb7a56172/copy-of-tfma_basic.ipynb).

The error you are facing looks like is because of Jupyter extensions missing. Please make sure Jupyter extensions are installed as shown in [tutorial](https://www.tensorflow.org/tfx/tutorials/model_analysis/tfma_basic#install_jupyter_extensions) and let us know if still you are facing issues. Thank you!

![image](https://github.com/tensorflow/model-analysis/assets/110809167/431b49ed-d9cf-4151-8561-57afec8c2164)




Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!
Repository: tensorflow/model-analysis
Issue: Feat: Have ability to return the HTML rather than creating the plots
Root cause: Unimplemented Feature Gap

This is a feature request so not following the template.
Let me know if I should anyways.

In tfdv there is the ability to get the HTML using `get_statistics_html` https://github.com/tensorflow/data-validation/blob/v1.13.0/tensorflow_data_validation/utils/display_util.py#L403-L466.
This is pretty handy as sometimes you do want to create HTML files you save on your computer.
tfma has a few functions:
- `render_slicing_attributions`
- `render_plot`
- `render_slicing_metrics`
- `render_time_series`

It would be very handy to be able to just return the HTML for these.
For context https://github.com/tensorflow/model-analysis/issues/10 solves `render_slicing_metrics` but not the others one as far as I am aware (thanks @Bobgy for the help)


Closing this as #10 solves it.
Repository: tensorflow/model-analysis
Issue: CVE in Pyarrow dependency
Root cause: External Dependency Breakage

### System information

-   **Have I written custom code**: no
-   **OS Platform and Distribution**:  N/A
-   **TensorFlow Model Analysis installed from**: binary
-   **TensorFlow Model Analysis version**: 0.45.0
-   **Python version**: N/A
-   **Jupyter Notebook version**: N/A
-   **Exact command to reproduce**: N/A

### Describe the problem

`pyarrow` in versions less than `14.0.1` contains the critical security vulnerability `CVE-2023-47248`.
If possible, please update the dependency of `pyarrow` to a version `>= 14.0.1`.


@vanHavel, Thank you for raising this feature request. We have already received this issue in other TFX child library and are working actively on updating the pyarrow dependency. We will update this thread once pyarrow dependency is updated. Thank you!
Repository: tensorflow/model-analysis
Issue: pip install tensorflow-model-analysis fails on WSL2 on Windows 11
Root cause: Environment Incompatibility

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: no
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 22.04.4 LTS in WSL in Windows 11
-   **TensorFlow Model Analysis installed from (source or binary)**: pip
-   **TensorFlow Model Analysis version (use command below)**:  tensorflow_model_analysis-0.45.0-py3-none-any.whl.metadata ?
-   **Python version**:   3.10.12
-   **Jupyter Notebook version**: n/a
-   **Exact command to reproduce**:  pip install tensorflow-model-analysis

### Describe the problem

pip install tensorflow-model-analysis fails on WSL2 on Windows 11

### Source code / logs

see the output here: https://pastebin.com/hSjycywU 
(it was too long for this editor box)

@akosmaroy,

I tried to install TFMA using `pip install tensorflow-model-analysis` in colab environment and it worked for me. Can you try to create a new virtual environment and try pip intall again. Also, you can try to pip install from https://pypi-nightly.tensorflow.org/ as shown in [installation steps](https://github.com/tensorflow/model-analysis?tab=readme-ov-file#installation). Thank you!

Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!
Repository: tensorflow/model-analysis
Issue: [Bug] Python 3.11 Support
Root cause: External Dependency Breakage

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: No
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 22.04
-   **TensorFlow Model Analysis installed from (source or binary)**: Source
-   **TensorFlow Model Analysis version (use command below)**: Commit `83cd413a68e8d743aca4b1319304cc525a736e88` (not unique to this commit)
-   **Python version**: 3.11
-   **Jupyter Notebook version**: N/A
-   **Exact command to reproduce**:
    ```bash
    $ python3.11 -m venv .venv
    $ source .venv/bin/activate
    $ pip install .
    $ python -c "import tensorflow_model_analysis as tfma"
    ```

### Describe the problem

There seems to be an issue with Python 3.11 support. For Python 3.11, protobuf 4.25.2 is installed, but the tfma protobuf modules seem to require protobuf 3.19 or earlier. This causes tfma to crash on import (shown below). 

### Source code / logs

```bash
$ python -c "import tensorflow_model_analysis as tfma"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/tmp.PTkD1g2WBK/model-analysis/tensorflow_model_analysis/__init__.py", line 30, in <module>
    from tensorflow_model_analysis.sdk import *
  File "/tmp/tmp.PTkD1g2WBK/model-analysis/tensorflow_model_analysis/sdk.py", line 60, in <module>
    from tensorflow_model_analysis.proto.config_pb2 import AggregationOptions
  File "/tmp/tmp.PTkD1g2WBK/model-analysis/tensorflow_model_analysis/proto/config_pb2.py", line 35, in <module>
    _descriptor.EnumValueDescriptor(
  File "/tmp/tmp.PTkD1g2WBK/model-analysis/.venv/lib/python3.11/site-packages/google/protobuf/descriptor.py", line 789, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
```
Repository: tensorflow/model-analysis
Issue: Error while using tfdv.generate_statistics_from_csv('path_to_csv_file')
Root cause: Validation Gap

I got a error while using this-

train_stats = tfdv.generate_statistics_from_csv('data_1.csv')
(the path to data is correct)

**Error-** ValueError: No line was found. [while running 'DecodeData/ParseCSVRecords']

@akki3d76  - Hi, there is a separate Github repository for Tensorflow data-validation. Request you to post the query in the tfdv repo(since it's related to tfdv). Please find the link to tfdv repo below.
[Link to tfdv repo](https://github.com/tensorflow/data-validation/issues)

Hence closing this issue.
Repository: tensorflow/model-analysis
Issue: Shows all zeros for model analysis
Root cause: Validation Gap

Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow-model-analysis

If you open a GitHub issue, here is our policy:

1.  It must be a bug, a feature request, or a significant problem with
    documentation (for small docs fixes please send a PR instead).
2.  The form below must be filled out.

**Here's why we have that policy**: TensorFlow Model Analysis developers respond
to issues. We want to focus on work that benefits the whole community, e.g.,
fixing bugs and adding features. Support only helps individuals. GitHub also
notifies thousands of people when issues are filed. We want them to see you
communicating an interesting problem, rather than being redirected to Stack
Overflow.

--------------------------------------------------------------------------------

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: Yes / No
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 16.04
-   **TensorFlow Model Analysis installed from (source or binary)**: PyPI
-   **TensorFlow Model Analysis version (use command below)**: 0.9.2
-   **Python version**: 2.7
-   **Jupyter Notebook version**: 1.0.0
-   **Exact command to reproduce**: Follow Chicago Taxi Example (local example) tutorial.

You can obtain the TensorFlow Model Analysis version with

`python -c "import tensorflow_model_analysis as tfma; print(tfma.VERSION_STRING)"`

### Describe the problem

I ran Chicago Taxi Example notebook but got all zeros when visualizing the sliced data, it looked like as  picture below.
![image](https://user-images.githubusercontent.com/44855457/48174905-bcd29a00-e344-11e8-994d-8fb3ba9b5c22.png)

Then I tried to use "Iris flower data set" as input and modified some code, but still got the same result shows as below.
![image](https://user-images.githubusercontent.com/44855457/48175152-ce687180-e345-11e8-8041-c95488433188.png)
I used the first column to sliced data.

I don't know whether import warning as below picture shows would affect the result or not.
![image](https://user-images.githubusercontent.com/44855457/48175822-944c9f00-e348-11e8-8b85-980f8c5aa9bd.png)
This import warning don't appeared after first run without restarting kernel.



Can you check two things:

(1) What are the example counts for each row in both cases?

(2) If you load the trained model and send predictions to it, does it give non-trivial predictions?



Sorry, typo: if you load the trained model and **get predictions from it**, does it give non-trivial predictions?

@yau991526  -  Hi,  could you please respond to the questions asked by @chuanyu  ?  Please keep us posted. Feel free to close if it is resolved.

Closing as it is in "awaiting response" status for long time. Please try with the latest versions and post your comments(if any). We will reopen. Thanks!