zhiminy/EvalEng huggingface dataset:
Repository: tensorflow/model-analysis
Issue: Documentation for run_model_analysis is outdated
GitHub labels: stat:awaiting tensorflower

type:docs

### System information
N/A

tfma version 0.13.0


### Describe the problem

The documentation for `run_model_analysis` (can be found [here](https://github.com/tensorflow/model-analysis/blob/master/g3doc/api_docs/python/tfma/run_model_analysis.md)) is not according to the latest version of tfma.  For instance, `model_location` parameter does not exist anymore.

### Source code / logs
N/A

Thanks for bringing this to our attention! It looks like we neglected to rebuild the docs before releasing. We will do so shortly.

Besides, I think I found a bug on https://github.com/tensorflow/model-analysis/blob/6b949f2dd979cf2d8cf08cecd243e2dbe5dcec6c/tensorflow_model_analysis/api/model_eval_lib.py#L568

I'm getting:

```
...

/usr/local/lib/python2.7/dist-packages/tensorflow_model_analysis/api/model_eval_lib.pyc in ExtractEvaluateAndWriteResults(examples, eval_shared_model, output_path, display_only_data_location, slice_spec, desired_batch_size, extractors, evaluators, writers, write_config, num_bootstrap_samples)
    566 
    567   example_weight_metric_key = metric_keys.EXAMPLE_COUNT
--> 568   if eval_shared_model.example_weight_key:
    569     example_weight_metric_key = metric_keys.EXAMPLE_WEIGHT
    570 

AttributeError: 'str' object has no attribute 'example_weight_key'

```
This happened when I called
```
tfma.run_model_analysis(eval_shared_model='WORKING_DIR/model_analysis',
                        data_location='WORKING_DIR')
```
I would like to create a mockup with all the emulation of this bug but I find it too difficult to reproduce (sorry). I think that from the last version of tfma to this one you did some changes around `example_weight_key` and forgot to create some tests for it. 

After exploring a bit more, we found that the cause seems to be just deprecated documentation. Here is the contradiction:

- The [documentation](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/run_model_analysis) says that run_model_analysis
  - has the argument `example_weight_key` 
  - The first argument is `model_location`
- The [actual implementation](https://github.com/tensorflow/model-analysis/blob/6b949f2dd979cf2d8cf08cecd243e2dbe5dcec6c/tensorflow_model_analysis/api/model_eval_lib.py#L593) of the function 
  - does not have the argument `example_weight_key`
  - The first argument is `eval_shared_model`, which in turn has an `example_weight_key`


Thanks for looking into this. Yes, the API docs are out of date, and while we work on updating them, you can instead refer to the docstrings on the functions, either by looking at the code or by using Python's `help` builtin.

Can you please fix this issue, it's been there for three months and it's in the Getting started guide, the very first thing that users read.

Fixed.
Repository: tensorflow/model-analysis
Issue: Support for regression models?
Root cause: Documentation Deficiency

Is tfma support  regression models?? Like `tf.estimator.DNNLinearCombinedRegressor`

Do you have any info on this? Are we missing something or is it not supported yet?

Yes, all canned Estimators are supported. Did you run into problems trying to get a regression model to work?

We have access only to average_loss, label/mean, post_export_metrics/example_count, prediction/mean. Can we have access to regression tf.metrics like mae, mse, rmse, mcd ?

EDIT : 
I found this in the DNNRegressor.evaluate() doc :
> For canned estimators, the dict contains the loss (mean loss per mini-batch) and the average_loss (mean loss per sample). Canned classifiers also return the accuracy. Canned regressors also return the label/mean and the prediction/mean

I also learned that DNNRegressor use L2 loss function, so the average_loss on 1 example is a mse. Can you (or is it the tf.estimator team?) provide off-the-shelf mae, rmse? Thanks

You can add metrics you want to the canned Estimators using [`tf.estimator.add_metrics`](https://www.tensorflow.org/api_docs/python/tf/contrib/estimator/add_metrics)

It's exactly what I want, thanks! Sorry for the inconvenience.

EDIT : Well I can't seem to get it work.
I do : 
`def my_mae(labels, predictions):
    return {'mean_absolute_error': tf.metrics.mean_absolute_error(labels, predictions)}`
`estimator = tf.estimator.add_metrics(estimator, my_mae)`

And I get the following error:
> ERROR - Failed to convert object of type <type 'dict'> to Tensor. Contents: {'predictions': <tf.Tensor 'dnn/logits/BiasAdd:0' shape=(?, 1) dtype=float32>}. Consider casting elements to a supported type.

I tried predictions['logits'], predictions['activation'] (similarly to the predictions['logistics'] for a classifier in the doc) without success. Can you help me? Thanks!

Hello, any updates on my edit? I can't find any info on the internet. Thanks!

Sorry, GitHub doesn't send notifications for edits, so I didn't realise you had made an update.

The `predictions` argument is the same as the `predictions` dictionary in the `EstimatorSpec` returned by your `model_fn`. It looks like it's a dictionary containing a single key named `predictions`, so you probably want to do `mean_absolute_error(labels, predictions['predictions'])`.

It works, thanks!

Closing the issue due to the lack of recent activity. Please feel free to add additional information and open the issue again. Thanks!
Repository: tensorflow/model-analysis
Issue: KeyError: TupleType for load_eval_result from GCS with TFX 0.13.0
Root cause: External Dependency Breakage

TFMA's `load_eval_result('gs://bucket/path/to/evaluator/output')` on TFX 0.13 (Kubeflow 0.4.1 on GCP with KFP 0.1.19 and Python 3.5) gives the following error:

```
KeyError                                  Traceback (most recent call last)
<ipython-input-7-4f9962d5bddb> in <module>
----> 1 result = tfma.load_eval_result('gs://bucket/path/to/evaluator/output')

/usr/lib/python3.5/site-packages/tensorflow_model_analysis/api/model_eval_lib.py in load_eval_result(output_path)
    194            for key, plot_data in plots_proto_list]
    195 
--> 196   eval_config = load_eval_config(output_path)
    197   return EvalResult(
    198       slicing_metrics=slicing_metrics, plots=plots, config=eval_config)

/usr/lib/python3.5/site-packages/tensorflow_model_analysis/api/model_eval_lib.py in load_eval_config(output_path)
    115       tf.python_io.tf_record_iterator(
    116           os.path.join(output_path, _EVAL_CONFIG_FILE)))
--> 117   final_dict = pickle.loads(serialized_record)
    118   _check_version(final_dict, output_path)
    119   return final_dict[_EVAL_CONFIG_KEY]

/usr/lib/python3.5/site-packages/dill/_dill.py in _load_type(name)
    575 
    576 def _load_type(name):
--> 577     return _reverse_typemap[name]
    578 
    579 def _create_type(typeobj, *args):

KeyError: 'TupleType'
```

CMLE's Python version is also 3.5 (as it was provided as a training input).

For TFX 0.12 on the same Kubeflow cluster it works as expected. I updated TFMA's version according to the [instructions](https://github.com/tensorflow/tfx#compatible-versions).

TFDV works without any issues (in the same notebook). And it also works fine when using Airflow for local development.

Could it be related to [this](https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype)?

Seems like it could be related to the Stackoverflow question you linked to (thanks for the pointer!): could you try the workaround suggested there and see if it fixes things?

Do something like:
```
dill._dill._reverse_typemap['TupleType'] = type
```

before calling `load_eval_result` and see if that makes the error go away.

I did, but that leads to `KeyError: 'ObjectType'`.

If I then add `ObjectType` in the same way with the reverse type map I get `TypeError: type.__new__(SingleSliceSpec): SingleSliceSpec is not a subtype of type`.

Any ideas what the problem might be?

Closing this as there has been no activity for some time.Please feel free to reopen.
Repository: tensorflow/model-analysis
Issue: WriteTFRecord includes full path to output file
Root cause: Algorithmic Error

This is not a very significant bug, but it is a bit unexpected behavior. In [the code](https://github.com/tensorflow/model-analysis/blob/545106a1725e4203788b6a1469108c97954f8a7d/tensorflow_model_analysis/api/model_eval_lib.py#L315) output files are written via a `beam.io.WriteToTFRecord` PTransform, but the name is assigned as `'WriteTFRecord(%s)' % output_file`. The problem is that the `output_file` is generally an absolute path to some file, using `/` as a delimiter. In the Google Cloud Dataflow UI it seems like `/` in the step name is used to group together similar operations, so you end up with a deeply-nested graph structure where `WriteTFRecord(gs://path/to/file)` gets mapped to `'WriteTFRecord(gs:' > '' > 'path' > 'to' > 'file'`.

The default writers now uses MetricsAndPlotsWriter which has a stage name of 'WriteMetricsAndPlots', so this is no longer an issue.
Repository: tensorflow/model-analysis
Issue: tfma.run_model_analysis only handles uncompressed tfrecords
Root cause: Architectural Constraint

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: No
-   **TensorFlow Model Analysis version (use command below)**:0.13.2 + master
### Describe the problem

The type of compression when reading TFRecords in `run_model_analysis` is [hard-coded](https://github.com/tensorflow/model-analysis/blob/master/tensorflow_model_analysis/api/model_eval_lib.py#L769) to compressed. However, beam handles correctly other kind of compression. The type CompressionTypes.AUTO should be used: " Default value is CompressionTypes.AUTO, in which case the file_path's extension will be used to detect the compression."
Repository: tensorflow/model-analysis
Issue: No Output in Jupyter Notebook
Root cause: Environment Incompatibility

Instrumented a model according to the docs. Created a notebook with a cell to render the tfma widget.

```python
eval_result = tfma.run_model_analysis(
  model_location=MODEL_LOCATION,
  data_location=EVAL_TFRECORDS_LOCATION,
  file_format='tfrecords')

tfma.view.render_slicing_metrics(eval_result)
```

This gives me some INFO log messages but no widget. I have installed the nb extensions as indicated in the docs. I used the `--user` flag for installations.

This output is received in the notebook:

```
INFO:tensorflow:Restoring parameters from ./data/tfma/1523504954/variables/variables

INFO:tensorflow:Restoring parameters from ./data/tfma/1523504954/variables/variables

INFO:tensorflow:Restoring parameters from ./data/tfma/1523504954/variables/variables

INFO:tensorflow:Restoring parameters from ./data/tfma/1523504954/variables/variables

INFO:tensorflow:Restoring parameters from ./data/tfma/1523504954/variables/variables

INFO:tensorflow:Restoring parameters from ./data/tfma/1523504954/variables/variables

INFO:tensorflow:Restoring parameters from ./data/tfma/1523504954/variables/variables

INFO:tensorflow:Restoring parameters from ./data/tfma/1523504954/variables/variables
```

Hi Nathan,

Since you are using --user, I suppose you are not using virtualenv. Could you please try it with virtualenv first?

If you are using virtualenv, please try running the [local playground](https://github.com/tensorflow/model-analysis/blob/master/examples/chicago_taxi/chicago_taxi_tfma_local_playground.ipynb) notebook. Does it work? If not, please check to see if any error occurred in the browser. Look up how to open developer toolbar in your choice of browser. (eg: press F12 if you are using Chrome on linux.)

Thanks!



Closing due to inactivity.

Hello, I am having the same issue. I am running the installation with --sys-prefix
My render code looks like that
`
tfma.view.render_slicing_metrics(tfma_result_1, slicing_column='trip_start_hour')
`

output:

`U2xpY2luZ01ldHJpY3NWaWV3ZXIoY29uZmlnPXsnd2VpZ2h0ZWRFeGFtcGxlc0NvbHVtbic6ICdwb3N0X2V4cG9ydF9tZXRyaWNzL2V4YW1wbGVfY291bnQnfSwgZGF0YT1beydtZXRyaWNzJzrigKY=
`
I have attached the code. I am using the Chicago taxi example to try to make it work.

Many thanks for any advice,
eilalan
[or_chicago_taxi_tfma_local_playground (1).ipynb.txt](https://github.com/tensorflow/model-analysis/files/2637481/or_chicago_taxi_tfma_local_playground.1.ipynb.txt)




same issue.  Doesn't render anything in Jupyter notebook. 

@pfyang  this is about running/rendering in Jupyter, not virtualenv (or Colab)
Repository: tensorflow/model-analysis
Issue: `static` folder missing when installing Jupyter notebook extensions
Root cause: Architectural Constraint

### System information
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 16.04
-   **TensorFlow Model Analysis installed from (source or binary)**: binary
-   **TensorFlow Model Analysis version (use command below)**: 0.13.2
-   **Python version**: 2.7 / 3.5
-   **Jupyter Notebook version**:  5.7.8
-   **Exact command to reproduce**:

### Describe the problem

I'm following the instructions to install the TFMA Jupyter extension via Conda and I realized that at `jupyter nbextension install --py --symlink tensorflow_model_analysis`:

```
% jupyter nbextension install --py --symlink tensorflow_model_analysis
/home/benjamintan/miniconda3/lib/python3.7/site-packages/apache_beam/__init__.py:84: UserWarning: Running the Apache Beam SDK on Python 3 is not yet fully supported. You may encounter buggy behavior or missing features.
  'Running the Apache Beam SDK on Python 3 is not yet fully supported. '

Installing /home/benjamintan/miniconda3/lib/python3.7/site-packages/tensorflow_model_analysis/static -> tfma_widget_js
Symlinking: /usr/local/share/jupyter/nbextensions/tfma_widget_js -> /home/benjamintan/miniconda3/lib/python3.7/site-packages/tensorflow_model_analysis/static
- Validating: problems found:
   X  require: /usr/local/share/jupyter/nbextensions/tfma_widget_js/extension.js
  OK section: notebook
Full spec: {'section': 'notebook', 'src': 'static', 'dest': 'tfma_widget_js', 'require': 'tfma_widget_js/extension'}

    To initialize this nbextension in the browser every time the notebook (or other app) loads:

          jupyter nbextension enable tensorflow_model_analysis --py
```

What I've found is that the `/home/benjamintan/miniconda3/lib/python3.7/site-packages/tensorflow_model_analysis/static` doesn't exist. This causes the widgets not to load.

However, when I manually copy the files from Github to the aforementioned directory, the widgets then load.

I've tried to use `--system` and `--sys-prefix` without any success.



Hi,

We currently do not support TFMA with conda. Could you please try using pip install with virtualenv?

Also, please use --sys-prefix when using virtualenv as suggested in the installation insrtuction:
  https://github.com/tensorflow/model-analysis#installation

Thanks!

we're also seeing this issue at Twitter and have PR #55 out to resolve, if someone could take a look.

Closing this as there has been no update lately.Please fee free to reopen based on above comment trace.
Repository: tensorflow/model-analysis
Issue: Missing npm package for v0.47.1
Root cause: Unimplemented Feature Gap

# Details

- There seems no npm package for tensorflow_model_analysis==0.47.1([link](https://www.npmjs.com/package/tensorflow_model_analysis?activeTab=versions))
- Currently we make HTML file of evaluation result with `ipywidgets` package(see below code)
- If we run below code with tensorflow_model_analysis==0.47.1 and then open that html with browser(using Chrome & Safari), we get error something like this
  > `cannot find https://unpkg.com/tensorflow_model_analysis@0.47.1/dist/index.js`
- I think it is caused by missing npm package for 0.47.1
- Can you update npm package for 0.47.1(Cannot use 0.47.0 or older version since we use TFX 1.16 and it requires tensorflow_model_analysis >= 0.47.0 and 0.47.0 has circular import error)

## Code for html
```python
import tensorflow_model_analysis as tfma
from ipywidgets.embed import embed_minimal_html

slicing_specs = ...
eval_result = tfma.load_eval_result(...)
metric_views = []
for slicing_spec in slicing_specs:
    metric_views.append(tfma.view.render_slicing_metrics(eval_result, slicing_spec=slicing_spec))

embed_minimal_html("result.html", metric_views)
```

Hey @dante-lee 
Sorry for addressing the issue late. Thank you for bringing this to our attention. We will work on this and keep you updated.
Thank you!

@janasangeetha 
Thank you for your consideration on this issue. Is there any update on this issue?

Hey @dante-lee 
We have released latest version [0.48.0](https://www.npmjs.com/package/tensorflow_model_analysis/v/0.48.0). Please try the same and let us know if you are still facing any issues.
Thank you!

@janasangeetha 
Thanks for your reply!

I have one more question. I'm currently using TFX 1.16.0 and according to [dependencies.py](https://github.com/tensorflow/tfx/blob/c4230755a5453fa0625118227fb1ed1b824bd4d9/tfx/dependencies.py#L119) in tfx source code, it seems like tfx==1.16.0 requires tensorflow-model-analysis 0.47.x which could lead to version conflict.

I wonder is it ok to use tensorflow-model-analysis 0.48.0 with TFX 1.16.0.(On my local machine, I was failed to install tfx 1.16 with tensorflow-model-analysis 0.48.0)

Hey @dante-lee 
We are working on upgrading TFX. We’ll keep you updated as soon as we have more information.
Appreciate your patience!

@janasangeetha 
OK thanks!
Repository: tensorflow/model-analysis
Issue: Exporter not available for tfma OSS
Root cause: Architectural Constraint

Currently, `tensorflow_model_analysis/export_only/__init__.py` imports `tensorflow_model_analysis.eval_saved_model.export` and `tensorflow_model_analysis.eval_saved_model.exporter`, but looking at `tensorflow_model_analysis/eval_saved_model/__init__.py`, we see that `eval_saved_model`

> is in maintenance mode. It is not available in TFMA OSS.

This creates two issues

- There is no way to use `tensorflow_model_analysis.export_only`
- This causes test failures because `tensorflow_model_analysis.export_only.export` gets imported when running tests in the CI. 

These imports were removed in #189 because they were breaking tests, and I think there's no real chance of these internal-only features getting merged into this repository at this point. I propose closing this issue - thoughts?

> These imports were removed in [#189](https://github.com/tensorflow/model-analysis/pull/189) because they were breaking tests, and I think there's no real chance of these internal-only features getting merged into this repository at this point. I propose closing this issue - thoughts?

I'd like to be optimistic that the developers of this repo will implement these exporter features. 

Tagging @embr for input - any chance these features can get merged here?

Hi @smokestacklightnin Thanks for reporting this isssue!

@mdreves, could you please take a look at this issue? `export_only` is currently unusable due to imports from `eval_saved_model`, which isn’t available in OSS. Thanks!
Repository: tensorflow/model-analysis
Issue: Source code released not Python2 compatible
Root cause: Validation Gap

### System information
Jupyter Python2 on linux and run 0.13.2 code downloaded from github

### Describe the problem

Issue also posted as: https://stackoverflow.com/questions/57047799/copy-of-tensorflow-model-analysis-code-does-not-run-in-python2/57048469#57048469

Released code version 0.13.2 and 0.13.1 both not python2 compatible

### Source code / logs

compiler error at tensorflow_model_analysis/types.py:54 because type hint not supported in python2

Adding more information from the stack overflow issue mentioned by the user:

I downloaded tfma source code 0.13.2 and ran it in jupyter notebook locally with python2, and it does not work

**Working code:**

```
import tensorflow as tf

import third_party.tensorflow_model_analysis.tensorflow_model_analysis.version as v
print('TFMA version: {}'.format(v.VERSION_STRING))

Output is: 

TFMA version: 0.13.1
```

**Not working code:**

```
import third_party.tensorflow_model_analysis.tensorflow_model_analysis.types as types
model_location = '/tmp/model0712/export/tfma/1562953448'

eval_shared_model = types.EvalSharedModel(model_path=model_location)

Output is: 

....third_party/tensorflow_model_analysis/tensorflow_model_analysis/types.py", line 54
    value: float,
         ^
SyntaxError: invalid syntax
```

Seems type hint thing is only supported in Python 3, so above error is reasonable to me.

However does it mean tfma code cannot be run in Python 2 although github page says that it is Python2.7 compatible?

Or did I missed anything?

Does TFMA work if you follow the instructions in https://github.com/tensorflow/model-analysis/blob/master/g3doc/get_started.md? 

The instructions seems not update to date, API changed for run_model_analysis (parameter model_location get replaced with EvalSharedModel), others seems still applies

Closing as documentation has been updated and shortly we will also no longer support python3.
Repository: tensorflow/model-analysis
Issue: `GenericChangeThreshold` relative validation inconsistency
Root cause: Documentation Deficiency

The implementation of relative threshold validation doesn't align with the documented/expected behaviour.

From [the docs](https://github.com/tensorflow/model-analysis/blob/master/docs/model_validations.md#genericchangethreshold):
> Aboslute change is calculated as the value diference between the metrics of the candidate and baseline model, namely, v_c - v_b where v_c denotes the candidate metric value and v_b denotes the baseline value. Relative value is the relative difference between the metric of the candidate and the baseline, namely, v_c/v_b.

However, the implementation instead calculates `(v_c - v_b) / v_b`, NOT `v_c/v_b` as the docs very clearly state.
https://github.com/tensorflow/model-analysis/blob/422dbd37b31e286bda3311028b5d9ff7b76df7ac/tensorflow_model_analysis/evaluators/metrics_validator.py#L88

The easiest fix would be to update the docs to correctly describe the implementation, but I'm not sure what the desired behaviour actually is.

Hi @stefandominicus-takealot, Thanks for pointing this out!

@mdreves, There’s an inconsistency between the docs and the implementation for relative threshold validation. Could you please clarify which behavior is intended? Thanks!