zhiminy/EvalEng huggingface dataset:
Repository: tensorflow/model-analysis
Issue: Multiclass confusion matrix / binarized metrics need class names, not just class IDs
Root cause: Unimplemented Feature Gap

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: Yes
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu
-   **TensorFlow Model Analysis installed from (source or binary)**: binary (PyPI)
-   **TensorFlow Model Analysis version (use command below)**: 0.22.2
-   **Python version**: 3.6.9
-   **Jupyter Notebook version**: 6.0.3
-   **Exact command to reproduce**:
```python
from tensorflow_model_analysis import EvalConfig
from tensorflow_model_analysis.metrics import default_multi_class_classification_specs
from google.protobuf.json_format import ParseDict

class = ['class_1', 'class_2', ...]

eval_config = {
    'model_specs': [
        {
            'name': 'rig_state',
            'model_type': 'tf_keras',
            'signature_name': 'serve_raw',
            'label_key': ...,
            'example_weight_key': 'sample_weight'
        }
    ],
    'metrics_specs': [
        {
            'metrics': [
                {
                    'class_name': 'MultiClassConfusionMatrixPlot',
                    'config': '"thresholds": [0.5]'
                },
                {'class_name': 'ExampleCount'},
                {'class_name': 'WeightedExampleCount'},
                {'class_name': 'SparseCategoricalAccuracy'},
            ],
        },
        {
            'binarize': {'class_ids': {'values': list(range(len(classes)))}},
            'metrics': [
                {'class_name': 'AUC'},
                {'class_name': 'CalibrationPlot'},
                {'class_name': 'BinaryAccuracy'},
                {'class_name': 'MeanPrediction'}
            ]
        }
    ],
    'slicing_specs': [...]
}
eval_config: EvalConfig = ParseDict(eval_config, EvalConfig())
```

### Describe the problem
Multiclass confusion matrices and binarized metrics should support class names, not just class IDs. Something like `'binarize': {'classes': [{'id': _id, 'name': name} for _id, name in enumerate(classes)]`. As it stands, having integer value IDs for the classes is meaningless to data scientists and business stakeholders looking at the TFMA visualizations.

We are looking into this, but don't yet have a clear solution. We would like to get the class id -> name mappings via the label vocab, but we don't always have access to the vocab so we are currently looking into getting the APIs we need.

Typically the vocab is computed/known in an upstream step... would it be the worst idea to update the EvalConfig proto to have a field for `vocab`?

@mdreves any thoughts about this suggestion?

The idea has been floated internally a few times and we are still considering it, but the preference is to find something that is bundled with the model so that the config is shared across components.

Closing this due to inactivity. Feel free to reopen and post your comments(if you still have queries on this).
Thank you!
Repository: tensorflow/model-analysis
Issue: Support for Python 3
Root cause: External Dependency Breakage

What's the timeline for supporting Python 3?

TensorFlow only runs on Python 3.5 and 3.6 on Windows. So those of us who work on Windows machines have a harder time trying out TFMA.

TensorFlow Model Analysis depends on Apache Beam, which is not compatible with Python 3 yet. I believe the Beam team is actively working on this ([see the bug on their end][0]). I'm not sure about the exact timeline, but my impression is that they might be able to get Python 3 support working towards the end of this year.

[0]: https://issues.apache.org/jira/browse/BEAM-1251

Apache Beam is almost Python 3 ready. I think this might be a good time to ensure the rest of model-analysis is as well. I'd be happy to give this a try once #12 is resolved.

Please keep tracking and post here once Apache Beam is Python 3 ready.

What is the approximate month/date that this will be ready? My team wants to know for planning purposes. Thanks.

As mentioned above, this is blocked by lack of Python 3 support in Beam. You can monitor the status on the Beam Python SDK Roadmap [page](https://beam.apache.org/roadmap/python-sdk/#python-3-support).

TFMA 0.13 has support for Python 3.5.

Other version of Python (like 3.6 and 3.7) will follow in future releases (after Beam officially supports them).

Should we close this?

There might still be an issue with Python 3.5 though: #46 

Closing this as [tfma](https://pypi.org/project/tensorflow-model-analysis/) supports 3.6,3.7,3.8 now.
Repository: tensorflow/model-analysis
Issue: Documentation for Slicer.SingleSliceSpec
Root cause: Documentation Deficiency

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: No
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: N/A
-   **TensorFlow Model Analysis installed from (source or binary)**: N/A
-   **TensorFlow Model Analysis version (use command below)**: N/A
-   **Python version**: N/A
-   **Jupyter Notebook version**: N/A
-   **Exact command to reproduce**: N/A

You can obtain the TensorFlow Model Analysis version with

`python -c "import tensorflow_model_analysis as tfma; print(tfma.version.VERSION)"`

### Describe the problem

Documentation for SingleSliceSpec is missing. The [link](https://www.tensorflow.org/tfx/tutorials/model_analysis/tfma_basic#run_tfma_and_render_metrics) from the tutorials goes to the following [page
](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/SingleSliceSpec
) and 404s

If documentation is available in any other format I'd love to look at it.

@pavanky,
In [Github TFMA Release.md](https://github.com/tensorflow/model-analysis/blob/master/RELEASE.md#bug-fixes-and-other-changes-2) it is specified that 

> Deprecated external use of tfma.slicer.SingleSliceSpec (tfma.SlicingSpec should be used instead).

So, please use [tfma.SlicingSpec](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/SlicingSpec) instead. Thanks!

@rmothukuru thanks! can the tutorial page be updated to the more proper usage?

Closing this issue as it has been resolved. Thanks!

Oky thanks for news

On Mon, Oct 26, 2020 at 2:29 PM rmothukuru <notifications@github.com> wrote:

> Closing this issue as it has been resolved. Thanks!
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/model-analysis/issues/82#issuecomment-716472830>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/APVZPWIS7ZXUBOFGL4KCMV3SMVJAHANCNFSM4QRAH4PQ>
> .
>
Repository: tensorflow/model-analysis
Issue: Module tensorflow_model_analysis, semver range 0.24.3 is not registered as a widget module
Root cause: External Dependency Breakage

Seems TFMA NPM module was not released for 0.24.3 (https://www.npmjs.com/package/tensorflow_model_analysis latest version is 0.24.2), which breaks UI. Console log:
```
manager.js:305 Uncaught (in promise) Error: Module tensorflow_model_analysis, semver range 0.24.3 is not registered as a widget module
```

TFMA npm module for 0.24.3 is available.

Can you please give a try and let us know if there are any issues.
Repository: truera/trulens
Issue: [BUG] Trulens does not evaluate the cost per inference in the case of Bedrock.
Root cause: Unimplemented Feature Gap

**Bug Description**
I'm trying to evaluate RAG using Trulense with the Bedrock Llama 2 model, but in the Trulense dashboard, it does not provide the cost of inference. It only shows $0. If I use OpenAI instead of Bedrock Llama 2, then it shows the price for every inference.

**To Reproduce**
```
llm = Bedrock( model="meta.llama2-70b-chat-v1", profile_name=profile_name)

tru_recorder = TruChain(chain,app_id='ChatApplication', feedbacks=[feedback function])

with tru_recorder as recording:
    llm_response = llm.chat("What's the capital of the USA?")
display(llm_response)
```

**Expected behavior**
Trulens should provide the cost for every inference in any LLM.

**Relevant Logs/Tracebacks**
Trulens dashboard using Bedrock Llama 2 model.
![image](https://github.com/truera/trulens/assets/163841019/cc520cea-f7f6-4d96-83a8-b1a69674254d)


**Environment:**
 - OS: Windows
 - Python Version: 3.10.1
 - TruLens version: 0.27.0
 - Versions of other relevant installed libraries: llama-index==0.10.25


<!-- Answer -->
It seems that Trulens does not support cost tracking for Bedrock models, including the Bedrock Llama 2 model you're using. This limitation is noted in the Trulens codebase, specifically in tests for Bedrock models where cost tracking is explicitly disabled. This is because Trulens currently only supports cost tracking for OpenAI models.

For Bedrock models, while there is an indication that cost tracking might be possible through the Litellm integration, it's mentioned that this could be inaccurate. This suggests that any cost tracking for Bedrock models would need to be implemented manually and might not align perfectly with Trulens' existing cost tracking features.

If cost tracking is critical for your use case with Bedrock models, you might need to look into custom solutions or workarounds that can approximate the cost based on your specific usage patterns and the pricing model provided by Bedrock.

## Sources
- [trulens_eval/tests/e2e/test_endpoints.py](https://github.com/truera/trulens/blob/main/trulens_eval/tests/e2e/test_endpoints.py)

<!-- Dosu Comment Footer -->
<details>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>


Hi @DhruvinKoshiya , do you know if Bedrock models are charged at some fixed rate that can be determined from model name?

@piotrm0 - Bedrock modes are charged per the input and output token. This [document](https://aws.amazon.com/bedrock/pricing/?refid=33b5edcf-0d26-4e1d-b868-603c42c06eaf) provides pricing of all the models.
Repository: truera/trulens
Issue: [FEAT] Feasibility to extend the record schema to accommodate custom fields
Root cause: Unimplemented Feature Gap

**Feature Description**
Currently the record schema has fixed set of fields, which are sufficient for generic use case, it would be convenient to have extended attributes like thread_id, session_id etc which makes it better fit for end to end solutioning.

**Reason**
As a workaround, I have tried subclassing Record class to accommodate additional fields, but adding those new fields to migrations has been the challenge as they are fixed in the first_revisions.py.

**Importance of Feature**
Very high for promoting trulens to production grade solution.
Repository: truera/trulens
Issue: [FEAT] Can I use a full local LLM instead of  openai?
Root cause: Documentation Deficiency

**Feature Description**
Describe the feature you are requesting. Try to reference existing implementations/papers/examples when possible.
I'd like to evaluate my RAG system using trulens with local LLM and embedding model, but I can not find any instructions to follow.
**Reason**
Why is this needed? Is there a workaround or another library/product you are using today to fulfill this gap?
My RAG system is deploying in an environment without networks, so I can't use openai.
**Importance of Feature**
What value does this feature unlock for you?
I can evaluate my RAG system in a local environment. 

Hi @Lauorie - you can do this today. See the [Ollama quickstart](https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/ollama_quickstart.ipynb)

Does this work for you?
Repository: truera/trulens
Issue: [BUG] Azure openai shows 404 error
Root cause: Interface Contract Mismatch

Hi guys, I'm trying to set Azure as a provider with the following code:
`azopenai = feedback.AzureOpenAI(deployment_name=COMPLETIONS_MODEL,base_url=endpoint, api_key=key, api_version=os.environ["OPENAI_API_VERSION"])`  and when I tries to apply feedback function:
`f_context_relevance(df.loc[[95]].question.values,df.loc[[95]].Context.values)
`

i got the error: `openai request failed <class 'openai.NotFoundError'>=Error code: 404 - {'statusCode': 404, 'message': 'Resource not found'}. Retries remaining=3.`
However, when I'm using the same parameters for LLM it works:
```
from langchain_openai import AzureChatOpenAI
llm = AzureChatOpenAI(
        azure_endpoint =endpoint,
        openai_api_version = os.environ["OPENAI_API_VERSION"],
        openai_api_key = key,
        openai_api_type = "azure",
        deployment_name = COMPLETIONS_MODEL,
        temperature = 0
    )
 
llm.invoke("what is ML?")
```
 Any ideas? (edited) 

<!-- Answer -->
The 404 error you're encountering with `feedback.AzureOpenAI` but not with `AzureChatOpenAI` is likely due to the `deployment_name` parameter not being correctly passed to the Azure OpenAI client in the `AzureOpenAI` class.

Ensure that the `deployment_name` is correctly set and passed to the Azure OpenAI client. The `deployment_name` should be included in the client arguments to avoid the 404 error.

Additionally, make sure that the environment variables `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY`, and `OPENAI_API_VERSION` are correctly set, as these are required for the `AzureOpenAI` class to function properly.

If you continue to experience issues, enable detailed logging to get more information about the error:

```python
import logging

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
```

This will ensure that any logging calls made within the `feedback.AzureOpenAI` class will output detailed information to the console or a log file, depending on your configuration.


<!-- Dosu Comment Footer -->
<details>
<summary>

#### Details
</summary>

**Sources**
- [trulens_eval/trulens_eval/feedback/provider/openai.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/openai.py)
- [trulens_eval/trulens_eval/feedback/provider/endpoint/openai.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/endpoint/openai.py)
- [docs/trulens_eval/api/provider/openai/azureopenai.md](https://github.com/truera/trulens/blob/main/docs/trulens_eval/api/provider/openai/azureopenai.md)

<br/>

**About Dosu**
This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
</details>

*To continue the conversation, mention @dosu.*
Repository: truera/trulens
Issue: [FEAT] ARES integration possible?
Root cause: Unimplemented Feature Gap

**Feature Description**
Is it possible to integrate Trulens with ARES (Stanford)? Is the recommended way to train the judges outside of Trulens and then use them in Trulens as custom feedback functions?

**Reason**
I'd like to have an easy way to compare ARES and Trulens stock evaluations.

**Importance of Feature**
To be honest, I'm not sure how much this is needed. I like the ARES approach theoretically, but I can't say whether it is better than the Trulens approach.

Hi @DallanQ - the ARES workflow seems a bit at odds with feedback functions, requiring a lot of setup and data to get started. If you've already got ARES results though, it wouldn't be that hard to generate TruLens evaluations from your data using the `TruVirtual` class ([example notebook](https://www.trulens.org/trulens_eval/getting_started/quickstarts/existing_data_quickstart/)) for comparison.

Thanks @joshreini1 - I'm still exploring this space, and that notebook helps.

### Follow-up
Hey @DallanQ,
- First off, we video called once about DL and family history like 2 years ago :) 
- Second, I don't see a PR or fork linked here, and searching '_`ares`_' in the repo doesn't bring anything up. Did you ever get this working? I'd love to see how if you did, so as to not reinvent the wheel.

Hi @DrewGalbraith nice to connect again :-)  I ended up going a different direction. I split retrieval evaluation and answer-synthesis evaluation into separate evaluation efforts and used a more traditional approach for retrieval evaluation.
Repository: truera/trulens
Issue: from trulens_eval.feedback import Groundedness Not working
Root cause: Documentation Deficiency

ImportError: cannot import name 'Groundedness' from 'trulens_eval.feedback' 

<!-- Greeting -->
Hey there, @saccpp! I'm here to help you with any bugs or questions you have while waiting for a human maintainer. Let's get started!

<!-- Answer -->
The `Groundedness` class is not part of the `trulens_eval.feedback` module. Instead, you should use the `GroundTruthAgreement` class. Update your import statement as follows:

```python
from trulens_eval.feedback import GroundTruthAgreement
```

If you intended to use a groundedness-related function, `GroundTruthAgreement` should serve your needs. Ensure your import statement matches the available classes or functions <sup>[[1]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/utils/imports.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/__init__.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Hey @saccpp - groundedness is now available from the feedback provider, similar to how the other feedback functions are called. This was changed to reduce the code you have to write and make it more straightforward to use.

The new syntax is just like this:

```python
provider = OpenAI() #or another feedback provider of your choice
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
)
```


Read more about the change in the 0.29 release notes here: https://github.com/truera/trulens/releases/tag/trulens-eval-0.29.0
Repository: truera/trulens
Issue: TruLens Docs (Feedback Functions) Graphic Improvement Suggestion
GitHub labels: documentation

enhancement

I'm not sure if this is the right place to suggest enhancements for the TruLens docs, but while [this graphic](https://www.trulens.org/trulens_eval/getting_started/core_concepts/feedback_functions/) is already helpful in explaining the surrounding text, it could be stronger.

### Suggestion
The graphic would be clear if the 'Meaningful' axis was a finite scale. It should range from the least meaningful evaluation (random binary classification, i.e. hallucination/not hallucination) to the most meaningful possible evaluation (a representative ground truth), approximately scaling everything else in between. Additionally, the upper bound of 'Scalable' would likely be that same random classification; I don't see a lower bound for this axis. The point placement along axes wouldn't feel so arbitrary if this finite scaling was worked into the graphic. 

### Disclaimer/Justification
I understand the most important part of the graphic is the general relationship of each point to each other as opposed to their placement relative to the scales. I contend with that in two ways:
1. Given the points aren't equidistant, especially along 'Scalable', there _does_ seem to be an effort to communicate size of disparity as opposed to just general direction.
2. Having 'Ground Truth Evals' sitting 2/3 along the image is unclear, suggesting there are even more informative/meaningful evaluations for a use-case not listed on this graphic. There aren't though, assuming ground truth is composed correctly.

Thanks for the feedback @DrewGalbraith - we'll take this into consideration
Repository: truera/trulens
Issue: [FEAT] Support for multi-modal RAG evaluation (including images, tables, etc.)
Root cause: Documentation Deficiency

**Feature Description**
Support for multi-modal RAG evaluation (including images, tables, etc.)

**Reason**
Multi-modal RAG is becoming more important.



Hi @joly-chen - multimodal RAG evaluation is already supported. Here's an example notebook you can adapt to your use case: https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/gemini_multi_modal.ipynb
Repository: truera/trulens
Issue: Python Quick Start (no notebook)
GitHub labels: enhancement

I don't use Jupyter notebook, but I do use LLMs. Could have a quick start that doesn't require Jupyter? 

Hi [krishangupta](https://github.com/krishangupta),

Can you see if the below is what you might be looking for? If so we can add it in.
https://github.com/truera/trulens/blob/quickstart_non_jupyter/trulens_eval/trulens_eval_quickstart.py

Note: don't forget to add API keys or you will run into authentication errors
At the start of the quickstart you will want to fill in environment variables, or you can set them outside the script too
```
os.environ["OPENAI_API_KEY"] = "..."
os.environ["HUGGINGFACE_API_KEY"] = "..."
```

This is now here in main
https://github.com/truera/trulens/blob/main/trulens_eval/generated_files/quickstart.py
Repository: truera/trulens
Issue: [BUG] LLAMA Index not working with Trullama
Root cause: External Dependency Breakage

**Bug Description**
When i tried to run the example notebook provided for llama index i'm facing so many errors

**To Reproduce**
![Screenshot from 2024-07-06 22-55-00](https://github.com/truera/trulens/assets/60032500/2552cab2-c3d2-4be9-8ac2-3ae3acc401a6)


**Tried uninstalling and reinstalling llam-index as well, doesnt help*

![image](https://github.com/truera/trulens/assets/60032500/883c2fed-da40-44b6-9c6c-be4f0841998d)



**Environment:**
 - Google Colab and local conda env both faced same issue
 - Python Version = 3.10

! pip install trulens_eval llama_index==0.10.11 -q

**Additional context**


Even though i could clearly import llama-index but with trullama it doesn't recognise llama-index at all

One of the worst libraries i ve used, full of code breaks and multiple dependencies errors





<!-- Greeting -->
Hey @Jaykumaran! 👋 I'm here to help you with any bugs, questions, or contributions. I'm a bot designed to assist you while you wait for a human maintainer.

<!-- Answer -->
The `ModuleNotFoundError` for the `llama-index` package occurs because the `trulens_eval` package uses a context-manager-based scheme to handle optional imports. This scheme provides clearer instructions when optional packages are not installed.

To resolve this issue, ensure that you have installed the `llama-index` package with a compatible version. The recommended command is:

```sh
pip install "llama-index>=v0.9.14.post3"
```

Additionally, there has been a recent update in the `trulens_eval` package (version 0.31.0) that addresses issues related to the `llama-index` integration. Make sure you are using the latest version of `trulens_eval`:

```sh
pip install --upgrade trulens_eval
```

This should align with the expected import paths and allow the `TruLlama` class to function correctly <sup>[[1]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/tru_llama.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/trulens_eval/RELEASES.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Thanks upgrading trulens-eval and changing Groundedness to GroundTruthAgreement  solved issue about llama-index not recognising error

Also,

!pip install "llama-index>=v0.9.14.post3" -q was helpful to avoid llamaindex.llms.openai import OpenAI not found error