zhiminy/EvalEng huggingface dataset:
Repository: truera/trulens
Issue: ValueError in the tensorflow 2 / keras notebook provided in the website.
Root cause: External Dependency Breakage

ValueError in the tensorflow 2 / keras [notebook](https://colab.research.google.com/drive/1f-ETsdlppODJGQCdMXG-jmGmfyWyW2VD?usp=sharing) provided in the [website](https://www.trulens.org/welcome/).

The error occurs when executing the 6th code cell, it contains following code:
```
infl = InputAttribution(model)
attrs_input = infl.attributions(x_pp)
```

Error Log:
```
ValueError                                Traceback (most recent call last)
<ipython-input-18-9b10bbb33c8d> in <module>()
      1 infl = InputAttribution(model)
----> 2 attrs_input = infl.attributions(x_pp)

/usr/local/lib/python3.7/dist-packages/trulens/nn/attribution.py in attributions(self, *model_args, **model_kwargs)
    269             to_cut=self.slice.to_cut,
    270             intervention=D,
--> 271             doi_cut=doi_cut)
    272         # Take the mean across the samples in the DoI.
    273         if isinstance(qoi_grads, DATA_CONTAINER_TYPE):

/usr/local/lib/python3.7/dist-packages/trulens/nn/models/keras.py in qoi_bprop(self, qoi, model_args, model_kwargs, doi_cut, to_cut, attribution_cut, intervention)
    430             intervention, DATA_CONTAINER_TYPE) else [intervention]
    431 
--> 432         Q = qoi(to_tensors[0]) if len(to_tensors) == 1 else qoi(to_tensors)
    433 
    434         doi_tensors, intervention = self._prepare_intervention_with_input(

/usr/local/lib/python3.7/dist-packages/trulens/nn/quantities.py in __call__(self, y)
    109 
    110     def __call__(self, y: TensorLike) -> TensorLike:
--> 111         self._assert_cut_contains_only_one_tensor(y)
    112 
    113         if self.activation is not None:

/usr/local/lib/python3.7/dist-packages/trulens/nn/quantities.py in _assert_cut_contains_only_one_tensor(self, x)
     76                 '`{}` expected to receive an instance of `Tensor`, but '
     77                 'received an instance of {}'.format(
---> 78                     self.__class__.__name__, type(x)))
     79 
     80 

ValueError: `MaxClassQoI` expected to receive an instance of `Tensor`, but received an instance of <class 'keras.engine.keras_tensor.KerasTensor'>
```

It looks like TensorFlow 2.6 introduced some breaking changes that caused the notebook to fail. I have a PR in progress that seems to fix these issues. When this has been resolved, we'll update TruLens on pypi, so you can upgrade your package with pip, or pull (if you're running from source). Once the pip package has been redeployed the colab notebook should work again as well.

The fix has been deployed and the notebook should work now. Please ensure that you're using trulens version 0.0.8 (e.g., with `pip list`) to make sure you get the fix.
Repository: truera/trulens
Issue: Allow get query parameters?
Root cause: Unimplemented Feature Gap

Hi, I have an application that uses Trulens scores and I would like to link that application to the Evaluations page.

Is it possible to enable query parameters so I can generate a url with the App Filters?

Would be nice to write the filters on the URL:
https://docs.streamlit.io/library/api-reference/utilities/st.experimental_set_query_params

Then get the filters back:
https://docs.streamlit.io/library/api-reference/utilities/st.experimental_get_query_params

Thanks!

Hey @llermaly - this should be doable. I'll take a look at it.

@llermaly this is merged in main and will be available in our next release, 0.13.0 (tomorrow).
Repository: truera/trulens
Issue: Quickstart example not actually logging
Root cause: Documentation Deficiency

The change to use a context manager doesn't actually log anything:

```
with tru_llm_standalone as recording:
    llm_standalone(prompt_input)
```

If instead I change that to

```
tru_llm_standalone.call_with_record(prompt_input)
```

I see records logged.

Hi @mdagost , we are in the middle of changing the quickstarts and documentation. Hopefully will be out today.

Please try

```
with tru_llm_standalone as recording:
    tru_llm_standalone.app(prompt_input)
```

The context manager does require a class to be logged; so we provide it via the `app` attribute

This was deployed yesterday:

https://colab.research.google.com/github/truera/trulens/blob/releases/rc-trulens-eval-0.13.0/trulens_eval/examples/colab/quickstarts/text2text_quickstart_colab.ipynb 

Note that in the solution above `app` is not `tru_llm_standalone` but rather an instance of a class that wraps that callable.
Repository: truera/trulens
Issue: API to retrieve feedback function results after a run
Root cause: Unimplemented Feature Gap

Currently, it's hard to retrieve feedback function results in a structured way. It would be nice to have a:
`get_feedbacks(record)` API

Added one option to get the feedback immediately after recording a record in this PR: https://github.com/truera/trulens/pull/480 .
Repository: truera/trulens
Issue: Log the production data into Bigquery and retrieve the analysis based on it
Root cause: Documentation Deficiency

Hello,
I'm wondering how we can log the production data into Bigquery database and then retrieve them later to analyze the results.

Hi @noshad-vida !

We have integration with SQLAlchemy that allows you to do this.

Check out our docs on using SQLAlchemy with TruLens: https://www.trulens.org/trulens_eval/where_to_log/

and this article on using BigQuery with SQLAlchemy:
https://googleapis.dev/python/pybigquery/0.7.0/


@joshreini1 thanks for your response. I just don't find a sample code to display the evaluation dashboard using the logged data.

Once you initialed Tru with bigquery:

from trulens_eval import Tru
tru = Tru(database_url=“your bigquery db url/auth here")

All logging will be done there, and tru.run_dashboard() will launch from the logs in that db. Let me know if you run into any snags doing this as you’re likely the first setting this up in BigQuery!


@noshad-vida were you able to use this? Closing for now unless you run into issues
Repository: truera/trulens
Issue: Handle single-entry dictionary output in default QoIs
Root cause: Validation Gap

It seems that the output of torchvision models is an OrderedDict and not a torch.Tensor, which by default has a single key, `'out'`. Currently a custom QoI is required to handle this type of output:

```python
class TorchvisionMaxClassQoI(MaxClassQoI):
    def __call__(self, y):
        super().__call__(y['out'])
```

It would be nice to handle this case by default instead of requiring a custom QoI. If the input, `y` to a QoI is a `dict` or `OrderedDict` with exactly one entry, then that entry should be used in place of `y` rather than raising an exception (as this is unambiguously the output we would care about).

There is also the cut accessor which can be used for this.
Repository: truera/trulens
Issue: The helpfuless prompt use the wrong prompt template (harmfullness instead of helfulness)
Root cause: Validation Gap

file_path : trulens_eval/trulens_eval/feedback/prompts.py
line 83 : LANGCHAIN_**HELPFULNESS**_PROMPT = v2.**Harmfulness**.prompt.template

Thanks @MirageMika !

Here's the fix: https://github.com/truera/trulens/pull/594

Will get this in the next release.
Repository: truera/trulens
Issue: Inconsistent evaluation metrics (groundedness) with OpenAI APIs
Root cause: Algorithmic Error

Hi Team,

I'm calculating Groundedness score which is not as expected for the below scenarios (over multiple runs):
1. Inconsistent over same user input & response
2. Wrong (0) over expected (correct) response, even if it's an exact match from the context
    1. On few initial runs, I was getting a score of 1
    3. For later runs with score of 0, the reason was - 'No evidence found'

I'm using 'gpt-3.5-turbo' model & as per my understanding from the source code, the below stands true:
1. OpenAI uses 'groundedness.groundedness_measure_with_cot_reasons' - a measure to track if the source material supports each sentence in the statement & also uses chain of thought methodology and emits the reasons
2. The prompt that goes to OpenAI requires only user input & response
{"messages": [{"role": "system", "content": "You are a INFORMATION OVERLAP classifier providing the overlap of information between a SOURCE and STATEMENT.\\nFor every sentence in the statement, please answer with this template:\\n\\nTEMPLATE: \\nStatement Sentence: <Sentence>, \\nSupporting Evidence: <Choose the exact unchanged sentences in the source that can answer the statement, if nothing matches, say NOTHING FOUND>\\nScore: <Output a number between 0-10 where 0 is no information overlap and 10 is all information is overlapping.\\nGive me the INFORMATION OVERLAP of this SOURCE and STATEMENT.}

What could be the possible reasons for the inconsistency in results with such a powerful model & what can I do to manage that? Are there any other OpenAI parameters that's being set in trulens source code?

Hi there, TruLens developer here and happy to help.

I'm taking a look at the issue in the next few days and will be adding a comprehensive benchmark dataset to smoke test some of the groundedness feedback function providers (essentially the evals on evals). Hopefully this helps uncover the root cause here. Any chance you might be able to share some screenshots or snippets of the query/response pairs that you were using so that I can try reproducing?

Thanks!
Repository: truera/trulens
Issue: Conciseness feedback does not take into account the data
Root cause: Validation Gap

Only the template of conciseness feedback passes through the provider, so it gives 10/10 every time.

Provider: OpenAI

Thanks @MirageMika ! Taking a look

This will be fixed by: https://github.com/truera/trulens/pull/636

Available in the next release.

thanks!
Repository: truera/trulens
Issue: [duplicated]
Root cause: Validation Gap

While experimenting with the `language_verification` feedback function (which uses Hugging Face models, not OpenAI) in the `trulens_eval` library, I encountered the following issue:

### Issue Explanation 
When using the `TruBasicApp` class and attempting to run it within a context manager, an error is thrown if an OpenAI API key is not provided, even when OpenAI functionalities are not explicitly invoked.
```
from trulens_eval import TruBasicApp
translate_recorder = TruBasicApp(translate_app, app_id="translate-app1", feedbacks=feedbacks)

with translate_recorder as recording: # issue here
    for prompt in prompts:
        print(prompt)
        _ = translate_recorder.app(prompt)
        
tru_record = recording.records[0]
```
I might be missing some option that allows me to disable all OpenAI functionalities, or introduce a parameter, such as `with_openai=False`, in the relevant methods. If so, please let me know how to implement it.

### (Possible) Root Cause
Upon investigation, the issue seems to stem from the `track_all_costs_tally()` static method (in `instruments.py`). This method, in turn, calls the track_all_costs() method with the default argument `with_openai=True`. The code should be able to handle a missing API key with the ApiKeyError exception handling. 

```    
def track_all_costs(
        thunk: Thunk[T],
        with_openai: bool = True,
        with_hugs: bool = True,
        with_litellm: bool = True
    ) -> Tuple[T, Sequence[EndpointCallback]]:
        """
        Track costs of all of the apis we can currently track, over the
        execution of thunk.
        """
        endpoints = []
        if with_openai:
            # TODO: DEPS
            from trulens_eval.feedback.provider.endpoint.openai import \
                OpenAIEndpoint

            try:
                e = OpenAIEndpoint()
                endpoints.append(e)
            except ApiKeyError:
                logger.debug(
                    "OpenAI API keys are not set. "
                    "Will not track usage."
                )
```

Instead, it raises openai's own exception:
```
OpenAIError("The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable")
```

From what I understand, the problem is that `_check_key("OPENAI_API_KEY", silent=True, warn=False)`, the function responsible for raising an ApiKeyError, is commented out and never being invoked during the initialization of the OpenAIEndpoint.
Repository: truera/trulens
Issue: Feedback function 'not_toxic()' returns score of 'toxic'
Root cause: Algorithmic Error

The feedback function `not_toxic()` is returning the score of 'toxic', instead of 'not-toxic'. So, if the input does not contain any toxic content, it's flagged in the evaluation tables.

Function definition:

```
    def not_toxic(self, text: str) -> float:
        # ...

        assert len(text) > 0, "Input cannot be blank."

        max_length = 500
        truncated_text = text[:max_length]
        payload = {"inputs": truncated_text}
        hf_response = self.endpoint.post(
            url=HUGS_TOXIC_API_URL, payload=payload
        )

        for label in hf_response:
            if label['label'] == 'toxic':  **## <-- HERE**
                return label['score']

        raise RuntimeError("LABEL_2 not found in huggingface api response.")

```


Thanks for catching @saradiazdelser ! Will fix this soon

Opened up a PR here: https://github.com/truera/trulens/pull/633

Will get this in the next release.
Repository: truera/trulens
Issue: Need clarification on three RAG evaluation metric

Hi:

Really interesting work on RAG evaluation metrics. However, I am unable to understand how these evaluation functions work. Could you please point me to some relevant documentation that explains the mathematical formula for all three:

1. Context Relevance
2. Groundedness
3. Answer Relevance

Would really appreciate your help...thanks much in advance.


Hi @vr25 - these are model-based metrics.

1. Context relevance can be measured with smaller BERT-style models, embedding distances, or with LLMs.
2. Groundedness can be measured using smaller NLI models or LLMs
3. Answer relevance can BERT-style models, embedding distances, or with LLMs.

TruLens gives you the flexibility to use these different options for metric generation.

Does that help?

@joshreini1 This helps...thanks for the clarification.
Repository: truera/trulens
Issue: version that satisfies the requirement trulens==0.18.1
Root cause: Documentation Deficiency

What version of trulens would be best for following the notebook examples?

[The example](https://github.com/truera/trulens/blob/main/trulens_eval/examples/quickstart/text2text_quickstart.ipynb)

![image](https://github.com/truera/trulens/assets/18591926/e220fa91-449c-410d-92b5-947a1e529b67)
```
ERROR: Could not find a version that satisfies the requirement trulens==0.18.1 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.11, 0.0.12, 0.0.12.1, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4)
ERROR: No matching distribution found for trulens==0.18.1
```

Hi @isayahc - will fix this in the notebook.

Please use `pip install trulens_eval==0.18.1`
Repository: truera/trulens
Issue: error: cannot initialize trulens_eval.Bedrock object 
Root cause: Interface Contract Mismatch

Version: 0.18.0

Code:

```
from trulens_eval import TruChain, Feedback, Tru, Bedrock
tru = Tru()
tru.reset_database()

# Initialize Huggingface-based feedback function collection class:
bedrock = Bedrock(model_id = "anthropic.claude-v2", region_name="us-west-2")
```

Error:

```
ValidationError                           Traceback (most recent call last)
Cell In[16], line 2
      1 # Initialize Huggingface-based feedback function collection class:
----> 2 bedrock = Bedrock(model_id = "anthropic.claude-v2", region_name="us-west-2")
      4 # Define a language match feedback function using HuggingFace.
      5 #f_relevance = Feedback(bedrock.relevance).on_input_output()
      6 # By default this will check language match on the main app input and main app
      7 # output.

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/trulens_eval/feedback/provider/bedrock.py:51, in Bedrock.__init__(self, model_id, region_name, *args, **kwargs)
     46 self_kwargs['region_name'] = region_name
     47 self_kwargs['endpoint'] = BedrockEndpoint(
     48     region_name=region_name, *args, **kwargs
     49 )
---> 51 super().__init__(
     52     **self_kwargs
     53 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py:43, in LLMProvider.__init__(self, *args, **kwargs)
     40 self_kwargs = dict()
     41 self_kwargs.update(**kwargs)
---> 43 super().__init__(
     44     **self_kwargs
     45 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py:27, in Provider.__init__(self, name, **kwargs)
     23 def __init__(self, name: str = None, **kwargs):
     24     # for WithClassInfo:
     25     kwargs['obj'] = self
---> 27     super().__init__(name=name, **kwargs)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/trulens_eval/utils/pyschema.py:617, in WithClassInfo.__init__(self, class_info, obj, cls, *args, **kwargs)
    614     class_info = Class.of_class(cls, with_bases=True)
    616 kwargs[CLASS_INFO] = class_info
--> 617 super().__init__(*args, **kwargs)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/pydantic/main.py:341, in pydantic.main.BaseModel.__init__()

ValidationError: 1 validation error for Bedrock
model_engine
  field required (type=value_error.missing)
```


Hey @arm-diaz looks like you figured this out based on your other ticket. The `model_id` arg just needs to be changed to `model_engine`. We'll change this in the example notebook too
Repository: truera/trulens
Issue: requires_gradients_ is being called on model_args, which are not guaranteed to be tensors nor be connected via gradients
Root cause: Validation Gap

in models.pytorch 
```
model_args = ModelWrapper._nested_apply(
                model_args, lambda model_args: model_args.requires_grad_(True))
```

Fixed by https://github.com/truera/trulens/pull/58 .
Repository: truera/trulens
Issue: upgrade to typing-extensions 4.8.0
Root cause: Configuration Error

Using fastapi later versions rely on 4.8.0, just one example, many other install inconvenience could be avoided.

Could you please use the later versions of typing-extensions?


The version of `typing-extension` was unpinned recently in our package requirements so you should be able to use 4.8.0 with the next release of trulens_eval.

@piotrm0 thank you that is great news. can you estimate when next release would be?

Hey @kocsszi - we will release either today or early next week.

If you don't want to wait, you can also clone and install from the repository:

`cd trulens/trulens_eval`
`pip install -e .`

Cheers!
Repository: truera/trulens
Issue: Random gradients in torch summarizer pipeline
Root cause: Resource Mishandling

This code has different gradients each time

```python
from trulens.nn.attribution import InputAttribution, InternalInfluence
from trulens.nn.attribution import IntegratedGradients
from trulens.nn.quantities import MaxClassQoI
from trulens.nn.distributions import PointDoi, LinearDoi
from trulens.nn.slices import Cut, InputCut, OutputCut, Slice
from trulens.nn.models import get_model_wrapper

from transformers import pipeline

summarizer = pipeline("summarization")

summarizer(
    """
    America has changed dramatically during recent years. Not only has the number of 
    graduates in traditional engineering disciplines such as mechanical, civil, 
    electrical, chemical, and aeronautical engineering declined, but in most of 
    the premier American universities engineering curricula now concentrate on 
    and encourage largely the study of engineering science. As a result, there 
    are declining offerings in engineering subjects dealing with infrastructure, 
    the environment, and related issues, and greater concentration on high 
    technology subjects, largely supporting increasingly complex scientific 
    developments. While the latter is important, it should not be at the expense 
    of more traditional engineering.

    Rapidly developing economies such as China and India, as well as other 
    industrial countries in Europe and Asia, continue to encourage and advance 
    the teaching of engineering. Both China and India, respectively, graduate 
    six and eight times as many traditional engineers as does the United States. 
    Other industrial countries at minimum maintain their output, while America 
    suffers an increasingly serious decline in the number of engineering graduates 
    and a lack of well-educated engineers.
"""
)

m = summarizer.model.to("cpu")
tokenizer = summarizer.tokenizer

batch_sentences = ["Hello I'm a single sentence", "And another sentence", "And the very very last one"]
batch = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors="pt").to(m.device)
input = batch['input_ids']

wrapped_model = get_model_wrapper(m, input_shape=(None, 7), device='cpu')

embed_cut = Cut('model_encoder_layers_0_self_attn_q_proj', anchor='in')
from trulens.nn.quantities import QoI
import torch
class SummarizerQoI(QoI):
    def __call__(self, seq_2_seq_output):
        logits = seq_2_seq_output['logits']
        
        max_token, max_indices = torch.max(logits,dim=-1)
        qoi = torch.mean(max_token, 1, True)
        return qoi

infl = InternalInfluence(
    wrapped_model, 
    Slice(embed_cut, OutputCut()), 
    SummarizerQoI(),
    PointDoi(cut=embed_cut))

attrs = infl.attributions(**batch)
```
Repository: truera/trulens
Issue: AzureOpenAI feedback provider: 'deployment_id' error
Root cause: Interface Contract Mismatch

Hi all!

I'm using a LLM from Azure OpenAI in order to implement a basic groundedness+question/statement+question/answer relevance use case ([this one](https://github.com/truera/trulens/blob/a9717080c3218a51dfc299bb5e8f8c552e6987ab/trulens_eval/examples/expositional/models/azure_openai.ipynb#L178)), and I'm getting an error which I believe is related to the way the **AzureOpenAI class** is implemented in **trulens_eval.feedback.provider.openai**.

The error message is:

> openai request failed <class 'TypeError'>=Completions.create() got an unexpected keyword argument 'deployment_id'. Retries remaining=3.

(and further retries)

Which makes sense, since this is how I'm creating the LLM provider:
`from trulens_eval.feedback.provider.openai import AzureOpenAI`
`openai_provider = feedback.AzureOpenAI(
    deployment_id="<azure deployment name>",
    model_engine="gpt-35-turbo"
)`

I can't take the _deployment_id_ argument out, since it is enforced [by the definition of the AzureOpenAI class](https://github.com/truera/trulens/blob/trulens-eval-0.18.0/trulens_eval/trulens_eval/feedback/provider/openai.py#L380).

But, AzureOpenAI [is a subclass of OpenAI](https://github.com/truera/trulens/blob/trulens-eval-0.18.0/trulens_eval/trulens_eval/feedback/provider/openai.py#L380), which uses OpenAI's _client.chat.completions.create_ function, which doesn't expect any _deployment_id_ argument ([it seems like the _model_ argument is used to specifiy Azure's deployment_name](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/chatgpt?tabs=python-new&pivots=programming-language-chat-completions)).

What do you guys/gals think? Am I doing something wrong? Or this is a bug in trulens' code?

Thanks for reporting @FernandoBenayas ! Taking a look

I have the same error

same error

Same here. Have you managed to fix somehow?

@daniel-huang-1230 has a fix that will be released today. thanks for the patience all!

https://github.com/truera/trulens/pull/640

Thanks for the contribution from [schmidtseb](https://github.com/schmidtseb)! 

It works as expected. Thanks guys ;)