zhiminy/EvalEng huggingface dataset:
Repository: truera/trulens
Issue: Metadata Error
Root cause: Validation Gap

I encountered the following error in trulens_eval 0.22.0 for records with metadata:

![image](https://github.com/truera/trulens/assets/38088099/a66ca4af-f638-4dfc-aa20-7f55397e7666)


@joshreini1 - Fixed the bug in PR https://github.com/truera/trulens/pull/888 

Closing - the fix from @aaronvarghese is merged into main and will be available in next release
Repository: truera/trulens
Issue: The code in bedrock.py is causing an error.
Root cause: Validation Gap

In bedrock.py, we have the below code. Although I have boto3 and botocore installed, it is complaining that those are not installed. I think this is because the assert error is checking for the installation outside the virtual environment. If I directly do the imports, it works

Code which is causing an error
----------------------------------
```
with OptionalImports(messages=REQUIREMENT_BEDROCK):
    # Here only to make sure we throw our message if bedrock optional packages
    # are not installed.
    import boto3

OptionalImports(messages=REQUIREMENT_BEDROCK).assert_installed(boto3)
```

Hey @rajib76 - I'm unable to reproduce this error. Is there something specific to your virtual environment that could be causing this issue?

Actually, nothing specific. If I just push the imports outside of the context manager and do not make it optional, it works. Is there a reason for having them as optional imports

There is - you can read more about why we're using optional imports here: https://github.com/truera/trulens/blob/main/trulens_eval/OPTIONAL.md

Can you share the code that you're using that encounters this error?

this is the code i am trying to run. this was running fine earlier. so, some recent changes broke it 

https://github.com/rajib76/true_lens_examples/blob/main/01_rag_triad_relevance.py

and this is my custom claude provider

https://github.com/rajib76/true_lens_examples/blob/main/providers/bedrock_claude_provider.py

Hi @rajib76 . I might have a solution. Can you take a look at https://github.com/truera/trulens/pull/892 and let us know whether that fixes your issue.

Hey @rajib76 - this fix is released now in https://github.com/truera/trulens/releases/tag/trulens-eval-0.22.2
Repository: truera/trulens
Issue: BUG : amazon.titan-text-express-v1 is not yet implemented as a feedback provider
Root cause: Unimplemented Feature Gap

Hello,

I have below simple code which failed with error `bedrock request failed <class 'NotImplementedError'>=The model selected, amazon.titan-text-express-v1, is not yet implemented as a feedback provider`.

As you can see in commented code, I tried anthropic.claude-v2 and other Bedrock models by cross referencing model_id from this document : https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html

But it fails for all models. I have Bedrock model enabled and it works when I independently invoke them as provided in screenshot. 

```
# Initialize provider class
tru = Tru()
tru.reset_database()

#ftitan = fBedrock(model_id = "anthropic.claude-v2")
ftitan = fBedrock()

# Define a groundedness feedback function
f_comprehensiveness = (
    Feedback(ftitan.comprehensiveness_with_cot_reasons, name = "Comprehensiveness")
    .on_input_output()
)

tru_recorder = TruChain(
    llm_chain_summary,
    app_id='Exp_1_Claim_Summary',
    feedbacks=[f_comprehensiveness]
)

with tru_recorder as recording:
    completion = llm_chain_summary({"context": notes}, return_only_outputs=True)
```
Error Screenshot: The multiple error tries comes from my multiple attemps by changing model_id. 

<img width="1239" alt="image" src="https://github.com/truera/trulens/assets/43816307/bcd48abb-40cf-43c2-ac7e-aea049f4cd92">


Working Code using Bedrock through Langchain:

<img width="1239" alt="image" src="https://github.com/truera/trulens/assets/43816307/0099ac4c-c760-453d-a02b-9c80ecf84acc">


Thanks @SanjuMLGeek - fixed in this #887. Can you install from that branch and give it a try?

The only model available in Bedrock not yet implemented in TruLens should be Llama-2

Hey @SanjuMLGeek - you're just missing the subdirectory. Try:

`pip install git+https://github.com/truera/trulens@josh/bedrock_fix#subdirectory=trulens_eval`

Hey @SanjuMLGeek - this change is released now in 0.22.2:

https://github.com/truera/trulens/releases/tag/trulens-eval-0.22.2

@joshreini1 - It solved the problem. Thank you.
Repository: truera/trulens
Issue: Notebook not found when using the open in Colab link in the moderation 
Root cause: Documentation Deficiency

When click on the "Open in Colab" link in the moderation example:

https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/expositional/use_cases/moderation.ipynb

I get a popup dialog that says "Notebook not found":

![image](https://github.com/truera/trulens/assets/4660487/b9a8b8c9-83ea-4e7a-bf30-e5e2129850de)

It then asked me to authorize via GitHub, which I did.  I then tried the link again but I still got that error.

HI I would like to assist you in this 


Correct link is: https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/expositional/use_cases/moderation.ipynb

Will update this in the notebook.

Thanks
Repository: truera/trulens
Issue: Migrate Llama-Index Optional Dependency to 0.10
Root cause: External Dependency Breakage

Solution proposed by Logan is to use a bridge file for imports, similar to what they did with [pydantic](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/bridge/pydantic.py), to maintain compatibility across llama-index versions

In the meantime, we've pinned the llama-index optional requirement to <0.10.

Note, that this is only an issue if your app uses llama-index >= 0.10. Llama-index is not a required TruLens dependency

This is released in 0.24.0!
Repository: truera/trulens
Issue: Tru() fails with `database_url` key
Root cause: Configuration Error

Getting the following error:
```
TypeError: Invalid argument(s) 'max_overflow','pool_use_lifo' sent to create_engine(), using configuration SQLiteDialect_pysqlite/SingletonThreadPool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.%
```
when trying to initialize Tru() with a database_url:
```
tru = Tru(
    # By default, Trulens leaves a `default.sqlite` file behind. Use an in-memory
    # database for unit tests so that we don't need to worry about cleaning it up.
    database_url="sqlite:///:memory:",
    database_redact_keys=True,
)
tru.reset_database()
```

trulens-eval version 0.22.0

Possible fix incoming here: https://github.com/truera/trulens/pull/904 .

Fix is in the next release.
Repository: truera/trulens
Issue: RuntimeError: cannot reuse already awaited coroutine
Root cause: Resource Mishandling

Not sure if related to llama-index v0.10.x, or not, but I've seen this issue reported a few times in our discord
https://discord.com/channels/1059199217496772688/1059200010622873741/1209156084036538388

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[20], line 2
      1 # or as context manager
----> 2 with tru_query_engine_recorder as recording:
      3     query_engine.query("What did the author do growing up?")

File c:\Users\TurnerZ\Documents\GitHub\amaliai-hr\.venv\Lib\site-packages\trulens_eval\app.py:842, in App.__exit__(self, exc_type, exc_value, exc_tb)
    839 self.recording_contexts.reset(ctx.token)
    841 if exc_type is not None:
--> 842     raise exc_value
    844 return

Cell In[20], line 3
      1 # or as context manager
      2 with tru_query_engine_recorder as recording:
----> 3     query_engine.query("What did the author do growing up?")

File c:\Users\TurnerZ\Documents\GitHub\amaliai-hr\.venv\Lib\site-packages\trulens_eval\instruments.py:633, in Instrument.tracked_method_wrapper.<locals>.tru_wrapper(*args, **kwargs)
    625 @functools.wraps(func)
    626 def tru_wrapper(*args, **kwargs):
    627     logger.debug(
    628         f"{query}: calling instrumented sync method {func} of type {type(func)}, "
    629         f"iscoroutinefunction={is_really_coroutinefunction(func)}, "
    630         f"isasyncgeneratorfunction={inspect.isasyncgenfunction(func)}"
...
--> 267         result = coro.send(None)
    268     else:
    269         result = coro.throw(exc)

RuntimeError: cannot reuse already awaited coroutine
```

This is from running the quickstart
https://www.trulens.org/trulens_eval/llama_index_quickstart/

To confirm, is this with llama_index >= 0.10.0 ? We currently do not support llama_index >= 0.10 . @joshreini1 

> To confirm, is this with llama_index >= 0.10.0 ? We currently do not support llama_index >= 0.10 . @joshreini1

I experience the same problem with trulens-eval 0.22.2 version and llama-index 0.9.17
UPD: it's interesting, that i don't have this problem when I create new index and not load it from `docstore.json` (chromadb)

Hey @julfr - this is an issue where the true error is being obstructed by this message about the coroutine. @piotrm0 has a fix that will be released today to show the true error.

@julfr Can you check if this is resolved?
Repository: truera/trulens
Issue: Found more than one `BaseRetriever` in app
Root cause: Documentation Deficiency

When i tried to select context from retrivalQA chain in:
    from trulens_eval.app import App
    context = App.select_context(query_engine)
 
i faced this issue 
ValueError: Found more than one `BaseRetriever` in app:
        <class 'langchain.retrievers.ensemble.EnsembleRetriever'> at retriever
        <class 'langchain_core.vectorstores.VectorStoreRetriever'> at retriever.retrievers[0]
       
Is there any alternative as i need the context to be added for the context relevance and groudnedness

Hey @AraHema - good question! Definitely an alternative. The `select_context` is intended as a shortcut for simple RAGs. For your case, you can use our `Select` method to gain more control on choosing the context to evaluate.

Here’s an example specifically for selecting the context fetched by Langchain ensemble retrievers https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/frameworks/langchain/langchain_ensemble_retriever.ipynb

i faced this issue raise RuntimeError(
RuntimeError: Could not locate app._get_relevant_documents.rets[:].page_content in app/record.


I used this Select.Record.app.combine_documents_chain._call.args.inputs.input_documents[:].page_content
but the scoring is not good as i expected for context relevance.

Do you have any suggestions?

Hey @AraHema - when you inspect the feedback display in the TruLens UI, are you seeing the expected documents being evaluated?

<!-- Answer -->
Hi, @AraHema,

I'm helping the trulens team manage their backlog and am marking this issue as stale. From what I understand, you encountered a ValueError when attempting to select context from a retrievalQA chain due to finding more than one `BaseRetriever` in the app. Joshreini1 provided an alternative solution using the `Select` method to gain more control over choosing the context to evaluate. However, you mentioned that the suggested solution did not yield the expected scoring for context relevance and sought further suggestions.

Could you please confirm if this issue is still relevant to the latest version of the trulens repository? If it is, please let the trulens team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you!




@AraHema please let us know if you still need help with this, closing this for now.
Repository: truera/trulens
Issue: Could not locate app.get_relevant_documents.rets[:].page_content in app/record.
Root cause: Validation Gap

during the use of this method https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/frameworks/langchain/langchain_ensemble_retriever.ipynb

i faced this issue 

KeyError: 'Key not in dictionary: get_relevant_documents'
RuntimeError: Could not locate app.get_relevant_documents.rets[:].page_content in app/record.

Can you share the full stack trace on where you are experiencing this issue @AraHema ?

@AraHema Can you please let us know if you still need help?

<!-- Answer -->
Hi, @AraHema

I'm helping the trulens team manage their backlog and am marking this issue as stale. It looks like you reported a KeyError and RuntimeError in the langchain_ensemble_retriever.ipynb file, specifically related to the 'get_relevant_documents' key not being found in the dictionary. Joshreini1 requested the full stack trace for further investigation, and yuvneshtruera asked if you still needed assistance.

Could you please confirm if this issue is still relevant to the latest version of the trulens repository? If it is, please let the trulens team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you!
Repository: truera/trulens
Issue: Evaluation metrics inference
GitHub labels: question

Hi,

I am using truelens to evaluate the performance of one of our chat applications. I need help in understanding the metrics being used. I am not able to understand how to interpret these metrics. For eg: The answer for my query is actually incorrect, the context relevance and groundedness parameter scores depict this to some extent but I get answer relevance as always 1. 

Can some one help me understand how to interpret these metrics or provide reference to content which explain these at a deeper level? 

Thanks in advance. 

Hi @FatemaD1577 !

We have material in our docs:
[Core Concept: The RAG Triad](https://www.trulens.org/trulens_eval/core_concepts_rag_triad/)

And you may also enjoy a short course on this topic we put together with Andrew Ng and Llama-Index:
[Building and Evaluating Advanced RAG](https://www.deeplearning.ai/short-courses/building-evaluating-advanced-rag/)

Thank you!! @joshreini1 for the prompt reply. Will check the course.

You're welcome!
Repository: understandable-machine-intelligence-lab/Quantus
Issue: New data domain: Quantus + NLP = True
Root cause: Unimplemented Feature Gap

**Goal**: make Quantus source code compatible with NLP tasks, including the addition of a tutorial to showcase how Quantus can help quantify the goodness of NLP explanation

- [ ] Determine toy dataset:
      - Suggestively use the following library https://pytorch.org/text/ -- with a tutorial: https://pytorch.org/text/stable/tutorials/sst2_classification_non_distributed.html and off-the-shelf model
- [ ] Go over Quantus source code to determine the scope of this PR


Updates are expected:
- source code
- tutorial
- pytests


Hi, 
so, here [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1yQ-RwL1rHmrOtnRbuNYT_pCw9jdV7DXV?usp=sharing) is a notebook demonstrating how NLP explanation evaluation with Quantus could look like in (near) future.

In the first iteration, the goal is to support models from Hugging face Hub with scope limited to text-classification tasks.

Hi @aaarrti , are there any updates about this? 

I presented myself in this thread #217, please let me know if I can help
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Loosen restrictions on number of steps for Pixel-Flipping
Root cause: Documentation Deficiency

Pixel-Flipping has a check called `assert_max_steps` related to the maximum number of steps passed as an argument in `max_steps_per_input`. This check seems too restrictive and I am not sure about its exact purpose or meaning; perhaps it's actually meant for Region Perturbation.

Instead, a natural check for the `max_steps_per_input` parameter would be an **upper bound limit** regarding  the **total number of pixels** in the input to avoid flipping the same pixels more than once.

https://github.com/understandable-machine-intelligence-lab/Quantus/blob/1ecb500195f985f7b7a6a7086cba82219d76c6c6/quantus/metrics/faithfulness_metrics.py#L1369-L1372

https://github.com/understandable-machine-intelligence-lab/Quantus/blob/1ecb500195f985f7b7a6a7086cba82219d76c6c6/quantus/helpers/asserts.py#L76-L81

Hmm, I think there is confusion here with what is the intended purpose of `max_steps_per_input`.

I originally implemented it as an alternative to specifying `features_in_step` to allow the user to automatically infer the number of `features_in_step` by splitting the input size by the maximum steps per input. I see that it is not well-documented in the library, so to avoid future confusion, I will remove this in future releases of the library. 

Fixed in PR: https://github.com/understandable-machine-intelligence-lab/Quantus/pull/114
Repository: understandable-machine-intelligence-lab/Quantus
Issue: taking absolute before normalisation makes normalise_by_negative defective
Root cause: Algorithmic Error

I'm not sure if this is intended, but all metrics first take the absolute of the attributions, then apply normalisation.

But if we want to normalise the positive and the negative sides differently like it's done in `normalise_by_negative` it would be more reasonable to normalise first, and then take the absolute.

Am I missing something, or is this a bug?

Agree, had planned to fix this before. It will be fixed in upcoming commits @dkrako 

Fixed in PR: https://github.com/understandable-machine-intelligence-lab/Quantus/pull/114
Repository: understandable-machine-intelligence-lab/Quantus
Issue: New metrics: implement Consistency and Sufficiency
Root cause: Unimplemented Feature Gap

Implement Consistency (that belongs to the Robustness category) and Sufficiency (that belongs to the Sufficiency category):

• **Consistency**: roughly, two instances x, x0 that get the same explanation should also have the same prediction. For instance, if two different images are assigned the same explanation, e(x) = e(x 0 ) = “contains a zebra”, then their assigned labels should also be the same. 
• **Sufficiency**: if x is assigned an explanation e(x) = π that also holds for another instance x 0 (even if e(x 0 ) 6= π), then x 0 should have the same label as x.

Paper: https://arxiv.org/pdf/2202.00734.pdf

@dilyabareeva 

To clarify changes excepted in the PR, the should be reflected in the following folders:
- [ ] `quantus/metrics/faithfulness_metrics.py` and `quantus/metrics/robustness_metrics.py`
- [ ] `tests/metrics/test_faithfulness_metrics.py` and `quantus/metrics/test_robustness_metrics.py`
- [ ] `tests/metrics/test_faithfulness_metrics.py` and `quantus/metrics/test_robustness_metrics.py`
- [ ] and in the tutorial: https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/tutorials/tutorial_basic_example_all_metrics.ipynb
- [ ] `tests/helpers/constants.py``
- [ ] `README.md`

Completed. https://github.com/understandable-machine-intelligence-lab/Quantus/issues/112
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Evaluating transformer explainability
GitHub labels: question

Hello @sebastian-lapuschkin ,

Thank you for the great work, 

Please suggest, is there any sample reference example, for using this framework with transformer explainability like this one
https://github.com/hila-chefer/Transformer-Explainability
& this one https://github.com/berniwal/swin-transformer-pytorch

Best Regards,
@jaiswati

@jaiswati Thanks for your feedback. Since the XAI method implementations you are referring to are not built into Quantus library, the best approach would be to pass your custom explanation function to metrics. Please refer to [README/Getting Started/Step 2. Load explanations/Employ customised functions](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/README.md#getting-started) for more detail.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: New metric: implement Relative Stability
Root cause: Unimplemented Feature Gap

Implement Relative Stability (belongs to the Robustness category).

• **Relative Stability**: leverages model information to evaluate the stability of an explanation with respect to the change in the a) input data, b) intermediate representations and c) output logits of the underlying prediction model.

Paper: https://arxiv.org/pdf/2203.06877.pdf

To clarify changes excepted in the PR, the should be reflected in the following folders:

- [ ] quantus/metrics/faithfulness_metrics.py and quantus/metrics/robustness_metrics.py
- [ ] tests/metrics/test_faithfulness_metrics.py and quantus/metrics/test_robustness_metrics.py
- [ ] tests/metrics/test_faithfulness_metrics.py and quantus/metrics/test_robustness_metrics.py
 and in the tutorial: https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/tutorials/tutorial_basic_example_all_metrics.ipynb
- [ ] `tests/helpers/constants.py``
- [ ] README.md

Hi @annahedstroem, I'd like to work on this feature
Repository: understandable-machine-intelligence-lab/Quantus
Issue: [Bug] NameError: name 'tf' is not defined
Root cause: Validation Gap

For `quantus==0.1.4`, 
the following example code snippet comes from `quantus.metrics.localisation_metrics.PointingGame`.

```python
import torch
from quantus.helpers.models import LeNet
import torchvision
from captum.attr import Saliency
from quantus import PointingGame

# Enable GPU.
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Load a pre-trained LeNet classification model (architecture at quantus/helpers/models).
model = LeNet()
model.load_state_dict(torch.load("tutorials/assets/mnist"))

# Load MNIST datasets and make loaders.
test_set = torchvision.datasets.MNIST(root='./sample_data', download=True)
test_loader = torch.utils.data.DataLoader(test_set, batch_size=24)

# Load a batch of inputs and outputs to use for XAI evaluation.
x_batch, y_batch = iter(test_loader).next()
x_batch, y_batch = x_batch.cpu().numpy(), y_batch.cpu().numpy()

# Generate Saliency attributions of the test set batch of the test set.
a_batch_saliency = Saliency(model).attribute(inputs=x_batch, target=y_batch, abs=True).sum(axis=1)
a_batch_saliency = a_batch_saliency.cpu().numpy()

# Initialise the metric and evaluate explanations by calling the metric instance.
metric = PointingGame(abs=True, normalise=False)
scores = metric(model=model, x_batch=x_batch, y_batch=y_batch, a_batch=a_batch_saliency, **{})
```

It produces the following error:

``` 
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_19546/1671044940.py in <module>
      1 import torch
----> 2 from quantus.helpers.models import LeNet
      3 import torchvision
      4 from captum.attr import Saliency
      5 from quantus import PointingGame

~/.pyenv/versions/3.8.5/envs/env/lib/python3.8/site-packages/quantus/__init__.py in <module>
----> 1 from .helpers import *
      2 from .metrics import *
      3 from .evaluation import *

~/.pyenv/versions/3.8.5/envs/env/lib/python3.8/site-packages/quantus/helpers/__init__.py in <module>
     18     from .models import *
     19 if __EXTRAS__:
---> 20     from .explanation_func import *

~/.pyenv/versions/3.8.5/envs/env/lib/python3.8/site-packages/quantus/helpers/explanation_func.py in <module>
     95 
     96 def generate_tf_explanation(
---> 97     model: tf.keras.Model, inputs: np.array, targets: np.array, **kwargs
     98 ) -> np.ndarray:
     99     """

NameError: name 'tf' is not defined
```

A similar error occurs in `quantus/helpers/utils.py:L200`, and seems to be related to type checking. I'm not sure if the best is to continue wrapping function declarations with `if util.find_spec("tensorflow"):`, or choose a different strategy.

Thanks so much for highlighting this bug @vedal.

It is fixed in this PR: https://github.com/understandable-machine-intelligence-lab/Quantus/pull/120