zhiminy/EvalEng huggingface dataset:
Repository: truera/trulens
Issue: Add docs for running evals on pre-logged data
Root cause: Documentation Deficiency


+1 Could you add docs for already existing OpenAI results evaluation instead of retrieving through LlamaIndex?

> +1 Could you add docs for already existing OpenAI results evaluation instead of retrieving through LlamaIndex?

Can you clarify further on what you mean by "already existing results evaluation"? @Niharika6442 

So I have few answers generated by Azure openAI along with the context and questions. I'd like to evaluate them directly without generating answers again using llamaindex as shown in quickstarts.

Coming soon!

Done! See: https://www.trulens.org/trulens_eval/feedback_functions_existing_data/

Where is the retriever_component defined for below line?
context_call = retriever_component.get_context

Also, Can you provide an example notebook as a quickstart?

> Where is the retriever_component defined for below line? context_call = retriever_component.get_context
> 
> Also, Can you provide an example notebook as a quickstart?

@joshreini1, could you please explain "context_call = retriever_component.get_context". In cases where already a dataframe of questions, answers and retrieved contexts are available, do we still need to pass the selector for a presumed context retrieval component's call?

@joshreini1 anything on this?

@prasad4fun the context_call definition is used to record the structure of your application. If you need to include a context as an intermediate step of your app - I'd recommend setting the context selector here and record the `context_call` `args` and `rets` like shown in the docs.

There's also a notebook that goes a bit deeper on these things here: [Virtual Example](https://github.com/truera/trulens/blob/main/trulens_eval/examples/experimental/virtual_example.ipynb)

Will put together more of a "quickstart" on this at some point soon

https://www.trulens.org/trulens_eval/evaluation/running_feedback_functions/existing_data/#:~:text=In%20many%20cases%2C%20developers%20have,data%2C%20independent%20of%20the%20recorder%20.
Repository: truera/trulens
Issue: Contradicting results: Getting NaN in the notebook but actual value shows in the Streamlit app
Root cause: Validation Gap

I am following the deeplearning.ai course in [Building and Evaluating Advanced RAG - part 3, Rag Triad of metrics](https://learn.deeplearning.ai/building-evaluating-advanced-rag/lesson/3/rag-triad-of-metrics).

When I execute

```
import pandas as pd

pd.set_option("display.max_colwidth", None)
records[["input", "output"] + feedback]
```

I get the below with alot of NaN values:


![Screenshot 2024-01-20 at 21 45 41](https://github.com/truera/trulens/assets/63347261/20cd69fa-e94a-4a0c-b819-9d4ff45204d2)


When I get the equivalent question in the Streamlit app, it actually has the values.

<img width="947" alt="Screenshot 2024-01-20 at 21 48 49" src="https://github.com/truera/trulens/assets/63347261/a5e321a3-e823-486c-ba68-4b358d4c16b1">



Seems likely that the feedback scores were not yet available when you created the records df. Please rerun, expect you will see the same values as in streamlit
Repository: truera/trulens
Issue: Groundedness has incorrect values
Root cause: Validation Gap

In the Groundedness we see =0.5, but actualy results are 0.5 and 0.0, shouldnt it be 0.25?
If we look at the Context relevance it took the average instead of the highest

<img width="989" alt="Screenshot 2024-01-20 at 21 55 47" src="https://github.com/truera/trulens/assets/63347261/2d621cb9-6d5c-4e32-b9e9-a629e437f18b">


In another cases, the Groundedness seems to have a random value


<img width="1041" alt="Screenshot 2024-01-20 at 21 56 54" src="https://github.com/truera/trulens/assets/63347261/3ab348a3-c3e7-480c-8d85-30becf5aa790">


Thank you for bringing this to our attention, we are looking at this.

@magedhelmy1 - can you please share how you defined the groundedness feedback functions?

Here it is, 


```
import os
from dotenv import load_dotenv, find_dotenv

import numpy as np

import nest_asyncio

nest_asyncio.apply()


def get_openai_api_key():
    _ = load_dotenv(find_dotenv())

    return os.getenv("OPENAI_API_KEY")

from trulens_eval import (
    Feedback,
    TruLlama,
    OpenAI
)

from trulens_eval.feedback import Groundedness

def get_prebuilt_trulens_recorder(query_engine, app_id):
    openai = OpenAI()

    qa_relevance = (
        Feedback(openai.relevance_with_cot_reasons, name="Answer Relevance")
        .on_input_output()
    )

    qs_relevance = (
        Feedback(openai.relevance_with_cot_reasons, name = "Context Relevance")
        .on_input()
        .on(TruLlama.select_source_nodes().node.text)
        .aggregate(np.mean)
    )

#     grounded = Groundedness(groundedness_provider=openai, summarize_provider=openai)
    grounded = Groundedness(groundedness_provider=openai)

    groundedness = (
        Feedback(grounded.groundedness_measure_with_cot_reasons, name="Groundedness")
            .on(TruLlama.select_source_nodes().node.text)
            .on_output()
            .aggregate(grounded.grounded_statements_aggregator)
    )

    feedbacks = [qa_relevance, qs_relevance, groundedness]
    tru_recorder = TruLlama(
        query_engine,
        app_id=app_id,
        feedbacks=feedbacks
    )
    return tru_recorder
```

And please let me know if you need more info

@magedhelmy1 please adjust your groundedness definition to:

```
groundedness = (
        Feedback(grounded.groundedness_measure_with_cot_reasons, name="Groundedness")
            .on(TruLlama.select_source_nodes().node.text.collect())
            .on_output()
            .aggregate(grounded.grounded_statements_aggregator)
    )
```

Note the addition of `.collect()`
Repository: truera/trulens
Issue: Summary with CoT Feedback function doesn't capture cost for Bedrock models and the reason for the score given
Root cause: Unimplemented Feature Gap

Hello,

I have implemented Summary of CoT feedback as below. The Dashboard shows it didn't capture the cost for Bedrock based model as well as the record level activity shows no "reason" for the score. In my case one record shows the score as "1" but I don't feel the summary covered all facts while other record shows "0.5" and I can't differentiate the reasoning.

Appreciate your help and attention.

`from trulens_eval import Bedrock as TBedrock
from trulens_eval.feedback.provider.base import Provider, LLMProvider
from trulens_eval.feedback.provider.bedrock import Bedrock as fBedrock
from trulens_eval.tru_custom_app import instrument

import numpy as np

tru = Tru()
ftitan = fBedrock()

f_summary_with_cot = (
    Feedback(ftitan.summary_with_cot_reasons, name = "SummaryCoT")
    .on_input_output()
)

llm = Bedrock(model_id="anthropic.claude-instant-v1")

PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context"]
)

chain = LLMChain(
    llm=llm,
    prompt=PROMPT,
    verbose=False
)
    
tru_recorder = tru.Chain(
    chain,
    app_id='Claim_Summary_Prompt2',
    feedbacks=[f_summary_with_cot]
)`


Here is the screenshot:

<img width="1324" alt="image" src="https://github.com/truera/trulens/assets/43816307/92782f48-da68-469d-8be0-8ef9d52ed93e">

<img width="785" alt="image" src="https://github.com/truera/trulens/assets/43816307/2804b4ae-559f-4dfe-9e54-342da26aac96">

<img width="785" alt="image" src="https://github.com/truera/trulens/assets/43816307/9262813a-7fe8-4507-85d5-57aa35d111c8">







Thanks @SanjuMLGeek - looking into this

Hey @SanjuMLGeek ! We made big improvements to the summarization feedback (now named comprehensiveness) and these changes are now released in 0.23.0. Please give it a shot!
Repository: truera/trulens
Issue: Typo in moderation_sexualminors in openai provider causing it to crash when used
Root cause: Validation Gap

In `moderation_sexualminors` method in openai.py, there's a typo that's not caught by any tests that causes a failure when called.

```python
        openai_response = self._moderation(text)

        return float(oopenai_response.category_scores.sexual_minors)
```
should be:
```python
        openai_response = self._moderation(text)

        return float(openai_response.category_scores.sexual_minors)
```
Notice the extra `o` in `oopenai_response` in the existing code.

A linter in the CI would have caught this.
Repository: truera/trulens
Issue: Groundedness metric surpasses the defined limit
Root cause: Algorithmic Error

I implemented the use case from this notebook of trulens: [Colab Link](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/quickstart/quickstart.ipynb), but for a list of 500 consecutive queries, the computed Groundedness is 7.259001e+09. Conversely, when using the exact same code for a list of 100 queries, the Groundedness falls within the expected limits (0 to 1). What could be the potential issue in the case of 500 queries?

Hi @kouskouss - it's likely the issue here is the LLM performing the evaluations. We've done some refactoring for groundedness that may help with this.

Can you install from github and retry?

Please let me know if this issue persists.

With new version from github it throws validation error and final groundedness is calculated properly!
But with this version total_tokens is not calculated correctly, it always returns 0

Hi @joshreini1, I tried your suggestion and now it works! Thank you.

Great news!

@Bartosz-Lewandowski - if you are seeing issues with the token tracking, please open a new issue including the provider/model you are using. Thanks
Repository: truera/trulens
Issue: Using a Open-Source LLM as Provider
Root cause: Documentation Deficiency

I'm trying to use an open-source LLM (models available on huggingface) as a Provider, locally running, instead to use OpenAI provider. Is it possible?

Yes, for example: https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/ollama_quickstart.ipynb
Repository: truera/trulens
Issue: Errors due do not exctly matching strings in LLMprovider class
Root cause: Validation Gap

In line 157 of the base.py file in the providerst folder (https://github.com/truera/trulens/blob/9eb94ad34bf8b1f27a715fa898336c171a51e54b/trulens_eval/trulens_eval/feedback/provider/base.py#L157) the line.index command does not use the exact same string as the if statement in L155/L144 (One has a ":", one doesnt). I try to use a local huggingface llm pipeline with llama_index and when trying to apply a feedbackfunction ("relevance_with_cot_reasoning") it crashes at this point with the error "ValueError: substring not found". 
Not sure if thats the only cause for that error but I think using the same string for the index command and the if statement will avoid possible unwanted behavior. 

@munila - thanks! This was resolved in trulens_eval==0.23.0
Repository: truera/trulens
Issue: import Tru requires openai
Root cause: Validation Gap

after pip install trulens-eval, import trulens-eval errors out with the following error:

![image](https://github.com/truera/trulens/assets/80770057/a11f51a5-0ee9-49ce-95d3-c1473c783624)

intalled version is 0.20.3

Reverting back to 0.18.0 resolves the issue

Taking a look
Repository: truera/trulens
Issue: TruLens with custom hosted model on SageMaker

Is there a way that we can use custom hosted model on Sagemaker to calculate all the evaluation scores?

This is not an issue, this is for information.

Thanks for the question @abdulhaseeb-nbs !

You can create a custom feedback function that uses any custom model. See related docs: https://www.trulens.org/trulens_eval/custom_feedback_functions/

However you would need to recreate the prompting for out-of-the-box evals. We don't currently have a way to run an arbitrary/custom provider with existing feedback implementations
Repository: truera/trulens
Issue: Facing Issue: Using an Custom OpenAI Gateway with trulens
Root cause: Documentation Deficiency

So in trulens we are having and OpenAI Object , which requires the api-key and stuff of Open AI.
from trulens_eval.feedback.provider.openai import OpenAI

But , if I want to send the request to my custom AI_Gateway, which is using a seperate AI model. How to use that with truelens?

Hey @krishmurarka 

Can you do this with our langchain provider? https://www.trulens.org/trulens_eval/api/langchain_provider/

Basically you just need to set up your AI_Gateway as a chain, and then pass it to the TruLens LangChain provider

Hey @joshreini1  the link you provided doesn't exist 
do you have any updated link ??

Also If I am using my custom model how the cost and token calculation would work ?? 

because for now I did monkey patching to make my model run through , but I am not getting cost and token

Hey @krish-murarka - here's the updated link: https://www.trulens.org/trulens_eval/api/provider/langchain/

We also made some fixes to our cost tracking in trulens-eval==0.24.1; would recommend upgrading if you're still on an older version

Feel free to share along your monkey-patching as well :)

@joshreini1 Thank you, ohh we were using 0.21 version for trulens-eval let me bump the package.


And For Monkey Patching, we were using the llamaIndex Open AI object , but as it has  pydantic checks of model 


For we monkey patched this function of Open AI in llama: def _get_model_kwargs(self, **kwargs: Any) -> Dict[str, Any]:

OpenAI._get_model_kwargs = Custom_model_kwargs_custom
Repository: truera/trulens
Issue: Additions to the Tru API for working with large databases
Root cause: Unimplemented Feature Gap

I've been working primarily with very large databases and I find the [Tru API](https://www.trulens.org/trulens_eval/api/tru/) to be a bit lacking. 

In particular, methods like the following would be very helpful:

- [ ] * `tru.list_records(app_id)`: Get a list of record_ids in an app.
- [ ] * `tru.get_record_and_feedback(record_id)`: Get a single record and its feedback results from the database.
- [ ] * `tru.delete_record(record_id)`: Delete a single record from the database. (Useful to remove duplicate records from an App without having to reset the database and start over)
- [x] * `tru.delete_app(app_id)`: Delete an app from the database. (without having to reset the database and start over) (Helpful when an App run didn't complete due to an exception) - done in #1061 by @arn-tru 

These methods would be helpful, but not as high priority:

- [ ] * `tru.get_incomplete_records(app_id, [feedback_function_names])`: Returns a list of record_ids where the passed feedback function names don't yet have a value. Basically a way to test if the app has a full set of data yet, and if not, return the records that need to be finished.

Finally, some paging functionality would be helpful on:

- [ ] * `tru.get_records_and_feedback(app_ids)`


https://github.com/truera/trulens/pull/939 ?

Hi @epinzur,

We're excited to announce that the delete_app functionality has been implemented and will be available in the next release!

<!-- Answer -->
Hi, @epinzur,

I'm helping the trulens team manage their backlog and am marking this issue as stale. From what I understand, you requested additional methods to be added to the Tru API to better support working with large databases. There have been some developments in the comments, including a pull request from priyalorha and the implementation of the delete_app functionality announced by arn-tru. 

Could you please confirm if this issue is still relevant to the latest version of the trulens repository? If it is, please let the trulens team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!
Repository: truera/trulens
Issue: Custom LangChain Retriever is not logged
Root cause: Validation Gap

In my Chain,I find that only LangChain Retriever can be logged,Custom Retriever is not logged.

##  LangChain Retriever 
![image](https://github.com/truera/trulens/assets/24437317/7611c1dc-e149-4142-a833-6f0aa1ff229e)
## Custom Retriever
```
class ContextExpandRetriver(BaseRetriever):
    def _get_relevant_documents(
        self,
        query: str,
        *,
        run_manager: CallbackManagerForRetrieverRun,
        **kwargs: Any,
    ) -> List[Document]:
        source_documents = []
        # do something to ContextExpandRetriver
        return source_documents
```
![image](https://github.com/truera/trulens/assets/24437317/fb37d0fc-cef4-4467-8ff6-5b44f751d403)

Just replace retriever in Example blow will reproduce the problem！
https://www.trulens.org/trulens_eval/langchain_quickstart/#load-documents

Because of the bug,the lens App.select_context returns , is not log to the db, and thus can not be dealed by the Feedback
![image](https://github.com/truera/trulens/assets/24437317/5868fd26-c842-4817-a70e-4b202435201c)


Hey @q2wxec ! Thanks for creating the issue.

I’m not quite following the issue here. Is it that the expected component is not showing in the timeline (not being logged at all), or is it just not being selected by `select_context`

@q2wxec this may be fixed by #873 - can you install from that branch and give it a try?

> @q2wxec this may be fixed by #873 - can you install from that branch and give it a try?

ok,i will take a shoot and reply

> @q2wxec this may be fixed by #873 - can you install from that branch and give it a try?

I think this problem has been solved.However some problem occour while open the dashbord.
## This problem solved by looking into the json
![1707534707713](https://github.com/truera/trulens/assets/24437317/7ca486ec-b756-481d-bbac-001d4484d9a2)
## dashbord problem 
![image](https://github.com/truera/trulens/assets/24437317/5df3f3f3-061b-4d5b-ba42-4c43b68026d6)


Awesome news. Re: the dashboard error, this is fixed in main (and 0.22.1) but wasn’t in this branch yet. Just pulled it in to the branch

> Awesome news. Re: the dashboard error, this is fixed in main (and 0.22.1) but wasn’t in this branch yet. Just pulled it in to the branch

thx for your effort

You’re welcome! Assuming it’s working now?

> You’re welcome! Assuming it’s working now?

yes

Will close this once it's merged
Repository: truera/trulens
Issue: Does Trulens assume Open AI Embedding access , is there a way to use it with local Hugging face embeddings like bge-large ?
Root cause: Documentation Deficiency

Does Trulens assume Open AI Embedding access , is there a way to use it with local Hugging face embeddings like bge-large ?

TruLens does not assume use of any particular provider for models (embedding or LLM). We are completely agnostic to your application component choice, and have many supported options (open source and proprietary) on which to run evals (known in our docs as feedback providers)
Repository: truera/trulens
Issue: Amazon Bedrock provider: boto3,botocore packages ModuleNotFoundError
Root cause: Validation Gap

I tried to initialize Bedrock as the feedback provider, but encountered ModuleNotFoundError. Even though my boto3 and botocore are installed and satisfy required versions (logs showing Requirement already satisfied; checked through boto3.__version__ and botocore.__version__). 

Error logs:
```
ModuleNotFoundError: 
boto3,botocore packages are required for using Bedrock models.
You should be able to install them with pip:

    pip install "boto3>=1.33.6" "botocore>=1.33.6"
```

Implementation:
```
from trulens_eval.feedback.provider.bedrock import Bedrock as fBedrock
import boto3

client = boto3.client("bedrock", region_name="us-east-1")
provider = fBedrock(model_id = "anthropic.claude-v2", client=bedrock_client)
```

Any help is appreciated!

This is causing issues for me as well. If I initiate boto3 client for Bedrock without trulens_eval installed, it works fine. Once I install trulens_eval, it messes up and can't repair it somehow.

Hey @SanjuMLGeek @mabelli96 - this should be fixed now in 0.22.2. Please upgrade your trulens version and let us know if this issue persists.

https://github.com/truera/trulens/releases/tag/trulens-eval-0.22.2
Repository: truera/trulens
Issue: Empty board from tru.get_leaderboard(app_ids=["RAG v1"])
Root cause: Documentation Deficiency

I am currently using the code provided in this [Colab notebook](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/quickstart/quickstart.ipynb#scrollTo=-HyRuVA2qR7H) to evaluate my RAG pipeline. While running the evaluation, I encountered an issue:

- Missing Metrics in Leaderboard:

While the previous code was working correctly, when I was attempting to retrieve the leaderboard metrics using the `tru.get_leaderboard(app_ids=["RAG v1"])` method, I encountered an issue where the returned array only contained the latency and total_cost metrics. The metrics groundedness, qa_relevance, and context_relevance were unexpectedly missing from the output.

This error started occurring from yesterday, and I didn't have the same problems again. Is there any possible solution?

Hey @kouskouss - the metrics take some time to generate, and may not be available when you first run `get_leaderboard`.

You can use `as_completed` to wait on the results:

```
from concurrent.futures import as_completed

for feedback_future in  as_completed(rec.feedback_results):
    feedback, feedback_result = feedback_future.result()
    
    feedback: Feedback
    feedbac_result: FeedbackResult

    display(feedback.name, feedback_result.result)
```

> Hey @kouskouss - the metrics take some time to generate, and may not be available when you first run `get_leaderboard`.
> 
> You can use `as_completed` to wait on the results:
> 
> ```
> from concurrent.futures import as_completed
> 
> for feedback_future in  as_completed(rec.feedback_results):
>     feedback, feedback_result = feedback_future.result()
>     
>     feedback: Feedback
>     feedbac_result: FeedbackResult
> 
>     display(feedback.name, feedback_result.result)
> ```

How could the piece of code, that you provided, be connected with the code example from google colab that I followed? I am asking because there is no object with .feedback_results attributes that has been created following this tutorial.


I am mentioning you in case you did not see my previous response ( @joshreini1 ), I am following the initial code from google colab as I mentioned in the beginning, and the board some times is fullfilled with all the metrics and other time it is empty, is there any explanation for that?
Repository: truera/trulens
Issue: Cost tracking for LiteLLM apps/provider
Root cause: Unimplemented Feature Gap

Add cost tracking for LiteLLM apps/provider. Currently cost is recorded as 0.

@piotrm0 is this solved by https://github.com/truera/trulens/pull/866 ?

Solved by https://github.com/truera/trulens/pull/910 .
Repository: truera/trulens
Issue: Add base LLM-generated evals to Huggingface Provider
Root cause: Unimplemented Feature Gap

Huggingface provider currently only contains feedback functions using specific model endpoints on huggingface. Add the option to use base feedback functions (in LLMProvider) through LLMs available on huggingface.

It should work with feedback factions with and without cot_reasons.

User flow should be similar to:

```
from trulens_eval import Huggingface
hugs_provider = Huggingface('mistralai/Mixtral-8x7B-Instruct-v0.1')

hugs_provider.relevance_with_cot_reasons('What is the capital of India?', 'New Delhi')
```
Repository: truera/trulens
Issue: Increase provider test coverage to other feedback providers
Root cause: Unimplemented Feature Gap

`test_providers.py` currently only covers OpenAI provider. Add testing for other feedback providers.

- [ ] AzureOpenAI
- [ ] Bedrock
- [ ] LiteLLM
- [x] LangChain done in #1062 by @arn-tru 
- [X] Huggingface - done in #919 by @venkatkakoju 

@joshreini1 Do we still need some more tests for the AzureOpenAI  or will the open ai test will go with the same.

We do still need tests on AzureOpenAI. Thanks if you're offering :)