zhiminy/EvalEng huggingface dataset:
Repository: truera/trulens
Issue: OPENAI_API_BASE env won't work
Root cause: External Dependency Breakage

OPENAI_API_BASE env won't work, both in environment, or below

```python
from trulens_eval.feedback.provider.openai import OpenAI as fOpenAI

fopenai = fOpenAI(
    api_base='https://xxxx/openai/v1'
)
```

when I use AzureOpenAI, deployment_id is required when create AzureOpenAI object. But with deployment_id filled, another error thrown when recording query

```python
from trulens_eval.feedback.provider.openai import AzureOpenAI

import os

os.environ['OPENAI_API_KEY'] = 'xxx'
os.environ['OPENAI_API_BASE'] = 'https://xx/v1'

fopenai = AzureOpenAI(deployment_id='xxx')
...

with tru_rag as recording:
    for query in tqdm(eval_questions):
        rag.query(query['question'])
```
`openai request failed <class 'TypeError'>=Completions.create() got an unexpected keyword argument 'deployment_id'. Retries remaining=3.` thrown when executing `rag.query`

package version:

trulens-eval==0.18.2
openai==1.3.7

Finally I got a workaround:

```python
from openai import OpenAI
import os

os.environ['OPENAI_API_KEY'] = 'xxx'

client = OpenAI(base_url='https://xxx/v1')
openai_client = OpenAIClient(client=client)
openai_endpoint = OpenAIEndpoint(client=openai_client)
fopenai = fOpenAI(endpoint=openai_endpoint)
```

Hi @Steven-Luo,  

Thanks for raising the issue and glad to hear you have gotten a workaround!

So the issue you were encountering in `AzureOpenAI` was due to the breaking changes introduced in the new v1.x openai api. 
Specifically, the removal of `openai.Deployment (Previously used for Azure OpenAI)
`
Here's some migration examples: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/migration?tabs=python%2Cdalle-fix#use-your-data. 

We'll be adding API-parity classes in the upcoming week as well. I hope this is helpful. 



@Steven-Luo I tried your fix but I couldn't get it to work. The url is just the Azure end point right?

@Steven-Luo @Ethan-Otto The fix for this is now released in trulens_eval==0.18.3. Please upgrade and let us know if you have any persistent issues.

Thanks!
Repository: truera/trulens
Issue: Poor readability of Chinese in Records section
Root cause: Documentation Deficiency

Run dashboard
```python
...
tru.run_dashboard()
```

Please check the image below:
<img width="961" alt="截屏2023-12-04 00 03 48" src="https://github.com/truera/trulens/assets/3992893/0f460ea2-2ba2-44ba-a4e2-6c6eb5b99c0a">

package version:
trulens-eval==0.18.2


+1
Repository: truera/trulens
Issue: Can this tool be used locally without external services?
Root cause: Documentation Deficiency

Came due to the course on deeplearning.ai. 

Yes, you can use local models for feedback evaluations.

Thanks!

Maybe I stated it incorrectly.

I would like to use Azure OpenAI, but I was referring to your logging services.

Hey @se-kon - logging is all local!

See for reference: https://www.trulens.org/trulens_eval/where_to_log/

thanks!
Repository: truera/trulens
Issue: trulens-eval meets hallucinations detection and reduction

Hi ,

I wonder what metric is proposed to measure hallucination rate . There are many definitions for this metric . I'd like to know what the vision of package creators and reasonable explanation. 

Thanks,
Nissim

Hey @snassimr !

In RAG contexts, we consider the RAG Triad of context relevance, groundedness and answer relevance to hold the key for measuring hallucination.

Would recommend checking out this section of the docs: https://www.trulens.org/trulens_eval/core_concepts_rag_triad/

Let us know if you have further questions - thanks!

Can I use  Answer Relevance * Context Relevance * Groundedness as sort of goodness of fit  similar to accuracy/auc in Binary classification problem ?

If you want just one metric I think that’s reasonable. I’d suggest you gain a lot more power in discovering failure modes/possible areas of improvement if you use them separately however.
Repository: truera/trulens
Issue: tru.run_dashboard() raised WinError, trulens_eval versions '0.18.1', '0.18.2' 
Root cause: Environment Incompatibility

>>> from trulens_eval import Feedback, OpenAI, Tru, TruLlama, Select, OpenAI as fOpenAI
>>> tru = Tru()
🦑 Tru initialized with db url sqlite:///default.sqlite .
🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `Tru` to prevent this.

>>> tru.run_dashboard()
Starting dashboard ...
Config file already exists. Skipping writing process.
Credentials file already exists. Skipping writing process.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "xxxxxxxxxxxx\anaconda3\envs\GenAI_Stable\lib\site-packages\trulens_eval\tru.py", line 543, in run_dashboard
    proc = subprocess.Popen(
  File "xxxxxxxxxxxx\anaconda3\envs\GenAI_Stable\lib\subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "xxxxxxxxx\anaconda3\envs\GenAI_Stable\lib\subprocess.py", line 1456, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
**FileNotFoundError: [WinError 2] The system cannot find the file specified**

Hey @haolxx !

What environment are you running this in? Can you share more details so we can reproduce?

> Hey @haolxx !
> 
> What environment are you running this in? Can you share more details so we can reproduce?

I'm running the code in a anaconda environment with python 3.10.13. thx

Assuming you're also on Windows based on the error?

> Assuming you're also on Windows based on the error?

Correct

Having trouble reproducing this. Is this a VM? Do you see the logging db (default.sqlite) in your local directory?

> Having trouble reproducing this. Is this a VM? Do you see the logging db (default.sqlite) in your local directory?

not a VM - just an anaconda environment in my local machine. Yes, I see those 2 lines of code created the default.sqlite file and the .streamlit folder in my working directory.  .streamlit folder has 2 files: config and credentials. 

I switched back to python 3.8 and it worked now... not sure why it failed for python 3.10.13...

Good to know - thanks! Will continue to look into why it's sometimes failing for 3.10.13
Repository: truera/trulens
Issue: get_records_and_feedback function doesn't return the `relevance` key
Root cause: Resource Mishandling

I'm running the following code to estimate answer relevance.

```
for item in qa_pairs:
    with tru_recorder as recording:
        question = item['question']
        tru_recorder.app(question)

records = self._tru.get_records_and_feedback([])[0]

return records['relevance'].tolist()
```

When I'm trying to retrieve the relevance score from the output, I'm getting `KeyError: 'relevance'`. However, if I make some delay before the return statement, the key `relevance` is presented. This code is executed within an async function. It seems that there is some issue with background update of the results.

Hi @artem-seraf - the first time a feedback function is used, it's feedback column wouldn't be added to the records db until the feedback is generated. Does that make sense?

Does this cause any issues for you downstream?

Hi @joshreini1 
Thanks for your reply. Such a behaviour causes an error in my app because I'm trying to score question/answer pairs and write scores to my local DB. Thus, I need a way to ensure that all feedbacks are generated before the work with the DB.
Now it seems that I managed how to deal with that in a hacky way. I revealed that the `recording` object contains `Future` objects that correspond to the result of the execution of the feedback. Thus, I'm trying to extract these `Future` objects and wait for the end of their execution.

```
all_future_objects = []
for item in qa_pairs:
	with tru_recorder as recording:
		question = item['question']
		tru_recorder.app(question)
		
		# Extract all Future objects that correspond to the execution results of a feedback
		all_future_objects.append(recording.records[0].feedback_results[0])

# Wait for the end of the execution of all feedbacks
concurrent.futures.wait(all_future_objects)
```


Hi @artem-seraf ; another way to do this is to use the "WITH_APP" FeedbackMode (a parameter to TruChain/TruLamma/*) which will configure it to block on your app invocation until feedbacks are computed (effectively the same thing you are doing now with waiting for the futures). Your solution is more efficient though as it allows them to operate in parallel.

Hi @piotrm0 
Thanks for this tip! Actually, the previous code wasn't working in all cases (sometimes, I was still getting the error realted with the key absence). I assume that there are more async operations except futures that are involved into the retrieval of updated records (maybe it is connected with Sqlite). The following code snippet seems to work in a stable way.

```
all_future_objects = []
for item in qa_pairs:
    with tru_recorder as recording:
        question = item['question']
        tru_recorder.app(question)
        all_future_objects.append(recording.records[0].feedback_results[0])

# Wait for Future objects and extract score directly
scores = []
for f in as_completed(all_future_objects):
    response = f.result()
    scores.append(response[1].result)
```
Repository: truera/trulens
Issue: Langchain provider for all types of LLMs
Root cause: Unimplemented Feature Gap

I'm trying to implement feedback functions with google endpoints in vertex AI the same way that works for OpenAI. 

I do not want to implement a provider that is solely for google, I would like to implement it for langchain directly but reusing the structure provided by [LLMProvider](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/base.py#L30) (which is not exposed by the way)

In this way we are not dependent of the particular LLMs implementation but use langchain as a middleware.

Is this something that is in the radar or do you think will be a good addition?


Hey @Nvillaluenga ! This would be really useful, @shayaks and I were discussing that earlier this afternoon. It would be better to have a single provider to maintain that works across models than our current state of maintaining multiple model providers.

Hey Josh, are you going to work on this? I think I can do it over the weekend

Please do!

I can review once you have a PR. Thanks @Nvillaluenga !!

Hey there, here is my [PR](https://github.com/truera/trulens/pull/664) I tried to keep it as simple as possible, let me know what do you think.

PS: I'm thinking on adding some unit test later in the week. 

Merged! Will be released on 0.19
Repository: truera/trulens
Issue: Inconsistent Eval scores across multiple runs
Root cause: Documentation Deficiency

Hi Team 

When I am evaluating a set of query engines(from llama index) across multiple runs on the same set of evaluation questions , I am getting different evaluation scores.

I have also tried with openai gpt-3.5-turbo  with temperature 0 , seed & system_fingerprint parameter set too.


<img width="1189" alt="Screenshot 2023-12-07 at 12 06 17 PM" src="https://github.com/truera/trulens/assets/137484406/4f1453c4-0cb4-4b47-a9a5-4c7fa9edec8b">

<br>
<br>

<img width="1189" alt="Screenshot 2023-12-07 at 12 11 58 PM" src="https://github.com/truera/trulens/assets/137484406/58656d17-0876-4c0a-b72d-41ed61f39335">



Hi [venkatdeelip](https://github.com/venkatdeelip) thanks for raising the issue!

From the description, the first thing I'd like to clarify is that when you said `"the same set of evaluation questions"`, were you referring to identical queries being submitted to the LlamaIndex query_engines? 

If so, then these inconsistent scores should make sense and it's indicating feedback functions are working as expected. 
The reason is that the evaluations from our feedback functions look at different components in the RAG pipelines (i.e. queries, retrieved context chunks, generated responses), and these, especially the retrieved context chunks, can be different across multiple runs.  
Here's a quick conceptual overview of how these 3 scores are generated https://www.trulens.org/trulens_eval/core_concepts_rag_triad/ 


One quick way to verify this is to click into each run and pick one score (say, `Groundedness`) and eyeball the retrieved contexts. These should look different across each run. 

let me know if this is helpful and happy to answer further questions
Repository: truera/trulens
Issue: AzureOpenAI does not work
Root cause: Interface Contract Mismatch

This is the only solution to make Azure openAI work with the library inspite of it having support for it.
```python
from openai import AzureOpenAI as az
from trulens_eval.feedback.provider.openai import OpenAI
llm = OpenAI(model_engine="vanilla-gpt-35-turbo-16k")

oai_client  = az(
    api_key=OPEN_AI_KEY,  
    api_version="2023-10-01-preview",
    azure_endpoint = BASE_URL
    )

llm.endpoint.client = oai_client
```

using AzureOpenAI from trulens eval gives error as there is no deployment_id argument supported on openai side



https://github.com/truera/trulens/pull/640

Hey @kocsszi ! Not exactly sure what happend with this open/closed issue, but we released a patch 0.18.3 that fixes support for AzureOpenAI. If you haven't already, please upgrade trulens_eval to 0.18.3.

Thanks!

Hey @joshreini1 ! Thanks for getting back, opened it then realized was duplicate and wanted to tag it if sy wanted a workaround. But the fix was pretty fast too.
Repository: truera/trulens
Issue: Supporting harmless metrics
Root cause: Documentation Deficiency

Hi , 

I wonder if trulens supports any harmless evaluations metrics ?

I see a reference to this https://github.com/truera/trulens/blob/main/core_concepts_honest_harmless_helpful_evals.md

but the document doesn't exists

hey there!

Yes there are a couple of built-in feedback functions that can be used to evaluate harmlessness 

please see https://github.com/truera/trulens/tree/main/trulens_eval/examples/expositional/use_cases/iterate_on_rag and let us know if these are helpful.

P.S. notebooks are a bit outdated but we are iterating fast on updating the examples along with the documentation so stay tuned! :)
Repository: truera/trulens
Issue: Ground truth score does not always range from 0 to 1
Root cause: Validation Gap

I noticed that the Ground Truth feedback function doesn't always provide scores between 0 and 1, which throws off the overall average estimate for the score (see image below, experiment was run on a labeled question / answer evaluation set)
![image](https://github.com/truera/trulens/assets/52364891/7c07578c-b094-41e4-b760-5c62cf3f69e9)

Looking through the utils for the GroundTruthAgreement class, it looks like there's two main causes:

1. The AGREEMENT_SYSTEM_PROMPT does not reliably get the LLM to return the score from 0-10 at the end of its response. Importantly, I noticed in these cases when the score is above 10, the generated answer is not necessarily correct, so the number is likely not an attempt at a score from the LLM.
2. There's no logical catch in the re_0_10_rating() util function (trulens_eval/utils/generated.py) for when the matched score is outside the expected bounds. 

IMO I think the prompt can be tweaked to more specifically instruct the LLM to return the score at the end of the response, and if the system still returns a score outside the 0-10 range, it should be considered as a failure to match and return -10 to avoid biasing the results in either direction. Let me know what you think!

Hi @andrec279, 

Thanks for raising the issue and appreciating you taking a deeper look into the root cause!

I've submitted a small PR to mitigate the issue, and we have plans incorporating [Instructor](https://github.com/jxnl/instructor) (or similar modules) to allow for re-asking and more robust rating generation.
Repository: truera/trulens
Issue: Unit test should not call LLMs
Root cause: Validation Gap

Unit test should not have dependencies with LLMs.

API calls should be mocked, to ensure the test are in fact unitary.

Furthermore this is producing inconsistent result in the pipelines due to fails in the api calls (For example due to quota exceeded with OpenAI)

That beeing said I think **We need to mock LLMs calls**


Hey @Nvillaluenga ! Just checking to see if you still planned to take this on?
Repository: truera/trulens
Issue: Integration with other vectorDBs
Root cause: Documentation Deficiency

@joshreini1 
Are you open to Integrate with other VectorDBs for giving flexibility to user to use TruLens with any vectorDB of their choice?

I can work on integrating with ChromaDB and LanceDB for scalibility.
Please let me know, What are your thoughts on this.



Hey @PrashantDixit0 ! Because we wrap the application itself, it’s agnostic to the vector db being used (no need to write integration code).

However if you want to add examples for other vector dbs, that’d be great!

Sure @joshreini1,
Thanks for replying
Repository: truera/trulens
Issue: Does TruLens work with Local LLM using Langchain without OpenAI Key?
Root cause: External Dependency Breakage

Does TruLens work with Local LLM using Langchain without OpenAI Key? (Example for Local LLM: https://colab.research.google.com/drive/1XVjur9cdYLhxe6mrQwQUlWIm4VJmiYlu?usp=sharing#scrollTo=wJ7k5mYXj0vG)

Getting below error while using TruLens with Local LLM using Langchain for above example

**_ImportError: cannot import name 'LLM' from 'llama_index.llms.base' (/usr/local/lib/python3.10/dist-packages/llama_index/llms/base.py)_**


Also which LLM used by TruLens (as internal) for to get expected feedback parameters result comparison with LLM App actual feedback result?

Hi; this may be due to recent changes in llama_index, can you update both the trulens_eval package and llama_index package?

Hey @navinchandrac ! The error message you posted about above is fixed/released, however it's unreleated to running feedback functions with a Local LLM. Can you post more info on that if you're still struggling?

Hey again @navinchandrac ! Just added a new example of using ollama to run feedback functions locally in this [PR](https://github.com/truera/trulens/pull/703). Feel free to check it out!

Closing this issue for now. Please create a new one or re-open if you run into new issues
Repository: truera/trulens
Issue: ImportError: cannot import name 'LLM' from 'llama_index.llms.base'
Root cause: External Dependency Breakage

I encounted "**ImportError**: cannot import name 'LLM' from 'llama_index.llms.base'" when running the following code:

`from trulens_eval import Tru`

I investigated 'llama_index.llms.base' and realised that the 'LLM' class had been renamed as 'BaseLLM'. I adjusted the source code and managed to get the code to work, but thought to report the issue here as well.

Thanks for reporting @wenhao89 ! This is already patched in main and will be released in 0.19 today.

Following up @wenhao89  - this change is released! Please upgrade to trulens_eval==0.19.1. Thanks!
Repository: truera/trulens
Issue: Is there anyway to specify streamlit  server.port and server.address properties when running the tru.run_dashboard()
Root cause: Unimplemented Feature Gap

Hi Team,

I am planning to deploy the Trulens-Eval dashboard on Azure App Server. Please let me if any documentation or example available to deploy trulens-eval in azure app service.

If not, let me know is there any way to pass **server.port** and **server.address** to **run_dashboard** function call

I am also looking for the same capability

Hi, we added `port` and `address` arguments to `Tru.run_dashboard`. You can use it now from the github version or wait until next release.

Hey @VikramAIS and @SanjuMLGeek ! This is released in trulens_eval==0.19.2. Please upgrade to get this feature, thanks!
Repository: truera/trulens
Issue: 0.18.3 trulens_eval fails when importing OpenAI
Root cause: Environment Incompatibility

I've upgraded llama_index to the latest 0.9.15 due to an [incompatibility](https://github.com/truera/trulens/issues/675) in trulens_eval 0.18.2 which is fixed in 0.18.3. However, 0.18.3 now fails when importing OpenAI:

```
Traceback (most recent call last):
    from trulens_eval import OpenAI
  File "/venv/lib/python3.9/site-packages/trulens_eval/__init__.py", line 83, in <module>
    from trulens_eval.feedback import Bedrock
  File "/venv/lib/python3.9/site-packages/trulens_eval/feedback/__init__.py", line 16, in <module>
    from trulens_eval.feedback.feedback import Feedback
  File "/venv/lib/python3.9/site-packages/trulens_eval/feedback/feedback.py", line 17, in <module>
    from trulens_eval.feedback.provider.endpoint.base import Endpoint
  File "/venv/lib/python3.9/site-packages/trulens_eval/feedback/provider/__init__.py", line 1, in <module>
    from trulens_eval.feedback.provider.base import Provider
  File "/venv/lib/python3.9/site-packages/trulens_eval/feedback/provider/base.py", line 8, in <module>
    from trulens_eval.feedback.provider.endpoint.base import Endpoint
  File "/venv/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/__init__.py", line 6, in <module>
    from trulens_eval.feedback.provider.endpoint.openai import OpenAIEndpoint, OpenAIClient
  File "/venv/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/openai.py", line 47, in <module>
    class OpenAIClient(SerialModel):
  File "/venv/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/openai.py", line 68, in OpenAIClient
    client: Optional[oai.OpenAI | oai.AzureOpenAI] = None,
TypeError: unsupported operand type(s) for |: 'type' and 'type'
```

Hey @benke-rl ! We merged a fix for this earlier this week and will release it with 0.19 tomorrow.

If you don’t want to wait, feel free to install from the repo. Thanks!

Following up @benke-rl  - this change is released! Please upgrade to trulens_eval==0.19.1. Thanks!
Repository: truera/trulens
Issue: Error while importing Tru from trulens_eval
Root cause: External Dependency Breakage

Hello,

I am running below code from Sagemaker Jupyter notebook and somehow when I run this code it throw error as attached. I checked the version of llama_index and its the latest 0.9.15 and also using latest versions of trulens  and trulens_eval. 
If I import LLM from llama_index.llm.base independently, I don't see any error.

`from trulens_eval import Tru`

<img width="1115" alt="Screenshot 2023-12-14 at 3 00 32 PM" src="https://github.com/truera/trulens/assets/43816307/98b89e9a-9507-48b9-a48d-170ecc78115d">


Same here

Actually the Trulens team already fixed it 2 days ago : https://github.com/truera/trulens/pull/671

So you just have to update your trulens imports locally until they release version 0.18.4 and update the install in their tutorials

@joshreini1 you contacted me on Twitter and that was my problem :)

Thanks Michael! Glad to hear you found the fix:)

Following up @michaelromagne @victormimo @SanjuMLGeek - this change is released! Please upgrade to trulens_eval==0.19.1. Thanks!

Thank you. I have tested it and working good now. Need help on issue [# 676](https://github.com/truera/trulens/issues/676) now..!!