zhiminy/EvalEng huggingface dataset:
Repository: truera/trulens
Issue: Example google collab cell has a error when running
Root cause: External Dependency Breakage

In the TruLens quickstart here: https://www.trulens.org/trulens_eval/quickstart/

For the collab:

![image](https://github.com/truera/trulens/assets/67878058/1199ebf7-e6fc-4eb4-8567-564b54a70a21)

I uncommented and ran the first line: 
![image](https://github.com/truera/trulens/assets/67878058/ea2f538b-bfbf-4cd3-bf13-af640e57b602)

Ran all the cells until this cell, and then got this error:

![image](https://github.com/truera/trulens/assets/67878058/559d1c3e-81cc-4724-95e3-f884fe20fa93)

The solution is to add llama_index to the very first cell: 
![image](https://github.com/truera/trulens/assets/67878058/0d556f3d-aa80-451b-b28d-3ef1fbeac303)

Now the cell runs fine:
![image](https://github.com/truera/trulens/assets/67878058/9479ee03-0906-463f-bdca-f6853b8fe174)

We should not require llama-index to import trulens here. Will take a look at the dependencies.

Added PR #714 that should fix this.

> Added PR #714 that should fix this.

Same problem different
Today on the latest version of trulens_eval

<img width="860" alt="image" src="https://github.com/truera/trulens/assets/26019476/c69788eb-34b6-480c-9c66-56fc17bd6f3f">


This is still a problem in the [Colab](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/quickstart/langchain_quickstart.ipynb#scrollTo=wJ2eXpo_7oRd) and in the [Langchain Quickstart on GitHub](https://github.com/truera/trulens/blob/main/trulens_eval/examples/quickstart/langchain_quickstart.ipynb)
The fix I'm using is: 
`pip install llama-index==0.9.21`

Without installing llama-index, i get the following error when running the Python cell below "Import from LangChain and TruLens":

```
TypeError: Dummy.__new__() takes 2 positional arguments but 4 were given

```

After `pip install llama-index` 

```
🦑 Tru initialized with db url sqlite:///default.sqlite .
🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `Tru` to prevent this.
```

Same issue as https://github.com/truera/trulens/issues/731

Please upgrade your trulens version if you are still experiencing this issue. Thanks!

@joshreini1 I just tried the collab right now from https://www.trulens.org/trulens_eval/quickstart/. This is the bottom of the output of the first cell: 

![image](https://github.com/truera/trulens/assets/67878058/e7983bdc-b80d-4634-8e13-c87400a45804)

I then run all the cells until the cell below and get this error:

![image](https://github.com/truera/trulens/assets/67878058/5cb34b72-a50f-44cf-a6cd-515af90041cf)

Dependency hell is fun isn't it xD


Let me create a new issue for this litellm requirement. Will tag you
Repository: truera/trulens
Issue: Getting error while while executing "tru.run_dashboard()" on Colab
Root cause: Validation Gap

Getting below error while executing **tru.run_dashboard()** code on **Colab**

**Error Message:** 

Starting dashboard ...
Config file already exists. Skipping writing process.
Credentials file already exists. Skipping writing process.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-45-4daf7c350606>](https://localhost:8080/#) in <cell line: 72>()
     70 #display(rec)
     71 
---> 72 tru.run_dashboard()

[/usr/local/lib/python3.10/dist-packages/trulens_eval/tru.py](https://localhost:8080/#) in run_dashboard(self, port, address, force, _dev)
    547 
    548         if Tru.dashboard_proc is not None:
--> 549             print("Dashboard already running at path:", Tru.dashboard_urls)
    550             return Tru.dashboard_proc
    551 

AttributeError: type object 'Tru' has no attribute 'dashboard_urls'

Hi; does this happen the first time tru().run_dashboard() is called? That is, can you check whether perhaps you are calling this twice, especially in close succession?

Yes. Getting error while calling first time tru().run_dashboard() 

Please find the attached Colab file for your reference with code.

[Local_LLM_using_Langchain (4).zip](https://github.com/truera/trulens/files/13748226/Local_LLM_using_Langchain.4.zip)


Hi; there was an error introduced in last release that caused this issue. A fix is in the next release (or you can use the github version).

@navinchandrac - can you please retry with the latest release (0.20.0)?
Repository: truera/trulens
Issue: Getting error when using AzureOpenAI for feedback functions
Root cause: Documentation Deficiency

I am running the tutorial from here: https://www.trulens.org/trulens_eval/quickstart/#set-up-feedback-functions

Only difference: I use AzureOpenAI instead of OpenAI.

For the feedback functions I bind the openai client like this:
```python
from trulens_eval.feedback.provider.openai import AzureOpenAI as fAzureOpenAI

azopenai = fAzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_KEY"),
    azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT"),
    api_version = os.getenv("AZURE_API_VERSION")
)
```

Once I run my evaluation questions through the app I get the following error:

Callback class OpenAICallback is registered for handling create **but there are no endpoints waiting to receive the result**.
Run of <lambda> in <Thread(TP.submit with debug timeout_1, started 7936)> failed with: Could not locate app.retrieve.args.query in app/record.

I am having the same issue. How did you resolve it?

> I am having the same issue. How did you resolve it?

I have encountered the same situation, do you have a solution？


When using the AzureOpenAI provider, you need to set the following as environmental variables rather than passing them to the provider. 

- AZURE_OPENAI_ENDPOINT
- AZURE_OPENAI_API_KEY
- OPENAI_API_VERSION

Then you just need to pass the deployment name only to the provider:

```python
from trulens_eval.feedback.provider.openai import AzureOpenAI
openai_provider = AzureOpenAI(deployment_name="...")

openai_provider.relevance(
    prompt="Where is Germany?",
    response="Poland is in Europe."
) # low relevance
```

You can see the API reference here: https://www.trulens.org/trulens_eval/api/provider/openai/azureopenai/#trulens_eval.feedback.provider.openai.AzureOpenAI

> When using the AzureOpenAI provider, you need to set the following as environmental variables rather than passing them to the provider.
> 
> * AZURE_OPENAI_ENDPOINT
> * AZURE_OPENAI_API_KEY
> * OPENAI_API_VERSION
> 
> Then you just need to pass the deployment name only to the provider:
> 
> ```python
> from trulens_eval.feedback.provider.openai import AzureOpenAI
> openai_provider = AzureOpenAI(deployment_name="...")
> 
> openai_provider.relevance(
>     prompt="Where is Germany?",
>     response="Poland is in Europe."
> ) # low relevance
> ```
> 
> You can see the API reference here: https://www.trulens.org/trulens_eval/api/provider/openai/azureopenai/#trulens_eval.feedback.provider.openai.AzureOpenAI

This method is effective, thank you very much for your help.
Repository: truera/trulens
Issue: wrong return statement in feedback criteria
Root cause: Algorithmic Error

the return uses the wrong criteria. should be prompts.LANGCHAIN_CORRECTNESS_PROMPT

https://github.com/truera/trulens/blame/5fa4b82567a2ffddf44bda82d0c6c514bbad8132/trulens_eval/trulens_eval/feedback/provider/base.py#L540
Repository: truera/trulens
Issue: QoI has no access to instance being explained
Root cause: Architectural Constraint

Quantities of interests receive output values on DoI samples but these are not necessarily the instance being explained, except in some cases like point. Because of this, one cannot define a QoI for "explained instance predicted class logits" because predicted class is not know unless the DoI sample is the same as the explained instance.

A related issue is that some may interpret QoI like the default max to mean what I described above but that is not what the max QoI implements because of the issue noted.
Repository: truera/trulens
Issue: change folder that stores the default.sqlite file and the .streamlit folder
Root cause: Documentation Deficiency

when we initialize Tru(), can we specify a folder to store the default.sqlite file and the .streamlit folder, instead of going with the current work directory by default? Thanks


from trulens_eval import Feedback, OpenAI, Tru, TruLlama, Select, OpenAI as fOpenAI
from trulens_eval.feedback import GroundTruthAgreement, Groundedness

tru = Tru()

no need - figured it out:

tru = Tru(database_url="sqlite:///./test_eval_db/trulens_eval.sqlite")
Repository: truera/trulens
Issue: Private LLM
Root cause: Documentation Deficiency

Can we use Trulens also with a private LLM? For example, all my models are running on AWS EC2. Can I run there trulens to evaluate my models?

Thanks

@alexandruvesa - are you wanting to use your private LLM for evaluation? Or simply to evaluate an app that uses a private LLM?

Evaluate an app that uses a private LLM @joshreini1 .

Any regular TruLens usage should work in an ec2. Feel free to share more details about your specific scenario

Closing this ticket. Please re-open if you have further questions.
Repository: truera/trulens
Issue: TypeError: BaseModel.validate() takes 2 positional arguments but 3 were given
Root cause: External Dependency Breakage

when use langchain as a provider,error occour
codes
```
from trulens_eval.feedback.provider.langchain import Langchain
provider = Langchain(chain=llm)
```
stack
```
Traceback (most recent call last):
  File "\codes\ai\langchain\lesson3\trulens.py", line 42, in <module>
    provider = Langchain(chain=llm)
  File "\AppData\Local\Programs\Python\Python310\lib\site-packages\trulens_eval\feedback\provider\langchain.py", line 43, in __init__
    self_kwargs["endpoint"] = LangchainEndpoint(
  File "\AppData\Local\Programs\Python\Python310\lib\site-packages\trulens_eval\feedback\provider\endpoint\langchain.py", line 64, in __init__
    super().__init__(*args, **kwargs)
  File "\AppData\Local\Programs\Python\Python310\lib\site-packages\trulens_eval\feedback\provider\endpoint\base.py", line 173, in __init__
    super(SerialModel, self).__init__(*args, **kwargs)
  File "\AppData\Local\Programs\Python\Python310\lib\site-packages\trulens_eval\utils\pyschema.py", line 610, in __init__
    super().__init__(*args, **kwargs)
  File "\AppData\Local\Programs\Python\Python310\lib\site-packages\pydantic\main.py", line 164, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
TypeError: BaseModel.validate() takes 2 positional arguments but 3 were given
```

Looking into this

@q2wxec - this was fixed by https://github.com/truera/trulens/pull/759 and merged to main. Will be released in 0.20.1 shortly
Repository: truera/trulens
Issue: Azure OpenAI summarize_provider within class Groundedness throws error
Root cause: Documentation Deficiency

On ["Initialize Feedback Function(s)" section of the Azure OpenAI example](https://github.com/truera/trulens/blob/1b7c5167032033448af74f35f43710c8884dddec/trulens_eval/examples/expositional/models/azure_openai.ipynb) the following code:

```
# groundedness of output on the context
groundedness = feedback.Groundedness(summarize_provider=azopenai, groundedness_provider=azopenai)
```
throws an error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[29], [line 18](vscode-notebook-cell:?execution_count=29&line=18)
     [13](vscode-notebook-cell:?execution_count=29&line=13) f_qs_relevance = Feedback(azopenai.qs_relevance, name = "Context Relevance").on_input().on(
     [14](vscode-notebook-cell:?execution_count=29&line=14)     TruLlama.select_source_nodes().node.text
     [15](vscode-notebook-cell:?execution_count=29&line=15) ).aggregate(np.mean)
     [17](vscode-notebook-cell:?execution_count=29&line=17) # groundedness of output on the context
---> [18](vscode-notebook-cell:?execution_count=29&line=18) groundedness = feedback.Groundedness(groundedness_provider=azopenai, summarize_provider=azopenai)
     [19](vscode-notebook-cell:?execution_count=29&line=19) f_groundedness = Feedback(groundedness.groundedness_measure, name = "Groundedness").on(TruLlama.select_source_nodes().node.text).on_output()

TypeError: Groundedness.__init__() got an unexpected keyword argument 'summarize_provider'
```

I believe this is because in the Class definition for `Groundedness` the Args list  a `summarize_provider` but it doesn't seem to be used (that I could find at least) 

```
Args:
            groundedness_provider (Provider, optional): groundedness provider options: OpenAI LLM or HuggingFace NLI. Defaults to OpenAI().
            summarize_provider (Provider, optional): Internal Usage for DB serialization.
        """
```


My workaround was to eliminate the `summarize_provider` from the groundedness var definition: `groundedness = feedback.Groundedness(groundedness_provider=azopenai)`


----
Note on versions:
`trulens-eval                             0.20.0`
`openai                                   1.3.7`


Hey @ingridstevens  - this is the right solution. Summarize_provider was removed a few versions back.

Thanks @joshreini1 for clarifying that the feature is no longer part of Trulens. I appreciate the information. However, I noticed that even though it has been removed, the example notebook for Azure still references to it.

I wanted to bring this to your attention, as it might cause confusion or errors for someone using the example Azure notebook at this link: https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/azure_openai.ipynb
Repository: truera/trulens
Issue: please change the import of PromptTemplate from langchain.prompts
Root cause: External Dependency Breakage

Getting a warning as below

```
UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain.prompts.PromptTemplate instead.
  warnings.warn(

```
This prompt template are now part of the prompts package. The change for this will be in the trulens_eval/feedback/v2/feedback.py module as below

```
# from langchain import PromptTemplate
from langchain.prompts import PromptTemplate
```

I see that the latest version has the change, I was pinning to old version
Repository: truera/trulens
Issue: Custom Feedback Provider
Root cause: Documentation Deficiency

Suppose I have an endpoint set up for Mistral 7B model. How to create a custom feedback provider which utilizes this endpoint/model?

Hey @htcml !

You can write a custom feedback function with any arbitrary callable: [DOCS](https://www.trulens.org/trulens_eval/custom_feedback_functions/)

If your endpoint is compatible with LangChain, you could also use our LangChain provider: [DOCS](https://www.trulens.org/trulens_eval/api/langchain_provider/)

> Hey @htcml !
> 
> You can write a custom feedback function with any arbitrary callable: [DOCS](https://www.trulens.org/trulens_eval/custom_feedback_functions/)
> 
> If your endpoint is compatible with LangChain, you could also use our LangChain provider: [DOCS](https://www.trulens.org/trulens_eval/api/langchain_provider/)

404 - Not found
Repository: truera/trulens
Issue: TypeError in trulens: issubclass() Argument Must Be a Class When Wrapping RAG with TruCustomApp
Root cause: Validation Gap

Hello,

I've encountered a `TypeError` while following the Quickstart guide on the TruLens documentation page (specifically the "Construct the app" section found at https://www.trulens.org/trulens_eval/quickstart/).

Here's the code snippet that is causing the issue:

```python
from trulens_eval import TruCustomApp

tru_rag = TruCustomApp(rag,
                       app_id='RAG v1',
                       feedbacks=[f_groundedness, f_qa_relevance, f_context_relevance])
```

However, when I run this snippet, I receive the following error message:

```
TypeError: issubclass() arg 1 must be a class
```

I have followed the installation steps exactly as described on the Quickstart page, including the necessary imports and environment setup. The error occurs when trying to instantiate TruCustomApp with my RAG model.

Could you please help me understand what might be going wrong here? Am I missing a step, or could there be an issue with the TruCustomApp class definition?

For reference, I'm using the following package versions:

```
trulens_eval: 0.2.0
chroma_db: 4.18
openai: 1.3.7
```

Any guidance on this would be greatly appreciated. Thank you for your support.


Hey @gryhkn - thanks for reporting. This is fixed by https://github.com/truera/trulens/pull/770 and will be available in the next release.

For instructions on installing from this branch, see:
https://www.trulens.org/welcome/#installation-and-setup

I tried installing branch https://github.com/truera/trulens/tree/piotrm/optionals_and_subclass_fixes, however not able to run it successfully.

Please find the error trace

File "c:\Code\tru_lens\feedback_func.py", line 1, in <module>
    from trulens_eval import Feedback, Select
  File "C:\Code\trulen_env\lib\site-packages\trulens_eval\__init__.py", line 83, in <module>
    from trulens_eval.feedback import Bedrock
  File "C:\Code\trulen_env\lib\site-packages\trulens_eval\feedback\__init__.py", line 17, in <module>
    from trulens_eval.feedback.groundedness import Groundedness
  File "C:\Code\trulen_env\lib\site-packages\trulens_eval\feedback\groundedness.py", line 11, in <module>
    from trulens_eval.feedback.provider.litellm import LiteLLM
  File "C:\Code\trulen_env\lib\site-packages\trulens_eval\feedback\provider\litellm.py", line 11, in <module>
    import litellm
ModuleNotFoundError: No module named 'litellm'

Hey @ZS4MLDL @gryhkn - these issues should be fixed in the new release, 0.20.2 available now. Please upgrade and report back if these issues continue. Closing for now.

I tried installing  0.20.2, however I was getting same error. Finally, I had to install litellm and boto3 to make it working..

pip install litellm
pip install boto3
Repository: truera/trulens
Issue: Issue with Trulens Evaluation and TruChain(using LangChain)
Root cause: Interface Contract Mismatch

I'm currently working on LLM evaluation using Trulens and Azure openai and encountering some issues with the code. Here's the relevant portion of my code:

 ```
retriever = vectorstore .as_retriever()

prompt = hub.pull("rlm/rag-prompt")

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)
```
When attempting to apply the Trulens recorder with the following code:
```

# TruLens Eval chain recorder
chain_recorder = TruChain(
    rag_chain, app_id="Chain1_ChatApplication", feedbacks=[f_qa_relevance, f_context_relevance, f_groundedness])
I encounter the error:
```

```
ValidationError: 1 validation error for TruChain
app
  Can't instantiate abstract class Chain with abstract methods _call, input_keys, output_keys (type=type_error)
```
I also tried applying TruChain with an llm Langchain chain:

```
chain = LLMChain(llm=llm, prompt=prompt)

qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever()
)
chain_recorder = TruChain(
    qa, app_id="contextual-chatbot", feedbacks=[f_qa_relevance, f_context_relevance, f_groundedness]
)
```
However, this results in the error:(in the chain_recorder part)

`TypeError: Object of type 'OpenAI' is not JSON serializable
`I'm using Python 3.10.11 and have also tested with Python 3.9. The relevant package versions are as follows:

langchain==0.1.0
langchain-community==0.0.9
langchain-core==0.1.7
langchainhub==0.1.14
trulens-eval==0.18.1

Hi @morhyak , can you run this on latest release?
Repository: truera/trulens
Issue: OpenAI account Terminated after using trulens library
Root cause: Others

Story in a nutshell. I am doing the [Building and Evaluating Advanced RAG](https://www.deeplearning.ai/short-courses/building-evaluating-advanced-rag/) course on a DeepLearning.AI. On the 3rd task, I tested the Window-context RAG locally with my OpenAI API key + trulens evaluation. Note, **I used the same pdf and question list as in the course**. I didn't use the API before , only chatGPT. I even added 5$ to get better RPM for the trulens. The night after I received the termination account email 

*We have determined that you or a member of your organization are using the OpenAI API in ways that violate our policies.*

Again, I used the API once for this course and received a ban after it. I suspect [trulens](https://github.com/truera/trulens) violates the policy of the api usage (maybe they updated them)? Maybe someone from the developer or community can clarify the situation? Because losing access is painful and I still can't get it back. 

Hey @kinivi - sorry to hear this. Can you reach out to OpenAI to get clarification?

I have not seen others experience this issue

Hi, @joshreini1 , yes, I reached the OpenAI but still waiting for more info why account was blocked. Will update when they message back. 

Thanks. Closing for now - please re-open if you discover it's an issue with TruLens.
Repository: truera/trulens
Issue: Arbitrary LLM endpoint as Provider 
Root cause: Documentation Deficiency

Hi! I am wondering if it is possible to use a self-deployed LLM endpoint as the provider of the Trulens feedback. I checked the code briefly and noticed that LiteLLM, google_vertex, bedrock, huggingface, open ai, and Azure open ai are supported. I have tried  the Azure open ai and integration was seamless. However, I am curious about possibility of programmatically connecting to arbitrary model endpoint via http/curl requests. I'd appreciate your help and thoughts on this. 

@lalehsg can you use the langchain provider for this?

Relevant docs: https://www.trulens.org/trulens_eval/api/langchain_provider/

@lalehsg Please let us know if this works for you?

Closing this for now. Please let us know if this needs reopening

hi sorry for the delay. thanks for the reply. the doc link you provided doesn't work. but i think i get the solution. using langchain wrapper functions for this purpose...