zhiminy/EvalEng huggingface dataset:
Repository: truera/trulens
Issue: Use of Custom Feedback Providers
Root cause: Documentation Deficiency

I have an endpoint hosting llama3-8b instruct model and would like to use that as feedback provider. 

I used Langchain as below:

```

  llm = OpenAI(model="meta/llama3-8b-instruct",
                    base_url= "https://llama3-8b-instruct.com/v1",
                    api_key='111',
                    http_async_client=AsyncClient(verify=False),
                   )
  provider = Langchain(chain = llm)

```
	
The above attempt was not successful. Has anyone tried the do the same successfully? Please advise if there is other way to do it. Thank you.

Hi @hmo1234 - can you share a small-as-possible end-to-end example I can reproduce?
Repository: truera/trulens
Issue: [FEAT] AWS Bedrock Claude 3 Support
Root cause: Unimplemented Feature Gap

**Feature Description**
Add support to run feedback functions using Bedrock Claude 3 models.

**Reason**
Discussion: https://github.com/truera/trulens/discussions/1293

**Importance of Feature**
Run feedback functions using the most powerful models on AWS Bedrock.
Repository: truera/trulens
Issue: [BUG] [DOCS] Stock feedback functions for HuggingFace Provider not showing in docs
Root cause: Documentation Deficiency

**Bug Description**
Stock feedback functions for HuggingFace Provider not showing in docs in the following locations:
- https://www.trulens.org/trulens_eval/api/provider/huggingface/
- https://www.trulens.org/trulens_eval/evaluation/feedback_implementations/stock/

**To Reproduce**
Visit links above

**Expected behavior**
Stock huggingface feedbacks should display in docs for both pages
Repository: truera/trulens
Issue: Have a Quickstart from Colab
Root cause: Documentation Deficiency


Agree with @rshih32 that it would be nice to have example (colab) notebook(s).  
I created one and wrote a small article about Trulens in general, because I really like the problem it solves.  
Links:
* [Colab notebook](https://github.com/darinkist/article_track_monitor_llms/blob/main/ColabDemo_Medium_Article_Evaluate_Monitor_LLMs.ipynb)
* [Evaluate and Monitor the Experiments With Your LLM App](https://pub.towardsai.net/evaluate-and-monitor-the-experiments-with-your-llm-app-df391c0f51c9) with a bit more explanation of each step.
  
Please note: To run TruLens or Streamlit in Colab, a small workaround is necessary.



Thanks @darinkist,
Used your example as a basis to get the tunnel working.

Implemented in: https://github.com/truera/trulens/pull/260 

This should make the 0.5.0 release
Repository: truera/trulens
Issue: [FEAT] Check typing with Mypy and add py.typed file
Root cause: Unimplemented Feature Gap

**Feature Description**
It seems the trulens project makes efforts to type the code.
I haven't found that types are checked in CI (are they?). It would be nice to do a check with a tool such as Mypy in CI.
Then a [PEP561 py.typed file](https://peps.python.org/pep-0561/#packaging-type-information) could be added to the package so types can be used in projects that import trulens.

**Reason**
This will increase quality and probably uncover some bugs.

**Importance of Feature**
Currently when importing trulens in a project and checking it with mypy, trulens types are not imported.


@coreyhu You might have added this in upcoming changes. Can you confirm?
Repository: truera/trulens
Issue: [BUG] Using Langchain Async Support on the dashboard will display two pieces of data.
Root cause: Algorithmic Error

**Bug Description**
What happened?
- Hi Trulens teams, I encountered the a question of receiving duplicate responses while using Trulens Async Support.
[link here](https://www.trulens.org/trulens_eval/tracking/instrumentation/langchain/)

**To Reproduce**
Which steps should someone take to run into the same error? A small, reproducible code example is useful here.
```jupyter-notebook
from langchain import LLMChain
from langchain.callbacks import AsyncIteratorCallbackHandler
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

from trulens_eval import TruChain

# Set up an async callback.
callback = AsyncIteratorCallbackHandler()

# Setup a simple question/answer chain with streaming ChatOpenAI.
prompt = PromptTemplate.from_template("Honestly answer this question: {question}.")
llm = ChatOpenAI(
    temperature=0.0,
    streaming=True, # important
    callbacks=[callback]
)
async_chain = LLMChain(llm=llm, prompt=prompt)
async_tc_recorder = TruChain(async_chain)

with async_tc_recorder as recording:
    await async_chain.ainvoke(input=dict(question="What is 1+2? Explain your answer."))
``` 
**Expected behavior**
A clear and concise description of what you expected to happen.
- When asking a question, only get one response.

**Relevant Logs/Tracebacks**
Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. If the issue is related to the TruLens dashboard, please also include a screenshot.

**Environment:**

- OS: [e.g. MacOS, Windows] MacOS
- Python Version 3.12
- TruLens version 0.33.0
- Versions of other relevant installed libraries

**Additional context**
Add any other context about the problem here.


Hi @zongyicoding - @sfc-gh-pmardziel is working on a fix

Hey @zongyicoding - we have a fixes coming for async and streaming in [this PR](https://github.com/truera/trulens/pull/1441).

Feel free to install it from the branch if you'd like to try it now, otherwise we'll have it in the next release.

Hey @zongyicoding - fixes for this were released in [TruLens 1.1](https://github.com/truera/trulens/releases/tag/trulens-1.1.0). Please give it a try and let us know if you have any issues.

```
pip install -U trulens-core trulens-apps-langchain
```
Repository: truera/trulens
Issue: [Request] Make code snippets copy-able on How to use TruLens page
GitHub labels: documentation

enhancement

## Background
- Currently, the [How to use TruLens page](https://www.trulens.org/how-to-use.html) displays 3 code snippets as PNGs. These PNGs are both static and resolution is impacted depending on the user's screen.

## Request
- Replace snippet PNGs with copy-able code snippets for ease of use, accessibility, and ease of maintenance (ideal), OR
- Add hyperlinks to respective files/gists with code represented in PNGs.

how-to-use trulens page is removed, and https://github.com/truera/trulens/pull/358 makes code blocks in all of the docs copyable.
Repository: truera/trulens
Issue: [FEAT] Inject a client record_id
Root cause: Documentation Deficiency

**Feature Description**
When using TruChain, inject a record_id (or an external record_id) when the chain is ran

```
with tru_recorder as recording:
    llm_response = rag_chain.invoke("What is Task Decomposition?")
```

**Reason**
I have all my inputs on a Pandas df. When i get the records dataframe it would be extremely useful to easily join the records df with the original df



Hey @ruifcruz - you can do this today! Try this out:

```python
with tru_recorder as recording:
    # inject the record metadata to records until this is changed.
    recording.record_metadata=dict(external_record_id=<your id here>)
    llm_response = rag_chain.invoke("What is Task Decomposition?")
    # inject a different id for a different record
    recording.record_metadata=dict(external_record_id=<a new record id>)
    llm_response = rag_chain.invoke("Who is George Washington?")
```

Thank you @sfc-gh-jreini 🙏 I would suggest to improve the docs with that info
Repository: truera/trulens
Issue: Request to add functionality to Trulens to measure Latency
Root cause: Unimplemented Feature Gap

By measuring latency, we can compare how effective these models are across different implementations. I believe it would be beneficial for evaluating and comparing the performance of this wrapper. Particularly, working with an e-GPU but in a Sequential process, it may help triage bottleneck issues.

 Latency will be captured automatically starting in the next release on Wednesday 6/14.
Repository: truera/trulens
Issue: [FEAT] Make trulens-app-llamaindex compatible with llamaindex>0.11?
Root cause: Configuration Error

**Feature Description**
I'm using poetry to manage my project and my `pyproject.toml` file looks like this:
```
[tool.poetry.dependencies]
# basic and demostration environment
python = "^3.12"

# llama-index environment
llama-index = "^0.11.10"
llama-index-llms-ollama = "^0.3.2"
llama-index-embeddings-huggingface = "^0.3.1"
trulens = "^1.0.2"
trulens-providers-openai = "^1.0.2"
```

When I run command `poetry add trulens-apps-llamaindex`, it will report error like this:
```
Because no versions of trulens-apps-llamaindex match >1.0.2,<2.0.0
 and trulens-apps-llamaindex (1.0.2) depends on llama-index (>=0.10,<0.11), trulens-apps-llamaindex (>=1.0.2,<2.0.0) requires llama-index (>=0.10,<0.11).
So, because rag demo depends on both llama-index (^0.11.10) and trulens-apps-llamaindex (^1.0.2), version solving failed.
```

It looks like trulens-app-llamaindex is incompatible with the newest llamaindex whose version is > 0.11.

**Reason**
I want to use new packages in my project because new packages may have new features and methods.


Thanks, this is likely a versioning issue in the `pyproject.toml`. llamaindex > 0.11 should be supported. Will make a fix shortly

Fixed in #1482 and merged. Will be available in the next release.
Repository: truera/trulens
Issue: Attributions fails in tf2 with graph activated through tf.function annotated call
Root cause: Interface Contract Mismatch

Describe the bug
The symptom occurs with error 'Tensor' has no attribute 'numpy'
The problem is because this call in tensorflow_v2 is expecting eager, but it is not in eager mode.

To Reproduce
implement a tensorflow.keras.Model

Override call() function with @tf.function annotation.

Quick hack fix is to remove the annotation, but this is not the correct longterm solution
```
from tensorflow.keras import Model
class CustomModel(Model):

def __init__ ....

@tf.function
def call()...
```
(Run attribution on this model)
Repository: truera/trulens
Issue: [BUG] why not found trulens.providers after pip install trulens 
Root cause: Documentation Deficiency

not found trulens.providers dic after use pip install trulens.  

i follow this flow https://www.trulens.org/getting_started/quickstarts/quickstart/

![image](https://github.com/user-attachments/assets/83b662c0-988b-4a11-a5e7-4d44a6f60551)


Hi @jianhuisu - you need to install `trulens-providers-openai`, this is included in the `pip install` cell at the top of the notebook.

> Hi @jianhuisu - you need to install `trulens-providers-openai`, this is included in the `pip install` cell at the top of the notebook.

thank you , i was careless
Repository: truera/trulens
Issue: [BUG] Snowflake Connector gives error when using active snowpark session in "Streamlit in Snowflake"
Root cause: Validation Gap

**Bug Description**
When I want to create a connection to the Snowflake within "Streamlit in Snowflake" (thus, during running a Streamlit App on Snowflake Warehouse) it gives the following error message

`ValueError: Connection parameters missing from provided `snowpark_session`: ['user']`

This error does not occur when using the same code line in e.g. Snowflake Notebooks.

**To Reproduce**
1. Create a a simple Streamlit App for "Streamlit in Snowflake"
2. Add a snowflake connection using the following code line 
```
from snowflake.snowpark.context import get_active_session
from trulens.connectors.snowflake import SnowflakeConnector
tru_snowflake_connector = SnowflakeConnector(
    snowpark_session=get_active_session(),
    database_redact_keys=True
)
```
3. Publish the streamlit app on snowflake

**Expected behavior**
The Trulens snowflake connector should use the active snowflake connection (or snowpark session) and create a trulens connection to the database.

**Relevant Logs/Tracebacks**
File "/usr/lib/python_udf/49145eaa7707ed0d820f39c30a83d26d83795c04529fe37a973e19b64ed3abd5/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 600, in _run_script
    exec(code, module.__dict__)
File "/tmp/appRoot/app.py", line 14, in <module>
    tru_snowflake_connector = SnowflakeConnector(
File "/usr/lib/python_udf/49145eaa7707ed0d820f39c30a83d26d83795c04529fe37a973e19b64ed3abd5/lib/python3.10/site-packages/trulens/connectors/snowflake/connector.py", line 101, in __init__
    raise ValueError(

**Environment:**

- OS: Linux
- Python Version: 3.10
- TruLens version: 1.1.0
- Versions of other relevant installed libraries
- streamlit: 1.35.0
- snowflake-snowpark-python: 1.18.0

**Additional context**
Here you will find a screenshot of the error and a screenshot for the same code in Snowflake Notebooks.

![image](https://github.com/user-attachments/assets/017b6051-547a-4438-93c5-e0253b9293b9)
![image](https://github.com/user-attachments/assets/8d91a57c-928f-4b40-866d-c12942f9ba90)




Hi @cwitt2013 -

Unless the role has the `READ SESSION ON ACCOUNT` privilege, Streamlit in Snowflake nulls out `CURRENT_USER` (in addition to a few others) which is leading to this issue. You can solve it by doing either of the following:

```
USE ROLE ACCOUNTADMIN;
GRANT READ SESSION ON ACCOUNT TO ROLE streamlit_owner_role;
```


Thank you @sfc-gh-jreini! That solved the issue. Do you have any documentation on the rights needed on Snowflake to run Trulens?

There are not blanket privileges needed to run TruLens on Snowflake - but happy to help if you run into any other issues.
Repository: truera/trulens
Issue: [FEAT] how can I use LLM from huggingface as the provider of the Trulens feedback?
Root cause: Unimplemented Feature Gap

I want to use the local model downloaded from Hugging Face as the provider of Trulens feedback for the RAG Evaluation project. If this feature is already available, how can I use it?
Thank you.


Hi @Anchang8 - we don't currently have this feature through models running locally from Hugggingface, but we can run local LLMs as feedback providers using Ollama (via LiteLLM) if that would meet your needs.

Here's an example notebook:

[Ollama Quickstart](https://www.trulens.org/cookbook/models/local_and_OSS_models/ollama_quickstart/)

@Anchang8 have you had a chance to give this a try?
Repository: truera/trulens
Issue: [BUG] Postgres fails to insert value as np.float64(..) for result 
Root cause: Environment Incompatibility

**Bug Description**
I'm running my evaluations in deferred mode with `tru.start_evaluator(restart=True, return_when_done=True)`. When updating the feedbacks in my postgres db I get an error as it can't insert `np.float64` values.

**Expected behavior**
Feebacks should be updated without any issue.

**Relevant Logs/Tracebacks**

```
Run of {'prepare_feedback'} in <Thread(TP.submit with debug timeout_19, started 69017842677312)> failed with: (psycopg2.errors.InvalidSchemaName) schema "np" does not exist
LINE 1: ...hat the QUESTION seeks.\n\nScore: 3"}}]}', result=np.float64...
                                                             ^

[SQL: UPDATE trulens_feedbacks SET last_ts=%(last_ts)s, status=%(status)s, calls_json=%(calls_json)s, result=%(result)s, cost_json=%(cost_json)s, multi_result=%(multi_result)s WHERE trulens_feedbacks.feedback_result_id = %(trulens_feedbacks_feedback_result_id)s]
[parameters: {'last_ts': 1733478034.682396, 'status': 'done', 'calls_json': '{"calls": [{"args": {"question": "<QUESTION>", "context": "<CONTEXT>"}}]}', 'result': np.float64(1.0), 'cost_json': '{"n_requests": 1, "n_successful_requests": 1, "n_completion_requests": 1, "n_classification_requests": 0, "n_classes": 0, "n_embedding_requests": 0,  ... (48 characters truncated) ... unks": 0, "n_prompt_tokens": 587, "n_completion_tokens": 84, "n_cortex_guardrails_tokens": 0, "cost": 0.00013844999999999998, "cost_currency": "USD"}', 'multi_result': 'null', 'trulens_feedbacks_feedback_result_id': 'feedback_result_hash_d5515cdbd278d1e10bcfa3b1ef531a65'}]
(Background on this error at: https://sqlalche.me/e/20/f405)
```

**Environment:**

- OS: Google Cloud Run Functions
- Python Version: 3.12
- TruLens version: 1.2.10



hi @theocnrds, thank you for the bug report!

This looks like a known issue with sqlalchemy ([GitHub thread](https://github.com/sqlalchemy/sqlalchemy/discussions/11712)). Based on the maintainer's response, could you try:
1. Ensuring that the connection string uses `postgresql+psycopg` to use psycopg 3
2. Ensuring that your environment has `psycopg >= 3.2`, which should support this out of the box ([psycopg docs](https://www.psycopg.org/psycopg3/docs/news.html#psycopg-3-2))

Let us know if this solves your issue!

cc @sfc-gh-dkurokawa 

Thanks, that helped!

hi @theocnrds, just wanted to follow up here - we updated the docs to recommend usage psycopg3, and I'll close off the issue for now. 

Happy new year :)
Repository: truera/trulens
Issue: [BUG] Using multiple tags for Record and App
Root cause: Validation Gap

**Bug Description**
Tags can only be a single string. I tried different formats (comma separated etc.) but the whole string is always split into separate characters, i.e. `tags="tag1,tag2"` leads to the tags `"t", "a", "g", "1", "2", ","`.

**Expected behavior**
Input values should be of type `str | list[str]`.
`tags=["tag1", "tag2"]` should lead to the tags `"tag1", "tag2"`

**Environment:**

- Python Version: 3.12
- TruLens version: 1.2.10
Repository: truera/trulens
Issue: [root] Internal Python error in the inspect module. Below is the traceback from this internal error.
Root cause: Validation Gap


### Discussed in https://github.com/truera/trulens/discussions/1733

<div type='discussions-op-text'>

<sup>Originally posted by **harshalDharpure** January 17, 2025</sup>
I am getting below error while following the Documents of Trulens with Snowflake --> https://quickstarts.snowflake.com/guide/getting_started_with_llmops_using_snowflake_cortex_and_trulens/index.html?index=..%2F..index#10


prompts = [
    "How do I launch a streamlit app?",
    "How can I capture the state of my session in streamlit?",
    "How do I install streamlit?",
    "How do I change the background color of a streamlit app?",
    "What's the advantage of using a streamlit form?",
    "What are some ways I should use checkboxes?",
    "How can I conserve space and hide away content?",
    "Can you recommend some resources for learning Streamlit?",
    "What are some common use cases for Streamlit?",
    "How can I deploy a streamlit app on the cloud?",
    "How do I add a logo to streamlit?",
    "What is the best way to deploy a Streamlit app?",
    "How should I use a streamlit toggle?",
    "How do I add new pages to my streamlit app?",
    "How do I write a dataframe to display in my dashboard?",
    "Can I plot a map in streamlit? If so, how?",
    "How do vector stores enable efficient similarity search?",
]


with tru_rag as recording:
    for prompt in prompts:
        rag.query(prompt)

tru_session.get_leaderboard()





 [root] Internal Python error in the inspect module.
Below is the traceback from this internal error.

ERROR [root] Internal Python error in the inspect module.
Below is the traceback from this internal error.

NameError: name 'Complete' is not defined

During handling of the above exception, another exception occurred:

AttributeError: 'NameError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

AssertionError
NameError: name 'Complete' is not defined

During handling of the above exception, another exception occurred:
</div>

Can anyone Please help me in resolving the above issue
Repository: truera/trulens
Issue: [BUG]TruLens Saves Data in Previously Used Schema Despite Configuration to Use New Schema
Root cause: Configuration Error

**Bug Description**
I was using the PUBLIC schema, and all the tables in the PUBLIC schema were created by TruLens. Now, I have created all the tables under the TRULENS schema and configured the session to use the TRULENS schema. However, despite confirming the active schema as TRULENS using the CURRENT_SCHEMA() query, TruLens still saves new data into the PUBLIC schema instead of the intended TRULENS schema.

I don’t want to move the old data into the new schema. I want the data in the PUBLIC schema to remain there while new data is saved in the TRULENS schema. What should I do to achieve this?

```
sf_settings = {
    'account': 'P',
    'database': 'da',
    'warehouse': 'OH',
    'role': 'T',
    'user': 'NG',
    'schema': 'TRULENS',
    'password': {'password': None},
    'paramstyle': 'qmark'
}
```

![Image](https://github.com/user-attachments/assets/bb387f2a-542d-4e87-88e0-002931cc62a1)

![Image](https://github.com/user-attachments/assets/f495a72a-ec2f-4b85-83f6-6fa6eed883bd)

**Environment:**

snowflake-sqlalchemy version: 1.7.3
trulens-eval version: 1.3.2
LiteLLM version: 1.57.8
"python.version" : 3.11.11,
"python.connector.version" : 3.12.1,
"os.name" : Linux

**Additional context**
Add any other context about the problem here.


Hi, when are you changing the schema from "PUBLIC" to "TRULENS"? It's important to do it before creating the `trulens.connectors.snowflake.SnowflakeConnector`.

Closing this for now. @HannaHUp let us know the answer to @sfc-gh-dkurokawa's question if you're still experiencing this issue