zhiminy/EvalEng huggingface dataset:
Repository: truera/trulens
Issue: [BUG] String type error in trulens dashboard
Root cause: Validation Gap

**Bug Description**
 I’ve created a feature branch of our production code and added the logic that utilizes the add_dataframe method to submit data to TruLens for evaluation.  It completes without error, but when I go open the dashboard for it using the run_dashboard option, the Leaderboard displays and indicates the correct number of records, but it doesn’t show any feedback columns/scores (just the version, number of records and the average latency).  Then if I try going to the Records option, I get this error:

```
File "C:\GitLab\eric.weist\data-science\.trulens.1.3.2.conda\Lib\site-packages\trulens\dashboard\pages\Records.py", line 607, in <module>

    records_main()

File "C:\GitLab\eric.weist\data-science\.trulens.1.3.2.conda\Lib\site-packages\trulens\dashboard\pages\Records.py", line 603, in records_main

    render_records(app_name)

File "C:\GitLab\user\data-science\.trulens.1.3.2.conda\Lib\site-packages\trulens\dashboard\pages\Records.py", line 549, in render_records

    df = _preprocess_df(

File "C:\GitLab\user\data-science\.trulens.1.3.2.conda\Lib\site-packages\trulens\dashboard\pages\Records.py", line 174, in _preprocess_df

    records_df["output"].str.encode("utf-8").str.decode("unicode-escape")

File "C:\GitLab\user\data-science\.trulens.1.3.2.conda\lib\site-packages\pandas\core\generic.py", line 6299, in __getattr__

    return object.__getattribute__(self, name)

File "C:\GitLab\user\data-science\.trulens.1.3.2.conda\lib\site-packages\pandas\core\accessor.py", line 224, in __get__

    accessor_obj = self._accessor(obj)

File "C:\GitLab\user\data-science\.trulens.1.3.2.conda\lib\site-packages\pandas\core\strings\accessor.py", line 191, in __init__

    self._inferred_dtype = self._validate(data)

File "C:\GitLab\user\data-science\.trulens.1.3.2.conda\lib\site-packages\pandas\core\strings\accessor.py", line 245, in _validate

    raise AttributeError("Can only use .str accessor with string values!")
```


**To Reproduce**
Which steps should someone take to run into the same error? A small, reproducible code example is useful here.

**Expected behavior**
A clear and concise description of what you expected to happen.

**Relevant Logs/Tracebacks**
Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. If the issue is related to the TruLens dashboard, please also include a screenshot.

**Environment:**

- OS: [e.g. MacOS, Windows]
- Python Version
- TruLens version
- Versions of other relevant installed libraries

**Additional context**
Add any other context about the problem here.
Repository: truera/trulens
Issue: [FEAT] Custom Logging Level Option for TruCustomApp
GitHub labels: bug

enhancement

When initializing a TruCustomApp, I'm unable to adjust the logging level. See verbose logs below for just instantiating the TruCustomApp. I'm also seeing similar levels of verbose feedback when calling the app

<img width="1394" alt="Image" src="https://github.com/user-attachments/assets/f349714a-620d-402e-8778-af831b76bd82" />




Fix merged and will be available in the next release!
Repository: truera/trulens
Issue: [BUG] Can't handle Japanese text inputs and outputs when storing records in Snowflake
Root cause: Validation Gap

**Bug Description**
I created RAG evaluation with TruLens and stored the log in Snowflake using `SnowflakeConnector`.
I followed the steps presented in the [Snowflake's quickstart](https://quickstarts.snowflake.com/guide/getting_started_with_llmops_using_snowflake_cortex_and_trulens/index.html#0).

![Image](https://github.com/user-attachments/assets/dfda85ed-4d05-46e8-8891-34193c1b0340)

I tried the RAG system in Japanese and found that The log 
**To Reproduce**
Follow the quickstart and try RAG with a prompt in Japanese

**Expected behavior**
The library should handle Japanese in proper format.

**Environment:**
- OS: MacOS
- Python 3.11
- TruLens-core, TruLens-eval 1.4.2



Fix will be released today - thanks @sfc-gh-dkurokawa for fixing!
Repository: truera/trulens
Issue: feedback_and_future_results is None [BUG]
Root cause: Validation Gap

**Bug Description**
I attempted to test the [langchain_quickstart.ipynb](https://colab.research.google.com/github/truera/trulens/blob/main/examples/quickstart/quickstart.ipynb) notebook, but I do not have an OpenAI key. Therefore, I decided to use Mistral as the model and provider, as it is supported in Langchain and Langchain listed in the providers list in your library. I made the following changes:

Used the free Mistral model:
`from langchain_mistralai import ChatMistralAI`
instead of:
`from langchain.chat_models import ChatOpenAI`

And used the MistralAI embeddings:
`from langchain_mistralai import MistralAIEmbeddings`
instead of:
`from langchain_openai import OpenAIEmbeddings`

All other configurations remained the same. 
However, I encountered a `TypeError: 'NoneType' object is not iterable`. 
This error occurred because `last_record.feedback_and_future_results is None`.


**To Reproduce**
You can reproduce this issue by checking the following notebook: [Langchain Mistral Quickstart Notebook](https://colab.research.google.com/drive/1LjUKy9Jz8nAumnnbjhBsoF1H42QBvVXF?usp=sharing)

**Relevant Logs/Tracebacks**

![Image](https://github.com/user-attachments/assets/d9a54db5-91c9-42e1-b5b3-c8602a12d93b)

**Environment:**
Google Colaboratory



Hi @DmitryRedko - looking into this.

Hey @DmitryRedko - thanks again for flagging this - we uncovered an issue related to using the Langchain provider & TruChain in combination and are working on a fix. You can track it here: https://github.com/truera/trulens/pull/1833

Hey @DmitryRedko - this is fixed by the above mentioned PR + https://github.com/truera/trulens/pull/1838

Planning to release this tomorrow.

Also I'd suggest making a small change to your notebook to use mistral-large instead of codestral for your LLM judge. Codestral seemed to be producing some issues in generating reliable feedback.

```python
feedback_llm = ChatMistralAI(
    model="mistral-large-latest",
    temperature=0,
)

# Initialize provider class
provider = Langchain(chain=feedback_llm)
```

Hi @sfc-gh-jreini! Thank you for your promptness and feedback! I'll try to use mistral-large-latest

Hey @DmitryRedko - sharing the update that 1.4.3 including the fixes mentioned is released!

https://github.com/truera/trulens/releases/tag/trulens-1.4.3
Repository: truera/trulens
Issue: Installation Failure of trulens Package on Hugging Face Spaces Due to Package Name Mismatch
Root cause: Environment Incompatibility

I'm encountering an issue when deploying my application on Hugging Face Spaces. The deployment fails during the installation of the trulens package from a specific branch and subdirectory of the repository. The error suggests a mismatch between the expected package name and the actual name defined in the package's metadata.

Steps to Reproduce:

Local Installation:

Using the following command, the package installs successfully on my local GPU environment:

`! pip install git+https://github.com/piotrm0/trulens.git@piotrm/jsonify-stack-size-protect#subdirectory=trulens_eval ` 


Deployment on Hugging Face Spaces:

Added the following line to requirements.txt:

**Add the following line to requirements.txt**
`git+https://github.com/piotrm0/trulens.git@piotrm/jsonify-stack-size-protect#egg=trulens&subdirectory=trulens_eval`


Upon deployment, the following import statements in app.py result in an error:
```
from trulens.apps.llamaindex import TruLlama
from trulens.providers.openai import OpenAI
from trulens.dashboard.display import get_feedback_result
from trulens.apps.llamaindex.guardrails import WithFeedbackFilterNodes
from trulens.core import TruSession
from trulens.core import Feedback
```

Observed Behavior:

During deployment, the following error is logged:

===== Application Startup at 2025-02-25 06:33:59 =====

Traceback (most recent call last):
  File "/home/user/app/app.py", line 19, in <module>
    from trulens.apps.llamaindex import TruLlama
ModuleNotFoundError: No module named 'trulens'
Traceback (most recent call last):
  File "/home/user/app/app.py", line 19, in <module>
    from trulens.apps.llamaindex import TruLlama
ModuleNotFoundError: No module named 'trulens'
```
Build File:
[Build.txt](https://github.com/user-attachments/files/18958694/Build.txt)


Hi @wahidur028 - what is the reason for using that branch? FYI there is no remote branch of that name, and this is what's likely causing your issue - perhaps you had pulled down a branch into your local that has since been deleted.

@sfc-gh-jreini Thanks for your response. As far as I remember, I encountered an issue earlier and was instructed to use this branch to resolve it. I've been using it since then.

Previous issue was:
`Getting RecursionError: maximum recursion depth exceeded while calling a Python object while creating tru_recorder. `

https://github.com/truera/trulens/issues/1100
https://github.com/truera/trulens/issues/1174

Thanks for linking the context - very helpful. Looks like these changes were merged a while back (apologies for not mentioning in original thread) so you should be fine to use the release versions 

Let me know if you have any issues with the release/pypi versions 

Closing this bc it was related to dev versions. If you run into new issues with the release versions, please open a new issue. Thanks!

@sfc-gh-jreini we are getting the same error https://github.com/truera/trulens/issues/1174

![Image](https://github.com/user-attachments/assets/f45971ad-93fd-44cb-8fe1-4163e862dfec)
Repository: truera/trulens
Issue: [BUG] How to get the app_id from the program to filter the records and leaderboard based on the app_id?
Root cause: Documentation Deficiency

**Bug Description**
What happened?

As far as I know, app_id is deterministically created using app_name and app_version. I dont have access to get it from any interface clearly.

I have checked the documentation and read through the source code.

```
for rec in virtual_records:

        print(rec.main_input, "-->", rec.main_output)

        for feedback, feedback_result in rec.wait_for_feedback_results().items():
            print("\t", feedback.name, feedback_result.result)
    app_id = rec.app_id

leaderboard = tru_session.get_leaderboard(app_ids=[app_id ])
records, feedbacks = tru_session.get_records_and_feedback(app_ids=[app_id ])
````

There should be someother way to get the app_id. Also, we are not allowed to create app_id to track it, right?
Could you please tell me the right way to get the app_id?


**Environment:**

- OS: [Windows]
- Python Version 3.12
trulens==1.4.6
trulens-providers-openai==1.4.0

**Additional context**
Add any other context about the problem here.


Hey @Sudhakar17 - what's the issue with using the code above, i.e. `record.app_id` ?

Hey @Sudhakar17 - please let me know your preferred solution, i.e. if you can use `record.app_id`
Repository: truera/trulens
Issue: [BUG] Please remove poetry from list of dependencies
Root cause: Configuration Error

Hello,

I noticed that _poetry_ is listed as regular dependency in [pyproject.toml](https://github.com/truera/trulens/blob/main/pyproject.toml#L30).

This is causing downstream issues in dependency resolution on our side (as poetry<2 depends on outdated packages).

The restriction to poetry<2 was added in [this](https://github.com/truera/trulens/pull/1709) MR, with the stated reason being failing Python 3.8 tests. Python 3.8 support ended 6 months ago (see [here](https://endoflife.date/python)). Further, according to the pyproject.toml file linked above, Python 3.8 is not supported by Trulens anyway.

Therefore, please remove the restriction to poetry<2 from pyproject.toml.

If the restriction to poetry <2 absolutely needs to be in the file at the moment, I strongly recommend to move it to dev dependencies, as it's only required for development purposes (and according to the original MRs description, it's only added due to failing tests). This way, it would not affect dependency resolution in projects that import Trulens.

We temporarily solved this issue on our side by downgrading from Trulens 1.4.9 to 1.2.11, but this is not a long-term solution.


@sfc-gh-chu has an open PR to solve this - thanks for reporting @drooms-sandrus !

https://github.com/truera/trulens/pull/1930
Repository: truera/trulens
Issue: RuntimeError: cannot join current thread
Root cause: Resource Mishandling

Hi, Getting this error at the langchain provider instantiation.

```
provider = Langchain(chain=self._get_llm())
Exception ignored in: <function App.del at 0x000002E063315800>Traceback (most recent call last):  File ".venv\Lib\site-packages\trulens\core\app.py", line 588, in del    self.stop_evaluator()  File ".venv\Lib\site-packages\trulens\core\app.py", line 2018, in stop_evaluator    self._evaluator.stop_evaluator()  File ".venv\Lib\site-packages\trulens\core\utils\evaluator.py", line 206, in stop_evaluator    self._thread.join(timeout=300)  File "C:\Users\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1109, in join    raise RuntimeError("cannot join current thread")RuntimeError: cannot join current thread
```

@poojatambe sorry for the very slow response on this. Can you try upgrading your trulens versions and let us know if you're still experiencing this issue? thanks
Repository: truera/trulens
Issue: [BUG] Bedrock: TypeError: cannot unpack non-iterable float object
Root cause: Interface Contract Mismatch

**Bug Description**
Getting error TypeError: cannot unpack non-iterable float object while using Bedrock provider's `groundedness_measure_with_cot_reasons` to calculate groundedness.

**To Reproduce**
Setup a notebook, install truelens dependencies, setup AWS credetials. Then run below code in a cell
```py
from trulens.providers.bedrock import Bedrock

provider = Bedrock(
    model_id="anthropic.claude-3-5-haiku-20241022-v1:0",
    region_name="us-west-2"
)

contexts = ["Starbucks represents the second wave of coffee culture in the US."]
response = "Starbucks represents the second wave of coffee culture."
question = "Who represents the second wave of coffee culture in US."

try:
    score = provider.groundedness_measure_with_cot_reasons(
        source=contexts,
        statement=response
    )

except Exception as e:
    print(e)
    print(f"Groundedness failed: {e}")
```

**Expected Behavior**
It should return the Groundedness score as a float.

**Relevant Logs/Tracebacks**
```
score = self.provider.groundedness_measure_with_cot_reasons(\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \".venv/lib/python3.11/site-packages/trulens/feedback/llm_provider.py\", line 2133, in groundedness_measure_with_cot_reasons\n    results.append(future.result())\n                   ^^^^^^^^^^^^^^^\n  File \"/opt/homebrew/Cellar/python@3.11/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py\", line 449, in result\n    return self.__get_result()\n           ^^^^^^^^^^^^^^^^^^^\n  File \"/opt/homebrew/Cellar/python@3.11/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py\", line 401, in __get_result\n    raise self._exception\n  File \"/opt/homebrew/Cellar/python@3.11/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \".venv/lib/python3.11/site-packages/trulens/core/utils/python.py\", line 630, in _future_target_wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \".venv/lib/python3.11/site-packages/trulens/feedback/llm_provider.py\", line 2108, in evaluate_hypothesis\n    score, reason = self.generate_score_and_reasons(\n    ^^^^^^^^^^^^^\nTypeError: cannot unpack non-iterable float object"}
```

**Environment:**

- OS: MacOS
- Python Version 3.11
- TruLens version 2.2.0



Hi @azeemirshad - @sfc-gh-dkurokawa fixed this issue in https://github.com/truera/trulens/pull/2193 and released it in TruLens 2.3.0. Let us know if you have any continued issues. Thanks!
Repository: truera/trulens
Issue: [STYLE] Inconsistent import style across codebase
GitHub labels: enhancement

Large portions of the codebase doesn't follow the proper styling for imports as outlined by the 
https://www.trulens.org/contributing/standards/ .I would like to get the ball rolling by first fixing the import formatting in src/providers/bedrock/trulens/bedrock folder. And moving outwards from there. Here is the current code snippet I am trying to improve.

import inspect
import logging
import pprint
from typing import Any, Callable, Iterable, Optional

import boto3
from botocore.client import ClientCreator
import pydantic
from trulens.core.feedback import endpoint as core_endpoint
from trulens.core.utils import python as python_utils

Here are some issues I noticed currently in the way modules are being currently imported. 
1.from trulens.core.feedback import endpoint as core_endpoint should be imported as a module, not directly.
2.from trulens.core.utils import python as python_utils has the same issue, should follow the _utils naming pattern.
3.import json appears inside a function of the file it is probably better practise to keep imports at the top unless absolutely necessary.
I would love to work on this issue as my first contribution to this project if it was ok.Let me know if there is something I need to be aware of.Thanks in advance for your time.


Hi @Labreo - this is a great first contribution. Please let me know if you need any assistance. Thanks!
Repository: truera/trulens
Issue: [BUG]
GitHub labels: bug

**Bug Description**
What happened?

**To Reproduce**
Which steps should someone take to run into the same error? A small, reproducible code example is useful here.

**Expected Behavior**
A clear and concise description of what you expected to happen.

**Relevant Logs/Tracebacks**
Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. If the issue is related to the TruLens dashboard, please also include a screenshot.

**Environment:**

- OS: [e.g. MacOS, Windows]
- Python Version
- TruLens version
- Versions of other relevant installed libraries

**Additional Context**
Add any other context about the problem here.


Hi @fazankhanbuzdar-create, please add details following the provided template for a bug you wish to report!

Closing this until details are added
Repository: truera/trulens
Issue: [Documentation] TruApp documentation page missing
GitHub labels: documentation

See https://www.trulens.org/reference/apps/

TruCustomApp page refers to the missing TruApp page, see https://www.trulens.org/reference/trulens/apps/custom/

<!-- Greeting -->
Hi @rvernica! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the trulens team.

<!-- Answer -->
There isn't a dedicated documentation page for TruApp at the referenced URLs, and the TruCustomApp docs currently point to a missing or upcoming page. The docs indicate that TruCustomApp is being deprecated in favor of the standardized TruApp framework, but they don't provide migration details or direct documentation for TruApp itself. 

However, you can find practical usage examples of TruApp in GitHub issues and discussions. For example, to use TruApp, you typically import it from `trulens.apps.app`, and instantiate it with your app, feedbacks, and a configured DBConnector:

```python
from trulens.core import DBConnector
from trulens.apps.app import TruApp

connector = DBConnector(...)  # configure your connector here
tru_app = TruApp(
    app=my_app,
    feedbacks=[...],
    connector=connector
)
```

This pattern is shown in [community discussions](https://github.com/truera/trulens/discussions/2060) and [issue examples](https://github.com/truera/trulens/issues/2207). While there isn't a formal migration guide, these examples reflect the current usage and transition to TruApp.


<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*

---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/50f1ca73-55de-4aa6-bb5a-eb22c3608a4a?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/50f1ca73-55de-4aa6-bb5a-eb22c3608a4a?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/50f1ca73-55de-4aa6-bb5a-eb22c3608a4a?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/50f1ca73-55de-4aa6-bb5a-eb22c3608a4a?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/50f1ca73-55de-4aa6-bb5a-eb22c3608a4a?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/50f1ca73-55de-4aa6-bb5a-eb22c3608a4a?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/50f1ca73-55de-4aa6-bb5a-eb22c3608a4a?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/e16d1378-a540-46b6-bd0e-816c493d2aee/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/truera/trulens/issues/2276)

Thanks for raising @rvernica - we'll get this updated!

Hey @rvernica - the fix is merged, will redeploy docs soon! Thanks again for raising the issue.
Repository: truera/trulens
Issue: [BUG]
Root cause: Interface Contract Mismatch

**Bug Description**
I'm using TruLens version 2.4.1 and import feedback v1. 
from trulens.core import Feedback
f_relevance = (     Feedback(         provider.relevance_with_cot_reasons,         name="Relevance Score"     ).on_input_output() ).
I'm facing the error: WARNING:trulens.feedback.computer:feedback_name=Maliciousness, record=bcde06ab-9499-497f-8f9b-3b5431031f90, span_group=None had an error during computation:
CriteriaOutputSpaceMixin.validate_criteria_and_output_space() missing 1 required positional argument: 'custom_instructions'
WARNING:trulens.feedback.computer:feedback_name=Helpfulness, record=bcde06ab-9499-497f-8f9b-3b5431031f90, span_group=None had an error during computation:
CriteriaOutputSpaceMixin.validate_criteria_and_output_space() missing 1 required positional argument: 'custom_instructions
**To Reproduce**


**Expected Behavior**
The feedback function should be run correctly and show metric on dashboard

**Environment:**
- OS: Windows
- Python Version: 3.12
- TruLens version: 2.4.2
- Versions of other relevant installed libraries


Hi, thanks for flagging this! I just pushed a [PR](https://github.com/truera/trulens/pull/2284) to fix this bug, so this should be resolved shortly! 

We just pushed the fix to the repo, please let us know if you encounter any other issues! Thanks :)
Repository: truera/trulens
Issue: trulens_eval.feedback.OpenAI with Azure
Root cause: Unimplemented Feature Gap

Hi there 👋 

I am using from `trulens_eval.feedback.OpenAI` but I need to pass extra args to use with azure, I don't understand the usage of  `OpenAIEndpoint` since I cannot see extra args to setup openai, usually you can do something like

```python
openai.api_type = "azure"
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_version = "2023-03-15-preview"
openai.api_key = os.getenv("OPENAI_API_KEY")
```

Not sure how to do it with trulens

Thanks a lot

Cheers,

Fra


Hi Fra,
We're looking into how to set Azure parameters in the openai API. We should be able to expose it in the OpenAI class soon.

I've made a PR #242 

merged, this should make the next release (in a day or so)