Repository: truera/trulens
Issue: UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 2: illegal multibyte sequence
Root cause: External Dependency Breakage

**Bug Description**
I am trying to use TruLens to evaluate metrics for my RAG (Retrieval-Augmented Generation) program. My program uses Chinese text, and I'm encountering encoding errors somewhere within TruLens. Could you please advise on how to modify this?

**To Reproduce**
# -*- coding: utf-8 -*-
from utils import *
from langchain.chains import LLMChain

query = "如何确保不同部门之间的协调合作以有效推进大规模设备更新计划？"

# 1. 创建LLM链
vdb = Chroma(
    persist_directory=os.path.join(PROJECT_PATH, './data/.nnew_document_db'),
    embedding_function=get_embeddings_model(),
)
# 召回并过滤文档
documents = vdb.similarity_search_with_relevance_scores(query, k=5)
query_result = [doc[0].page_content for doc in documents if doc[1] > 0.5]
if len(query_result) == 0:
    policy_result = '未检索到相关的政策信息。'
    print(policy_result)
# 填充提示词并总结答案
prompt = PromptTemplate.from_template(RETRIVAL_PROMPT_TPL_01)
retrival_chain = LLMChain(
    llm = get_llm_model(),
    prompt = prompt,
    verbose = VERBOSE,
)
inputs = {
    'query': query,
    'query_result': '\n\n'.join(query_result) if len(query_result) else '没有查到',
}

# 创建TruLens反馈函数
from trulens_eval import TruChain, Tru, Feedback, OpenAI as TruOpenAI

tru_openai = TruOpenAI()
relevance = Feedback(tru_openai.relevance).on_input_output()
sentiment = Feedback(tru_openai.sentiment).on_output()

# 使用TruChain包装LangChain应用
truchain = TruChain(
    retrival_chain,
    app_id="LangChain_Assistant",
    feedbacks=[relevance, sentiment]
)

# 使用应用并记录结果
response = truchain.with_(retrival_chain, inputs)

# 运行仪表板查看结果
tru = Tru()
tru.run_dashboard(port=8502)

**Expected behavior**
A clear and concise description of what you expected to happen.

**Relevant Logs/Tracebacks**
Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. If the issue is related to the TruLens dashboard, please also include a screenshot.

**Environment:**

- Windows 10
- Python Version：3.10.9
- TruLens version：0.24.1



<!-- Answer -->
I found a similar closed issue that might be relevant to your problem. The issue titled "[Poor readability of Chinese in Records section](https://github.com/truera/trulens/issues/629)" was resolved by a pull request that escaped Unicode for the records table <sup>[[1]](https://github.com/truera/trulens/issues/629)</sup>.

Additionally, TruLens provides utility functions for encoding conversion or validation, which might help you handle encoding errors. Specifically, the `trulens.core.utils.json` module includes functions such as `encode_httpx_url` and `encode_openai_timeout`. The `trulens.core.utils.serial` module offers various classes and methods for serialization and validation, such as `JSONized`, `Step`, `GetAttribute`, `GetIndex`, `GetItem`, and `Lens` <sup>[[2]](https://github.com/truera/trulens/blob/main/src/trulens_eval/trulens_eval/utils/json.py)</sup><sup>[[3]](https://github.com/truera/trulens/blob/main/src/core/trulens/core/utils/serial.py)</sup>.

You can try using these utilities to handle encoding issues in your code. Here is an example of how you might use one of these functions:

```python
from trulens.core.utils.json import encode_httpx_url

# Example usage
encoded_url = encode_httpx_url("http://example.com/如何确保不同部门之间的协调合作以有效推进大规模设备更新计划？")
print(encoded_url)
```

This might help you identify and resolve the encoding errors you are encountering.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I didn't understand. Please provide the complete code.

<!-- Answer -->
To help you handle the `UnicodeDecodeError` in your code, you can use the utility functions provided by TruLens for encoding and decoding. Below is a complete example of how you might modify your code to handle encoding issues:

```python
# -*- coding: utf-8 -*-
import os
from utils import *
from langchain.chains import LLMChain
from trulens.core.utils.json import json_str_of_obj

query = "如何确保不同部门之间的协调合作以有效推进大规模设备更新计划？"

# 1. 创建LLM链
vdb = Chroma(
    persist_directory=os.path.join(PROJECT_PATH, './data/.nnew_document_db'),
    embedding_function=get_embeddings_model(),
)
# 召回并过滤文档
documents = vdb.similarity_search_with_relevance_scores(query, k=5)
query_result = [doc[0].page_content for doc in documents if doc[1] > 0.5]
if len(query_result) == 0:
    policy_result = '未检索到相关的政策信息。'
    print(policy_result)
# 填充提示词并总结答案
prompt = PromptTemplate.from_template(RETRIVAL_PROMPT_TPL_01)
retrival_chain = LLMChain(
    llm = get_llm_model(),
    prompt = prompt,
    verbose = VERBOSE,
)
inputs = {
    'query': query,
    'query_result': '\n\n'.join(query_result) if len(query_result) else '没有查到',
}

# 创建TruLens反馈函数
from trulens_eval import TruChain, Tru, Feedback, OpenAI as TruOpenAI

tru_openai = TruOpenAI()
relevance = Feedback(tru_openai.relevance).on_input_output()
sentiment = Feedback(tru_openai.sentiment).on_output()

# 使用TruChain包装LangChain应用
truchain = TruChain(
    retrival_chain,
    app_id="LangChain_Assistant",
    feedbacks=[relevance, sentiment]
)

# 使用应用并记录结果
response = truchain.with_(retrival_chain, inputs)

# 处理编码问题
encoded_response = json_str_of_obj(response)
print(encoded_response)

# 运行仪表板查看结果
tru = Tru()
tru.run_dashboard(port=8502)
```

In this example, the `json_str_of_obj` function from the `trulens.core.utils.json` module is used to handle encoding issues by converting the response to a JSON string. This should help you avoid `UnicodeDecodeError` when dealing with Chinese text or other non-ASCII characters <sup>[[1]](https://github.com/truera/trulens/blob/main/src/core/trulens/core/utils/json.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/src/trulens_eval/trulens_eval/utils/json.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@beatG123 The encoding issue should be resolved in a later release. Could you try upgrading to `trulens-eval==0.33.1` or  `trulens==1.0.2`? 

> @beatG123 The encoding issue should be resolved in a later release. Could you try upgrading to `trulens-eval==0.33.1` or `trulens==1.0.2`?

Sure, I'll try it. If you have a better solution, please tell me. Thank you.

@beatG123 did this help?

@beatG123 closing this for now, pls reopen if you need help again