Repository: truera/trulens
Issue: Evaluate local deployment finetuned LLM, LLM of openai interface as feedback provider, RunTimeError occurred
Root cause: Configuration Error

```
from langchain_openai import ChatOpenAI
import pandas as pd

glm4_base_client = ChatOpenAI(model="glm-4v-9b",
                          api_key="your_api_key",
                          base_url="http://0.0.0.0:8000/v1/",
                          )
glm4_finetune_client = ChatOpenAI(model="glm-4v-9b",
                          api_key="your_api_key",
                          base_url="http://0.0.0.0:8001/v1/",
                          )

test_dataset = pd.read_csv("test_output.csv")
test_dataset.rename(columns={"Input": "query", "GT Response": "response"}, inplace=True)
golden_set = test_dataset[["query", "response"]].to_dict(orient="records")

feedback_provider = OpenAI(
                           model_engine="qwen-max",
                           endpoint="https://dashscope.aliyuncs.com/compatible-mode/v1",
                           api_key="******",
                           )

ground_truth = GroundTruthAgreement(golden_set, provider=feedback_provider)

f_groundtruth = (
    Feedback(ground_truth.agreement_measure, name="Ground Truth Agreement")
    .on_input_output()
)

f_answer_relevance = (
    Feedback(feedback_provider.relevance_with_cot_reasons, name="Answer Relevance")
    .on_input_output()
)

def base_model(query):
    response = glm4_base_client.invoke(query).content
    return response

def finetuned_model(query):
    response = glm4_finetune_client.invoke(query).content
    return response

base_recorder = TruBasicApp(
    base_model,
    app_name="LLM",
    app_version="base",
    feedbacks=[
        f_groundtruth,
        f_answer_relevance,
    ],
)

finetuned_recorder = TruBasicApp(
    finetuned_model,
    app_name="LLM",
    app_version="finetuned",
    feedbacks=[
        f_groundtruth,
        f_answer_relevance,
    ],
)

for i in range(len(test_dataset)):
    with base_recorder as recording:
        base_recorder.app(test_dataset["query"][i])

    with finetuned_recorder as recording:
        finetuned_recorder.app(test_dataset["query"][i])
```

ERROR AS FOLLOWS:
```
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
Feedback Function exception caught: Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 895, in run
    mod_base_endpoint.Endpoint.track_all_costs_tally(
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 524, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 504, in track_all_costs
    return Endpoint._track_costs(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 601, in _track_costs
    result: T = __func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/feedback/groundtruth.py", line 223, in agreement_measure
    ground_truth_response = self._find_response(prompt)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/feedback/groundtruth.py", line 161, in _find_response
    responses = [
                ^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/feedback/groundtruth.py", line 162, in <listcomp>
    qr["expected_response"]
    ~~^^^^^^^^^^^^^^^^^^^^^
KeyError: 'expected_response'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 910, in run
    raise RuntimeError(
RuntimeError: Evaluation of Ground Truth Agreement failed on inputs: 
{'prompt': '波纹黄尺蛾在广西南宁一带哪个时期发生数量较多？',
 'response': '波纹黄尺蛾在广西南宁一带通常在春季和夏季的温暖季节发生数量较多。具体来说，主要集中在4月至6月期间，这段时间内气候适宜，有利于其繁殖和活动。'}.

```
I try to use local deployment llm (base and finetuned), feedback provider use openai llm with endpoint and api_key, and evaluate in my dataset. 

I want to know how to make sure that the feedback provider can be called normally, and how to fix the error with my part of the code causing an error(reference [)](https://github.com/truera/trulens/blob/main/examples/expositional/models/bedrock/bedrock_finetuning_experiments.ipynb) to build my code. 

Any helpful help would be greatly appreciated.

<!-- Answer -->
To ensure that the feedback provider can be called normally and fix the `RuntimeError` related to the `expected_response` KeyError, you need to ensure that the `expected_response` key is properly handled in your code. Here is how you can modify your code to handle this key appropriately:

1. **Modify the `golden_set` dictionary**: Ensure that each item in the `golden_set` dictionary includes the `expected_response` key. If your dataset does not have this key, you can set it to `None`.

2. **Update the `_create_chat_completion` method**: Add a check to handle the `expected_response` key.

Here is an example of how you can modify your code:

### Step 1: Modify the `golden_set` Dictionary
Ensure that each item in the `golden_set` dictionary includes the `expected_response` key:

```python
test_dataset = pd.read_csv("test_output.csv")
test_dataset.rename(columns={"Input": "query", "GT Response": "response"}, inplace=True)
test_dataset["expected_response"] = None  # Add this line to include the expected_response key
golden_set = test_dataset[["query", "response", "expected_response"]].to_dict(orient="records")
```

### Step 2: Update the `_create_chat_completion` Method
Modify the `_create_chat_completion` method to handle the `expected_response` key:

```python
def _create_chat_completion(
    self,
    prompt: Optional[str] = None,
    messages: Optional[Sequence[Dict]] = None,
    **kwargs
) -> str:
    if 'model' not in kwargs:
        kwargs['model'] = self.model_engine

    if 'temperature' not in kwargs:
        kwargs['temperature'] = 0.0

    if 'seed' not in kwargs:
        kwargs['seed'] = 123

    # Handle the 'expected_response' key
    if 'expected_response' not in kwargs:
        kwargs['expected_response'] = None

    if messages is not None:
        completion = self.endpoint.client.chat.completions.create(
            messages=messages, **kwargs
        )

    elif prompt is not None:
        completion = self.endpoint.client.chat.completions.create(
            messages=[{
                "role": "system",
                "content": prompt
            }], **kwargs
        )

    else:
        raise ValueError("`prompt` or `messages` must be specified.")

    return completion.choices[0].message.content
```

### Step 3: Ensure `GroundTruthAgreement` Handles `expected_response`
Ensure that the `GroundTruthAgreement` class handles the `expected_response` key properly. The `expected_response` key might be missing (i.e., set to `None`) in ground truth datasets, as explicitly mentioned in the class definition.

```python
expected_response: Optional[str] = (
    None  # expected response can be empty in GT datasets
)
```

The `_find_response` method in the `GroundTruthAgreement` class should handle cases where the `expected_response` key might be missing:

```python
def _find_response(self, prompt: str) -> Optional[str]:
    if self.ground_truth_imp is not None:
        return self.ground_truth_imp(prompt)

    responses = [
        qr["response"]
        for qr in self.ground_truth
        if qr["query"] == prompt
    ]
    if responses:
        return responses[0]
    else:
        return None
```

By adding the check for the `expected_response` key and ensuring that your `golden_set` dictionary includes this key, you can ensure that the feedback provider can be called normally and fix the `RuntimeError` related to the `expected_response` KeyError <sup>[[1]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/openai.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/bedrock.py)</sup><sup>[[3]](https://github.com/truera/trulens/blob/main/src/core/trulens/core/schema/groundtruth.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu 
OK, I followed your advice, add `test_dataset["expected_response"] = None` and `golden_set = test_dataset[["query", "response", "expected_response"]].to_dict(orient="records")`. But no other code changes were suggested because the code I provided did not include the part of the code you mentioned
```
test_dataset = pd.read_csv("test_output.csv")
test_dataset.rename(columns={"Input": "query", "GT Response": "response"}, inplace=True)
test_dataset["expected_response"] = None  # Add this line to include the expected_response key
golden_set = test_dataset[["query", "response", "expected_response"]].to_dict(orient="records")
```
```
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=3.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=3.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=2.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=3.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=2.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=3.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=1.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=2.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=1.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=2.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=3.
Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=1.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 895, in run
    mod_base_endpoint.Endpoint.track_all_costs_tally(
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 524, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 504, in track_all_costs
    return Endpoint._track_costs(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 601, in _track_costs
    result: T = __func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 565, in relevance_with_cot_reasons
    return self.generate_score_and_reasons(system_prompt, user_prompt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 218, in generate_score_and_reasons
    response = self.endpoint.run_in_pace(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 323, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint openai request failed 4 time(s): 
	Connection error.
	Connection error.
	Connection error.
	Connection error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 910, in run
    raise RuntimeError(
RuntimeError: Evaluation of Answer Relevance failed on inputs: 
{'prompt': '为什么需要促使新梢抽发整齐健壮？',
 'response': '促使新梢抽发整齐健壮对于果树生长和产量有重要意义，具体原因如下：\n'
             '\n'
             '1. **均匀光照**：整齐的.

Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=2.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=1.
openai request failed <class 'openai.APIConnectionError'>=Connection error.. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 895, in run
    mod_base_endpoint.Endpoint.track_all_costs_tally(
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 524, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 504, in track_all_costs
    return Endpoint._track_costs(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 601, in _track_costs
    result: T = __func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 565, in relevance_with_cot_reasons
    return self.generate_score_and_reasons(system_prompt, user_prompt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 218, in generate_score_and_reasons
    response = self.endpoint.run_in_pace(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 323, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint openai request failed 4 time(s): 
	Connection error.
	Connection error.
	Connection error.
	Connection error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 910, in run
    raise RuntimeError(
RuntimeError: Evaluation of Answer Relevance failed on inputs: 
{'prompt': '蠹虫主要危害荔枝的哪个部位？',
 'response': '**1. 荔枝嫩梢**\n'
             '蠹虫主要以荔枝树的嫩梢为食，它们会咬食嫩梢的顶端部分，导致新梢生长受阻。\n'
             '\n.

Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
```
but now the ERROR are reported as above. How can I fix it?

<!-- Answer -->
To fix the `APIConnectionError` and the issue with the `OpenAICallback` class when using the feedback provider in your code, you should update your imports to use the new module as the current one is deprecated.

Replace:
```python
from trulens_eval.trulens_eval.feedback.provider.endpoint.openai import OpenAICallback
```

With:
```python
from trulens.providers.openai.endpoint import OpenAICallback
```

Additionally, ensure that you are correctly initializing the `OpenAI` class from the `trulens.providers.openai.provider` module. Here is an example of how to create an `OpenAI` provider:

```python
from trulens.providers.openai.provider import OpenAI
openai_provider = OpenAI()
```

This change should resolve the issues related to the `APIConnectionError` and the `OpenAICallback` class <sup>[[1]](https://github.com/truera/trulens/blob/main/src/trulens_eval/trulens_eval/feedback/provider/endpoint/openai.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/openai.py)</sup><sup>[[3]](https://github.com/truera/trulens/blob/main/src/trulens_eval/trulens_eval/feedback/provider/openai.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
If I use 
```
from trulens.providers.openai.provider import OpenAI
openai_provider = OpenAI()
```
How do I set it up model_name, base_url and apy_key to call the llm?

<!-- Answer -->
To set up `model_name`, `base_url`, and `api_key` when calling the LLM using `from trulens.providers.openai.provider import OpenAI` and `openai_provider = OpenAI()`, you can pass these parameters as keyword arguments when initializing the `OpenAI` class. Here is an example:

```python
from trulens.providers.openai import OpenAI

openai_provider = OpenAI(
    model_engine="your_model_name",  # Set your model name here
    base_url="your_base_url",        # Set your base URL here
    api_key="your_api_key"           # Set your API key here
)
```

Replace `"your_model_name"`, `"your_base_url"`, and `"your_api_key"` with your actual model name, base URL, and API key respectively. This will configure the `OpenAI` class to use the specified model, endpoint, and API key for making requests <sup>[[1]](https://github.com/truera/trulens/blob/main/src/providers/openai/trulens/providers/openai/provider.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/openai.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
Now I pass parameters when iitializing OpenAI class. Test sample as follow:
```
from trulens.providers.openai import OpenAI
feedback_provider = OpenAI(
                           model_engine="glm-4v-9b",
                           endpoint="http://0.0.0.0:8000/v1/",
                           api_key="your_api_key",
                           )
```
I use the local deployment llm, "http://0.0.0.0:8000/v1/"  is what I published through api server. 
Which code methods should I call to test whether feedback_provider can call my locally deployed llm?

<!-- Answer -->
To test whether the `feedback_provider` can call your locally deployed LLM using the `OpenAI` class with the specified parameters, you should call the `test_llmcompletion` method. This method checks that the feedback functions produce a value in the 0-1 range and ensures that chain of thought reasons feedback functions produce criteria and supporting evidence.

Here is the relevant code snippet:

```python
@optional_test
def test_llmcompletion(self):
    """
    Check that LLMProvider feedback functions produce a value in the 0-1
    range only. Also check to make sure chain of thought reasons feedback functions
    produce criteria and supporting evidence. Only checks each feedback function
    once for each model.
    """
    from trulens.providers.openai import OpenAI

    models = ["gpt-3.5-turbo"]
    provider_models = [
        (OpenAI(model_engine=model), model) for model in models
    ]
    for provider, model in provider_models:
        with self.subTest(f"{provider.__class__.__name__}-{model}"):
            tests = get_llmprovider_tests(provider)
            funcs = set()

            for imp, args, _ in tests:
                # only one test per feedback function per model:
                if (imp, model) in funcs:
                    continue
                funcs.add((imp, model))
```

You will need to modify the `models` list to include your specific model engine and endpoint:

```python
models = ["glm-4v-9b"]
provider_models = [
    (OpenAI(model_engine=model, endpoint="http://0.0.0.0:8000/v1/", api_key="your_api_key"), model) for model in models
]
```

This will configure the `OpenAI` class to use your locally deployed LLM <sup>[[1]](https://github.com/truera/trulens/blob/main/tests/e2e/test_providers.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
```
from trulens.providers.openai import OpenAI
feedback_provider = OpenAI(
                           model_engine="glm-4v-9b",
                           endpoint="http://0.0.0.0:8000/v1/",
                           api_key="your_api_key",
                           )
```
Can you test whether feedback_provider works on my code base？

<!-- Answer -->
To test whether `feedback_provider` works on your code base using the provided setup, you can create an instance of the `OpenAI` class and call one of its methods. Here is an example of how you can do this:

```python
from trulens.providers.openai import OpenAI

# Initialize the feedback provider
feedback_provider = OpenAI(
    model_engine="glm-4v-9b",
    endpoint="http://0.0.0.0:8000/v1/",
    api_key="your_api_key",
)

# Example usage: create a chat completion
try:
    response = feedback_provider._create_chat_completion(prompt="Hello, how are you?")
    print("Chat completion response:", response)
except Exception as e:
    print("Error:", e)
```

This code initializes the `OpenAI` feedback provider with the specified parameters and then attempts to create a chat completion using the `_create_chat_completion` method. If the setup is correct and the endpoint is working, it will print the response from the chat completion. If there is an error, it will print the error message.

Additionally, you can run more comprehensive tests similar to those found in the `trulens_eval/tests/e2e/test_providers.py` file. For example, you can test various moderation and feedback functions to ensure they produce values in the expected range:

```python
from trulens.providers.openai import OpenAI
from trulens_eval.tests.e2e.test_providers import get_openai_tests, get_llmprovider_tests

# Initialize the feedback provider
feedback_provider = OpenAI(
    model_engine="glm-4v-9b",
    endpoint="http://0.0.0.0:8000/v1/",
    api_key="your_api_key",
)

# Run OpenAI moderation tests
tests = get_openai_tests(feedback_provider)
for imp, args, expected in tests:
    try:
        actual = imp(**args)
        assert 0.0 <= actual <= 1.0, f"Test failed for {imp.__name__} with args {args}"
        print(f"Test passed for {imp.__name__} with args {args}")
    except Exception as e:
        print(f"Error in {imp.__name__} with args {args}: {e}")

# Run LLMProvider feedback tests
tests = get_llmprovider_tests(feedback_provider)
for imp, args, expected in tests:
    try:
        actual = imp(**args)
        assert 0.0 <= actual <= 1.0, f"Test failed for {imp.__name__} with args {args}"
        print(f"Test passed for {imp.__name__} with args {args}")
    except Exception as e:
        print(f"Error in {imp.__name__} with args {args}: {e}")
```

This code will run a series of tests on the `feedback_provider` to ensure that various moderation and feedback functions are working correctly and producing values within the expected range <sup>[[1]](https://github.com/truera/trulens/blob/main/trulens_eval/tests/e2e/test_providers.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
thank you for the test code:
from trulens.providers.openai import OpenAI

```
# Initialize the feedback provider
feedback_provider = OpenAI(
    model_engine="glm-4v-9b",
    endpoint="http://0.0.0.0:8000/v1/",
    api_key="your_api_key",
)

# Example usage: create a chat completion
try:
    response = feedback_provider._create_chat_completion(prompt="Hello, how are you?")
    print("Chat completion response:", response)
except Exception as e:
    print("Error:", e)
```
unfortunatlly, it return `Error: Connection error.`
I think its not my lllm's problem, beceuse when I run:
```
from langchain_openai import ChatOpenAI

glm4_base_client = ChatOpenAI(model="glm-4v-9b",
                          api_key="your_api_key",
                          base_url="http://0.0.0.0:8000/v1/",
                          )
glm4_base_client.invoke("hello")
```
llm can respond normally. What do you think?

<!-- Answer -->
The `Connection error` issue with the `OpenAI` feedback provider might be due to differences in how the `OpenAI` class and the `ChatOpenAI` client handle the connection and API requests. Here are a few steps to troubleshoot and resolve this issue:

1. **Check Endpoint Configuration**:
   Ensure that the endpoint URL and API key are correctly configured in the `OpenAI` class. Double-check for any typos or incorrect values.

2. **Verify Dependencies**:
   Ensure that all required dependencies are installed and up-to-date. The `OpenAI` class relies on the `openai` and `httpx` packages. Make sure these packages are installed with the correct versions:
   ```sh
   pip install openai>=1.1.1,<2 httpx>=0.23.3
   ```

3. **Debugging Connection**:
   Add logging to the `OpenAI` class to capture more details about the connection error. This can help identify the root cause of the issue.

4. **Compare Implementations**:
   Compare the implementation of the `ChatOpenAI` client and the `OpenAI` class to identify any differences in how they handle the connection and API requests.

Here is an example of how you can add logging to the `OpenAI` class to capture more details about the connection error:

```python
import logging
from trulens.providers.openai import OpenAI

# Configure logging
logging.basicConfig(level=logging.DEBUG)

# Initialize the feedback provider
feedback_provider = OpenAI(
    model_engine="glm-4v-9b",
    endpoint="http://0.0.0.0:8000/v1/",
    api_key="your_api_key",
)

# Example usage: create a chat completion
try:
    response = feedback_provider._create_chat_completion(prompt="Hello, how are you?")
    print("Chat completion response:", response)
except Exception as e:
    logging.error("Error: %s", e)
```

Additionally, you can compare the `ChatOpenAI` client and the `OpenAI` class implementations to identify any differences. The `ChatOpenAI` client might be using a different configuration or handling the connection differently, which could explain why it works while the `OpenAI` class does not.

If the issue persists, you might want to consider using the `ChatOpenAI` client directly for your feedback provider, or further investigate the `OpenAIEndpoint` and `OpenAIClient` classes used by the `OpenAI` feedback provider to ensure they are correctly configured.

For reference, here is how the `OpenAIEndpoint` and `OpenAIClient` classes are configured:

### `OpenAIEndpoint` Class
```python
class OpenAIEndpoint(Endpoint):
    def __init__(
        self,
        name: str = "openai",
        client: Optional[
            Union[oai.OpenAI, oai.AzureOpenAI, OpenAIClient]
        ] = None,
        rpm: Optional[int] = None,
        pace: Optional[Pace] = None,
        **kwargs: dict,
    ):
        # Initialization logic
```

### `OpenAIClient` Class
```python
class OpenAIClient(SerialModel):
    def __init__(
        self,
        client: Optional[Union[oai.OpenAI, oai.AzureOpenAI]] = None,
        client_cls: Optional[Class] = None,
        client_kwargs: Optional[dict] = None,
    ):
        # Initialization logic
```

Ensure that these classes are correctly configured and that the `OpenAI` class is using them appropriately <sup>[[1]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/openai.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/src/providers/openai/trulens/providers/openai/provider.py)</sup><sup>[[3]](https://github.com/truera/trulens/blob/main/src/providers/openai/trulens/providers/openai/endpoint.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
I took your advice, use logging to find out problem:
```
import logging
from trulens.providers.openai import OpenAI

# Configure logging
logging.basicConfig(level=logging.DEBUG)

# Initialize the feedback provider
feedback_provider = OpenAI(
    model_engine="glm-4v-9b",
    endpoint="http://0.0.0.0:8000/v1/",
    api_key="your_api_key",
)

# Example usage: create a chat completion
try:
    response = feedback_provider._create_chat_completion(prompt="Hello, how are you?")
    print("Chat completion response:", response)
except Exception as e:
    logging.error("Error: %s", e)
```
The DEBUG as follows:
```
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/usr/lib/ssl/certs/ca-certificates.crt'
DEBUG:trulens.core.feedback.endpoint:module <module 'openai.resources' from '/root/anaconda3/envs/trulens/lib/python3.11/site-packages/openai/resources/__init__.py'> already instrumented for create
DEBUG:trulens.core.feedback.endpoint:module <module 'openai.resources.chat' from '/root/anaconda3/envs/trulens/lib/python3.11/site-packages/openai/resources/chat/__init__.py'> already instrumented for create
DEBUG:trulens.core.feedback.endpoint:Calling instrumented method <function Completions.create at 0x7ff21ddbf420> of type <class 'function'>, iscoroutinefunction=False, isasyncgeneratorfunction=False
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'system', 'content': 'Hello, how are you?'}], 'model': 'glm-4v-9b', 'seed': 123, 'temperature': 0.0}}
DEBUG:openai._base_client:Sending HTTP Request: POST https://api.openai.com/v1/chat/completions
DEBUG:httpcore.connection:connect_tcp.started host='api.openai.com' port=443 local_address=None timeout=5.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(101, 'Network is unreachable'))
DEBUG:openai._base_client:Encountered Exception
Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
    yield
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/root/anaconda3/envs/trulens/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/openai/_base_client.py", line 973, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 926, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "/root/anaconda3/envs/trulens/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 101] Network is unreachable
DEBUG:openai._base_client:1 retry left
INFO:openai._base_client:Retrying request to /chat/completions in 0.967390 seconds
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'system', 'content': 'Hello, how are you?'}], 'model': 'glm-4v-9b', 'seed': 123, 'temperature': 0.0}}
DEBUG:openai._base_client:Sending HTTP Request: POST https://api.openai.com/v1/chat/completions
DEBUG:httpcore.connection:connect_tcp.started host='api.openai.com' port=443 local_address=None timeout=5.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(101, 'Network is unreachable'))
DEBUG:openai._base_client:Encountered Exception
Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
    yield
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/root/anaconda3/envs/trulens/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/openai/_base_client.py", line 973, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 926, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "/root/anaconda3/envs/trulens/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 101] Network is unreachable
DEBUG:openai._base_client:0 retries left
INFO:openai._base_client:Retrying request to /chat/completions in 1.826524 seconds
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'system', 'content': 'Hello, how are you?'}], 'model': 'glm-4v-9b', 'seed': 123, 'temperature': 0.0}}
DEBUG:openai._base_client:Sending HTTP Request: POST https://api.openai.com/v1/chat/completions
DEBUG:httpcore.connection:connect_tcp.started host='api.openai.com' port=443 local_address=None timeout=5.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(101, 'Network is unreachable'))
DEBUG:openai._base_client:Encountered Exception
Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
    yield
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/root/anaconda3/envs/trulens/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/openai/_base_client.py", line 973, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 926, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "/root/anaconda3/envs/trulens/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/anaconda3/envs/trulens/lib/python3.11/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 101] Network is unreachable
DEBUG:openai._base_client:Raising connection error
ERROR:root:Error: Connection error.
```
It's that means the code cannot connect to the server? By the way, the llm glm-4v-9b was deployment on my server, connection failure is not possible, the test code prove that.
```
from langchain_openai import ChatOpenAI

glm4_base_client = ChatOpenAI(model="glm-4v-9b",
                          api_key="your_api_key",
                          base_url="http://0.0.0.0:8000/v1/",
                          )
glm4_base_client.invoke("hello")
```


<!-- Answer -->
The issue seems to be that the `OpenAI` class is attempting to connect to the default OpenAI API endpoint (`https://api.openai.com/v1/chat/completions`) instead of your locally deployed endpoint (`http://0.0.0.0:8000/v1/`). This discrepancy is likely due to how the `OpenAI` class is configured to handle the endpoint parameter.

To ensure that the `OpenAI` class uses your specified endpoint, you need to correctly set up the `OpenAIEndpoint` and pass it to the `OpenAI` class. Here is how you can do it:

1. **Create an `OpenAIEndpoint` instance with your local endpoint configuration.**
2. **Pass this endpoint instance to the `OpenAI` class.**

Here is the updated code:

```python
from trulens.providers.openai.endpoint import OpenAIEndpoint
from trulens.providers.openai import OpenAI

# Create an OpenAIEndpoint instance with your local endpoint configuration
endpoint_config = {
    'api_base': 'http://0.0.0.0:8000/v1',  # Your local endpoint
    'api_key': 'your_api_key',             # Your API key
}

openai_endpoint = OpenAIEndpoint(**endpoint_config)

# Initialize the OpenAI class with the endpoint
feedback_provider = OpenAI(endpoint=openai_endpoint, model_engine="glm-4v-9b")

# Example usage: create a chat completion
try:
    response = feedback_provider._create_chat_completion(prompt="Hello, how are you?")
    print("Chat completion response:", response)
except Exception as e:
    print("Error:", e)
```

This setup ensures that the `OpenAI` class uses your local endpoint for making requests. The `OpenAIEndpoint` class is responsible for handling the connection to the specified endpoint, and by passing it to the `OpenAI` class, you ensure that all requests are directed to your local server.

This should resolve the connection issue and allow the `OpenAI` class to communicate with your locally deployed LLM <sup>[[1]](https://github.com/truera/trulens/blob/main/src/providers/openai/trulens/providers/openai/provider.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/src/trulens_eval/trulens_eval/feedback/provider/endpoint/openai.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@2500035435 to confirm, are you using an Azure OpenAI deployment? If so you can try

```python
from trulens.providers.openai.provider import AzureOpenAI

feedback_provider = AzureOpenAI(
    deployment_name="glm-4v-9b",
    azure_endpoint="http://0.0.0.0:8000/v1/",
    api_key="your_api_key",
    api_version="api_version" # optionally
)
```

You can find an [example](https://www.trulens.org/examples/models/azure/azure_openai_langchain/#import-from-trulens) and [API reference](https://www.trulens.org/reference/trulens/providers/openai/provider/#trulens.providers.openai.provider.AzureOpenAI)


To use an OpenAI provider with a different base URL, you can run

```python
from trulens.providers.openai.provider import OpenAI

feedback_provider = OpenAI(
    model_engine="glm-4v-9b",
    base_url="http://0.0.0.0:8000/v1/",
    api_key="your_api_key"
)
```

@2500035435 closing this for now. pls reopen if you need more help