Repository: truera/trulens
Issue: Feedback functions returing None : API langchain request failed 4 time(s)
Root cause: Environment Incompatibility

We are using langchain and open ai ( no access to embeddings ) . 
We have sucessfully implemented RAG chains in this environment. However , facing issues when integrating with truelens. 

I am following the code base here : 
https://www.trulens.org/trulens_eval/langchain_quickstart/#explore-in-a-dashboard

and even though I am  the RAG execution is sucessful and results are fetched 

with tru_recorder as recording:
    llm_response = rag_chain.invoke(query)

display(llm_response)

Feedback functions are returning None : 

    display(feedback.name, feedback_result.result)


'groundedness_measure_with_cot_reasons'
None
'qs_relevance'
None
'relevance'
None

_Originally posted by @arkacisco in https://github.com/truera/trulens/issues/833#issuecomment-1920299834_
 


On further debugging , here is the final error from the records :

API langchain request failed 4 time(s)..

Please note we have an AzureChatOpenAI endpoint and used it in the chain argument of the langchain provider and defined the feedback callbacks with that provider . 

llm = AzureChatOpenAI(deployment_name="gpt-35-turbo", 
                      base_url  = 'https://chat-ai.cisco.com/openai', 
                      api_key=token_response.json()["access_token"],  
                      api_version="2023-08-01-preview",
                      model_kwargs=dict(
                      user=f'{{"appkey": "{app_key}"}}'
                    )
       ) 
from trulens_eval.feedback.provider.langchain import Langchain

 langchain_provider = Langchain(chain = llm)

from trulens_eval.feedback import Groundedness
grounded = Groundedness(groundedness_provider=langchain_provider)
# Define a groundedness feedback function
f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons)
    .on(context.collect()) # collect context chunks into a list
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)


Here is the complete error :

FeedbackResult(feedback_result_id='feedback_result_hash_78971a0a21dd7ff9f4284477eb48f957', record_id='record_hash_f2af4562df7046022be19265ed889b11', feedback_definition_id='feedback_definition_hash_aa08be1911c9532808078c267ce61b3d', last_ts=datetime.datetime(2024, 2, 1, 19, 16, 12, 821320), status=<FeedbackResultStatus.FAILED: 'failed'>, cost=Cost(n_requests=0, n_successful_requests=0, n_classes=0, n_tokens=0, n_stream_chunks=0, n_prompt_tokens=0, n_completion_tokens=0, cost=0.0), name='relevance', calls=[], result=None, error='Traceback (most recent call last):\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 542, in run\n result_and_meta, part_cost = sync(\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/utils/asynchro.py", line 114, in sync\n return loop.run_until_complete(awaitable)\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete\n return f.result()\n File "/opt/conda/lib/python3.10/asyncio/futures.py", line 201, in result\n raise self._exception.with_traceback(self._exception_tb)\n File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 234, in __step\n result = coro.throw(exc)\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 436, in atrack_all_costs_tally\n result, cbs = await Endpoint.atrack_all_costs(\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 421, in atrack_all_costs\n return await Endpoint._atrack_costs(thunk, with_endpoints=endpoints)\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 510, in _atrack_costs\n result: T = await desync(thunk)\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/utils/asynchro.py", line 82, in desync\n res = await asyncio.to_thread(func, *args, **kwargs)\n File "/opt/conda/lib/python3.10/asyncio/threads.py", line 25, in to_thread\n return await loop.run_in_executor(None, func_call)\n File "/opt/conda/lib/python3.10/asyncio/futures.py", line 285, in await\n yield self # This tells Task to wait for completion.\n File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup\n future.result()\n File "/opt/conda/lib/python3.10/asyncio/futures.py", line 201, in result\n raise self._exception.with_traceback(self._exception_tb)\n File "/opt/conda/lib/python3.10/concurrent/futures/thread.py", line 58, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/utils/python.py", line 248, in _future_target_wrapper\n return func(*args, **kwargs)\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 544, in \n lambda: self.imp(**ins)\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py", line 286, in relevance\n self.endpoint.run_me(\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 272, in run_me\n raise RuntimeError(\nRuntimeError: API langchain request failed 4 time(s).\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 548, in run\n raise RuntimeError(\nRuntimeError: Evaluation of relevance failed on inputs: \n{'prompt': '',\n 'response': 'To overcome the bug< redacted>, you can fix it by u\nAPI langchain request failed 4 time(s)..\n', multi_result=None))



This issue was closed by mistake. reopened.

Hey @arkacisco - can you try testing your langchain provider on its own?

You can use the following:

`langchain_provider.relevance('What's the capital of Japan?','Tokyo')`

Please ignore my previous comment , its not working .. here are the details : 

Code executed : 

from trulens_eval.feedback.provider.langchain import Langchain
from langchain.llms import OpenAI, AzureOpenAI

gpt3_llm = AzureOpenAI(model="gpt-35-turbo", base_url = 'https://chat-ai.cisco.com', 
                      api_key=token_response.json()["access_token"],  
                      api_version="2023-08-01-preview",
                         model_kwargs=dict(
                      user=f'{{"appkey": "{app_key}"}}'
                    )
                    )
langchain_provider = Langchain(chain = gpt3_llm)


llm.predict("tell me a joke")

"Sure, here's a classic one for you:\n\nWhy don't scientists trust atoms?\n\nBecause they make up everything!"

langchain_provider.relevance('What\'s the capital of Japan?','Tokyo',)


Error Log : 


langchain request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'fault': {'faultstring': 'The Token has expired: policy(JWT-validateToken)', 'detail': {'errorcode': 'steps.jwt.TokenExpired'}}}. Retries remaining=3.
langchain request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'fault': {'faultstring': 'The Token has expired: policy(JWT-validateToken)', 'detail': {'errorcode': 'steps.jwt.TokenExpired'}}}. Retries remaining=2.
langchain request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'fault': {'faultstring': 'The Token has expired: policy(JWT-validateToken)', 'detail': {'errorcode': 'steps.jwt.TokenExpired'}}}. Retries remaining=1.
langchain request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'fault': {'faultstring': 'The Token has expired: policy(JWT-validateToken)', 'detail': {'errorcode': 'steps.jwt.TokenExpired'}}}. Retries remaining=0.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[171], line 1
----> 1 langchain_provider.relevance('What\'s the capital of Japan?','Tokyo',)

File /mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py:286, in LLMProvider.relevance(self, prompt, response)
    253 def relevance(self, prompt: str, response: str) -> float:
    254     """
    255     Uses chat completion model. A function that completes a
    256     template to check the relevance of the response to a prompt.
   (...)
    283         "relevant".
    284     """
    285     return re_0_10_rating(
--> 286         self.endpoint.run_me(
    287             lambda: self._create_chat_completion(
    288                 prompt=str.format(
    289                     prompts.PR_RELEVANCE, prompt=prompt, response=response
    290                 )
    291             )
    292         )
    293     ) / 10.0

File /mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py:272, in Endpoint.run_me(self, thunk)
    269             sleep(retry_delay)
    270             retry_delay *= 2
--> 272 raise RuntimeError(
    273     f"API {self.name} request failed {self.retries+1} time(s)."
    274 )

RuntimeError: API langchain request failed 4 time(s).





Does the LLM you have here work outside of the TruLens Langchain provider?

Yes @joshreini1  , it does . We normally use one of the the following options to connect and use the the LLM endpoint : 

-- Azure Chat OpenAI : 
from langchain.chat_models import AzureChatOpenAI
app_key  =<redacted>
llm = AzureChatOpenAI(deployment_name="gpt-35-turbo", 
                      base_url  = 'https://chat-ai.cisco.com/openai', 
                      api_key=token_response.json()["access_token"],  
                      api_version="2023-08-01-preview",
                      model_kwargs=dict(
                      user=f'{{"appkey": "{app_key}"}}'
                    )
       ) 

llm.predict("Tell me  a joke")

"Why don't scientists trust atoms?\n\nBecause they make up everything!"


------------AzureOpenAI---------------


from openai import AzureOpenAI
azure_openai_llm = AzureOpenAI(#deployment_name="gpt-35-turbo", 
                      azure_endpoint = 'https://chat-ai.cisco.com', 
                      api_key=token_response.json()["access_token"],  
                      api_version="2023-08-01-preview",
                     
                    )
from openai.types.chat import ChatCompletionMessageParam
message_with_history: list[ChatCompletionMessageParam] = [
    {"role": "system", "content": "You are a chatbot"},
    {
        "role": "user",
        "content": "Tell me a joke",
    },
]
response = azure_openai_llm.chat.completions.create(
    model="gpt-35-turbo",
    messages=message_with_history,
    temperature=0,
    # stop=["
"],
    user='{"appkey": <redacted>}',
    max_tokens=1000,
    # stream=True,
)
print("Reponse:", response)
for chunk in response:
    print("Chunk:", chunk) 

Reponse: ChatCompletion(id='chatcmpl-8pLUvV3agEQuGTFymuKMWqNWr3fb9', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="Sure, here's a joke for you:\n\nWhy don't scientists trust atoms?\n\nBecause they make up everything!", role='assistant', function_call=None, tool_calls=None), content_filter_results={'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}})], created=1707248057, model='gpt-35-turbo', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=22, prompt_tokens=20, total_tokens=42), prompt_filter_results=[{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}], prompt_sent_to_chatgpt=[{'role': 'system', 'content': 'You are a chatbot'}, {'role': 'user', 'content': 'Tell me a joke'}], user='{"appkey": "redacted", "session_id": "1a709f11-ae05-4ed4-98d6-6c2662663169", "user": "", "prompt_truncate": "yes"}')
Chunk: ('id', 'chatcmpl-8pLUvV3agEQuGTFymuKMWqNWr3fb9')
Chunk: ('choices', [Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="Sure, here's a joke for you:\n\nWhy don't scientists trust atoms?\n\nBecause they make up everything!", role='assistant', function_call=None, tool_calls=None), content_filter_results={'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}})])
Chunk: ('created', 1707248057)
Chunk: ('model', 'gpt-35-turbo')
Chunk: ('object', 'chat.completion')
Chunk: ('system_fingerprint', None)
Chunk: ('usage', CompletionUsage(completion_tokens=22, prompt_tokens=20, total_tokens=42))
Chunk: ('prompt_filter_results', [{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}])
Chunk: ('prompt_sent_to_chatgpt', [{'role': 'system', 'content': 'You are a chatbot'}, {'role': 'user', 'content': 'Tell me a joke'}])
Chunk: ('user', '{"appkey": "redacted", "session_id": "1a709f11-ae05-4ed4-98d6-6c2662663169", "user": "", "prompt_truncate": "yes"}')



Can you use our AzureOpenAI provider directly? https://www.trulens.org/trulens_eval/api/azureopenai_provider/

Here's a usage example notebook: https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/azure_openai.ipynb

Using the AzureOpenAI provider directly is more robust and more widely used feedback provider compared to the Langchain provider.

I tried this before using langchain provider and it did not work . 

First of all it does not take the app_key as an argument : 

from trulens_eval.feedback.provider import OpenAI, AzureOpenAI

tru_openai = AzureOpenAI(deployment_name="gpt-35-turbo", 
                      base_url = 'https://chat-ai.cisco.com', 
                      api_key=token_response.json()["access_token"],  
                      api_version="2023-08-01-preview",
                      model_kwargs=dict(
                      user=f'{{"appkey": "{app_key}"}}'
                    )
                    )


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[176], line 3
      1 # Initialize provider class
      2 # openai = OpenAI()
----> 3 tru_openai = AzureOpenAI(deployment_name="gpt-35-turbo", 
      4                       base_url = 'https://chat-ai.cisco.com/', 
      5                       api_key=token_response.json()["access_token"],  
      6                       api_version="2023-08-01-preview",
      7                       model_kwargs=dict(
      8                       user=f'{{"appkey": "{app_key}"}}'
      9                     )
     10                     )

File /mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/openai.py:429, in AzureOpenAI.__init__(self, deployment_name, endpoint, **kwargs)
    425 else:
    426     # but include in provider args
    427     kwargs['model_engine'] = deployment_name
--> 429 kwargs["client"] = OpenAIClient(client=oai.AzureOpenAI(**client_kwargs))
    431 super().__init__(
    432     endpoint=None, **kwargs
    433 )

TypeError: AzureOpenAI.__init__() got an unexpected keyword argument 'model_kwargs'


If I use it without the app_key : 

# Initialize provider class
tru_openai = AzureOpenAI(deployment_name="gpt-35-turbo", 
                      base_url = 'https://chat-ai.cisco.com', 
                      api_key=token_response.json()["access_token"],  
                      api_version="2023-08-01-preview",
            
                   )

I get the same errror : 

tru_openai.relevance('What\'s the capital of Japan?','Tokyo',)

openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'fault': {'faultstring': 'The Token has expired: policy(JWT-validateToken)', 'detail': {'errorcode': 'steps.jwt.TokenExpired'}}}. Retries remaining=3.
openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'fault': {'faultstring': 'The Token has expired: policy(JWT-validateToken)', 'detail': {'errorcode': 'steps.jwt.TokenExpired'}}}. Retries remaining=2.
openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'fault': {'faultstring': 'The Token has expired: policy(JWT-validateToken)', 'detail': {'errorcode': 'steps.jwt.TokenExpired'}}}. Retries remaining=1.
openai request failed <class 'openai.AuthenticationError'>=Error code: 401 - {'fault': {'faultstring': 'The Token has expired: policy(JWT-validateToken)', 'detail': {'errorcode': 'steps.jwt.TokenExpired'}}}. Retries remaining=0.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[187], line 1
----> 1 tru_openai.relevance('What\'s the capital of Japan?','Tokyo',)

File /mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py:286, in LLMProvider.relevance(self, prompt, response)
    253 def relevance(self, prompt: str, response: str) -> float:
    254     """
    255     Uses chat completion model. A function that completes a
    256     template to check the relevance of the response to a prompt.
   (...)
    283         "relevant".
    284     """
    285     return re_0_10_rating(
--> 286         self.endpoint.run_me(
    287             lambda: self._create_chat_completion(
    288                 prompt=str.format(
    289                     prompts.PR_RELEVANCE, prompt=prompt, response=response
    290                 )
    291             )
    292         )
    293     ) / 10.0

File /mnt/nfs/shared/arkghosh/LLM/BCS KB/notebooks/venv/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py:272, in Endpoint.run_me(self, thunk)
    269             sleep(retry_delay)
    270             retry_delay *= 2
--> 272 raise RuntimeError(
    273     f"API {self.name} request failed {self.retries+1} time(s)."
    274 )

RuntimeError: API openai request failed 4 time(s).





Working on a fix for this @arkacisco 

Thanks !