Repository: truera/trulens
Issue: Run open-souce-LLM-based evaluations with LiteLLM proxy
Root cause: Documentation Deficiency

Hello,

My name is Enrique. I work for VMware/Broadcom developing reference architectures that help our customers get started with smart app development based on local (mainly open-source) LLMs. We'd like to include TruLens as the evaluation framework. 

It looks like TrueLens requires the use of LiteLLM to let users carry on evaluations based on models like Llama 2, Mistral 7b (x8), Zephyr, etc., that run within a company's data center.

I managed to replicate the examples from [Smoke Tests/Answer Relevance](https://www.trulens.org/trulens_eval/answer_relevance_smoke_tests/) by making multiple tweaks that are not documented and probably no even supported. Here the code:

```
# Import relevance feedback function
from trulens_eval.feedback import GroundTruthAgreement, OpenAI, LiteLLM
from trulens_eval import TruBasicApp, Feedback, Tru, Select
from test_cases import answer_relevance_golden_set

Tru().reset_database()

import litellm
from trulens_eval.feedback import provider

litellm.set_verbose = True
litellm.api_base = "http://localhost:8010/v1"
litellm.api_key = "NO_EVAL_KEY"


COMPLETE_ENGINE="openai/HuggingFaceH4/zephyr-7b-alpha"

litellm_provider = provider.litellm.LiteLLM(model_engine=COMPLETE_ENGINE)

zephyr_7b_alpha = LiteLLM(model_engine=COMPLETE_ENGINE)

def wrapped_relevance_zephyr(input, output):
    return zephyr_7b_alpha.relevance(input, output)

tru_wrapped_relevance_zephyr = TruBasicApp(wrapped_relevance_zephyr,
                                           app_id = "answer relevance zephyr 7b alpha",
                                           feedbacks=[f_mae])

for i in range(len(answer_relevance_golden_set)):
    prompt = answer_relevance_golden_set[i]["query"]
    response = answer_relevance_golden_set[i]["response"]

    with tru_wrapped_relevance_zephyr as recording:
        tru_wrapped_relevance_zephyr.app(prompt, response)

Tru().get_leaderboard(app_ids=[]).sort_values(by='Mean Absolute Error')

```

This code runs without errors, however the `litellm.api_base` value is not a [LiteLLM gateway](https://docs.litellm.ai/docs/proxy/quick_start) service but a vLLM server running in OpenAI API- compatible mode. Here is a list of why this is not convenient.

- First and foremost, TrueLens is supposed to leverage LiteLLM's endpoints/engines. In my case, although the code runs without errors, vLLM is not a LiteLLM endpoint and the code might break eventually.
- Even if the vLLM endpoint works as a provider for TrueLens the need import and set the `litellm.api_base` value for different LLM inference sources is kind for fragile.

It'd be more convenient for users to get TrueLens properly integrated with LiteLLM proxy services so they can take advantage of its telemetry and routing features.

I have a working LitteLLM gateway configured in the following way:

```
# LiteLLM proxy config.yaml file
model_list: 
  - model_name: vllm-model
    litellm_params:
      api_key: no_key
      model: openai/HuggingFaceH4/zephyr-7b-alpha 
      api_base: http://localhost:8010/v1 # e.g. http://0.0.0.0:3000

```

If I run a completion request targeting the `api_base` from this config file, the following code runs successfully:

```
from llama_index.llms import OpenAILike

llm = OpenAILike(model="openai/HuggingFaceH4/zephyr-7b-alpha",
              temperature=0.1,
              max_tokens=300,
              api_base="http://0.0.0.0:8000",
              api_key="NOKEY",
              is_chat_model=True,
              )

llm.complete("what is a gravity?")

CompletionResponse(text='Gravity is a fundamental force of nature that causes objects with mass to attract each other. It is one of the four fundamental forces in physics, along with electromagnetism, the strong nuclear force, and the weak nuclear force. Gravity is the force that keeps the Earth and other planets in orbit around the Sun, and it is what causes objects to fall to the ground when dropped. The strength of gravity decreases with distance, and it is inversely proportional to the square of the distance between two objects.', additional_kwargs={}, raw={'id': 'cmpl-fdf963508170456da371f0536dcc8a96', 'choices': [Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Gravity is a fundamental force of nature that causes objects with mass to attract each other. It is one of the four fundamental forces in physics, along with electromagnetism, the strong nuclear force, and the weak nuclear force. Gravity is the force that keeps the Earth and other planets in orbit around the Sun, and it is what causes objects to fall to the ground when dropped. The strength of gravity decreases with distance, and it is inversely proportional to the square of the distance between two objects.', role='assistant', function_call=None, tool_calls=None))], 'created': 698858, 'model': 'HuggingFaceH4/zephyr-7b-alpha', 'object': 'chat.completion', 'system_fingerprint': None, 'usage': CompletionUsage(completion_tokens=107, prompt_tokens=22, total_tokens=129), '_response_ms': 1686.105}, delta=None)

```

However if I try to use the LiteLLM proxy endpoint (by setting the COMPLETE_ENGINE variable to the proxy's URL) to define the function required by TrueLens smoke test code, it would not work. 

```
litellm_provider = provider.litellm.LiteLLM(model_engine=COMPLETE_ENGINE)

zephyr_7b_alpha = LiteLLM(model_engine=COMPLETE_ENGINE)

def wrapped_relevance_zephyr(input, output):
    return zephyr_7b_alpha.relevance(input, output)

```

You might say that I have not set the `endpoint` parameter but just like other people have commented, the parameter does not seen to have any effect.

Finally, I think you might want to consider supporting vLLM as a direct provider as it is very pervasive across the GenAI open-source community.

Thanks in advance for your help. 



Hey @vecorro - can you use our langchain provider for this?

Relevant docs: https://www.trulens.org/trulens_eval/api/langchain_provider/

Hey @vecorro, 

Why use litellm here? vllm is already openai compatible. 

Also, can you share any error message you're seeing? Trying to debug if this is a litellm proxy issue (context - i'm the co-maintainer of litellm)

Thanks @joshreini1 

I'm happy to try the LangChain provider, however to @krrishdholakia 's question, the initial reason to use LiteLLM is the [Answer Relevance](https://www.trulens.org/trulens_eval/answer_relevance_smoke_tests/) and [Context Relevance](https://www.trulens.org/trulens_eval/context_relevance_smoke_tests/) TruLens smoke tests get exemplified by extensibly using LiteLLM. Based on that fact, I though that would be an easy way to get started with TruLens.

On the other hand,  likely enterprise settings would combine the use of local (open-source) LLMs and commercial ones (Cohere, Anthropic, OpenAI, Google, etc.) so the LiteLLM proxy's value proposition looks interesting as it can bring different types of LLMs together under a unified API service. In addition, the proxy's load balancing, logging, and health checks could be valuable and worth exploring. vLLM is great but it does not offer many of these features.

That said, here the detail of what I'm doing:
```
# Import relevance feedback function
from trulens_eval.feedback import GroundTruthAgreement, OpenAI, LiteLLM
from trulens_eval import TruBasicApp, Feedback, Tru, Select
from test_cases import answer_relevance_golden_set

Tru().reset_database()

import litellm
from trulens_eval.feedback import provider

litellm.set_verbose = True
litellm.api_base = "http://0.0.0.0:8000" # The working LiteLLM proxy URL base 
litellm.api_key = "NO_EVAL_KEY"

COMPLETE_ENGINE="vllm-model"

#COMPLETE_ENGINE="openai/HuggingFaceH4/zephyr-7b-alpha"

litellm_provider = provider.litellm.LiteLLM(model_engine=COMPLETE_ENGINE,
                                            # endpoint="/chat/completions",
                                            )
zephyr_7b_alpha = LiteLLM(model_engine=COMPLETE_ENGINE,
                          # endpoint="/chat/completions"
                          )
def wrapped_relevance_zephyr(input, output):
    return zephyr_7b_alpha.relevance(input, output)
    
# Create a Feedback object using the numeric_difference method of the ground_truth object
ground_truth = GroundTruthAgreement(answer_relevance_golden_set,
                                    provider=litellm_provider)

# Call the numeric_difference method with app and record and aggregate to get the mean absolute error
f_mae = (Feedback(ground_truth.mae, name = "Mean Absolute Error")
         .on(Select.Record.calls[0].args.args[0])
         .on(Select.Record.calls[0].args.args[1])
         .on_output())
tru_wrapped_relevance_zephyr = TruBasicApp(wrapped_relevance_zephyr,
                                           app_id = "answer relevance zephyr 7b alpha",
                                           feedbacks=[f_mae])
```

If I set `COMPLETE_ENGINE="vllm-model"`, I get the following error:
```
litellm request failed <class 'litellm.exceptions.APIConnectionError'>=LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=vllm-model
 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers. Retries remaining=3.

```
On the other hand, if I set `COMPLETE_ENGINE="openai/HuggingFaceH4/zephyr-7b-alpha"` then I get the following error:

```
litellm request failed <class 'litellm.exceptions.APIError'>=OpenAIException - Error code: 400 - {'error': {'message': "OpenAIException - LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=HuggingFaceH4/zephyr-7b-alpha\n Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers", 'type': None, 'param': None, 'code': 400}}. Retries remaining=3.
```

Thanks in advance for any help on this. LiteLLM seems to have multiple interesting features and it'd be great to use them in conjunction with TruLens. 

@krrishdholakia assuming you can help @vecorro in figuring out what's going wrong here?

Hi @vecorro 

This just looks like your litellm proxy doesn't know what those names are. 

How did you set it up? Can you share your config? 

Thanks both. I posted the gateway config in at the middle of my initial comment. Here the re-post for you convenience @krrishdholakia 

```
# LiteLLM proxy config.yaml file
model_list: 
  - model_name: vllm-model
    litellm_params:
      api_key: no_key
      model: openai/HuggingFaceH4/zephyr-7b-alpha 
      api_base: http://localhost:8010/v1 # e.g. http://0.0.0.0:3000

```

As I also mentioned in the initial post, the following code that targets the LiteLLM proxy works properly:

```
from llama_index.llms import OpenAILike

llm = OpenAILike(model="openai/HuggingFaceH4/zephyr-7b-alpha",
              temperature=0.1,
              max_tokens=300,
              api_base="http://0.0.0.0:8000",
              api_key="NOKEY",
              is_chat_model=True,
              )

llm.complete("what is a gravity?")

CompletionResponse(text='Gravity is a fundamental force of nature that causes objects with mass to attract each other. It is one of the four fundamental forces in physics, along with electromagnetism, the strong nuclear force, and the weak nuclear force. Gravity is the force that keeps the Earth and other planets in orbit around the Sun, and it is what causes objects to fall to the ground when dropped. The strength of gravity decreases with distance, and it is inversely proportional to the square of the distance between two objects.', additional_kwargs={}, raw={'id': 'cmpl-fdf963508170456da371f0536dcc8a96', 'choices': [Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Gravity is a fundamental force of nature that causes objects with mass to attract each other. It is one of the four fundamental forces in physics, along with electromagnetism, the strong nuclear force, and the weak nuclear force. Gravity is the force that keeps the Earth and other planets in orbit around the Sun, and it is what causes objects to fall to the ground when dropped. The strength of gravity decreases with distance, and it is inversely proportional to the square of the distance between two objects.', role='assistant', function_call=None, tool_calls=None))], 'created': 698858, 'model': 'HuggingFaceH4/zephyr-7b-alpha', 'object': 'chat.completion', 'system_fingerprint': None, 'usage': CompletionUsage(completion_tokens=107, prompt_tokens=22, total_tokens=129), '_response_ms': 1686.105}, delta=None)

```

Hi @vecorro can we hop on a call to better understand your problem ?

Sharing my calendly for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Thank, just scheduled the call.

@vecorro Is this is issue still relevant?

Hi,

The LitteLLM team helped me figure out a solution. Thanks a lot.

Hi @vecorro 
Can you share the solution