Repository: truera/trulens
Issue: [BUG] No return of feedback when using LiteLLM with Mistral
Root cause: Configuration Error

**Bug Description**

Actually I am using HuggingFacePipeline as model to my RAG on Langchain (I have no problem and I get an output) and for evaluating I am using TruLens with LiteLLM. The model : mistralai/Mistral-7B-Instruct-v0.2

**To Reproduce**

```
from trulens_eval.feedback.provider.litellm import LiteLLM
from trulens_eval import TruChain,Feedback,Tru
from trulens_eval.app import App
litellm_provider = LiteLLM(model_engine = "huggingface/mistralai/Mistral-7B-Instruct-v0.2")
tru = Tru()
tru.reset_database()
context = App.select_context(chain)
f_answer_relevance = (
    Feedback(litellm_provider.relevance)
    .on_input_output()
)

with chain_recorder as recording:
    chain.invoke("What is the capital of USA?")

rec = recording.records[0]

for feedback, feedback_result in rec.wait_for_feedback_results().items():
    print(feedback.name, feedback_result.result)
```

My output : `relevance None`


**Expected behavior**
I want a feedback to be returned and not None


**Environment:**
 - OS: Linux
 - Python Version 3.10.12
 - TruLens version 0.27.2
 - Running on Colab

**Additional context**
I try to see all docs from your site but unfortunately I don't run something that works so I come here for your help.


<!-- Answer -->
It looks like the issue you're encountering with receiving `None` for feedback when using LiteLLM with the `mistralai/Mistral-7B-Instruct-v0.2` model might be due to how the response from the model is being handled or expected. Ensure that the response from the `completion` function in the `LiteLLM` class matches the expected structure, particularly that it contains a `"choices"` key with a list of dictionaries, and each dictionary should have a `"message"` key with a dictionary containing a `"content"` key.

To troubleshoot and potentially resolve this issue, consider the following steps:

1. **Check the Response Structure**: Confirm that the response from the `completion` function aligns with the expected format. The response should be a dictionary with a `"choices"` key, where `"choices"` is a list of dictionaries, and each of those contains a `"message"` key with a dictionary that has a `"content"` key.

2. **Review Model Documentation**: Make sure you've provided all necessary parameters to the model as per its documentation and that you're handling its response correctly.

3. **Implement Error Handling**: Add error handling around the `completion` function call to catch any unexpected responses or issues.

4. **Add Debugging Logs**: Temporarily insert logging before processing the response to inspect the structure and content of the `comp` variable.

These steps should help you identify and fix the issue with receiving feedback. If you've tried these and are still facing problems, providing more details about the error or unexpected behavior could help in offering further assistance.

## Sources
- [trulens_eval/trulens_eval/feedback/provider/litellm.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/litellm.py)

<!-- Dosu Comment Footer -->
<details>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>


Hi @Kev744,

Happy to help - 

The example code you provided does not seem to be complete, so I've taken the liberty to fill in the blanks. Below is a minimal example I ran with TruLens 0.27.2 and Python 3.10, and I am able to get the relevance score output. Could you perhaps try and compare it with your current setup? Let me know if this is helpful


```# %%
from trulens_eval.feedback.provider.litellm import LiteLLM
from trulens_eval import TruChain,Feedback,Tru
from trulens_eval.app import App


# %%
litellm_provider = LiteLLM(model_engine = "huggingface/mistralai/Mistral-7B-Instruct-v0.2")
tru = Tru()
tru.reset_database()


# %%
import os
os.environ["OPENAI_API_KEY"] = "sk-..."

# %%
from langchain_core.runnables import RunnablePassthrough
from langchain.schema import StrOutputParser
from langchain import hub
from langchain.chat_models import ChatOpenAI
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.document_loaders import WebBaseLoader
import bs4


loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
splits = text_splitter.split_documents(docs)


vectorstore = Chroma.from_documents(
    documents=splits,
    embedding=OpenAIEmbeddings()
)

# %%
retriever = vectorstore.as_retriever()

prompt = hub.pull("rlm/rag-prompt")
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# %%
context = App.select_context(chain)
f_answer_relevance = (
    Feedback(litellm_provider.relevance)
    .on_input_output()
)

chain_recorder = TruChain(chain, app_id="test_mistral", feedbacks=[f_answer_relevance])

with chain_recorder as recording:
    chain.invoke("What is the capital of USA?")

rec = recording.records[0]

for feedback, feedback_result in rec.wait_for_feedback_results().items():
    print(feedback.name, feedback_result.result)
```

Hello, so I try your code but replacing it by a Mistral open-source model that I use to my project instead of OpenAI. Check all the codes please because I need to register the model (the log said me that I need to register the model for custom pricing) on litellm for the relevance working but token number, total cost and n_requests stay at 0.

Here is the link of Colab attached to show you the logs

[Link to Colab Project](https://colab.research.google.com/drive/1cvmojcAIn8ZpogjnGk5uXR79_KjYZ85E?usp=sharing
)

@Kev744 Happy to take a look - could you grant me read access to the Colab notebook for my email daniel@truera.com so that I can view it? 

@daniel-huang-1230 it's done thanks

Hi @Kev744,

I've reviewed and run your notebook and I believe the root cause for the cost tracking metrics not showing up correctly is due to the LiteLLM's model provider.

In your notebook, you are using the below model (model name "mistralai/Mistral-7B-Instruct-v0.2" from litellm_provider "huggingface"
```
register_model({
        "mistralai/Mistral-7B-Instruct-v0.2": {
        "max_tokens" : 32000,
        "input_cost_per_token": 50,
        "output_cost_per_token": 0.000000,
        "litellm_provider" : "huggingface",
        "mode":"chat"
    },
})
```

However, the Mistral-7B-instructo-v0.2 with huggingface provider does not yet seem to be available from the community-maintained list in LiteLLM. (ref: https://litellm.vercel.app/docs/completion/token_usage#7-model_cost)

In order to get out-of-the-box cost tracking in TruLens with LiteLLM, you'll need to pick  from list below and use Mistral-7B-instruct from one of the below available providers (i.e. Bedrock)
https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json#L1588

> In your notebook, you are using the below model (model name "mistralai/Mistral-7B-Instruct-v0.2" from litellm_provider "huggingface"
> 
So explain me why it worked for you with GPT as LLM for LangChain and HuggingFace as provider with LiteLLM ? Because I do exactly the same for the provider...
> 
> However, the Mistral-7B-instructo-v0.2 with huggingface provider does not yet seem to be available from the community-maintained list in LiteLLM. (ref: https://litellm.vercel.app/docs/completion/token_usage#7-model_cost)
> 
If isn't supported, why it supported for you in the first provided code ? And why by registering the model in "model_prices_and_context_window.json", I am able to get the feedbacks for example and not the total tokens or cost ? Because actually by registering the model and call the LiteLLM completion_cost method, I get a desired cost.

Thanks



> > In your notebook, you are using the below model (model name "mistralai/Mistral-7B-Instruct-v0.2" from litellm_provider "huggingface"
> 
> So explain me why it worked for you with GPT as LLM for LangChain and HuggingFace as provider with LiteLLM ? Because I do exactly the same for the provider...
> 
> > However, the Mistral-7B-instructo-v0.2 with huggingface provider does not yet seem to be available from the community-maintained list in LiteLLM. (ref: https://litellm.vercel.app/docs/completion/token_usage#7-model_cost)
> 
> If isn't supported, why it supported for you in the first provided code ? And why by registering the model in "model_prices_and_context_window.json", I am able to get the feedbacks for example and not the total tokens or cost ? Because actually by registering the model and call the LiteLLM completion_cost method, I get a desired cost.
> 
> Thanks

My apology for the confusion - just realized after taking a second pass on your Colab notebook that you were using Huggingface pipeline + Langchain as the main LLM in rag_chain but also just happened to use the same model (Mistral-7b-instruct) wrapped with LiteLLM as Trulens' feedback provider. So updating cost info using LiteLLM's `register_model` has no effect on you main LLM being instrumented by TruLens. 

So to clarify, there are two issues we are looking at there:
1. relevance feedback function returning `None`: I believe this is resolved once you register model with LiteLLM, and this gives proper feedback scores with your relevance function.
2. missing cost and token tracking: the root cause is that while you were registering and overriding model costs with LiteLLM, 
   the main instrumented chain (rag_chain of class TruChain) is not aware of / not implemented as of today to track costs of models that are not OpenAI (and a few others). Cost tracking worked in my example as I'm using OpenAI's chat model and TruLens is able to extract cost/token-usage information from OpenAI endpoint by looking at Langchain's OpenAICallbackHandler ([see source code in Trulens](https://github.com/truera/trulens/blob/debug-mistral-output/trulens_eval/trulens_eval/feedback/provider/endpoint/openai.py#L190) and [source code in Langchain](https://api.python.langchain.com/en/latest/callbacks/langchain_community.callbacks.openai_info.OpenAICallbackHandler.html#langchain_community.callbacks.openai_info.OpenAICallbackHandler.on_llm_end)) 



To help you unblock and get cost tracking working in your specific use case, next steps can be: 
1. For the LLM used in your RAG, switch from using HuggingFace pipeline to LiteLLM with the same or similar Mistral model, this way you can continue using  `register_model` to override costs as you'd like. 
2. Our team will internally discuss this week about a larger feature update for **cost tracking for other common models** so that we can get HuggingFace pipeline to work with cost tracking natively.  Also feel free to open a PR if you'd like.

> To help you unblock and get cost tracking working in your specific use case, next steps can be:
> 
> 1. For the LLM used in your RAG, switch from using HuggingFace pipeline to LiteLLM with the same or similar Mistral model, this way you can continue using  `register_model` to override costs as you'd like.

We can't do it because they don't have a wrapper for loading LiteLLM models on Langchain for RAG, unless you have an idea ? so unfortunately I think the best way is to implement a code to track cost from HuggingFace.

> 2. Our team will internally discuss this week about a larger feature update for **cost tracking for other common models** so that we can get HuggingFace pipeline to work with cost tracking natively.  Also feel free to open a PR if you'd like.

Ok, thanks, I will create one

> > To help you unblock and get cost tracking working in your specific use case, next steps can be:
> > 
> > 1. For the LLM used in your RAG, switch from using HuggingFace pipeline to LiteLLM with the same or similar Mistral model, this way you can continue using  `register_model` to override costs as you'd like.
> 
> We can't do it because they don't have a wrapper for loading LiteLLM models on Langchain for RAG, unless you have an idea ? so unfortunately I think the best way is to implement a code to track cost from HuggingFace.
> 
> > 2. Our team will internally discuss this week about a larger feature update for **cost tracking for other common models** so that we can get HuggingFace pipeline to work with cost tracking natively.  Also feel free to open a PR if you'd like.
> 
> Ok, thanks, I will create one

re: "We can't do it because they don't have a wrapper for loading LiteLLM models on Langchain for RAG, unless you have an idea ? so unfortunately I think the best way is to implement a code to track cost from HuggingFace." 

Might be a bit late to this response but want to include this before closing the issue: 

One approach I've personally taken, if you can make do without using Langchain everywhere, is to build a small RAG pipeline from scratch and that way it's a breeze to load any models supported / available in LiteLLM in your RAG pipeline. 

TruLens has several example notebooks for this type of "rag from scratch instrumentations". Example code included here:
```
import litellm

class RAG_from_scratch:
    @instrument
    def retrieve(self, query: str) -> list:
        """
        Retrieve relevant text from vector store.
        """
        results = vector_store.query(
        query_texts=query,
        n_results=2
    )
        return results['documents'][0]

    @instrument
    def generate_completion(self, query: str, context_str: list) -> str:
        """
        Generate answer from context.
        """
        completion = litellm.completion(
        model="mistral/mistral-small",
        temperature=0,
        messages=
        [
            {"role": "user",
            "content": 
            f"We have provided context information below. \n"
            f"---------------------\n"
            f"{context_str}"
            f"\n---------------------\n"
            f"Given this information, please answer the question: {query}"
            }
        ]
        ).choices[0].message.content
        return completion

    @instrument
    def query(self, query: str) -> str:
        context_str = self.retrieve(query)
        completion = self.generate_completion(query, context_str)
        return completion

rag = RAG_from_scratch()
```
and here's a recent PR using the above construct from our team with an end-to-end notebook:
https://app.reviewnb.com/truera/trulens/pull/1109/.