Repository: truera/trulens
Issue: how can I use LLM from huggingface as the provider of the Trulens feedback? 
Root cause: Unimplemented Feature Gap

I want to use local model that download from huggingface as the provider of the Trulens feedback for RAG Evaluation of Llama-Index project.How to set it and custom prompt?
Thank you. 

<!-- Greeting -->
Hey @rushmj, nice to see you here again! Hope you're doing well.

<!-- Answer -->
To use a local model downloaded from HuggingFace as the provider for Trulens feedback in the RAG Evaluation of the Llama-Index project and customize the prompt, you can follow these steps:

1. **Install Necessary Packages**: Make sure you have all the required packages installed. This includes `torch`, `transformers`, `datasets`, `loralib`, `sentencepiece`, `bitsandbytes`, `accelerate`, `langchain`, `xformers`, and `trulens-eval`.

2. **Set Up the Environment Variable for OpenAI API Key**: Before creating the feedback function, set the `OPENAI_API_KEY` environment variable with your OpenAI API key.

    ```python
    import os
    os.environ["OPENAI_API_KEY"] = "your_openai_api_key_here"
    ```

3. **Create the Feedback Function**: Use the `Feedback` class from `trulens_eval.feedback` to define the qualities of the model you care about. For example, to check if the statement returned by the LLM is relevant to the user's query, you can use the `Feedback_OpenAI` class for setting up a feedback function for query-statement relevance.

    ```python
    from trulens_eval.feedback import Feedback, OpenAI as Feedback_OpenAI
    feedback_openai = Feedback_OpenAI()
    qa_relevance = Feedback(feedback_openai.relevance, name = "Answer Relevance").on_input_output()
    ```

4. **Load the Local Model**: Use the `HuggingFacePipeline.from_model_id` method to load the Alpaca7B model locally. You can specify the model ID, task, and model kwargs such as `temperature`, `top_p`, and `max_length` according to your requirements.

    ```python
    from langchain import HuggingFacePipeline

    local_llm = HuggingFacePipeline.from_model_id(model_id="chavinlo/alpaca-native",
        task="text-generation",
        model_kwargs={"temperature":0.6, "top_p":0.95, "max_length":256})
    ```

5. **Set Up a Chat with Memory**: Use `ConversationChain` and `ConversationTokenBufferMemory` (or `ConversationBufferWindowMemory` for window-based memory) to set up an AI assistant with memory. Customize the prompt template to prime the model as a gardening expert or any other domain you prefer.

    ```python
    from langchain.chains import ConversationChain
    from langchain.memory import ConversationTokenBufferMemory

    token_memory = ConversationTokenBufferMemory(llm = local_llm, max_token_limit=400)

    conversation = ConversationChain(
        llm=local_llm, 
        verbose=True, 
        memory=token_memory
    )

    conversation.prompt.template = '''Your customized prompt here'''
    ```

6. **Wrap with TruChain to Instrument Your Chain**: Use `tru.Chain` to wrap your conversation chain, specifying the app ID and feedbacks.

    ```python
    from trulens_eval import tru_chain
    from trulens_eval.tru import Tru

    tru = Tru()
    tc_conversation = tru.Chain(conversation, app_id='YourAppID', feedbacks=[qa_relevance])
    ```

7. **Interact with Your AI Assistant**: Use the `with_record` method of `tc_conversation` to interact with your AI assistant and pass your custom prompts.

    ```python
    response, record = tc_conversation.with_record(conversation, "Your custom prompt here")
    display(response)
    ```

8. **Adjust the Model and Memory Settings as Needed**: Based on the feedback and the performance of your AI assistant, you may need to adjust the model's `max_length` or the memory's `max_token_limit` to ensure the AI does not run out of tokens in memory.

By following these steps, you can effectively use a local model from HuggingFace as the provider for Trulens feedback in your project and customize the prompt according to your needs. You can refer to the [trulens_eval/examples/expositional/models/alpaca7b_local_llm.ipynb](https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/alpaca7b_local_llm.ipynb) for more details.

Please note that the local model from HuggingFace needs to be compatible with Trulens feedback. This includes the model output format, input preprocessing, API compatibility, error handling, concurrency and performance, and type checking and validation. You can refer to the [trulens_eval/trulens_eval/feedback/provider/hugs.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/hugs.py) for more details.

To download a model from HuggingFace and load it locally for use with Trulens, you can follow the steps provided in the context. Please note that the exact code and steps might vary depending on the specific model you choose from HuggingFace and whether you're using TensorFlow, PyTorch, or another backend supported by Trulens. The provided steps are based on a general approach for a PyTorch model from HuggingFace's Transformers library. You can refer to the [trulens_explain/README.md](https://github.com/truera/trulens/blob/main/trulens_explain/README.md) for more details.

I hope this helps! If you have any further questions, feel free to ask.

## Sources
- [trulens_eval/examples/expositional/models/alpaca7b_local_llm.ipynb](https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/alpaca7b_local_llm.ipynb)
- [trulens_eval/trulens_eval/feedback/provider/hugs.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/hugs.py)
- [trulens_explain/README.md](https://github.com/truera/trulens/blob/main/trulens_explain/README.md)

<!-- Dosu Comment Footer -->
<details open>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>

We don't have this support right now. We can add this to our backlog as a feature request.

@rushmj You can choose to write a custom feedback function as described here: https://www.trulens.org/trulens_eval/evaluation/feedback_implementations/custom_feedback_functions/

You could also extend the base Provider class to create a new provider.

thanks！