Repository: explodinggradients/ragas
Issue: Using Local Model Instead of API Key in Langchain
Root cause: Documentation Deficiency

I'm currently exploring the Langchain library and want to configure it to use a local model instead of an API key. Could you guide me on how to achieve this?

For instance, in my current setup, I'm utilizing the ChatOpenAI class from langchain.chat_models and the LangchainLLM wrapper from ragas.llms. My goal is to replace the reliance on an API key with a local model.

```py
from langchain.chat_models import ChatOpenAI
from ragas.llms import LangchainLLM

inference_server_url = "http://localhost:8080/v1"

# create vLLM Langchain instance
chat = ChatOpenAI(
    model="HuggingFaceH4/zephyr-7b-alpha",
    openai_api_key="no-key",
    openai_api_base=inference_server_url,
    max_tokens=5,
    temperature=0,
)

# use the Ragas LangchainLLM wrapper to create a RagasLLM instance
vllm = LangchainLLM(llm=chat)
```

Could you provide guidance on how to adapt this code to use a locally stored model instead of the API key? I've already downloaded the model files locally.

Thanks for your assistance!

I was also wondering how to use the local model.

I'm using huggingface'smodel without OpenIA KEY.

This my Chat implementation :

```python
from langchain_community.chat_models.huggingface import ChatHuggingFace
from langchain_community.llms import HuggingFaceHub
from langchain_core.language_models.chat_models import BaseChatModel

def chat_factory() -> BaseChatModel:

    llm = HuggingFaceHub(
        repo_id="HuggingFaceH4/zephyr-7b-beta",
        task="text-generation",
        model_kwargs={
            "max_new_tokens": 512,
            "top_k": 30,
            "temperature": 0.1,
            "repetition_penalty": 1.03,
        },
    )
    chat = ChatHuggingFace(llm=llm)
    return chat

```

and I use it 

```python
# Add custom llms and embeddings
generator_llm = LangchainLLM(llm= chat_factory())
critic_llm = LangchainLLM(llm= chat_factory())
embeddings_model = embeddings_factory()

# Change resulting question type distribution
testset_distribution = {
    "simple": 1.0,
    "reasoning": 0.0,
    "multi_context": 0.0,
    "conditional": 0.0,
}

# percentage of conversational question
chat_qa = 0.2


test_generator = TestsetGenerator(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings_model=embeddings_model,
    testset_distribution=testset_distribution,
    chat_qa=chat_qa,
)
```

@youen Thank you for providing the information! I'm very grateful for the approach without requiring an OpenAI key. I'm still curious about how to invoke the local model without an OpenAI key on GitHub.

hey @Tenshi0x0 are you still facing this issue? there is another way to do this if your installing from source. Let me know if you want to me share it

> hey @Tenshi0x0 are you still facing this issue? there is another way to do this if your installing from source. Let me know if you want to me share it

@jjmachan Yes! Thank you so much for mentioning this issue in my question and providing your solutions.

> hey @Tenshi0x0 are you still facing this issue? there is another way to do this if your installing from source. Let me know if you want to me share it

Yes!

Hi guys, great you see our users helping each others out.

1. Install ragas from source using `pip install git+https://github.com/explodinggradients/ragas.git`
2. Refer to on how to use local models using langchain wrapper https://docs.ragas.io/en/latest/howtos/customisations/llms.html
3. If you're using a metric that requires embeddings, set your embeddings using https://docs.ragas.io/en/latest/howtos/customisations/embeddings.html

@Tenshi0x0 feel free to pitch in and continue the conversation.

> Hi guys, great you see our users helping each others out.
> 
> 1. Install ragas from source using `pip install git+https://github.com/explodinggradients/ragas.git`
> 2. Refer to on how to use local models using langchain wrapper https://docs.ragas.io/en/latest/howtos/customisations/llms.html
> 3. If you're using a metric that requires embeddings, set your embeddings using https://docs.ragas.io/en/latest/howtos/customisations/embeddings.html
> 
> @Tenshi0x0 feel free to pitch in and continue the conversation.

Docs can now be found here --> https://docs.ragas.io/en/latest/howtos/customisations/bring-your-own-llm-or-embs.html

> I'm using huggingface'smodel without OpenIA KEY.
> 
> This my Chat implementation :
> 
> ```python
> from langchain_community.chat_models.huggingface import ChatHuggingFace
> from langchain_community.llms import HuggingFaceHub
> from langchain_core.language_models.chat_models import BaseChatModel
> 
> def chat_factory() -> BaseChatModel:
> 
>     llm = HuggingFaceHub(
>         repo_id="HuggingFaceH4/zephyr-7b-beta",
>         task="text-generation",
>         model_kwargs={
>             "max_new_tokens": 512,
>             "top_k": 30,
>             "temperature": 0.1,
>             "repetition_penalty": 1.03,
>         },
>     )
>     chat = ChatHuggingFace(llm=llm)
>     return chat
> ```
> 
> and I use it
> 
> ```python
> # Add custom llms and embeddings
> generator_llm = LangchainLLM(llm= chat_factory())
> critic_llm = LangchainLLM(llm= chat_factory())
> embeddings_model = embeddings_factory()
> 
> # Change resulting question type distribution
> testset_distribution = {
>     "simple": 1.0,
>     "reasoning": 0.0,
>     "multi_context": 0.0,
>     "conditional": 0.0,
> }
> 
> # percentage of conversational question
> chat_qa = 0.2
> 
> 
> test_generator = TestsetGenerator(
>     generator_llm=generator_llm,
>     critic_llm=critic_llm,
>     embeddings_model=embeddings_model,
>     testset_distribution=testset_distribution,
>     chat_qa=chat_qa,
> )
> ```

@youen Thank you for providing this information. However, to better understand the context and provide a more comprehensive solution, would it be possible for you to share the full script? 

try this:

https://github.com/Microflow-IO/modsecurity-for-anylog/blob/main/ragas_local_llm.ipynb