Repository: explodinggradients/ragas
Issue: Facing error for evaluate() for Langchain instance LLM and Embedding models
Root cause: Configuration Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

** Facing error with using Langchain wrapped hugging face models**
I am using "Qwen/Qwen2-1.5B-Instruct" llm and "sentence-transformers/all-MiniLM-L6-v2" embedding model instead of default OpenAI model, both the models are initialized with langchain, but i am unable to generate score or use any metric,  getting "ERROR:ragas.executor:Exception raised in Job[0]: TimeoutError()" error.

**Here is the entire code**
from langchain.embeddings import HuggingFaceEmbeddings
from langchain_huggingface.llms import HuggingFacePipeline
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, AutoModel
from ragas.metrics import faithfulness
from ragas import evaluate
from datasets import Dataset 


model_id = "Qwen/Qwen2-1.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=200)
llm_model = HuggingFacePipeline(pipeline=pipe)

embedding_model = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")

data_samples = {
    'question': [
        'What is the capital of France?'
    ],
    'answer': [
        'The capital of France is Paris.'
    ],
    'contexts': [
        ['France is a country in Western Europe. Paris is its capital and most populous city.']
    ],
}

dataset = Dataset.from_dict(data_samples)
results = evaluate(dataset, metrics=[faithfulness], llm=hf, embeddings=embedding_model)

**Here is the snapshot of my issue**
I ma using Colab for implementation so i cannot use bigger
![raga_error](https://github.com/user-attachments/assets/e563ec69-e07a-46de-bfcb-20fac2a5fb59)
 models 


<!-- Greeting -->
Hey @kartik-angadi! I'm here to help you with any bugs, questions, or contributions you might have. Let's tackle this together!

<!-- Answer -->
I found similar issues that might help you resolve the `TimeoutError` when using Langchain-wrapped Hugging Face models with the `evaluate()` function in the `ragas` library.

1. **Timeout Configuration**:
   You need to set the `timeout` parameter in the `RunConfig` and ensure it is passed to both the LLM and the embeddings. Here is an example of how to do it:

   ```python
   from ragas.run_config import RunConfig
   from ragas.llms.base import LangchainLLMWrapper
   from ragas.embeddings.base import LangchainEmbeddingsWrapper
   from langchain_huggingface.llms import HuggingFacePipeline
   from langchain.embeddings import HuggingFaceEmbeddings
   from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
   from ragas.metrics import faithfulness
   from ragas import evaluate
   from datasets import Dataset

   # Create a RunConfig with a specified timeout
   run_config = RunConfig(timeout=60)  # Set timeout to 60 seconds

   # Initialize the LLM with the run_config
   model_id = "Qwen/Qwen2-1.5B-Instruct"
   tokenizer = AutoTokenizer.from_pretrained(model_id)
   model = AutoModelForCausalLM.from_pretrained(model_id)
   pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=200)
   llm_model = HuggingFacePipeline(pipeline=pipe, run_config=run_config)

   # Initialize the embeddings with the run_config
   embedding_model = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
   embedding_wrapper = LangchainEmbeddingsWrapper(embedding_model, run_config)

   # Create the dataset
   data_samples = {
       'question': ['What is the capital of France?'],
       'answer': ['The capital of France is Paris.'],
       'contexts': [['France is a country in Western Europe. Paris is its capital and most populous city.']],
   }
   dataset = Dataset.from_dict(data_samples)

   # Use the evaluate function
   results = evaluate(dataset, metrics=[faithfulness], llm=llm_model, embeddings=embedding_wrapper)
   ```

2. **Known Issues and Limitations**:
   There are known issues with using Langchain-wrapped Hugging Face models in the `ragas` library, such as the lack of support for asynchronous callbacks and specific configurations required for different types of models <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/integrations/langchain.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/embeddings/base.py)</sup>.

3. **Debugging and Logging**:
   Enable debugging and logging options to diagnose the `TimeoutError`. Set the `log_tenacity` parameter in `RunConfig` to `True` and enable the `RAGAS_DEBUG` environment variable:

   ```python
   import os
   os.environ["RAGAS_DEBUG"] = "true"

   run_config = RunConfig(timeout=60, log_tenacity=True)
   ```

   This will provide comprehensive logging and debugging information <sup>[[4]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/run_config.py)</sup><sup>[[5]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/utils.py)</sup>.

By following these steps, you should be able to resolve the `TimeoutError` and successfully use the `evaluate()` function with your Langchain-wrapped Hugging Face models.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*