Repository: explodinggradients/ragas
Issue: Error using HuggingFacePipelines from Langchain to perform evaluations. 
Root cause: Interface Contract Mismatch

**Describe the bug**
Trying to use HuggingFacePipelines from Langchain to perform evaluations. 

Ragas version: 0.0.21
Python version: 3.9

**Code to Reproduce**
Define the HF Pipeline
```
tokenizer = AutoTokenizer.from_pretrained(hugging_face_model, use_auth_token=access_token)

llm_model = AutoModelForCausalLM.from_pretrained(hugging_face_model, #meta-llama/Llama-2-13b-chat-hf
                                                     load_in_4bit=True,
                                                     device_map='balanced_low_0',
                                                     torch_dtype=torch.float16,
                                                     low_cpu_mem_usage=True,
                                                     use_auth_token=access_token
                                                    )
max_len = 8192
llm_task = "text-generation"
T = 0.1

llm_pipeline = pipeline(
    task=llm_task,
    model=llm_model, 
    tokenizer=tokenizer, 
    max_length=max_len,
    temperature=T,
    top_p=0.95,
    repetition_penalty=1.15
)

llm = HuggingFacePipeline(pipeline=llm_pipeline)
```

Setup Ragas
```
from ragas.llms import LangchainLLM
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

ragas_llm = LangchainLLM(llm=llm)

faithfulness.llm = ragas_llm
answer_relevancy.llm = ragas_llm
context_precision.llm = ragas_llm
context_recall.llm = ragas_llm
harmfulness.llm = ragas_llm
```

Run evaluation
```
from ragas import evaluate

result = evaluate(dataset, metrics=[faithfulness])
```


**Error trace**
```
  0%|                                                     | 0/1 [00:00<?, ?it/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[31], line 1
----> 1 result = evaluate(dataset, metrics=[faithfulness])

File ~/.local/lib/python3.9/site-packages/ragas/evaluation.py:105, in evaluate(dataset, metrics, column_map)
    103         binary_metrics.append(metric.name)
    104     print(f"evaluating with [{metric.name}]")
--> 105     scores.append(metric.score(dataset).select_columns(metric.name))
    107 # log the evaluation event
    108 metrics_names = [m.name for m in metrics]

File ~/.local/lib/python3.9/site-packages/ragas/metrics/base.py:76, in Metric.score(self, dataset, callbacks)
     74 with trace_as_chain_group(f"ragas_{self.name}", callback_manager=cm) as group:
     75     for batch in tqdm(self.get_batches(len(dataset))):
---> 76         score = self._score_batch(dataset.select(batch), callbacks=group)
     77         scores.extend(score)
     79 return dataset.add_column(f"{self.name}", scores)

File ~/.local/lib/python3.9/site-packages/ragas/metrics/_faithfulness.py:88, in Faithfulness._score_batch(self, ds, callbacks, callback_group_name)
     84     human_prompt = LONG_FORM_ANSWER_PROMPT.format(question=q, answer=a)
     85     prompts.append(ChatPromptTemplate.from_messages([human_prompt]))
---> 88 result = self.llm.generate(prompts, callbacks=batch_group)
     89 list_statements: list[list[str]] = []
     90 for output in result.generations:
     91     # use only the first generation for each prompt

File ~/.local/lib/python3.9/site-packages/ragas/llms/langchain.py:199, in LangchainLLM.generate(self, prompts, n, temperature, callbacks)
    197     self.llm.model_kwargs = {"temperature": temperature}
    198 else:
--> 199     self.llm.temperature = temperature
    201 if self.llm_supports_completions(self.llm):
    202     return self._generate_multiple_completions(prompts, n, callbacks)

File /usr/local/lib/python3.9/site-packages/pydantic/main.py:357, in pydantic.main.BaseModel.__setattr__()

ValueError: "HuggingFacePipeline" object has no field "temperature"
```


**Expected behavior**
Following this [document](https://docs.ragas.io/en/latest/howtos/customisations/llms.html), expect the evaluation to use HuggingFacePipelines from Langchain for evaluation


**Additional context**
No additional context to add, please let me know if you need any more details.


@tinomaxthayil Can you check if this is something that interests you? I can help with the solution. 

I can check this issue.

Hi, I am facing the same error. Are there any updates regarding this issue please?

Facing the same issue when trying to use other LLMs from langchain. Will be glad to hear updates!
Thanks!

Same problem here, so I removed the two lines in [`src/ragas/llms/langchain.py`](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/langchain.py)

In the `generate` function:
```python
    def generate(
        self,
        prompts: list[ChatPromptTemplate],
        n: int = 1,
        temperature: float = 1e-8,
        callbacks: t.Optional[Callbacks] = None,
    ) -> LLMResult:
        # set temperature to 0.2 for multiple completions
        temperature = 0.2 if n > 1 else 1e-8
        if isBedrock(self.llm) and ("model_kwargs" in self.llm.__dict__):
            self.llm.model_kwargs = {"temperature": temperature}
        elif isAmazonAPIGateway(self.llm) and ("model_kwargs" in self.llm.__dict__):
            self.llm.model_kwargs = {"temperature": temperature}
        else: #remove
            self.llm.temperature = temperature #remove

        ......
```
Removed the two lines below:
```python
else:
    self.llm.temperature = temperature
```

And it works for using the `context_precision`, `context_recall`, `faithfulness`. Still not working to use the `answer_relevancy` metric. This metric seems to still utilize OpenAI's embedding or something else; I'm not entirely sure.

When I use the `answer_relevancy`, I receive the following error message:
```
> Traceback (most recent call last):
>   File "/hokken/Chinese-LLaMA-Alpaca-2/scripts/langchain/langchain_RAGAs_test.py", line 197, in <module>
>     result = evaluate(
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/ragas/evaluation.py", line 101, in evaluate
>     scores.append(metric.score(dataset).select_columns(metric.name))
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/ragas/metrics/base.py", line 76, in score
>     score = self._score_batch(dataset.select(batch), callbacks=group)
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/ragas/metrics/_answer_relevance.py", line 136, in _score_batch
>     cosine_sim = self.calculate_similarity(question, gen_questions)
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/ragas/metrics/_answer_relevance.py", line 145, in calculate_similarity
>     question_vec = np.asarray(self.embeddings.embed_query(question)).reshape(1, -1)
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/langchain/embeddings/openai.py", line 585, in embed_query
>     return self.embed_documents([text])[0]
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/langchain/embeddings/openai.py", line 556, in embed_documents
>     return self._get_len_safe_embeddings(texts, engine=engine)
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/langchain/embeddings/openai.py", line 432, in _get_len_safe_embeddings
>     response = embed_with_retry(
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/langchain/embeddings/openai.py", line 107, in embed_with_retry
>     return embeddings.client.create(**kwargs)
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/openai/resources/embeddings.py", line 106, in create
>     return self._post(
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/openai/_base_client.py", line 1088, in post
>     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/openai/_base_client.py", line 853, in request
>     return self._request(
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/openai/_base_client.py", line 916, in _request
>     return self._retry_request(
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/openai/_base_client.py", line 958, in _retry_request
>     return self._request(
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/openai/_base_client.py", line 916, in _request
>     return self._retry_request(
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/openai/_base_client.py", line 958, in _retry_request
>     return self._request(
>   File "/root/anaconda3/envs/suisei/lib/python3.9/site-packages/openai/_base_client.py", line 930, in _request
>     raise self._make_status_error_from_response(err.response) from None
> openai.RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}
```




Actually done something similar, I work with some wrapper around that object, so instead of removing this exact two lines I modified them to set the temperature in some parameters attribute.

I wonder though if it is something we can have some more flexibility with

Alternatively, we can retain the `else` condition and make some minor modifications to ensure that our model can get `temperature` by the conditions.


Replace `self.llm.temperature = temperature` with `self.llm.model_kwargs = {"temperature": temperature}`
```python
    def generate(
        self,
        prompts: list[ChatPromptTemplate],
        n: int = 1,
        temperature: float = 1e-8,
        callbacks: t.Optional[Callbacks] = None,
    ) -> LLMResult:
        # set temperature to 0.2 for multiple completions
        temperature = 0.2 if n > 1 else 1e-8
        if isBedrock(self.llm) and ("model_kwargs" in self.llm.__dict__):
            self.llm.model_kwargs = {"temperature": temperature}
        elif isAmazonAPIGateway(self.llm) and ("model_kwargs" in self.llm.__dict__):
            self.llm.model_kwargs = {"temperature": temperature}
        else: 
            #self.llm.temperature = temperature 
            self.llm.model_kwargs = {"temperature": temperature} #modified

        ......
```


In this way, our model can still get the `temperature` from this condition:
```python
# set temperature to 0.2 for multiple completions
        temperature = 0.2 if n > 1 else 1e-8
```

And, here is my code snippet for constructing the pipeline:
```python
pipeline = pipeline(
        "text-generation",
        model=model_4bit,
        tokenizer=tokenizer,
        use_cache=True,
        device_map="auto",
        max_length=4096,
        do_sample=True,
        num_return_sequences=1,
        top_k= 30,
        top_p= 0.9,
        repetition_penalty= 1.1,
        #temperature=0.2
)

#llm = HuggingFacePipeline(pipeline=pipeline) 
llm = HuggingFacePipeline(pipeline=pipeline, model_kwargs={'temperature':0.2}) #modified
```

I added the `model_kwargs={'temperature':0.2}` to the HuggingFacePipeline() and remove the `temperature` from the pipeline().