Repository: explodinggradients/ragas
Issue: RAGAS returning NAN as output for context precision with hugging face models and embeddings
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I am trying to use hugging face open source llm model and embedding, ragas metric return NAN as output . Tried with various models still hitting the same issue

Ragas version: 0.1.5
Python version: 3.10

**Code to Reproduce**
Share code to reproduce the issue

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.chat_models.huggingface import ChatHuggingFace
from langchain_community.llms import HuggingFaceHub

embedding_model = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")
hugging_llm = HuggingFaceHub(
    repo_id="HuggingFaceH4/zephyr-7b-beta",
    task="text-generation",
    model_kwargs={
        "max_new_tokens": 512,
        "top_k": 30,
        "temperature": 0.1,
        "repetition_penalty": 1.03,
    },
)


llm = LangchainLLMWrapper(hugging_llm)
embeddings = LangchainEmbeddingsWrapper(embedding_model)

response_dataset = Dataset.from_dict({
    "question" : ["What is the capital of India"],
    "answer" : ["Capital of India is Delhi"],
    "contexts" :  [["Capital of India is Delhi"]],
    "ground_truth" : ["Capital of India is Delhi"]
    })

from ragas import evaluate
from ragas.metrics import context_precision
result = evaluate(
    dataset=response_dataset,
    llm=llm,
    embeddings=embeddings,
    metrics=[
        context_precision,
    ],
)
print(result)

**Error trace**
Evaluating:   0%|                                                                                   | 0/1 [00:00<?, ?it/s]Invalid response format. Expected a list of dictionaries with keys 'verdict'
Evaluating: 100%|███████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.37it/s]
/home/sdp/miniconda3/envs/vdms-langchain/lib/python3.10/site-packages/ragas/evaluation.py:276: RuntimeWarning: Mean of empty slice
  value = np.nanmean(self.scores[cn])
**Expected behavior**

Valid output for context precision



**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I was encountering the same error when working with a custom huggingface model. 

For me the issue was, that the temperature had been overwritten by ragas. 
The model I am using does not respond with the overridden temperature so I had to change the code 
concerning the temperature directly in ragas.
 

> I was encountering the same error when working with a custom huggingface model.
> 
> For me the issue was, that the temperature had been overwritten by ragas. The model I am using does not respond with the overridden temperature so I had to change the code concerning the temperature directly in ragas.

Can you point me to the file where you made the change ?

sure 

[https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py](url)

there you can change the temperature manually.

> sure
> 
> [https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py](url)
> 
> there you can change the temperature manually.

Thanks, and may I know what model you used ?

the model I am using: 
[https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1](url)

Do you understand assessment indicator ==Context Precision==, please?