zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Incorrect Faithfulness Value Mapping
Root cause: Validation Gap

https://github.com/explodinggradients/ragas/blob/e1e05f8844635f46f60564ae6e3ae406e7e070fb/src/ragas/metrics/_faithfulness.py#L153

Shouldn't this mapping instead be:

```python
 verdict_score_map = {"1": 1, "0": 0, "-1": np.nan} 
```

Since the prompt and examples tell the model to return a "-1" verdict if the statement is invalid? 


Oh, yes @pmbaumgartner I would have forgotten to change the mapping after changing the prompt for adaptation. Would you like to raise a PR on this?
Repository: explodinggradients/ragas
Issue: Separate query and context encoding models
Root cause: Unimplemented Feature Gap

[X] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Hi, I would like to use separate models from HuggingFace to prepare embeddings of context and queries. Is there any option to do so in the current version of Ragas?

**Additional context**
A perfect option would be to create two separate arguments (e.g. query_embeddings, context_embeddings, to be able to pass pre-computed embeddings by HuggingFace model into evaluation process).


Hey @dkoterwa We don't have that now. I would love to know why you would like to do that?



Some of the metrics require to create an embedding of question, e.g. Answer Relevance. There are dense retrievers (such as DRAGON) which have separate encoders to produce embeddings for query/context. That's why using them with ragas may not be aligned with the reasoning behind training those retrievers.
Repository: explodinggradients/ragas
Issue: Can ragas use LLMs in Ollama?
Root cause: Interface Contract Mismatch

I have deployed several LLMs in ollama, can I evaluate by LLMs in ollama using RAGAs?


Yes https://docs.ragas.io/en/latest/howtos/customisations/bring-your-own-llm-or-embs.html @Shihui628 

thank you! really enjoy this nice tool

successfully use ragas with ollama by simple code:

`from langchain_community.chat_models import ChatOllama
from langchain_community.embeddings import OllamaEmbeddings

ollama_llm = "mistral"
model = ChatOllama(model=ollama_llm)
embeddings_ollama = OllamaEmbeddings(model="mistral") `

With Ollama I am running into this error, AttributeError: 'Ollama' object has no attribute 'set_run_config'.  I am using llama_index.

@Shihui628 I'm getting this issue for the same
https://github.com/explodinggradients/ragas/issues/1466
Repository: explodinggradients/ragas
Issue: ModuleNotFoundError: No module named 'openai.error'
Root cause: External Dependency Breakage

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Working in Jupyterlab hosted on GCP, I have tried to check if Ragas can help to evaluate MS Azure AI Search and OpenAI.
However I have run into issue immediately during importin Ragas module. 

Ragas version: 0.0.6
Python version: 3.7.12
Openai version: 1.13.3

**Code to Reproduce**
Share code to reproduce the issue
import openai
from datasets import Dataset 
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness

**Error trace**
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/tmp/ipykernel_651/2981097807.py in <module>
     15 from datasets import Dataset
     16 from ragas import evaluate
---> 17 from ragas.metrics import faithfulness, answer_correctness

~/.local/lib/python3.7/site-packages/ragas/metrics/__init__.py in <module>
      1 from ragas.metrics.answer_relevance import AnswerRelevancy, answer_relevancy
----> 2 from ragas.metrics.context_relevance import ContextRelevancy, context_relevancy
      3 from ragas.metrics.faithfulnes import Faithfulness, faithfulness
      4 
      5 __all__ = [

~/.local/lib/python3.7/site-packages/ragas/metrics/context_relevance.py in <module>
     12 
     13 from ragas.metrics.base import Metric
---> 14 from ragas.metrics.llms import openai_completion
     15 
     16 CONTEXT_RELEVANCE = """

~/.local/lib/python3.7/site-packages/ragas/metrics/llms.py in <module>
      6 import backoff
      7 import openai
----> 8 from openai.error import RateLimitError
      9 
     10 openai.api_key = os.environ.get("OPENAI_API_KEY")

ModuleNotFoundError: No module named 'openai.error'

**Expected behavior**
Should work ;) 


**Additional context**
None

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Ok, I see now that this is caused most likely by outdated Ragas version I have available in artifactory. Need to confirm.

After upgrade to python3.10 and latest ragas build everthing is fine.
Repository: explodinggradients/ragas
Issue: Hard coded path when saving filters
Root cause: Architectural Constraint

This hard coded path was breaking the demos for me since `/Users/shahules/Myprojects/ragas/experiments/{filename}.json` didn't exist on my machine. I just replaced it with `{filename}.json` and everything worked.

https://github.com/explodinggradients/ragas/blob/7ba0c28b07cd5742c7067697d898efd7a4638d49/src/ragas/testset/filters.py#L29


Hey @jimgoo damn, that was very poor review from my side, it was an experimental branch that somehow got mixed up. Just fixing it now.
Repository: explodinggradients/ragas
Issue: Allow generating test dataset in different languages
Root cause: Documentation Deficiency

I'm impressed by the quality of the dataset which got generated. Well done and the prompts you use <3

FR: it would be amazing to support the generation of the dataset with different languages. 

We are using german documents, but the generated questions are still english. A MVP would be to generate questions in the same language. The northstar would be to allow specifying the target language, which can be achieved through a simple addition in the prompt.

Hey @manzke It's already supported using automatic language adaptation. 

https://docs.ragas.io/en/stable/howtos/applications/use_prompt_adaptation.html#language-adaptation-for-testset-generation

feel free to reopen the issue if it's what not you're looking for

Awesome! Couldn't find it in the docs. Will test it.
Repository: explodinggradients/ragas
Issue: Inconsistency between documentation and column validation step
Root cause: Documentation Deficiency

Hi,

So based on the updated ragas documentation, it says that `ground_truth` is a `list[list]`. ([here](https://docs.ragas.io/en/stable/howtos/applications/data_preparation.html)). 

However, even if I converted the evaluation dataset to use such a key, I got the following error:

```
ValueError: Dataset feature "ground_truth" should be of type string
```

I did a bit of investigation, and it seems in the step to validate the types of the columns, `ground_truth` is added to the same check as `question` and `answer`, which are expected to be strings. See image and I also link the code line [here](https://github.com/explodinggradients/ragas/blob/746d723b4ca16faf697b048bba70b6fcad2f449a/src/ragas/validation.py#L43).

Therefore, I am a bit confused now. I open a similar issue (#655 ) previously, but I believe I was not clear there.

Am I missing something? 🤔 


![image](https://github.com/explodinggradients/ragas/assets/57979150/7c00cc53-b3c2-41a1-a9f9-a06c18ce5a0b)


I am using `ragas==0.1.3`.

Thanks!!


Hey @lsantosq thanks for sharing this with us. Ground truth in an older version of ragas was list[str] but now we use str. The doc was out of date. Just raised a PR to fix this.

If you think the PR resolves your confusion, feel free to close the issue @lsantosq
Repository: explodinggradients/ragas
Issue: New metrics about answers
Root cause: Unimplemented Feature Gap

In our practical use, we have found that in addition to the existing metrics of RAGAS, we also place great importance on metrics related to the quality of the answers generated by LLM, such as the Semantic Diversity and Lexical Diversity of the answers. These metrics that measure whether the answer is comprehensive and whether it elaborately explains the questions posed by the users are also an important reference for our evaluation. Moreover, we have already implemented this capability based on the existing framework of RAGAS. With Semantic Diversity, We referred to the paper https://arxiv.org/pdf/2205.01497.pdf, and https://arxiv.org/pdf/2307.04626.pdf with Lexical Diversity. Could you please inform me if the community is considering adding metrics to RAGAS? If needed, we are very willing to share our code.



Hi,

It will be great help to the community if you can share the code.

Regards,
Saby

Hey @prime-7 Yes of course, we are always open to adding new metrics that help the community. Please do a PR on it, if you want to chat with me before that feel free to book a [slot](https://calendly.com/shahules/30min) here. 

Just briefly checkout out both papers, would love to chat and know your use-case for the metrics.  @prime-7 

OK, please give me some time, there are some works to be done, such as matching latest version, changing the language of  prompt in English.

Sure @prime-7
Repository: explodinggradients/ragas
Issue: ImportError: cannot import name 'LangchainLLM' from 'ragas.llms'
Root cause: Interface Contract Mismatch

I am using llama2 model for synthetic data generation and evaluation.


''' 
    
    def chat_factory() -> BaseChatModel:

    llm = HuggingFaceHub(
        repo_id="meta-llama/Llama-2-7b-chat-hf",
        task="text-generation",
        huggingfacehub_api_token='#################',
        model_kwargs={
            "max_new_tokens": 512,
            "top_k": 30,
            "temperature": 0.1,
            "repetition_penalty": 1.03,
        },
    )
    chat = ChatHuggingFace(llm=llm)
    return chat
'''

  '''
     from ragas.testset.generator import TestsetGenerator
     from ragas.llms import LangchainLLM
     from ragas.embeddings import embedding_factory
     from langchain_core.language_models import BaseLanguageModel
  
  
    generator_llm=LangchainLLM(llm= chat_factory())
    critic_llm = LangchainLLM(llm= chat_factory())
    embeddings_model = embedding_factory()
  
  
    testset_distribution = {
        "simple": 1.0,
        "reasoning": 0.0,
        "multi_context": 0.0,
        "conditional": 0.0,
    }
  
  
    chat_qa = 0.2
  
  
    test_generator = TestsetGenerator(
        generator_llm=generator_llm,
        critic_llm=critic_llm,
        embeddings_model=embeddings_model,
        testset_distribution=testset_distribution,
        chat_qa=chat_qa,
    )


'''



Ragas version:0.1.2
Python version:3.10.12

i am getting error while running this above snippet


try 

from ragas.llms import LangchainLLMWrapper

I get the same error 
`ImportError: cannot import name 'LangchainLLM' from 'ragas.llms' `

Using `LangchainLLMWrapper` as dropin replacement.
`TypeError: LangchainLLMWrapper.__init__() got an unexpected keyword argument 'llm'`

ragas.version: '0.1.8.dev13+gc5eac53'
langchain '0.1.17'
```
inference_server_url = "http://localhost:8080/v1"
# create vLLM Langchain instance
chat = ChatOpenAI(
    model="meta-llama/Meta-Llama-3-8B",
    openai_api_key="no-key",
    openai_api_base=inference_server_url,
    max_tokens=5,
    temperature=0,
)

# use the Ragas LangchainLLM wrapper to create a RagasLLM instance
vllm = LangchainLLMWrapper(llm=chat)
```

vllm = LangchainLLMWrapper(langchain_llm=chat)

closing this since @mridulBanik112 suggested the right solution. 

you can also just use it as
```py
vllm = LangchainLLMWrapper(chat)
```

to simplify it.
Repository: explodinggradients/ragas
Issue: Are the Context Precision Examples and Calculation Correct?
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
When calculating context precision, don't you need to provide each sentence/statement in the context individually in the prompt rather than all joined together? Right now all of the examples in the prompt for context precision concatenate the context strings into a single string. This means when calculating the context, the denominator is always going to be 1 because there's only 1 document in the context, which is the entire context.

**Code Examples**
https://github.com/explodinggradients/ragas/blob/c7302724a6203966c7d0c16db8f09ed5ce2a1a75/src/ragas/metrics/_context_precision.py#L19-L54

**Additional context**
Anything else you want to share with us? 


Hey @pmbaumgartner context precision is computed over multiple chunks. So denominator here is the number of retrieved contexts which can be one or more as you set in your RAG.

That makes sense for the calculation, but I'm still a bit lost. I think I'm confused because that data structure is _different_ than what's provided in the few-shot examples. If it were the same as the calculation, I would expect to see one "example" per document/sentence in the context, not the entire context concatenated together. Does that make sense? 

I'm not sure it actually matters for the performance of the LLM on this task, but the mismatch here is a bit confusing.

Can you check the calculation example of [context_precision](https://docs.ragas.io/en/stable/concepts/metrics/context_precision.html) and see if the code makes sense to you then?
Repository: explodinggradients/ragas
Issue: What are the specific inputs and corresponding outputs used in the evaluation?
Root cause: Documentation Deficiency

How should I obtain specific inputs and outputs? Because I encountered strange results, I wanted to find the reason from specific user inputs and LLM outputs. The model I am using is GPT-3.5 packaged in Langchain.

Hey @hanlv15 this is totally possible if you're using any tracing tools. for example https://docs.ragas.io/en/stable/howtos/integrations/index.html and https://docs.ragas.io/en/stable/howtos/applications/tracing.html
Repository: explodinggradients/ragas
Issue: Evaluation of RAG pipeline using RAGAS
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I want to evaluate the RAG pipeline using RAGAS. Using below code as reference to implement the metrics, but I don't understand that without passing explicit OpenAI key, contexts how are the metrics getting calculated?

**Code Examples**
from langchain.document_loaders import WebBaseLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

# load the Wikipedia page and create index
loader = WebBaseLoader("https://en.wikipedia.org/wiki/New_York_City")
index = VectorstoreIndexCreator().from_loaders([loader])

# create the QA chain
llm = ChatOpenAI()
qa_chain = RetrievalQA.from_chain_type(
    llm, retriever=index.vectorstore.as_retriever(), return_source_documents=True
)

# testing it out
question = "How did New York City get its name?"
result = qa_chain({"query": question})
result["result"]

# output
# 'New York City got its name in 1664 when it was renamed after the Duke of York, who later became King James II of England. The city was originally called New Amsterdam by Dutch colonists and was renamed New York when it came under British control.'

from ragas.metrics import faithfulness, answer_relevancy, context_relevancy, context_recall
from ragas.langchain import RagasEvaluatorChain


# make eval chains
eval_chains = {
    m.name: RagasEvaluatorChain(metric=m)
    for m in [faithfulness, answer_relevancy, context_relevancy]
}


# evaluate
for name, eval_chain in eval_chains.items():
    print('name:',name)
    print('eval_chain:',eval_chain)
    score_name = f"{name}_score"
    print(f"{score_name}: {eval_chain(result)[score_name]}")

**Additional context**
Anything else you want to share with us? 


@dosu-bot can you help here?
Repository: explodinggradients/ragas
Issue: Mistral-8x7b answer_correctness metric prompt fix
Root cause: Validation Gap

[X ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Is there any prompt modification such that Mistral-8x7b does not give nan when evaluating answer_correctness metric?

**Code Examples**
I do not have any code snippets as answer_correctness does not prompt any warning or message, it just outputs None.

**Additional context**
The exact model I'm using is: mistralai/Mixtral-8x7B-Instruct-v0.1


Have you tried setting the raise_exceptions parameter to True?
Repository: explodinggradients/ragas
Issue: Support positively grading 'I don't know' answers when the context is not relevant
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Hi there,
I read all metric descriptions and realized that there is no support for the following case:

Suppose the LLM is asked a question, and there is no way for the answer to be retrieved by the RAG because it is not documented, but the ground truth does exist. 
In this case I would want the LLM to say "I don't know", and to ignore the non-relevant context which is retrieved.
There is no metric that would evaluate this case positively (correct me if I'm wrong).

**Why is the feature important for you?**
I would like to use the package for analyzing an LLM I have, but it does not cover the case above.

**Additional context**
I'll give you an example:

Suppose the use case of the LLM is to answer questions about documentation of a product.
The user asks "Can this product do {something that it cannot do}?
The context retrieved will never include the ground truth, which is 'No, it is not supported'.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hey @mayalinetsky-kryon currently most of these cases return NaN. Because we think that there is no point in scoring when the system is refusing to answer. Check out the same with faithfulness, etc. 

Do you have anything other in mind?

I was thinking of some conditional metric:
If the context is not relevant, or the answer cannot be found in context, AND the LLM's response is 'IDK'\'couldn't find an answer', then a score would be high, because that's the best case scenario.
Repository: explodinggradients/ragas
Issue: why context_recall=0,but context_precision=1?
Root cause: Algorithmic Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
why context_recall=0,but context_precision=1?Shouldn't this be 0?


**Code Examples**
ds = Dataset.from_dict({"question": question_list,
                        "contexts": contexts_list,
                        "answer": answer_list,
                        "ground_truth": ground_truth_list}
                       )

result = evaluate(
    ds,
    metrics=[
        context_recall,
        context_precision,
        faithfulness,
        answer_relevancy,
        # answer_similarity,
        # answer_correctness,
    ], raise_exceptions=False
)
**Additional context**
Anything else you want to share with us?
Repository: explodinggradients/ragas
Issue: Context_recall returns only nan when using gpt-4-0125-preview and gpt-4-1106-preview
Root cause: Interface Contract Mismatch

**Describe the bug**
When using gpt-4-0125-preview and gpt-4-1106-preview as llm I obtain only nan values for the context_recall metrics.

Ragas version: 0.1.5
Python version: 3.11.7

**Code to Reproduce**
```
result_ragas = evaluate(
    data,
    metrics=[
        context_recall
    ],
    llm=ChatOpenAI(model_name="gpt-4-0125-preview", temperature=0.0)
)



same issue for me!

The problem comes from the parsing of the output. gpt-4-0125-preview always responds with:
{"classification": []}

Changing this line in _context_recall.py
```response = response if isinstance(response, list) else [response]```
with
```response = response['classification']```
solves the problem

> The problem comes from the parsing of the output. gpt-4-0125-preview always responds with: {"classification": []}
> 
> Changing this line in _context_recall.py `response = response if isinstance(response, list) else [response]` with `response = response['classification']` solves the problem

no, it can not be solved, because the dict generated by GPT always not strictly follow the example, and randomly generated, so the context recall and context precision should not be generated in this manner

Hey @frischzenger We are about to merge a release that would fix most of the NaN related issues. #807
Repository: explodinggradients/ragas
Issue: ModuleNotFoundError: No module named 'ragas.testset'; 'ragas' is not a package
Root cause: Environment Incompatibility

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Just trying to import the package while pip show ragas correctly shows the package

Ragas version: 0.1.4
Python version: 3.10.11

**Code to Reproduce**
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context

**Error trace**
Traceback (most recent call last):
  File "/Users/x/Documents/x/test/ragas.py", line 7, in <module>
    import ragas
  File "/Users/x/Documents/x/test/ragas.py", line 11, in <module>
    from ragas.testset.generator import TestsetGenerator
ModuleNotFoundError: No module named 'ragas.testset'; 'ragas' is not a package

**Additional context**
I just installed the package (working in VSCode)

I get this error with a fresh install of Python 3.11. How did you resolve this?

```
Traceback (most recent call last):
  File "/usr/local/google/home/jeremyafisher/Code/rag/models/ragas.py", line 7, in <module>
    from ragas.testset.generator import TestsetGenerator
  File "/usr/local/google/home/jeremyafisher/Code/rag/models/ragas.py", line 7, in <module>
    from ragas.testset.generator import TestsetGenerator
ModuleNotFoundError: No module named 'ragas.testset'; 'ragas' is not a package
```

Hi I am facing the same issue. 
Python 3.10.12 and Ragas 0.1.10
Traceback (most recent call last):
  File "/home/ubuntu/scp-analyzer/rag-old/ragas.py", line 2, in <module>
    from ragas.testset.generator import TestsetGenerator
  File "/home/ubuntu/scp-analyzer/rag-old/ragas.py", line 2, in <module>
    from ragas.testset.generator import TestsetGenerator
ModuleNotFoundError: No module named 'ragas.testset'; 'ragas' is not a package

same issue on python 3.11