zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: QA Chains and Bring Your Own LLMs and Embeddings
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
How can I evaluate my RAG (Retrieval-Augmented Generation) built with Transformers and using embeddings from Hugging Face? In the documentation, I see how to evaluate with your own LLM and your embeddings, as well as how to integrate RAGs with Chain, but not how to do both at the same time. I don't want to use the OpenAI API.

**Code Examples**
import transformers, torch
from transformers import pipeline
from langchain import HuggingFacePipeline
generate_text = pipeline(
    model=model, tokenizer=tokenizer,
    return_full_text=True,
    task='text-generation',
    do_sample=True,
    top_p=0.98,
    temperature=0.4,
    max_new_tokens=512,
    repetition_penalty=1.1  
)

llm = HuggingFacePipeline(pipeline=generate_text)
qa = RetrievalQA.from_chain_type(
    llm=llm,chain_type='stuff',
    retriever=vectorstore.as_retriever(search_kwargs={"k":2})
)

embed_model_id = "distiluse-base-multilingual-cased-v1"

device = f"cuda:{cuda.current_device()}" if cuda.is_available() else "cpu"

embed_model = HuggingFaceEmbeddings(
    model_name=embed_model_id,
    model_kwargs={"device": device},
    encode_kwargs={"device": device, "batch_size": 32}
)



Yes, ofc.
please checkout https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html

Feel free to open the issue if it's not working for you.


I continue without knowing how to do it
Repository: explodinggradients/ragas
Issue: Errors in passing docstore while generating data
Root cause: Validation Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I am using this to load my text (as dataframe loader). I am using Azure open AI for my llm and embeddings. I am not using an wrapper as mentioned in the documents
```

documents = loader.load()

generator = TestsetGenerator(generator_llm=client,critic_llm=client,embeddings=client2,docstore = documents)

```
```
**Code Examples**
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[51], line 8
      2 documents = loader.load()
      4 generator = TestsetGenerator(generator_llm=client,critic_llm=client,embeddings=client2,docstore = documents)
----> 8 testset = generator.generate(documents, raise_exceptions=False, with_debugging_logs=False,
      9                                                  distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}) 

File ~/.local/lib/python3.10/site-packages/ragas/testset/generator.py:223, in TestsetGenerator.generate(self, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    221 if run_config is None:
    222     run_config = RunConfig(max_retries=15, max_wait=90)
--> 223 self.docstore.set_run_config(run_config)
    225 # init filters and evolutions
    226 for evolution in distributions:

AttributeError: 'list' object has no attribute 'set_run_confi

```
**Additional context**
Anything else you want to share with us?
Repository: explodinggradients/ragas
Issue: Politely ask about the roadmap
GitHub labels: enhancement

question

**Your Question**
Hi, team. I learned about the other evaluation projects and I'm curious about your roadmap: do you have any plan about the tracking, for example, the RAG methods contain retrival, rank and generation, if ragas can track these(including all the retrival context) and display to the users, that will be very helpful.

**Additional context**
https://github.com/Arize-ai/phoenix, other project to refer to.

Thanks very much


Hey bro @muruan01 thanks for the FR, I am marking it to consider for future releases. We have a discord community in which we will plan monthly office hours to plan roadmaps.
Repository: explodinggradients/ragas
Issue: erorr on windows
Root cause: Environment Incompatibility

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

i ran following code of ragas on windows 10


Ragas version: 0.1.4
Python version: 3.9

**Code to Reproduce**
``
from datasets import Dataset 
import os
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness

os.environ["OPENAI_API_KEY"] = "your-openai-key"

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset,metrics=[faithfulness,answer_correctness])
score.to_pandas()
``

**Error trace**
RunTime error: event loop is closed.





Facing the same issue

facing the same issue

Facing the same issue

I encountered the same problem in the following environment:
Operating system: win 11
python version: 3.8
ragas version: 0.1.8

When I changed the python version to 3.10, the problem was solved.

Is this still an issue with latest ragas version?
Repository: explodinggradients/ragas
Issue: RAGAs Async or Multiprocessing
Root cause: Documentation Deficiency

[X] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

Hello,

Would just like to ask if there are examples of RAGAs being used in an Async or Multiprocessing context. I've tried experimenting with both and I keep encountering the `_thread.Rlock` error. I've also tried enabling the `is_async` flag but I'm not sure if I'm using it correctly (it doesn't work with `await`).

Can you share any examples/sample code using RAGAs with asyncio, multiprocessing, or multithreading. My understanding might be incorrect and any clarification will help. Thanks!

ragas==0.1.4
Repository: explodinggradients/ragas
Issue: Low quality generated synthetic dataset
Root cause: Algorithmic Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
When I try to generate a synthetic dataset with my docs, I mostly get the same questions/answers (from 20 questions/answers, 12 are the same), which leads to a dataset that is practically useless.

**Code Examples**
testset = generator.generate_with_langchain_docs(documents, test_size=20, distributions={simple: 0.5, reasoning: 0.5})


Hey @WoutDeRijck Can you explain some of the issues you have seen with the generations? 

Like I said, when trying to generate questions, say for example 20 questions/answers, I get 12 the same questions and another 8 of the same question (identical), so practically I end up with two different questions.

That is weird @WoutDeRijck can you share your code? So that I can try reproduce.

Hey @WoutDeRijck were you able to solve this? I am also facing this same issue!


hey @Nandakishore-Thekkadathu , since this a duplicate and closing this, lets chat in the other one 🙂
Repository: explodinggradients/ragas
Issue: Negative faithfulness result
Root cause: Algorithmic Error

I got the result for faithfulness as -1.5 for one of the rows of my dataset even though its defined scale is 0 - 1. I am currently using ragas version 0.0.9.  
This did not happen when I previously ran the version 0.0.7 on the same dataset. 

![image](https://github.com/explodinggradients/ragas/assets/56498312/819e63fc-6ca8-48de-9bf3-e6213baaf8f3)

Following is the piece of code I used : 
```
gpt3 = ChatOpenAI()
faithfulness_gpt3 = Faithfulness(
    name="faithfulness_gpt3", llm=gpt3, batch_size=3
)
subset = df.iloc[:3]
hg_dataset_1 = Dataset(pa.Table.from_pandas(subset))
result = evaluate(hg_dataset_1, metrics=[faithfulness_gpt3, context_relevancy, answer_relevancy])```


Thanks for reporting this issue @PseudoKush . This is very intriguing, we also haven't observed such results before. I know this would be difficult but if there is any way for you to share the row data, I can try and reproduce the issue. 

I have used my own dataset that was created on a list of Q&A I did within my LLM pipeline. I am not sure how can I share that since it is not available on platforms like Hugging face. What do you suggest to share the data for the particular row ?

Update on the issue : I re-ran the cell (without changing version or anything else) couple of times and score came back in the defined range of 0 - 1. 

If you wish to share the row you can do it as a JSON file. `dataset[idx].to_json("...")` 
This could have been caused due to inherent stochastic nature of LLM generations.

[value_anamoly.csv](https://github.com/explodinggradients/ragas/files/12228392/value_anamoly.csv)

I have attached the csv format for the particular row. It contains question, answer, context. Using the below command, I transformed it into required form:
``` 
df["contexts"] = df.apply(lambda x: x['contexts'][58:].split('passage: ')[1:], axis=1)
hg_dataset_1 = Dataset(pa.Table.from_pandas(df))
```

Thanks a lot, @PseudoKush . I was able to reproduce the error. We will raise a fix soon #80
Repository: explodinggradients/ragas
Issue: Context_relevancy metric returns 1 unexpectedly
Root cause: Algorithmic Error

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I noticed that when my RAG system can not retrieve a relevant context for a question from my evaluation set, ie the returned context is empty or just ['N'], the computed context_relevancy by the library is 1.

Ragas version:
ragas==0.1.5

Python version:

**Code to Reproduce**
```
dct = {'question': 'What is the role of WWSE Finance?',
 'answer': '1. Based on the context provided, there is no information given about a company or entity named "WWSE Finance." Therefore, I cannot provide a detailed step-by-step answer about its role.\n2. Without further context or information, it is impossible to determine what WWSE Finance does, what industry it operates in, or what responsibilities and functions it has.\n3. In order to provide an accurate answer, I would need more context or information about WWSE Finance, such as its mission statement, services offered, or relevant news articles.\n\nIn summary, I don\'t know the role of WWSE Finance because there is not enough information provided in the context.',
 'contexts': ['N'],
 'ground_truth': "WWSE Finance supports internal studios in their creation of great games and improving the company's bottom line by analyzing and predicting future scenarios or by recording and reporting past financial transactions. Its aim is to play an integral part in the internal studio business, whether through scenario planning with a studio team or managing month's bills or expenses."}

ds = Dataset.from_dict(dct)

fiqa_eval = DatasetDict({"baseline": ds})

chat_model = ChatDatabricks(endpoint=...)

from ragas import evaluate
result = evaluate(fiqa_eval["baseline"], metrics=metrics, embeddings=fast_embeddings, llm = chat_model)
result

```





**Error trace**

**Expected behavior**
I expect that in these cases the context_relevancy returns as 0 because technically there's no retrieved context. 


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hey @lalehsg Context_relevancy metric is deprecated and we don't maintain it anymore. As you must have seen the warning. 

got it! thanks!

@shahules786 I still see, context relevancy in [documentation](https://docs.ragas.io/en/latest/concepts/metrics/context_relevancy.html) may I know in which version we are going to deprecate context relevancy metric
Repository: explodinggradients/ragas
Issue: Generate Synthetic Dataset from pinecone index
Root cause: Unimplemented Feature Gap

As an RAG developer, I already have a huge data ingestion pipeline managing data ingestion for me in the Pinecone. Since ETL processes are computationally expensive, one would want to refrain from redoing them. However, I now want to generate questions from the chunked and stored documents within my Pinecone index. But ragas takes only langachain documents or llamaindex documents/nodes. How to approach this?

Hey @rnbokade This is something we might introduce in the near future. There are some challenges associated with it, including bypassing the chunk size bias. For example, if we create questions out of predefined chunks the questions will be mostly of poor quality and will be obviously biased towards the chunk size you have defined. This will prevent ragas from forming high-quality QA pairs.

For now I can create a preliminary docstore implementation with add , similaroty search, fetch and all those abstract methods. Maybe we can fine tune it once people start using it.

Hey @rnbokade that makes sense. Would love to see a PR if you are able to do it.
Repository: explodinggradients/ragas
Issue: Faithfulness return NaN when it should be zero
Root cause: Algorithmic Error

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
when no context is retrieved and the llm answer starts with "i'm sorry..." or "i'm unable...", the metric score become NaN
 



**Expected behavior**
I believe in these cases we the score should be zero. 



<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hey @lalehsg This is intentional. Since the system refuses to answer it does not make sense to score it. We can't say it's less faithful or fully faithful in this situation therefore it's NaN

ok so I see inconsistent results for the same samples with GPT4 and GPT3.5 turbo for faithfulness. GPT3.5 is returning 0 and 0.33 in 2 such samples. GPT4 is returning NaN for both cases. curious!
Repository: explodinggradients/ragas
Issue: metric or expansion of aspect critique to detect problematic urls?
Root cause: Unimplemented Feature Gap

**Describe the Feature**
I'm currently working with an LLM based application that returns urls in its answers sometimes. 

Sometimes it returns urls that are busted in some way. Typically 404'd links and such.

I'm currently making a check outside of ragas during testing to flag if our bot is throwing bad urls, which made me wonder if there would be a place in ragas to have a way to also scan answers for urls. Probably using a combination of the `re` and `requests` libraries in python to grab urls, and test each one.  

If the urls return error codes or some sort of issue, lower the score.  Good status codes, increase the score.  No urls in the answer could be neutral or whatever.

**Why is the feature important for you?**
No code really to share at this time. I'm half tempted to make a PR on this, but curious if it fits in ragas to begin with.

**Additional context**
N/A
Repository: explodinggradients/ragas
Issue: Context Recall returning NaN when using GPT-4 models
Root cause: Interface Contract Mismatch

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When using any gpt-4 model as an evaluator, the context recall metric returns an NaN result and the following warning for almost every single question:
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'

I have tried this with my own dataset, as well as following the instructions in https://docs.ragas.io/en/stable/getstarted/evaluation.html simply changing the evaluator to any of the GPT-4 models (gpt-4-0125-preview, gpt-4-1106-preview and gpt-4). From the 10 questions in the testset, I got on average 9 NaN results for that metric. The other metrics work correctly.

Ragas version: 0.1.5
Python version: 3.10

**Code to Reproduce**
Follow the code in https://docs.ragas.io/en/stable/getstarted/evaluation.html simply changing the evaluator to any of the GPT-4 models (gpt-4-0125-preview, gpt-4-1106-preview and gpt-4).

**Error trace**
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'

Hey, can you please share some data points that I can use to reproduce the issue?
I'll raise a fix - this is mostly an issue related to the JSON formating which we are working on. 

from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
    answer_correctness,
    context_relevancy
)
from ragas import evaluate
from langchain.chat_models import ChatOpenAI
from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
amnesty_qa

gpt4 = ChatOpenAI(model_name="gpt-4-0125-preview")
#gpt4 = ChatOpenAI(model_name="gpt-4")

result = evaluate(
    #experiment_dataset,
    amnesty_qa["eval"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
        context_relevancy,
        answer_correctness
    ],
    llm=gpt4
)

result
df = result.to_pandas()

df.head(10)

Running this code from your website I am getting 9/10 values of NaN for context recall:
[recall_error.json](https://github.com/explodinggradients/ragas/files/14733350/recall_error.json)

I would like to add it works better with the gpt-4 simple model and works almost perfectly with the gpt-3.5 models. But I need to run the evaluation with the gpt4 models. 
Moreover, I have tried installing previous RAGAS versions and it still returns the same problems. Which is very odd, since yesterday I was being able to run the evaluations correctly.

Hi, I think this is still relevant. Context Precision and context recall return Nan for Gpt-4o models 

As of today, the issue still exists. Context recall returns Nan while other metrics are ok.

Still exists, not sure why
Repository: explodinggradients/ragas
Issue: How to get the calls Ragas issues to the LLM?
Root cause: Documentation Deficiency

[ x ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I'm generating a synthetic dataset. I'd like to log all the calls Ragas makes to the LLM. How do I do this?
I tried to step through with a debugger in vscode, but as soon as I hit [this line](https://github.com/explodinggradients/ragas/blob/8a56282d226a3093a894a00e8b4d199eab81d7ea/src/ragas/executor.py#L132) the code executes asynchronously, and I have no way of seeing what it is doing. I do see the tqdm loop progressing. This is despite the fact that I turned off is_async:

So the questions are:
1. How can I get the sequence of LLM calls RAGAS makes?
2. Where is the debugging log?
3. How can I turn off async execution, so I can step through with a debugger?

Thanks!

Ragas version: 0.1.0
Python version: 3.11.7

**Code to Reproduce**
Share code to reproduce the issue
```
    testset = generator.generate_with_langchain_docs(documents, test_size=1, with_debugging_logs=True, is_async=False,
                                                     distributions={simple: 1, reasoning: 0, multi_context: 0})
```

**Error trace**
No error.

**Expected behavior**
Expecting to find the debug log, and for the code to execute synchronously.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: Fix output parsing of Faithfuness
Root cause: Validation Gap

The CoT followed by the final answering mechanism in the prompt used [here](https://github.com/explodinggradients/ragas/blob/4a2804e7ef58218ffbd86abfcdc0c934538d33d8/src/ragas/metrics/faithfulnes.py#L37) is prone to errors. This has to be fixed without affecting the performance.  

Also refere #79
Repository: explodinggradients/ragas
Issue: Evaluation hangs w/ langchain's vllm wrapper
Root cause: Interface Contract Mismatch

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I want to use a local language model with vllm to evaluate an example. 
When I used langchain's vllm wrapper, the program hangs without any error message.
```Evaluating:   0%|          | 0/2 [00:00<?, ?it/s] ...```
Is this usage not expected?

CUDA version: 12.1	
python version: 3.10.0
ragas version: 0.1.5
vllm version: 0.3.3

**Code to Reproduce**

```
import torch 
from langchain_community.llms import VLLM
from ragas import evaluate
from ragas.embeddings.base import HuggingfaceEmbeddings
from ragas.metrics import answer_relevancy
from datasets import Dataset

MODEL_DIR = f"/PATH/TO/LLM/"
EMBEDDING_DIR = f"/PATH/TO/EMB/"

def main():
    data_samples = {
        'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
        'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
        'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
        ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
        'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
    }
    dataset = Dataset.from_dict(data_samples)
    
    llm = VLLM(
        model=f"{MODEL_DIR}/Mistral-7B-v0.1",
        tensor_parallel_size=torch.cuda.device_count(), 
        trust_remote_code=True, 
        max_new_tokens=1000,
    )
    embeddings = HuggingfaceEmbeddings(model_name=f"{EMBEDDING_DIR}/bge-small-en-v1.5") 

    score = evaluate(
        dataset,
        metrics=[answer_relevancy],
        llm=llm,
        embeddings=embeddings,
    )
    print(score.to_pandas())

if __name__ == '__main__':
    main()
```

.....
<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hello I am experiencing the same issue but I'm using Mistral via Ollama

Is the issue solved new version of ragas ?