zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Merge node error for multi_context evolutions
Root cause: Validation Gap

**Describe the bug**
Runner in Executor raised an exception
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/executor.py", line 58, in _aresults
    r = await future
  File "/home/user/anaconda3/envs/ragas/lib/python3.10/asyncio/tasks.py", line 575, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/user/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/home/user/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 163, in evolve
    return await self.generate_datarow(
  File "/home/user/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 210, in generate_datarow
    merged_nodes = self.merge_nodes(relevant_context)
  File "/home/user/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 75, in merge_nodes
    embed_dim = len(nodes.nodes[0].embedding) if nodes.nodes[0].embedding else None
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()


Ragas version: 0.1.1
Python version: 3.10.0

This only happens for `multi_context` evolutions.


also getting the same error

Hey @srikanthmalla @daveyko I have fixed this issue, please install ragas from source :) Will make a release soon
Sorry for the trouble guys

> Hey @srikanthmalla @daveyko I have fixed this issue, please install ragas from source :) Will make a release soon Sorry for the trouble guys

Hi! I tried one more time with the source. It does not raise an error, but it still doesn't produce multi-context questions.


Hey @AnnaKelmanson Can you share the code you used? And make sure that the distribution you used has multi_context in it? 

This particular error has been fixed #617 , so closing the issue. Feel free reopen if required. 

> Hey @AnnaKelmanson Can you share the code you used? And make sure that the distribution you used has multi_context in it? 

I used the code from the documentation examples for test set generation.
Repository: explodinggradients/ragas
Issue: Text-to-SQL Support Evaluation ?
Root cause: Unimplemented Feature Gap

I have a use case for text-to-SQL with RAG using LangChain. Is there any example or guide to evaluate the SQL result? Is the metric the same as regular text RAG? Thanks in advance

Hey bro @Putnug1122 apologies for the late reply. But this is something we would love to add to ragas. If  you're into text-SQL applications, I would love to pick you brain on it to formulate metrics. 

Popular text-to-SQL benchmarks like ([bird-sql](https://bird-bench.github.io/) and [spider](https://yale-lily.github.io/spider)) propose Execution Accuracy as the main metric since string similarity or exact match metrics often don't correlate to SQL correctness.

I will be happy to help with the integration of such benchmarks or with a bring-your-db approach.

@bcarvalho-via Sorry this is probably very late for you. But now we are adding more metrics to cover more use-cases into ragas including agentic workflows. So regarding SQL - 

we found out that metrics used to evaluate text to SQL pipelines can be broadly put into
1) Execution based: This approach executes the output query, collects the output, and compares the output rows with real rows using some ways (LLM-based or non-LLM-based)
2) Non-execution-based: executing every query to evaluate the pipeline can be expensive. So if there is a way to compare predicted_sql and reference_sql one could use that instead. Here also there are LLM-based and non-LLM-based ways to do it.
What do you think? I would like to hear your thoughts on what metrics might benefit you here.
Repository: explodinggradients/ragas
Issue: getting error in _faithfulness metric
Root cause: Validation Gap

Getting below error for faithfulness metric after executing the code
from ragas import evaluate
```
result = evaluate(
    fiqa_eval["baseline"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
        harmfulness,
    ],
)
result
```



![image](https://github.com/explodinggradients/ragas/assets/150421011/e6f46f02-b4d9-473f-8163-8c10c21085b2)
![image](https://github.com/explodinggradients/ragas/assets/150421011/26935f60-6f21-4e03-a549-1873533b7349)


Hey @bhaskarturkar can you please share the ragas version?

Hi @shahules786,  
I am getting this error for ragas version 0.1.2.


i am also getting this error

here are the distributions

distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}

interestingly this doesn't happen for non multi_context

Hey @daveyko this is a different issue, your issue will be resolved once you update to the latest version.

@shahules786 its happening to me on the latest version 0.1.2

@daveyko are you facing the same issue as described above? 

yup same issue im getting 
assert isinstance(statements, dict), "Invalid JSON response"

in  
File "/Users/davidko/.pyenv/versions/3.10.13/lib/python3.10/site-packages/ragas/metrics/_faithfulness.py", line 190, in _ascore

this is when i run evaluate
Repository: explodinggradients/ragas
Issue: Evaluation Dataset - Ground truth Warnings
Root cause: Documentation Deficiency

Hi,

I would like to raise a potential error around the dataset used in the evaluation method (from `evaluation.py`).

I got the evaluation method working when I set the following dataset:

```
Dataset.from_dict(
        {
            'question': list,
            'contexts': list[list],
            'ground_truths': list[list],
            'answer': list
        }
    )
```

However, I was receiving the following warning every time the pipeline ran:

```
passing column names as 'ground_truths' is deprecated and will be removed in the next version, please use 'ground_truth' instead. Note that `ground_truth` should be of type string and not Sequence[string] like `ground_truths`
```

Comparing to the documentation present [here](https://docs.ragas.io/en/stable/howtos/applications/data_preparation.html) and in the evaluation docstrings [here](https://github.com/explodinggradients/ragas/blob/317a2d88c2917fecc24dd4804cb7e12d79a6bac5/src/ragas/evaluation.py#L57), `ground_truth` key is expected to be a `list[list]`.

Therefore, is this warning correct? If so, to generate a string, we simply concatenate the list of strings into a single string? Separating by space, comma?

Then, on a quick experimentation, I changed the dataset key from `ground_truths` to `ground_truth`, but passing still `list[list]`, it did not work. This makes the example given [here](https://docs.ragas.io/en/stable/howtos/applications/data_preparation.html) invalid, I guess.

Thanks! I happy to provide any additional info!

I am using: `ragas==0.1.0`


seems like the documentation has to be updated - thanks for pointing it out . 

the correct format is 
```py
Dataset.from_dict(
        {
            'question': list,
            'contexts': list[list],
            'ground_truth': list,
            'answer': list
        }
    )

```
Repository: explodinggradients/ragas
Issue: .to_pandas() method erases randomly some ground_truth answers in testset generation
Root cause: Validation Gap

**Describe the bug**
[ragas_test.csv](https://github.com/explodinggradients/ragas/files/14418285/ragas_test.csv)

After generation of a testset and using to_pandas() some cells in ground_truth column turned out to be empty. Also it seems like it doesn't save info uniformly (in some cases question are in quotes, in some without them). I attach the csv.

Ragas version: source
Python version: 3.12.2

**Code to Reproduce**
```
def ragas_testset(file_path, num_ques):
    docs = load_documents(file_path)
    generator = TestsetGenerator.with_openai()

    distributions = {
        simple: 0.4,
        multi_context: 0.2,
        reasoning: 0.4
    }

    testset = generator.generate_with_langchain_docs(docs, num_ques, 
                                                    distributions,
                                                    with_debugging_logs=False,  
                                                    is_async=False,  
                                                    raise_exceptions=False)
    testset_df = testset.to_pandas()
    return testset_df
```

It should have returned a df without empty cells.



Hey @AnnaKelmanson Some ground-truth rows turns about to be empty due some limitation of LLM itself. I can raise a PR to resolve the formating issue now.
Repository: explodinggradients/ragas
Issue: Ground truth vs answer vs context 

Can anyone explain the difference between ground_truth and answer in more deep vs context

Does https://docs.ragas.io/en/stable/howtos/applications/data_preparation.html or https://github.com/explodinggradients/ragas/issues/655 answer your question? 

Hi, Closing this PR due to lack of activity. I hope your doubt was resolved.
Repository: explodinggradients/ragas
Issue: LlamaIndex ModuleNotFoundError
Root cause: Unimplemented Feature Gap

Getting a module not found error when trying to import `ragas.llama_index` as shown in the documentation. 

Ragas version: 0.1.2
Python version: 3.10.12

**Code to Reproduce**
`from ragas.llama_index import evaluate`

**Error trace**
```
ModuleNotFoundError: No module named 'ragas.llama_index'
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
File <command-467099862076163>, line 1
----> 1 from ragas.llama_index import evaluate

ModuleNotFoundError: No module named 'ragas.llama_index'
```

**Expected behavior**
Successful import of the `evaluate` module for use with LlamaIndex query engines as detailed [here](https://docs.ragas.io/en/latest/howtos/integrations/llamaindex.html)


I am facing the same issue. Any update on fix?

The files got deleted with the update of v0.1.0 3 weeks ago, but there is #557, so it seems to be an open ticket.
It's still possible to use the v0.0.22 for the evaluation. Of course there are newer features like adopt() are missing.

I'm using this for now, in case it's useful:

```python
    def pandas_to_dataset(df):
        if "contexts" in df.columns:
            df["contexts"] = df["contexts"].apply(
                lambda x: [x] if isinstance(x, str) else x
            )
        return Dataset.from_pandas(
            df,
            features=Features(
                {
                    "question": Value("string"),
                    "contexts": Sequence(Value("string")),
                    "ground_truth": Value("string"),
                    "evolution_type": Value("string"),
                    "episode_done": Value("bool"),
                    "answer": Value("string"),
                }
            ),
        )    

    dataset = pandas_to_dataset(df) # df should contain a column with the answers retrieved using query
    result = evaluate(
        dataset,
        metrics=[answer_correctness, faithfulness, answer_relevancy, context_recall],
    )
```

Which versions of llamaindex and ragas am I required to use in order to make this work?

added it in the new version 
docs: https://docs.ragas.io/en/stable/howtos/integrations/llamaindex.html

what is the solution? which version is working one?
Repository: explodinggradients/ragas
Issue: faithfulness score AttributeError
Root cause: Validation Gap

**Describe the bug**
```
File /opt/conda/lib/python3.11/site-packages/ragas/metrics/_faithfulness.py:178, in <genexpr>(.0)
    175 output = json_loader.safe_load(output[0].text, self.llm)
    176 output = output if isinstance(output, list) else []
    177 faithful_statements = sum(
--> 178     verdict_score_map.get(dict.get("verdict", "").lower(), np.nan)
    179     for dict in output
    180 )
    181 num_statements = len(output)
    182 if num_statements:

AttributeError: 'str' object has no attribute 'get'
```
The above error occurs in facefulness, is it a problem solved in v1.0.2?

Ragas version: 0.0.11
Python version: 3.11

**Code to Reproduce**
```python
...
gpt4 = ChatOpenAI(model_name="gpt-4-0125-preview")
gpt4_wrapper = CustomLangchainLLM(llm=gpt4)

_faithfulness = Faithfulness()
_faithfulness.llm = gpt4_wrapper

RagasEvaluatorChain(metric=_faithfulness).evaluate(examples, predictions)
...
```

**Error trace**

**Expected behavior**
no error occured


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


@shahules786 thanks for your works.
could you check this issue plz,,,

Hey @rere950303 please reinstall from source once the PR is merged :)
Repository: explodinggradients/ragas
Issue: POST Call to  https://t.explodinggradients.com:443?
GitHub labels: question

Hi,

I saw in my debug logs that there is POST call to https://t.explodinggradients.com:443.  Could you please tell me what is being sent? and how do I stop it?  if there is a setting or so?  We work on regulated environment and data, so this is important to know.

2024-02-28,17:04:59 urllib3.connectionpool:549 - DEBUG:https://t.explodinggradients.com:443 "POST / HTTP/1.1" 200 15


set  `RAGAS_DEBUG_TRACKING` to true
read more [here](https://github.com/explodinggradients/ragas/blob/main/src/ragas/_analytics.py)  
duplicate #630

Thank you

Hi @gauravdd @stydxm Just for additional info,  we do not track any sensitive information as you can verify in the code. The tracking is just to see how many samples are being evaluated. 
Closing the issue on that remark. 

I think you should set `RAGAS_DO_NOT_TRACK = True` too
Repository: explodinggradients/ragas
Issue: Support `langchain-google-genai`
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Support Google Gemini API like how OpenAI API is supported as first class citizen.

**Why is the feature important for you?**
I need to test both OpenAI and Google Gemini APIs, but currently I need to repeat a big chunk of `TestsetGenerator.with_openai()` for the equivalent of `langchain-google-genai` code.

**Additional context**
`temperature` setting also corrected to prevent error described in #656 


PR raised: #677, but breaks Python <=3.8 so careful review is needed.


I have same error when using gemini an Ragas, any suggestions? 

@RazHadas @joy13975 How have you used gemini with Ragas? I need this. Kindly help me out.

@nehas0326  here is an example of using gemini with Ragas: https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb

Above code has some very basic missing parts and errors, here is a working one: https://github.com/anshupandey/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb
Repository: explodinggradients/ragas
Issue: answer_relevancy
Root cause: Documentation Deficiency

**Your Question**
what model  is used by ragas for answer relevancy score calculation ? 
I didn't mentioned any model name or inference API till ragas generating an evaluation report ? 

**Additional context**

**If Rags follow this below steps to calculate the answer relevancy score what model is used to generate the question in step-1 and which embedding model is used for generating embeddings which in turn is used for calculating cosine similarity ?**

To calculate the relevance of the answer to the given question, we follow two steps:

Step 1: Reverse-engineer ‘n’ variants of the question from the generated answer using a Large Language Model (LLM). For instance, for the first answer, the LLM might generate the following possible questions:

Question 1: “In which part of Europe is France located?”

Question 2: “What is the geographical location of France within Europe?”

Question 3: “Can you identify the region of Europe where France is situated?”

Step 2: Calculate the mean cosine similarity between the generated questions and the actual question.
The underlying concept is that if the answer correctly addresses the question, it is highly probable that the original question can be reconstructed solely from the answer.

Ragas by default uses gpt-3-5 and ada-2 embeddings. 
Please close the issue if it's clear :)
Repository: explodinggradients/ragas
Issue: openai.NotFoundError: Error code: 404 - {'error': {'message': 'The model `gpt-4` does not exist or you do not have access to it.
Root cause: Configuration Error

Hi, 
I'm trying to use RAGAs library for synthetic test set generation, I do not have access to gpt4. The critic_lllm in TestsetGenerator.with_openai() uses gpt4 by default, so I changed it to gpt3.5. However, it throws the error gpt4 not available. 

I've attached the code with the error below. Pls let me know if there's a solution for this. Thank you!

Ragas version: 0.1.2
Python version: 3.9

**Code to Reproduce**
```
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context

# generator with openai models
generator = TestsetGenerator.with_openai(generator_llm = "gpt-3.5-turbo-16k", critic_llm = "gpt-3.5-turbo-16k")

# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
```

**Error trace**
openai.NotFoundError: Error code: 404 - {'error': {'message': 'The model `gpt-4` does not exist or you do not have access to it. Learn more: https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4.', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}

If there are ways to generate the test set with open source LLMs, please help me with the same.

Hey @gowrijsuria do you have access to gpt-4 model? Seems like you don't have it. Can you checkout customising test set generation document for using any llm of your choice? https://docs.ragas.io/en/latest/howtos/customisations/bring-your-own-llm-or-embs.html

I have access to gpt3.5 , hence I passed that as the param. However, it doesn't work. Pls check this issue
Repository: explodinggradients/ragas
Issue: More verbose output in evaluate
Root cause: Documentation Deficiency

**Your Question**
Like I asked in https://github.com/explodinggradients/ragas/issues/108, is there a way to have more verbose output in evaluate?

It would be nice to see how ragas arrived to a score.

**Code Examples**
Currently, I can hack in this functionality by adding in a print statement here for faithfulness:

https://github.com/explodinggradients/ragas/blob/30c830377644dce94e410f3ce3a8aa4339bb74a9/src/ragas/metrics/_faithfulness.py#L201

But it would be nice to turn this on or off with a configuration option or something.

If not, I'd be happy to whip up a PR for this.


Hey, please refer to callbacks and tracing for doing this https://docs.ragas.io/en/latest/howtos/applications/tracing.html
Repository: explodinggradients/ragas
Issue: Any way to use llamaindex docstore nodes and embeddings?
Root cause: Documentation Deficiency

Is there a way to pass ragas llamaindex's StorageContext.docstore object to use already generated nodes and embeddings? When dealing with large number of really big documents - waiting for re-indexing is a problem. There's docstore parameter in TestsetGenerator class but I am confused with the documents parameter in generate_with_llamaindex_docs() method.
```
def get_docstore():
    config = load_toml_configs("data.toml")
    storage = config['data']['storage_path']
    storage_context = StorageContext.from_defaults(persist_dir=storage)
    return storage_context.docstore

def main():
    #...
    generator = TestsetGenerator.with_openai(
        docstore=get_docstore(),
        generator_llm=llm,
        critic_llm=critic_llm,
        embeddings=embeddings_llm
        )

    #...
    question_dataset = generator.generate_with_llamaindex_docs(
        documents=documents,
        test_size=number_of_questions,
        distributions=distributions,
        is_async=False,
        raise_exceptions=False
        )
```



I would be also interested in reusing nodes/chunks and embedding that I already have in database.

Can we somehow "fuse" RAGAS test set generator to bypass documents loading, chunking, embedding and just inject nodes with embeddings and other things that are required to start generating?
Repository: explodinggradients/ragas
Issue: max retries exceeded for SimpleEvolution 
Root cause: Validation Gap

from ragas.testset import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context

test_generator = TestsetGenerator(
    generator_llm=bedrock_model,
    critic_llm=bedrock_model,
    embeddings=bedrock_embeddings,
    docstore=docstore,
)

distributions = {simple: 0.5, reasoning: 0.4, multi_context: 0.1}

testset = test_generator.generate_with_langchain_docs(
    documents=documents, test_size=15, distributions=distributions
)

after running this code the embeddings are created but there seems to be issue with generating the test data.
![image](https://github.com/explodinggradients/ragas/assets/121536451/4c53b116-ebb2-429e-894f-17b2599f77a3)

I'm trying to create the test dataset using llama2 from amazon bedrock. 

@jjmachan could you please look into this issue 

Hey @AshuKhandave Sorry for the late reply. Which models/ragas version are you using? 
Consider joining our discord channel for quick help. 

Hey @shahules786 I'm using meta_llama2_13b and Amazon Titan for embedding using AWS bedrock.

I see @AshuKhandave  would you be able to use better models? The thing with synthetic data generation is that only high-quality models like gpt 3.5 or on par with gpt 3.5 could produce high-quality results out of the box. 

I'm getting the same error when trying on OpenAI 

Is this still happening with current version?