zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Ragas Using Azure OpenAI ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Value'>
Root cause: Validation Gap

**Describe the bug**
I am trying to get results and I am getting this error ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Value'>

Ragas version:
Python version:
3

**Code to Reproduce**
I am working on a custom model i am have three different csv that i am stitching together. and trying to retieve information

**Error trace**
```
import pandas as pd
import numpy as np

# Load each CSV file into a separate DataFrame
hr_queries = pd.read_csv('test1.csv')
ground_truth = pd.read_csv('ground_truth.csv')
context = pd.read_csv('context.csv')

questions = test1['questions']  
answers = test1['answers']  

# Extract the first column from the 'ground_truth' DataFrame
ground_truth = ground_truth.iloc[:, 0]

# Extract the 'context' column from the 'context' DataFrame
context = context['context'] 

# Create a data dictionary
data_dict = {
    'question': questions,
    'answer': answers,
    'ground_truths': ground_truth,
    'contexts': context,
}

# Concatenate the DataFrames in the data dictionary
df = pd.concat(data_dict, axis=1)

# Save the concatenated DataFrame to a new CSV file
df.to_csv('test2.csv', index=False)

from datasets import load_dataset
with open('test2.csv') as f:
  data = f.read()
dataset = load_dataset('csv', data_files='test2.csv')


print(dataset)
```

```
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

# list of metrics we're going to use
metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
    harmfulness,
]

```

```
or m in metrics:
    m.__setattr__("llm", ragas_azure_model)

```

```
from ragas import evaluate

result = evaluate(
    dataset["train"],  
    metrics=metrics,
)

result

```
ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Value'>



**Expected behavior**
I should be to able to evaluating with [context_recall]
<img width="853" alt="Screenshot 2023-12-28 at 12 49 46" src="https://github.com/explodinggradients/ragas/assets/47422016/7c2be1d8-daef-4611-b91e-c162bb8a8f12">



**Additional context**
Add any other context about the problem here.


Closing this issue as this is duplicate to #286 and #413
Repository: explodinggradients/ragas
Issue: ragas_score removed from Result class in November 2023 but still in documentation
Root cause: Documentation Deficiency

**Describe the bug**
Ragas documentation [here](https://docs.ragas.io/en/latest/references/evaluation.html#ragas.evaluation.evaluate) states the `ragas_score` can be retrieved directly from result of `evaluate`. However, this throws a `KeyError`
Is there an intended workaround or was this removed with the intention that we would calculate harmonic mean using output?

Ragas version:
Python version: 3.10.13, 3.12.1

**Code to Reproduce**
```
# OPENAI API key set here...

from datasets import load_dataset
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
    context_relevancy
)
from ragas.metrics.critique import harmfulness
from ragas import evaluate

fiqa_eval = load_dataset("explodinggradients/fiqa", "ragas_eval")


result = evaluate(
    fiqa_eval["baseline"].select(range(3)),
    metrics=[
        answer_relevancy,
        faithfulness,
        context_recall,
        context_precision,
        context_relevancy,
        harmfulness
    ]
)

print(result['ragas_score'])
```


**Expected behavior**
The result object would have a key/value pair for `ragas_score` as described in documentation.



Hi @mwertheim-zco , you're right. We are letting the users decide on what aggregation they want to do on the score level. Once you have the raw scores you could write simple aggregate functions to do any of these calculations. 
We should remove `ragas_score` from docs. would you like to help? 

Hi @shahules786 i can help with this one. Please assign me the issue. Thanks.

Thank you. Assigned to you @prufrock349 

@shahules786  how can we get ragas_score then?

Hi @atursunov , you can combine the scores in results dict using any aggregation function you prefer.
Repository: explodinggradients/ragas
Issue: TestsetGenerator issue with small Q&A FAQ dataset - multiple 'no neighbors exists' errors and limited generation output
Root cause: Algorithmic Error

Hi,

I've been using the TestsetGenerator with default and other adjusted question type distributions, but in nearly all cases using a a test_size of 20-50, I get very few results returned. I see multiple errors such as:

/home/vscode/.local/lib/python3.11/site-packages/ragas/testset/testset_generator.py:269: UserWarning: No neighbors exists
  warnings.warn("No neighbors exists")
and
/home/vscode/.local/lib/python3.11/site-packages/ragas/testset/utils.py:16: UserWarning: Invalid json
  warnings.warn("Invalid json")

I've looked through the documentation but there's nothing that covers these issues. The documents are short/simple FAQ question/answers, so relatively small in size on a per document basis. An example is:

Do I get scores for all the activities that I do?
No, some activities are designed to help show language applied to real-life situations, like games or videos.

This would be a sample document as it represents a page in an FAQ - this may not be relevant, but I've added it in case the context is useful. There's 67 of these FAQ documents in the data I'm working with and I'm running:

test_generator.generate(documents, test_size=30)

Typically I'm getting between 4-7 results back, which is not many - but I assume the errors are having an impact in some way that I can't determine.

apologies for the hard time but this will be fixed with #380 
I'll update this as soon as its merged in 🙂 

@jjmachan about when do you think this will be fixed/merged? 

hopefully next week 🤞🏽 

@jjmachan Any updates on this? :)

Anyone has work around method to use it with v0.0.22 ?
It's not return enough as test size.

hey @thenextmz, @lamkhatinh and @Norfolkmag 
we have fixed this in main - you can use it now - do take it for a spin and let us know (you will have to install from source for now but we are going to release v0.1 in a couple of hours, going through the issues to see if we missed any major ones and final touch ups 🙂 )

you can see the updated docs here https://docs.ragas.io/en/latest/concepts/testset_generation.html
Repository: explodinggradients/ragas
Issue: Missing components
Root cause: Documentation Deficiency

**Describe the bug**
The latest commit has deleted a bunch of components and code references, this does not seem intentional / is not documented anywhere.

Ragas version: commit/bad3a8e1d115e69bb4acbd9887ca5b6cb6a03fba
Python version: 3.10

e.g. `langchain/` package is entirely missing


@jjmachan 

hey welcome to the community :)
really sorry about that but we are removing these extra modules in preparation for our v0.1 release which is a major revamp. While the happy case has been kept migration free since a lot of internals have changed which will need to be migrated. 

I would love to help you migrate (by debugging on call) this since you are on the dev branch and hopefully it will get you unblocked and will help we write the migration docs for others as well - what do you think

@jjmachan hey sure that’s okay. For starters, how is the EvaluatorChain migrated (from the missing langchain package)?

```
from ragas.langchain.evalchain import RagasEvaluatorChain
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

# create evaluation chains
faithfulness_chain = RagasEvaluatorChain(metric=faithfulness)
answer_rel_chain = RagasEvaluatorChain(metric=answer_relevancy)
context_rel_chain = RagasEvaluatorChain(metric=context_precision)
context_recall_chain = RagasEvaluatorChain(metric=context_recall)
```

hey @nils-hde could you try this notebook https://colab.research.google.com/drive/1C1Epju1lVkXTQi2jBq1njrOrmkfg0eQS?usp=sharing

I know it's too late - sorry about that
Repository: explodinggradients/ragas
Issue: max sequence length in context relevancy. 
Root cause: Algorithmic Error

Currently, if the sequence length of context is more than max_length (512) it will be truncated before scoring relevancy. Instead, such contexts should be chunked to sequence < 512 tokens before scoring and then averaged. 
Changes for this could be made [here](https://github.com/explodinggradients/ragas/blob/e0324d96788c401b71ed19d688213b7096313b82/src/ragas/metrics/context_relevance.py#L38) 

This was addressed in #59
Repository: explodinggradients/ragas
Issue: [Question] why context is used in question generation prompt used in answer relevancy calculation
Root cause: Documentation Deficiency

Hello,

I've been exploring the methodology for calculating answer relevancy and noticed that the context is provided alongside the answer during the evaluation process. 

As I understand it, the approach involves generating questions from the given answer using a LLM and then comparing these generated questions with the original question to assess relevancy.

My question pertains to the necessity and impact of including context in this process. What is the rationale behind using context in the relevancy calculation, and how does it enhance the results?

To further investigate, I conducted an experiment where I modified the prompt to instruct the language model to disregard the provided context. The idea was to evaluate if excluding context might be a more effective approach. Unfortunately, RAGAS, doesn't currently allow for the exclusion of context without altering the prompt itself.

Here are the results from my experiment on the FIQA dataset:
![image](https://github.com/explodinggradients/ragas/assets/33887466/2b4431b5-bfd5-4656-886f-af9f334c498b)

From these findings, it appears that the questions generated without considering context are more closely aligned with the original questions than those generated with context. 
This raises the question: Is providing context to the language model truly beneficial for the purpose of calculating answer relevancy?

I would appreciate any insights or explanations regarding the use of context in this process. Also, if there is any possibility to exclude context without modifying the prompt in future iterations of RAGAS, that information would be valuable.

Thank you for your time and assistance.


Wow, your visualization looks great @praveenck06 . thanks for sharing your analysis, your results indeed has high correlation. The rationale behind using context in question generation is to ensure that it works for situations like keyphrase extraction. For example
question: Extract an emotional keyphrases from the given context
answer: keyphrase1, keyphrase2
context: context containing keyphrase1 and keyphrase2

In situations similar to this question generation can benefit from context. Does that make sense?

@shahules786 since we have already seperate metric for relevancy of context, my worry is that if we pass the context for question generation then answer relevancy would be combination of answer generation and context retrival.
would it better to have seperate metric for answer generation and context retrival ?

And also need help on understanding noncommital part of the calculation

@praveenck06 the noncommittal part is used to identify answers from LLM like "I don't know about x", "My knowledge cutoff is y", etc. These answers should have zero relevancy. 

Hi @praveenck06 feel free to close the issue if your concern was addressed.
Repository: explodinggradients/ragas
Issue: Cannot access ragas.evaluation.evaluate() is_async argument
Root cause: Unimplemented Feature Gap

Hi I am not sure if this is intentional or not (hence why I am not marking it as a bug), but it seems like I am not able to set the _is_async_ argument in ragas.evaluation.evaluate function as True even though it seems like based on this [documentation page](https://docs.ragas.io/en/latest/references/evaluation.html) it is an argument for the function. I am using ragas version 0.0.22.

ohh @anthonyivn2 that is because it is only available in the `main` branch right now. You will have to install it from the source.

do give it a go if you want - does have some breaking changes though. 

Thanks also for raising the issue and helping us with keeping ragas top-notch 🙂 

I'm closing this for now - feel free to reopen in case you need it

ps this will also be generally available with our `v0.1` release

Ok got it, would the v0.1 release comes with the changes where _context precision_ would require _ground truths_?

@anthonyivn2 Yes but it will be still optional. You can try this out now by installing ragas from source.
Repository: explodinggradients/ragas
Issue: `KeyError: 'file_name'` when running synthetic data generation example
Root cause: Validation Gap

**Describe the bug**
When running the synthetic data generation example from https://docs.ragas.io/en/latest/getstarted/testset_generation.html#get-started-testset-generation, I get the error:

`KeyError: 'file_name'`

on the line:

 `testset = testsetgenerator.generate(documents, test_size=test_size)`.

Ragas version: `0.0.23.dev37+g041b20c`
Python version: `3.11`

**Code to Reproduce**

```
import os
os.environ["OPENAI_API_KEY"] = "KEY GOES HERE"

from llama_index import download_loader

SemanticScholarReader = download_loader("SemanticScholarReader")
loader = SemanticScholarReader()
# Narrow down the search space
query_space = "large language models"
# Increase the limit to obtain more documents
documents = loader.load_data(query=query_space, limit=10)


from ragas.testset import TestsetGenerator

testsetgenerator = TestsetGenerator.from_default()
test_size = 10
testset = testsetgenerator.generate(documents, test_size=test_size)
```

**Error trace**

```
testset = testsetgenerator.generate(documents, test_size=test_size)
  0%|          | 0/10 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "C:\Users\DMBO\AppData\Local\Anaconda3\envs\docgpt_ragas_py311\Lib\site-packages\IPython\core\interactiveshell.py", line 3553, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-62b08db69f64>", line 1, in <module>
    testset = testsetgenerator.generate(documents, test_size=test_size)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DMBO\AppData\Local\Anaconda3\envs\docgpt_ragas_py311\Lib\site-packages\ragas\testset\testset_generator.py", line 427, in generate
    neighbor_nodes = doc_nodes_map[curr_node.metadata["file_name"]]
                                   ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'file_name'
```

**Expected behavior**
I expected the test set to be generated without error.



we have actually added a note for that [here](https://docs.ragas.io/en/latest/getstarted/testset_generation.html#data-generation) can you check that.

the reason is that you need to provide a "file_name" in the metadata which is right now a manual process


Got it, thanks!
Repository: explodinggradients/ragas
Issue: Safe loader issue with ragas from source
Root cause: Algorithmic Error

**Describe the bug**
https://github.com/explodinggradients/ragas/blob/041b20cb00eb200bb6e62df8104baf0ac801b976/src/ragas/llms/json_load.py#L78-L95

This function does not seem to work as expected.. 

Ragas version: [041b20](https://github.com/explodinggradients/ragas/commit/041b20cb00eb200bb6e62df8104baf0ac801b976) (`pip install  git+https://github.com/explodinggradients/ragas.git`)
Python version: 3.9

**Code to Reproduce**
```python
from ragas.llms.json_load import json_loader
from ragas.llms import llm_factory

s ="""
    {
        "statement_1": "The USA Supreme Court ruling on abortion can have significant global implications.",
        "reason": "The context mentions that the ruling has had impacts beyond national borders due to the geopolitical and cultural influence wielded by the USA globally and the aid it funds.",
        "verdict": "1"
    }
    {
        "statement_2": "The ruling can serve as a reference point for other countries dealing with their own abortion laws.",
        "reason": "The context mentions that the ruling has laid the groundwork for anti-abortion legislative and policy attacks in other countries.",
        "verdict": "1"
    }
"""

json_loader.safe_load(s, llm_factory())

```

**Output**
```
{'statement_1': 'The USA Supreme Court ruling on abortion can have significant global implications.',
 'reason': 'The context mentions that the ruling has had impacts beyond national borders due to the geopolitical and cultural influence wielded by the USA globally and the aid it funds.',
 'verdict': '1'} 
```

**Error trace**
No error

**Expected behavior**
The entire string loaded, not only "the first part"

**Additional context**
This issue seems to have an impact on the entire calculation of scores
Repository: explodinggradients/ragas
Issue: Support llm_factory ChatOpenAI response_format arg
Root cause: Unimplemented Feature Gap

Allow to set `response_format` args in `llm_factory` https://github.com/explodinggradients/ragas/blob/041b20cb00eb200bb6e62df8104baf0ac801b976/src/ragas/llms/__init__.py#L11-L12

Setting`response_format` to `{"type": "json_object}` ensure that the response to be in json format (**does not seem to be the case now**)

To use the `json_object` response_format, one has to use the openai model `gpt-3.5-turbo-1106` or `gpt-4-1106`. Also, the input should contains the word `json`. 

For instance:

```
LONG_FORM_ANSWER_PROMPT = Prompt(
    name="long_form_answer",
    instruction="Create one or more statements from each sentence in the given answer. Answer with json.",
    examples=[
        {
            "question": "Who was  Albert Einstein and what is he best known for?",
            "answer": "He was a German-born theoretical physicist, widely acknowledged to be one of the greatest and most influential physicists of all time. He was best known for developing the theory of relativity, he also made important contributions to the development of the theory of quantum mechanics.",
            "statements": {
                "statements": [
                    "Albert Einstein, a German-born theoretical physicist, is renowned for being one of the most influential physicists in history.",
                    "Albert Einstein was best known for his theory of relativity.",
                    "Einstein's contributions significantly advanced the field of quantum mechanics",
                    "Recognized globally, Einstein's work has profoundly impacted the scientific community",
                    "Einstein's groundbreaking theories continue to shape our understanding of physics today.",
                ]
            },
        },
        {
            "question": "Cadmium Chloride is slightly soluble in this chemical, it is also called what?",
            "answer": "alcohol",
            "statements": {
                "statements": ["Cadmium Chloride is slightly soluble in alcohol."]
            },
        },
        {
            "question": "Were Hitler and Benito Mussolini of the same nationality?",
            "answer": "Sorry, I can't provide answer to that question.",
            "statements": {"statements": []},
        },
    ],
    input_keys=["question", "answer"],
    output_key="statements",
    output_type="JSON",
)
```


Could be related to #445
Repository: explodinggradients/ragas
Issue: Azure has not provided the response due to a content filter being triggered
Root cause: Unimplemented Feature Gap

**Describe the bug**
Trying to run ragas evaluations as mentioned here: https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html. In some cases it runs OK, but on other cases the same code throws following error:

![image](https://github.com/explodinggradients/ragas/assets/7220691/64064f0d-dcc2-418a-b67f-b16b6d7582e8)

![image](https://github.com/explodinggradients/ragas/assets/7220691/94ff05c4-4c1f-45b0-b633-d2da8f70634e)

It looks like the issue is not in ragas but wondering is there a retry option in ragas, so we can re-run code if it fails?

Ragas version: 0.0.22
OpenAI version: 1.6.1
LangChain version: 0.1.0
Python version: 3.10



Hi @atursunov, did you managed to resolve it?


Yes, it is more on Langchain and OpenAI side. Later will post a workaround.

@atursunov could you please share the solution? I am working to implement this with Azure OpenAI and getting the same error as you but with latest Ragas package.
Repository: explodinggradients/ragas
Issue: OPENAI key error even on open-source llm
Root cause: Validation Gap

**Describe the bug**
OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable

Ragas version:
Python version:3.10

**Code to Reproduce**
(https://docs.ragas.io/en/stable/howtos/customisations/llms.html)
**Error trace**

**Expected behavior**
OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable
Repository: explodinggradients/ragas
Issue: Expose LLM
Root cause: Unimplemented Feature Gap

Hi,

Great piece of work here, really well done.

It would be fantastic if we could any supported LLM from langchain to do the evaluation. 

I have a number of use cases which require sovereignty, which means essentially either using on prem LLMs or Azure OpenAPI locked to a certain region.

Happy to help wherever I can!

hey, that for raising the issue and offering to help :)

We were a bit hesitant about adding langchain dependency because of how fast it keeps changing but if you could explain a bit more about your use cases that would help. 
Basically, what are you trying to improve by bringing in other LLMs? reduce cost? easily try out other LLMs for evaluation? If you could explain a bit about our thought process that would be awesome!

No problem!

Yes exactly - for a given use case in the enterprise there needs to be multiple choices factoring in performance, cost and the quality of results. In addition Azure OpenAPI offers privacy and compliance for the enterprise which is a cut above openAPI, hence it will become the "go to" for at least the next 6 months - at the moment a lot of organizations have closed off ChatGPT / OpenAPI together due to privacy and IP concerns. 

alrighty, I will start on a draft to address this shortly. Thank you for the input @HazK 🙂 

That sounds great thank you
________________________________
From: Jithin James ***@***.***>
Sent: Saturday, June 24, 2023 6:09:33 AM
To: explodinggradients/ragas ***@***.***>
Cc: HazK ***@***.***>; Mention ***@***.***>
Subject: Re: [explodinggradients/ragas] Expose LLM (Issue #46)


alrighty, I will start on a draft to address this shortly. Thank you for the input @HazK<https://github.com/HazK> 🙂

—
Reply to this email directly, view it on GitHub<https://github.com/explodinggradients/ragas/issues/46#issuecomment-1605270542>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHYOU5UYESNF3N6ZLHXUWOTXMZZA3ANCNFSM6AAAAAAZRPS4TI>.
You are receiving this because you were mentioned.Message ID: ***@***.***>


hey @HazK this is fixed. Thanks for bringing this up and do let us know when you try it
here is the example notebook: https://github.com/explodinggradients/ragas/blob/main/docs/guides/llms.ipynb