Repository: explodinggradients/ragas
Issue: datasets.features.features.Sequence
Root cause: Validation Gap

**Describe the bug**
I'm getting this error either with my own dataset or your dataset in the documentation, it keeps giving this error:
ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Sequence'>

Ragas version:
Python version:

**Code to Reproduce**
from ragas import evaluate

result = evaluate(
    fiqa_eval["baseline"],
    metrics=metrics,
)

result
Copy code

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.


Hey, can you share the Ragas and Python versions? and make sure it's the latest

Ragas: Version: 0.0.20
Openai: Version: 1.3.0
py: Version 3.11


> Hey, can you share the Ragas and Python versions? and make sure it's the latest

Now I'm encountering this error:
NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}

While evaluating relevancy score:

evaluating with [faithfulness]
100%|██████████| 7/7 [09:02<00:00, 77.46s/it]
evaluating with [answer_relevancy]
  0%|          | 0/7 [00:07<?, ?it/s]
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
[<ipython-input-149-c87895f9a55b>](https://localhost:8080/#) in <cell line: 3>()
      1 from ragas import evaluate
      2 
----> 3 result = evaluate(fiqa_eval['train'], metrics=metrics)

11 frames
[/usr/local/lib/python3.10/dist-packages/openai/_base_client.py](https://localhost:8080/#) in _request(self, cast_to, options, remaining_retries, stream, stream_cls)
    875             # to completion before attempting to access the response text.
    876             err.response.read()
--> 877             raise self._make_status_error_from_response(err.response) from None
    878         except httpx.TimeoutException as err:
    879             if retries > 0:

NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}


the same problem, so how to fix it?

> the same problem, so how to fix it?

which problem is it u are considering? last one or the one at the top? (second one is embedding problem, I used opensource embedding(jina-ai) it worked but for prod I need openai embedding to work out.
if it 1st one let me know, I have a solution.

> **Describe the bug** I'm getting this error either with my own dataset or your dataset in the documentation, it keeps giving this error: ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Sequence'>
> 
> Ragas version: Python version:
> 
> **Code to Reproduce** from ragas import evaluate
> 
> result = evaluate( fiqa_eval["baseline"], metrics=metrics, )
> 
> result Copy code
> 
> **Error trace**
> 
> **Expected behavior** A clear and concise description of what you expected to happen.
> 
> **Additional context** Add any other context about the problem here.

1st problem, thank you very much

@samedovzaur @guankaisi It's really helpful if you can produce full, end-to-end, code examples.

Here's an example that works fine for me:

```python
from ragas import evaluate
from datasets import load_dataset

from ragas.metrics import (
    context_precision,
    answer_relevancy, 
    faithfulness,
    context_recall,
)

# list of metrics we're going to use
metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
]

fiqa_eval = load_dataset("explodinggradients/fiqa", "ragas_eval")

result = evaluate(
fiqa_eval["baseline"],
metrics=metrics,
)

# takes a long time to run but works fine
```


When I have previously observed your error (`ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Sequence'>`), it's because `ragas` is expected a sequence of list of strings for the `AnswerCorrectness` metric, but was fed a Sequence of strings. 

The error is here: https://github.com/explodinggradients/ragas/blob/65b77f5d8a300107ab964de6af060a8207dff260/src/ragas/validation.py#L30C1-L31C72

I can typically fix it by ensuring that `ds.features[column_names].feature.dtype` is indeed of type string, e.g. 
`dataset['baseline'] = dataset['baseline'].astype(str)` 



> **Describe the bug** I'm getting this error either with my own dataset or your dataset in the documentation, it keeps giving this error: ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Sequence'>
> 
> Ragas version: Python version:
> 
> **Code to Reproduce** from ragas import evaluate
> 
> result = evaluate( fiqa_eval["baseline"], metrics=metrics, )
> 


Same error for me

Ragas version: 0.0.21
Python Version: 3.11.5

@MattiaBaldinetti Can you post your entire code end-to-end please? 

@shahules786 This interface/error, is not an issue with the codebase but is biting python/pandas newbies - one idea is I could do a small PR to make the error more explicit and show exactly what is going wrong...thoughts? 

@nelsonauner We are planning to remove huggingface dataset in Ragas v0.1 . We think adding HF dataset was a bit of over-engineering, what are your thoughts? @nelsonauner 
@jjmachan 

we could do a pydantic validation 
hey @nelsonauner let me know if you interested in contributing, I was just about to do some designing and if you want we can do this together, should be a small fix

I'm interested. I'll try to contact you on the discord

> @MattiaBaldinetti Can you post your entire code end-to-end please?

Hi @nelsonauner, this is my code:

```
import os
os.environ["OPENAI_API_KEY"] = "my-Key"


dataset = load_dataset("mattobaldo/personal_dataset")

Output:
DatasetDict({
        train: Dataset({
        features: ['question', 'ground_truths', 'answer', 'contexts'],
        num_rows: 30
    })
})


from ragas.metrics import (
    answer_relevancy,
    faithfulness, 
    context_recall, 
    context_precision, 
)


from ragas import evaluate
result = evaluate(
    dataset["train"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall],)
result
```
Return this error : 
ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Value'>


@MattiaBaldinetti - I think the issue is with your own dataset, not ragas. 
I found very similar code to yours in the ragas documentation [here](https://github.com/explodinggradients/ragas/blob/41e9e54b204566425908c1a437d7c1c798578ad3/docs/howtos/integrations/zeno.ipynb#L33)


It runs fine for me, and is almost identical to your code except for your _personal_ dataset. 

Before reporting issues, you should make sure they're really an issue with ragas, preferably by using examples in the documentation and standardized datasets, not your own datasets (that might have data issues we can't expect ragas volunteers to fix!) 

```python

from datasets import load_dataset
# This gives an error "DatasetGenerationError: An error occurred while generating the dataset""
# dataset = load_dataset("mattobaldo/personal_dataset")

# So instead I use one from the tutorial:
dataset = load_dataset("explodinggradients/fiqa", "ragas_eval")

# the rest is the same as @MattiaBaldinetti  issue in https://github.com/explodinggradients/ragas/issues/286:

from ragas.metrics import (
    answer_relevancy,
    faithfulness, 
    context_recall, 
    context_precision, 
)


from ragas import evaluate
result = evaluate(
    dataset["baseline"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall],)
        
        
# {'ragas_score': 0.6483, 'context_precision': 0.6333, 'faithfulness': 0.7384, 'answer_relevancy': 0.9251, 'context_recall': 0.4639}
```

> @MattiaBaldinetti - I think the issue is with your own dataset, not ragas. I found very similar code to yours in the ragas documentation [here](https://github.com/explodinggradients/ragas/blob/41e9e54b204566425908c1a437d7c1c798578ad3/docs/howtos/integrations/zeno.ipynb#L33)
> 
> It runs fine for me, and is almost identical to your code except for your _personal_ dataset.
> 
> Before reporting issues, you should make sure they're really an issue with ragas, preferably by using examples in the documentation and standardized datasets, not your own datasets (that might have data issues we can't expect ragas volunteers to fix!)
> 
> ```python
> from datasets import load_dataset
> # This gives an error "DatasetGenerationError: An error occurred while generating the dataset""
> # dataset = load_dataset("mattobaldo/personal_dataset")
> 
> # So instead I use one from the tutorial:
> dataset = load_dataset("explodinggradients/fiqa", "ragas_eval")
> 
> # the rest is the same as @MattiaBaldinetti  issue in https://github.com/explodinggradients/ragas/issues/286:
> 
> from ragas.metrics import (
>     answer_relevancy,
>     faithfulness, 
>     context_recall, 
>     context_precision, 
> )
> 
> 
> from ragas import evaluate
> result = evaluate(
>     dataset["baseline"],
>     metrics=[
>         context_precision,
>         faithfulness,
>         answer_relevancy,
>         context_recall],)
>         
>         
> # {'ragas_score': 0.6483, 'context_precision': 0.6333, 'faithfulness': 0.7384, 'answer_relevancy': 0.9251, 'context_recall': 0.4639}
> ```

Hello.
Yes the code is similar because I followed the Ragas own guide to use a personal dataset. There was a problem with the format of a column. Strange because I had made copy and paste of dataset used by Ragas.
Thanks the same for the answer
Have a good day

hi @MattiaBaldinetti did you get any resolution on this? As im facing the exact same issue but couldnt find a solution its been weeks now.
I am also using my personal dataset and getting error
`ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Value'>
`
Please support if you got a solution

hi @nelsonauner @shahules786 @samedovzaur please let me know what is the resolution to this. I am desperately looking for a solution..
Looking forward to your kind support.

This is not working as expected for personal dataset whereas for the sample dataset it is working fine

> hi @MattiaBaldinetti did you get any resolution on this? As im facing the exact same issue but couldnt find a solution its been weeks now. I am also using my personal dataset and getting error `ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Value'> ` Please support if you got a solution

Hi!
I decided to load the data into a json in the following way:
```
{
   "question" : "..." ,
   "ground_truths" : [ "..." ] ,
   "answer" : "..." ,
   "contexts" : [ "..." ]
}
```

To import it the right way you just do:
```
from dataset import load_dataset

dataset = load_dataset('json', 'path_of_your_json')
```

See if this works for you !!

@MattiaBaldinetti so do you mean i store my data in json instead of csv file?

> @MattiaBaldinetti so do you mean i store my data in json instead of csv file? 

I had problems with csv, so I used a json. 
Try turning your csv file into json using the standard above

hi @MattiaBaldinetti this is not working for me. Please let me know what is going wrong?

```
import pandas as pd

# Read the CSV file into a DataFrame.
df = pd.read_csv('hr_queries_ragas.csv')

# Convert the DataFrame to a JSON string.
json_string = df.to_json()

# Write the JSON string to a file.
with open('hr_queries_ragas.json', 'w') as f:
    f.write(json_string)
```
```
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

# list of metrics we're going to use
metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
    harmfulness,
]
```

```
for m in metrics:
    m.__setattr__("llm", ragas_azure_model)

from ragas import evaluate
```

```
dataset = load_dataset('json', 'queries_ragas')

result = evaluate(
    dataset["train"],
    metrics=metrics,
)
```


<img width="1147" alt="Screenshot 2023-12-28 at 6 52 22 PM" src="https://github.com/explodinggradients/ragas/assets/47422016/b0a92cad-1efa-496c-90e6-fd3d02b75a97">


Hi @mariamaslam 

Have you checked that the dataset is correct?
There seems to be a problem with this. You should check if the csv is properly formatted and if the final json is correct


@mariamaslam Last time when i was using ragas, created this dataset on huggingface as they did in the ragas docs, maybe you can take a look at it and feel free to use if that works for you then try to crate your own on hugginface then imply the dataset: https://huggingface.co/datasets/zaursamedov1/ragas_SS

*It's been a lot last time I had used ragas in my code that's why i ve forgot code part unf.

This is working guys...
I have changed the contexts column in seq form


from datasets import Dataset, Value, Sequence

# Assuming your CSV file is named 'your_data.csv'
csv_file_path = 'eval.csv'

# Read the CSV file into a Pandas DataFrame
import pandas as pd
df = pd.read_csv(csv_file_path)

df['contexts'] = df['contexts'].apply(lambda x: eval(x) if isinstance(x, str) else [])

# Create a dictionary with the specified features
data_dict = {
    'question': df['question'].tolist(),
    # 'ground_truths': df['ground_truths'].tolist(),
    'contexts': df['contexts'].tolist(),
    'answer': df['answer'].tolist(),
    'evolution_type': df['evolution_type'].tolist(),
    'episode_done': df['episode_done'].tolist()
    
}

# Create a Dataset using the dictionary
fiqa_dataset = Dataset.from_dict(data_dict)

# Print the information about the created dataset
print(fiqa_dataset)


Don't know if it is helpful now, but I was plagued by this issue for a long time. Then the following worked for me:
`"contexts": DF["chain_context"].apply(lambda x: [x])`

Ragas ver. 0.1.5 

This is my way to figure out this error.
I compared my data with the example. even the dataset is same as 
Dataset({
    features: ['question', 'ground_truth', 'answer', 'contexts'],
    num_rows: 10
})
First check the 'context' in datasets, found I processed the context in RAG, so the context is not in list format. For RAGAS, use contexts.append([r.payload['page_content'] for r in results]) for dataset dictionary. Then it works.

> Don't know if it is helpful now, but I was plagued by this issue for a long time. Then the following worked for me: `"contexts": DF["chain_context"].apply(lambda x: [x])`
> 
> Ragas ver. 0.1.5

This worked for me !

> e I followed the Ragas own guide to use a personal dataset. There was a problem with the format of a column. Strange because I had made copy and paste of dataset used by Ragas.
> Thanks the same for the ans

So basically contexts needs to be a list of strings rather than a string?

> > e I followed the Ragas own guide to use a personal dataset. There was a problem with the format of a column. Strange because I had made copy and paste of dataset used by Ragas.
> > Thanks the same for the ans
> 
> So basically contexts needs to be a list of strings rather than a string?

Hi Tommy,
yes I used a list of string for the contexts and it worked for me. 

Basically, the fact that the variable is in plural, makes me think it is made for more contexts. 
So I see as perfect solution the list of strings.