zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Function is_base64 falsely classifies text as image
Root cause: Algorithmic Error

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
While using MultiModalFaithfulness, class ImageTextPromptValue(PromptValue) has is_base64 function. Given a text "Test" is marks it as True even tho it's not even close.

Ragas version: 0.2.6
Python version: 3.9

**Code to Reproduce**
```python
import base64

# Copied from class ImageTextPromptValue(PromptValue) and removed `self`
def is_base64(s):
    try:
        if isinstance(s, str):
            # Try to decode the string
            if base64.b64encode(base64.b64decode(s)).decode("utf-8") == s:
                return True
        return False
    except Exception:
        return False


print(is_base64("Test"))
```

**Error trace**

**Expected behavior**
Expected this to be marked as False


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: Invalid \escape and StringIO AttributeError in RAGAS Evaluation
Root cause: Validation Gap

**Describe the bug**
When evaluating datasets using RAGAS with a custom LLM like mixtral-8x7b-instruct and the vllm inference backend, several exceptions related to escape characters and StringIO objects are raised.

Ragas version: 0.2.6
Python version: 3.11

**Code to Reproduce**
from datasets import load_dataset
from ragas.metrics import faithfulness, answer_relevancy, answer_similarity, answer_correctness
from langchain_openai import ChatOpenAI
from langchain_community.embeddings import HuggingFaceEmbeddings
import pandas as pd
from ragas.llms import LangchainLLMWrapper
from ragas import evaluate

##custom model 
openai_api_key = "Empty"
openai_api_base = "mixtral model url here" ##hosted internally, vllm serving
model="mixtral-8x7b-instruct"

eval_llm = ChatOpenAI(
    model=model,
    api_key=openai_api_key,  
    base_url=openai_api_base,
    # max_tokens=max_tokens,
    temperature=0.1
)

##load dataset 
dataset = load_dataset(
    "explodinggradients/amnesty_qa",
    "english_v3",
    trust_remote_code=True
)

eval_dataset = EvaluationDataset.from_hf_dataset(dataset)

evaluator_llm = LangchainLLMWrapper(eval_llm)
metrics = [
    Faithfulness(llm=evaluator_llm)
]

##eval 
results = evaluate(dataset=eval_dataset, metrics=metrics)

**Error trace**
![image](https://github.com/user-attachments/assets/062572a8-90c7-4729-8e04-70bc276730c5)

**Expected behavior**
[evaluated dataframe as shown in docs
](https://docs.ragas.io/en/stable/getstarted/rag_evaluation/#exporting-and-analyzing-results)
**Additional context**
following the example from docs https://docs.ragas.io/en/stable/getstarted/rag_evaluation/#exporting-and-analyzing-results
<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I have the same problem, is there any update?
Repository: explodinggradients/ragas
Issue: [R-315] Integration for SCIPE metric 
Root cause: Unimplemented Feature Gap

> SCIPE is a powerful tool for evaluating and diagnosing LLM (Large Language Model) graphs or chains. It assesses LLM responses and employs a custom algorithm to identify problematic nodes within the LLM chain.

We need to integrate it by
- implementing the error and probability caclculation into Ragas
- make the LLM evaluator and aspect_critic so that it can be aligned with ragas optimize


ref: https://github.com/garg-ankush/scipe

<sub>[R-315](https://linear.app/exploding-gradients/issue/R-315/integration-for-scipe-metric)</sub>

This looks interesting. I would like to first document an approach for implementing it here. Can you assign this issue to me?

Hey @ayulockin are you working on it? If not can I take it, it looks interesting to me.

Hey Parthib, yes I am working on it. I will let you know if I need
collaboration.

I am trying to come up with a design for this metric.

On Sun, 1 Dec 2024, 12:25 Parthib Datta, ***@***.***> wrote:

> Hey @ayulockin <https://github.com/ayulockin> are you working on it? If
> not can I take it, it looks interesting to me.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/explodinggradients/ragas/issues/1702#issuecomment-2509603239>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHNS4ZY2FC4I3TUVHMDY7OD2DKXFNAVCNFSM6AAAAABSOIMOQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGYYDGMRTHE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Ooh ok. Let me know if you need any collaboration.
Thank you.
Repository: explodinggradients/ragas
Issue: _reproducibility in faithfulness not being used
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
The _reproducibility parameter is defined for faithfulness metric but never being used.

Ragas version: main branch

**Expected behavior**
It should be used like in other metrics such as context recall.
Repository: explodinggradients/ragas
Issue: Evaluate graph RAG LLM application
Root cause: Documentation Deficiency

**Your Question**
Hi, I developed a graph RAG LLM application and want to do evaluation for it like [here](https://docs.ragas.io/en/stable/getstarted/rag_evaluation/) and [here](https://docs.ragas.io/en/stable/howtos/integrations/_llamaindex/#building-the-testset) in the documentation. But it seems that the current evaluation is for text document RAG? May I know is there a graph RAG LLM application evaluation feature? My question is actually similar to [this guy in stakeoverflow](https://stackoverflow.com/questions/78881336/best-approach-to-evaluate-a-graph-rag-pipeline-using-metrics) but there is no solution either. Thanks!




Hi @sahusiddharth Is there any update on this? Or can we do the evaluation just like [here](https://docs.ragas.io/en/stable/howtos/integrations/_llamaindex/) with prepared reference graph node data and the retrieved ones? 

@hliuci @Howie-Arup I'm not sure why we might need a different set of metrics other than the standard ones already present in Ragas?

We have
- [Context Precision - Ragas](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)
- [Context Recall - Ragas](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/)

which should do the job right? all you have to do is convert the nodes to text before passing it into evaluate.

let me know if you need more help here - happy to help 🙂 

Got it @jjmachan, thanks a lot for your help!

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Same metrics function with different content must be rewrite the function ??
Root cause: Architectural Constraint

Now I want to apply one of the metrics functions in the evaluation metrics to different content.
For example, the SemanticSimilarity function by default takes {"reference", "response"} to measure the similarity between two pieces of content. Now I want to measure the similarity between {"user_input", "retrieved_contexts"}. 
How should I proceed? Does it mean that I have to rewrite the whole method myself?

hey there @YYH211 sadly this is not possible in metrics other than general purpose metrics sadly https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/

but I can make this fix for you today and merge it in 🙂 

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: How to set the ground_truth datatype correctly when creating a DatasetDict from scratch?
Root cause: Documentation Deficiency

Hi,

After successfully trying the fiqa example, I'm now trying to get ragas to work on my own dataset. Below is a toy example I'm working on:

```
d = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on January 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'ground_truths': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
df = pd.DataFrame(data=d)

test = DatasetDict()
test['test'] = Dataset.from_list(
    mapping=df[["question", "answer", "ground_truths"]] \
                    .to_dict(orient='records'),
     # features={"ground_truths": Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
)
# test['test'] = test['test'].map(lambda example: {'ground_truths': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)})

test_result = evaluate(
    test['test'],
    metrics=[answer_relevancy]
)
```

It works fine if I take out 'ground_truths' and just include the question and answer in Dataset.from_list(), but when I try to include the ground_truth I get an error about the datatype. I've tried setting the type with the 'features' argument in the Dataset call and afterwards with a map function but both didn't seem to work. Any tips or hints on how to set the datatype correctly?


Regards, Koen

Hey @kpeters , thanks for raising the issue. `ground_truths` is a list of list. So if you change your dict to 
```python
d = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on January 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'ground_truths': [['The first superbowl was held on January 15, 1967'], ['The New England Patriots have won the Super Bowl a record six times']]
}
df = pd.DataFrame(data=d)

````
it should work. 
The idea is that there can be multiple correct answers to a single question. Does that make sense?


@shahules786, Thanks, that works indeed! I'm not used to the possibility of having multiple correct answers, but I'll take it for now.
Repository: explodinggradients/ragas
Issue: ragas on AWS
Root cause: Documentation Deficiency

Hi,
Facing error while installing ragas both on EC2 and SageMaker studio. In EC2 it is failing with No space left on device. In Sagemaker studio, the code is not able to identify the code ragas library after installation (and restart). It will be good if you can publish documentation on how to install ragas on AWS.

Thanks
Repository: explodinggradients/ragas
Issue: Test Set Generation Failure
Root cause: Validation Gap

Issue Description
The test set generation process failed at 74% completion, throwing an output parser error. Specifically, the issue raised while applying transformations (SummaryExtractor, HeadlinesExtractor) using fix_output_format and headlines_extractor_prompt.

Error Details
The output parser failed to parse the response from the Azure OpenAI deployment.
The error is repetitive for multiple transformations in the pipeline.

Error trace

2024-12-04 15:09:26,387 httpx _send_single_request 1773 INFO: 
HTTP Request: POST 
Applying [SummaryExtractor, HeadlinesExtractor]:  74% | 528/716 [09:17<05:48, 1.85s/it]
 
2024-12-04 15:09:28,267 ragas.prompt.pydantic_prompt generate_multiple 207 ERROR: 
Prompt fix_output_format failed to parse output: The output parser failed to parse the output after 0 retries.
 
2024-12-04 15:09:28,267 ragas.prompt.pydantic_prompt generate_multiple 207 ERROR: 
Prompt headlines_extractor_prompt failed to parse output: The output parser failed to parse the output after 0 retries.
 
2024-12-04 15:09:28,267 ragas.testset.transforms.engine run_coroutines 54 ERROR: 
Unable to apply transformation: The output parser failed to parse the output after 0 retries

Hey @abivarshinit-intechive Is there a way I could reproduce this error? Few documents or something. 

I have the same issue recently.. are you able to resolve it? 

Kindly share your code to reproduce @zoey-lyu 
Thanks
Repository: explodinggradients/ragas
Issue: Multi-lingual support for question generation and evaluation in Ragas v2
Root cause: Unimplemented Feature Gap

**Describe the Feature**
A way to generate test sets (and run evaluations) in languages other than English.

**Why is the feature important for you?**
I do not live in an English-speaking country. 
The source material I am using is multi-lingual

**Additional context**
When using Ragas v1, there was an option to "_adapt_" the generator to different languages:

Example v1 code:
`generator.adapt('french', evolutions=[simple, reasoning, conditional, multi_context], cache_dir=cache_dir)`

I am not able to use this code in the new version of Ragas
(this generates the following error: `AttributeError: 'TestsetGenerator' object has no attribute 'adapt'`)

**Add any other context about the feature you want to share with us.**
When using non-English source material the resulting question-answer pairs are a mix between the source language and English. 

I also cannot seem to find how to do this in the v2 documentation. 
If there is no translation feature present, could it be possible to append something to the generation / evaluation prompt?


Hey @stilkin-pxl We are working on this. Will try to ship it with this week's release it possible.  Meanwhile, we have shipped items that enables you to create custom single and multi hop queries, refer docs https://docs.ragas.io/en/latest/howtos/customizations/

Hey @stilkin-pxl Just shipped it here https://github.com/explodinggradients/ragas/pull/1734

Closing the issue with PR merged. Refer docs https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_language_adaptation/#intialize-required-models
Repository: explodinggradients/ragas
Issue: Error while generating synthetic testset
Root cause: Validation Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

I encountered an error while trying to generate a synthetic testset using the TestsetGenerator from the ragas library. Below is the code snippet and the corresponding error message.

What could be causing this error?
Are there specific requirements or filters for the documents or embeddings that I might have missed?
Could this issue be related to the compatibility of the LLM or embedding model?
Thank you.

from langchain.document_loaders import DirectoryLoader, PyPDFLoader
from langchain_ollama import ChatOllama
from langchain_ollama import OllamaEmbeddings
from ragas.testset import TestsetGenerator

loader = DirectoryLoader(
  r'./ragtestgen',
  glob='**/*.pdf',
  loader_cls=PyPDFLoader
)

docs = loader.load()
print(f"Loaded {len(docs)} documents.")

llm = ChatOllama(model="llama3.1:8b")
embedding_model = OllamaEmbeddings(model="mxbai-embed-large:335m")

generator = TestsetGenerator.from_langchain(llm=llm, embedding_model=embedding_model)

testset = generator.generate_with_langchain_docs(documents=docs, testset_size=2)
![image](https://github.com/user-attachments/assets/f75b8a49-5170-4c93-a1ce-c440e41b6c48)



I'm getting other problems now
Any help?
Thank you.
![image](https://github.com/user-attachments/assets/4d43729c-4440-4df6-a058-7ceb312de7ee)
![image](https://github.com/user-attachments/assets/9cd269ba-52a1-4b15-abc7-202f4effdca4)
Repository: explodinggradients/ragas
Issue: Relying on pySBD is not feasible- package hasn't been updated in 3 years (sentence segmentation)
Root cause: External Dependency Breakage

**Describe the Feature**
One of RAGAS dependencies is the package pySBD. This package has several bugs but it seems to be largely unmaintained at this point. The last release was 3 years ago. This will expose RAGAS to bugs, security vulnerabilities and more.

**Why is the feature important for you?**
I found a bug related to pySBD and opened an issue there. I noticed that the repo seems to have no activity anymore. It's an extremely bad idea to have that as one of the main dependencies in my opinion. 

**Additional context**
Switching to a better known and more well-maintained package will avoid headaches for RAGAS maintainers in the future.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hey @diegodebrito thanks for reporting this. Any alternative package you can recommend to do just segmentation? 


Hi @shahules786, sorry for taking so long to answer. I am not a specialist on that, but maybe just standard Spacy would work for that purpose?

Something along these lines
```
import spacy

    split_sentences = []
    for answer in answers:
        doc = nlp(answer)
        sentences_with_index = {i: sent.text for i, sent in enumerate(doc.sents)}
        split_sentences.append(sentences_with_index)
```

What was the reason you went with pySBD in the first place?

Hey @diegodebrito We have removed the need for segmenting from ragas with the latest PR. Thank you.
Repository: explodinggradients/ragas
Issue: LangchainLLMWrapper not compatible with some BaseLanguageModel implementations (ChatOllama)
Root cause: Interface Contract Mismatch

Hi,

LangchainLLMWrapper [passes temperature to the](https://github.com/explodinggradients/ragas/blob/7eec750d841611827ac1f39dee6fc1a4c1d0f69f/src/ragas/llms/base.py#L191) generate_prompt method even though the [generate_prompt method from BaseLanguageModel does not explicitly define this parameter.](https://github.com/langchain-ai/langchain/blob/18386c16c7201e54dfaacbc18eed51da7965a7a5/libs/core/langchain_core/language_models/base.py#L166)

This leads to an issue with ChatOllama, which expects the temperature to be set upon object instantiation and not on the generate_prompt method as the kwargs are passed down up until the [ollama package method which does not accept kwargs](https://github.com/ollama/ollama-python/blob/4b10dee2b2a73242bb3ac89332f634cda2b6f633/ollama/_client.py#L288) and raises an exception because there is no such parameter defined. 

I wonder if this is something to tackle in ragas llm, langchain_ollama, or Ollama?

Best regards,
John

Hello John, 
Have you solved this problem?

Hi @shoxyouness ,

as a workaround I commented out passing the temperature argument in generate_text but I assume this may lead to unexpected behavior if the ragas library changes temperature on subsequent generate_text calls.

If i get it right...
Working with LlamaIndexLLMWrapper solves problem for me

```
from ragas.llms import LlamaIndexLLMWrapper
from llama_index.llms.ollama import Ollama

llm = Ollama(model="llama3.1" request_timeout=120.0) 
evaluator_llm = LlamaIndexLLMWrapper(llm)
```
Repository: explodinggradients/ragas
Issue: Problem with Azure OpenAI endpoint.
Root cause: Documentation Deficiency

Very interesting framework! I am struggling right now to try it out though:
I followed your guidelines (https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/quickstart-azure-openai.ipynb) to be able to run evaluate() with my Azure OpenAI endpoint. However, I get the following error: 'InvalidRequestError: The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.' .. even though my endpoint is up and running and works perfectly when I call it from my own code. There seems to be an issue with the langchain functionality AzureChatOpenAI of some kind. Any ideas, how I could resolve/debug this?

I managed to make it work with model `gpt-35-turbo`, as the deployment of `text-davinci-003` does not support `The chatCompletion operation`. Also, I have changed `deployment` to `deployment_name` in the `AzureChatOpenAI` definition:

```
azure_model = AzureChatOpenAI(
    deployment_name=os.getenv("OPENAI_LLM_DEPLOYMENT"),
    openai_api_base=os.getenv("OPENAI_API_BASE"),
    openai_api_type=os.getenv("OPENAI_API_TYPE"),
)
```

closing this since we also have an updated documentation site that documents this https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html

the change to deployment_name solved my issue. thanks!
Repository: explodinggradients/ragas
Issue: V0.2.7 Context Recall only returning 1 or 0 values
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
After upgrading the version to 0.2.7, the Context Recall metrics only return 1 or 0 values. Is this intentional? The previous version would return values between 0 and 1.

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


Hi @timelesshc,

Are you still facing this issue? 

I tested with a simple example, I received fractional values. The results you're seeing might be related to your data. I've included the code snippet I used below for reference:

```python
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import LLMContextRecall

sample = SingleTurnSample(
    user_input="Tell me about the Great Wall of China.",
    response="The Great Wall of China is in northern China and was built to protect against invasions.",
    reference="The Great Wall of China is in northern China, spans thousands of miles, and was built to protect against invasions.",
    retrieved_contexts=[
        "The Great Wall of China is a historic structure located in northern China.",
        "The wall was built over centuries to protect against invasions from nomadic tribes."
    ],
)

context_recall = LLMContextRecall(llm=evaluator_llm)
await context_recall.single_turn_ascore(sample)
```

Output
```
0.6666666666666666
```

In the above example:

1. The claim "The Great Wall of China is in northern China" can be derived from the first retrieved context.
2. The claim "The wall was built to protect against invasions" can be derived from the second retrieved context.
3. The claim "spans thousands of miles" is not directly supported by the retrieved contexts.

Hi @sahusiddharth , thanks for the follow up. 

Yes, it seems the result was related to my data. My "reference" narrative was too short to have multiple claims, so that using below equation would result a binary value. If I were to make a longer "reference" or break up the "retrieved_contexts" a bit more, the result seems to be fractional values.

<img width="245" alt="image" src="https://github.com/user-attachments/assets/dc65b2f2-893e-4678-9479-dbb2115ed1b5" />
Repository: explodinggradients/ragas
Issue: Generate score explanation during evaluation
Root cause: Documentation Deficiency

[X] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Is it possible to get the score explanation as part of the evaluation output? E.g. explain why the score is 0.


Hey, @Sergio-Zavota 
One way to do it is to export traces as shown in https://docs.ragas.io/en/stable/howtos/applications/_metrics_llm_calls/

Another way is to use a tool like app.ragas.io that we just launched https://docs.ragas.io/en/stable/howtos/customizations/metrics/train_your_own_metric/#review-and-annotate

We will add more docs to it soon.  Is this what you're looking for?