zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Is there a leaderboard related to the Ragas (RAG Assessment) metrics?
Root cause: Unimplemented Feature Gap


I tested my RAG application using Ragas and gained scores for various metrics. However, I wish there was a leaderboard that would allow me to see where my scores stand in comparison. Is there such a leaderboard available?

<!-- Greeting -->
Привет, @njw-bupt! Я здесь, чтобы помочь вам с вашей проблемой. Я бот и могу помочь вам решить ошибки, ответить на вопросы и даже стать контрибьютором.

<!-- Answer -->
The Ragas repository does not include a built-in feature or module for comparing scores against a leaderboard <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md)</sup>. However, you can submit your evaluation scores to Tonic Validate using the `RagasValidateApi` class, which allows you to upload results for leaderboard purposes <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/tonic-validate.ipynb)</sup>.

Additionally, you can visualize and explore your evaluation results using platforms like Zeno and Arize's Phoenix. These platforms provide tools for visualizing the data and tracking performance over time <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/tonic-validate.ipynb)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/zeno.ipynb)</sup><sup>[[4]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/arize.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


There is no leaderboard or plans to add it sadly. The goal of the project was to give you the tools to make your own leaderboard and benchmark improvements for your usecases 🙂 .
Repository: explodinggradients/ragas
Issue: Where can I find the default prompt used in TestsetGenerator?
GitHub labels: question

answered


<!-- Answer -->
The default prompt used in the `TestsetGenerator` can be found in the `src/ragas/testset/prompts.py` file. Here are some examples of prompts defined in that file:

```python
reasoning_question_prompt = Prompt(
    name="reasoning_question",
    instruction="""Complicate the given question by rewriting question into a multi-hop reasoning question based on the provided context.
    Answering the question should require the reader to make multiple logical connections or inferences using the information available in given context.
    Rules to follow when rewriting question:
    1. Ensure that the rewritten question can be answered entirely from the information present in the contexts.
    2. Do not frame questions that contains more than 15 words. Use abbreviation wherever possible.
    3. Make sure the question is clear and unambiguous.
    4. phrases like 'based on the provided context','according to the context',etc are not allowed to appear in the question.""",
    examples=[
        {
            "question": "What is the capital of France?",
            "context": "France is a country in Western Europe. It has several cities, including Paris, Lyon, and Marseille. Paris is not only known for its cultural landmarks like the Eiffel Tower and the Louvre Museum but also as the administrative center.",
            "output": "Linking the Eiffel Tower and administrative center, which city stands as both?",
        },
        {
            "question": "What does the append() method do in Python?",
            "context": "In Python, lists are used to store multiple items in a single variable. Lists are one of 4 built-in data types used to store collections of data. The append() method adds a single item to the end of a list.",
            "output": "If a list represents a variable collection, what method extends it by one item?",
        },
    ],
    input_keys=["question", "context"],
    output_key="output",
    output_type="str",
    language="english",
)
```

This is one of the prompts defined in the file, and similar prompts are defined for different types of questions and contexts <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/prompts.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Custom prompt/seed for question generation
Root cause: Documentation Deficiency

Is it possible to feed in a custom prompt into TestsetGenerator via the API? Alternatively, is it possible to modify the seed question prompt?

The only way I see of doing so is by modifying internal RAGAS code, would be useful to do so via the external API.

<!-- Greeting -->
Hi @ableiweiss! I'm a bot here to help you with any questions or issues you might have. Let's tackle this together!

<!-- Answer -->
Yes, it is possible to feed a custom prompt into `TestsetGenerator` via the API. You can modify the `seed_question_prompt` by defining a new `Prompt` object with your specific instructions and examples. Here is an example:

```python
from ragas.llms.prompt import Prompt
from ragas.testset.generator import TestsetGenerator
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Define the custom prompt
custom_prompt = Prompt(
    name="custom_seed_question",
    instruction="Generate a custom question that can be fully answered from the given context. The question should be formed using the provided topic.",
    examples=[
        {
            "context": "Quantum computing leverages the principles of quantum mechanics to perform computations at speeds unattainable by classical computers.",
            "keyphrase": "Quantum computing",
            "question": "What are the advantages of quantum computing over classical computing?",
        },
        {
            "context": "Machine learning algorithms enable computers to learn from data and make decisions based on patterns and insights.",
            "keyphrase": "Machine learning",
            "question": "How do machine learning algorithms improve decision-making processes?",
        },
    ],
    input_keys=["context", "keyphrase"],
    output_key="question",
    output_type="str",
)

# Initialize the TestsetGenerator with the custom prompt
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# Replace the default prompt with the custom prompt
simple.seed_question_prompt = custom_prompt

# Generate test sets using the custom prompt
documents = load_your_documents()  # Replace with your document loading logic
testset = generator.generate_with_langchain_docs(documents, 10, {simple: 1.0}) 
testset.to_pandas()
```

This example demonstrates how to create a custom prompt and integrate it into the `TestsetGenerator` class for generating test sets <sup>[[1]](https://github.com/explodinggradients/ragas/issues/1121)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: get keyphrases from TestGenerator
GitHub labels: enhancement

stale

Is it get the keyphrases extracted by TestGenerator using the API?


<!-- Answer -->
To get the keyphrases extracted by `TestGenerator` using the API, you can use the `KeyphraseExtractor` class and its `extract` method. Here is an example of how you can do this:

```python
from ragas.testset.extractor import KeyphraseExtractor
from ragas.llms.base import BaseRagasLLM
from ragas.testset.docstore import Node

# Assuming you have an instance of BaseRagasLLM and a Node
llm = BaseRagasLLM()  # Initialize your LLM instance
node = Node(page_content="Your text content here")  # Initialize your Node with the text content

# Create an instance of KeyphraseExtractor
keyphrase_extractor = KeyphraseExtractor(llm=llm)

# Extract keyphrases
keyphrases = await keyphrase_extractor.extract(node=node, is_async=True)

print(keyphrases)
```

This code initializes the `KeyphraseExtractor` with an instance of `BaseRagasLLM` and a `Node` containing the text content. It then calls the `extract` method to get the keyphrases. The `extract` method returns a list of keyphrases extracted from the text content.

Additionally, the `TestsetGenerator` class provides a convenient way to initialize the `KeyphraseExtractor` when creating a document store. Here is an example of how you can use `TestsetGenerator` to set up the keyphrase extraction:

```python
from ragas.testset.generator import TestsetGenerator
from ragas.llms import BaseRagasLLM
from ragas.embeddings.base import BaseRagasEmbeddings
from ragas.testset.docstore import InMemoryDocumentStore

# Initialize your LLM and embeddings instances
generator_llm = BaseRagasLLM()
critic_llm = BaseRagasLLM()
embeddings = BaseRagasEmbeddings()

# Create a TestsetGenerator instance
testset_generator = TestsetGenerator(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings=embeddings,
    docstore=InMemoryDocumentStore()
)

# Now you can use the testset_generator to handle keyphrase extraction
```

This approach ensures that the `KeyphraseExtractor` is properly integrated into the document store, making it easier to manage and use within your RAG pipeline <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/extractor.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/prompts.py)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/generator.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@ableiweiss that is an interesting suggestion. How are you planning to use it?

We are working on a rewamp for testset generation #1016 will add it to that but if you could explain your reasoning and usecase it will help us a lot 🙂
Repository: explodinggradients/ragas
Issue: Include 'Reason/Justification' in the result along with scores
Root cause: Documentation Deficiency

I would like to know how to include justifications in the results along with the scores when using the evaluate function in the RAGAS framework. Currently, the function only returns the scores without any explanations or justifications, which makes it difficult to understand the reasoning behind the scores.

@DincyKajol are you using any tracing tools? can you check out the docs [Explainability through Logging and tracing | Ragas](https://docs.ragas.io/en/stable/howtos/applications/tracing.html) and let me know if that helps?

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Is is possible to input a string of page content in TestGenerator instead of DirectoryLoader
Root cause: Documentation Deficiency

In other words, is it possible to pass in a single page content string into TestGenerator instead of directory of documents?


<!-- Answer -->
Yes, it is possible to input a single page content string into `TestGenerator` instead of using `DirectoryLoader`, which requires a directory of documents. You can use the `InMemoryDocumentStore` to handle individual documents.

Here is an example of how you can achieve this:

```python
from langchain_openai.chat_models import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings
from ragas.testset.docstore import InMemoryDocumentStore
from ragas.testset.extractor import KeyphraseExtractor
from ragas.testset.generator import TestsetGenerator
from langchain.text_splitter import TokenTextSplitter

# Initialize the necessary components
generator_llm_model = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm_model = ChatOpenAI(model="gpt-4")
embeddings_model = OpenAIEmbeddings(model="text-embedding-ada-002")
keyphrase_extractor = KeyphraseExtractor(llm=generator_llm_model)

# Create an InMemoryDocumentStore
splitter = TokenTextSplitter(chunk_size=1024, chunk_overlap=0)
docstore = InMemoryDocumentStore(
    splitter=splitter,
    embeddings=embeddings_model,
    extractor=keyphrase_extractor,
)

# Add a single document to the docstore
single_document_content = "Your single page content string here."
docstore.add_documents([Document(content=single_document_content)])

# Create the TestsetGenerator
testset_generator = TestsetGenerator(
    generator_llm=generator_llm_model,
    critic_llm=critic_llm_model,
    embeddings=embeddings_model,
    docstore=docstore,
)

# Now you can use the testset_generator to generate test sets
```

This code snippet demonstrates how to initialize the `TestsetGenerator` with a single document content string using the `InMemoryDocumentStore` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/generator.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: ragas evaluate with llama_index and langchain doesn't seem to work with Azure OpenAI
Root cause: Configuration Error

Azure OpenAI requires the special parameter `deployment` or `deployment_id`

The langchain wrappers seem to mostly have been updated to accomodate for this but it doesn't seem to work with ragas.

 I ended up getting faithfulness working but updating the `generate` method with:

from 
```

    elif isinstance(llm, BaseChatModel):
        ps = [p.format_messages() for p in prompts]
        result = llm.generate(ps, callbacks=callbacks)
        
        
```
to

```
    elif isinstance(llm, BaseChatModel):
        ps = [p.format_messages() for p in prompts]
        result = llm.generate(ps, callbacks=callbacks, deployment_id='<my_id>', api_version='<my_version>')

```

but with `answer_relevancy` I hit the same issue when it tries to run: 

```
│    91 │   def calculate_similarity(                                                              │
│    92 │   │   self: t.Self, question: str, generated_questions: list[str]                        │
│    93 │   ):                                                                                     │
│ ❱  94 │   │   question_vec = np.asarray(self.embedding.embed_query(question)).reshape(1, -1)     │
│    95 │   │   gen_question_vec = np.asarray(                                                     │
│    96 │   │   │   self.embedding.embed_documents(generated_questions)                            │
│    97 │   │   )
``` 

Any ideas? 

Hi @Data-drone , It's possible to use AzureOpenAI with ragas. You have to init a new `AzureChatOpenAI` LLM as mentioned in this [notebook](https://github.com/explodinggradients/ragas/blob/main/docs/guides/llms.ipynb)

To know how to init AzureOpenAI using Langchain you can refer this [guide](https://python.langchain.com/docs/integrations/llms/azure_openai) 

I have setup AzureOpenAI and Llama_index is working fine. But it seems like when you call the service context from ragas it is losing the Azure config somehow?  

We did have a running thread where we fixed this one time but I would love to get on a call with you when you are available and help debug this. The issue is that we haven't gotten access to Azure Openai endpoints yet so can't debug from our end

the original thread: https://discord.com/channels/1119637219561451644/1123849273562386453/1148224350915407962

it would be great if @Data-drone could commend in that thread, We'll debug this in discord?

Sure should I try the latest master?

you could try that but we've released a 0.0.13 version too. Do try it out and let us know.
you might have to configure the AzureOpenAI separately for ragas metrics. You can use [this guide](https://github.com/explodinggradients/ragas/blob/main/docs/guides/llms.ipynb) to see how.
Repository: explodinggradients/ragas
Issue: What are "contexts" in the generated Testset?
GitHub labels: question

answered

[+] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
When I create the testset using the test set generator, the resulting testset contains "contexts". What are these contexts?
Are these the list of chunks retrieved while generating the ground truth answers for each question?

<!-- Greeting -->
Hey @neerajchhimwal! 👋 I'm here to assist you with any questions or issues you have. Let's tackle this together!

<!-- Answer -->
The "contexts" in the generated testset are the retrieved contexts corresponding to each question. These contexts are indeed the list of chunks retrieved while generating the ground truth answers for each question. Each question can retrieve multiple text chunks, which are then used to formulate comprehensive answers <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/prepare_data.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/testset_generation.md)</sup>.

checkout https://docs.ragas.io/en/stable/concepts/testset_generation.html for more info

<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Support vLLM
Root cause: Unimplemented Feature Gap

```
Exception: n=1 was passed to generate but the LLM VLLM
Params: {} does not support it. Raise an issue if you want support for {llm}.
```

Following the instructions from the exception. 

Thank you for raising this and we are working on adding the feature to support this. Are you using this https://github.com/vllm-project/vllm directly or via huggingface?

Also, another thing to keep in mind is that all the tests have been done with openai so other opensource models might perform differently, which models are you considering using? 

@sam-h-bean made a PR for this https://github.com/explodinggradients/ragas/pull/118
Repository: explodinggradients/ragas
Issue: [R-283] Adding more `token_usage_parser`s for various LLM providers
Root cause: Unimplemented Feature Gap

There are a lot more LLM providers for whom we don't have parsers. We would really apprecite you inputs here in 
1. suggesting parsers that you need. This will help us prioritize which ones most used.
2. contribute your own. If you make any for your usecase do consider contributing back too 🙂 

### Supported Currently
(will keep this updated as we add more)
- OpenAI
- Anthropic

### Need Support for
(will use this to tally the requests from the community)
- AWS Bedrock
- GCP and Gemni

<sub>[R-283](https://linear.app/exploding-gradients/issue/R-283/adding-more-token-usage-parsers-for-various-llm-providers)</sub>
Repository: explodinggradients/ragas
Issue: [R-285] add a decorator to track percentage of evaluations and testset failures
Root cause: Unimplemented Feature Gap



<sub>From [SyncLinear.com](https://synclinear.com) | [R-285](https://linear.app/exploding-gradients/issue/R-285/add-a-decorator-to-track-percentage-of-evaluations-and-testset)</sub>

fixed with #1171
Repository: explodinggradients/ragas
Issue: Update dependancy to Pydantic to 2.X.X
Root cause: External Dependency Breakage


with #209 we have removed the limit of `pydantic<2` from our end but a lot of tools in the ecosystem still have that limit but hopefully pip should handle that

do test out the latest release and check if this solved your issue and feel free to reopen this issue if it's still not fully fixed 🙂 

Cheers
Jithin
Repository: explodinggradients/ragas
Issue: Swiftide integration

Hey everyone, amazing work on Ragas! Recently I've been working on [Swiftide](https://github.com/bosun-ai/swiftide), and I'm looking to integrate with an evaluation framework. Rust is great at performance, Python is king at data anytics.

We don't have python bindings yet, but I'd be happy to do everything on the Swiftide side to put down something cool.

@timonv thanks for raising this suggestion - would love to do it, Swiftide looks great! 
How should we do this though? One way could be with #1011 and making some rust bindings? 

but I think the easiest would be to create a notebook/documentation with utils code that makes it easier
1. create the dataframe needed by ragas with swiftide
2. evaluate with ragas
3. show results

would love to hear your suggestions 🙂 

@jjmachan Is the dataset basically question, contexts (retrieved documents?) and an answer? #1011 would be amazing, my guess is that most users would prefer to start out with a python notebook so they can analyze and compare different setups.

Any ideas on how we would run it from a notebook? Full python bindings are a bit of an effort as we are heavy on the less trivial features. Maybe I can create a tiny wrapper support library that can call a fully constructed pipeline from python.

An even simpler poc approach could be to simply call a compiled binary and have the dataframe output to a file or stdout.


alrighty maybe lets start with the notebook then and take it up from there? would love to have a quick chat with you before so we can plan?
Repository: explodinggradients/ragas
Issue: Answer relevancy use custom embedding
Root cause: Unimplemented Feature Gap

At the moment,  this metrics use OpenAIEmbedding by default.
Can we allow this to use custom embedding instead ?

Hi @starrywheat , which metrics are you using? and which embedding are you planning to use instead of OpenAI embeddings? 

Hi @shahules786,  thanks for the prompt response. As the title suggests, I am looking at the answer relevancy  which I believe require sentence embedding for the answers comparison. 
And besides OpenAI, I want to be able to use open source embedding such as BAAI/bge-base-en. This would be a reasonable and better choice than text-embedding-ada-002, according to the [HF leaderboard](https://huggingface.co/spaces/mteb/leaderboard). Plus, having flexibility is often good. 



@shahules786, I made it worked in my local branch by modifying a couple lines in the answer_relevancy.py.
I am happy to contribute to this.

@starrywheat  great, you can raise PR if you already have the code. I will have to do some ablation studies before merging it, and the shorter max sequence length of Bge embedding is a concern. Even though I am sure that it's better than Ada. 

@shahules786 sounds  great I will do that. 
And happy to further discuss the pros and cons of various embeddings  and how best to integrate  into the metrics  in  the PR

@shahules786  I don't think I have access right to create new branch for PR. Can you help me with that

@starrywheat Are you working on a forked version? If not I would suggest to fork the repo and then make your changes on it. Then you should be able to raise a PR from your forked ragas version to here. 
Here is the [guide](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork)

Hi all, I am interested in using bge-small-en-v1.5 instead of open ai ones, as mentioned in this issue. Seems that this capability has been added. I am looking for an example code snippet to how to wrap/calll the custom embeddings. I'd appreciate your help! 

Hi @lalehsg , please install ragas from source and follow https://docs.ragas.io/en/latest/howtos/customisations/embeddings.html

Hi @shahules786 This was very helpful and I was able to run locally easily. Now, I need to run it on a more restricted env. so, can't really do install from source. was wondering if this has been part of a recent version release. so that i just pip install, the usual way and get the feature. thanks again in advance!