Repository: explodinggradients/ragas
Issue: Synthetic Data generation from the vector database
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I would like to create a Evaluation dataset in other words Synthetic data set. I would like to create these from the vector database.

**Additional context**
What is the right approach to create Synthetic data set, is it from the data source directory or from the vector database?


<!-- Greeting -->
Hey @saikrishna321! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

<!-- Answer -->
The right approach to create a synthetic dataset for evaluation from a vector database involves using the Ragas framework to generate high-quality and diverse data points. Here is a step-by-step guide:

1. **Install Dependencies and Import Libraries**:
   ```python
   !pip install "ragas<0.1.1" pypdf arize-phoenix "openinference-instrumentation-llama-index<1.0.0" "llama-index<0.10.0" pandas
   import pandas as pd
   pd.set_option("display.max_colwidth", None)
   ```

2. **Configure Your OpenAI API Key**:
   ```python
   import os
   from getpass import getpass
   import openai

   if not (openai_api_key := os.getenv("OPENAI_API_KEY")):
       openai_api_key = getpass("🔑 Enter your OpenAI API key: ")
   openai.api_key = openai_api_key
   os.environ["OPENAI_API_KEY"] = openai_api_key
   ```

3. **Download and Read Dataset**:
   ```python
   !git clone https://huggingface.co/datasets/explodinggradients/prompt-engineering-papers
   from llama_index import SimpleDirectoryReader

   dir_path = "./prompt-engineering-papers"
   reader = SimpleDirectoryReader(dir_path, num_files_limit=2)
   documents = reader.load_data()
   ```

4. **Generate Synthetic Test Dataset**:
   ```python
   from ragas.testset.generator import TestsetGenerator
   from ragas.testset.evolutions import simple, reasoning, multi_context
   from langchain_openai import ChatOpenAI, OpenAIEmbeddings

   TEST_SIZE = 25

   generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
   critic_llm = ChatOpenAI(model="gpt-4")
   embeddings = OpenAIEmbeddings()

   generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)

   distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}
   testset = generator.generate_with_llamaindex_docs(
       documents, test_size=TEST_SIZE, distributions=distribution
   )
   test_df = testset.to_pandas()
   test_df.head()
   ```

This approach leverages Ragas to synthetically generate a high-quality and diverse test dataset, which can then be used for evaluation purposes. Ensure that each document's metadata includes a key called `filename` to identify chunks belonging to the same document, which is crucial for the generation process <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/arize.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/testset_generation.md)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/testset_generation.md)</sup><sup>[[4]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/prepare_data.ipynb)</sup><sup>[[5]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Thanks for the code. I would like to generate the data set by connecting to Vector database and not from the document directory

<!-- Answer -->
To generate a synthetic dataset for evaluation by connecting directly to a vector database instead of using a document directory, you can modify the process to load data from your vector database. Here is an example of how you can achieve this:

1. **Connect to your vector database and retrieve documents.**
2. **Initialize the `TestsetGenerator` with the corresponding generator and critic LLMs.**
3. **Generate the test set using the retrieved documents.**

Here is a sample code snippet to illustrate this process:

```python
from llama_index.core import VectorStoreIndex
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Connect to your vector database and retrieve documents
# This is a placeholder function. Replace it with your actual database connection and retrieval logic.
def load_documents_from_vector_db():
    # Example: Replace with actual code to fetch documents from your vector database
    documents = [
        {"id": 1, "content": "Document content 1"},
        {"id": 2, "content": "Document content 2"},
        # Add more documents as needed
    ]
    return documents

documents = load_documents_from_vector_db()

# Initialize the TestsetGenerator object with the corresponding generator and critic LLMs
generator_llm = OpenAI(model="gpt-3.5-turbo-16k")
critic_llm = OpenAI(model="gpt-4")
embeddings = OpenAIEmbedding()

generator = TestsetGenerator.from_llama_index(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings=embeddings,
)

# Generate the test set
testset = generator.generate_with_llamaindex_docs(
    documents,
    test_size=5,
    distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},
)

# Convert the test set to a pandas DataFrame
df = testset.to_pandas()
df.head()
```

This code snippet demonstrates how to connect to a vector database, retrieve documents, and generate a synthetic dataset for evaluation using the `TestsetGenerator` from the `ragas` framework <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/llamaindex.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


What is your recommendations? Should I read from document or vector database?

<!-- Answer -->
It is recommended to read from a document to generate a synthetic dataset. This approach has been highlighted as reducing the time and effort in curating test data by 90% and allows for the creation of a wide variety of challenging test cases tailored to your retrieval and generation tasks <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/arize.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/applications/compare_embeddings.md)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/testset_generation.md)</sup><sup>[[4]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/testset_generation.md)</sup><sup>[[5]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/openlayer.ipynb)</sup>.

Here is a sample code snippet to illustrate this process:

```python
from llama_index import SimpleDirectoryReader
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Load documents from a directory
dir_path = "./prompt-engineering-papers"
reader = SimpleDirectoryReader(dir_path, num_files_limit=2)
documents = reader.load_data()

# Initialize the TestsetGenerator
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)

# Set question type distribution
distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}

# Generate test set
testset = generator.generate_with_llamaindex_docs(documents, test_size=25, distributions=distribution)
test_df = testset.to_pandas()
test_df.head()
```

This method ensures high-quality and diverse synthetic datasets, enhancing the robustness of your evaluation process.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Can I give a seed data and from that can ragas generate the evaluation data set?

<!-- Answer -->
Yes, you can give seed data and Ragas can generate the evaluation dataset from it.

Here is an example of how to do this:

```python
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Load your documents
# documents = load your documents

# Initialize the generator with OpenAI models
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# Define the distribution of question types
distributions = {
    simple: 0.5,
    multi_context: 0.4,
    reasoning: 0.1
}

# Generate the test set
testset = generator.generate_with_langchain_docs(documents, 10, distributions) 
testset.to_pandas()
```

This code demonstrates how to load documents and generate an evaluation dataset using Ragas. The framework uses an evolution-based synthetic data generation paradigm to ensure high-quality and diverse questions, significantly reducing the time and effort in curating test data <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/testset_generation.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/arize.ipynb)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/prepare_data.ipynb)</sup><sup>[[4]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I would give it a sample example and then generate dataset