Repository: explodinggradients/ragas
Issue: How can i set config parameters for test data generation
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
How can i set config parameters for test data generation for not getting connection timeout error

**Code Examples**
generator = TestsetGenerator.from_langchain(
    generator_llm=llm, critic_llm=llm, embeddings=embeddings
)

# Generate the test set
testset = generator.generate_with_langchain_docs(
documents=documents,
test_size=10,  # Example test size
distributions={simple: 0.5, reasoning: 0.3, multi_context: 0.2},
with_debugging_logs=False,
is_async=True,
raise_exceptions=False,  # Set to False to avoid ExceptionInRunner error
run_config= RunConfig,
)

error I am getting
embedding nodes:  50%
 2/4 [01:23<00:00,  3.49it/s]
---------------------------------------------------------------------------
ConnectError                              Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py](https://localhost:8080/#) in map_httpcore_exceptions()
     71     try:
---> 72         yield
     73     except Exception as exc:

54 frames
ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

ConnectError                              Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py](https://localhost:8080/#) in map_httpcore_exceptions()
     87 
     88         message = str(exc)
---> 89         raise mapped_exc(message) from exc
     90 
     91 

ConnectError: All connection attempts failed

**Additional context**
from langchain_huggingface import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(model_name="sangmini/msmarco-cotmae-MiniLM-L12_en-ko-ja")

RunConfig = {"timeout":1800, "max_retries":10, "max_wait":200, "max_workers":16}


<!-- Answer -->
To set configuration parameters for test data generation and avoid connection timeout errors when using `TestsetGenerator` with LangChain and HuggingFaceEmbeddings, you need to configure parameters like `max_workers`, `rate_limits`, and `timeouts`. Here is an example of how you can do this:

```python
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.run_config import RunConfig

# Initialize the generator with OpenAI models
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()

# Configure the generator with custom parameters to avoid timeouts
run_config = RunConfig(
    max_workers=5,  # Adjust the number of workers
    rate_limits={"requests_per_minute": 60},  # Set rate limits
    timeouts={"connect_timeout": 10, "read_timeout": 30}  # Set timeouts
)

generator = TestsetGenerator.from_langchain(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings=embeddings,
    run_config=run_config
)

# Generate the test set
testset = generator.generate_with_langchain_docs(
    documents=documents,
    test_size=10,  # Example test size
    distributions={simple: 0.5, reasoning: 0.3, multi_context: 0.2},
    with_debugging_logs=False,
    is_async=True,
    raise_exceptions=False,  # Set to False to avoid ExceptionInRunner error
    run_config=run_config
)
```

This configuration should help mitigate connection timeout errors by controlling the concurrency and rate of requests <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/testset_generation.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu what configuration should i set if i want to send 200 pages pdf file?

my current config is:
from ragas.run_config import RunConfig

# Configure the generator with custom parameters to avoid timeouts
run_config = RunConfig(
    max_wait=2000,
    max_retries=1000,
    max_workers=20,  # Adjust the number of workers
    timeout=6000,
)

<!-- Answer -->
To handle a 200-page PDF file efficiently, you should ensure that your configuration can manage the large volume of data. Here are some recommendations for your `RunConfig`:

1. **Increase `max_workers`**: This will allow more concurrent processing.
2. **Adjust `timeout` and `max_wait`**: Ensure these values are high enough to handle the processing time required for large documents.
3. **Set `max_retries`**: This should be high enough to handle potential transient errors.

Here is an example configuration:

```python
from ragas.run_config import RunConfig

# Configure the generator with custom parameters to handle large documents
run_config = RunConfig(
    max_wait=2000,
    max_retries=1000,
    max_workers=64,  # Increase the number of workers for better concurrency
    timeout=6000,
)
```

Additionally, ensure that your document handling mechanism, such as chunking, is properly configured to manage large documents. The `TestsetGenerator` uses chunking to handle large documents effectively:

```python
from langchain.text_splitter import TokenTextSplitter

splitter = TokenTextSplitter(chunk_size=1024, chunk_overlap=0)
```

This setup should help you manage the processing of a 200-page PDF file efficiently <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/run_config.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/generator.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu Give me step by step code for synthetic data generation using any open source hugging face llm model and any open source hugging face embedding model

@dosu Give me step by step code for synthetic data generation using any open source hugging face llm model and any open source hugging face embedding model

<!-- Answer -->
Here is a step-by-step code example for synthetic data generation using an open-source Hugging Face LLM model and an open-source Hugging Face embedding model:

```python
import os
from llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex
from llama_index.llms import HuggingFaceInferenceAPI
from llama_index.embeddings import HuggingFaceInferenceAPIEmbedding
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, answer_correctness

# Load documents
reader = SimpleDirectoryReader("./arxiv-papers/", num_files_limit=30)
documents = reader.load_data()

# Define the LLM and embedding models
llm_model_name = "HuggingFaceH4/zephyr-7b-alpha"  # Example LLM model
embedding_model_name = "sentence-transformers/all-MiniLM-L6-v2"  # Example embedding model

# Initialize the LLM and embedding models
llm = HuggingFaceInferenceAPI(model_name=llm_model_name, token="Your Hugging Face token")
embedding = HuggingFaceInferenceAPIEmbedding(model_name=embedding_model_name, token="Your Hugging Face token")

# Build the query engine
def build_query_engine(llm, embedding):
    vector_index = VectorStoreIndex.from_documents(
        documents, service_context=ServiceContext.from_defaults(chunk_size=512, llm=llm, embed_model=embedding)
    )
    query_engine = vector_index.as_query_engine(similarity_top_k=2)
    return query_engine

query_engine = build_query_engine(llm, embedding)

# Generate responses
def generate_responses(query_engine, test_questions, test_answers=None):
    responses = [query_engine.query(q) for q in test_questions]
    answers = [r.response for r in responses]
    contexts = [[c.node.get_content() for c in r.source_nodes] for r in responses]
    
    dataset_dict = {
        "question": test_questions,
        "answer": answers,
        "contexts": contexts,
    }
    if test_answers is not None:
        dataset_dict["ground_truth"] = test_answers
    
    ds = Dataset.from_dict(dataset_dict)
    return ds

# Example test questions and answers
test_questions = ["What is the impact of large language models on AI research?"]
test_answers = [["Large language models have significantly advanced the field of AI research."]]

# Generate the dataset
result_ds = generate_responses(query_engine, test_questions, test_answers)

# Define metrics
metrics = [faithfulness, answer_relevancy, answer_correctness]

# Evaluate the model
result = evaluate(result_ds, metrics=metrics)
print(result)
```

This code demonstrates how to generate synthetic data using Hugging Face models and evaluate it using the Ragas framework <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/applications/compare_llms.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*