Repository: explodinggradients/ragas
Issue: The runner thread which was running the jobs raised an exeception.
Root cause: External Dependency Breakage

---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
Cell In[56], [line 22](vscode-notebook-cell:?execution_count=56&line=22)
     [11](vscode-notebook-cell:?execution_count=56&line=11) embeddings = client.embeddings.create(
     [12](vscode-notebook-cell:?execution_count=56&line=12)     input=inputs,
     [13](vscode-notebook-cell:?execution_count=56&line=13)     model="RAG_text-embedding-3-large" 
     [14](vscode-notebook-cell:?execution_count=56&line=14) )
     [16](vscode-notebook-cell:?execution_count=56&line=16) generator = TestsetGenerator.from_langchain(
     [17](vscode-notebook-cell:?execution_count=56&line=17)     generator_llm,
     [18](vscode-notebook-cell:?execution_count=56&line=18)     critic_llm,
     [19](vscode-notebook-cell:?execution_count=56&line=19)     embeddings
     [20](vscode-notebook-cell:?execution_count=56&line=20) )
---> [22](vscode-notebook-cell:?execution_count=56&line=22) testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

File c:\Users\10742161\AppData\Local\anaconda3\envs\rag\lib\site-packages\ragas\testset\generator.py:206, in TestsetGenerator.generate_with_langchain_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    [204](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:204) distributions = distributions or {}
    [205](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:205) # chunk documents and add to docstore
--> [206](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:206) self.docstore.add_documents(
    [207](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:207)     [Document.from_langchain_document(doc) for doc in documents]
    [208](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:208) )
    [210](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:210) return self.generate(
    [211](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:211)     test_size=test_size,
    [212](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:212)     distributions=distributions,
   (...)
    [216](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:216)     run_config=run_config,
    [217](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/generator.py:217) )

File c:\Users\10742161\AppData\Local\anaconda3\envs\rag\lib\site-packages\ragas\testset\docstore.py:214, in InMemoryDocumentStore.add_documents(self, docs, show_progress)
    [209](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:209) # split documents with self.splitter into smaller nodes
    [210](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:210) nodes = [
    [211](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:211)     Node.from_langchain_document(d)
    [212](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:212)     for d in self.splitter.transform_documents(docs)
    [213](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:213) ]
--> [214](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:214) self.add_nodes(nodes, show_progress=show_progress)

File c:\Users\10742161\AppData\Local\anaconda3\envs\rag\lib\site-packages\ragas\testset\docstore.py:253, in InMemoryDocumentStore.add_nodes(self, nodes, show_progress)
    [251](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:251) results = executor.results()
    [252](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:252) if not results:
--> [253](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:253)     raise ExceptionInRunner()
    [255](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:255) for i, n in enumerate(nodes):
    [256](file:///C:/Users/10742161/AppData/Local/anaconda3/envs/rag/lib/site-packages/ragas/testset/docstore.py:256)     if i in nodes_to_embed.keys():

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Collecting ragas==0.1.9

**Describe the bug**
While generating synthetic dataset : ExceptionInRunner error appeared as below:
 ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Ragas version: 0.1.9
Python version: 3.10

**Code to Reproduce**
---------------------------------------------------
from langchain_community.document_loaders import DirectoryLoader
loader = DirectoryLoader(r"Data\Rag_data")
documents = loader.load()
----------------------------------------------------
for document in documents:
    document.metadata['filename'] = document.metadata['source']

--------------------------------------------------------
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain.llms import AzureOpenAI
generator_llm = AzureOpenAI(api_key= open_api_key,
                     api_version= azure_api_version,
                     azure_endpoint=azure_endpoint)
critic_llm = AzureOpenAI(api_key= open_api_key,
                     api_version= azure_api_version,
                     azure_endpoint=azure_endpoint)

embeddings = client.embeddings.create(
    input=inputs,
    model="RAG_text-embedding-3-large" 
)

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

**Error trace**

**Expected behavior**
Expected behavior to be generation of Synthetic/Ground Truth in Json format as provided with "filename" 


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hey @tushitdave! I'm here to help you out with any bugs or questions you have while waiting for a human maintainer.

<!-- Answer -->
A similar issue was reported in the Ragas repository, where the `ExceptionInRunner` was raised. The suggested solution for this issue involves ensuring that the `OPENAI_API_KEY` environment variable is set correctly with a valid OpenAI API key. You can set the API key using the following code snippet:

```python
import os
from getpass import getpass
import openai

# Set your OpenAI API key if it is not already set as an environment variable
if not (openai_api_key := os.getenv("OPENAI_API_KEY")):
    openai_api_key = getpass("🔑 Enter your OpenAI API key: ")
openai.api_key = openai_api_key
os.environ["OPENAI_API_KEY"] = openai_api_key
```

Additionally, you can pass `raise_exceptions=False` in the `evaluate` function to show only a warning message instead of raising an exception:

```python
from datasets import Dataset 
from ragas import evaluate
from ragas.metrics import context_recall

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)

# Ensure the API key is correct
os.environ["OPENAI_API_KEY"] = "your-valid-openai-key"

score = evaluate(dataset, metrics=[context_recall], raise_exceptions=False)
score.to_pandas()
```

This approach will prevent the `ExceptionInRunner` from stopping your execution and will only show a warning message <sup>[[1]](https://github.com/explodinggradients/ragas/issues/1055)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu

I did the same as you suggested, but error remain same. Please note I am generating synthetic dataset and using AzureOpenAI instead of OpenAI.

from langchain_community.document_loaders import DirectoryLoader
loader = DirectoryLoader(r"Data\Rag_data")
documents = loader.load()

for document in documents:
    document.metadata['filename'] = document.metadata['source']

from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain.llms import AzureOpenAI

from getpass import getpass
import openai

if not (openai_api_key := os.getenv("OPENAI_API_KEY")):
    openai_api_key = getpass("Used my azureopenAI key here")
openai.api_key = openai_api_key
os.environ["OPENAI_API_KEY"] = openai_api_key

generator_llm = AzureOpenAI(api_key= open_api_key,
                     api_version= azure_api_version,
                     azure_endpoint=azure_endpoint)

critic_llm = AzureOpenAI(api_key= open_api_key,
                     api_version= azure_api_version,
                     azure_endpoint=azure_endpoint)

embeddings = client.embeddings.create(
    input=inputs,
    model="RAG_text-embedding-3-large" 
)

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},raise_exceptions=False)

In above mentioned line, I am using langchain for synthetic data generation as mentioned in below link:

https://docs.ragas.io/en/v0.1.12/getstarted/testset_generation.html#get-started-testset-generation

Only change, I am doing here is- I am using AzureOpenAI client.


<!-- Answer -->
When using AzureOpenAI with the Ragas library, ensure that you have configured the Azure OpenAI settings correctly. Here are the steps and configurations you need to verify:

1. **Installation**: Make sure you have installed the `langchain_openai` package:
   ```bash
   pip install langchain_openai
   ```

2. **Configuration**: Ensure that you have set the Azure OpenAI parameters correctly:
   - `base_url`: The endpoint URL for your Azure OpenAI service.
   - `model_deployment`: The deployment name for your model.
   - `model_name`: The name of your model.
   - `embedding_deployment`: The deployment name for your embeddings.
   - `embedding_name`: The name of your embedding model, typically "text-embedding-ada-002".

3. **Code Setup**: Initialize the Azure OpenAI models and embeddings using the provided configuration and wrap them with `LangchainLLMWrapper` and `LangchainEmbeddingsWrapper`:
   ```python
   from langchain_openai.chat_models import AzureChatOpenAI
   from langchain_openai.embeddings import AzureOpenAIEmbeddings
   from ragas.llms import LangchainLLMWrapper
   from ragas.embeddings import LangchainEmbeddingsWrapper

   azure_configs = {
       "base_url": "https://<your-endpoint>.openai.azure.com/",
       "model_deployment": "your-deployment-name",
       "model_name": "your-model-name",
       "embedding_deployment": "your-deployment-name",
       "embedding_name": "text-embedding-ada-002",
   }

   azure_llm = AzureChatOpenAI(
       openai_api_version="2023-05-15",
       azure_endpoint=azure_configs["base_url"],
       azure_deployment=azure_configs["model_deployment"],
       model=azure_configs["model_name"],
       validate_base_url=False,
   )

   azure_embeddings = AzureOpenAIEmbeddings(
       openai_api_version="2023-05-15",
       azure_endpoint=azure_configs["base_url"],
       azure_deployment=azure_configs["embedding_deployment"],
       model=azure_configs["embedding_name"],
   )

   azure_llm = LangchainLLMWrapper(azure_llm)
   azure_embeddings = LangchainEmbeddingsWrapper(azure_embeddings)
   ```

Ensure that all these configurations are correctly set up in your code. If the error persists, double-check the Azure OpenAI service configurations and ensure that the API keys and endpoints are correctly configured <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customizations/customize_models.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@Dosu
As You suggested, I did applied, however, errors are as follows:
ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Code:
`from langchain_openai.chat_models import AzureChatOpenAI
from langchain_openai.embeddings import AzureOpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context`

`embedding_deployment = "RAG_text-embedding-3-large"
azure_llm = AzureChatOpenAI(
    openai_api_version=azure_api_version,
    azure_endpoint=azure_endpoint,
    azure_deployment=deployment,
    model='gpt-4o-mini',
    validate_base_url=False,
)`

`critic_azure_llm = AzureChatOpenAI(
    openai_api_version=azure_api_version,
    azure_endpoint=azure_endpoint,
    azure_deployment=deployment,
    model='gpt-4o-mini',
    validate_base_url=False,
)`

`azure_embeddings = AzureOpenAIEmbeddings(
    openai_api_version=embedding_api_version,
    azure_endpoint=embedding_endpoint,
    azure_deployment=embedding_deployment,
    model="text-embedding-3-large",
)`

`azure_llm = LangchainLLMWrapper(azure_llm)
critic_azure_llm = LangchainLLMWrapper(critic_azure_llm)
azure_embeddings = LangchainEmbeddingsWrapper(azure_embeddings)`

`from langchain_community.document_loaders import DirectoryLoader
loader = DirectoryLoader(r"Data\Rag_data")
documents = loader.load()`

`for document in documents:
    document.metadata['filename'] = document.metadata['source']`

`generator = TestsetGenerator.from_langchain(
    azure_llm,
    critic_azure_llm,
    azure_embeddings
)

testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},
                                                 raise_exceptions=False)`

`
I am not able to generate synthetic dataset.

<!-- Answer -->
The `ExceptionInRunner` error you're encountering indicates that an exception occurred in the executor thread. Here are a few steps to help troubleshoot and resolve the issue:

1. **Check Azure Configuration**: Ensure that all Azure configurations are correct, including `azure_endpoint`, `azure_deployment`, and `model` names. Double-check that these match your Azure OpenAI service settings.

2. **Validate API Versions**: Make sure the `openai_api_version` and `embedding_api_version` are set correctly. They should match the versions supported by your Azure OpenAI deployment.

3. **Model and Embedding Names**: Verify that the model names (`gpt-4o-mini` and `text-embedding-3-large`) are correctly specified and available in your Azure deployment.

4. **Debugging Logs**: Since you have set `raise_exceptions=False`, the error might not stop execution, but it will still log warnings. Check the logs for any additional error messages or warnings that might provide more context.

5. **Environment Variables**: Ensure that your environment variables, especially `OPENAI_API_KEY`, are set correctly and accessible in your script.

6. **Network and Permissions**: Check if there are any network issues or permission settings that might be preventing access to the Azure OpenAI service.

If these steps do not resolve the issue, consider enabling debugging logs by setting `with_debugging_logs=True` in the `generate_with_langchain_docs` method to get more detailed output, which might help identify the root cause of the problem.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu : All above mentioned details/step from 1 to 6 are complied with the requirement. Nothing wrong 

@tushitdave
Same issue here.
I've run into the same issue multiple times with Ragas. It seems to be a common problem—I've seen several open tickets on this and no easy solution. I believe the root cause might be incompatible dependencies, 

The main challenge is the lack of clear documentation or locked dependency versions. Without locked versions, it's tough to resolve issues with Langchain/Ragas dependencies. Unfortunately, neither the docs nor the repo provide the correct versions for reference. 

For example, the pyproject.toml file here [link](https://github.com/explodinggradients/ragas/blob/main/pyproject.toml) doesn't specify the versions for most dependencies.

it's impossible to generate synthetic datasets. You need some luck to build the correct depenencies. which I haven't had lately. 

<!-- Answer -->
This is a tough nut to crack, could really use your help on this, @jjmachan.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


This issue is related to RAGAS v0.1.9, which is now significantly outdated. The testset generation API has been completely redesigned in version 0.3.8+.

Please upgrade and follow docs for latest usage.