Repository: explodinggradients/ragas
Issue: Testset Generation Failure: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Root cause: External Dependency Breakage

**Description:**

I am encountering multiple warnings and errors while attempting to generate a test set using the `ragas` library with LangChain, Pydantic v2, and OpenAI. The main issues stem from deprecated imports and namespace conflicts. Despite following potential fixes and utilizing `nest_asyncio`, the issue persists.

**Dependencies:**

Here are the relevant dependencies from my `pyproject.toml`:

```toml
[tool.poetry.dependencies]
python = ">=3.12,<3.13"
ragas = "^0.1.18"
langchain = "^0.3.0"
langchain-community = "^0.3.0"
langchain-openai = "^0.2.0"
bs4 = "^0.0.2"
beautifulsoup4 = "^4.12.3"
unstructured = "^0.15.12"
libmagic = "^1.0"
python-magic-bin = "^0.4.14"
nest-asyncio = "^1.6.0"
```

**Steps to Reproduce:**

1. Set up the environment using Python 3.12 and the dependencies listed above.
2. Use the following code to load documents from a webpage and generate a test set:

```python
import os
from langchain_community.document_loaders import WebBaseLoader
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

import nest_asyncio
nest_asyncio.apply()

os.environ["OPENAI_API_KEY"] = "openai-api-key"

loader = WebBaseLoader("https://www.nba.com/stats/players")
documents = loader.load()

generator_llm = ChatOpenAI(model="gpt-4o-mini-2024-07-18")
critic_llm = ChatOpenAI(model="gpt-4o-mini-2024-07-18")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

try:
    test_set = generator.generate_with_langchain_docs(documents, test_size=3, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)
    data_frame = test_set.to_pandas()
    data_frame.to_csv('testset.csv', index=False)
except Exception as e:
    print(f"Error during test set generation: {e}")
```

**Error Output:**

```
USER_AGENT environment variable not set, consider setting it to identify your requests.
/.../pydantic/_fields.py:132: UserWarning: Field "model_name" in _VertexAIBase has conflict with protected namespace "model_".
/.../pydantic/_fields.py:132: UserWarning: Field "model_name" in _VertexAICommon has conflict with protected namespace "model_".
LangChainDeprecationWarning: As of langchain-core 0.3.0, LangChain uses pydantic v2 internally...
Error during test set generation: The runner thread which was running the jobs raised an exception. Read the traceback above to debug it.
```

**What I Have Tried:**

- Added `nest_asyncio.apply()` to manage async conflicts.
- Checked compatibility between LangChain and Pydantic, replacing deprecated imports where applicable.
- Attempted to use `raise_exceptions=False` as suggested and `is_async=False`, but the problem persists.

**Environment:**

- Python Version: 3.12
- Pydantic Version: 2.x
- LangChain Version: 0.3.0
- Ragas Version: 0.1.18

**Expected Behavior:**

The test set should be generated without warnings or namespace conflicts, and it should create the `testset.csv` file successfully.

**Actual Behavior:**

The process halts with a deprecation warning and a namespace conflict related to `pydantic` and `model_`. It raises an exception during test set generation.

Any guidance on how to resolve these namespace conflicts and LangChain compatibility issues would be greatly appreciated.

<!-- Greeting -->
Hey @Zolastic! I'm here to help you tackle any bugs, answer questions, and even become a contributor. Let's work together to resolve the issue you're facing.

<!-- Answer -->
I found a similar closed issue that might be relevant to your problem: [running quickstart.ipynb fails](https://github.com/explodinggradients/ragas/issues/144). This issue was closed by a pull request: [fix: llamaIndex issues](https://github.com/explodinggradients/ragas/pull/156) on September 26, 2023 <sup>[[1]](https://github.com/explodinggradients/ragas/issues/144)</sup>.

To resolve the namespace conflicts and LangChain compatibility issues when generating a test set using the `ragas` library with LangChain, Pydantic v2, and OpenAI, you need to ensure that the correct imports and class references are used. Specifically, you should use the appropriate wrappers and ensure that the correct classes from LangChain and OpenAI are referenced.

Here is the relevant code snippet that shows how to correctly set up the `TestsetGenerator` class to handle LangChain and OpenAI models:

```python
from langchain_openai.chat_models import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings.base import LangchainEmbeddingsWrapper
from ragas.testset.docstore import InMemoryDocumentStore
from ragas.testset.extractor import KeyphraseExtractor
from ragas.llms.prompt import Prompt

@dataclass
class TestsetGenerator:
    generator_llm: BaseRagasLLM
    critic_llm: BaseRagasLLM
    embeddings: BaseRagasEmbeddings
    docstore: DocumentStore

    @classmethod
    def from_langchain(
        cls,
        generator_llm: LangchainLLM,
        critic_llm: LangchainLLM,
        embeddings: LangchainEmbeddings,
        docstore: t.Optional[DocumentStore] = None,
        run_config: RunConfig = RunConfig(),
        chunk_size: int = 1024,
    ) -> "TestsetGenerator":
        generator_llm_model = LangchainLLMWrapper(generator_llm)
        critic_llm_model = LangchainLLMWrapper(critic_llm)
        embeddings_model = LangchainEmbeddingsWrapper(embeddings)

        keyphrase_extractor = KeyphraseExtractor(llm=generator_llm_model)
        if docstore is None:
            from langchain.text_splitter import TokenTextSplitter

            splitter = TokenTextSplitter(chunk_size=chunk_size, chunk_overlap=0)
            docstore = InMemoryDocumentStore(
                splitter=splitter,
                embeddings=embeddings_model,
                extractor=keyphrase_extractor,
                run_config=run_config,
            )
        return cls(
            generator_llm=generator_llm_model,
            critic_llm=critic_llm_model,
            embeddings=embeddings_model,
            docstore=docstore,
        )

    @classmethod
    @deprecated("0.1.4", removal="0.2.0", alternative="from_langchain")
    def with_openai(
        cls,
        generator_llm: str = "gpt-4o-mini",
        critic_llm: str = "gpt-4o",
        embeddings: str = "text-embedding-ada-002",
        docstore: t.Optional[DocumentStore] = None,
        chunk_size: int = 1024,
    ) -> "TestsetGenerator":
        generator_llm_model = ChatOpenAI(model=generator_llm)
        critic_llm_model = ChatOpenAI(model=critic_llm)
        embeddings_model = OpenAIEmbeddings(model=embeddings)

        return cls.from_langchain(
            generator_llm=generator_llm_model,
            critic_llm=critic_llm_model,
            embeddings=embeddings_model,
            docstore=docstore,
            chunk_size=chunk_size,
        )
```

Additionally, you can leverage the `Prompt` class from the `ragas.llms.prompt` module to create various prompts for generating and refining questions, extracting keyphrases, and more. This setup ensures that the `TestsetGenerator` class can handle both LangChain and OpenAI models by using the appropriate wrappers and class references <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/generator.py)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/prompts.py)</sup>.

By following these steps, you should be able to resolve the namespace conflicts and compatibility issues you're encountering.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Unfortunately, this does not fix the error that appears when running 
``` python 
generator.generate_with_langchain_docs(documents, test_size=3, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)
```

error:
```
The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass raise_exceptions=False incase you want to show only a warning message instead.
```

I am learning Ragas and am referring to the [documentation](https://docs.ragas.io/en/latest/getstarted/testset_generation.html) and this [GitHub file](https://github.com/mosh98/RAG_With_Models/blob/main/evaluation/RAGAS%20Testset%20GEN%20DEMO.ipynb)

**Full Error Traceback:**

I'm encountering the following error when trying to generate a test set using `generate_with_langchain_docs`. It appears related to the `ragas` docstore, and despite using `raise_exceptions=False`, the issue persists.

**Error Traceback:**

```bash
Traceback (most recent call last): 
  File "/Users/username/Documents/projects/learning/ragas-basics/testset-generation-2.py", line 32, in <module>
    testset = generator.generate_with_langchain_docs(documents, test_size=2, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/Documents/projects/learning/ragas-basics/.venv/lib/python3.12/site-packages/ragas/testset/generator.py", line 206, in generate_with_langchain_docs
    self.docstore.add_documents(
  File "/Users/username/Documents/projects/learning/ragas-basics/.venv/lib/python3.12/site-packages/ragas/testset/docstore.py", line 214, in add_documents
    self.add_nodes(nodes, show_progress=show_progress)
  File "/Users/username/Documents/projects/learning/ragas-basics/.venv/lib/python3.12/site-packages/ragas/testset/docstore.py", line 253, in add_nodes
    raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` in case you want to show only a warning message instead.
```

Here is the code I’m using:

```python
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

from langchain_community.document_loaders import WebBaseLoader

import nest_asyncio
nest_asyncio.apply()

loader = WebBaseLoader("https://www.nba.com/stats/players")
documents = loader.load()

# generator with openai models
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()


generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=2, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}, raise_exceptions=False)

df = testset.to_pandas()
print(df)   
```

I’ve tried the following:
- Setting `raise_exceptions=False`
- Using `nest_asyncio` and `is_async=False`

Still facing the issue. Would appreciate any help to debug this!

@dosu: If you can provide any insights or assistance here, that would be great! Others are welcome to chime in as well.

<!-- Answer -->
To resolve the `ExceptionInRunner` error when generating a test set using `generate_with_langchain_docs` in the `ragas` library, despite setting `raise_exceptions=False` and using `nest_asyncio`, you need to ensure that the `test_data_rows` are properly handled even when exceptions are raised. The relevant code snippet shows that `ExceptionInRunner` is raised if `test_data_rows` is empty:

```python
try:
    test_data_rows = exec.results()
    if not test_data_rows:
        raise ExceptionInRunner()
except ValueError as e:
    raise e
```

Since `raise_exceptions` is set to `False`, the `Executor` should not raise exceptions, but it seems that the `ExceptionInRunner` is still being raised if `test_data_rows` is empty. To handle this, you can modify the code to check for `raise_exceptions` before raising `ExceptionInRunner`:

```python
try:
    test_data_rows = exec.results()
    if not test_data_rows and raise_exceptions:
        raise ExceptionInRunner()
except ValueError as e:
    raise e
```

This way, `ExceptionInRunner` will only be raised if `raise_exceptions` is `True`.

Here is the modified code snippet:

```python
try:
    test_data_rows = exec.results()
    if not test_data_rows and raise_exceptions:
        raise ExceptionInRunner()
except ValueError as e:
    raise e
```

This change ensures that the `ExceptionInRunner` is not raised when `raise_exceptions` is set to `False`. Additionally, the `ExceptionInRunner` class in the `ragas` library is designed to provide a warning message when `raise_exceptions=False`, which aligns with this approach:

```python
class ExceptionInRunner(RagasException):
    """
    Exception raised when an exception is raised in the executor.
    """

    def __init__(self):
        msg = "The runner thread which was running the jobs raised an exception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` in case you want to show only a warning message instead."
        super().__init__(msg)
```

By making this adjustment, you should be able to avoid the `ExceptionInRunner` error when `raise_exceptions` is set to `False` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/generator.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/exceptions.py)</sup>.

This is a tough nut to crack, could really use your help on this, @jjmachan.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


**Issue Update:**

I was able to solve the issue by editing the `docstore.py` file under the `testset` folder. However, I'm not entirely sure if my solution is the best or if there’s a more appropriate way to resolve it. Here’s a summary of the changes I made:

### 1. Changes to `Node` Class:

#### Before (Original Code):
```python
class Node(Document):
    keyphrases: t.List[str] = Field(default_factory=list, repr=False)
    relationships: t.Dict[Direction, t.Any] = Field(default_factory=dict, repr=False)
    doc_similarity: t.Optional[float] = Field(default=None, repr=False)
    wins: int = 0
```

#### After (My Changes):
```python
class Node(Document):
    keyphrases: t.List[str] = Field(default_factory=list, repr=False)
    relationships: t.Dict[Direction, t.Any] = {}  # Changed Field to a standard dict
    doc_similarity: t.Optional[float] = Field(default=None, repr=False)
    wins: int = 0
```

### 2. Changes to `add_documents` Method:

#### Before (Original Code):
```python
def add_documents(self, docs: t.Sequence[Document], show_progress=True):
    # split documents with self.splitter into smaller nodes
    nodes = [
        Node.from_langchain_document(d)
        for d in self.splitter.transform_documents(docs)
    ]
    self.add_nodes(nodes, show_progress=show_progress)
```

#### After (My Changes):
```python
def add_documents(self, docs: t.Sequence[Document], show_progress=True):
    """
    Add documents in batch mode.
    """
    assert self.embeddings is not None, "Embeddings must be set"

    # split documents with self.splitter into smaller nodes
    docs = self.splitter.transform_documents(docs)

    # Create the nodes and set the filename for each one
    nodes = []
    for i, d in enumerate(docs):
        node = Node.from_langchain_document(d)
        node.metadata['filename'] = f"document_{i}.txt"  # Set custom filename
        node.embedding = None  # Clear embedding for each node
        node.keyphrases = []   # Clear keyphrases for each node
        nodes.append(node)

    print("nodes in add_documents: ", nodes[0].embedding)
    self.add_nodes(nodes, show_progress=show_progress)
```

### Why I Made These Changes:
1. **`Node` Class Change:** 
   I was encountering an issue where modifying the `relationships` field led to errors because `Field` (from Pydantic) does not support item assignment. By changing `relationships` to a standard dictionary (`{}`), I was able to avoid this error.

2. **`add_documents` Method Change:**
   - I explicitly cleared the `embedding` and `keyphrases` for each node to ensure they start with fresh values before processing.

### Question:
Are these changes the correct approach for fixing the issue? Is there a better or more standard way to address these issues within the library’s architecture?

@dosu, do you have any insights or suggestions on how to approach this? Any feedback or guidance from others would be greatly appreciated!

<!-- Answer -->
Your changes address the issues you encountered, but let's review them to ensure they align with the library's architecture and best practices.

### 1. Changes to `Node` Class

#### Original Code:
```python
class Node(Document):
    keyphrases: t.List[str] = Field(default_factory=list, repr=False)
    relationships: t.Dict[Direction, t.Any] = Field(default_factory=dict, repr=False)
    doc_similarity: t.Optional[float] = Field(default=None, repr=False)
    wins: int = 0
```

#### Your Changes:
```python
class Node(Document):
    keyphrases: t.List[str] = Field(default_factory=list, repr=False)
    relationships: t.Dict[Direction, t.Any] = {}  # Changed Field to a standard dict
    doc_similarity: t.Optional[float] = Field(default=None, repr=False)
    wins: int = 0
```

### Analysis:
- **Issue with `Field`**: The original use of `Field` with `default_factory=dict` is correct for Pydantic models. The error you encountered might be due to how the `relationships` field is being modified elsewhere in the code.
- **Standard Dictionary**: Changing `relationships` to a standard dictionary (`{}`) avoids the Pydantic-specific behavior but loses some benefits of Pydantic's validation and serialization.

### Recommendation:
Instead of changing `relationships` to a standard dictionary, ensure that any modifications to this field are done in a way that Pydantic supports. For example, if you need to update the dictionary, use methods that Pydantic models support.

### 2. Changes to `add_documents` Method

#### Original Code:
```python
def add_documents(self, docs: t.Sequence[Document], show_progress=True):
    # split documents with self.splitter into smaller nodes
    nodes = [
        Node.from_langchain_document(d)
        for d in self.splitter.transform_documents(docs)
    ]
    self.add_nodes(nodes, show_progress=show_progress)
```

#### Your Changes:
```python
def add_documents(self, docs: t.Sequence[Document], show_progress=True):
    """
    Add documents in batch mode.
    """
    assert self.embeddings is not None, "Embeddings must be set"

    # split documents with self.splitter into smaller nodes
    docs = self.splitter.transform_documents(docs)

    # Create the nodes and set the filename for each one
    nodes = []
    for i, d in enumerate(docs):
        node = Node.from_langchain_document(d)
        node.metadata['filename'] = f"document_{i}.txt"  # Set custom filename
        node.embedding = None  # Clear embedding for each node
        node.keyphrases = []   # Clear keyphrases for each node
        nodes.append(node)

    print("nodes in add_documents: ", nodes[0].embedding)
    self.add_nodes(nodes, show_progress=show_progress)
```

### Analysis:
- **Clearing Embeddings and Keyphrases**: Explicitly clearing `embedding` and `keyphrases` ensures that each node starts with fresh values, which can help avoid issues with stale data.
- **Setting Custom Filenames**: Adding a custom filename to each node's metadata is a good practice for tracking and debugging.

### Recommendation:
Your changes to the `add_documents` method are reasonable. However, ensure that clearing `embedding` and `keyphrases` aligns with the intended use of the library. If these fields are expected to be set elsewhere, clearing them might not be necessary.

### Final Thoughts:
- **`Node` Class**: Consider keeping the `Field` for `relationships` and ensure modifications are Pydantic-compatible.
- **`add_documents` Method**: Your changes are fine, but double-check if clearing `embedding` and `keyphrases` is necessary for your use case.

If you encounter further issues, it might be helpful to review how `relationships` are being modified and ensure that any updates to Pydantic fields are done correctly.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I've encountered the same error. It seems that the default field values of ragas Node and Document, which inherit from langchain_core.pydantic_v1 BaseModel, might be causing the issue. In langchain v0.3, pydantic v0.2 is used internally, and there might be a problem with combining this with langchain_core.pydantic_v1.

Thank you for the suggestion! 🙏

I changed my `langchain` version to `0.2.11` and `langchain-openai` to `0.1.8`, and it worked perfectly. It seems the issue was indeed related to the version mismatch with `pydantic`. I’ll proceed with these versions for now.

Thanks again for the help!