Repository: explodinggradients/ragas
Issue: agenerate_prompt error when using Langchain Ollama for Synthetic Test Set Generation
Root cause: Interface Contract Mismatch

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

# Describe the bug
When attempting to generate a synthetic test set following [this doc](https://docs.ragas.io/en/stable/getstarted/testset_generation.html) and attempting to bring my own llm (`mistral` on ollama) following [this doc](https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html) I get the following error:
```
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/llms/base.py", line 177, in agenerate_text
    result = await self.langchain_llm.agenerate_prompt(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LangchainLLMWrapper' object has no attribute 'agenerate_prompt'. Did you mean: 'agenerate_text'?
```

as `ChatOllama` delegates to `BaseChatModel` it should have `agenerate_prompt`— and when debugging the ipynb cell I'm running this in, I can see the following:
<img width="473" alt="Screenshot 2024-03-15 at 11 53 17 AM" src="https://github.com/explodinggradients/ragas/assets/3477690/46e81183-d170-4a1d-becc-17f95c55c481">


Ragas version: `0.1.4`
Python version: `3.12.2` (and previously with `3.9.6`)
Langchain-Core version: `0.1.32`

# Code to Reproduce
using `mistral` specifically due to [this example](https://github.com/Ritesh-2001/Evaluate_RAG/blob/a0f671e66d6139bf5dc801e5cfb30a2b09b59128/arxiv_rag_with_ragas.ipynb#L337) (which is testing evaluate)
<details>
<summary>show code</summary>

```
# Generate Synthetic Test Set for RAGAs Evaluation
## Load documents
### Deps
from langchain.document_loaders import DirectoryLoader
### Loading
loader = DirectoryLoader(f'../{input_path}') # input_path defined in project setup
documents = loader.load()
## Update metadata
### Encapsulate
def update_metadata():
  for document in documents:
      document.metadata['filename'] = document.metadata['source']
### Run
update_metadata()
## Build Models for use in generator
### Deps
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_community.chat_models import ChatOllama
from langchain_community.embeddings import OllamaEmbeddings
from langchain.text_splitter import TokenTextSplitter
from ragas.testset.extractor import KeyphraseExtractor
from ragas.testset.docstore import InMemoryDocumentStore

### Build
ragas_llm = LangchainLLMWrapper(langchain_llm=ChatOllama(model="mistral"))
embeddings = LangchainEmbeddingsWrapper(OllamaEmbeddings(model="mistral"))
keyphrase_extractor = KeyphraseExtractor(llm=ragas_llm)
splitter = TokenTextSplitter(chunk_size=2500, chunk_overlap=100)

doc_store = InMemoryDocumentStore(
    splitter=splitter, extractor=keyphrase_extractor, embeddings=embeddings
)
## Generate Test Set
### Deps
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
### Generate
generator = TestsetGenerator.from_langchain(
    generator_llm=ragas_llm,
    critic_llm=ragas_llm,
    embeddings=embeddings,
    docstore=doc_store,
)

testset = generator.generate_with_langchain_docs(documents, test_size=10, raise_exceptions=False, with_debugging_logs=True, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
```

</details>

# Error trace
<details>
<summary>show error trace</summary>

```

Runner in Executor raised an exception
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/executor.py", line 79, in _aresults
    r = await future
        ^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
           ^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/testset/evolutions.py", line 141, in evolve
    ) = await self._aevolve(current_tries, current_nodes)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/testset/evolutions.py", line 458, in _aevolve
    simple_question, current_nodes, _ = await self.se._aevolve(
                                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/testset/evolutions.py", line 288, in _aevolve
    passed = await self.node_filter.filter(merged_node)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/testset/filters.py", line 54, in filter
    results = await self.llm.generate(prompt=prompt)
...
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/ragas/llms/base.py", line 177, in agenerate_text
    result = await self.langchain_llm.agenerate_prompt(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LangchainLLMWrapper' object has no attribute 'agenerate_prompt'. Did you mean: 'agenerate_text'?

```

</details>


# Expected behavior
I expected `generate_with_langchain_docs` to finish and create a set of test data based on the passed documents, even with a non OpenAI model


# Additional context
N/A

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Can you check with `langchain-core==0.1.19` @fongelias 

Hey @fongelias you're passing an instance of `LanghchainLLMwrapper` where instance of `BaseLanguageModel` is expected. 
To fix your code remove wrapper models from langchain with `LanghchainLLMwrapper`

```
ragas_llm = ChatOllama(model="mistral"))
embeddings = OllamaEmbeddings(model="mistral")

``

please reopen if you face any issues

@fongelias While evaluating what all metrics were you able to generate with llm = ChatOllama(model="mistral")) and
embeddings = OllamaEmbeddings(model="mistral")?
Its giving errors like - Invalid response format. Expected a list of dictionaries with keys 'verdict'
Invalid JSON response. Expected dictionary with key 'Attributed' and RuntimeWarning: Mean of empty slice
  value = np.nanmean(self.scores[cn]) which resulted in lot NaN outputs
  
  @shahules786 Is there any way to specifically prompt mistral to avoid these?

To close the loop on this issue—

## Changes Made
I followed this message below and removed some of the wrappers I thought were required for non-OpenAI models:

> Hey @fongelias you're passing an instance of `LanghchainLLMwrapper` where instance of `BaseLanguageModel` is expected. To fix your code remove wrapper models from langchain with `LanghchainLLMwrapper`
> 
> ```
> ragas_llm = ChatOllama(model="mistral"))
> embeddings = OllamaEmbeddings(model="mistral")
> 
> ``
> 
> please reopen if you face any issues
> ```

I did not update my dependency to `langchain-core==0.1.19` as instructed— for no other reason than I prioritized the other fix

## Results
`generate_with_langchain_docs` runs without error— however it is not able to gather anything for `ground_truth` (received a bunch of `NaN`s) and generally, the end output is not useful

## Discussion and Next Steps
I don't believe this issue is the fault of the framework— a lot of smaller models have issues handling constraints and formatting— I'm using local `mistral` on a M1Max Mac

I'll probably try a few other local models overnight and see if I get better results— then if nothing works I'll give OpenAI a try. For those trying to keep things free as possible before using something paid, I'll write something here if I get any good non-OpenAI results

## Final Code
<details>
<summary>show code</summary>

```python
# Generate Synthetic Test Set for RAGAs Evaluation
## Load documents
### Deps
from langchain.document_loaders import DirectoryLoader
### Loading
loader = DirectoryLoader(f'../{input_path}') # input_path defined in project setup
documents = loader.load()
## Update metadata
### Encapsulate
def update_metadata():
  for document in documents:
      document.metadata['filename'] = document.metadata['source']
### Run
update_metadata()
## Build Models for use in generator
### Deps
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_community.chat_models import ChatOllama
from langchain_community.embeddings import OllamaEmbeddings
from langchain.text_splitter import TokenTextSplitter
from ragas.testset.extractor import KeyphraseExtractor
from ragas.testset.docstore import InMemoryDocumentStore

### Build
ragas_llm = ChatOllama(model="mistral")
embeddings = OllamaEmbeddings(model="mistral")
keyphrase_extractor = KeyphraseExtractor(llm=ragas_llm)
splitter = TokenTextSplitter(chunk_size=2500, chunk_overlap=100)
## Generate Test Set
### Deps
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
### Generate
generator = TestsetGenerator.from_langchain(
    generator_llm=ragas_llm,
    critic_llm=ragas_llm,
    embeddings=embeddings,
)

testset = generator.generate_with_langchain_docs(documents, test_size=10, raise_exceptions=False, with_debugging_logs=True, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
```

</details>



> @fongelias While evaluating what all metrics were you able to generate with llm = ChatOllama(model="mistral")) and embeddings = OllamaEmbeddings(model="mistral")? Its giving errors like - Invalid response format. Expected a list of dictionaries with keys 'verdict' Invalid JSON response. Expected dictionary with key 'Attributed' and RuntimeWarning: Mean of empty slice value = np.nanmean(self.scores[cn])
> 
> @shahules786 Is there any way to specifically prompt mistral to avoid these?

I was able to run through things without 'error'— but tl;dr on the writeup above is that I ended up getting a ton of `NaNs` from mistral.

I suspect adapting the prompting for different models would require significant legwork on the RAGAs team's end, not too sure how we can alter things as end users here

I was facing this for some time. After seeing the comment about langchain-core being a potential issue I reinstalled all my libraries with the below and it seemed to resolve (where it did not before). Assuming an update was made to something.

`pip install langchain-core langchain_community langchain langchain_experimental ragas --force-reinstall`

Hi @IAmStoxe 
I'm facing the same error, I re-installed  all the libraries but it didn't work 
Could you share more about you solution?
Thanks

Hi @IAmStoxe , I used your command to reinstall those packages but I got a new error:
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Could you please share your requirements.txt file? Thanks.

Hello @Josh-Chen-CanaDREAM, does `ChatOllama(model='mistral')` and `OllamaEmbeddings(model='mistral')` work for you rn. I needed a fix for it or a way around for open source llm and llm embeddings.