Repository: explodinggradients/ragas
Issue: Testset Generation - TypeError: generate() missing 1 required positional argument: 'prompts'
Root cause: Interface Contract Mismatch

**Describe the bug**
When trying to generate a synthetic dataset using HF open-source downloaded models, I get `TypeError: generate() missing 1 required positional argument: 'prompts'`

Ragas version: ragas-0.1.0rc2.dev23+g1bbb8d5 (installed from source today)
Python version: 3.8.18

**Code to Reproduce**
```py
from ragas.testset import TestsetGenerator
from ragas.embeddings import HuggingfaceEmbeddings
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
from langchain.llms import HuggingFacePipeline
from langchain.text_splitter import RecursiveCharacterTextSplitter
from ragas.testset.docstore import InMemoryDocumentStore
from ragas.testset.extractor import keyphraseExtractor
from ragas.run_config import RunConfig
from ragas.testset.evolutions import simple, reasoning, multi_context, conditional

# load documents here [...]

# Initialize HuggingFaceEmbeddings
e5_embeddings = HuggingfaceEmbeddings(model_name='intfloat/multilingual-e5-large')
e5_embeddings.set_run_config(run_config= RunConfig)

# critic model, should be more capable than generator?
model_name = 'mistralai/Mixtral-8x7B-Instruct-v0.1' 
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
transformers_pipeline = pipeline('text-generation', model=model, tokenizer=tokenizer)
critic_llm = HuggingFacePipeline(pipeline=transformers_pipeline)

# generator model, should be less capable than critic_llm?
model_name = 'mistralai/Mistral-7B-Instruct-v0.2' 
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
transformers_pipeline = pipeline('text-generation', model=model, tokenizer=tokenizer)
generator_llm = HuggingFacePipeline(pipeline=transformers_pipeline)

# Create an instance of TextSplitter
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=512, chunk_overlap=100)
# Create an instance of a keyphrase extractor
keyphrase_extractor = keyphraseExtractor(llm=critic_llm)
# Create an instance of InMemoryDocumentStore
docstore = InMemoryDocumentStore(splitter=text_splitter, embeddings=e5_embeddings, extractor=keyphrase_extractor)
# generator with e5 embeddings and local models
generator = TestsetGenerator(embeddings=e5_embeddings,
                             critic_llm=critic_llm,
                             generator_llm=generator_llm,
                             docstore=docstore
                             )

# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=20, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.10, conditional: 0.15})
test_df = testset.to_pandas()
test_df.head()
```

**Error trace**
```
Traceback (most recent call last):
  File "/.pyenv/versions/3.8.18/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/code/rag-evaluation/.venv/ragas/src/ragas/executor.py", line 66, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/.pyenv/versions/3.8.18/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/code/rag-evaluation/.venv/ragas/src/ragas/executor.py", line 54, in _aresults
    raise e
  File "/code/rag-evaluation/.venv/ragas/src/ragas/executor.py", line 49, in _aresults
    r = await future
  File "/.pyenv/versions/3.8.18/lib/python3.8/asyncio/tasks.py", line 619, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/code/rag-evaluation/.venv/ragas/src/ragas/executor.py", line 82, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/code/rag-evaluation/.venv/ragas/src/ragas/testset/extractor.py", line 49, in extract
    results = await self.llm.generate(prompt=prompt, is_async=is_async)
TypeError: generate() missing 1 required positional argument: 'prompts'
Traceback (most recent call last):
  File "/code/rag-evaluation/ragas-testsetgeneration.py", line 70, in <module>
    testset = generator.generate_with_langchain_docs(documents, test_size=20, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.10, conditional: 0.15})
  File "/code/rag-evaluation/.venv/ragas/src/ragas/testset/generator.py", line 149, in generate_with_langchain_docs
    self.docstore.add_documents(
  File "/code/rag-evaluation/.venv/ragas/src/ragas/testset/docstore.py", line 209, in add_documents
    self.add_nodes(nodes, show_progress=show_progress)
  File "/code/rag-evaluation/.venv/ragas/src/ragas/testset/docstore.py", line 250, in add_nodes
    n.embedding = results[nodes_to_embed[i]]
IndexError: list index out of range
```

**Additional context**
If I try to fix the issue by changing the problematic line to `results = await self.llm.generate(prompts=[prompt], is_async=is_async)`, I get a different error:
```
Traceback (most recent call last):
  File "/.pyenv/versions/3.8.18/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/code/rag-evaluation/.venv/ragas/src/ragas/executor.py", line 66, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/.pyenv/versions/3.8.18/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/code/rag-evaluation/.venv/ragas/src/ragas/executor.py", line 54, in _aresults
    raise e
  File "/code/rag-evaluation/.venv/ragas/src/ragas/executor.py", line 49, in _aresults
    r = await future
  File "/.pyenv/versions/3.8.18/lib/python3.8/asyncio/tasks.py", line 619, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/code/rag-evaluation/.venv/ragas/src/ragas/executor.py", line 82, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/code/rag-evaluation/.venv/ragas/src/ragas/testset/extractor.py", line 49, in extract
    results = await self.llm.generate(prompts=[prompt], is_async=is_async)
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/langchain_core/language_models/llms.py", line 703, in generate
    output = self._generate_helper(
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/langchain_core/language_models/llms.py", line 567, in _generate_helper
    raise e
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/langchain_core/language_models/llms.py", line 554, in _generate_helper
    self._generate(
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/langchain_community/llms/huggingface_pipeline.py", line 203, in _generate
    responses = self.pipeline(batch_prompts)
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/transformers/pipelines/text_generation.py", line 219, in __call__
    return super().__call__(text_inputs, **kwargs)
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1143, in __call__
    outputs = list(final_iterator)
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
    data = self._next_data()
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/transformers/pipelines/pt_utils.py", line 19, in __getitem__
    processed = self.process(item, **self.params)
  File "/code/rag-evaluation/.venv/lib/python3.8/site-packages/transformers/pipelines/text_generation.py", line 233, in preprocess
    prefix + prompt_text,
TypeError: can only concatenate str (not "PromptValue") to str
Traceback (most recent call last):
  File "/code/rag-evaluation/ragas-testsetgeneration.py", line 70, in <module>
    testset = generator.generate_with_langchain_docs(documents, test_size=20, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.10, conditional: 0.15})
  File "/code/rag-evaluation/.venv/ragas/src/ragas/testset/generator.py", line 149, in generate_with_langchain_docs
    self.docstore.add_documents(
  File "/code/rag-evaluation/.venv/ragas/src/ragas/testset/docstore.py", line 209, in add_documents
    self.add_nodes(nodes, show_progress=show_progress)
  File "/code/rag-evaluation/.venv/ragas/src/ragas/testset/docstore.py", line 250, in add_nodes
    n.embedding = results[nodes_to_embed[i]]
IndexError: list index out of range
```

Any ideas? 
Thanks!!

hey @gonznm thanks for the clear writeup :)

could you try wrapping the generator_llm with the `LangchainLLMWrapper` as so

```py
from ragas.llms.base import LangchainLLMWrapper
generator_llm_wrapped = LangchainLLMWrapper(generator_llm)
critic_llm_wrapped = LangchainLLMWrapper(critic_llm)


# and now you use pass it as such
generator = TestsetGenerator(embeddings=e5_embeddings,
         critic_llm=critic_llm_wrapped,
         generator_llm=generator_llm_wrapped,
         docstore=docstore
)
```

the embeddings might also be an issue because HF embeddings doesn't like async

You mean wrapping the HuggingFacePipeline like this, right?
```
# critic model, should be more capable than generator?
model_name = 'mistralai/Mixtral-8x7B-Instruct-v0.1' 
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
transformers_pipeline = pipeline('text-generation', model=model, tokenizer=tokenizer)
critic_llm = HuggingFacePipeline(pipeline=transformers_pipeline)
critic_llm_wrapped = LangchainLLMWrapper(critic_llm)

# generator model, should be less capable than critic_llm?
model_name = 'mistralai/Mistral-7B-Instruct-v0.2' 
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
transformers_pipeline = pipeline('text-generation', model=model, tokenizer=tokenizer)
generator_llm = HuggingFacePipeline(pipeline=transformers_pipeline)
generator_llm_wrapped = LangchainLLMWrapper(generator_llm)
```

It throws the same error 😔

got it - this is the issue
```
TypeError: can only concatenate str (not "PromptValue") to str
```

There must be some peculiarities with the HFPipeline, I'll have to take some time to reproduce this myself. Or we could debug this in your system. Let me know if your up for that

also, we just released v0.1 so try it out. This won't fix the errro sadly 

@jjmachan I don't really have time at the moment 😕
If you are not able to reproduce it yourself before, let's talk next week!