Repository: explodinggradients/ragas
Issue: Issues regarding ragas custom model
GitHub labels: bug

enhancement

Hi folks, please report issues regarding the ragas custom critic model in this thread.

Docs : https://docs.ragas.io/en/latest/howtos/customisations/ragas_custom_model.html
Model: https://huggingface.co/explodinggradients/Ragas-critic-llm-Qwen1.5-GPTQ

Hi there! Thank you for your work! 

I have a little question - is it possible to use this critic model for evaluation (not for test data generation)? I mean pass it as evaluate(llm=critic_llm), for example to calculate faithfulness. 

If not, do you have plans to fine-tune model to replace gpt-4 in evaluation?

Hey @valer1435  the critic model is now fine-tuned only to act in test generation. 

Why is it so hard to find a full working example of how to use a custom LLM for test set generation ?

I'm assuming this didn't help @projects-g -> https://docs.ragas.io/en/stable/howtos/customisations/ragas_custom_model.html

where are you stuck at?

@jjmachan The documentation page you shared is not found.

Hey @wanjeakshay https://docs.ragas.io/en/latest/howtos/customizations/customize_models/#examples
You can initialise any model using this approach and use it
Where is the issue? 


 @shahules786 can we use huggingface model for test set generation, as generator as well as critic llm ?


@wanjeakshay there is no generator and critic in test generation 0.2. Just one llm. 
Yes you may use any LLM of your choice as shown in the doc https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/
please check the docs. 

@jjmachan  I am using ollama for generating testest, it is not working, for evaluation also i am getting Nan as response.
```py
from langchain_community.llms import Ollama
from langchain_community.embeddings import OllamaEmbeddings

# model_name = "llama3.2:1b-instruct-q2_K"
model_name = "llama3.2:3b"
langchain_llm = Ollama(model=model_name)
langchain_embeddings = OllamaEmbeddings(model="nomic-embed-text")

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
generator_llm = LangchainLLMWrapper(langchain_llm)
generator_embeddings = LangchainEmbeddingsWrapper(langchain_embeddings)

from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
df = generator.generate_with_langchain_docs(docs[:2], testset_size=3)
```

The error i am getting,
```
ValueError                                Traceback (most recent call last)
Cell In[72], [line 4](vscode-notebook-cell:?execution_count=72&line=4)
      [1](vscode-notebook-cell:?execution_count=72&line=1) from ragas.testset import TestsetGenerator
      [3](vscode-notebook-cell:?execution_count=72&line=3) generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
----> [4](vscode-notebook-cell:?execution_count=72&line=4) df = generator.generate_with_langchain_docs(docs[:2], testset_size=3)

File c:\Users\Lenovo\Desktop\Nimbusnext\Ragas\myenv\lib\site-packages\ragas\testset\synthesizers\generate.py:188, in TestsetGenerator.generate_with_langchain_docs(self, documents, testset_size, transforms, transforms_llm, transforms_embedding_model, query_distribution, run_config, callbacks, with_debugging_logs, raise_exceptions)
    [185](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:185) apply_transforms(kg, transforms)
    [186](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:186) self.knowledge_graph = kg
--> [188](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:188) return self.generate(
    [189](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:189)     testset_size=testset_size,
    [190](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:190)     query_distribution=query_distribution,
    [191](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:191)     run_config=run_config,
    [192](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:192)     callbacks=callbacks,
    [193](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:193)     with_debugging_logs=with_debugging_logs,
    [194](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:194)     raise_exceptions=raise_exceptions,
    [195](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:195) )

File c:\Users\Lenovo\Desktop\Nimbusnext\Ragas\myenv\lib\site-packages\ragas\testset\synthesizers\generate.py:369, in TestsetGenerator.generate(self, testset_size, query_distribution, num_personas, run_config, batch_size, callbacks, token_usage_parser, with_debugging_logs, raise_exceptions)
    [366](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:366)     patch_logger("ragas.experimental.testset.transforms", logging.DEBUG)
    [368](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:368) if self.persona_list is None:
--> [369](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:369)     self.persona_list = generate_personas_from_kg(
    [370](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:370)         llm=self.llm,
    [371](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/synthesizers/generate.py:371)         kg=self.knowledge_graph,
...
     [97](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/persona.py:97)     )
     [99](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/persona.py:99) summaries = [node.properties.get("summary") for node in nodes]
    [100](file:///C:/Users/Lenovo/Desktop/Nimbusnext/Ragas/myenv/lib/site-packages/ragas/testset/persona.py:100) summaries = [summary for summary in summaries if isinstance(summary, str)]

ValueError: No nodes that satisfied the given filer. Try changing the filter.
```

Can you tell what I am doing wrong
Ragas version- ragas-0.2.6
python- 3.10


@wanjeakshay I have the same problem with same message
@shahules786 Can you help us to solve this?

hey @DatTNH23 are you also using ollama? we don't support that yet sadly

Hi @jjmachan,
I create my own Ollama Class and it is a child from BaseLanguageModel. It works only a few times and fails almost

`class OllamaLLM(BaseLanguageModel):
    model: str = "llama3.1"
    def generate_prompt(
        self,
        prompts: List[PromptValue],
        stop: Optional[List[str]] = None,
        callbacks: Optional[List[callable]] = None,
        **kwargs,
    ) -> LLMResult:
        
        print("run normal generate_prompt")
        try:
            prompt = prompts[0].to_string()
            model_response = self.predict(prompt)
            res = Generation(text=model_response)

            return LLMResult(generations=[[res]])
        except requests.exceptions.RequestException as e:
            print(f"Error making request to Ollama API: {e}")
            return LLMResult(generations=[[]])

    async def agenerate_prompt(
        self,
        prompts: List[PromptValue],
        stop: Optional[List[str]] = None,
        callbacks: Optional[List[callable]] = None,
        **kwargs,
    ) -> LLMResult:
        print("run agenerate_prompt")
        results = self.generate_prompt(prompts, stop=stop, **kwargs)
        print("FINISH agenerate_prompt")
        return results
        #return LLMResult(generations=results)
    
    #deprecated
    def predict(
        self, text: str, *, stop: Optional[Sequence[str]] = None, **kwargs: Any
    ) -> str:
        try: 
            res = ollama.generate(model=self.model, prompt=text)
            print('make predict', self.model)
            model_response = res["response"]
        except Exception as e:
            print(e)
        return model_response
    #deprecated
    def predict_messages(
        self,
        messages: List[BaseMessage],
        *,
        stop: Optional[Sequence[str]] = None,
        **kwargs: Any,
    ) -> BaseMessage:
        return None
    #deprecated
    async def apredict(
        self, text: str, *, stop: Optional[Sequence[str]] = None, **kwargs: Any
    ) -> str:
        return ""
    #deprecated
    async def apredict_messages(
        self,
        messages: List[BaseMessage],
        *,
        stop: Optional[Sequence[str]] = None,
        **kwargs: Any,
    ) -> BaseMessage:
        return None
    def invoke():
        return None

 def evaluate_data():
    model_name = "llama3.1"
    embedding_model = "bge-m3"
    langchain_llm = OllamaLLM(model=model_name)
    llm_wrapper = LangchainLLMWrapper(langchain_llm)
    embedding_wrapper = LangchainEmbeddingsWrapper(langchain_embeddings)
    evaluator_llm = llm_wrapper

    # Sample 1
    sample1 = SingleTurnSample(
        user_input="What is the capital of Germany?",
        retrieved_contexts=["Berlin is the capital and largest city of Germany."],
        response="The capital of Germany is Berlin.",
        reference="Berlin",
    )

    # Sample 2
    sample2 = SingleTurnSample(
        user_input="Who wrote 'Pride and Prejudice'?",
        retrieved_contexts=["'Pride and Prejudice' is a novel by Jane Austen."],
        response="'Pride and Prejudice' was written by Jane Austen.",
        reference="Jane Austen",
    )

    # Sample 3
    sample3 = SingleTurnSample(
        user_input="What's the chemical formula for water?",
        retrieved_contexts=["Water has the chemical formula H2O."],
        response="The chemical formula for water is H2O.",
        reference="H2O",
    )
    dataset = EvaluationDataset(samples=[sample1,sample2,sample3])

    # eval_dataset = EvaluationDataset.from_hf_dataset(dataset)
    metrics = [
        # LLMContextRecall(llm=evaluator_llm),
        FactualCorrectness(llm=evaluator_llm),
    ]

    results = evaluate(dataset=dataset, metrics=metrics, raise_exceptions=True)
    df = results.to_pandas()
    df.head()
    print(df)
    print("FINISH")


def main():
   evaluate_data()


if __name__ == "__main__":
    main()

`
The error is:
`Evaluating:  33%|██████████████████████████████████████████████████                                                                                                    | 1/3 [02:25<04:50, 145.16s/it]
Traceback (most recent call last):
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/langchain_core/output_parsers/pydantic.py", line 28, in _parse_obj
    return self.pydantic_object.model_validate(obj)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/pydantic/main.py", line 596, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        obj, strict=strict, from_attributes=from_attributes, context=context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
pydantic_core._pydantic_core.ValidationError: 1 validation error for StringIO
text
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 402, in parse_output_string
    result = super().parse(jsonstr)
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/langchain_core/output_parsers/pydantic.py", line 83, in parse
    return super().parse(text)
           ~~~~~~~~~~~~~^^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/langchain_core/output_parsers/json.py", line 97, in parse
    return self.parse_result([Generation(text=text)])
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/langchain_core/output_parsers/pydantic.py", line 72, in parse_result
    raise e
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/langchain_core/output_parsers/pydantic.py", line 68, in parse_result
    return self._parse_obj(json_object)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/langchain_core/output_parsers/pydantic.py", line 36, in _parse_obj
    raise self._parser_exception(e, obj) from e
langchain_core.exceptions.OutputParserException: Failed to parse StringIO from completion {}. Got: 1 validation error for StringIO
text
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/OUTPUT_PARSING_FAILURE 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nguyenthanhdat/Documents/Projekte/test/evaluate_new.py", line 200, in <module>
    main()
    ~~~~^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/evaluate_new.py", line 194, in main
    evaluate_data()
    ~~~~~~~~~~~~~^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/evaluate_new.py", line 185, in evaluate_data
    results = evaluate(dataset=dataset, metrics=metrics, raise_exceptions=True, callbacks=[TestCallback()],
                       run_config=RunConfig(timeout=10,max_retries=1,max_wait=60,max_workers=1))
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/_analytics.py", line 227, in wrapper
    result = func(*args, **kwargs)
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/evaluation.py", line 315, in evaluate
    raise e
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/evaluation.py", line 295, in evaluate
    results = executor.results()
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/executor.py", line 213, in results
    results = asyncio.run(self._process_jobs())
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/nest_asyncio.py", line 30, in run
    return loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
           ~~~~~~~~^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/futures.py", line 200, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/tasks.py", line 304, in __step_run_and_handle_result
    result = coro.send(None)
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/executor.py", line 141, in _process_jobs
    await self._process_coroutines(
        self.jobs, internal_pbar, results, max_workers
    )
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/executor.py", line 191, in _process_coroutines
    result = await future
             ^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/tasks.py", line 634, in _wait_for_one
    return f.result() if resolve else f
           ~~~~~~~~^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/futures.py", line 200, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/tasks.py", line 304, in __step_run_and_handle_result
    result = coro.send(None)
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/executor.py", line 48, in sema_coro
    return await coro
           ^^^^^^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/executor.py", line 100, in wrapped_callable_async
    raise e
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/executor.py", line 96, in wrapped_callable_async
    result = await callable(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/metrics/base.py", line 502, in single_turn_ascore
    raise e
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/metrics/base.py", line 495, in single_turn_ascore
    score = await asyncio.wait_for(
            ^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
    )
    ^
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/metrics/_factual_correctness.py", line 287, in _single_turn_ascore
    reference_claims = await self.decompose_claims(reference, callbacks)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/metrics/_factual_correctness.py", line 254, in decompose_claims
    result = await self.claim_decomposition_prompt.generate(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        data=prompt_input, llm=self.llm, callbacks=callbacks
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 126, in generate
    output_single = await self.generate_multiple(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 212, in generate_multiple
    raise e
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 200, in generate_multiple
    answer = await parser.parse_output_string(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 410, in parse_output_string
    fixed_output_string = await fix_output_format_prompt.generate(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 126, in generate
    output_single = await self.generate_multiple(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 212, in generate_multiple
    raise e
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 200, in generate_multiple
    answer = await parser.parse_output_string(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 410, in parse_output_string
    fixed_output_string = await fix_output_format_prompt.generate(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 126, in generate
    output_single = await self.generate_multiple(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 212, in generate_multiple
    raise e
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 200, in generate_multiple
    answer = await parser.parse_output_string(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 410, in parse_output_string
    fixed_output_string = await fix_output_format_prompt.generate(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 126, in generate
    output_single = await self.generate_multiple(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 212, in generate_multiple
    raise e
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 200, in generate_multiple
    answer = await parser.parse_output_string(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/Users/nguyenthanhdat/Documents/Projekte/test/myenv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 422, in parse_output_string
    raise RagasOutputParserException()
ragas.exceptions.RagasOutputParserException: The output parser failed to parse the output including retries.`
I am using ragas **0.2.9**

My class work very good with version 0.1.x, but when I try to migrate to 0.2.x, then it never succeed.






hey @DatTNH23 unfortunatly we dont' have a solution for you yet 🙁. Since I don't have access to a good GPU haven't been able to get a good solution for this either but do keep an eye on https://github.com/explodinggradients/ragas/issues/1170 

sorry about that

I am facing the same issue: `ValueError: No nodes that satisfied the given filer. Try changing the filter.`, via LLM model loaded via `vllm` and `HuggingFaceEndpoint`, ref:

Llm:
```
from ragas.llms import LangchainLLMWrapper
from vllm import LLM, SamplingParams
# Create an LLM.
llm = LLM(model="ibm-granite/granite-3.1-8b-instruct", max_model_len=8192)
generator_llm = LangchainLLMWrapper(llm)
```
and, 
```
repo_id = "mistralai/Mistral-7B-Instruct-v0.2"

llm = HuggingFaceEndpoint(
    repo_id=repo_id,
    temperature=0.5,
    huggingfacehub_api_token=HUGGINGFACEHUB_API_TOKEN,
)
generator_llm = LangchainLLMWrapper(llm)
```

Embedding:
```
from langchain_huggingface import HuggingFaceEmbeddings
model_name = "sentence-transformers/all-mpnet-base-v2"
# model_name = "nomic-ai/nomic-embed-text-v1.5"
model_kwargs = {'device': 'cpu', 'trust_remote_code': True}
encode_kwargs = {'normalize_embeddings': False}
hf = HuggingFaceEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)
```

Testset generation via Ragas:
```
from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm, embedding_model=hf)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
```

Results when LLM loaded via VLLM:
```
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'

unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node

unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'
unable to apply transformation: 'LLM' object has no attribute 'agenerate_prompt'

unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'

unable to apply transformation: Node 6bb4f54c-b9bb-4c7c-85d3-c2d9c77d2146 has no summary_embedding

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[6], line 4
      1 from ragas.testset import TestsetGenerator
      3 generator = TestsetGenerator(llm=generator_llm, embedding_model=hf)
----> 4 dataset = generator.generate_with_langchain_docs(docs, testset_size=10)

File ~/.env/lib64/python3.11/site-packages/ragas/testset/synthesizers/generate.py:188, in TestsetGenerator.generate_with_langchain_docs(self, documents, testset_size, transforms, transforms_llm, transforms_embedding_model, query_distribution, run_config, callbacks, with_debugging_logs, raise_exceptions)
    185 apply_transforms(kg, transforms)
    186 self.knowledge_graph = kg
--> 188 return self.generate(
    189     testset_size=testset_size,
    190     query_distribution=query_distribution,
    191     run_config=run_config,
    192     callbacks=callbacks,
    193     with_debugging_logs=with_debugging_logs,
    194     raise_exceptions=raise_exceptions,
    195 )

File ~/.env/lib64/python3.11/site-packages/ragas/testset/synthesizers/generate.py:369, in TestsetGenerator.generate(self, testset_size, query_distribution, num_personas, run_config, batch_size, callbacks, token_usage_parser, with_debugging_logs, raise_exceptions)
    366     patch_logger("ragas.experimental.testset.transforms", logging.DEBUG)
    368 if self.persona_list is None:
--> 369     self.persona_list = generate_personas_from_kg(
    370         llm=self.llm,
    371         kg=self.knowledge_graph,
    372         num_personas=num_personas,
    373         callbacks=callbacks,
    374     )
    375 else:
    376     random.shuffle(self.persona_list)

File ~/.env/lib64/python3.11/site-packages/ragas/testset/persona.py:95, in generate_personas_from_kg(kg, llm, persona_generation_prompt, num_personas, filter_fn, callbacks)
     93 nodes = [node for node in kg.nodes if filter_fn(node)]
     94 if len(nodes) == 0:
---> 95     raise ValueError(
     96         "No nodes that satisfied the given filer. Try changing the filter."
     97     )
     99 summaries = [node.properties.get("summary") for node in nodes]
    100 summaries = [summary for summary in summaries if isinstance(summary, str)]

ValueError: No nodes that satisfied the given filer. Try changing the filter.
```


Results when LLM loaded via HuggingFaceEndpoint:
```
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt headlines_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt headlines_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt headlines_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt headlines_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt headlines_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.

unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node
unable to apply transformation: 'headlines' property not found in this node

Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt summary_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt summary_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt summary_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt summary_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
Prompt summary_extractor_prompt failed to parse output: The output parser failed to parse the output including retries.
unable to apply transformation: The output parser failed to parse the output including retries.

unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'

unable to apply transformation: Node 65e9d592-d0d5-4c5e-bfb2-31eb058405c1 has no summary_embedding

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[18], line 4
      1 from ragas.testset import TestsetGenerator
      3 generator = TestsetGenerator(llm=generator_llm, embedding_model=hf)
----> 4 dataset = generator.generate_with_langchain_docs(docs, testset_size=10)

File ~/.env/lib64/python3.11/site-packages/ragas/testset/synthesizers/generate.py:188, in TestsetGenerator.generate_with_langchain_docs(self, documents, testset_size, transforms, transforms_llm, transforms_embedding_model, query_distribution, run_config, callbacks, with_debugging_logs, raise_exceptions)
    185 apply_transforms(kg, transforms)
    186 self.knowledge_graph = kg
--> 188 return self.generate(
    189     testset_size=testset_size,
    190     query_distribution=query_distribution,
    191     run_config=run_config,
    192     callbacks=callbacks,
    193     with_debugging_logs=with_debugging_logs,
    194     raise_exceptions=raise_exceptions,
    195 )

File ~/.env/lib64/python3.11/site-packages/ragas/testset/synthesizers/generate.py:369, in TestsetGenerator.generate(self, testset_size, query_distribution, num_personas, run_config, batch_size, callbacks, token_usage_parser, with_debugging_logs, raise_exceptions)
    366     patch_logger("ragas.experimental.testset.transforms", logging.DEBUG)
    368 if self.persona_list is None:
--> 369     self.persona_list = generate_personas_from_kg(
    370         llm=self.llm,
    371         kg=self.knowledge_graph,
    372         num_personas=num_personas,
    373         callbacks=callbacks,
    374     )
    375 else:
    376     random.shuffle(self.persona_list)

File ~/.env/lib64/python3.11/site-packages/ragas/testset/persona.py:95, in generate_personas_from_kg(kg, llm, persona_generation_prompt, num_personas, filter_fn, callbacks)
     93 nodes = [node for node in kg.nodes if filter_fn(node)]
     94 if len(nodes) == 0:
---> 95     raise ValueError(
     96         "No nodes that satisfied the given filer. Try changing the filter."
     97     )
     99 summaries = [node.properties.get("summary") for node in nodes]
    100 summaries = [summary for summary in summaries if isinstance(summary, str)]

ValueError: No nodes that satisfied the given filer. Try changing the filter.
```

I am not sure if model load via VLLM/HuggingFaceEndpoint is supported or not, and if not then how can I load a model of my choice for `testset generation`, as specified under [Others](https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/#__tabbed_1_4) under choose your LLM.


Plz lemme know if anyone was able to resolve the issue, loading model other OpenAI, AWS/Azure. Thanks!

hey @justjais did you figure this out? 

I'm not sure if this is a langchain LLM object
> from vllm import LLM, SamplingParams 

which you are wrapping