Repository: explodinggradients/ragas
Issue: [R-313] Adding more LLM/Embedding Providers
Root cause: Unimplemented Feature Gap

A running issue to get feedback on which LLM or Embedding Providers we need to add for you

<sub>[R-313](https://linear.app/exploding-gradients/issue/R-313/adding-more-llmembedding-providers)</sub>

If the output of my model is obtained through a method call, for example, def query_model(query): return response. Then how should I write the code to load the model? In addition, the Embedding model is also generated through a similar method call, providing methods for converting words into vectors and comparing vector similarities.

hey @kent-william007, in that case you will have to implement https://github.com/explodinggradients/ragas/blob/1e7121be03b5ed5a3ee24bfa4aaf5a875337068d/src/ragas/llms/base.py#L64-L83

on your own and make sure make sure that the result is of the type https://python.langchain.com/api_reference/core/outputs/langchain_core.outputs.llm_result.LLMResult.html

similarly for https://github.com/explodinggradients/ragas/blob/main/src/ragas/embeddings/base.py

let me know if you have further doubts, we'll help you out 🙂 

btw are you on discord?

Is it possible to support model invocation through an API? This way, we can leverage the model capabilities provided by the server without needing to worry about which underlying model is being used.


Trying to use Ragas with Google's Gemini models always gives me some headache.

Whether I'm [Evaluating Using Metrics](https://docs.ragas.io/en/latest/getstarted/rag_evaluation/) or trying to [Generate Synthetic Testset for RAG](https://docs.ragas.io/en/latest/getstarted/rag_testset_generation/) I keep getting errors such as `The LLM generation was not completed. Please increase try increasing the max_tokens and try again.`

Same result with different models e.g. `gemini-1.5-pro-001`,`gemini-1.5-pro-002`,  `gemini-1.5-flash`.

I just commented on [#1573 ](https://github.com/explodinggradients/ragas/issues/1573#issuecomment-2523303332) about getting the same thing.

Gemini support, please.
🟦🟥🟨🟩

@matheusft could you check this PR and docs: https://github.com/explodinggradients/ragas/pull/1728 this should help you.

let me know if it solves your issue or I'll look further into it

@jjmachan  I was having the same issue using google`s api. Seens like the #1728 works here with the googles api.

> @matheusft could you check this PR and docs: #1728 this should help you.
> 
> let me know if it solves your issue or I'll look further into it

It does not work for me as `LangchainLLMWrapper` does not implement the `.invoke()` method, so it's not working with my LangChain RAG.

I'm now getting
```
Expected a Runnable, callable or dict.Instead got an unsupported type: <class 'ragas.llms.base.LangchainLLMWrapper'>
```

i get this error

`AttributeError: 'GoogleGenerativeAIEmbeddings' object has no attribute 'set_run_config'
`
I used the LangchainLLMWrapper for the llm, but there seems to be an issue with embeddings.
I don't know why most libraries strictly enforces OPENAI when not all uses Open AI
Note: I am using gemini llm and embeddings

> > @matheusft could you check this PR and docs: #1728 this should help you.
> > let me know if it solves your issue or I'll look further into it
> 
> It does not work for me as `LangchainLLMWrapper` does not implement the `.invoke()` method, so it's not working with my LangChain RAG.
> 
> I'm now getting
> 
> ```
> Expected a Runnable, callable or dict.Instead got an unsupported type: <class 'ragas.llms.base.LangchainLLMWrapper'>
> ```

wrapped classes are only intended to be used inside Ragas, it is recommended to not use it anywhere else

> Trying to use Ragas with Google's Gemini models always gives me some headache.
> 
> Whether I'm [Evaluating Using Metrics](https://docs.ragas.io/en/latest/getstarted/rag_evaluation/) or trying to [Generate Synthetic Testset for RAG](https://docs.ragas.io/en/latest/getstarted/rag_testset_generation/) I keep getting errors such as `The LLM generation was not completed. Please increase try increasing the max_tokens and try again.`
> 
> Same result with different models e.g. `gemini-1.5-pro-001`,`gemini-1.5-pro-002`, `gemini-1.5-flash`.
> 
> I just commented on [#1573 ](https://github.com/explodinggradients/ragas/issues/1573#issuecomment-2523303332) about getting the same thing.
> 
> Gemini support, please. 🟦🟥🟨🟩

So would you have any suggestion on how to use Ragas with Gemini and Langchain that actually works?

hey @matheusft let me check this and get back to you. We had a community PR that should have helped fix this but I think that didn't solve the issue your facing? let me setup the google gemini workflow and get back to you by today 🙂 

Does anyone know how else on can evaluate a rag system without ragas?

hey @matheusft could you check this PR out: https://github.com/explodinggradients/ragas/pull/1759

I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂 

> hey @matheusft could you check this PR out: #1759
> 
> I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂

Fantastic!
I’ll give it a go this week and will let you know the outcome.

Thanks in advance.

> Hey @matheusft could you check this PR out: #1759
> 
> I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂

Hey,

I just finished testing it.
The latest version (`0.2.7`) is working fine for running `evaluate` ([this code](https://github.com/explodinggradients/ragas/issues/1573#issuecomment-2523303332)) using `Gemini`, no warnings or errors. 👏🏻

However, still getting `ERROR unable to apply transformation: The LLM generation was not completed. Please increase try increasing the max_tokens and try again.` when trying to run [Generate Testset](https://docs.ragas.io/en/latest/getstarted/rag_testset_generation/#choose-your-llm) code with `Gemini`.


```python

    from langchain_google_vertexai import VertexAI
    from langchain_google_vertexai import VertexAIEmbeddings
    from ragas.embeddings import LangchainEmbeddingsWrapper
    from ragas.llms import LangchainLLMWrapper
    from ragas.testset import TestsetGenerator
    
    llm = VertexAI(model_name="MY_LLM_MODEL")

    embedding_function = VertexAIEmbeddings(
        model_name="MY_EMBEDDING_MODEL",
        project_id = "MY_PROJECT",
        location = "MY_LOCATION",
        request_parallelism=1,
    )

    evaluator_llm = LangchainLLMWrapper(llm)
    evaluator_embeddings = LangchainEmbeddingsWrapper(embedding_function)
    
    generator = TestsetGenerator(llm=evaluator_llm, embedding_model=evaluator_embeddings)
    dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
```




> > Hey [@matheusft](https://github.com/matheusft) could you check this PR out: [#1759](https://github.com/explodinggradients/ragas/pull/1759)
> > I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂
> 
> Hey,
> 
> I just finished testing it. The latest version (`0.2.7`) is working fine for running `evaluate` ([this code](https://github.com/explodinggradients/ragas/issues/1573#issuecomment-2523303332)) using `Gemini`, no warnings or errors. 👏🏻
> 
> However, still getting `ERROR unable to apply transformation: The LLM generation was not completed. Please increase try increasing the max_tokens and try again.` when trying to run [Generate Testset](https://docs.ragas.io/en/latest/getstarted/rag_testset_generation/#choose-your-llm) code with `Gemini`.
> 
>     from langchain_google_vertexai import VertexAI
>     from langchain_google_vertexai import VertexAIEmbeddings
>     from ragas.embeddings import LangchainEmbeddingsWrapper
>     from ragas.llms import LangchainLLMWrapper
>     from ragas.testset import TestsetGenerator
>     
>     llm = VertexAI(model_name="MY_LLM_MODEL")
> 
>     embedding_function = VertexAIEmbeddings(
>         model_name="MY_EMBEDDING_MODEL",
>         project_id = "MY_PROJECT",
>         location = "MY_LOCATION",
>         request_parallelism=1,
>     )
> 
>     evaluator_llm = LangchainLLMWrapper(llm)
>     evaluator_embeddings = LangchainEmbeddingsWrapper(embedding_function)
>     
>     generator = TestsetGenerator(llm=evaluator_llm, embedding_model=evaluator_embeddings)
>     dataset = generator.generate_with_langchain_docs(docs, testset_size=10)

Hey @matheusft, are Ragas metrics available for gemini model without using langchain?

hey @kailashp19 not right now sadly - you have to use either langchain or llamaindex

how do you use the API right now?

@matheusft really sorry about the late response but were you able to figure it out? seeemed like out of context error - we did some improvements there but could be a model configuration issue too?

Hello, no worries.
No updates on my side, apologies.
I've moved as as I could not get past this error.

ahh my bad :( @matheusft 
what are u using for evals now?

I’m still looking for something to be fair 

> hey [@kailashp19](https://github.com/kailashp19) not right now sadly - you have to use either langchain or llamaindex
> 
> how do you use the API right now?
> 
> [@matheusft](https://github.com/matheusft) really sorry about the late response but were you able to figure it out? seeemed like out of context error - we did some improvements there but could be a model configuration issue too?

Actually, our team is trying to use RAGAs metrics with other LLMs like gemini, groq api. However it is not supporting with other LLMs. Any plans when RAGAs metrics will be usable for other LLMs?

@kailashp19 we do support other LLMs as well via Langchain and Llamaindex, do checkout this notebook doc https://docs.ragas.io/en/stable/howtos/customizations/customize_models/ ?

let me know if you're still facing trouble

@matheusft can I suggest maybe coming for an office hour? we would love to help you setup evals and get you unblocked if you give us a chance again 🙂 (link https://cal.com/team/ragas/office-hours)

any plans on adding support for watsonx.ai ?

Does anyone have any idea if I can use free hugging face models for RAGAS metrics?

Hey any one  knows how to use opensource models for RAGAS metric?

How can I use local model

I wish to use openRouter provider. Is it possible  ? 

These are already supported with `llm_factory` in latest ragas.