Repository: explodinggradients/ragas
Issue: Gemini supported llm and embedding docs
Root cause: Configuration Error

**Describe the Feature**I
I have access to Gemini for llm_judge and embedding model I was using mixedbread-ai/mxbai-embed-large-v1
I am facing issue in 3 metrics - faithfulness, response_relevancy, context_recall
I am attaching the ss for your reference:

<img width="720" height="567" alt="Image" src="https://github.com/user-attachments/assets/7c920062-289c-4717-9689-f1458818c77b" />

**Why is the feature important for you?**
As our organization has access to Gemini key so I want ragas to be compatible with gemini models

**Additional context**
Add any other context about the feature you want to share with us.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


@bsatapat-jpg Can you please share which ragas version are you using and a sample code to replicate the bug and also your logs as to what you're facing.

is the AnswerAccuracy() works with  Gemini
` metrics['answer_accuracy'] = AnswerAccuracy(llm=self.llm)`
as i am try using it but it always give me that is is not supported by Gemini 
could you let me How it works ?

> is the AnswerAccuracy() works with Gemini ` metrics['answer_accuracy'] = AnswerAccuracy(llm=self.llm)` as i am try using it but it always give me that is is not supported by Gemini could you let me How it works ?

Could you share the entire code snippet?
Also, the error log that you're getting.


> > is the AnswerAccuracy() works with Gemini ` metrics['answer_accuracy'] = AnswerAccuracy(llm=self.llm)` as i am try using it but it always give me that is is not supported by Gemini could you let me How it works ?
> 
> Could you share the entire code snippet? Also, the error log that you're getting.


```
from typing import Dict, Any, Optional
import pandas as pd

from ragas import evaluate, EvaluationDataset
from ragas.metrics import AspectCritic
from ragas.metrics.collections import ContextPrecision
from ragas.llms import llm_factory
import google.generativeai as genai

# -----------------------------
# 1️⃣ Gemini API Key & Client
# -----------------------------
GEMINI_API_KEY = "Gemini api key"
genai.configure(api_key=GEMINI_API_KEY)
# Create Gemini client
gemini_client = genai.GenerativeModel("gemini-2.0-flash")

# Attempt to create RAGAS LLM wrapper using Gemini
try:
    gemini_llm = llm_factory(
        model_name="gemini-2.0-flash",
        provider="google",
        client=gemini_client
    )
    print("✅ Gemini LLM created via llm_factory")
except Exception as e:
    print("❌ LLM creation failed:", e)
    gemini_llm = None

# Try initializing metrics
metrics = {}
try:
    metrics["context_precision"] = ContextPrecision(llm=gemini_llm)
    print("✅ ContextPrecision metric initialized")
except Exception as e:
    print("❌ ContextPrecision initialization failed:", e)

try:
    metrics["accuracy"] = AspectCritic(
        name="accuracy",
        llm=gemini_llm,
        definition="Check if the response is factually correct."
    )
    print("✅ AspectCritic metric initialized")
except Exception as e:
    print("❌ AspectCritic initialization failed:", e)

# Check if the metrics have a .measure method
for name, metric in metrics.items():
  print(f"Metric: {name}, has .measure:", hasattr(metric, "measure"))

```
 Error appear : 
   LLM creation failed: llm_factory() missing 1 required positional argument: 'model'
  ❌ ContextPrecision initialization failed: Collections metrics only support modern InstructorLLM. Found: NoneType. Use: llm_factory('gpt-4o-mini', client=openai_client)
  Metric: accuracy, has .measure: False


Problem: Gemini LLM cannot be used with ContextPrecision or AnswerAccuracy
Reason: RAGAS metrics require modern InstructorLLM with .measure(), which Gemini currently does not provide.
Impact: Any attempt to use Gemini as the LLM in reference-based metrics fails — metrics are invalid.

@Farahala Thanks for the code. Yes, instructor doesn't work. We've a workaround using litellm that works. Will add it properly to ragas for easier switch later.

Try using `LiteLLMEmbeddings`. Something like this:

```py
import asyncio
from litellm import OpenAI
from ragas.llms import llm_factory
from ragas.embeddings import LiteLLMEmbeddings
from ragas.metrics.collections import ContextPrecision, AnswerCorrectness
from ragas import aevaluate
from datasets import Dataset

async def main():
    # Create LiteLLM client for Gemini
    client = OpenAI(
        api_key="your-gemini-api-key",
        model="gemini-2.0-flash"
    )

    # Create LLM with LiteLLM provider
    llm = llm_factory(
        model="gemini-2.0-flash",
        provider="litellm",
        client=client
    )

    # Create embeddings
    embeddings = LiteLLMEmbeddings(model="text-embedding-3-small")

    # Create metrics (now works with Gemini!)
    metrics = {
        "context_precision": ContextPrecision(llm=llm),
        "answer_correctness": AnswerCorrectness(llm=llm, embeddings=embeddings),
    }

    # Evaluate (uses Gemini through LiteLLM)
    result = await aevaluate(
        dataset,
        metrics=list(metrics.values()),
        llm=llm,
    )

    print(result.to_pandas())

asyncio.run(main())
```

Hello @anistark ,

I see that there is a new version of Ragas and I tried the exact example provided in docs/howtos/integrations/gemini.md:

```
import os
from datasets import Dataset
import google.generativeai as genai
from ragas import evaluate
from ragas.llms import llm_factory
from ragas.metrics import (
    AnswerCorrectness,
    ContextPrecision,
    ContextRecall,
    Faithfulness
)

# Initialize Gemini client
genai.configure(api_key=os.environ.get("GOOGLE_API_KEY"))
client = genai.GenerativeModel("gemini-2.0-flash")
llm = llm_factory("gemini-2.0-flash", provider="google", client=client)

# Create sample evaluation data
data = {
    "question": ["What is the capital of France?"],
    "answer": ["Paris is the capital of France."],
    "contexts": [["France is a country in Western Europe. Paris is its capital."]],
    "ground_truth": ["Paris"]
}

dataset = Dataset.from_dict(data)

# Define metrics
metrics = [
    ContextPrecision(llm=llm),
    ContextRecall(llm=llm),
    Faithfulness(llm=llm),
    AnswerCorrectness(llm=llm)
]

# Run evaluation
results = evaluate(dataset, metrics=metrics)
print(results)
```


However, even though I am using my Gemini API key, I still get the following error:

`openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable`


It seems that the metrics (ContextPrecision, ContextRecall, Faithfulness, AnswerCorrectness) are still trying to use OpenAI embeddings internally. Could you advise how to run evaluation fully with Gemini only without requiring an OpenAI API key?

Thanks @Farahala for checking in. You're right, the mapping was broken. Fixed in #2454 


Hello @anistark
is it fixed ? if Not when it can be fixed
 And if Yes ? I am still getting the same error  when  I try Option 1: Default Embeddings (Recommended)
 `openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable`

> Hello [@anistark](https://github.com/anistark) is it fixed ? if Not when it can be fixed And if Yes ? I am still getting the same error when I try Option 1: Default Embeddings (Recommended) `openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable`

Are you using from `main` branch or the released version 0.3.9?


> > Hello [@anistark](https://github.com/anistark) is it fixed ? if Not when it can be fixed And if Yes ? I am still getting the same error when I try Option 1: Default Embeddings (Recommended) `openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable`
> 
> Are you using from `main` branch or the released version 0.3.9?

form the released  version : 0.3.10

> form the released version : 0.3.10

There's no released 0.3.10 version. Could you share how you're installing and calling it?

Here's all the [releases](https://pypi.org/project/ragas/#history).

> > form the released version : 0.3.10
> 
> There's no released 0.3.10 version. Could you share how you're installing and calling it?
> 
> Here's all the [releases](https://pypi.org/project/ragas/#history).

oki i uninstall it and then reinstall it again 
Ragas version : 0.3.9
But iam still getting the same error `openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable`

Here is the code :
`import os
import google.generativeai as genai
from ragas.llms import llm_factory
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import AnswerCorrectness, ContextPrecision, ContextRecall, Faithfulness
import ragas

print(ragas.__version__)

genai.configure(api_key="YourGeminiAPIkey")
client = genai.GenerativeModel("gemini-2.0-flash")
llm = llm_factory("gemini-2.0-flash", provider="google", client=client)


data = {
    "question": ["What is the capital of France?"],
    "answer": ["Paris is the capital of France."],
    "contexts": [["France is a country in Western Europe. Paris is its capital."]],
    "ground_truth": ["Paris"]
}

dataset = Dataset.from_dict(data)

metrics = [
    ContextPrecision(llm=llm),
    ContextRecall(llm=llm),
    Faithfulness(llm=llm),
    AnswerCorrectness(llm=llm) 
]


results = evaluate(dataset, metrics=metrics)
print(results)`

@Farahala 
The PR I had merged above is not released yet. You can test it from repo main branch if you want. Or wait for upcoming release next week.

@anistark  I uninstall and re-install it now from the main branch using : pip install --upgrade --force-reinstall git+https://github.com/vibrantlabsai/ragas.git@main , but still  getting the same error 
` openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable`

@Farahala 
There seem to still be some issue with `genai` module. Can you try this PR #2466 to test if it works for you?