Repository: explodinggradients/ragas
Issue: Support for Running RAGAS Evaluation with Local Model (`openai/gpt-oss-20b`) Instead of OpenAI API
Root cause: Documentation Deficiency

Hi RAGAS team,

I am currently using the RAGAS framework to validate AI model outputs for metrics such as:

- **Faithfulness**
- **Answer Relevancy**
- **Context Precision**
- **Context Recall**

At present, these evaluations are being done via **OpenAI API calls**.
However, I would like to run these same evaluations **completely offline** using a **locally downloaded LLM**, specifically [`[openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b)`](https://huggingface.co/openai/gpt-oss-20b).

**My questions are:**

1. How can I configure RAGAS to point to a **local model instance** (Hugging Face Transformers pipeline) instead of calling the OpenAI API?
2. Are there any code examples or adapter interfaces in RAGAS that allow custom model backends for evaluation tasks?
3. For tasks like Faithfulness or Context Recall that rely on prompt-based evaluation, do I need to reimplement the prompt templates for a local model, or can I reuse the existing ones?
4. Are there any **performance or scoring differences** I should expect when moving from OpenAI API models to a local model?

Thanks for your help, and happy to contribute if this requires a code addition or documentation update.

+1 
Hey , i need this too .
Not everyone uses openapi , for example im running my project with gemini models.
It would be great if for evaluation using LLMs , you are using a global wrapper that supports the majority of LLMs with default the openapi if you like.

It's already supported.
Use `LangchainLLMWrapper` to wrap any LangChain LLM

You can use gemini as well.

```py
from langchain_google_genai import ChatGoogleGenerativeAI
from ragas.llms import LangchainLLMWrapper

gemini_llm = ChatGoogleGenerativeAI(model="gemini-pro")
ragas_llm = LangchainLLMWrapper(gemini_llm)

faithfulness = Faithfulness(llm=ragas_llm)
```



Thanks for quick response @anistark 

tried and fails , trying to read open api key , when trying to generate some ebbedings.

```python


from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
)
from langchain_google_genai import ChatGoogleGenerativeAI 
from ragas.llms import LangchainLLMWrapper 
gemini_llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash", google_api_key="<ENTER YOUR KEY HERE>")
ragas_llm = LangchainLLMWrapper(gemini_llm)

data = [
    {
        "question": "What was one of the earliest recorded forms of cryptographic schemes, and which ancient civilization is noted for employing it?",
        "contexts": ["One of the earliest recorded forms was hieroglyphic substitution ciphers, employed by the Egyptians."],
        "answer": "One of the earliest recorded forms was hieroglyphic substitution ciphers, employed by the Egyptians.",
        "reference": "One of the earliest recorded forms was hieroglyphic substitution ciphers, employed by the Egyptians.",
    }]

ragas_dataset = Dataset.from_list(data)

result = evaluate(
    ragas_dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_recall,
        context_precision,
    ],
    llm=ragas_llm
)
print("RAGAS evaluation completed.")
# Convert the result to pandas DataFrame and then to JSON
print(result.to_pandas().to_json(orient="records", indent=2))

```


Error : 

```
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1760529983.752517 1084559 alts_credentials.cc:93] ALTS creds ignored. Not running on GCP and untrusted ALTS is not enabled.
Traceback (most recent call last):
  File "<redacted>/src/dummy.py", line 25, in <module>
    result = evaluate(
             ^^^^^^^^^
  File "<redacted>/venv/lib/python3.12/site-packages/ragas/_analytics.py", line 227, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/venv/lib/python3.12/site-packages/ragas/evaluation.py", line 203, in evaluate
    embeddings = embedding_factory()
                 ^^^^^^^^^^^^^^^^^^^
  File "<redacted>/venv/lib/python3.12/site-packages/ragas/embeddings/base.py", line 377, in embedding_factory
    openai_embeddings = OpenAIEmbeddings(model=model)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/venv/lib/python3.12/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/venv/lib/python3.12/site-packages/langchain_openai/embeddings/base.py", line 331, in validate_environment
    self.client = openai.OpenAI(**client_params, **sync_specific).embeddings  # type: ignore[arg-type]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/venv/lib/python3.12/site-packages/openai/_client.py", line 137, in __init__
    raise OpenAIError(
openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
```

@Kos-M  The issue is that answer_relevancy metric requires embeddings, and it's defaulting to OpenAI embeddings. You need to provide local embeddings explicitly.

```py
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall,
context_precision
from langchain_google_genai import ChatGoogleGenerativeAI,
GoogleGenerativeAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

# Gemini LLM
gemini_llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash",
google_api_key="<YOUR_KEY>")
ragas_llm = LangchainLLMWrapper(gemini_llm)

# Gemini Embeddings (uses same API key)
gemini_embeddings = GoogleGenerativeAIEmbeddings(
    model="models/embedding-001",
    google_api_key="<YOUR_KEY>"
)
ragas_embeddings = LangchainEmbeddingsWrapper(gemini_embeddings)

# Configure metrics with custom LLM and embeddings
faithfulness.llm = ragas_llm
answer_relevancy.llm = ragas_llm
answer_relevancy.embeddings = ragas_embeddings  # Critical: answer_relevancy
needs embeddings
context_recall.llm = ragas_llm
context_precision.llm = ragas_llm

data = [{
    "question": "What was one of the earliest recorded forms of cryptographic
schemes?",
    "contexts": ["One of the earliest recorded forms was hieroglyphic
substitution ciphers, employed by the Egyptians."],
    "answer": "One of the earliest recorded forms was hieroglyphic substitution
ciphers, employed by the Egyptians.",
    "ground_truth": "One of the earliest recorded forms was hieroglyphic
substitution ciphers, employed by the Egyptians.",
}]

ragas_dataset = Dataset.from_list(data)

result = evaluate(
    ragas_dataset,
    metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
)

print("RAGAS evaluation completed.")
print(result.to_pandas().to_json(orient="records", indent=2))
```

Alternative: Use local HuggingFace embeddings (completely offline):

```py
from ragas.embeddings import HuggingFaceEmbeddings

# Local embeddings (no API key needed)
local_embeddings = HuggingFaceEmbeddings(
    model="sentence-transformers/all-MiniLM-L6-v2",
    use_api=False
)

answer_relevancy.embeddings = local_embeddings
```



Oh i see what i was doing wrong , 
Thank you so much @anistark  , it works like a charm <3

Thanks @Kos-M 
Closing this issue as done. Feel free to reopen or new issue if there's any further block.