Repository: confident-ai/deepeval
Issue: Support  integrating Ollama as a model provider
Root cause: Unimplemented Feature Gap

Hi, 
Is there a way to  use Ollama as an lLlm provider 

--terminal
deepeval set-local-model --model-name=llama3:8b --base-url="http://localhost:11434/v1/" --api-key="ollama"

Is there any way to do this programatically from Python?

> Is there any way to do this programatically from Python?

after you set it to use local model : 
```shell
deepeval set-local-model --model-name=llama3.1:8b \
    --base-url="http://localhost:11434/v1/" \
    --api-key="ollama"
```
you can call relevant function without specifying the model, it will automatically use the local model when need LLM 
```python
metric = SummarizationMetric(
    threshold=0.5,
    assessment_questions=[
        "Is the coverage score based on a percentage of 'yes' answers?",
        "Does the score ensure the summary's accuracy with the source?",
        "Does a higher score mean a more comprehensive summary?",
    ],
    include_reason=True,
    
)
```

this is how i run ollama local models:


```python 
from typing import Optional, Tuple, List, Union, Any
from langchain_openai import ChatOpenAI
from langchain_core.messages import AIMessage, BaseMessage
from langchain_core.outputs import ChatResult
from deepeval.models import DeepEvalBaseLLM
from deepeval.metrics.utils import trimAndLoadJson
from langchain_community.callbacks import get_openai_callback

class CustomChatOpenAI(ChatOpenAI):
    format: str = None

    def __init__(self, format: str = None, **kwargs):
        super().__init__(**kwargs)
        self.format = format

    async def _acreate(
        self, messages: List[BaseMessage], **kwargs
    ) -> ChatResult:
        if self.format:
            kwargs["format"] = self.format
        return await super()._acreate(messages, **kwargs)

class OllamaLLM(DeepEvalBaseLLM):
    
    def __init__(
        self,
        model_name: str,
        base_url: str = "http://localhost:11434/v1/",
        json_mode: bool = True,
        temperature: float = 0,
        *args,
        **kwargs
    ):

        self.model_name = model_name
        self.base_url = base_url
        self.json_mode = json_mode
        self.temperature = temperature
        self.args = args
        self.kwargs = kwargs
        super().__init__(model_name)

    def load_model(self) -> CustomChatOpenAI:
        """Load and configure the Ollama model."""
        return CustomChatOpenAI(
            model_name=self.model_name,
            openai_api_key="ollama",  
            base_url=self.base_url,
            format="json" if self.json_mode else None,
            temperature=self.temperature,
            *self.args,
            **self.kwargs,
        )

    def generate(self, prompt: str, schema: Any = None) -> Union[Any, Tuple[str, float]]:

        chat_model = self.load_model()
        with get_openai_callback() as cb:
            res = chat_model.invoke(prompt)
            if schema is not None:
                try:
                    # Try to parse the response using the schema
                    data = trimAndLoadJson(res.content, None)
                    return schema(**data)
                except Exception:
                    # If schema parsing fails, return parsed JSON
                    return trimAndLoadJson(res.content, None)
            return res.content, 0.0

    async def a_generate(self, prompt: str, schema: Any = None) -> Union[Any, Tuple[str, float]]:

        chat_model = self.load_model()
        with get_openai_callback() as cb:
            res = await chat_model.ainvoke(prompt)
            if schema is not None:
                try:
                    # Try to parse the response using the schema
                    data = trimAndLoadJson(res.content, None)
                    return schema(**data)
                except Exception:
                    # If schema parsing fails, return parsed JSON
                    return trimAndLoadJson(res.content, None)
            return res.content, 0.0

    def get_model_name(self) -> str:
        """Get the name of the current model."""
        return self.model_name

    @property
    def __class__(self):
        from deepeval.models import GPTModel  
        return GPTModel
```

> this is how i run ollama local models:
> 
> ```python
> from deepeval.models.gpt_model import CustomChatOpenAI
> 
> model_local = CustomChatOpenAI(
>                 model_name="llama3",
>                 openai_api_key="ollama",
>                 base_url="http://localhost:11434/v1/",
>                 # format=json to enable Ollama JSON mode
>                 temperature=0,
>             )
> 
> evaluator = DeepEvalFaithfulnessEvaluator(
>     model=model_local
> )
> ```

Do you also use this with synthesizer.generate* functions? If so, an example would be appreciated! 

> > this is how i run ollama local models:
> > ```python
> > from deepeval.models.gpt_model import CustomChatOpenAI
> > 
> > model_local = CustomChatOpenAI(
> >                 model_name="llama3",
> >                 openai_api_key="ollama",
> >                 base_url="http://localhost:11434/v1/",
> >                 # format=json to enable Ollama JSON mode
> >                 temperature=0,
> >             )
> > 
> > evaluator = DeepEvalFaithfulnessEvaluator(
> >     model=model_local
> > )
> > ```
> 
> Do you also use this with synthesizer.generate* functions? If so, an example would be appreciated!

i edited my original answer. 

i found a workaround by adding `def __class__(self)` at the end, now I'm able to use it as `synthesizer` in the following example: 

```python

from deepeval.synthesizer import Synthesizer

model_local = OllamaLLM(model_name=model_name)

synthesizer1= Synthesizer(model=model_local)
synthesizer.generate_goldens_from_contexts(
    contexts=[
        ["The Earth revolves around the Sun.", "Planets are celestial bodies."],
        ["Water freezes at 0 degrees Celsius.", "The chemical formula for water is H2O."],
    ]
)
```

> > > this is how i run ollama local models:
> > > ```python
> > > from deepeval.models.gpt_model import CustomChatOpenAI
> > > 
> > > model_local = CustomChatOpenAI(
> > >                 model_name="llama3",
> > >                 openai_api_key="ollama",
> > >                 base_url="http://localhost:11434/v1/",
> > >                 # format=json to enable Ollama JSON mode
> > >                 temperature=0,
> > >             )
> > > 
> > > evaluator = DeepEvalFaithfulnessEvaluator(
> > >     model=model_local
> > > )
> > > ```
> > 
> > 
> > Do you also use this with synthesizer.generate* functions? If so, an example would be appreciated!
> 
> i edited my original answer.
> 
> i found a workaround by adding `def __class__(self)` at the end, now I'm able to use it as `synthesizer` in the following example:
> 
> ```python
> from deepeval.synthesizer import Synthesizer
> 
> model_local = OllamaLLM(model_name=model_name)
> 
> synthesizer1= Synthesizer(model=model_local)
> synthesizer.generate_goldens_from_contexts(
>     contexts=[
>         ["The Earth revolves around the Sun.", "Planets are celestial bodies."],
>         ["Water freezes at 0 degrees Celsius.", "The chemical formula for water is H2O."],
>     ]
> )
> ```

Thanks for the quick reply! I will try to modify my code using your example. 

I'm wondering, do you also run `deepeval set-local-model` through the terminal before using your code? I'm getting `openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable` errors if i don't. 

Also, i don't understand what `deepeval set-local-model` does. Is it a declaration for deepeval to use my local ollama directly or is it tells deepeval that i will use local models somewhere in my code? 

my understanding is that after running `deepeval set-local-model`,  a file named `.deepeval` containing your custom configuration will be created in the project home directory, which looks something like:

```
# .deepeval
{"LOCAL_MODEL_NAME": "llama3", "LOCAL_MODEL_BASE_URL": "http://localhost:11434/v1/", "LOCAL_MODEL_API_KEY": "ollama", "LOCAL_MODEL_FORMAT": "json", "USE_LOCAL_MODEL": "YES", "USE_AZURE_OPENAI": "NO"}
``` 

when you instantiate a new model object from the `GPTModel` class, a class method called `should_use_local_model()` is triggered to check if `.deepeval` exists. If it doesnt exist, the class will return a `ChatOpenAI` object which reuqires an `OPENAI_API_KEY` to be provided. 




for anybody getting this error
ollama._types.ResponseError: 404 page not found (status code: 404)

You can try removing the v1 from the ollama base url
deepeval set-local-model --model-name=llama3.2:1b --base-url="http://localhost:11434/" --api-key="ollama"

The .deepeval file can also be directly updated if it already got created.