Repository: confident-ai/deepeval
Issue: Using custom models as input for metrics
Root cause: Documentation Deficiency

**Describe the bug**
It seems that model input for different metrics cannot accept anything but gpt model. Is this intended? Working specifically with `AnswerRelevancyModel`, if I pass this as an instance to `AnswerRelevancyMetric` I get `AttributeError: 'AnswerRelevancyModel' object has no attribute 'get_model_name'`. 

**To Reproduce**
Steps to reproduce the behavior:
```python
from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.models.answer_relevancy_model import AnswerRelevancyModel

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = AnswerRelevancyMetric(
    threshold=0.7,
    model=AnswerRelevancyModel(),
    include_reason=True
)
```

**Expected behavior**
SInce `AnswerRelevancyModel()` is an instance of `DeepEvalBaseModel` I would expect this to work

**Desktop (please complete the following information):**
 - OS: Debian 12

**Additional context**
I'm wondering about how easy it to use custom models in general and integerate with deepeval. For example if I wanted to not rely on gpt and instead use another model like prometheus.


Yup this is intended, our docs have the type each model parameter accepts, for example for answer relevancy: https://docs.confident-ai.com/docs/metrics-answer-relevancy#example

The Answer relevancy model simply exists right now, and was used previously for the answer relevancy metric, not anymore

Ok but then I find it weird to have these first 2 lines in this conditional:
```python
        if isinstance(model, DeepEvalBaseModel):
            self.model = model
        else:
            self.model = GPTModel(model=model)
```
in answer_relevancy.py but I'm supposing now that this is has become legacy and not updated yet.

I guess the most important question I have is if it is worth it for me to look into DeepEval at all if I do not want to rely on openai? 

Edit: I'm not necessarily concerned with answer relevancy btw but a framework such a deepeval that defines several metrics for evaluation of LLM apps in production but also gives the option of using custom models when it comes to LLM evaluators

Ah I see why you’re confused now. We’re just using the same base model for
both custom LLMs and regular models. So while AnswerRelevancyModel also
inherits from the same base model as the one we used as a custom LLM as
shown in the docs, they are intended to be different. It was me being
sloppy, will separate them in the future.

Here is the docs to create a custom LLM. You can use ANY LLM you wish. But
whether your own custom LLM can follow instructions or not, such as
outputting a valid JSON, is up to you :-)

https://docs.confident-ai.com/docs/metrics-introduction#using-a-custom-llm

So in short yes, deepeval is extremely flexible and does not depend on
openAI.


On Wed, 14 Feb 2024 at 17:06, Yasmin ***@***.***> wrote:

> Ok but then I find it weird to have these first 2 lines in this
> conditional:
>
>         if isinstance(model, DeepEvalBaseModel):
>             self.model = model
>         else:
>             self.model = GPTModel(model=model)
>
> in answer_relevancy.py but I'm supposing now that this is has become
> legacy and not updated yet.
>
> I guess the most important question I have is if it is worth it for me to
> look into DeepEval at all if I do not want to rely on openai?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/confident-ai/deepeval/issues/473#issuecomment-1943255466>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BCFQK667Z6DKZLKCS7OX3HDYTRWANAVCNFSM6AAAAABDGJYJQCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGI2TKNBWGY>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


> https://docs.confident-ai.com/docs/metrics-introduction#using-a-custom-llm

Yes this looks like what I'm looking for thx!

Ok it's more clear now, good to see that it is flexible as such because it seems like a great tool

No problem! Feel free to raise any further issues, appreciate the feedback