Repository: explodinggradients/ragas
Issue: [R-243] docs: Add documentation on the best approach to define custom metrics.
Root cause: Documentation Deficiency

[X] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

I was exploring the possibility to create custom metrics.
It seems that it is possible, by subclassing either [Metric](https://github.com/explodinggradients/ragas/blob/e9418237d0dd06f640b0f6bbae7a5a3f9c1029ea/src/ragas/metrics/base.py#L52), [MetricWithLLM](https://github.com/explodinggradients/ragas/blob/e9418237d0dd06f640b0f6bbae7a5a3f9c1029ea/src/ragas/metrics/base.py#L127) or [MetricWithEmbeddings](https://github.com/explodinggradients/ragas/blob/e9418237d0dd06f640b0f6bbae7a5a3f9c1029ea/src/ragas/metrics/base.py#L144).

For example I created a dummy example computing the answer length:
Note that I don't need this specific metric, I used it to have a simple example.

```python
import typing as t
from datasets import Dataset
from ragas import evaluate
from ragas.metrics.base import Metric, EvaluationMode
from langchain_core.callbacks import Callbacks
from ragas.run_config import RunConfig

class AnswerLength(Metric):
    """Simple example of a custom metric. Returning the answer length."""
    name: str = "answer_length"
    evaluation_mode: EvaluationMode = EvaluationMode.qa
    
    async def _ascore(
        self: t.Self, row: t.Dict, callbacks: Callbacks, is_async: bool
    ) -> float:
        return len(row["answer"])
    def init(self, run_config: RunConfig):
        """do nothing"""

answer_length = AnswerLength()

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
}

dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset, metrics=[answer_length])
score.to_pandas()
```

My questions are:

- Do you recommend creating custom metrics with ragas? Or is it preferable to exclusively rely on the [pre-existing metrics offered by Ragas](https://docs.ragas.io/en/latest/concepts/metrics/index.html)?
- If yes, is the approach described above correct?

I am not sure if the possibility of creating custom metrics is an intended feature or not. I want to make sure that my custom metric implementations do not break when ragas evolves. I am interested in knowing the vision about supporting and documenting the possibility of having custom metrics in the future.

<sub>[R-243](https://linear.app/exploding-gradients/issue/R-243/docs-add-documentation-on-the-best-approach-to-define-custom-metrics)</sub>

hey @ouphi thanks for brining this up.

you current take is correct, we'll be adding more documentation for making this easier too down the line but gist is
implement `_ascore` and `init` and rest should be fine.


Out of curiosity, what were the models you were planning to add?

How can I add a custom prompt and then evaluate ?

you could try out [Aspect Critique | Ragas](https://docs.ragas.io/en/stable/concepts/metrics/critique.html)

the prompts there are very extendable but let me know if that was what you had in mind