Repository: explodinggradients/ragas
Issue: Request for multi-language support
Root cause: Unimplemented Feature Gap

Hi,

This request might be way to early in the development process, but I notice some metrics like answer_similarity and faithfulness (applied to answer and ground_truth like in answer_correctness) have way higher scores than I would expect. Roughly around 0.7 +/- 0.2. 
When translating my answer and ground_truth to English and recalculating the metrics they are way lower, around 0.25 +/- 0.1 (more in line with expectations)

Would it be possible for classes like faithfulness to allow users to set their own LONG_FORM_ANSWER_PROMPT  and NLI_STATEMENTS_MESSAGE  ? That way, scores might better reflect reality.

For the similarity-score I'm not so sure yet how that could be improved since I don't see much language specific inputs there. Or could that be handled by passing your own embedding? If you have any ideas, please let me know

Regards, Koen

Maybe this shouldn't be called 'multi language support', but more like 'passing own metric prompts'.
In practice, I would just translate your current prompts to my own language and insert them as is.

Looking into this a bit more, it maybe doesn't have to be much work. For answer_relevance, I can already access QUESTION_GEN and give it a different value due to the various names used in the python file.

I'll try to explain.
The file (or module) is called answer_relevance.py
In this file is a class AnswerRelevancy
and an instance answer_relevancy

Because each is named slightly differently, all types (module, class and instance) are exposed. Allowing me to:
```
import ragas
ragas.metrics.answer_relevance.QUESTION_GEN = <set humanprompttemplate in own language>
```

When looking at faithfulness,
The file (or module) is called faithfulness.py
the class is Faithfulness
and the instance is faithfulness

Because the file and instance have the same name, only the latter (instance) is exposed.

If you could introduce slight naming variations for all metrics in filename, classname and instancename this would already solve the problem (I think)


Hope it helps, Koen

Hey @kpeters, seems like a very interesting experiment to me. Would be able to share some more information?
1. How many samples were used to measure this drift? 
2. What languages were you using?

Most LLMs are fine-tuned on cross-lingual tasks, so my assumption was that OpenAI should perform well on other languages too even if the prompts were written in English. 

 

> Maybe this shouldn't be called 'multi language support', but more like 'passing own metric prompts'.
In practice, I would just translate your current prompts to my own language and insert them as is.

This and the solution you suggested seem very good to me.  I think this will even help users who want to use OSS LLMs and need prompt tuning. @jjmachan 

@shahules786 Thanks for responding. Let me check internally if there are any samples I'm allowed to share, I'll get back to you on that after the weekend.

To answer your questions:
1. the sample size is currently very small, I use 5 validation questions (expected to grow in the future)
2. The language I'm working with is Dutch

In my experience (no evidence), it handles different languages quite well as long as you don't mix them within a prompt OR provide explicit instructions. e.g. 
```
"""Analyse the Dutch language provided within tripple tildes and tell me ... <whatever you want to have analysed>
~~~{input}~~~"""
```

@kpeters thanks for sharing your insights. Would you be open to hoping on a call sometime next week and discuss this further? Here is my [calendly](https://calendly.com/shahules/30min)

This will be fixed with #208 

#208  and #243 have been closed, looking forward to the release