Repository: explodinggradients/ragas
Issue: Language Not Supported
Root cause: Unimplemented Feature Gap

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I want to use RAGAS as my RAG evaluation framework, but I cannot find supported language other than `RAGAS_SUPPORTED_LANGUAGE_CODES` in this line https://github.com/explodinggradients/ragas/blob/v0.2.8/src/ragas/metrics/base.py#L707

Which after tracing the code, it come from here:
* https://github.com/explodinggradients/ragas/blob/v0.2.8/src/ragas/utils.py#L14-L22
* https://github.com/nipunsadvilkar/pySBD/blob/v0.3.4/pysbd/languages.py#L26-L50

The pySBD last commit is 3 years ago, which I also have question why prefer use that library? 

My ultimate question is: How to add language support which don't supported by pySBD (and not supported by RAGAS)?
I see that the list is too limited, not a single language from Southeast Asia is supported. 

**Additional context**
If I can extend the language support which don't "natively" supported by RAGAS, where I can find the example to create an Adapter Language?

Thank you!


hey @yusufsyaifudin thanks for sharing this - which language are you are you planning to use? other that pySBD which other tools do you work with that have support which you mentioned?

@shahules786 should be able to provide you better information too

Thanks @jjmachan for your reply

> which language are you are you planning to use?

I am work with Bahasa Indonesia and have tried to run the RAGAs with default settings (I assume it English) in three proprietary model: claude-3-haiku-20240307, claude-3-5-haiku-20241022 and claude-3-5-sonnet-20241022.

The claude-3-haiku-20240307 always return the faithfulness score to 1.0 (I only test with two data) which I can confirm that it should be near to 0. The other two models return 0.0, at this point I starting to think that maybe it just because the Haiku old version is "bad" at reasoning.

But, I think by using the same language in the prompt for testing (Bahasa Indonesia in my case), probably it would be have better reasoning.

> other that pySBD which other tools do you work with that have support which you mentioned?

Actually I don't know any alternative,  maybe we still in the state that none package supports all language for sentence boundary extractor.

But, imho, if we can create some "abstraction" regarding the sentence segmentation and prompt, we can achieve multi-language support easily? Probably using nltk, or other package.

For example, in my project I use https://github.com/yusufsyaifudin/id-sentence-segmenter which forked version from the https://yudanta.github.io/posts/indonesian-simple-sentence-segmentation/ (which is part of his theses work https://etd.repository.ugm.ac.id/penelitian/detail/103174).

If I want to extend or create abstraction for this, which file and line of code as the the starting point that I can read? Maybe @shahules786 can help me to point this out.

:bow:

Hey @yusufsyaifudin thanks for reporting this. There have been some issues regarding Pysbd. Let me revisit it this week and figure out an alternative solution. We use Pysdb for segmenting text into sentences that is used in few metrics like faithfulness. 

The simplest and most straightforward fix would be to remove the need for sentence segmentation altogether and let the LLM handle it when creating statements. For example, the current pipeline looks like this:

**text → split into sentences (Pysbd) → use the LLM on each sentence to create one or more statements.**

Alternatively, it could be simplified to:

**text → use the LLM to create statements directly.**

I think, especially with the [Align your evaluators automatically](https://github.com/explodinggradients/ragas/issues/docs.ragas.io/en/latest/howtos/customizations/metrics/train_your_own_metric/) feature, this approach can work well, as the number and granularity of statements created from the text can be fine-tuned by providing feedback to the evaluators.