Repository: explodinggradients/ragas
Issue: Concerning the WARNING in faithfulness
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 
[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When executing the faithfulness metric with Japanese data, I receive the following warning message:
`WARNING:ragas.metrics._faithfulness:No statements were generated from the answer.`

Ragas version:0.1.20
Python version:3.10.12

**Code to Reproduce**
`from datasets import Dataset
from ragas.metrics import faithfulness
from ragas import evaluate

data_samples = {
    'question': ['最初のスーパーボウルはいつですか？', '最も多くのスーパーボウルを制したチームはどこですか？'],
    'answer': ['最初のスーパーボウルは1967年1月15日に開催されました', '最も多くのスーパーボウルを制したのはニューイングランド・ペイトリオッツです'],
    'contexts': [
        ['最初のAFL–NFLワールドチャンピオンシップゲームは、1967年1月15日にロサンゼルスのロサンゼルス・メモリアル・コロシアムで開催されたアメリカンフットボールの試合でした。'],
        ['グリーンベイ・パッカーズはウィスコンシン州グリーンベイに本拠地を置く...', 'パッカーズは...フットボールカンファレンスに所属しています。'],
    ],
}

dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset, metrics=[faithfulness])
print(score)
score.to_pandas()
`

**Error trace**
`WARNING:ragas.metrics._faithfulness:No statements were generated from the answer.`

**Expected behavior**
I expected the faithfulness metric to compute scores without warnings when using Japanese data.

**Additional context**
It appears that the faithfulness metric may not handle Japanese text correctly. Any guidance on how to resolve this issue would be greatly appreciated.

<!-- Greeting -->
Hey @TakutoIyanagi-littletree! I'm here to help you with bug fixes, answer your questions, and guide you on becoming a contributor. Let's squash those bugs together! 🤖

<!-- Answer -->
To resolve the warning "No statements were generated from the answer" when using the faithfulness metric with Japanese data in Ragas version 0.1.20, you need to ensure that the sentence segmenter is correctly set up for the Japanese language. Here is how you can adapt the `FaithfulnessExperimental` metric to Japanese:

```python
from ragas.experimental.metrics._faithfulness import FaithfulnessExperimental
from ragas.metrics.base import get_segmenter

faithfulness = FaithfulnessExperimental()
faithfulness.sentence_segmenter = get_segmenter(language="japanese", clean=False)
faithfulness.llm = your_language_model  # Set your language model here

# Use the adapted faithfulness metric in your evaluation
score = evaluate(dataset, metrics=[faithfulness])
print(score)
score.to_pandas()
```

By setting the sentence segmenter to handle Japanese, the metric will be able to correctly segment Japanese sentences, which should help in generating the necessary statements and avoid the warning <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_faithfulness.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Thanks for the reply.
However, after running it I got an error again.
Is there a wrong way to specify the language model?

**Describe the bug**
`AttributeError: 'dict' object has no attribute 'get_sample_type'`

**Code to Reproduce**
```
import openai
from ragas.experimental.metrics._faithfulness import FaithfulnessExperimental
from ragas.metrics.base import get_segmenter
from ragas.llms import llm_factory
from ragas.metrics import (
    faithfulness,
)

LLM_NAME = "gpt-4o-mini"
ragas_llm = llm_factory(model=LLM_NAME)

dataset = {
    'question': ['最初のスーパーボウルはいつですか？', '最も多くのスーパーボウルを制したチームはどこですか？'],
    'answer': ['最初のスーパーボウルは1967年1月15日に開催されました', '最も多くのスーパーボウルを制したのはニューイングランド・ペイトリオッツです'],
    'contexts': [['最初のAFL–NFLワールドチャンピオンシップゲームは、1967年1月15日にロサンゼルスのロサンゼルス・メモリアル・コロシアムで開催されたアメリカンフットボールの試合でした。'],
    ['グリーンベイ・パッカーズ...ウィスコンシン州グリーンベイ。', 'パッカーズは...フットボールカンファレンスに所属しています。']],
}

faithfulness = FaithfulnessExperimental()
faithfulness.sentence_segmenter = get_segmenter(language="japanese", clean=False)
faithfulness.llm = ragas_llm 

score = evaluate(dataset, metrics=[faithfulness])
print(score)
score.to_pandas()
```

<!-- Answer -->
To resolve the `AttributeError: 'dict' object has no attribute 'get_sample_type'` when specifying the language model in the `FaithfulnessExperimental` metric, you need to ensure that the dataset is correctly formatted as a `Dataset` object from the `datasets` library, not as a plain dictionary.

Here is the corrected code:

```python
import openai
from ragas.experimental.metrics._faithfulness import FaithfulnessExperimental
from ragas.metrics.base import get_segmenter
from ragas.llms import llm_factory
from ragas import evaluate
from datasets import Dataset

# Ragasの評価用LLMの設定
LLM_NAME = "gpt-4o-mini"
ragas_llm = llm_factory(model=LLM_NAME)

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts': [
        ['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
        ['The Green Bay Packers...Green Bay, Wisconsin.', 'The Packers compete...Football Conference']
    ],
}

# Convert the dictionary to a Dataset object
dataset = Dataset.from_dict(data_samples)

faithfulness = FaithfulnessExperimental()
faithfulness.sentence_segmenter = get_segmenter(language="japanese", clean=False)
faithfulness.llm = ragas_llm  # Set your language model here

# Use the adapted faithfulness metric in your evaluation
score = evaluate(dataset, metrics=[faithfulness])
print(score)
score.to_pandas()
```

This should resolve the `AttributeError` and allow the `FaithfulnessExperimental` metric to work correctly with your Japanese data <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/experimental/metrics/_faithfulness.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Thank you. I was able to solve it.

```
import openai
from ragas.experimental.metrics._faithfulness import FaithfulnessExperimental
from ragas.metrics.base import get_segmenter
from ragas.llms import llm_factory
from ragas import evaluate
from datasets import Dataset

LLM_NAME = "gpt-4o-mini"
ragas_llm = llm_factory(model=LLM_NAME)

data_samples = {
    'question': ['最初のスーパーボウルはいつですか？', '最も多くのスーパーボウルを制したチームはどこですか？'],
    'answer': ['最初のスーパーボウルは1967年1月15日に開催されました', '最も多くのスーパーボウルを制したのはニューイングランド・ペイトリオッツです'],
    'contexts': [['最初のAFL–NFLワールドチャンピオンシップゲームは、1967年1月15日にロサンゼルスのロサンゼルス・メモリアル・コロシアムで開催されたアメリカンフットボールの試合でした。'],
    ['グリーンベイ・パッカーズ...ウィスコンシン州グリーンベイ。', 'パッカーズは...フットボールカンファレンスに所属しています。']],
}

dataset = Dataset.from_dict(data_samples)

faithfulness = FaithfulnessExperimental()
faithfulness.sentence_segmenter = get_segmenter(language="japanese", clean=False)
faithfulness.llm = ragas_llm  # Set your language model here

# Use the adapted faithfulness metric in your evaluation
score = evaluate(dataset, metrics=[faithfulness])
print(score)
score.to_pandas()
```

great that you got this solved 🙂 !

It seems the issue was answered, closing this now.