zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Language of the Testset
Root cause: Algorithmic Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Why the testset language is in english, also the references are in english

**Code Examples**
from ragas.testset.synthesizers.single_hop.specific import (
    SingleHopSpecificQuerySynthesizer,
)

distribution = [
    (SingleHopSpecificQuerySynthesizer(llm=generator_llm), 1.0),
]

for query, _ in distribution:
    prompts = await query.adapt_prompts("spanish", llm=generator_llm)
    query.set_prompts(**prompts)

**Additional context**
In theory this should solve my problem, but it did not, in the https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_language_adaptation/#generate you use this cell to transform the queries into spanish, but you obtain a french sentence:
Query: Quelles sont les caractéristiques du Bronx en tant que borough de New York?
Reference: Le Bronx est l'un des cinq arrondissements de New York, qui est la plus grande ville des États-Unis. Bien que le contexte ne fournisse pas de détails spécifiques sur le Bronx, il mentionne que New York est une ville cosmopolite avec de nombreux quartiers ethniques, ce qui pourrait inclure des caractéristiques culturelles variées présentes dans le Bronx.


I have the same issue! I solve it by not relying on language adaptation from ragas but by having a separate step of translating the generated questions and answers into the necessary language.
Repository: explodinggradients/ragas
Issue: why await no support from demo
Root cause: Documentation Deficiency

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
run demo in https://docs.ragas.io/en/latest/getstarted/evals/#evaluating-using-a-llm-based-metric

Ragas version:
Python version: 3.10
Ragas Version: ragas       0.2.13

**Code to Reproduce**
same as demo

**Error trace**
File "D:\PyInvo\ragas_demo.py", line 271
    await metric.single_turn_ascore(SingleTurnSample(**test_data))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: 'await' outside function

And in my pycharm, also have error:

![Image](https://github.com/user-attachments/assets/f895e416-a9fd-43c6-a241-0b285cd6fa1f)

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hi @landhu,

I tried reproducing the error with the same Python and Ragas versions, but I wasn't able to replicate the issue. Could you please share the full error trace or any additional context that might help identify the root cause?

The code needs to be in inside a`async def` function.  See this pull request for an example https://github.com/explodinggradients/ragas/pull/1983


thanks again @yesidc for raising this - we are closing this now
Repository: explodinggradients/ragas
Issue: Getting scores with llamaindex using custom model as bedrock or gemini
Root cause: Unimplemented Feature Gap

**Describe the Feature**
I would like to do evaluation using Bedrock or Gemini with llamaindex and not just langchain (https://docs.ragas.io/en/stable/howtos/customizations/customize_models/).

i have already tried to use to use `LlamaIndexLLMWrapper` and `LlamaIndexEmbeddingsWrapper`, but it does not work. Indeed I got error or a big loop with:
   
    from llama_index.llms.bedrock_converse import BedrockConverse
    from llama_index.embeddings.bedrock import BedrockEmbedding
    from ragas.llms import LlamaIndexLLMWrapper
    from ragas.embeddings import LlamaIndexEmbeddingsWrapper
    from ragas.metrics import ResponseRelevancy
   
    bedrock_model = BedrockConverse(...)
    bedrock_embeddings = BedrockEmbedding(...)
    response_relevancy_scorer = ResponseRelevancy(
        llm=LangchainLLMWrapper(bedrock_model),
        embeddings=LangchainEmbeddingsWrapper(bedrock_embeddings)
    )
    out = await response_relevancy_scorer.single_turn_ascore(sample)

May you, please, consider to improve the code with such features?

**Why is the feature important for you?**
I normally use llamaindex and not langchain.

@jjmachan may you or a colleague of yours help me, please?

@shahules786 maybe you can help me?

Any update on this ticket?
Repository: explodinggradients/ragas
Issue: Spanish implementation for ragas 0.1.21 with method adapt() that actually works well
Root cause: Validation Gap

Here's a well-structured issue report based on your problem:

---

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

### **Describe the bug**
I am experiencing an issue while generating a test set in Spanish using `ragas` version 0.1.21. The error occurs when calling `generate_with_langchain_docs()`, and it appears to be related to JSON validation. The error message suggests that the output is not in a valid JSON format.

### **Ragas version**
0.1.21  

### **Python version**
(Provide your Python version, e.g., `Python 3.10.6`)

### **Code to Reproduce**
```python
def generateTestset(
        chunks: list[Document],
        embedding_function=None,
        llm_type: str = "ollama",
        llm_model: str = "mistral"
):

    generator_llm = llm_model
    critic_llm = llm_model
    embeddings = embedding_function

    if llm_type == "ollama":
        generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)
    elif llm_type == "openai":
        generator = TestsetGenerator.with_openai()

    # adapt to language
    language = "spanish"

    generator.adapt("spanish", evolutions=[simple, reasoning, multi_context])
    generator.save(evolutions=[simple, reasoning, multi_context])

    testset = generator.generate_with_langchain_docs(
        chunks, 
        test_size=10, 
        distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}
    )
    final_testset = testset.to_pandas()
```

### **Error trace**
```
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Prompt
root
output in example 1 is not in valid json format: Expecting value: line 1 column 1 (char 0) (type=value_error)
```

### **Expected behavior**
I expected `generate_with_langchain_docs()` to return a valid test set without JSON-related errors.

### **Additional context**
- I am using `ollama` with `mistral` as the model.
- The error suggests that the generated output might not be properly formatted JSON.
- I followed the official documentation for `ragas` version 0.1.21.

Any guidance on resolving this issue would be greatly appreciated!
Repository: explodinggradients/ragas
Issue: SimpleCriteriaScore does not set `definition` correctly
Root cause: Validation Gap

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I think there's a bug in the `SimpleCriteriaScore` class, where it does not set `self.definition` correctly and the setter does not end up getting called. Specifically, I think this line:

`self._definition = definition` 

should be this instead:

`self.definition = definition` 

Here is an example from the docs, where I've edited the `reference` to be the same as the `response`. The expected result should be 5, but 10 is returned.

![Image](https://github.com/user-attachments/assets/eacd31af-60d3-4c32-ab9e-7604f164be58)

I then manually set the definition and get the expected result

![Image](https://github.com/user-attachments/assets/a49fa14b-9dd6-4c42-a505-06426c07552c)

Ragas version: 0.2.13
Python version: 3.10.16

**Code to Reproduce**
Attached the code to reproduce above.

**Error trace**

**Expected behavior**
If the `reference` and `response` are the same string, a `description` of `"Score 0 to 5 by similarity"` should return 5, and not 10.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: LangchainLLMWrapper sets a default temperature even with models that don't support it (ex o3-mini)
Root cause: Configuration Error

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When I try to create an LLM instance using LangchainLLMWrapper and specify a model that doesn't support the "temperature" parameter, I cannot get any responses from the API. 

Ragas version: 0.2.13
Python version: 3.10.16

**Code to Reproduce**

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model_name="o3-mini"))
sample = SingleTurnSample(
    response="The Eiffel Tower is located in Paris.",
    reference="The Eiffel Tower is located in Paris. I has a height of 1000ft."
)

scorer = FactualCorrectness(llm = evaluator_llm)
await scorer.single_turn_ascore(sample)

**Error trace**
openai.BadRequestError: Error code: 400 - {'error': {'message': "Unsupported parameter: 'temperature' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'temperature', 'code': 'unsupported_parameter'}}

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I guess the solution provided won't work if using AzureChatOpenAI(`from langchain_openai.chat_models import AzureChatOpenAI`). 

1. if we pass `temperature ` as `None` -> AzureChatOpenAI throws error ('temperature' must be valid float)
2. if we remove `temperature ` attributes from `LangchainLLMWrapper ` object  -> AzureChatOpenAI throws error (AttributeError('AzureChatOpenAI' object has no attribute 'temperature'))

Need to get a Generic solution to work well with all existing clients for reasoning models(o1, o3-mini etc.) 

Quick fix - 
_.tesla_rasa_eval/lib/python3.11/site-packages/ragas/llms/base.py_
change the 'get_temperature' function - 
```
    def get_temperature(self, n: int) -> float:
        """Return the temperature to use for completion based on n."""
        return 0.3 if n > 1 else n
```
* change ->  replaced `1e-8` to `n`

and pass temperature = 0 for reasoning models(o1, o3-mini etc.)

I am encountering similar issues referring to these 2:
1. [https://github.com/Azure/azure-sdk-for-python/issues/39938]()
2. [https://github.com/langchain-ai/langchain/issues/30126]()

Maybe we could just add a conditional statement here specifically for like you suggested @mukul1609 for o1 and o3 family. There is also an article experimenting reasoning models for evaluation pipeline: https://www.reddit.com/r/Rag/comments/1ixs5wx/we_evaluated_if_reasoning_models_like_o3mini_can/?rdt=60212.
Repository: explodinggradients/ragas
Issue: Why  AnswerCorrectness & SemanticSimilarity need reference in dataset?
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Why  AnswerCorrectness & SemanticSimilarity need reference in dataset?

From answer_correctness & semantic_similarity descript:
`Scores the semantic similarity of ground truth with generated answer.
    cross encoder score is used to quantify semantic similarity.`

`Measures answer correctness compared to ground truth as a combination of
    factuality and semantic similarity.`

Ragas version:python3.10
Python version:0.2.14

**Code to Reproduce**
`dataset = []
dataset.append(
        {
            "user_input":'how many devices added yesterday?',
           # "retrieved_contexts":["Albert Einstein proposed the theory of relativity, which transformed our understanding of time, space, and gravity."],
            "response": "This is a topic I can't help with.\n        \n    You can ask questions like",
            #"reference":"Albert Einstein proposed the theory of relativity, which transformed our understanding of time, space, and gravity."
        }
    )

result = evaluate(dataset=evaluation_dataset, metrics=[SemanticSimilarity()]）
`

**Error trace**
`   result = evaluate(dataset=evaluation_dataset, metrics=[SemanticSimilarity()], llm=evaluator_llm, embeddings=silcom_embedding)
  File "D:\Program Files\Python310\lib\site-packages\ragas\_analytics.py", line 227, in wrapper
    result = func(*args, **kwargs)
  File "D:\Program Files\Python310\lib\site-packages\ragas\evaluation.py", line 181, in evaluate
    validate_required_columns(dataset, metrics)
  File "D:\Program Files\Python310\lib\site-packages\ragas\validation.py", line 63, in validate_required_columns
    raise ValueError(
ValueError: The metric [semantic_similarity] that is used requires the following additional columns ['reference'] to be present in the dataset.`


**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
The previous version was not required

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: What to include in context precision/recall for RAG LLM evaluation
Root cause: Documentation Deficiency

Hi, I am doing evaluation for my RAG LLM application using ragas. I have the prompt `instruction` to describe some rules, the `retrieved content` from my retriever, and `chat history` together for the LLM to do question answering. When I do the evaluation for `context precision` or `context recall` in the [documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/), should I use the `instruction+retrieved content+chat history` or just `retrieved content`? Thanks!

**Code Examples**
For example, for the parameter `retrieved_contexts` in the example codes below, should I use `instruction+retrieved content+chat history` or just `retrieved content`?
```
from ragas import SingleTurnSample
from ragas.metrics import LLMContextPrecisionWithoutReference

context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm)

sample = SingleTurnSample(
    user_input="Where is the Eiffel Tower located?",
    response="The Eiffel Tower is located in Paris.",
    retrieved_contexts=["The Eiffel Tower is located in Paris."], 
)


await context_precision.single_turn_ascore(sample)
```



Hi @hliuci,

Context precision measures whether the retrieved chunks were actually helpful in generating the response. In other words, it evaluates the impact of the retrieved content on the final output. If your RAG architecture uses instruction, chat history, and the retrieved context together during generation, then your evaluation should consider the combined context. However, if you’re solely evaluating the effectiveness of the retrieval component, then you’d use just the retrieved content.

Similarly, context recall follows the same logic. It assesses whether all the relevant retrieved information was captured. So, your evaluation approach should align with how your system integrates these components during generation.

@sahusiddharth Got it, thank you for your help!
Repository: explodinggradients/ragas
Issue: Can code similar to app.ragas.io dashboard be open sourced?
Root cause: Unimplemented Feature Gap


**Your Question**
May I ask if code similar to the app.ragas.io dashboard can be open sourced? 
Can we deploy the dashboard locally? 
How to deploy a local dashboard and do you have any relevant guidance?


Not planned at the moment.


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Enforcing json format
Root cause: Validation Gap

- [x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**My Question**
I'm getting exception 
```
 File "/my/path/to/ragas/prompt/pydantic_prompt.py", line 423, in parse_output_string
    raise RagasOutputParserException()
ragas.exceptions.RagasOutputParserException: The output parser failed to parse the output including retries.
```
And I'm pretty sure it's happening because the LLM is getting cut off when it's generating the answer in the JSON format. Right now I have the max tokens for my LLM to generate set to 1024. I can increase the max tokens to help avoid that problem. But I'm wondering what happens if the LLM has a longer generation and still runs into that max token count before it finishes the JSON structure. I know the prompt says to follow the JSON schema, so there's an effort to make sure that doesn't happen, but could there be a more concrete way for RAGAS to make sure that JSON format is finished? Or is that just too complicated?

I might also decrease my temperature and tweak my top_p to decrease the likelihood of that over generation. Maybe there could be a reminder/guidance on settings like that somewhere in the documentation. That might give people some trouble shooting ideas and a reminder that tweaking those settings might give them better performance.

**Code Examples**
After line 201 in pydantic_prompt.py I'm printing out the output string to see what the LLM generates. That's where I saw that the JSON schema wasn't getting finished.
Here's a shortened example of what it looks like:
```
 {
    "statements": [
        {
            "statement": "The crash of Yeti Airlines Flight 691 in Pokhara, Nepal was caused by a technical issue with the plane.",
            "reason": "The context explicitly states that the crash was caused by the aircraft's propellers being feathered, which is a technical issue.",
            "verdict": 1
        },
        {
            "statement": "The weather was clear at the time of the crash.",
            "reason": "There is no"
```
Repository: explodinggradients/ragas
Issue: Need a demo about deepseek.
GitHub labels: question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I guess that I should use LangchainLLMWrapper to add deepseek api, but still need an official demo to make up my weak coding ability.
Help, please!

closing this since it's a duplicate of https://github.com/explodinggradients/ragas/issues/1895

but will get something working 🙂
Repository: explodinggradients/ragas
Issue: FactualCorrectness false negative calculation
Root cause: Algorithmic Error



**Describe the bug**

The calculation of answer_correctness metric (0.75*factual_correctness+0.25*semantic_similarity), does not use the same implementation of factual_correctness as when FactualCorrectness is computed directly. This discrepancy leads to inconsistent results in which answer_correcness can be lower than both, factual_correctness and semantic_similarity.


**Code to Reproduce**
`score = evaluate(dataset, metrics=[answer_correctness, answer_similarity, FactualCorrectness()], llm=self.azure_model, embeddings=self.azure_embeddings, run_config=self.run_config)`
**Error trace**
No error given.

**Expected behavior**
I would expect to factual correctness is the same inside answer_correctness and when calculated with FactualCorrectness directly.




How is `fp = fn`? `fp` is sum of `~reference_response` and `fn` is sum of `~response_reference`. They are sum of negation of different array. 

> How is `fp = fn`? `fp` is sum of `~reference_response` and `fn` is sum of `~response_reference`. They are sum of negation of different array.

Yes, that is true. I edit the issue. Anyway, in some cases I get different valué from FactualCorrecness and from the inside metric fractal correctness calculated inside answer_correctness.

I confirm the same situation. The reason seems to be that the factual_correctness used in answer_correctness has a different prompt compared to the one used in factual_correctness alone. It would be really nice to have also this version of the factual correctness available.
Repository: explodinggradients/ragas
Issue: Clarification of answer relevancy

Is answer relevancy supposed to be used to compare "only prompt (only the question) and answers" or "prompt + retrieved context (full input to an LLM) and answers?

@ogencoglu that's interesting. Although the initial idea was to enable compare <question, answer>  I am now thinking it should also consider `context` as in many instances that can be helpful in recreating the question back. I'm working on some improvements in answer_relevancy which I can include too. 

If you like to contribute to this, please let me know. 
To get an in-depth idea of the paradigm refer https://docs.ragas.io/en/latest/concepts/metrics/answer_relevance.html
Repository: explodinggradients/ragas
Issue: Initializing FactualCorrectness metric has side effects on other metrics
Root cause: Algorithmic Error

[x ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Initializing a Factual Correctness metrics has side effects on the examples of other metrics: for example, a Rubrics Score initialized afterwards has `ClaimDecompositionInput` in its examples.

Ragas version: 0.2.13
Python version: 3.12.4

**Code to Reproduce**
```
from ragas.metrics import FactualCorrectness, RubricsScore

rubrics = RubricsScore()
print(f"Rubrics Examples: \n {rubrics.single_turn_scoring_prompt.examples}")
# Rubrics Examples: 
#  []

factual_correctness = FactualCorrectness()
print(f"Factual Correctness Examples: \n {factual_correctness.claim_decomposition_prompt.examples}")
# Factual Correctness Examples: 
#  [(ClaimDecompositionInput(response='Charles Babbage was a French mathematician, philosopher, and food critic.'), ...

rubrics = RubricsScore()
print(f"Rubrics Examples: \n {rubrics.single_turn_scoring_prompt.examples}")
# Rubrics Examples: 
#  [(ClaimDecompositionInput(response='Charles Babbage was a French mathematician, philosopher, and food critic.'), ...
```

**Expected behavior**
The examples in the rubrics prompt are always empty.

**Additional context**
I suspect that the error is because in the `__post_init__` in `FactualCorrectness`  the examples of `PydanticPrompt` are modified in place using `extend`.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->