zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: How to get Similarity metrics
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I already have golden data and I would like to get a similarity metric by comparing the output from the application with the golden sample data.


<!-- Answer -->
To get a similarity metric by comparing the output from your application with the golden sample data, you can use the `answer_correctness` metric from the `ragas` library. Here is an example of how to do this:

```python
from datasets import Dataset 
from ragas.metrics import answer_correctness
from ragas import evaluate

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset, metrics=[answer_correctness])
score.to_pandas()
```

This code will calculate the answer correctness score between the generated answers and the ground truth answers, providing a metric that ranges from 0 to 1, where a higher score indicates a better alignment.

Make sure you have your OpenAI API key set in your environment, as the `answer_correctness` metric uses OpenAI's API to compute the score:

```python
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
```

You can also explore other metrics provided by the `ragas` library, such as `faithfulness`, `context_precision`, and `answer_relevancy`, to get a comprehensive evaluation of your RAG pipeline.

To install the `ragas` library, you can use pip:

```bash
pip install ragas
```

Alternatively, you can install it from the source:

```bash
pip install git+https://github.com/explodinggradients/ragas
```

For more detailed information and examples, refer to the [Ragas documentation](https://docs.ragas.io/) <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/semantic_similarity.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/answer_correctness.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: 异步刷新accessToken报连接不上服务器
Root cause: Validation Gap

openapi_requestor.py:377 [t:140096597632832]: async requesting llm api endpoint: /chat/completions
INFO:qianfan:async requesting llm api endpoint: /chat/completions
[INFO] [09-09 10:08:13] oauth.py:229 [t:140096597632832]: trying to refresh access_token for ak `yf7AZx***`
INFO:qianfan:trying to refresh access_token for ak `yf7AZx***`
[INFO:qianfan:sucessfully refresh access_token for ak `yf7AZx***`](oauth.py:240 [t:140096597632832]: refresh access token failed with exception Cannot connect to host aip.baidubce.com:443 ssl:default [Network is unreachable])

<!-- Greeting -->
@ming1028 你好！我是[Dosu](https://dosu.dev)，一个可以帮助你解决bug、回答问题并成为贡献者的机器人。我在这里帮助你，等待人类维护者的到来。

<!-- Answer -->
The error message indicates that the asynchronous request to refresh the access token failed due to a network issue ("Network is unreachable"). Here are some key points and potential steps to address this issue:

1. **Network Connectivity**: Ensure that the network connection is stable and that there are no firewall or proxy settings blocking access to `aip.baidubce.com`.

2. **Retry Mechanism**: The `ragas` framework uses the `tenacity` library to handle retries for network-related exceptions. Ensure that the retry mechanism is properly configured in your `RunConfig` settings, including the maximum number of retries and the wait time between retries <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/run_config.py)</sup>.

3. **Logging**: The `ragas` framework logs detailed information about network failures and retries using the `tenacity` library's `after_log` function. Ensure that `log_tenacity` is enabled in your `RunConfig` to capture detailed logs for debugging <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/run_config.py)</sup>.

4. **Exception Handling**: The `ragas` framework defines specific exceptions for handling retries, such as `MaxRetriesExceeded`. Ensure that these exceptions are properly handled in your code to provide meaningful error messages and take appropriate actions <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/exceptions.py)</sup>.

5. **Configuration**: Verify that the configuration settings for network connectivity, such as timeouts and retry limits, are correctly set in your environment.

By addressing these points, you can improve the robustness of the token refresh process and handle network-related exceptions more effectively.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


异步情况下没办法使用重试
base.py:406 [t:140096597632832]: retry is not available when stream is enabled
WARNING:qianfan:retry is not available when stream is enabled
服务器使用同步方式可以请求到，也可以ping通
Repository: explodinggradients/ragas
Issue: No evaluation possible with context_relevancy/answer_relevancy/context_recall/harmfullness for Azure OpenAI
Root cause: Validation Gap

Hi guys,

I am using Azure OpenAI and the only evaluation method that is currently working is "faithfullness". All the others fail with the same error ```Exception: n=3 was passed to generate but the LLM AzureOpenAI
Params: {'deployment_name': '.........', 'model_name': 'text-davinci-003', 'temperature': 0.2, 'max_tokens': 256, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'n': 1, 'request_timeout': None, 'logit_bias': {}} does not support it. Raise an issue if you want support for {llm}.```

Ah I guess the problem is coming from here (within llms.py):

 ```
if isinstance(llm, OpenAI) or isinstance(llm, ChatOpenAI)
 ```
i have included 
 ```
if isinstance(llm, OpenAI) or isinstance(llm, ChatOpenAI) or isinstance(llm, AzureOpenAI)
 ```
to the mix. Seems to do the job. Are you guys incorporating that or should I open a merge request?


thank you for pointing the error out and fixing it. We haven't gotten access to Azure openAI yet so haven't been able to test it out. if you can make a PR after testing it, it would be really helpful and much appreciated :)

I think we can also add `AzureChatOpenAI` too as a check. 

Hey @gabriead I've added a fix for this in that  PR but like I said since we don't have access to Azure instances yet I can't run the tests at my end. I would really appreciate it if you could test it on your end if it fixes your issue.

Would really help us a lot 😄  

Hey guys, just pulled and that fixes the issue :-)
Repository: explodinggradients/ragas
Issue: local variable 'result' referenced before assignment
Root cause: Validation Gap

Issue in src/ragas/metrics/llms.py
The result variable is not initiated outside the if elif else blocks.

thanks a lot for pointing that out, Is it causing a runtime exception for you? Which Python version are you using. 

will fix this shortly

Yes I was using Google colab (python 3.7) and I got this runtime error. I was trying to use FLAN-T5 as the llm in evaluate.


issue has been fixed
Repository: explodinggradients/ragas
Issue: protobuf version incompatible with tf
Root cause: External Dependency Breakage

i am facing problems during the installation phase. It seems that the protobuf version required does not match dependencies required by tensorflow >= 2.13.0.
![image](https://github.com/explodinggradients/ragas/assets/50795192/3c09bdb1-2784-4f47-bfe6-bef5309fe957)


Hey @albiross thanks for bringing this up. I will be releasing the fix shortly. Please do test it out and see if it works.

Thanks again :)
Repository: explodinggradients/ragas
Issue: How can I calculate total tokens consumption for this evaluation process, in Azure Open AI.
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
How to calculate total tokens consumption for evaluating our RAG model? What parameter should I pass in token_usage_parser for Azure Open AI?

**Code Examples**
result_2 = evaluate(
    dataset1, metrics=metrics, llm=azure_model, embeddings=azure_embeddings, token_usage_parser =model_name
)

**Additional context**
Anything else you want to share with us? 


<!-- Answer -->
To calculate the total tokens consumption for the evaluation process in Azure Open AI, you can use the `total_tokens()` method on the `result` object. The parameter you should pass in `token_usage_parser` is `get_token_usage_for_openai`.

Here is the relevant code:

```python
from ragas import evaluate
from ragas.cost import get_token_usage_for_openai

result_2 = evaluate(
    dataset1,
    metrics=metrics,
    llm=azure_model,
    embeddings=azure_embeddings,
    token_usage_parser=get_token_usage_for_openai,
)

total_tokens = result_2.total_tokens()
```

This will give you the total tokens consumed during the evaluation process. The `get_token_usage_for_openai` function is used to parse the token usage from the LLM result, ensuring accurate token counting <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/applications/cost.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/tests/unit/test_cost.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Token Usage Structure:
- `TokenUsage.input_tokens` - Prompt tokens
- `TokenUsage.output_tokens` - Completion tokens
- `TokenUsage.model` - Model name used
- Methods: `cost()`, `__add__()` for aggregation

Reference Links:
- https://docs.ragas.io/en/latest/howtos/customizations/customize_models/#azure-
openai
- https://github.com/explodinggradients/ragas/blob/main/src/ragas/cost.py

Closing this as resolved.
Repository: explodinggradients/ragas
Issue: Quality of metrics questionable with Azure OpenAI
Root cause: Others

Hi guys,
I am evaluating the quality of the metrics in your framework for my RAG use case. So far I a am afraid to say that most prompts and responses are not delivering the expected result and are therefore not reliable. I am using the Azure OpenAI endpoint with GPT 3.5 so it is very possible that it has everything to do with that. Can anyone else confirm this observation?

I can offer a debugging session with the authors/contributors if that's helpful.

Hi @gabriead , thanks for raising this issue. We are continuously working to improve the quality of metrics. We do have some updates coming soon but would certainly benefit from your input. If you wish to do this over a call I can personally write to you over an email. 😊 

Hey @gabriead as shahul said we are continuously working to improve the quality of the metrics in ragas and really appreciate your offer to help us debug with a combined debugging session - would be super helpful for us. I've sent a mail to the mail-id in your github but if that is not the correct one do let us know your mail id or write to us at team@explodinggradients.com 

looking forward to meeting you Adrian and having a nice debugging session :)

@gabriead we have made a lot of improvements to ragas now for reproducibility so this should have improved. Please feel free to reopen if this is still an issue.

@shahules786 shall we close this?
Repository: explodinggradients/ragas
Issue: I cannot reproduce the results of the baseline because its context_relevancy is too high.
Root cause: Documentation Deficiency

I cannot reproduce the results of the fiqa baseline in this notebook:
https://github.com/explodinggradients/ragas/blob/main/experiments/baselines/fiqa/dataset-exploration-and-baseline.ipynb

At the end of this notebook, is shows the score:
{'NLI_score': 0.8655555555555556, 'answer_relevancy': 0.8737666666666667, 'context_ relevancy': 0.8181444444444443, 'ragas_score': 0.8517704492684051}

But when I test, the score I get is:
```json
{
      "context_ relevancy": 0.10368744449698047,
      "faithfulness": 1.0,
      "answer_relevancy": 0.9286177818722253,
      "context_recall": 0.6370370370370371,
      "harmfulness": 0.0,
      "ragas_score": 0.300955397960847,
}
```
I noticed that my context_relevancy is very low. I know that in the latest pr, the prompt used to test context_relevancy has been modified, but I am not running with this latest version. But I don’t think it’s caused by this, because this jupyter notebook seems to be running with an old version.
In addition to the context_relevancy value, there seem to be some gaps in the other values. So I'm wondering, what could be causing this?



Hi @zc277584121 ,  the results for the quickstart notebook were generated using an older version of fiqa dataset which contained very few rows.  So if you run it now on the entire dataset the scores will be different. 
If it's a point of concern we can update the notebook.  😊

I saw that it was using the first 30 rows. My results were also generated using the first 30 rows, not the entire data set.

@shahules786 , You mean, it uses an older version of the dataset and it has less than 30 rows?

In the example, it selects the first 3 rows of the dataset `fiqa_eval["baseline"].select(range(3))`, not 30. I can run and update the notebook with the newer version. Would that help?  

Thank you very much for your answer. I suggest you provide a stable and reproducible baseline so that other RAG frameworks can be compared with it. 😊


Sure @zc277584121 . Could you tell me what you're trying to achieve? Is this issue a blocker for your use case?
 I'm asking because since we are updating the code to improve metrics, there can be changes in the values of scores depicted in the quickstart. 

@shahules786 , This will not block our use. Even if the reproduced results are different from those in the notebook, I will currently use the results of my local run instead of those in the notebook. I was originally worried that there was something wrong with my reproduction process, but now it seems there is no problem.

@zc277584121 Thank you.
Repository: explodinggradients/ragas
Issue: What is the purpose of [sentence for sentence in sentences if sentence.strip().endswith(".")]
GitHub labels: question

module-metrics

**What is the purpose of this code in function __create_statements_prompt_ at file __answer_correctness.py_?**

```
sentences = [
            sentence for sentence in sentences if sentence.strip().endswith(".")
        ]
```

[ ] I checked the [_answer_correctness.py](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py#L210) and related resources and couldn't find an answer to my question.


<!-- Greeting -->
Hello @Tunnello! I'm here to help you out. I'm a bot that can assist you with bug fixes, answer your questions, and guide you on becoming a contributor. Let me know how I can assist you today!

<!-- Answer -->
The purpose of the code `sentence for sentence in sentences if sentence.strip().endswith(".")` in the function `__create_statements_prompt_` in the file `_answer_correctness.py` is to filter out sentences that do not end with a period. This ensures that only complete sentences are included in the final list of sentences <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


The current metrics no longer have this filtering logic. Please upgrade to latest version.

Closing this as obsolete.
Repository: explodinggradients/ragas
Issue: Test data generation for function calling
Root cause: Unimplemented Feature Gap

Hey,
I was wondering if you think it would be possible to create a synthetic dataset for function calling tasks?
I would like to use that dataset for a finetuning experiment.

Thanks for any guidance that you might have!

<!-- Greeting -->
Hey @alexHeu! I'm a bot designed to assist you with bugs, questions, and contributions. How can I help you today?

<!-- Answer -->
Yes, it is possible to create a synthetic dataset for function calling tasks to use in a finetuning experiment. Ragas provides tools for synthetic test data generation, which can be applied to function calling tasks. Here is a relevant code snippet for generating a synthetic test dataset:

```python
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

TEST_SIZE = 25

# generator with openai models
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)

# set question type distribution
distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}
# generate testset
testset = generator.generate_with_llamaindex_docs(
    documents, test_size=TEST_SIZE, distributions=distribution
)
test_df = testset.to_pandas()
test_df.head()
```

Ragas employs an evolutionary generation paradigm to create diverse and complex questions from provided documents. This approach ensures comprehensive coverage of various question types, including reasoning, conditioning, and multi-context, which are crucial for robust evaluation and finetuning experiments <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/arize.ipynb)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/testset_generation.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@alexHeu No, Ragas currently does NOT support generating synthetic datasets for function calling/tool use tasks. Ragas synthetic data generation is limited to RAG-style Q&A scenarios.

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Context Precision Metric Binary values
Root cause: Documentation Deficiency

I have noticed that if you look at the evaluated data set, the metric is determined individually for each data point of the dataset. The output of the console is then the mean value of all calculated metrics.
With Context Precision, I have noticed that the individual values (values per data point) are always binary, i.e. 0 or 1. Is this normal? Is there only this range of values for the individual data points, or have I made a mistake?

The whole thing can be reproduced if you use the small test set from the RAGAS documentation and save the generated dataframe as a CSV.


<!-- Answer -->
Yes, it is normal for the individual values of the Context Precision metric to always be binary (0 or 1) for each data point. This is confirmed by the `ContextPrecisionVerification` class, which defines the `verdict` field as a binary (0/1) value:

```python
class ContextPrecisionVerification(BaseModel):
    """Answer for the verification task whether the context was useful."""

    reason: str = Field(..., description="Reason for verification")
    verdict: int = Field(..., description="Binary (0/1) verdict of verification")
```

Additionally, the prompt for the `CONTEXT_PRECISION` metric instructs to give a verdict as "1" if useful and "0" if not:

```python
CONTEXT_PRECISION = Prompt(
    name="context_precision",
    instruction="""Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output.""",
    ...
)
```

The Context Precision metric evaluates whether all of the ground-truth relevant items present in the `contexts` are ranked higher or not. This metric is computed using the `question`, `ground_truth`, and the `contexts`, with values ranging between 0 and 1, where higher scores indicate better precision. The relevance indicator at each rank, denoted as $v_k$, is binary (0 or 1), confirming that individual values are indeed binary <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_precision.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/context_precision.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


This is expected behavior, not a bug. Context Precision uses binary verdicts (0/1) for individual context evaluations, but the final per-row score you see is a continuous value (0.0-1.0) calculated using Average Precision, not the raw binary verdicts.



It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Why is Context Relevancy removed from official documentation?
GitHub labels: question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Why is Context Relevancy removed from official documentation?

I can see it in the older version of doc.
https://docs.ragas.io/en/v0.1.0/concepts/metrics/context_relevancy.html

But not in the latest doc. 
https://docs.ragas.io/en/stable/concepts/metrics/context_relevancy.html

Why is that?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


<!-- Greeting -->
Hello @linzzzzzz! I'm here to help you with any bugs, questions, or contributions you have. Let's dive into the issue you're experiencing!

<!-- Answer -->
The "Context Relevancy" metric was removed from the latest version of the Ragas documentation because it has been deprecated. The `ContextPrecision` class, which was previously used for context relevancy, is now marked as deprecated with the following decorator:

```python
@deprecated(
    since="0.2", removal="0.3", alternative="LLMContextPrecisionWithReference"
)
```

This indicates that the `ContextPrecision` metric is deprecated since version 0.2 and will be removed in version 0.3, with `LLMContextPrecisionWithReference` suggested as the alternative <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_precision.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


What you said doesn't tell the "why".


<!-- Answer -->
The specific reasons for the removal of the "Context Relevancy" metric from the documentation are not explicitly provided in the available resources. However, the deprecation of the `ContextPrecision` metric, which was used for context relevancy, suggests a shift towards more advanced and accurate metrics like `LLMContextPrecisionWithReference`. This new metric offers improvements such as the inclusion of a reference answer for more accurate evaluation, prompt customization, and reproducibility control, making it a more robust and reliable choice for evaluating context precision <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_precision.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@linzzzzzz context relevancy was depreciated from v0.1 in favor of context utilisation. https://docs.ragas.io/en/stable/concepts/metrics/context_utilization.html