zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Context Precision Example
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I was reviewing the [context precision](https://docs.ragas.io/en/stable/concepts/metrics/context_precision.html) example on the website and noticed that `answer` is being provided as an input even though it's not being used actively as far as we can tell in the context precision metric. We even see a couple spots where the answer key is being overwritten with the `ground_truth` key. 

My question is, for the examples are they supposed to be minimal and required arguments or just a data dump of everything that can be provided?

I know that using answer is going to be deprecated soon and according to https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_precision.py#L117 it will fall back, I was just curious if it having both on the example is intended or not.

EDIT: Also if the answer version is being deprecated soon, is the plan to switch all references from `answer` to `ground_truth`?

<!-- Greeting -->
Hello @MatthewSH! I'm a bot here to help you with any issues or questions you have. I can assist with bug fixes, answer inquiries, and guide you on becoming a contributor. How can I assist you today?

<!-- Answer -->
The examples for context precision are supposed to be minimal and required arguments, not a data dump of everything that can be provided. The provided examples focus on the necessary inputs: `question`, `context`, and `answer`, and demonstrate how the context was useful in arriving at the given answer <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/context_precision.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_precision.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Checkout from latest [docs on ContextPrecision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Prompt for Question Creation or Am I missing something
Root cause: Documentation Deficiency

**Question:**
Where is the prompt generating the actual question.   I am using Claude Sonnet for generating the question test data set and it seems like the prompt can use some tweaking.   The issue currently is the question is generated with a lot of extra data that is not needed.  See the examples for what extra data is coming in.  

Do I need to manually go in and tweak the prompt (if so where)?
or 
Is there something i am missing or a set to resolve this?

**Example**
Questions will include things like 

Based on the given context and keyphrase, here is a question that can be answered from the information provided:

Rewritten multi-hop reasoning question: 

Based on the given context and keyphrase 'XXXX', a relevant question could be:


Thank you for any help you may provide

I have also tried on sonnet 3.5 and opus and have the same results

This issue has been resolved in recent versions of ragas through an architectural refactoring of the question generation system.

The current implementation uses Pydantic-based structured prompts with JSON schema validation.

If you're still experiencing this issue with the latest version, please reopen with details about your ragas version and LLM configuration.
Repository: explodinggradients/ragas
Issue: [Feature Request] OpenLLMetry Integration (OpenTelemetry)
Root cause: Unimplemented Feature Gap

**Describe the Feature**
OpenTelemetry is a de facto standard for reporting metrices and OpenLLMetry uses OpenTelemetry under the hood. It would be better to have an integration to send such metrices for RAG evaluation to OpenTelemetry collectors as well.

**Why is the feature important for you?**
Since OpenTelemetry is a common standard which would be used easily in any existing systems.

**Additional context**
NA

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


hey @amitjoy thanks for the feat request

I have been keeping an eye on OpenLLMetry for a while and with 
- [feat(langchain): use callbacks by tibor-reiss · Pull Request #1170 · traceloop/openllmetry](https://github.com/traceloop/openllmetry/pull/1170)
- [fix(langchain): improve callbacks by tibor-reiss · Pull Request #1317 · traceloop/openllmetry](https://github.com/traceloop/openllmetry/pull/1317)

there should be some progress and it should be easier. I'll take a closer look at this and see what would the easiest 🙂 

in the mean time do you use OpenLLMetry for your LLM application tracing also? do you use any other tools to visualize the traces? 

Hi @jjmachan,

Thank you for your prompt response and the detailed information regarding the integration with ragas. Yes, I do use OpenLLMetry and publish the metrics and spans to Sentry, as it can also function as an OpenTelemetry Collector. Since I already utilize Sentry for monitoring unexpected runtime issues, it seemed logical to extend its use to OpenTelemetry traces as well.
Repository: explodinggradients/ragas
Issue: Discord server link in README.md is expired

Current link in `README.md` https://github.com/explodinggradients/ragas/blob/main/README.md points to https://discord.gg/5djav8GGNZ, which returns an invalid invite.

Hopefully you can refresh the invite link :). If you don't want people to join, probably should remove the link

<img width="987" alt="image" src="https://github.com/explodinggradients/ragas/assets/3402261/25ff8775-1053-4e62-b87d-6b553b081f8a">


Hi @nelsonauner, Sorry for the inconvenience. I can see that the link is still active. Could you perhaps try again now? 

Works, great, thanks!
Repository: explodinggradients/ragas
Issue: Default Weights of semantic similarity and F1 score to calculate Answer Correctness
Root cause: Documentation Deficiency

As per text: Once we have the semantic similarity, we take a weighted average of the semantic similarity and the factual similarity calculated above to arrive at the final score. 

What are the default Weights of semantic similarity and factual similarity to calculate Answer Correctness?

<!-- Greeting -->
Hey @ekaanshkhosla! I'm here to assist you with any questions or issues you have. Let's work together to resolve your query.

<!-- Answer -->
The default weights of semantic similarity and factual similarity used to calculate Answer Correctness are 0.75 for factuality and 0.25 for semantic similarity <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


closing this since dosu answered it correctly. feel free to reopen of needed 🙂
Repository: explodinggradients/ragas
Issue: ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it
Root cause: Resource Mishandling

I am using RAGS evaluate method to do evaluation on 35 Test dataset with groundtruth. Its completing the evaluation and failing at the last step with this error. I have added "raise_exceptions=False" also, Still same issue.It was working 1 week before, suddenly is any update happened? I am running in Databrick


@prabham17 I'm not sure what the issue could be since we didn't release anything recently. Could you share the stack trace if possible?

I am facing similar problem. Here is the description:
I am using "mistralai/Mixtral-8x22B-Instruct-v0.1" running on VLLM on OpenAI like server. Also I am running the inference on 4 A-100 GPU cluster.
The model is not able to evaluate the following ragas metrics successfully and giving the ExceptionInRunner error:
     
    faithfulness, ##working for small #for long getting ExceptionInRunner
    answer_correctness, #working for small #for long getting ExceptionInRunner
    
The error appears as given below:
![image](https://github.com/explodinggradients/ragas/assets/112294417/62d9b609-d235-49d8-84a9-5b384bc5d80d)
![image](https://github.com/explodinggradients/ragas/assets/112294417/621bac52-ceb5-4340-976a-fa0f00b65ef2)

Also tried with a slight change of setting is_async=True, still got the same result: 
![image](https://github.com/explodinggradients/ragas/assets/112294417/83aa8236-545d-4c14-b2fa-12d43a14fbfd)


It's worth mentioning here that the aforementioned metrics are successfully running for smaller set of input samples, but giving the cited error for bigger sample size. For example: When I was experimenting earlier using only 5 samples, all three metrics were working just fine. However, when I ran them over 150 samples of data, they are not working. 

set raise_exceptions=False may be helpful

I did try that. But it was not working. The current fix is to increase the timeout to '600' in ragas.metrics.base.py (setting high value). It's working for the most part, but still facing ExceptionInRunner problem, rarely tho. 
![image](https://github.com/explodinggradients/ragas/assets/112294417/a61be0b6-1fd5-4679-9c1c-98cdceadd125)


hope it was fixed with the above PR 

let us know if you face any more issues! sorry about this one
Repository: explodinggradients/ragas
Issue: feat(benchmark): Implement flexible model selection in TestsetGenerator for improved customization and consistency
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Currently, the `tests/benchmarks/benchmark_testsetgen.py` script has hardcoded LLM models for `generator_llm`, `critic_llm`, and `embeddings`. However, other Ragas scripts have the ability to override these defaults when called from a Jupyter notebook (etc.).

**Why is the feature important for you?**

Current code in benchmark_testsetgen.py:
```python
# hardcoding of values in tests/benchmarks/benchmark_testsetgen.py
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)
```

Because of the hardcoding in the benchmark_testsetgen.py script it negates the ability to dynamically set global defaults in notebooks / scripts for the generator, critic, and embedding models when calling Ragas:

```python
# setting of defaults in Jupyter Notebook or script calling Ragas
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-0125")
critic_llm = ChatOpenAI(model="gpt-4o")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
```

**Additional context**
Modify the `benchmark_testsetgen.py` script to mirror `src/ragas/testset/generator.py` and support optional parameters to allow for consistent overriding of default settings for `generator_llm`, `critic_llm`, and `embedding_model`.  This will allow users to set global and consistent defaults when calling Ragas from a script / notebook:

```python
# updates to benchmark_testsetgen.py
def initialize_testset_generator(
    generator_llm="gpt-3.5-turbo-16k",
    critic_llm="gpt-4",
    embeddings="text-embedding-ada-002"
):
    generator_llm = ChatOpenAI(model=generator_llm)
    critic_llm = ChatOpenAI(model=critic_llm)
    embeddings = OpenAIEmbeddings(model=embeddings)

    return TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)

# Usage
generator = initialize_testset_generator()
```

I was surprised when I was running the Ragas scripts from a notebook that it ignored the GPT-4o settings for the critic_llm, and used the much more expensive and older base GPT-4 model.  A number of better variants on the approach above, but this should be sufficient.

The essential requirement is consistency and transparency of models used during specific steps of the process.

hey @donbr I'm not sure if I understood the idea but we can add that quite easily to the benchmark. but I was not able to understand the usecase you had in mind. 

the `tests/benchmark` directory is meant as a test suite for developing, hence the hard coded LLMs. what was the usecase for calling them directly? are you running the benchmarks yourself for different LLMs too?

in that case we can easily add the change you proposed! it would improve it like you said
Repository: explodinggradients/ragas
Issue: Are Multi-context questions generated by using related docs or related chunks of the same doc?
GitHub labels: question

stale




Are Multi-context questions generated by using related docs or related chunks of the same doc?

<img width="916" alt="image" src="https://github.com/explodinggradients/ragas/assets/123583648/06f07d05-8cad-4e3f-b88c-a74ae11f6837">


<!-- Greeting -->
Hey there, @nikithamary! I'm Dosu, a bot here to help you with bugs, questions, and becoming a contributor. I'll assist you while waiting for a human maintainer.

<!-- Answer -->
Multi-context questions are generated by rephrasing the question in a manner that necessitates information from multiple related sections or chunks to formulate an answer. This involves merging nodes from different documents or sections and generating a question based on the combined context <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/testset_generation.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/evolutions.py)</sup><sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/prompts.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: answer_correctness giving inconsistent result
Root cause: Algorithmic Error

**Describe the bug**
The answer correctness does not seems consistent.

**Code to Reproduce**
question = ["1","2","3","4","5"]
answers = ["yes", "no", "no", "yes", "no"]
contexts = [["1","2","3","4"],["1","2","3","4"],["1","2","3","4"],["1","2","3","4"],["1","2","3","4"]]
ground_truth = ["yes", "no","yes","no", "no"]

ds = Dataset.from_dict({
    "question" : question,
    "answer" : answers,
    "contexts" : contexts,
    "ground_truth" : ground_truth
})

result_ragas = evaluate(ds,metrics=metrics,raise_exceptions=False)
result_ragas

results = result_ragas.to_pandas()
results.head()

**result**
![image](https://github.com/explodinggradients/ragas/assets/51988290/34d68b6a-31e8-46ac-94c1-d2dae2538c51)


How come row 1 and row 4 does not share the same answer correctness ? 


<!-- Greeting -->
Hey @chboudry! I'm here to assist you with any bugs, questions, or contributions. Let's tackle this issue together!

<!-- Answer -->
The difference in the "answer_correctness" value between row 1 and row 4, despite having similar data, can be attributed to several factors:

1. **F1 Score Calculation**: The F1 score is computed based on the presence of true positive (TP), false positive (FP), and false negative (FN) statements. Even slight differences in these classifications can lead to different F1 scores.

2. **Answer Similarity Score**: If the weights for factuality and semantic similarity are not zero, the answer similarity score is also considered. Differences in semantic similarity between the answers and the ground truth can affect the final score.

3. **Weighted Average**: The final "answer_correctness" score is a weighted average of the F1 score and the similarity score. Variations in either of these components can lead to different final scores.

4. **Factual and Semantic Similarity**: Answer correctness encompasses both factual and semantic similarity between the generated answer and the ground truth. Differences in either of these aspects can lead to variations in the final score <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/answer_correctness.md)</sup>.

Therefore, even if the data appears similar, differences in the classification of statements, the calculation of similarity scores, and the weighted combination of these factors can result in different "answer_correctness" values for row 1 and row 4.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Try with new `metrics.collections` in latest ragas release:

`from ragas.metrics.collections._answer_correctness import AnswerCorrectness`

Let me know if you still face issues.

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Why do you use future annotations in evaluation.py ?
GitHub labels: question

In src/ragas/evaluation.py, why are you using future annotations instead of just defining the class Result at first ?


<!-- Answer -->
The `from __future__ import annotations` statement is used to enable postponed evaluation of type annotations. This allows the use of the `Result` class in type hints before it is actually defined. This is particularly useful in the `evaluate` function where the return type is specified as `Result` before the `Result` class is defined later in the code <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu yes, but you can just define Result at the begining of the file

<!-- Answer -->
Defining `Result` at the beginning of the file is indeed an alternative approach. However, using `from __future__ import annotations` can improve readability and maintainability by keeping related code together. This way, the logical flow of the code is preserved, and type hints can still reference classes defined later in the file. It also helps avoid potential circular import issues in larger projects.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: Can anybody tell me the testset generator code given in RAGAS website what columns will it actually create i think we can only get questions and ground_truth ans from testset generator and the context and answer will be generated from our rag architecture
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


<!-- Greeting -->
Hey @Saurabh8255, I'm here to help you with any questions or issues you might have. Let me know how I can assist you!

<!-- Answer -->
The testset generator code from the RAGAS website creates the following columns:

- `episode_done`

This is evident from the `_to_records` method in the `TestDataset` class, which converts each `DataRow` into a dictionary and adds the `"episode_done"` key with a value of `True`.

```python
def _to_records(self) -> t.List[t.Dict]:
    data_samples = []
    for data in self.test_data:
        data_dict = dict(data)
        data_dict["episode_done"] = True
        data_samples.append(data_dict)
    return data_samples
```

<sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/generator.py)</sup>


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


fixed with #1176. sry for the confusion 🙌🏽
Repository: explodinggradients/ragas
Issue: Exact versions of libraries
Root cause: Environment Incompatibility

HuggingFace is famous for moving fast and breaking things. Would be great to have the exact dependency versions of all the python libraries. Something like "`datasets==2.14.3`" instead of `datasets`.

Btw getting `tiktoken` dependency error while running your example. Coming from langchain apparently but for some reason not installed while langchain is installed. Might want to add it as well.

```
lib/python3.9/site-packages/langchain/embeddings/openai.py:321, in OpenAIEmbeddings._get_len_safe_embeddings(self, texts, engine, chunk_size)
    319     import tiktoken
    320 except ImportError:
--> 321     raise ImportError(
    322         "Could not import tiktoken python package. "
    323         "This is needed in order to for OpenAIEmbeddings. "
    324         "Please install it with `pip install tiktoken`."
    325     )
    327 tokens = []
    328 indices = []

ImportError: Could not import tiktoken python package. This is needed in order to for OpenAIEmbeddings. Please install it with `pip install tiktoken`.
```

Hi @ogencoglu, thanks for raising this issue. 
@jjmachan 

thanks again for raising the issue and help make ragas more robust, I really appreciate it Oguzhan :)
while I do agree that it is a chance for error, I don't think pinning a version is a solution in the long term. We will miss out on important bug fixes and security updates that might be release later. But I do think ragas has to do a better job at testing the dependencies so that if and when they change stuff, we make the required changes as well and don't break things. 

as for tiktoken, I don't think we use that internally in ragas and hence not a dependency, do correct me if I'm wrong @shahules786 @ogencoglu. could you share a bit more about the error you're facing so I know where its coming from.

do let me know your thoughts Oguzhan, much appreciated 

I'll close this for now. My hypothesis is that you will have more work dealing with users issues due to not fixing the versions than the work to test the latest versions.
Repository: explodinggradients/ragas
Issue: Can you provide human assessment data mentioned in RAGAS paper?
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Can you could provide the human assessment data collected for bechmarking RAGAS metrics against human evaluations in your [paper](https://arxiv.org/pdf/2309.15217)?

**Why is the feature important for you?**
The [paper](https://arxiv.org/pdf/2309.15217) only benchmarks ChatGPT against human evaluation. This feature would establish a standard dataset for benchmarking any LLM-as-judge models against human evaluation.

**Additional context**
It would be great if you could provide a standard dataset containing question, ground truth, context, human labels for benchmarking all RAGAS metrics for different judge models.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


@awsvmaringa sorry for the delay but are you still looking for it?

@jjmachan I also need it. Could you please provide the data?