zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: My faithfulness and context_recall is incorrect。
Root cause: Validation Gap

I set OPENAI_API_BASE to my own deployed model, and then there were some errors in the data evaluated by Ragas.
First, the same data produces different results each time it's evaluated. Second, there are values outside the range, like -1, in faithfulnessn.
Is there something wrong with my model?how can i locate the bug
![5Y{E2@R{L K{AFO65W21E1Q](https://github.com/explodinggradients/ragas/assets/129468785/8dad95c1-9227-4033-beaa-b279a8aa1c13)


Hi @PurpleLikeNero , the quality of results is highly dependent on the quality of the model. The default prompts in Ragas are tuned for OpenAI, you might want to tune it for your model. Are you using any opensource model?  

> Hi @PurpleLikeNero , the quality of results is highly(高度) dependent(依赖) on the quality of the model. The default prompts in Ragas are tuned for OpenAI, you might want to tune it for your model. Are you using any opensource model?

yes, my using opensource model named [Qwen-7B](https://github.com/QwenLM/Qwen-7B)

It's an interesting model. Do you use [Langsmith](https://github.com/explodinggradients/ragas/blob/main/docs/integrations/langsmith.ipynb) along with ragas? In that case, it would be easier to debug and change the prompt appropriately for your model.

> It's an interesting model. Do you use [Langsmith](https://github.com/explodinggradients/ragas/blob/main/docs/integrations/langsmith.ipynb) along with ragas? In that case, it would be easier to debug and change the prompt appropriately(适当) for your model.

ok,i will try this later.

@PurpleLikeNero did you manage to get better results? I am facing the same issue using as model Llama-2-13B-chat-GGUF (q6) and I think the problem could be with the prompts

I had an error in ContextRecall in _score_batch where the model (non-gpt) response didn't conform to the downstream processing requirements (Attributed vs [Attributed]). I would like to use a Pydantic output parser so that the responses can be better guided. i want to put it in _score_batch immediately after responses are collected or deeper in the LangchainLLM generate code which generates those responses. Any ideas or hints? Thanks

BTW, I was able to resolve the error with some prompt tuning, but I still would like the ability easily (via Pydantic output parser) to control this

@AbsoluteGravity we actually made some improvements to reproducibility by using forcing the evaluator LLM to output it in json. @tinomaxthayil is the one working on this. @tinomaxthayil we could also look into Pydantic Output Parser and see how it compares to json output parser. 

i think json would be better though because the model will have been trained on json outputs and error messages (more likely than pydantic). would that be the case @shahules786 ?
Repository: explodinggradients/ragas
Issue: ground truth
Root cause: Validation Gap

hello, i read the document about the quickstart. ground_truth just required if you are using context_recall.if I do not need context_recall, i do not parpare the context_recall. i do not add context_recall in my metrics,but something wrong happen "Column ground_truths not in the dataset. Current columns in the dataset: ['question', 'answer', 'contexts']"

Hi @bellyqpp , Can you share your code and error trace?

hello.
just pip install ragas==0.0.13
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
prepare my dataset.just 3 columns .question,answer,contexts.
from ragas.metrics import (
    context_relevancy,
    answer_relevancy,
    faithfulness,
)(## i delete the context recall,because i have no ground_truth)
from ragas.metrics.critique import harmfulness
from ragas import evaluate

result = evaluate(
    fiqa_eval["baseline"],
    metrics=[
        context_relevancy,
        faithfulness,
        answer_relevancy,
        harmfulness,
    ],
)

result
 in this step.something wrong happen "Column ground_truths not in the dataset. Current columns in the dataset: ['question', 'answer', 'contexts']"

Hi @bellyqpp , This bug was fixed in the last release so please update to the latest version using `pip install -U ragas`  

hello,there was something wrong.

from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)
error information:cannot import name 'context_precision' from 'ragas.metrics' (c:\python\lib\site-packages\ragas\metrics\__init__.py).


so ,in sep 2023,
from ragas.metrics import (
    context_relevancy,
    answer_relevancy,
    faithfulness,
    context_recall,
)

there have some changes. so I really want to know the difference. as the same ,i do not have ground_truth.
Repository: explodinggradients/ragas
Issue: LLMContextPrecisionWithReference cannot be imported from ragas.metrics
Root cause: Documentation Deficiency

[ X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
`LLMContextPrecisionWithReference` cannot be imported from ragas.metrics. "without reference" version can be. Looks like with reference is not in the \_\_init\_\_, can be imported directly from ragas.metrics._context_precision

Ragas version: 3.10.13
Python version: 0.2.0

**Code to Reproduce**
`from ragas.metrics import LLMContextPrecisionWithReference`

**Expected behavior**
Should import correctly

**Additional context**
none

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hello @mhmdmodan! I'm here to assist you with your bug and any other questions you may have. Let's work together to resolve the issue.

<!-- Answer -->
A similar issue has been reported where certain imports are not available directly from `ragas.metrics`. In this case, `LLMContextPrecisionWithReference` is not included in the `__init__.py` file of the `ragas.metrics` module, which is why it cannot be imported directly from there. Instead, you can import it directly from `ragas.metrics._context_precision`.

Here's how you can modify your import statement:

```python
from ragas.metrics._context_precision import LLMContextPrecisionWithReference
```

This should resolve the import issue. There is no specific reason mentioned in the documentation or code comments for why it is not included in the `__init__.py` file <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_precision.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Hi @mhmdmodan,

I’m closing this issue as Dosu has already answered the question.
Repository: explodinggradients/ragas
Issue: Add cost tracking to dataset generator and allow dataset size control
Root cause: Unimplemented Feature Gap

**Describe the Feature**
There are 2 problems with the current dataset generator approach:
- it doesn't support cost tracking
- it chunks and embeds the full document EVEN if the testset_siz == 1. The problem is that I, as a user, want to be able to efficiently generate smaller datasets, while current approach doesn't consider the testset_size when chunking and embedding the documents

**Why is the feature important for you?**
- Cost tracking is important to be able to monitor evaluation costs end-to-end:
 a) First part - costs of dataset generation (not supported)
 b) Second part - costs of a evaluation runs (already supported)
- More efficient dataset generation approach is important because currently dataset generation can be extremely inefficient as it doesn't consider the testset_size. There has to be a way to decide in a smart way whether to chunk & embed the full document or not based on the number of test questions needed to be generated.

**Additional context**
Add any other context about the feature you want to share with us.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hey @Twist333d Very valid concerns. 
1) We will plan and add support to cost estimation for test generation soon.
2) While efficiently generating ( and regenerating) test data points from a single document set, we also have to make sure that the generated points were not already generated before. So we intend to make this process efficient by doing a one-time preprocessing of documents and then letting you persist with the intermediate form (KG). In that case, one could repeatedly sample data points from the same corpus w/o redoing the preprocessing step. What do you think?

We will be continuously improving the new test gen, and would love to chat and understand more from you. 
https://cal.com/shahul-ragas/30min

@shahules786 thanks for a prompt response! I think what you describe does indeed address the two core concerns I had:
- be able to monitor costs for both dataset generation and evaluation
- ensure efficient dataset generation even if testsize is small

On the proposal above, how would a user manage the intermediate form? Would I need to save it / manage it locally?
Repository: explodinggradients/ragas
Issue: `ALL_METRICS` variable does not contain all metrics in v0.2
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
The `ALL_METRICS` variable in the `ragas.metrics` module only has 8 of the metrics, but there are clearly far more than this.

Ragas version: 0.2.0
Python version: 3.12

**Code to Reproduce**
```python
>>> from ragas.metrics import ALL_METRICS
>>> len(ALL_METRICS)
8
```

**Error trace**
No error.

**Expected behavior**
There seem to be roughly 30 metrics, so a number closer to that. I appreciate some metrics are duplicates of others with a different name.

**Additional context**
None.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hey @edjeffery! 👋 I'm here to help you squash those bugs and answer your questions. Let's tackle this together!

<!-- Answer -->
The issue you're experiencing with the `ALL_METRICS` variable not containing all metrics might be due to how metrics are defined and registered in the `ragas` library. The `ALL_METRICS` list is created by inspecting the module's members and including only those that are in the `__all__` list and are not classes or built-in objects <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/__init__.py)</sup>.

If the `__all__` list does not include all the metrics you expect, or if some metrics are defined in a way that excludes them from being added to `ALL_METRICS`, this could explain why you're seeing fewer metrics than expected. You might want to check the `__all__` list in the `ragas.metrics` module to ensure it includes all the metrics you are expecting. Additionally, ensure that the metrics are not being filtered out due to being classes or built-in objects.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@edjeffery I think we'll remove that - it is not expected to work like that. Do check the docs and only pick the metrics you want :)
Repository: explodinggradients/ragas
Issue: evaluate() needs optional delay to prevent rate limiting by API 
Root cause: Documentation Deficiency

**Describe the Feature**
see title 
**Why is the feature important for you?**
avoid rate limiting 
**Additional context**
no 
<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


we actually have something for that - could you check out https://docs.ragas.io/en/stable/howtos/customizations/_run_config/  and let me know if it is working 🙂 

ah ok thanks. Are these ""Exception raised in Job[200]: TimeoutError()
Exception raised in Job[198]: TimeoutError()"" self recovering, or is this saying it will skip over the data? 

it will skip over

check the `raise_exceptions` flag if you have doubts 🙂 

I just mean it as a question, do these try the call again, or just skip them? 

edited

Ah ok. By default, do you think it should try again? Skipping over examples due to rate limiting seems to skew the results of the entire evaluation. Do you think it should wait to try again?
Repository: explodinggradients/ragas
Issue: [R-299] Add cost estimation to testset generator
Root cause: Unimplemented Feature Gap

**Describe the Feature**
RAGAS supports cost estimation during evaluation; it would also be useful to know how much it costs to construct the synthetic dataset.

I would like to have the ability to use the CostCallbackHandler in both KnowledgeGraph creation and in `TestsetGenerator.generate()` calls.  Specifically, it would be useful to track them separately and jointly:

- if I create a KnowledgeGraph with `apply_transforms()`, I would like to know the token count / cost
- if I call `TestsetGenerator.generate()` with that previously created KnowledgeGraph, I would like to know just the token count / cost of scenario generation and question synthesis
- if I call `TestsetGenerator.generate_with_langchain_docs()`, I would like to know the token count / cost of both creating the KnowledgeGraph _and_ scenario + question generation

**Why is the feature important for you?**
Cost management is critical especially if source material is dynamic and requires frequent dataset synthesis.


<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


<sub>[R-299](https://linear.app/exploding-gradients/issue/R-299/add-cost-estimation-to-testset-generator)</sub>

+1

that is a very good suggestion @ahgraber - will get this sorted for you, hopefully by next week 🙂 

Thanks for adding this feature, however, it does not yet taken into account the cost of generating the KG used for testset generation, which is probably the most costly part.

hey @Kefan-pauline that is true - just created https://github.com/explodinggradients/ragas/issues/1671 to track this. Will get to this soon 🙂
Repository: explodinggradients/ragas
Issue: [R-298] Support Passing Multiple Evaluator Models to reduce bias 
Root cause: Unimplemented Feature Gap

**Describe the Feature**
I think it could be useful to support multiple evaluator models, and average performance across models to reduce bias. 
**Why is the feature important for you?**
It seems like using multiple evaluator models could be less biased. 
**Additional context**

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


<sub>[R-298](https://linear.app/exploding-gradients/issue/R-298/support-passing-multiple-evaluator-models-to-reduce-bias)</sub>

Hey @bdytx5 , this is a good idea. I think there also have been some recent papers showcasing evidence for it. I'll add to the list of improvements to be completed before the next major release. Thank you. 

If I'm understanding this right, you can already kind of do this by specifying a metric per evaluator model:

```py
# llms is a list of LangChain adapters; use `llm.model` for LlamaIndex
metrics = [
    [Faithfulness(  # response wrt retrieved_context
        name=f"faithfulness_{llm.model_name}",
        llm=LangchainLLMWrapper(llm),
    ) for llm in llms],
    [ResponseRelevancy(  # response wrt input
        name=f"answer_relevance_{llm.model_name}",
        llm=LangchainLLMWrapper(llm),
        embeddings=LangchainEmbeddingsWrapper(em),
    ) for llm in llms],
]
...
results = evaluate(dataset=dataset, metrics=metrics)
```

@ahgraber Yes, this is one way to achieve this now.
Repository: explodinggradients/ragas
Issue: Encounter Error: ZeroDivisionError: division by zero (context_relevance)
Root cause: Validation Gap

Got the issues while Loading Dataset of Size 250(Question, Answer, GT, Context), and some of the context were null. 


python3.9/site-packages/ragas/metrics/context_relevance.py", line 162, in _score_batch
    score = min(len(indices) / len(context_sents), 1)
ZeroDivisionError: division by zero

File: myenv_rfp/lib/python3.9/site-packages/ragas/metrics/context_relevance.py
Repository: explodinggradients/ragas
Issue: [R-302] Support function-calling / json mode / structured generation for testset generation
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Most service APIs now support enforcing schema outputs through function calling, json mode, or structured generation.
It would be really useful to have an option that would use the service API to enforce schema constraints rather than hoping chat prompts follow the expected format.

**Why is the feature important for you?**
With OpenAI, synthetic generation works flawlessly 99% of the time.
With Anthropic or Llama models, I get frequent parse errors, which end up retrying and ultimately failing. This uses a lot of tokens (and therefore $). 
Concretely, generating a testset of 100 questions, gpt-4o-mini uses ~660k input and produces ~13k output tokens.  When I attempt to generate a testset from the same knowledge graph with Anthropic Claude 3.5 sonnet, the generation fails from parse errors but I still end up using ~850k input and ~22.5k output tokens due to the retries!

**Additional context**
Given most of the responses are being parsed with Pydantic, it should be fairly trivial to turn the desired Pydantic object into a jsonschema (hint: `openai provides openai.pydantic_function_tool()` to convert Pydantic models to openai-compatible subset jsonschema)

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


<sub>[R-302](https://linear.app/exploding-gradients/issue/R-302/support-function-calling-json-mode-structured-generation-for-testset)</sub>

@ahgraber thanks for the suggestion - we should definitely do that as the default for the services that do support it

ref: https://python.langchain.com/v0.1/docs/modules/model_io/chat/structured_output/
something on top of this should work

also would love to chat sometime with you too Alex and get more feedback. I've send you an email to connect. Are you on discord btw

cheers ❤️ 
Jithin

Related:

For evals, many of the prompts seem to request a numeric answer (Context Recall -> "Attributed=0/1", Context Precision -> "Verdict=0/1"
While structured generation would work here, perhaps an even better option would be constraining the outputs to just tokens '0' and '1'?
OpenAI supports this with the `logit_bias` parameter (see [openai docs](https://platform.openai.com/docs/api-reference/chat/create#chat-create-logit_bias) and [AAAzzam's twitter thread](https://x.com/AAAzzam/status/1669753721574633473)); I'm not sure how it's integrated into LangChain/Llamaindex and whether it is supported for all/most models.


Is this still not planned? I think it would be useful for both schema enforcement and for cost and eval time from retries.
Repository: explodinggradients/ragas
Issue: Request to update Ragas 0.1.21 dependencies to resolve incompatibility with langchain-google-community
Root cause: External Dependency Breakage

- [x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

Is Ragas able to bump the langchain dependencies to include 0.3.* versions ([langchain/releases](https://github.com/langchain-ai/langchain/releases))

```
poetry show ragas  
 name         : ragas  
 version      : 0.1.21 
 description  :        

dependencies
 - appdirs *
 - datasets *
 - langchain <0.3
 - langchain-community <0.3
 - langchain-core <0.3
 - langchain-openai *
 - nest-asyncio *
 - numpy *
 - openai >1
 - pysbd >=0.3.4
 - tiktoken *
```

**Code Examples** / **Additional context**
I was trying to install [`langchain-google-community`](https://pypi.org/project/langchain-google-community/) but this depends on`langchain-core` `(>=0.3.0,<0.4)` and `langchain-core` `(>=0.3.0,<0.4)`

```
poetry add langchain-google-community
Using version ^2.0.1 for langchain-google-community

Updating dependencies
Resolving dependencies... (0.1s)


Because no versions of ragas match >0.1.20,<0.1.21 || >0.1.21,<0.2.0
 and ragas (0.1.20) depends on langchain-core (<0.3), ragas (>=0.1.20,<0.1.21 || >0.1.21,<0.2.0) requires langchain-core (<0.3).
And because ragas (0.1.21) depends on langchain-community (<0.3)
 and langchain-google-community (2.0.1) depends on langchain-community (>=0.3.0,<0.4.0), if ragas (>=0.1.20,<0.2.0) and langchain-google-community (2.0.1) then langchain-core (<0.3).
And because langchain-google-community (2.0.1) depends on langchain-core (>=0.3.0,<0.4)
 and no versions of langchain-google-community match >2.0.1,<3.0.0, ragas (>=0.1.20,<0.2.0) is incompatible with langchain-google-community (>=2.0.1,<3.0.0).
So, because ana-ragas-explore depends on both ragas (^0.1.20) and langchain-google-community (^2.0.1), version solving failed.
```

 


Nvm! Awww I see this is pinned to `langchain<0.3` and I should use Ragas `0.2.*`

https://github.com/explodinggradients/ragas/releases/tag/v0.1.21
Repository: explodinggradients/ragas
Issue: [R-301] make llm and embeddigns required
Root cause: Architectural Constraint

currently we use a lot of default llms - this makes Ragas very hard to change. We need to make all the llms and embeddings required so that users explicitly pass them

<sub>From [SyncLinear.com](https://synclinear.com) | [R-301](https://linear.app/exploding-gradients/issue/R-301/make-llm-and-embeddigns-required)</sub>

100% agree. There have been a lot of issues opened regarding this problem and the issues its causing that are preventing people from producing testsets. I believe we should remove all the default llms and require users to pass in their llms/embeddings. I am happy to help in this effort, but would like to talk to a maintainer about the vision going forward. 


hey @hunter-walden2113 definetly - this is my cal cal.com/jjmachan/huddle - feel free to book some time there. Lookin forward to chatting 🙂 

hey @hunter-walden2113 while this PR is would help a lot - it will also break a lot of code we have now. So we are putting this on pause and will take this up before v0.3 

Sounds good. Let me know when you want to take it back up.