zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: little wrong in this page <https://docs.ragas.io/en/stable/concepts/metrics/overview/#different-types-of-metrics>?
GitHub labels: documentation

question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources ,and i find little wrong in this page <https://docs.ragas.io/en/stable/concepts/metrics/overview/#different-types-of-metrics>?

**Your Question**
The code example has a typo in the word。

**Code Examples**
the page says:

```
from ragas.metrics import AgentGoalAccuracy
from ragas import MultiTurnSample

scorer = AgentGoalAccuracy()
await metric.multi_turn_ascore(sample)
```

but, i think it should be like this:

```
from ragas.metrics import AgentGoalAccuracy
from ragas import MultiTurnSample

scorer = AgentGoalAccuracy()
await scorer.multi_turn_ascore(sample)
```

`metric.multi_turn_ascore` should be replace of  `scorer.multi_turn_ascore`.



Hey @earfanfan , nice catch. Would you like to raise a PR?

thanks for raising this @earfanfan will be closed with #1825 🙂
Repository: explodinggradients/ragas
Issue: logprobs to assess the confidence of verdict prediction
Root cause: Unimplemented Feature Gap

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**

I would like to use [logprobs](https://cookbook.openai.com/examples/using_logprobs) to assess the confidence of verdict predictions in ragas, the implementation will likely be using callbacks. Before implementing it, I would like to know if anyone has already experimented with this and what were your takes on it?

My hope is to reduce the indeterministic nature of the scores from one run to another, by e.g., reducing hallucinations in verdict predictions (i.e. if the confidence is very low for verdict prediction, then rerun the prediction).

@jjmachan 
@shahules786 


hey @Kefan-pauline that is a very good idea and we would love to work with you and help implement it. We can get on a call and discuss this more if you want

but I think the implementation will be similar to how we have it with https://docs.ragas.io/en/stable/howtos/applications/_cost which could be an inspiration but we can discuss it more

Hi @jjmachan, is there any kind of development on this idea. I would like to learn and contribute to this interesting feature.
Repository: explodinggradients/ragas
Issue: 'StringIO' object has no attribute 'TP'
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
  'StringIO' object has no attribute 'TP'

Ragas version:
Python version:0.2.7



hey @richness thanks for reporting this 🙂 
could you share a bit more context about this? ideally the error trace so that I can figure out from where the error is coming?

closed with https://github.com/explodinggradients/ragas/pull/1864 will be released with v0.2.12 🙂

I'm closing this for now but if the issue is still persisting, please do let me know - really sorry about the delay
Repository: explodinggradients/ragas
Issue: What is Different Between LLMContextPrecisionWithoutReference and Faithfulness?
GitHub labels: question

I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

What is Different Between LLMContextPrecisionWithoutReference and Faithfulness?
First,we knew that LLMContextPrecisionWithReference analyze relationship between retrieved_contexts and reference(ground truth answer) and user_input to calculate precision of retrieved_contexts.

But when using LLMContextPrecisionWithoutReference, it mean alayze relationship between retrieved_contexts and response(LLM response),  when retrieved_contexts highly relevant with response(LLM response) the Precision@k  would be 1.

 I knew that the formula of LLMContextPrecisionWithoutReference and Faithfulness are different but if using structure below,
I think that LLMContextPrecisionWithoutReference is also analyze Context->Response like Faithfulness.Am I Correct?
<img width="616" alt="rag tried" src="https://github.com/user-attachments/assets/b4f79c2b-b638-470f-89c5-82ed377c4dd1" />


hey @Jimmy-web169 thanks for raising this 🙂 
@shahules786 could you help him out?

Hey, @Jimmy-web169 
Yes, both use (context, response) as data points to score. 
- Context precision without reference: Checks if each of the retrieved chunks was useful to arrive at the response (score each chunk as 0/1 based on this question). Then, it uses the Precision@k formula to quantify how good was the retrieved context.  (Context -> Response)
- Faithfulness: Here it breaks down the response to claims and then sees if each of these claims can be attributed to any of the given contexts. (Response -> Context) 

I hope this clears things up for you. Please close the PR if it helps :) 



Thank you! i will close the PR
Repository: explodinggradients/ragas
Issue: Langgraph integration not working
Root cause: Documentation Deficiency

 I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.


Bug: 
Referring to https://docs.ragas.io/en/stable/howtos/integrations/_langgraph_agent_evaluation/#agent-goal-accuracy
I was trying to integrate the RAGAS into my Langgraph project but it seems like the coroutines are not working for `await scorer.multi_turn_ascore(sample)` 

Here's the output I'm getting

`
>>> await scorer.multi_turn_ascore(sample)
  File "<stdin>", line 1
SyntaxError: 'await' outside function
>>>
`

`
>>> scorer.multi_turn_ascore(sample)
<coroutine object MultiTurnMetric.multi_turn_ascore at 0x16bd3b480>
>>>
`

Ragas version:v0.2.8

Python version:

**Code to Reproduce**
[RAGAS DOCS](https://docs.ragas.io/en/stable/howtos/integrations/_langgraph_agent_evaluation/#agent-goal-accuracy)

**Error trace**
SyntaxError: 'await' outside function

**Expected behavior**
Should return a float value


<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


hey @imvaibhav28  thanks for raising this issue
@sahusiddharth could you check this one out?

Hello @imvaibhav28,

I’ve tried to recreate the issue on my end but haven’t been able to replicate it.

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: Unable to instantiate HuggingfaceEmbeddings
Root cause: Unimplemented Feature Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Unable to instantiate `HuggingfaceEmbeddings`, because it doesn't implement asynchronous embed methods of base `BaseRagasEmbeddings` class.

Ragas version: 0.2.9
Python version: 3.12.8

**Code to Reproduce**
Code is taken from the docs basically.

```python
from ragas.dataset_schema import SingleTurnSample
from ragas.embeddings import HuggingfaceEmbeddings
from ragas.metrics import SemanticSimilarity

sample = SingleTurnSample(
    response="The Eiffel Tower is located in Paris.",
    reference="The Eiffel Tower is located in Paris. I has a height of 1000ft."
)
ragas_embeddings = HuggingfaceEmbeddings(model_name="BAAI/bge-reranker-v2-m3")
scorer = SemanticSimilarity(embeddings=ragas_embeddings, is_cross_encoder=True)

await scorer.single_turn_ascore(sample)
```

**Error trace**

```sh
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File /home/magnesium/sadaivault/prrag/kilt_eval.py:[1](https://file+.vscode-resource.vscode-cdn.net/home/magnesium/sadaivault/prrag/kilt_eval.py:1)
----> 1 ragas_embeddings = HuggingfaceEmbeddings(model_name="BAAI/bge-reranker-v2-m3")

TypeError: Can't instantiate abstract class HuggingfaceEmbeddings without an implementation for abstract methods 'aembed_documents', 'aembed_query'
```

**Expected behavior**
I expected the object to be instantiated and functional.


hey @sadaisystems could you maybe check out fastembed https://github.com/qdrant/fastembed it had async support last time I checked, which was a while back, could you check it out?

let me know if you need help. I'll try it out in my end and share the code snippet too

Try this

```
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import SemanticSimilarity
from langchain_huggingface import HuggingFaceEmbeddings

sample = SingleTurnSample(
    response="The Eiffel Tower is located in Paris.",
    reference="The Eiffel Tower is located in Paris. I has a height of 1000ft."
)
langchain_hf_embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
ragas_embeddings = LangchainEmbeddingsWrapper(langchain_hf_embeddings)
scorer = SemanticSimilarity(embeddings=ragas_embeddings, is_cross_encoder=True)

await scorer.single_turn_ascore(sample)
```
Repository: explodinggradients/ragas
Issue: LCDocument import error due to TYPE_CHECKING condition
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When I run https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/llamaindex.ipynb in google colab, I get the following error:

NameError                                 Traceback (most recent call last)
[<ipython-input-9-27a007be868c>](https://localhost:8080/#) in <cell line: 2>()
      1 # generate testset
----> 2 testset = generator.generate_with_llamaindex_docs(
      3     documents,
      4     testset_size=5,
      5 )

1 frames
[/usr/local/lib/python3.10/dist-packages/ragas/testset/synthesizers/generate.py](https://localhost:8080/#) in <listcomp>(.0)
    239             # create the transforms
    240             transforms = default_transforms(
--> 241                 documents=[LCDocument(page_content=doc.text) for doc in documents],
    242                 llm=llm_for_transforms,
    243                 embedding_model=embedding_model_for_transforms,

NameError: name 'LCDocument' is not defined

Ragas version: 0.2.9
Python version: 3.10

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
In `ragas/testset/synthesizers/generate.py`, `LCDocument` is imported under a `TYPE_CHECKING` condition, and the value of TYPE_CHECKING is false.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


hey @netbuddy thanks for reporting this 🙂 

we merged a fix for this in #1804 could you check it out? it will be out in this weeks release. 

If it solves it feel free to close this issue.
Repository: explodinggradients/ragas
Issue:  howtos:  comparing LLMs using ragas
Root cause: Documentation Deficiency

The idea is to build a how-to notebook comparing two LLMs using ragas
- The work can be organized similar to https://docs.ragas.io/en/latest/howtos/applications/compare_embeddings.html
- The metrics to use here are `faithfulness` and `answer_relevancy`
- Some suggestions for models to compare [falcon](https://huggingface.co/tiiuae/falcon-7b-instruct) vs [zyphyr](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha). You can use the free HF inference API for these models. 



Hi @shahules786, I'd like to take this up

finished by @tinomaxthayil here https://docs.ragas.io/en/latest/howtos/applications/compare_llms.html

When using the above notebook and running the below code block  for comparing LLMs, I am running into the TypeError. Any pointers ?

```
# Use zephyr model using HFInference API

zephyr_llm = HuggingFaceInferenceAPI(model_name="HuggingFaceH4/zephyr-7b-alpha", token="*****" )
qe1 = build_qe(zephyr_llm)
result_ds = generate_responses(qe1, test_questions, test_answers)
result_zephyr = evaluate(result_ds, metrics=metrics )
```

Error:
TypeError: evaluate() missing 1 required positional argument: 'questions'



Hi, can you verify if your result_ds is of the format:
Dataset({
    features: ['question', 'answer', 'contexts', 'ground_truths']
})

Hi @tinomaxthayil , thanks for your reply. I figured I was using the wrong evaluate function. (`ragas.llama_index.evaluate` instead of `ragas.evaluate`. The code works fine now. 

Thanks for creating this super useful notebook.
Repository: explodinggradients/ragas
Issue: Is there any plan to support `reason` in LLMMetrics and EvaluationResult?
Root cause: Unimplemented Feature Gap

[ * ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
For evaluation explainability, it is very valuable to also include the reason into the result so that we understand better how LLM metrics making decisions and improve the metrics accordingly. 

Is there any plan to support it? Thanks!


**Code Examples**
NA

**Additional context**
Anything else you want to share with us? 


@parkerzf you are right and we currently do have something like https://docs.ragas.io/en/stable/howtos/applications/_metrics_llm_calls to help with it. Can you check and see if it works for your usecase?

Hey @jjmachan Thanks for the reply! I think it is very close to what I am looking for. 

Ideally, I would like to use it as following:
```python
from datasets import load_dataset
from ragas import EvaluationDataset
from ragas import evaluate
from ragas.metrics._aspect_critic import harmfulness

dataset = load_dataset("explodinggradients/amnesty_qa", "english_v3")


eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"])

results = evaluate(eval_dataset[:5], metrics=[harmfulness])
results.to_pandas(including_trace=True)
```

The output dataframe contains the following columns:
["user_input", "response", "harmfulness", "harmfulness_reason"] 


What is the ETA of this new feature? I would like to try it out. 


Closing after 8 days of waiting for the additional info requested.

hey @parkerzf I'm don't think we will add it to the pandas dataframe, it will be redundant and I don't thing we can show multiple steps here - like for faithfulness or metrics where LLM is just one part of the process - like context recall

what we can do this for is aspect critic, since the verdict and reason are the only fields it shows. But in order to do that you can write a simple function. I can write it up for you if you want
Repository: explodinggradients/ragas
Issue: The metric [context_precision] that is used requires the following additional columns ['reference']
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
ValueError: The metric [context_precision] that is used requires the following additional columns ['reference'] to be present in the dataset.




Hello @Handapao-0,

The context precision metric is available in two variations: one with a reference and one without. For detailed information on their usage, please feel free to check the documentation [here](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/).

i have a new question. The metric [non_llm_context_recall] that is used requires the following additional columns ['reference_contexts'] to be present in the dataset.

 My dataset contains the following:

data = {
    "question": questions,
    "answer": answers,
    "contexts": contexts,
    "ground_truths": ground_truths
}
 
dataset = Dataset.from_dict(data)


Hello @Handapao-0,

You can modify the key names to align with the format used in Ragas. In your case, something like this should work:

```python
data = {
  "user_input": questions,
  "response": answers,
  "retrieved_contexts": contexts,
  "reference": ground_truths
}
```


It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: [R-316] Feedback for Ragas App (app.ragas.io)
GitHub labels: enhancement

We launched the Ragas App to help as a tool to better analyze your evaluations and testsets generated. You can checkout the docs see it in actions
- https://docs.ragas.io/en/stable/getstarted/evals/#analyzing-results
- https://docs.ragas.io/en/stable/getstarted/rag_eval/#analyze-results
- https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/#analyzing-the-testset

Over the coming months we will be shipping for features and we would love to hear your feedback on these 🙂 

Cheers
The Ragas Team

<sub>[R-316](https://linear.app/exploding-gradients/issue/R-316/feedback-for-ragas-app-appragasio)</sub>
Repository: explodinggradients/ragas
Issue: 'StringIO' object has no attribute <some-attribute>
Root cause: Algorithmic Error

**Describe the bug**

I believe there is a bug in [`RagasOutputParser`](https://github.com/explodinggradients/ragas/blob/main/src/ragas/prompt/pydantic_prompt.py#L387) when OutputParserException is raised.

While we try to generate new prompt,[ we forget to parse the output again.](https://github.com/explodinggradients/ragas/blob/main/src/ragas/prompt/pydantic_prompt.py#L417). 


**Specificaly in**

```python
class RagasOutputParser(PydanticOutputParser[OutputModel]):
    async def parse_output_string(
        self,
        output_string: str,
        prompt_value: PromptValue,
        llm: BaseRagasLLM,
        callbacks: Callbacks,
        retries_left: int = 1,
    ):
        callbacks = callbacks or []
        try:
            jsonstr = extract_json(output_string)
            result = super().parse(jsonstr)
        except OutputParserException:
            if retries_left != 0:
                retry_rm, retry_cb = new_group(
                    name="fix_output_format",
                    inputs={"output_string": output_string},
                    callbacks=callbacks,
                )
                fixed_output_string = await fix_output_format_prompt.generate(
                    llm=llm,
                    data=OutputStringAndPrompt(
                        output_string=output_string,
                        prompt_value=prompt_value.to_string(),
                    ),
                    callbacks=retry_cb,
                    retries_left=retries_left - 1,
                )
                retry_rm.on_chain_end({"fixed_output_string": fixed_output_string})
                result = fixed_output_string
            else:
                raise RagasOutputParserException()
        return result
```

my chain of thoughts:

-  the `OutputModel` of `fix_output_format_prompt` is `StringIO`
=> `fixed_output_string` will be of type `StringIO`
=> given `result = fixed_output_string`, the `result` will be of type `StringIO` no matter of what the original `OutputModel` is


Ragas version: 0.2.10
Python version: 3.10.13

----

I believe [this issue](https://github.com/explodinggradients/ragas/issues/1798), may be related


I encounter the same error on my end!

Ugh. Encountering it myself right now. This worked fine a while back. I dont recall ever changing the `ragas` version, so it is likely coming from some hidden dependency, or similar.

that is a stupid bug to have. Thanks @vesely-david for raising this and pin-pointing exactly what the problem was too - I'm really really grateful for that ❤ 

will be released shortly

Is the version with this fix released yet? What is the ragas version which has this fix?

I think from v0.2.12 but do check out v0.2.13
Repository: explodinggradients/ragas
Issue: Issues with await
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Running the example code for LLM based metric, I run into issues with await. Not sure if this is a bug or not. Running a non-async version works fine.

**Code Examples**
```python
metric = AspectCritic(
    name="summary_accuracy",
    llm=evaluator_llm,
    definition="Verify if the summary is accurate.",
)
test_data = SingleTurnSample(**test_data)
await metric.single_turn_score(test_data) <--- SyntaxError: 'await' outside function
metric.single_turnscore(test_data)            <--- This works fine
```


Since await is being used, it is important that you use this in an asynchronous context. try 
score = await metric.single_turn_score(test_data)

thanks for helping out @IqraSiddiqui ❤ 
hoping this clears your doubt @MichaelMMeskhi ?
Repository: explodinggradients/ragas
Issue: Test data generation 
Root cause: Algorithmic Error

- [x] Multi context not working due to a bug in documents to node mapping
- [x] Error in [doc filtering](https://github.com/explodinggradients/ragas/blob/068cd548e9a2e7a5351cd3588563e112396de1cf/src/ragas/testset/testset_generator.py#L176)  
- [x] Allow control over test distribution through from_defaults. 
- [ ] Ensure distribution of resulting questions is specified
- [x] Add question filtering
Repository: explodinggradients/ragas
Issue: Document too long at transforms
Root cause: Validation Gap

- [x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I have a really long pdf (475 pages), it raises [this](https://github.com/explodinggradients/ragas/blob/b3c768b5e0526e2051a597094f3f4e45e9c1143c/src/ragas/testset/transforms/default.py#L160) error Saying "Documents appears to be too short (ie 100 tokens or less). Please provide longer documents." 
I think the problem is not the document being short but the oposite.

[code](https://github.com/explodinggradients/ragas/blob/b3c768b5e0526e2051a597094f3f4e45e9c1143c/src/ragas/testset/transforms/default.py#L77C5-L79C64) to classify documents based on size:
```py
bin_ranges = [(0, 100), (101, 500), (501, 100000)]
result = count_doc_length_bins(documents, bin_ranges)
result = {k: v / len(documents) for k, v in result.items()}
```

debuggin my code the size of my document gives 390000. The size exceeds the upper limit of the last bin. So there is no bin to place this, falling for the default condition which raises the above exception.

I think the Document should be process as those in the last bean or raise an Exception saying the document is too big.


Hey, you're right about the error. It should have been the other way. But on the other side, if you feed in a document that is 475 pages long that would cost you very much on the extraction side of things, for example, it will try to extract headings from every page in the document,etc. The smart thing to do is split the documents into several small documents and then process a part of them for test generation. We understand this is a hack, we will take this into consideration when we roll out the next big iteration on test generation. 

It seems the issue was answered, closing this now.