zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Printing documentation to PDF
GitHub labels: documentation

enhancement

good first issue

**Describe the Feature**
An option to print to PDF would help go over all the features of the package.

**Why is the feature important for you?**
Moving through pages makes it hard to return to the right place and understand what was already covered.


<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hey @pretty-speeches 👋🏽 

you can use this plugins like 
- https://github.com/adrienbrignon/mkdocs-exporter
- https://timvink.github.io/mkdocs-print-site-plugin/ 

to do it but I am doubtful if we should add it as a core functionality. Marking this as good first issue to get more opinions during the Pycon India 25 devsprint

Hey @jjmachan . Looks cool to me. Can I work on it? I don't know pretty much anything about ragas (except that it's written in Python) but I'm confident I can figure something out and get back to you soon.

@Sweetdevil144 Feel free to take a dig at it.


@Sweetdevil144 Did you get a chance to take a dig at this?

Hey. Been busy with sems until noe. Taking a look in it now

@anistark here is what I found :
- https://github.com/adrienbrignon/mkdocs-exporter : explicitly requires users downloading PDF to have playwright browser compatibility. Let's rule out this Option.
- https://pypi.org/project/mkdocs-with-pdf/ : Create clean and neat PDFs. However, it uses WeasyPrint which requires users to priorly do 'brew install weasyprint' for WeasyPrint to do its magic. Same issue with https://github.com/zhaoterryy/mkdocs-pdf-export-plugin
- https://timvink.github.io/mkdocs-print-site-plugin/ : Create a single Page which users need to manually print via 'Ctrl+P' option. Reference : https://github.com/mkdocs/mkdocs/discussions/3912. This seems to be our best option for now.

Let me know if you know of any other alternatives.
Repository: explodinggradients/ragas
Issue: ResponseRelevancy in 0.3.3
Root cause: External Dependency Breakage

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I updated this morning to the new Ragas version (0.3.3) and during my evaluation run i got an error during the metric answer_relevancy. 
When i downgrade the Ragas to 0.3.2 it works fine again.
Other than ResponseRelevancy i also use [LLMContextRecall, Faithfulness, ContextRelevance, AnswerAccuracy, FactualCorrectness(mode="recall"), FactualCorrectness(mode="precision")]. They work as expected. 

Ragas version: 0.3.3
Python version: 3.13.7

**Code to Reproduce**
 result_t = evaluate(
            dataset=evaluation_dataset,
            metrics=[
                LLMContextRecall(),
                Faithfulness(),
                ContextRelevance(),
                ResponseRelevancy(),
                AnswerAccuracy(),
                FactualCorrectness(mode="recall"),
                FactualCorrectness(mode="precision"),
            ],
            llm=evaluator_llm,
            embeddings=embedding_model,
            run_config=RunConfig(timeout=1000),
            raise_exceptions=True,
        )

**Error trace**

  File "/.local/share/uv/python/cpython-3.13.7-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/src/testing_service/.venv/lib/python3.13/site-packages/ragas/metrics/_answer_relevance.py", line 135, in _single_turn_ascore
    return await self._ascore(row, callbacks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/testing_service/.venv/lib/python3.13/site-packages/ragas/metrics/_answer_relevance.py", line 142, in _ascore
    responses = await self.question_generation.generate_multiple(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        data=prompt_input, llm=self.llm, callbacks=callbacks, n=self.strictness
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "src/testing_service/.venv/lib/python3.13/site-packages/ragas/prompt/pydantic_prompt.py", line 231, in generate_multiple
    output_string = resp.generations[0][i].text


PS: the website is kinda broken since the update to 0.3.3

Thanks for reporting this bug @KMSJulianBeck 

Just to confirm, are you using langchain llm?

@anistark yeah i use langchain llm. 

> [@anistark](https://github.com/anistark) yeah i use langchain llm.

Thanks. Identified it in that case. Should be fixed by next release.
Repository: explodinggradients/ragas
Issue: BaseChatModel.agenerate() missing 1 required positional argument: 'messages'
Root cause: Interface Contract Mismatch

✅ I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**

This is what **dataset** looks like
```
[
  {
    "user_input": "test",
    "retrieved_contexts": ["test"
    ],
    "response": "test",
    "reference": "test"
  }
]
```
I'm sure the format is right, cuz I followed the doc.
But it keeps showing **TypeError(BaseChatModel.agenerate() missing 1 required positional argument: 'messages')** and metrics are all NaN.

**Here's the error message**
```
/Users/arianx/PycharmProjects/bigproj/.venv/bin/python /Users/arianx/PycharmProjects/bigproj/test__.py 
EvaluationDataset(features=['user_input', 'retrieved_contexts', 'response', 'reference'], len=1)
Evaluating:   0%|          | 0/1 [00:00<?, ?it/s]Exception raised in Job[0]: TypeError(BaseChatModel.agenerate() missing 1 required positional argument: 'messages')
Evaluating: 100%|██████████| 1/1 [00:00<00:00, 528.85it/s]
{'context_recall': nan}
  user_input retrieved_contexts response reference  context_recall
0       test             [test]     test      test             NaN

Process finished with exit code 0
```
Can someone fix that for me, I'd appreciate it !

**Code Examples**

``` 
  with open("tmpds.json", "r", encoding="utf-8") as f:
        dataset = json.load(f)

    from ragas import EvaluationDataset
    evaluation_dataset = EvaluationDataset.from_list(dataset)
    
    from ragas import evaluate

    from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness
    result = evaluate(dataset=evaluation_dataset, metrics=[LLMContextRecall()])
    print(result)


    df = result.to_pandas()
    print(df)
```

Thanks for reporting this bug @arianx55678 

Just to confirm, are you using langchain llm ?

Yes I was.
I switch into llama-index llm and my problem was fixed



> Yes I was. I switch into llama-index llm and my problem was fixed

Thanks for the confirmation. 
We'll probably do a patch release with the fix.
Repository: explodinggradients/ragas
Issue: Error in ragas==0.3.3: IndexError (list index out of range) causing answer_relevancy to be NaN
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When using ragas==0.3.3, I encountered the following error:
ERROR - Exception raised in Job[10]: IndexError(list index out of range)  
This resulted in the inability to output answer_relevancy, with the final evaluation results as follows:

{'faithfulness': 0.9348, 'context_precision': 1.0000, 'answer_relevancy': nan, 'context_recall': 0.8333}  

After downgrading to ragas==0.3.1, the issue was resolved and the evaluation proceeded normally.



Ragas version: 0.3.3
Python version: 3.11

**Code to Reproduce**
metrics_list = [  
    Faithfulness(llm=self.ragas_llm),  
    ContextPrecision(llm=self.ragas_llm),  
    AnswerRelevancy(llm=self.ragas_llm, embeddings=self.ragas_embeddings),  
    ContextRecall(llm=self.ragas_llm)  
]  

evaluate_result = evaluate(  
    dataset=datasets,  
    metrics=metrics_list  
)  

return evaluate_result  




Thanks for reporting @784404847 
We're taking a look at it. 

For a little more context are you using langchain llm ?

also encounter this error, I use langchain-deepseek, is this langchain specific issues @anistark ?

> also encounter this error, I use langchain-deepseek, is this langchain specific issues [@anistark](https://github.com/anistark) ?

Most likely, yes. I could reproduce for langchain llm. Will fix it.
Repository: explodinggradients/ragas
Issue: Please stop using  deprecated/archived `nest_asyncio` package
Root cause: External Dependency Breakage

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
[nest_asyncio](https://github.com/erdewit/nest_asyncio) has been archived by the owner for over a year now.

This has created problems with asyncio support in newer versions of Python and other packages which depend on these newer features. Specifically Python 3.12+ added an optional [`loop_factory`](https://docs.python.org/3.12/library/asyncio-runner.html#asyncio.run) parameter, however the patched version of `asyncio.run` from `nest_asyncio`'s patched [run method](https://github.com/erdewit/nest_asyncio/blob/master/nest_asyncio.py#L25) doesn't pass-through this keyword argument resulting in errors like:

```
> asyncio_run(amain(), loop_factory=get_loop_factory())
E TypeError: _patch_asyncio.<locals>.run() got an unexpected keyword argument 'loop_factory'
```

Ragas version: v0.2.15 (however this dependency exists in the `main` branch: https://github.com/explodinggradients/ragas/blob/main/pyproject.toml
Python version: 3.12

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Thanks for raising the issue @dagardner-nv 
It's a valid concern. Do you've any suggestions as well?


Considering a few options:
1. Replace with [anyio](https://pypi.org/project/anyio/).
2. Change approach to thread-based.


@anistark `anyio` is interesting I wasn't familiar with it until you suggested it. I looked into this a bit, and unless I'm missing something I don't think it provides the needed functionality (with the default `asyncio` or `trio` backends). 

This was my quick-test:
```python
import anyio

class DoAsyncStuff:
    async def do_cool_stuff(self):
        print(self)

    async def get_info(self):
        return "Async Info"

    def __str__(self):
        return anyio.run(self.get_info, backend='asyncio')

def main():
    stuff = DoAsyncStuff()
    print("Should work")
    print(stuff)
    print("Won't work, can't call anyio.run from within a loop")
    anyio.run(stuff.do_cool_stuff, backend='asyncio')

if __name__ == '__main__':
    main()

```

I've made a fork of `nest-asyncio` to fix support for Python 3.12 and 3.14: [nest-asyncio2](https://github.com/Chaoses-Ib/nest-asyncio2#comparison-with-nest_asyncio). Since it's also used in my own project, I'm going to maintain it for a long time. There is no behavior change on Python < 3.12, and it's already tested with https://github.com/NVIDIA/NeMo-Agent-Toolkit, https://github.com/pyvista/pyvista, https://github.com/Chaoses-Ib/ComfyScript, so it should be a safe replacement.
Repository: explodinggradients/ragas
Issue: Passing retrieved contexts as list of strings to LangSmith integration silently fails context precision calculation
Root cause: Documentation Deficiency

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
The [current LangSmith integration documentation](https://docs.ragas.io/en/stable/howtos/integrations/langsmith/#getting-the-dataset) instructs users to pass `retrieved_contexts` as a list of strings. However, the current version of the LangSmith integration [expects a list of LangChain Document objects (`LCDocument`)](https://github.com/explodinggradients/ragas/blob/main/src/ragas/integrations/langchain.py#L114).

Ragas version: 0.3.2
Python version: 3.11.8

**Code to Reproduce**
Pass `retrieved_contexts` as a list of anything that's _not_ an `LCDocument`, and `context_precision` will be consistently calculated as 0.0 because the items will silently fail the `if isinstance(doc, LCDocument)` check.

**Error trace**
None. The check skips the item (silently fails).

**Expected behavior**
Context precision is calculated properly and returns a non-zero value when applicable.

**Additional context**
N/A

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: Discrepancy between paper and repo implementation of Context Relevance
Root cause: Documentation Deficiency

[ x ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I found that the definition of *Context Relevance* in the Ragas paper differs from the current GitHub implementation.

- **In the paper (Section 3, Context Relevance):**
  > The context c(q) is considered relevant to the extent that it exclusively contains information that is needed to answer the question.  
  > This metric aims to penalize the inclusion of redundant information.  
  > To estimate context relevance, given a question q and its context c(q), the LLM extracts a subset of sentences, S_ext, from c(q) that are crucial to answer q, using the following prompt:  
  > *Please extract relevant sentences from the provided context that can potentially help answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information". While extracting candidate sentences you’re not allowed to make any changes to sentences from given context.*  
  > The context relevance score is then computed as:  
  > CR = (number of extracted sentences) / (total number of sentences in c(q))


- **In the repo (`ContextRelevance` class):**
  The LLM is prompted to directly assign a discrete score:  
  *0 = irrelevant, 1 = partially relevant, 2 = fully relevant*  
  which is then normalized to {0.0, 0.5, 1.0}.

My question:  
- Was this change intentional (e.g., a simplification for efficiency)?  
- If so, is there any plan to provide the original “sentence-extraction ratio” version as described in the paper?  
- Should users consider the repo’s implementation as the canonical version of Context Relevance, or the paper’s?

**Additional context**

Paper reference: [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://arxiv.org/abs/2309.15217)

In the paper, Context Relevance is defined as:

> The context c(q) is considered relevant to the extent that it exclusively contains information that is needed to answer the question. This metric aims to penalize the inclusion of redundant information. To estimate context relevance, given a question q and its context c(q), the LLM extracts a subset of sentences S_ext from c(q) that are crucial to answer q. If no relevant sentences are found, the model returns "Insufficient Information". The context relevance score is then computed as:
> 
> CR = (number of extracted sentences) / (total number of sentences in c(q))

Repo reference: [`ContextRelevance` implementation](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_nv_metrics.py), where the score is instead based on discrete LLM judgments (0 = irrelevant, 1 = partially relevant, 2 = fully relevant).


The issue is difference between paper and docs. Code seems aligned.

Thanks for bringing this up @noah-13 🙌🏼 

It has undergone a few updates and hence the change you noticed.
Repository: explodinggradients/ragas
Issue: the web: app.ragas.io can not access
Root cause: Unimplemented Feature Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I rember ragas have a websit(app.ragas.io) can store datase. but now I cannot access

Does it not support in 0.3.3?


hey @landhu 

Thank you for participating in the beta version of app.ragas.io — we hope you found it valuable.
We've made the decision to sunset the app as we focus on building our new platform. If you'd like help recovering any of your data, please reach out and we'll be happy to assist.

The good news is that most of the app's capabilities are now available through open-source ragas. With the 0.3 release, you can:
Use local CSVs, Google Drive, and more to manage datasets
Export experiment results
Build and run experiments directly with ragas
We're still refining these features and adding new tutorials to the docs — so stay tuned for more updates!

Thanks again for being part of our journey,
The Ragas Team 🙂 

I'm closing this for now but feel free to reopen it or ask questions here
Repository: explodinggradients/ragas
Issue: Documentation Bug: Non-LLM metrics inheritance described incorrectly
GitHub labels: bug

documentation

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
In the documentation section [Types of Metrics in Ragas](https://docs.ragas.io/en/latest/concepts/metrics/overview/#types-of-metrics-in-ragas), there is a mistake in the description of Non-LLM-based metrics.

It currently says:
> All LLM based metrics in ragas are inherited from Metric class.

This is incorrect. Based on the codebase and usage:

- **LLM-based metrics** inherit from `MetricWithLLM`
- **Non-LLM-based metrics** inherit from `Metric`

So the line should read:
> All Non-LLM-based metrics in ragas are inherited from Metric class.

**Ragas version:**
N/A (documentation bug)

**Python version:**
N/A

**Code to Reproduce**
Not applicable (documentation bug).

**Error trace**
Not applicable.

**Expected behavior**
The documentation should correctly describe that Non-LLM-based metrics inherit from `Metric`, while LLM-based metrics inherit from `MetricWithLLM`.

**Additional context**
This may cause confusion for users implementing their own metrics.



I’d be happy to raise a PR to fix this documentation mistake in the Metrics Overview page. Please let me know if that works and I can go ahead with the change.

> I’d be happy to raise a PR to fix this documentation mistake in the Metrics Overview page. Please let me know if that works and I can go ahead with the change.

That'd be great if you can open an PR. Will get it merged asap.
Repository: explodinggradients/ragas
Issue: Documentation Out of Date for Testset Generation for RAG
Root cause: Documentation Deficiency

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Referring to this documentation: https://docs.ragas.io/en/latest/concepts/test_data_generation/rag/#example_1

Which says that when running
`
from ragas.testset.transforms.extractors import NERExtractor

extractor = NERExtractor()
output = [await extractor.extract(node) for node in sample_nodes]
output[0]
`

It will: "Returns a tuple of the type of the extractor and the extracted information."

That should look like:
`('entities',
 {'ORG': [],
  'LOC': [],
  'PER': ['Einstein'],
  'MISC': ['theory of relativity',
   'space',
   'time',
   "observer's frame of reference"]})`

Instead it returns a flat list that looks like:
`('entities',
 ['UCL',
  'Computer Science',
  'Erasmus',
  'Exchange',
  'Programme Director',
  'Module Leader',
  'Academic Year Coordinator',
  'Personal Tutor',
  'Module Catalogue',
  'Portico'])`

Ragas version: 0.3.4
Python version: 3.11.1

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
It's very frustrating that the documentation and what is in the repo is constantly out of sync. 

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


hey @malikbou thank you for taking the time to raise this 🙂. We are participating in a devsprint for pycon which is why I marked this as good-first-issue and hopefully we'll fix this soon.

I am working on a fix for this.
Repository: explodinggradients/ragas
Issue: context_relevancy not available?
Root cause: Unimplemented Feature Gap

Has context_relevancy been depricated? I am unable to import it

Hi @chukwuderaMoj , thanks for raising the issue. This will be fixed by #229 , please update ragas from the source once it is merged. We would also recommend you to try out `context_precision`, it is same as average precision mentioned [here](https://amitness.com/2020/08/information-retrieval-evaluation/#2-average-precisionap
)
Repository: explodinggradients/ragas
Issue: Unable to install NumPy
Root cause: Environment Incompatibility

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

Unable to install NumPy.

Ragas version: 0.3.4
Python version: 3.13
OS : macOS ARM (Darwin 24.6.0)

**Code to Reproduce**

followed the same steps as in the DEVELOPMENT.md - Quickstart (for developers). Fails at Make install.

**Error trace**

**Expected behavior**

NumPy should get installed without raising any C++ errors.

**Additional context**

Python 3.13 (dev/early release) on macOS ARM (darwin) and NumPy 2.0.2 is trying to build from source because there isn’t a prebuilt wheel for that Python version yet.


hey @AbhinavGavireddi would you check if #2282 fixes this for you?