zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: embedding nodes:   0%|       Segmentation fault (core dumped)
Root cause: Resource Mishandling

hi，my code is as follows:

```python
device = "cuda:2"

loader = DirectoryLoader("./knowledge_base/DT_test/content")
documents = loader.load()

for document in documents:
    document.metadata['filename'] = document.metadata['source']

pipe = pipeline(
    "text-generation",
    model=AutoModelForCausalLM.from_pretrained("/home/models/chatglm3-6b", trust_remote_code=True),
    tokenizer=AutoTokenizer.from_pretrained("/home/models/chatglm3-6b", trust_remote_code=True),
    device=device
)
generator_llm = HuggingFacePipeline(pipeline=pipe)

embeddings = HuggingFaceBgeEmbeddings(
    model_name="/home/models/bge-large-zh",
    model_kwargs={'device': device},
    encode_kwargs={'normalize_embeddings': True},
)

generator = TestsetGenerator.from_langchain(
    generator_llm=generator_llm,
    critic_llm=generator_llm,
    embeddings=embeddings
)

testset = generator.generate_with_langchain_docs(documents, test_size=3,
                                                distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},
                                                is_async=False)

print(testset.to_pandas())
```


I am getting the following error：

```shell
embedding nodes:   0%|                                                                                                                                          | 0/28 [00:00<?, ?it/s]

Segmentation fault (core dumped)
```

may I ask what this is all about?

extremely grateful！


Hey @WGS-note are you still facing this. This is likey because of how the async is iterfacing with the local model and causing a crash. Is there any chance you can run this in an vLLM instance?

closing this now. We have some fixes for this already but if your still facing this please let us know :)
Repository: explodinggradients/ragas
Issue: Answer Correctness giving wrong results for batches and single records
Root cause: Algorithmic Error

Answer correctness is giving wrong results for batches and single records. What could be the reason?

I'm affraid that could be a serious bug, could you give me more info so that we can debug? Ideally the dataset is what I would need but let me try it on my end as well

closing this since we need more info for this
Repository: explodinggradients/ragas
Issue: Is it possible to add an argument to the evaluate() function to configure the group name?
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Add an argument to the evaluate() function to define the group name instead of using the default value "ragas evaluation".

**Why is the feature important for you?**
When using langsmith to display evaluation results, different configurations or other situations may be used under the same project. 
When using the new_group() in evaluate.py, is it more flexible to pass arguments to determine the value of the argument 'name' instead of the default 'ragas evaluation'?The langsmith interface will also distinguish different results more clearly.

**Additional context**
ragas:0.1.7 
https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py
Original version：
![image](https://github.com/explodinggradients/ragas/assets/75374117/f78f6d95-ca4e-4912-b9b2-7a6d112a2dfe)
![image](https://github.com/explodinggradients/ragas/assets/75374117/1490f10a-7b74-4dd1-9782-72495f4818b2)
Modified version：
![image](https://github.com/explodinggradients/ragas/assets/75374117/07fa111b-3d36-465c-9639-7c226b9755fa)
![image](https://github.com/explodinggradients/ragas/assets/75374117/4b887460-6d6a-4251-ac24-2bd76f8ed6a9)


<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


That is a very smart suggetion, Thank you @zzzmc, we'll add it shortly and let you know 🙂
Repository: explodinggradients/ragas
Issue: Un-deprecate multiple ground truth answers?
Root cause: Unimplemented Feature Gap

**Describe the Feature**

Per the deprecation message I receive in v0.1.7:

```
passing column names as 'ground_truths' is deprecated and will be removed in the next version,
please use 'ground_truth' instead. Note that `ground_truth` should be of type string and not
Sequence[string] like `ground_truths`
```

...It seems like providing multiple alternative ground-truth answers to a question *used to be* supported but is being removed? 

I couldn't quite figure out the situation on this from the docs or issues (maybe I just missed something somewhere?), but I'd like for Ragas to support multiple reference answers per question, if possible.

**Why is the feature important for you?**

I have a document-based question answering dataset where we've seen a few cases of questions which could have multiple correct answers - sometimes very similar, but in other cases quite semantically distinct. We don't necessarily have to score the alternative GT answers separately, but it seems like we need to clarify to the evaluator LLM that either of the options are equally valid, to receive reliable judgments... And it's not clear to me how to do that going forward, if `ground_truth` needs to become a single string?

**Additional context**

Thanks for your great work on the library! Really appreciate having an evaluation tool that maintains some separation from the actual orchestration of the chain/RAG.

hey @athewsey, thanks for the kind works and that is infact a very good features request. The reason why we dropped it was because we thought you can easy split the ground truths into multiple rows and then compute it. Have you tried that approach?

The con with that approach is that now its on the users side to process the different ground truths for the same question, answer pairs. I can get on a call and set it up for you if you want and then we can add it back as a doc update.what do you think of that?
Repository: explodinggradients/ragas
Issue: Grammar and punctuation improvements in critique prompts
GitHub labels: bug

module-metrics

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I've noticed a few grammar and punctuation issues in the [critique prompts](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/critique.py). The verbs need to be in their base form to match the auxiliary verbs "does" and "is," and one sentence is missing a question mark. Fixing these will make the questions clearer and more consistent.

Ragas version: Latest (on `main` branch)
Python version: N/A


Please assign me. I'll make a PR soon for this.

thanks @ruankie for taking this up and fixing it 🙂 

btw are you on our discord channel, would love to add you to our contributors group if you want (just as a thank you for the PR)

closing this for now! 

Hey @jjmachan, glad this PR helped.
That would be great, thanks!
I am not on your Discord server at the moment. Can you please send a link?
Repository: explodinggradients/ragas
Issue: [R-256] make better example dataset for getting started
Root cause: Unimplemented Feature Gap

Add better testset using the testset generation features of 2 dataset

1. Githlab Handbook
2. Arxiv Papers on LLMs

Need to figure out the number of testsets needed

<sub>From [SyncLinear.com](https://synclinear.com) | [R-256](https://linear.app/exploding-gradients/issue/R-256/make-better-example-dataset-for-getting-started)</sub>
Repository: explodinggradients/ragas
Issue: [R-259] Which is the best LLM for evaluation?
Root cause: Documentation Deficiency

 I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Do RAGAS prompts work equally well with other LLMs like Claude 3 Sonnet and Llama 3? If not which model to choose?
Also, is there a way to print and modify the prompts?

**Additional context**

- I can see a big variation in scores across different models. 
- GPT-3.5 gives me higher scores compared to others for all metrics. Claude 3 Sonnet is like the average of Llama 3 and GPT-4 Turbo. Llama 3 and Cohere Command often give NaN output for some metrics.
- My evaluation set has 19 records.


**Radar chart comparison of scores:**
![ragas_radar_model_comp](https://github.com/explodinggradients/ragas/assets/16662309/e18ae327-ae4e-43d2-9a0f-ad0c80183c8b)


<sub>[R-259](https://linear.app/exploding-gradients/issue/R-259/which-is-the-best-llm-for-evaluation)</sub>

Hey @yadavshashank This is a very interesting analysis. This is also a problem we have been thinking about. Fundamentally what matters here is which of these LLMs would suit your level of alignment between automated scoring vs manual scoring. This is the problem we can solve by adding some form of UI component before automated scoring that would allow developers to do some level of manual checking and make sure that scores align with their judgment.  We will be working in this direction, but that will only come later.
As of today, my best recommendation is to go for the best overall performing LLMs like gpt-4 / Claude if closed source and llama-3 if opensource. 

Would love to hop on a call and help/chat with you if you're open. My cal is [here](https://cal.com/shahul-ragas/30min).
Repository: explodinggradients/ragas
Issue: is_async missing in context_relevancy in ragas 0.1.8
Root cause: Validation Gap

In ragas 0.1.8 in _context_relevancy.py

```python
    async def _ascore(self, row: t.Dict, callbacks: Callbacks, is_async: bool) -> float:
        assert self.llm is not None, "LLM is not initialized"

        if self.show_deprecation_warning:
            logger.warning(
                "The 'context_relevancy' metric is going to be deprecated soon! Please use the 'context_precision' metric instead. It is a drop-in replacement just a simple search and replace should work."  # noqa
            )

        question, contexts = row["question"], row["contexts"]
        result = await self.llm.generate(
            self.context_relevancy_prompt.format(
                question=question, context="\n".join(contexts)
            ),
            callbacks=callbacks
        )
        return self._compute_score(result.generations[0][0].text, row)

```

I think  self.llm.generate is missing is_async=is_async. I don't know if this is a bug or not, but found it weird it was the only one who lacked the argument and in my setup is failing only this metric. 

Thank you in advance!


hey @abetatos nice catch - that should be something we missed but  `is_async` is an only artifact which we will be removing with #975. so I'll clean it up with that.

so I'm guessing context _relavancy is not working for you for some other reason? could you share your logs or any more info about it
Repository: explodinggradients/ragas
Issue: division by zero when computing output score
Root cause: Validation Gap

Good time of the day,

Here https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/filters.py#L60 is a division by zero, which I encounter when using ragas with ollama+llama3-8b.

To solve it, I can replace:

```
output["score"] = sum(output.values()) / len(output.values())
```

Keeps failing at the final stage. Can be fixed with

```
if len(output.values()) == 0:
    output['score'] = .0
else:
    output["score"] = sum(output.values()) / len(output.values())
```

Hello! I'm facing the same issue. Any updates on this so far?

any update ?


Closing this as resolved.

Use `TestsetGenerator` in the new version.
Repository: explodinggradients/ragas
Issue: [R-262] ImportError: llama_index must be installed to use this function. Please, install it with `pip install llama_index`.
Root cause: Validation Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I am following ragas official document for testset generation. I am using custom llm (huggingfacepipeline) and huggingface embedding model, but whenever I am running this code, I am getting this error : ImportError: llama_index must be installed to use this function. Please, install it with `pip install llama_index`.


**Code Examples**
generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# generate testset
testset = generator.generate_with_langchain_docs(overall, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

**Additional context**
I have already install llama_index and ragas version 0.0.22. 

How can I solve this problem ?

<sub>[R-262](https://linear.app/exploding-gradients/issue/R-262/importerror-llama-index-must-be-installed-to-use-this-function-please)</sub>

that is a bug and we will fix that for you 🙂 

thanks for reporting this ❤️
Repository: explodinggradients/ragas
Issue: [R-282] use callbacks to trace reasoning of `Prompt`
Root cause: Documentation Deficiency

Hi, at first, I want to say thanks this wonderful work.

Actually, I want save the reason of evaluation.
I mean, line 69 to line 106 at https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py
Actually, it already has save function and it seems save line 69~106.
Ah, of course it is example, but I think this module generate something like this line.
I just want to save it

But I tried use this save function, it returned 'None'

So, how can I save this?
I have to build new module for this?

Thanks!

<sub>[R-282](https://linear.app/exploding-gradients/issue/R-282/use-callbacks-to-trace-reasoning-of-prompt)</sub>

hey @JinSeoung-Oh thank you for raising this issue - that is indeed a good suggestion!

can I ask you a couple of questions to better understand this?
1. where and how would you like to save this? Would just text be enough? are you expecting it to be part of the `Result` object?
2. are you using any Tracing tools? If you are, the reasoning is actually saved for you automatically today
3. how will you visualize the saved data, there will be a lot of data points to go through.

@jjmachan I have been looking for such a feature as well (for faithfulness metric, though). I would like to see the statements and respective verdict for two reasons: 

1. I would like to roughly evaluate if the numbers returned by the evaluate function are based on reasonable extracted statements and verdicts. 
2. I am considering to add those details to a labeling task such that annotators get hints as to possible errors. 

I reckoned the information would be available if I use fallbacks, but this is a fairly complex setup especially given the early stage of my investigations. Referring to the [getting started example](https://docs.ragas.io/en/stable/getstarted/evaluation.html#get-started-evaluation), if the information could be returned as part of the result of the `evaluate` function, so `result.to_pandas()` then would add it to the dataframe, that would be a handy solution for me.

@kengelbrecht got it! will use this issue to track this

Hi @jjmachan any update on this? The issue is resolved but I can't see the reasoning returned by `evaluate` function. Please let me know if I am missing anything.

Also, if that has not be been implemented yet, how can we get the reasoning?

so the recommended way is to have some sort of logging/tracing tools available ie https://docs.ragas.io/en/latest/howtos/customizations/metrics/tracing

you see the traces inside `Result.traces` object - its a bit harder to work with - which is why we are not recommending it but it's available for you to use it

Do let us know if this helps you 🙂 

Ah, I totally forgot it. 
I solved this problem built custom evaluate function.
I cannot share my code, because my company used it.

Have to check Result class in eval.
Thanks!
Repository: explodinggradients/ragas
Issue: Using context utilization without ground truth throws error message
Root cause: Validation Gap

[ x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

I am getting the error message below when not providing ground truth and using context utilization metric:

"Looks like you're trying to use 'context_precision' without ground_truth. Please use consider using  `context_utilization' instead."

Ragas version: 0.1.8
Python version: 3.10.10

**Code to Reproduce**

`
result = evaluate(dataset=dataset, llm=llm, embeddings=embeddings, metrics=[faithfulness, context_utilization])

`

This is from validation.py:

`
if (
                isinstance(m, ContextPrecision)
                and "ground_truth" not in available_columns
            ):
                extra_msg = "Looks like you're trying to use 'context_precision' without ground_truth. Please use consider using  `context_utilization' instead."

`

From _context_precision.py it can be seen the context_utilization is a subclass of context_precision so either this error message is wrong or context_utilization also does need ground truth to be specified.

`

class ContextUtilization(ContextPrecision):
    name: str = "context_utilization"
    evaluation_mode: EvaluationMode = EvaluationMode.qac

    def get_dataset_attributes(self, dataset: Dataset):
        return dataset["question"], dataset["contexts"], dataset["answer"]


context_precision = ContextPrecision()
context_utilization = ContextUtilization()

`

**Expected behavior**

I expect not getting this error message if I use context_utilization metric only without ground truth OR the error message should indicate that ground truth is required for context_utilization as well OR ground truth should not be required for context utilization.





Closing this as resolved.

Refer to latest ragas version.
Repository: explodinggradients/ragas
Issue: Reasoning behind Context Precision @ K
Root cause: Documentation Deficiency

I am wondering about the reasoning behind the definition of context precision @ K, defined [here](https://docs.ragas.io/en/stable/concepts/metrics/context_precision.html)

1. Why do you calculate a sum of precision@k for k=1,...,K instead of just using precision@K?
I understand that this takes into account the ranking order, but isn't it misleading compared to the typical definition of precision @ K? Shouldn't it be named something like summed precision @ K?
2. The sum is divided by the total number of relevant documents in the top K results, instead of K. This means that the precision @ K can be larger than 1. If one takes the example in the link where it is defined and calculates precision @ 2 for the high context example, the result is 1.5 (if one sets v_k = 1 for k=1,2). Nevertheless, the above text states that the value is between 0 and 1.

Am I missing something? Thanks!!!

Edit: I checked the code and got it now:
v_k is not something you can set yourself, but v_k is 1 if context k is relevant and 0 else. So if context k is irrelevant, precision@k * v_k = 0. And then sum_k v_k is actually the number of relevant documents in top K.
Repository: vllm-project/guidellm
Issue: Automatic performance stabilizization - so no need to set max-requests or max-seconds
Root cause: Unimplemented Feature Gap

**Description**
It would be great if we could enable automatic performance stabilization so they user would not have to estimate max-requests or max-seconds and potentially still not find the point where performance plateaus and stabilizes. it would be great to add in some automatic logic for - Something along the lines of ensuring a combination across requests and time. 

**Acceptance Criteria**
Add in automated logic to ensure GuideLLM is able to find the performance stabilization point when running a benchmark. 

Proposal: 
Can set max-seconds to accept an `auto` flag, and we can have it measure when the benchmarks have stabilized, then will stop sending new requests and times out/stops benchmarking.
Repository: vllm-project/guidellm
Issue: Enable Batch Inferencing Benchmarking Support
Root cause: Unimplemented Feature Gap

**Purpose:**
The purpose of this ticket is to support batch inference requests for GuideLLM. We will need to add in a batch-size parameter to enable the user to dictate the batch size of inference requests. 

We will want to add non-streaming batched inferencing like this PR from llm-load-test https://github.com/openshift-psap/llm-load-test/pull/66  

Acceptance Criteria:
- Add a `batch-size` parameter in for the user to dictate the batch size of inference requests 
- This should be non-streaming batching 
- Once set, send requests in batches as dictated by the parameter 
Example:
`--batch-size=16, --batch-size=32
`
Repository: vllm-project/guidellm
Issue: [Feature Request] Testing with defined prefix lengths
Root cause: Unimplemented Feature Gap

With the v1 engine of vLLM enabling prefix caching by default, we would need a way to consistently test this from the client side. vLLM built-in benchmarks already have support for this so that can serve as a sample. 

The throughput improvements as we know are considerable with a good hit rate. 

Linking to vLLM v1 blog: https://blog.vllm.ai/2025/01/27/v1-alpha-release.html#:~:text=3.%20Zero%2DOverhead%20Prefix%20Caching
Repository: vllm-project/guidellm
Issue: [Feature Request] Max error rate parameter
Root cause: Unimplemented Feature Gap

**Feature Description**
Please add a `--max-error-rate` parameter which would stop the execution of a single benchmark.

**Why is this needed?**
When running benchmarks we want to fail as early as possible in order to save costs of either GPU machine time or remote API calls s.a ChatGPT API.

**Further Description**
From benchmarks that we've been running a reasonable default value would be `0.05` i.e `5%` max error rate.
Also, if a benchmark fails due to reaching the max error rate it should be reflected in the report generated by GuideLLM.
Repository: vllm-project/guidellm
Issue: [FeatureRequest] Constraint input & output sizes no matter the dataset
Root cause: Unimplemented Feature Gap

**Description**
Currently the user can limit the prompt input size and model output size only when using emulated data type.
Please separate the input & output sizes configuration from the type of the data.

**Suggested Implementation**
Add `--data-max-input-tokens` and `--max-output-tokens` parameters to the CLI.
If the data-type is `file` or `transformers` and the input size is larger than `data_max_input_tokens` it should cut-off the input at the limit.


@markVaykhansky just following up here. How does the following proposal work?

If we need to limit the output tokens, pre-process the dataset. For input tokens, we have an environment variable `GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS`

cc @markurtz @sjmonson 

Closing this out since this was addressed in #162
Repository: vllm-project/guidellm
Issue: [Feature Request] TTFT & ITL Timeouts
Root cause: Unimplemented Feature Gap

**Description**
In addition to  `--timeout` please add two timeout related CLI arguments:
ttft_timeout_s - The max amount of seconds after which a request fails if no tokens were received
itl_timeout_s - The max amount of seconds between two tokens, after the first token is generated, above which the request fails.

**Motivation**
Currently, there’s a timeout setting for each request. However, additional timeout such as `ttft_timeout` & `itl_timeout` could be useful as they also indicate a situation where we might wanna consider the request a failure.

@markVaykhansky is this still a needed feature or has the general timeout control been able to work out?