zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: SemanticSimilarity metric does not use specified embeddings model
Root cause: Configuration Error

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I am using a huggingface/sentence-transformers embedding model with SemanticSimilarity.  When I run the evals, I see token use and fees for OpenAI text-embedding-ada-002 model. 

Ragas version: 0.2.2
Python version: 3.11

**Code to Reproduce**
```py
from langchain_huggingface import HuggingFaceEmbeddings
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.metrics import SemanticSimilarity

my_embedding_model = HuggingFaceEmbeddings(
    model_name="nomic-ai/nomic-embed-text-v1.5",  # default context 2048 tokens
    model_kwargs={
        "device": device.type,
        "trust_remote_code": True,
        "tokenizer_kwargs": {"model_max_length": 8192},  # extend to 8192 tokens
        "model_kwargs": {"rotary_scaling_factor": 2},
        "prompts": {
            "text": "search_document: ",
            "query": "search_query: ",
            "clustering": "clustering: ",
            "classification": "classification: ",
        },
    },
    encode_kwargs={
        "normalize_embeddings": True,
        "prompt_name": "clustering",
    },
)

dataset = ...
metrics = [SemanticSimilarity(embeddings=LangchainEmbeddingsWrapper(my_embedding_model))]
evals = evaluate(dataset=dataset, metrics=metrics)
```

**Error trace**
n/a

**Expected behavior**
When a LLM or Embedding model is specified, it should override the ragas global default config.  
In this specific case, providing a custom huggingface embedding model should mean that openai is not used for embeddings.

**Additional context**
This is the specific case when I have proof of this, but I think there is a systemic issue where the ragas global default config is given priority over other specified models.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Closing as resolved (user error).  I did also ran AnswerRelevance, which _also_ requires semantic similarity, but for which I had not set an embedding model.

Suggestion: I would prefer explicit model assignments to prevent this kind of mistaken use
Repository: explodinggradients/ragas
Issue: TooManyRequests error. Even after setting Run Config
Root cause: Configuration Error

[Y ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

Trying to run Testset generation on a dataset with ~8000 llamaindex documents using Cohere's line of models. 

I constantly keep getting `cohere.errors.too_many_requests_error.TooManyRequestsError: status_code: 429, body: data=None message='You are past the per minute request limit, please wait and try again later.'`

How do I ensure rate-limiting is applied within the ragas framework? I tried setting a RunConfig like shown below, but it doesn't really help. 

```
my_run_config = RunConfig(max_workers=2, timeout=180, max_wait=120)
```

Its okay if it is slow but I just need my testset generation to run. 



Ragas version: 0.1.18
Python version: 3.11

**Code to Reproduce**

Share code to reproduce the issue
```
documents = SimpleDirectoryReader(input_files=[input_file]).load_data()
 generator_llm, critic_llm, embed_model = setup_models() #Cohere objects
 

    generator = TestsetGenerator.from_llama_index(
        generator_llm=generator_llm,
        embeddings=embed_model,
        critic_llm=critic_llm
    )

    generator.adapt(language="english", evolutions=[simple, multi_context, reasoning])
    testset = generator.generate_with_llamaindex_docs(
        documents,
        num_samples,
        distributions={simple: 0.6, reasoning: 0.2, multi_context: 0.2},
        is_async=False,
        with_debugging_logs=True,
        run_config=my_run_config
    )

```
**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


hey @tarunn2799 would it be possible to update to the latest version v0.2? we have batching in that version that should help

you can check the migration guide here: https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v01_to_v02/

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Answer correctness
Root cause: Unimplemented Feature Gap

Answer correctness given (generated_answer, ground_truth) can be measured are the linear combination of semantic similarity and factual correctness.  
TODO

- [x] Implement Semantic similarity metrics - using cross encoder
- [x] Implement Answer correctness as  _x * semantic similarity + y * factual correctness._
- [x] Verify faithfulness works with answer<->GT
Repository: explodinggradients/ragas
Issue: Issues related to the poor evaluation of long texts
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
(1) For cases where answer, context and groud_truth are long (about 500 words each), how can the code be modified to make the assessment more effective, currently there are some metrics such as context_recall, faithfulness results are empty (not 0.0).
(2) a large number of answer_relevancy value of 0.0 is normal
(3) Is it also normal to have only two context_precision values of 0.9999999 and 0.0?
Thanks for your answer!

Are you using the openai models?

> 你在使用 openai 模型吗？

No, I guess the prompt words for the evaluation indicator given in the code are in English and don't work well for other languages. I translated the answer to the question into English and now the problem is mostly solved.

@Superyeahh did you use the adapt function? 

should we close this now?

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Error occurred on basic LLMContextRecall usage, "reference   Input should be a valid string"
Root cause: Validation Gap

## input:
```
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness
from langchain_openai.chat_models import ChatOpenAI
from ragas.llms import LangchainLLMWrapper
from ragas import evaluate
from datasets import Dataset
import pandas as pd

# Example dataset
data = {
    "question": ["What is the capital of France?"],
    "answer": ["The capital of France is Paris."],
    # "retrieved_contexts": [["Paris is the capital city of France."]],
    "retrieved_contexts": [["Paris is the capital city of France."]],
    "ground_truth": [["Paris is the capital city of France."]]
}
 
df = pd.DataFrame(data)
eval_dataset = Dataset.from_dict(df)


model_name = "gpt-4o"
# model_name = "deepseek-v2"
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model=model_name))
metrics = [LLMContextRecall(), FactualCorrectness(), Faithfulness()]
results = evaluate(dataset=eval_dataset, metrics=metrics, llm=evaluator_llm)

print(results)
```

## output:
```
File "/opt/homebrew/anaconda3/envs/my_rag_checker/lib/python3.10/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for SingleTurnSample
reference
  Input should be a valid string [type=string_type, input_value=['Paris is the capital city of France.'], input_type=list]
    For further information visit https://errors.pydantic.dev/2.9/v/string_type

```

## version info
- system: mac air 2021 m1
```
>>>  python -V
Python 3.10.15

>>> pip list | grep ragas
ragas                     0.2.2
```

Actually , change `"ground_truth": [["ground_truth --- Paris is the capital city of France."]]` to `"ground_truth": ["ground_truth --- Paris is the capital city of France."]` will solve the problem!
over.
Repository: explodinggradients/ragas
Issue: Bug with context precision evaluator `LLMContextPrecisionWithoutReference`
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
The LLM-based context precision without reference class (`LLMContextPrecisionWithoutReference`) still try to access the reference answer in the SingleTurnSample instance.

```
File "/Users/miniconda3/envs/chatbot/lib/python3.11/site-packages/ragas/metrics/_context_precision.py", line 164, in _ascore
    answer=row["reference"],
```

Ragas version: 0.2.2
Python version: 3.11

**Code to Reproduce**
```
from ragas import SingleTurnSample
from ragas.metrics import LLMContextPrecisionWithoutReference

context_precision = LLMContextPrecisionWithoutReference()

sample = SingleTurnSample(
    user_input="Where is the Eiffel Tower located?",
    response="The Eiffel Tower is located in Paris.",
    retrieved_contexts=["The Eiffel Tower is located in Paris."], 
)

context_precision.single_turn_score(sample)
```

**Error trace**
```
Traceback (most recent call last):
  File "/Users/langchain-rag/test_ragas.py", line 38, in <module>
    context_precision.single_turn_score(sample)
  File "/Users/miniconda3/envs/chatbot/lib/python3.11/site-packages/ragas/metrics/base.py", line 258, in single_turn_score
    raise e
  File "/Users/miniconda3/envs/chatbot/lib/python3.11/site-packages/ragas/metrics/base.py", line 252, in single_turn_score
    score = loop.run_until_complete(
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/miniconda3/envs/chatbot/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/miniconda3/envs/chatbot/lib/python3.11/site-packages/ragas/metrics/_context_precision.py", line 147, in _single_turn_ascore
    return await self._ascore(row, callbacks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/miniconda3/envs/chatbot/lib/python3.11/site-packages/ragas/metrics/_context_precision.py", line 164, in _ascore
    answer=row["reference"],
           ~~~^^^^^^^^^^^^^
KeyError: 'reference'
```

**Expected behavior**
Output should be a context precision score


this has been fixed with #1567 so closing this now
do try out the latest version v0.2.3 and see if you get it working 🙂 

Seems like the doc is outdated - Shouldn't the method `LLMContextPrecisionWithoutReference` be precisely working without a reference ?

<img width="713" alt="Screenshot 2024-12-02 at 16 47 46" src="https://github.com/user-attachments/assets/f83879dc-f6f2-41b9-ae7f-5eed7b98b798">

[Source](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)

Thanks for clarifying !
Repository: explodinggradients/ragas
Issue: Exception raised in Job[300]: ValueError({'code': '3001', 'message': 'HTTP外部调用超时'})
Root cause: Documentation Deficiency

how can i check and fix this valueError and TimeoutError?
Evaluating: 77% 476/616 [09:54<‬01:58, 1.18it/s]2024-10-30 10:48:00.000 | INFO |
Evaluating: 77% 477/616 [09:57<‬03:09, 1.37s/it]2024-10-30 10:48:01.612 | INFO | 
Evaluating: 79% 484/616 [10:05<‬03:00, 1.36s/it]2024-10-30 10:48:11.612 | INFO | 
Evaluating: 79% 486/616 [10:08<‬02:53, 1.34s/it]Exception raised in Job[391]: ValueError({'code': '3001', 'message': 'HTTP外部调用超时'})
Evaluating: 80% 490/616 [10:14<‬03:06, 1.48s/it]2024-10-30 10:48:21.612 | INFO | 
Evaluating: 80% 491/616 [10:18<‬04:33, 2.19s/it]Prompt fix_output_format failed to parse output: The output parser failed to parse the output after 0 retries.
Prompt fix_output_format failed to parse output: The output parser failed to parse the output after 0 retries.
Prompt correctness_classifier failed to parse output: The output parser failed to parse the output after 0 retries.
Exception raised in Job[76]: RagasOutputParserException(The output parser failed to parse the output after 0 retries.)
Evaluating: 80% 493/616 [10:19<‬03:01, 1.48s/it]Exception raised in Job[132]: TimeoutError()

hey @snailmiss which model are you using for evaluation?

you can also checkout https://docs.ragas.io/en/stable/howtos/customizations/_run_config/ for how to configure timeouts

> you can also checkout https://docs.ragas.io/en/stable/howtos/customizations/_run_config/ for how to configure timeouts

@jjmachan how to set maxworks and timeout，i set maxworks=64，timeout=180，still have ValueError  and TimeoutError。

@snailmiss maybe you could try `batch` parameter for `evaluate()` function too?
![image](https://github.com/user-attachments/assets/03266464-6533-426d-b2f4-77d5c3a9ac7a)
Repository: explodinggradients/ragas
Issue: [R-311] Adding exact match caching
Root cause: Unimplemented Feature Gap

add exact match catching so to save obvious LLM requests. While semantic caching might add noise exact match caching will avoid that. It also makes testset generation and evaluation options easily re-runnable - solving the issue of lost tokens in case of an error somewhere in the pipeline

refference issues

* [periodically save current progress and restart from last save · Issue #1522 · explodinggradients/ragas](https://github.com/explodinggradients/ragas/issues/1522)

<sub>From [SyncLinear.com](https://synclinear.com) | [R-311](https://linear.app/exploding-gradients/issue/R-311/adding-exact-match-caching)</sub>

Hey @jjmachan can you assign this issue to me. I would like to take a stab at this.
Repository: explodinggradients/ragas
Issue: Hello, can RAGAS support Qwen2.5 model？
Root cause: Documentation Deficiency

Hello, can RAGAS support Qwen2.5 model？

hey @13416157913 you can support it via vllm
check out https://docs.ragas.io/en/stable/howtos/customizations/customize_models/ for more info

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: [R-312] support adding metadata into testset generation
Root cause: Unimplemented Feature Gap

add support for passing metadata into `Nodes` which exist and returning them as part of the generated testset's reference nodes for further analysis.

ref: https://discord.com/channels/1119637219561451644/1300329780041875517

<sub>[R-312](https://linear.app/exploding-gradients/issue/R-312/support-adding-metadata-into-testset-generation)</sub>

I am adding a document ID to the metadata but since this is lost in the Testset I can no longer know which document was used! Looking forward to have this implemented

As a workaround, I've updated the `SingleTurnSample` schema to include the sources field (or any other required document metadata).

During sample generation (in the `_generate_sample` function), while constructing the `reference_context`, we can also extract the specified metadata from the nodes and include it in the output.

For cases where `NodeType.CHUNK` nodes are created during splitting and don’t inherit metadata from their parent nodes, we can implement a custom transform that copies the parent metadata to each chunk node. This transform can be applied right after the splitting step.
Repository: explodinggradients/ragas
Issue: Removing `.DS_Store`

https://github.com/explodinggradients/ragas/blob/v0.0.16/tests/.DS_Store

It would be good to get these Mac artifacts out of the repo, consider adding to the `.gitignore`.

Here's a nice way to make a `.gitignore`:

```bash
curl -s \
  https://raw.githubusercontent.com/github/gitignore/master/{Global/JetBrains,Global/Vim,Global/VisualStudioCode,Global/macOS,Python}.gitignore \
  > ~/.gitignore
```
Repository: explodinggradients/ragas
Issue: Can the generated dataset be filled in the blank or true/false questions
Root cause: Documentation Deficiency

I notice that the generated test dataset is Q&A, and it means that there is a question and an answer.

I want to know whether I can use Ragas to generate other typies of queston, such as fill in the blank, true/false, or multiple-choice questions.

If it can do it, how should I write the code? Because I didn't see any relevant settings in the document.

Unfortunately I didn't find concrete answers on the internet but here is some information that might help you.

[Here](https://pixion.co/blog/ragas-test-set-generation-breakdown) it says

> From true/false and multiple choice questions all the way to the opinion-based, interpretation, comparison and critical thinking questions.

And in this [issue](https://github.com/explodinggradients/ragas/issues/609) where the problem actually has a `question_type` atribute

```
"quiz": [
{
"question": "True or False",
"answer": "false",
"question_type": "true_or_false"
}
```

Hey @SummerDream233 Yes, this can be done. Let me add to the documentation issues and create a doc on how to do it this week itself.
Repository: explodinggradients/ragas
Issue: Issues with metrics in version 0.1.18
Root cause: Documentation Deficiency

I am migrating from version 0.1.18 to version 0.2.3. I want to find information on some metrics, but I can't find the documentation for `answer_correctness` and `answer_relevancy` metrics on the website [https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/). Are these metrics being deprecated? Previously, these metrics were stored at [https://docs.ragas.io/en/stable/concepts/metrics/answer_correctness.html](https://docs.ragas.io/en/stable/concepts/metrics/answer_correctness.html) and [https://docs.ragas.io/en/stable/concepts/metrics/answer_relevance.html](https://docs.ragas.io/en/stable/concepts/metrics/answer_relevance.html), but now these links are no longer accessible.

answer relavance is: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance - it has been renamed to response relavency

answer_correctness: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness

hope it helps 🙂 


thank you~
Repository: explodinggradients/ragas
Issue: In NoiseSensitivity, the `.` should be deleted as the split condition.
Root cause: Algorithmic Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
After the sentence is split, i think the `.` should no longer be used as a split condition. Many other languages ​​do not use `.` as a split condition. Just follow the result of `sentence_segmenter`.

**Code Examples**
https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_noise_sensitivity.py#L104C13-L104C46

**Additional context**
I will provide a pr.
Repository: explodinggradients/ragas
Issue: Creating ground truth from dynamic data
Root cause: Unimplemented Feature Gap

Neat project!

Is there a way to generate the ground truth from dynamic functions, [similar to Langsmith](https://github.com/langchain-ai/langsmith-cookbook/blob/main/testing-examples/dynamic-data/testing_dynamic_data.ipynb)?

My need: I'm evaluating an agent that interacts with external APIs (specifically AWS) by generating code snippets that are then evaluated, returning JSON. The resulting JSON is fed back into the LLM and the LLM generates a natural language answer.

I'd like to have ground truth reference a function that is known to generate the data correctly for the question. This helps me test if the code generation is correct (as this is where things generally go wrong). I'd then compare the the LLM-generated JSON result to the result returned by the validated function.

...not sure if that makes sense?



Hey @itsderek23 Thank you. We have a[ diverse test data generator](https://github.com/explodinggradients/ragas/blob/main/docs/Testsetgeneration.ipynb) for RAGs. But in your case, it is for the agent use case which we are also interested in exploring. Your idea certainly makes sense to me for testing agents but there can be some pitfalls we should avoid while doing so. How can we understand your use case in detail?  I can reach out to you in person.

Thanks @shahules786!

It's for [OpsTower.ai](https://github.com/opstower-ai/llm-opstower), a DevOps AI Assistant. You can see how we do the evaluation [here](https://github.com/opstower-ai/devops-ai-open-leaderboard) (shows the datasets, a reference function, and prompts).

Happy to chat! Email is on my @itsderek23 profile.
Repository: explodinggradients/ragas
Issue: failed to parse the output
Root cause: Validation Gap

@shawnmittal @shahules786 @jannikmaierhoefer @jjmachan 

In ragas version 0.1.6
how can I fix the error : 
Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Evaluating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00,  2.44it/s] 
c:\Users\Amin\git\ragas-prototype\.venv\Lib\site-packages\ragas\evaluation.py:277: RuntimeWarning: Mean of empty slice
  value = np.nanmean(self.scores[cn])
RAGAS Evaluation Report:
{'context_utilization': nan}


I checked and the answer exist in my dataset. 
i used this code also:
    
    new_ragas_dataset = Dataset.from_list(new_ragas_dataset)