zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: How to check my embedding llama3 model is running for evaluate? Is Wrapper necessary?
GitHub labels: question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

For the Following Code:
llm = ChatOllama(model="llama3", base_url="http://localhost:11434/")
llm = ChatOllama(model="llama3")
embedding_model = OllamaEmbeddings(model="llama3")

result = evaluate(
dataset=dataset_subset,
llm=llm,
embeddings=embedding_model,
metrics=[
context_precision
],
run_config=RunConfig(timeout=180.0, max_workers=16)
)

**Q1. Ollama run llama 3 is working and I can talk in the chat, so do we need to input the base_url where Ollama 3 is running?
Q2. How to verify the embedding model is running as expected?
Q3. Is the LangchainLLMWrapper necessary in my case?** 

llm = LangchainEmbeddingsWrapper(embedding_model)
embedding_model = LangchainLLMWrapper(llm)
**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


<!-- Answer -->
To address your questions:

**Q1. Is it necessary to input the `base_url` where Ollama 3 is running?**

If your `ChatOllama` model is functioning correctly without specifying the `base_url`, then it might not be necessary. However, if your setup requires a specific endpoint for the model to communicate with, you should provide the `base_url` to ensure proper connectivity.

**Q2. How to verify the embedding model is running as expected?**

To verify that your embedding model is running correctly, ensure that the `evaluate` function is set up with the necessary parameters, including the dataset and metrics. The function will handle the evaluation process and return an `EvaluationResult` object containing the scores of each metric. This will help you confirm that the embedding model is functioning as expected <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py)</sup>.

**Q3. Is the `LangchainLLMWrapper` necessary in my case?**

The `LangchainLLMWrapper` is necessary if you are using a language model from Langchain, as it helps integrate the LLM with the evaluation framework. If you are not using a Langchain language model, then this wrapper is not required <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/extra/components/choose_evaluvator_llm.md)</sup>. Since you are using `OllamaEmbeddings`, which is not a Langchain model, the `LangchainLLMWrapper` is not necessary for your setup.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


there seems to be some issues with using ollama models do keep an eye on #1170 also
Repository: explodinggradients/ragas
Issue: [BUG] Issue with the Test Generation
Root cause: External Dependency Breakage

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

I was using the old version of Ragas to generate the evaluation data but it is not working anymore `generator_llm = AzureChatOpenAI(model="gpt-4o", api_version="2024-02-01")
critic_llm = AzureChatOpenAI(model="gpt-4o", api_version="2024-02-01")

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    aoai_embeddings
)

# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
testset_df = testset.to_pandas()`

The issue is even when I follow the new documentation and the example in it, it doesn't work because i encounter this error folowing the test generation in the README
ValueError: Node 5219d5c4-2785-4cee-96a0-75c5cd73b3a5 has no summary_embedding

I need to deliver an evaluation set but I can't, is because none of these is working, can you tell me which version should I install to solve my issue


<!-- Greeting -->
Hey @sarahzouinina! I'm here to help you with any bugs, questions, or contributions you have. Let's work on getting this issue sorted out together.

<!-- Answer -->
The error you're encountering, `ValueError: Node 5219d5c4-2785-4cee-96a0-75c5cd73b3a5 has no summary_embedding`, indicates that the node in your knowledge graph is missing the required 'summary_embedding' property. In the Ragas framework, each node of type `NodeType.DOCUMENT` must have a 'summary_embedding' to calculate cosine similarity between nodes <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/transforms/relationship_builders/cosine.py)</sup>.

To resolve this issue, ensure that your documents are processed with the `EmbeddingExtractor` class configured as `summary_embedder` to generate the necessary embeddings. This step is crucial in the transformation pipeline to create the 'summary_embedding' for each document node <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/transforms/relationship_builders/cosine.py)</sup>.

Regarding the version compatibility, there isn't specific information available about a version of Ragas that addresses this issue directly. However, you should ensure that you are using the latest version of Ragas, as it includes updates and changes that might affect the TestsetGenerator functionality, such as the introduction of a new Knowledge Graph and Transforms <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/migrations/migrate_from_v01_to_v02.md)</sup>. 

Make sure your setup aligns with these updates, and verify that all necessary transformations are applied to your documents before generating the test set.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


What specific version of Ragas are you using in the code you provided? Also, in the code you provided it looks like you never defined an embedding client.
Repository: explodinggradients/ragas
Issue: [R-304] support more providers for `is_finished()` logic
Root cause: Unimplemented Feature Gap

Suggest improvements to the parser for figuring out if the model finish reasons so that the default parser can be improved

you can also define your own custom parser and parse it too

<sub>[R-304](https://linear.app/exploding-gradients/issue/R-304/support-more-providers-for-is-finished-logic)</sub>

Llama models hosted on TogetherAI use `{"finish_reason": "eos"}`

```
{
  "content": "The letter 'r' appears twice in the word 'strawberry'.",
  "additional_kwargs": {
    "refusal": null
  },
  "response_metadata": {
    "token_usage": {
      "completion_tokens": 16,
      "prompt_tokens": 54,
      "total_tokens": 70,
      "completion_tokens_details": null,
      "prompt_tokens_details": null
    },
    "model_name": "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    "system_fingerprint": null,
    "finish_reason": "eos",
    "logprobs": null
  },
  "type": "ai",
  "name": null,
  "id": "run-16ae060b-4bc2-4624-b07e-dce650033d6c-0",
  "example": false,
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 54,
    "output_tokens": 16,
    "total_tokens": 70,
    "input_token_details": {},
    "output_token_details": {}
  }
}
```

hey @ahgraber thanks for reporting this as always 🙂 
I'll add it to that

While you're at it, IBM watsonx LLMs give `eos_token` when generation finishes:
```
generation_info={'finish_reason': 'eos_token'}
```

@spackows not sure if you are still waiting for an update or if you already figured it out, but I found [this solution](https://github.com/explodinggradients/ragas/pull/1728) and just added to it the 'eos_token'

hey @malikaltakrori would you be interested in adding that as a PR? would really appreciate it - if not I'll make it 🙂 

Hi @jjmachan, 
Sorry for the late reply. Github emails still go to my school's inbox. 
I would actually love to do so! (if you havn't already). 

@jjmachan Done!

thanks a million @malikaltakrori ❤
Repository: explodinggradients/ragas
Issue: Add BERTScore as potential "non-LLM" metric / MetricWithEmbedding for context-recall and context-precision
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Add BERTScore as additional evaluation metric scorer for context-precision and context-recall.  

**Why is the feature important for you?**

As a RAGAS user trying to evaluate/benchmark my RAG applications, I would like metrics that are more deterministic than LLM-as-a-Judge and more advanced and indicative of actual recommendation performance than naive string matching.  BERTscore uses embeddings to compare documents and is flexible wrt. the embedding model used.  This does not require invoking LLMs (so can be run on an analyst machine locally) and provides research-validated scores that are replicable.

**Additional context**

- [[1904.09675] BERTScore: Evaluating Text Generation with BERT](https://arxiv.org/abs/1904.09675)
- [bert_score/bert_score/score.py at master · Tiiiger/bert_score](https://github.com/Tiiiger/bert_score)
- [BERT Score - a Hugging Face Space by evaluate-metric](https://huggingface.co/spaces/evaluate-metric/bertscore)

The BERTscore package has not been updated recently; the recommended `microsoft/deberta-xlarge-mnli` model only allows ~500 tokens.  Models with longer context (`allenai/longformer-large-4096-finetuned-triviaqa` and `microsoft/deberta-v3`) do not perform as well, and modern embedding models (`nomic-embed-*`, `baii/bge-*`) are untested.

It would be interesting to see what happens using the same embedding model used in the RAG pipeline; that way you only have to embed the answer since you already have access the chunk embeddings from the RAG lookup.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


@shahules786 what do you think? could we rework https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/semantic_similarity/ for this?

I take back my suggestion re: being able to reuse embeddings -- BERTScore is calculated based on token-level embeddings.  
To prevent adding a dependency on `bert-score`, I may just try to hack a custom Metric that calls BertScore internally.

@ahgraber I have used BERTSCORE in the past, just that with the state of improvement in embeddings /LLMs I feel it doesn't make sense to adopt it anymore. It's not deterministic either. The idea of non-LLM metrics is to provide highly deterministic model-free metrics. 

But if users feel adding it to ragas adds value, we can take a look at that.  If not please feel free to close the issue. 

@shahules786  In your experience, does [cosine] similarity over text embeddings effectively replace BERTscore?
Repository: explodinggradients/ragas
Issue: [R-309] add llamaindex support for evaluation and testset generatin in v0.2
Root cause: Unimplemented Feature Gap

Tried reading through the [docs](https://docs.ragas.io/) and source-code, but couldn't understand if Llama-Index support has been removed.

**Your Question**
Was working through the `ragas` with `llama-index` [tutorial](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/llamaindex.ipynb), but `generator = TestsetGenerator.from_llama_index()` and `generator.generate_with_llamaindex_docs()` is no longer supported?
Is only `Langchain` supported hereon?

**Additional context**
Installed `ragas==0.2.2` using `!pip install git+https://github.com/explodinggradients/ragas`.


<sub>[R-309](https://linear.app/exploding-gradients/issue/R-309/llama-index-support-removed)</sub>

we didn't have port this integration yet but I can make this happen for you next week, would that be enough? 🙂 

@vikrantdeshpande09876 this is only for testset generation right?

Yes correct, testgeneration for now.
Repository: explodinggradients/ragas
Issue: [Bug] `AgentGoalAccuracyWithoutReference` is not implementing `_ascore` method
Root cause: Validation Gap

- [X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Trying out this example from the docs https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#without-reference

I cannot instantiate the metrics `AgentGoalAccuracyWithoutReference` because this class doesn't implement `_ascore` method from its parent's abstract class `Metric` (in fact, both `AgentGoalAccuracyWithoutReference` and `AgentGoalAccuracyWithReference` are not implementing this method.

Ragas version: 0.2.2
Python version: 3.9

**Code to Reproduce**
Share code to reproduce the issue

```python
from ragas.dataset_schema import  MultiTurnSample
from ragas.messages import HumanMessage,AIMessage,ToolMessage,ToolCall
from ragas.metrics import AgentGoalAccuracyWithoutReference


sample = MultiTurnSample(user_input=[])
scorer = AgentGoalAccuracyWithoutReference(llm=some_llm)
await scorer.multi_turn_ascore(sample)
```

**Error trace**
```bash
TypeError                                 Traceback (most recent call last)
Cell In[23], line 4
      1 from ragas.metrics import AgentGoalAccuracyWithoutReference
      3 sample = MultiTurnSample(user_input=sample_input_4, reference_topics=list(["science"]))
----> 4 scorer = AgentGoalAccuracyWithoutReference()
      6 scorer.multi_turn_score(sample)

TypeError: Can't instantiate abstract class AgentGoalAccuracyWithoutReference with abstract method _ascore
```
**Expected behavior**
I can instantiate the metric class normally


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I am seeing the same issue.

Looks like we are introducing more specific derived subclasses `MultiTurnMetric` and `SingleTurnMetric` to solve this Interface Segregation Principle violation , perhaps a workaround is just to implement that `_ascore` with sensible feedback to the invoker to unblock users before we fully deprecate `_ascore` e.g.?

```python
    async def _ascore(self, row: dict, callbacks: Callbacks) -> float:
        pass
```

### Version
"ragas>=0.2.1"


will be fixed shortly - thanks a lot for reporting this ❤️
Repository: explodinggradients/ragas
Issue: LLMContextPrecisionWithoutReference KeyError: 'reference'
Root cause: Validation Gap

[ yes] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
LLMContextPrecisionWithoutReference not working as mentioned
https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/context_precision/

Ragas version:
Python version:

**Code to Reproduce**
```
import asyncio
from ragas.dataset_schema import SingleTurnSample
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI
from ragas.metrics import LLMContextPrecisionWithoutReference

import os
os.environ["OPENAI_API_KEY"] = "Key"

sample = SingleTurnSample(
    user_input="Where is the Eiffel Tower located?",
    response="The Eiffel Tower is located in Paris.",
    retrieved_contexts=["The Eiffel Tower is located in Paris."], 
)
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4"))
context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm)
async def call_context():
    score = await context_precision.single_turn_ascore(sample)
    print(score)

asyncio.run(call_context())
```

**Error trace**

```
python3.10/site-packages/ragas/metrics/_context_precision.py", line 163, in _ascore
    answer=row["reference"],
KeyError: 'reference'
```

**Expected behavior**
It should give score without requiring reference.





Glad to see this error reported. I am facing a similar issue. 

this was fixed with the PR - will be out next week - thanks for raising the issue 🙂
Repository: explodinggradients/ragas
Issue: Testset generation feedback
Root cause: Documentation Deficiency

Hello,

I would like to open this issue to discuss tips and guidance related to test set generation. If you don't think this is the right place, feel free to close this issue.

Recently, I tested with Ragas 0.2, making some modifications, and generated tests for Vietnamese. I do very simple setup. I use `PyPDFLoader` to load pdf docs.

Here are some observations:

1. It generates questions for short, unattractive contexts. Perhaps some rule-based filters could help address this issue.

![image](https://github.com/user-attachments/assets/a037ce81-4464-466d-8ca0-2199fde30eb0)
![image](https://github.com/user-attachments/assets/a48e96a5-5bed-4769-b782-d438b4c5c07c)

2. I noticed that short, simple questions were generated even for long contexts. Tweaking the prompt slightly might improve the quality of the questions.

![image](https://github.com/user-attachments/assets/486ae739-da57-412a-ad1e-06acd4f6f30d)
3. I feel that the generated questions are too generic. Perhaps adding a global theme or topic could enhance the question generation process.
4. Ultimately, I believe that having the ability to use custom prompts to control the question generation would be very beneficial.

Thanks



Hey @tuan3w 
thanks for your feedback. Can you also share how many documents you used and also describe the nature of the data?
Some of these are documentation related issues, for example item 4 - you can already customise prompts using `set_prompts` and `get_prompts` method as in [here](https://docs.ragas.io/en/stable/howtos/customizations/metrics/_modifying-prompts-metrics/) but it's not documented for testset. 
Others would need bit more polishing. Test generation v3 is very capable, but it surely needs more polishing to make it better.

Also I just created #1577 to track feedbacks for it. Feel free to engage in conversations, and post your feedbacks there. 

@shahules786 can you share an example of how this would be done for testset? I use `set_prompts` to modify the prompt for SpecificQuerySynthesizer and verify that it changes the instruction and examples, but when I generate it doesn't have an effect.
Repository: explodinggradients/ragas
Issue: 对于长文本评估指标较差的问题：
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
（1）对于answer、context和groud_truth较长的情况（每项约500字），如何修改代码使评估效果提升，目前存在一些指标如context_recall、faithfulness结果为空（不是0.0）的情况。
（2）大量的answer_relevancy值为0.0是正常的吗
（3）context_precision值只有0.9999999和0.0两种情况的现象也是正常的吗？
感谢您的回答！

hey @Superyeahh could you translate this to english - would help us 🙂 

> 嘿，你能把这个翻译成英文吗 - 会帮助我们 🙂

I have re-asked this question in English in issue#1572, thank you for your reply and I look forward to your answer!

ohh got it - closing this in favour of that then 🙂
Repository: explodinggradients/ragas
Issue: Poor indicators for the assessment of long texts
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
(1) For cases where answer, context and groud_truth are long (about 500 words each), how can the code be modified to make the assessment more effective, currently there are some metrics such as context_recall, faithfulness results are empty (not 0.0).
(2) a large number of answer_relevancy value of 0.0 is normal
(3) Is it also normal to have only two context_precision values of 0.9999999 and 0.0?
Thanks for your answer!


hey @Superyeahh where you able to clear you doubts?

the recommended thing to do here is conect a logging and tracing tool like: https://docs.ragas.io/en/stable/howtos/customizations/metrics/tracing so that you results are more explainable
Repository: explodinggradients/ragas
Issue: [R-310] Test set generation improvements 
Root cause: Unimplemented Feature Gap

For Ragas 0.2, we released our third iteration of [synthetic test generation for RAG.](https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/#example_1) While developing this new approach we have kept some important feedback that we gathered from earlier versions

1. Ability to generalize over more data formats and domains. 
2. Ability to customize and create scenarios related to one's own domain. 
3. Ability to persist and resample Q&A from same list of documents w/o redoing preprocessing
4. Lower cost and faster iteration. 

We have already noted some feedback from the community and we want to welcome more, hence this discussion thread.  Please feel free share your thoughts and queries on the same. This will help us greatly improve the feature in coming weeks. 

<sub>[R-310](https://linear.app/exploding-gradients/issue/R-310/test-set-generation-improvements)</sub>

## Known issues

**Feature enhancement**
- Seeding test generation with queries #1642 

**Quality enhancement**  
- #1568
- #1477
- Add support for synthesising multiturn conversation by @ahgraber 
- https://github.com/explodinggradients/ragas/issues/1614

**Documentation improvements**
- Language adaptation for test generation    
    - #1488 
    -  #1485
- Custom prompts for test generation 
     - also noted in #1568 
     - custom True/False query generation #1620 


TODO

- [x] Improve quality of generated questions
- [x] Add node filter to avoid query creation from nodes with poor information quality. 
- [x] Heading splitter - add minium and max chunk size
- [x] Make sure default settings work in small document set (PubMed summaries,etc, <500 tokens per doc)
- [x] Fix long context issues with extraction (OOC)
- [ ] Adjust transforms/query generation for small number of documents (n=1)
- [ ] Improve sampling algorithm to sample diverse scenarios from available scenarios.
- [ ] Simplify code for writing custom scenarios 
- [ ] Add documentation for prompt adaptation in test set generation
- [ ] Add documentation for writing custom scenarios and query types, For example Yes/No questions


#1621 
#1618 
Both these items addresses quality-related issues raised in issues like #1568 

Personas and improvements to multihop look really good, I'll have to spend some time with the new multihop generation.

Update on node filter and headline splitter
- Added custom node filter to remove nodes w/o good content for query creation - such as references in research papers. One could modify and their own filters
- Tuned headline splitter for better node creation

merged #1645 

@ahgraber with new simplified way of query generation in multi hop, I am hoping it would also be easier to create multi turn conversations. Will explore this soon. 

This week's updates
1) #1661 make default settings work with different types of input docs
2) #1680 handle long context extraction

language adaptation for testgen https://github.com/explodinggradients/ragas/pull/1734
Repository: explodinggradients/ragas
Issue: RAGAs compatibility with GISKARD
Root cause: External Dependency Breakage

Currently trying to evaluate a RAG pipeline using generated tests using GISKARD and further integrating RAGAs metrics. However, after going through the documentation of GISKARD and implementation an error reading 

Package ragas.llms.prompt is missing, it is required for the computation of RAGAS metrics. You can install it with `pip install ragas.llms.prompt`.

however, ragas.llms.prompt does not exist 

hey @aranchod490 with v0.2 that is no longer available, could you use v0.1 in that case or raise an issue in GISKARD to fix this?

It seems the issue was answered, closing this now.