zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Difference between context_utilization metric and context_relevancy metrics?
GitHub labels: question

answered

Initially context_precision work without ground truth. This have been updated where now context_precision required ground truth. Below log observed when we try to use context_utilization metrics:
Using 'context_precision' without ground truth will be soon depreciated. Use 'context_utilization' instead

Just curious what the difference between context_utilization as compared to context_relevancy as not much documentation info found for context_utilization metrics.

And also now using "context_utilization" will show the above "warning" logs, is it expected?

`context_precision` helps you answer the question "what is the signal to noise ratio of the contexts my retriever retrieved". the key is how signal is calculated
1. for `context_precision` signal is from ground_truth
2. for `context_utilisation` the signal is from the answer generated. Basically it shows how much information was actually used by the llm to generate the given answer from all the context that was provided.

I hope it answer the questions 🙂 

Ok Thanks for the information. So now i will use context_utilization metric to calculate the score between context and the answer generated by llm.
Repository: explodinggradients/ragas
Issue: Metrics answer_similarity and answer_correctness not working with VertexAIEmbeddings
Root cause: Interface Contract Mismatch

**Describe the bug**
The issue appears when using any of the above mentioned metrics with VertexAIEmbeddings. The langchain.embeddings VertexAIEmbeddings does not implement embed_text and RAGAS code attempts to call to the embedding model in the stated way.

Ragas version: 0.1.7
Python version: 3.10.12

**Code to Reproduce**
```
from langchain.chat_models import ChatVertexAI
from langchain.embeddings import VertexAIEmbeddings
from ragas.llms.base import LangchainLLMWrapper
from ragas import evaluate

from ragas.metrics import (
    answer_similarity,
    answer_correctness,
)

metrics = [
    answer_similarity,
    answer_correctness,
]

ragas_vertexai_llm = ChatVertexAI(credentials=creds)

llm_wrapper = LangchainLLMWrapper(ragas_vertexai_llm)

vertexai_embeddings = VertexAIEmbeddings(credentials=creds)

for m in metrics:
    m.__setattr__("llm", llm_wrapper)
    if hasattr(m, "embeddings"):
        m.__setattr__("embeddings", vertexai_embeddings)

result = evaluate(
    evaluation_dataset,
    metrics=metrics,
    llm=llm_wrapper,
    embeddings=vertexai_embeddings
)
```

**Error trace**

Runner in Executor raised an exception
Traceback (most recent call last):
  File "/Users/xxx/miniconda3/envs/env_3.10.12/lib/python3.10/site-packages/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/Users/xxx/miniconda3/envs/env_3.10.12/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/Users/xxx/miniconda3/envs/env_3.10.12/lib/python3.10/site-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/Users/xxx/miniconda3/envs/env_3.10.12/lib/python3.10/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/Users/xxx/miniconda3/envs/env_3.10.12/lib/python3.10/site-packages/ragas/metrics/base.py", line 116, in ascore
    raise e
  File "/Users/xxx/miniconda3/envs/env_3.10.12/lib/python3.10/site-packages/ragas/metrics/base.py", line 112, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
  File "/Users/xxx/miniconda3/envs/env_3.10.12/lib/python3.10/site-packages/ragas/metrics/_answer_similarity.py", line 65, in _ascore
    embedding_1 = np.array(await self.embeddings.embed_text(ground_truth))
AttributeError: 'VertexAIEmbeddings' object has no attribute 'embed_text'


I was able to fix the issue as follows:

```
from ragas.embeddings import LangchainEmbeddingsWrapper
vertexai_embeddings = LangchainEmbeddingsWrapper(VertexAIEmbeddings(credentials=creds))
```
Repository: explodinggradients/ragas
Issue: [R-232] Faithfullness metrics stability issue, non zero output for response with no statement
Root cause: Algorithmic Error

I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Faithfullness metrics is coming even if answer is "IDK", Ideally no statements can be generated from "IDK" (manually confirmed with using faithfullness prompt), but still faithfullness metrics is coming greater than zero, even in some cases coming 1.
Observed the issue with the latest version which is  0.1.7.

Ragas version:  0.1.7
Python version: 3.10.9

**Code to Reproduce**
```python

import ragas
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    context_relevancy,
    answer_correctness
)

from datasets import Dataset 
from ragas import evaluate
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

metrics = [ faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    context_relevancy,
    answer_correctness]

embed_model =  OpenAIEmbeddings(model="text-embedding-3-small")

data_samples = {
            'question': df_res[query_col].to_list(),
            'answer': df_res.rag_answer.to_list(),
            'contexts' : df_res.context.to_list(), 
            'ground_truth': df_res[truth_ans_col].to_list()
        }
dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset, metrics=metrics,
                        embeddings=embed_model)
score.to_pandas()
```

Sample OUTPUT -
<google-sheets-html-origin><!--td {border: 1px solid #cccccc;}br {mso-data-placement:same-cell;}-->
question | answer | contexts | faithfulness | answer_relevancy | answer_correctness
-- | -- | -- | -- | -- | --
What is the impact of foreign exchange rates o... | IDK | [Service sales primarily represent third-party... | 0.952611 | NaN | 0
What did Microsoft report as its net cash from... | IDK | [PART I\nItem 1\n \nSegment revenue and operat... | 0.847241 | NaN | 0
Were there any significant acquisitions or div... | IDK | [To mitigate credit risk, the Company generall... | 0.823912 | NaN | 0
Has Microsoft announced any major new company ... | IDK | [ \nExhibit 15.1 \nApril 25, 2023\nThe Board o... | 0.689186 | NaN | 0
How does Amazon's R&D expenditure in the most ... | IDK | [Changes in foreign exchange rates negatively ... | 0.67289 | NaN | 0




**Error trace**

**Expected behavior**
Faithfullness metics should come NaN or 0.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<sub>[R-232](https://linear.app/exploding-gradients/issue/R-232/faithfullness-metrics-stability-issue-non-zero-output-for-response)</sub>
Repository: explodinggradients/ragas
Issue: Query about calculation of metrics
Root cause: Documentation Deficiency

Hey,

Thanks for creating and maintaining this repository.

I assume that you would be using an LLM to get out the scores for each metric. Or are you guys using some bespoke model for each metric like coherence, faithfulness?

If you rely on an LLM, how do you get the score? Do you ask the LLM to spit out a score?

Please let me know!

Thanks!

Hi, @anubhav562 , thanks for asking. We do not use LLM directly to score the model since it can be biased instead we design paradigms that leverage LLMs to give certain outputs and then use algorithms to quantify the output. You can observe that all of ragas scores are between 0 and 1.  
If you want to read more about metrics refer to our [docs](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md).
Repository: explodinggradients/ragas
Issue: [R-231] `generate_text` in `LangchainLLMWrapper` ignores the value of the temperature parameter
Root cause: Algorithmic Error

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
The method `generate_text` in `LangchainLLMWrapper` doesn't take into account the temperature parameter.

Ragas version: '0.1.dev368+g8a5ff9f'
Python version: indifferent

**Code to Reproduce**
```python
    async def agenerate_text(
        self,
        prompt: PromptValue,
        n: int = 1,
        temperature: float = 1e-8,
        stop: t.Optional[t.List[str]] = None,
        callbacks: Callbacks = None,
    ) -> LLMResult:
        temperature = self.get_temperature(n=n)  # <- this function sets a temperature value regardless of the one passed as a parameter
```

**Error trace**
This "bug" won't throw any exception but the code won't work as expected.

**Expected behavior**
Take the given temperature value into account if explicitly provided. Now, since the temperature has a default value already in the signature, it is not straightforward to know whether the param has been defined, or it implicitly has taken the default value. Therefore the signature should be slightly modified (at the "typing" level).

**Additional context**
Since `agenerate_text` is a concrete implementation of an abstract method in `BaseRagasLLM`, probably the abstract class signature would need to be updated too. I don't see in the repo any place where removing the default value in the signature would cause any problem.

I will open myself a PR with a solution proposal.

<sub>[R-231](https://linear.app/exploding-gradients/issue/R-231/generate-text-in-langchainllmwrapper-ignores-the-value-of-the)</sub>

closed with PR, thanks again @HerrIvan 🙂 ❤️
Repository: explodinggradients/ragas
Issue: [R-230] Adapt prompts "in place" also for cached prompts
Root cause: Algorithmic Error

**Describe the Feature**
Adapt Prompts "in place", also in case they are retrieved from the cache.

**Why is the feature important for you?**
If you want to generate a testset in a non-english language, the right approach is tofirst adapt all the generation/filtering prompts and then run the generation. 
However, this works only if you adapt the prompts the first time. In that case, the prompt instances get changed "in place", that means that when the `testset.generate` accesses the prompts, these prompts are the translated ones. However, if the prompts are retrieved from the cache, they are not changed "in place", which means that when TestsetGenerator.generate accesses the prompts, it uses the original ones in English.

**Additional context**
This is half a feature request, half a bug, since the behavior of the "adapt" function is not consistent: Prompts in the cache are not modified "in place", while instances of prompts not in the cache are modified.

See below in `src/ragas/llms/prompt.py`, when a prompt is "translated/adapted" the translated values are stored in the prompt instance. On top of that, the instance is returned.

```python
        for i, example in enumerate(grouped_results):
            [...]
            self.examples[i] = example_dict

        self.language = language

        # TODO:Validate the prompt after adaptation

        return self
```

However, in the same function, whenever there is a hit in the `cache` the current prompt instance isn't modified (ie, nothing is assigned to the `self` var:

```python
        if os.path.exists(os.path.join(cache_dir, language, f"{self.name}.json")):
            return self._load(language, self.name, cache_dir)
```

(The function `self._load` is a class method which doesn't modify the instance).

I will open a PR to address this issue.


<sub>[R-230](https://linear.app/exploding-gradients/issue/R-230/adapt-prompts-in-place-also-for-cached-prompts)</sub>

Hey @HerrIvan thanks for pointing this out. Seems like a bug to me. 

also tagging #890 

This has been fixed with v0.2 - I know finally 😅 🎉

do checkout the docs here: https://docs.ragas.io/en/stable/howtos/customizations/metrics/_metrics_language_adaptation/
reference here: https://docs.ragas.io/en/stable/references/prompt/#ragas.prompt.PromptMixin

and if you're migrating from v0.1 check out the migration docs here: https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v01_to_v02

could you check it out and verify if not feel free to comment here and I'll help you out - really sorry again that it tool this while
Repository: explodinggradients/ragas
Issue: [R-223] Ragas Langfuse Integration is not working with latest version of Ragas
Root cause: External Dependency Breakage

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Using the code sample as mentioned here: https://docs.ragas.io/en/latest/howtos/integrations/langfuse.html
Getting the error `AttributeError: 'Faithfulness' object has no attribute 'score_single'`

Full stack trace:
`
Ragas version:
Python version:

**Code to Reproduce**
https://docs.ragas.io/en/latest/howtos/integrations/langfuse.html

**Error trace**
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[15], line 23
     15 trace.span(
     16         name="generation",
     17         input={"question": question, "contexts": contexts},
     18         output={"answer": answer}
     19     )
     22 # compute scores for the question, context, answer tuple
---> 23 ragas_scores = score_with_ragas(question, contexts, answer)
     24 ragas_scores
     25 for m in metrics:

Cell In[10], line 5, in score_with_ragas(query, chunks, answer)
      3 for m in metrics:
      4     print(f"calculating {m.name}")
----> 5     scores[m.name] = m.score_single(
      6         {"question": query, "contexts": chunks, "answer": answer}
      7     )
      8 return scores

AttributeError: 'Faithfulness' object has no attribute 'score_single'

**Expected behavior**
It should run error free and the trace should be pushed to langfuse.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<sub>[R-223](https://linear.app/exploding-gradients/issue/R-223/ragas-langfuse-integration-is-not-working-with-latest-version-of-ragas)</sub>
Repository: explodinggradients/ragas
Issue: [R-258] Question about official docs of Context Precision.
Root cause: Documentation Deficiency

What I understood is that Following case is calculation of Context Precision at K=2.

For that case, Last part's denominator must be 2 not 1.

Is this right?

If i didn't understand well, could you explain more about this?? 
Following is what i'm thinking.

1. precision at 1
FP at 1 =1, TP at 1 =0 -> precision at 1 = 0
2. precision at 2
FP at 2 =1 , TP at 1 =1 -> precision at 2 = 1/2 =0.5 (since we consider top 2 resulte in context chunk items)
3 finally, we have
Context Precision at K=2 is
(1/2)x(preicision at 1 x indicator at 1 + preicision at 2x indicator at 2) = (1/2)x(0x1+0.5x1)=0.25

thanks.  

![ragas_docs](https://github.com/explodinggradients/ragas/assets/131431642/8d2dcfaf-a8de-4b63-a252-ca4cdb3f3a8e)


<sub>[R-258](https://linear.app/exploding-gradients/issue/R-258/question-about-official-docs-of-context-precision)</sub>

Hi, do you understand? I have this confusion too.

> Hi, do you understand? I have this confusion too.

No.. I'm stuck in same situation.

hey @WGS-note and @AxelJongoPark, thank you for bringing this up 🙌🏽 

@shahules786 could you help them out?

Metrics were updated as part of v0.2. Please check it out https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/
Repository: explodinggradients/ragas
Issue: AzureOpenAIEmbeddings with custom endpoint fails
Root cause: Configuration Error

[ x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Not able to use Embeddings based metrics as I'm getting the message above. I am using a custom Azure OpenAI endpoint and leverage langchain, which otherwise works perfectly well with my setup.

I wonder if this might be a regression of:
[https://github.com/explodinggradients/ragas/issues/545](https://github.com/explodinggradients/ragas/issues/545)

Ragas version: 0.1.7
Python version: 3.9.6

**Code to Reproduce**

```
from ragas.metrics import (
    answer_similarity,
    answer_correctness
)

from ragas import evaluate, adapt


chat_model = AzureChatOpenAI(
    openai_api_base=xxxxxx,
    openai_api_version=xxxxxx,
    deployment_name=xxxxxx,
    openai_api_key=xxxxxx,
    max_tokens=5000,
    temperature=0.0
    )

embeddings_model = AzureOpenAIEmbeddings(
    azure_deployment=xxxxxx,
    azure_endpoint=xxxxxx,
    api_version=xxxxxx,
    api_key=xxxxxx
)

metrics = [
    answer_similarity,
    answer_correctness
]

result = evaluate(
    dataset_evaluation_1,
    metrics=metrics,
    llm=chat_model,
    embeddings=embeddings_model
)
```

**Error trace**
The issue seems to be that it is trying to connect to the Azure OpenAI stardard endpoint (to which I don't have network access) rather than the one configured in the code:

 OSError: Tunnel connection failed: 407 Proxy Authentication Required.
The above exception was the direct cause of the following exception:
urllib3.exceptions.ProxyError: ('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Authentication Required'))

urlib3.exception.MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Authentication Required'))


**Additional context**
Have tried wrapping the model in `LangchainEmbeddingsWrapper` and that still results in the 407 because it tries to access the default Azure OpenAI endpoint (openaipublic.blob.core.windows.net) rather than my custom one.

Have also tried passing the models per metric:

```
for metric in metrics:
   metric.embeddings=embeddings_model
```

 and that results in the error:
`'AzureOpenAIEmbeddings' object has no attribute 'embed_texts'`
Repository: explodinggradients/ragas
Issue: [R-224] Ragas integration with Langfuse to trace both llm outputs and scores in the same place
Root cause: Unimplemented Feature Gap

[x ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.
I checked and tested the integration of `ragas` with `Langfuse` as demonstrated [here](https://docs.ragas.io/en/latest/howtos/integrations/langfuse.html). Although a lot of the code in that page did not work, I used the `evaluate` function to get the scores, and I logged all the scores in `Langfuse`. 

**My Question: how to customize llm traces with ragas**

Is there any way we can log the llm generations when evaluating the results using some given `trace_ids` ?
I already have the generations logged into langfuse, but the problem is I want to log these generations into some other corresponding trace_ids I already generated. 
Is this possible ? 

**Code Examples**
```
        score = evaluate(
            dataset,
            metrics=[
                context_precision,
                context_recall,
                context_relevancy,
            ],
            llm=llm_ragas,
            embeddings=embedding_model_ragas,
        )

       # at this point, the traces of all the llm generations done by ragas are already logged into langfuse. 
       # is there a way we can log them using some given trace ids ? 
        ....
        for k, v in score_row.items():
            try:
                if v not math.isnan(v):
                    langfuse.score(trace_id=trace_id, name=k, value=v)
                else:
                    logger.error(f"Score {k} is None")
            except Exception as e:
                logger.error(f"Error logging score: {e}")
....

```

I believe these issues are related: https://github.com/explodinggradients/ragas/issues/896, https://github.com/explodinggradients/ragas/issues/893

**Additional context**
Here are the requirements used. 

langfuse==2.26.3 
litellm==1.35.12
ragas==0.1.7

<sub>[R-224](https://linear.app/exploding-gradients/issue/R-224/ragas-integration-with-langfuse-to-trace-both-llm-outputs-and-scores)</sub>

Hey @databill86 thanks for raising this - will be fixing these issues shortly and doing a release with the fixes for you 🙂 

cheers
Repository: explodinggradients/ragas
Issue: [R-233] How to generate a testset with VertexAI
Root cause: Documentation Deficiency

Currently getting an error. See https://github.com/explodinggradients/ragas/issues/735 for more info on the error.

If anyone has working code for testset generation using VertexAI i would love to see it. Thanks!

<sub>[R-233](https://linear.app/exploding-gradients/issue/R-233/how-to-generate-a-testset-with-vertexai)</sub>
Repository: explodinggradients/ragas
Issue: Prevent hallucination in candidate sentence extraction
Root cause: Validation Gap

For measuring context_relevancy, the LLM tries to extract sentences from a given context that is actually useful to answer the question. But in this process, there is a slight chance that the LLM might hallucinate and give out a sentence that is not exactly present in the given context.

- It would be good to have some kind of [check](https://github.com/explodinggradients/ragas/blob/1cda1c7dea115fad524fa90a3273070169dd6868/src/ragas/metrics/context_relevance.py#L185) to ensure that only sentences present in the given context are selected by the LLM. 

- Another way to look at it is to change the [prompt](https://github.com/explodinggradients/ragas/blob/1cda1c7dea115fad524fa90a3273070169dd6868/src/ragas/metrics/context_relevance.py#L18) so that the LLM outputs the indices of candidate sentences from a given context. This could bypass the need for an extra check. 

Would like to work on this issue!
Repository: explodinggradients/ragas
Issue: Duplicate reference
Root cause: Validation Gap

https://github.com/explodinggradients/ragas/blame/65b77f5d8a300107ab964de6af060a8207dff260/src/ragas/validation.py#L31

Shouldn't it be referenced like this: 
ds.features[column_names].dtype
not with two times feature(s) ?

Eg. this for me works:
ds.features["contexts"].dtype == "string"

while as it is implemented atm throws an undesired error:
ds.features["contexts"].feature.dtype == "string"

hey @CvH2020 - welcome to the community 🙂 

could you give me some more information about this though 
1. ragas, dataset version
2. trace of the error message
3. anything else you feel will help us 

I ran the code you suggested which throws list not string for contexts

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: [R-221] test issue

this is a test issue

<sub>From [SyncLinear.com](https://synclinear.com) | [R-221](https://linear.app/exploding-gradients/issue/R-221/test-issue)</sub>

this is a test commend from github

this is a test commend from linear

<!-- From jithin on Linear. LinearCommentId:8889f27b-94e2-49b7-aa42-6f188a6f132b: -->