zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Testset generator with custom LLM 
Root cause: Documentation Deficiency

Hi Team i like the way how test set generation is working with openai !
 is it possible to fit an custom LLM to generate the test set ?


Hey @VpkPrasanna , it's totally possible. Wrapping an LLM or embedding of your choice is very similar to as shown [here](https://docs.ragas.io/en/stable/howtos/customisations/index.html). You can use any LLM of your choice. for example, using bedrock

```
from langchain_community.chat_models import BedrockChat
from langchain_community.embeddings import BedrockEmbeddings
from ragas.embeddings.base import BaseRagasEmbeddings, LangchainEmbeddingsWrapper
from ragas.llms import BaseRagasLLM, LangchainLLMWrapper


config = {
   your-config
}

bedrock_model = BedrockChat(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
    endpoint_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    model_id=config["model_id"],
    model_kwargs=config["model_kwargs"],
)
bedrock_model = LangchainLLMWrapper(bedrock_model)
# init the embeddings
bedrock_embeddings = BedrockEmbeddings(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
)

bedrock_embeddings = LangchainEmbeddingsWrapper(bedrock_embeddings)


###Add custom llms and embeddings
generator_llm = bedrock_model
critic_llm = bedrock_model

from ragas.testset.extractor import KeyphraseExtractor
from langchain.text_splitter import TokenTextSplitter
from ragas.testset.docstore import InMemoryDocumentStore


splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)
keyphrase_extractor = KeyphraseExtractor(llm=generator_llm)
docstore = InMemoryDocumentStore(
splitter=splitter,
embeddings=bedrock_embeddings,
extractor=keyphrase_extractor,
)
from ragas.testset import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context

test_generator = TestsetGenerator(
generator_llm=generator_llm,
critic_llm=critic_llm,
embeddings=bedrock_embeddings,
docstore=docstore,
)

testset = test_generator.generate_with_llamaindex_docs(documents=documents[:5],
test_size=10,distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
```

It will great to add these into our documentation, it's a good first PR if you're interested. If you're interested let me know. 

I have this ready @shahules786 , in addition to  complete LLM comparison for bedrock models(Similar to the ones in Applications) . How can I help to add this to the documentation?

Docs added 

@shahules786 same not working with Azure openai models, What can be the reason?
getting error:
AttributeError: 'LangchainLLMWrapper' object has no attribute 'agenerate_prompt'
Repository: explodinggradients/ragas
Issue: [Question] Context Recall different scores for the same context (questions and GT)
Root cause: Algorithmic Error

I am wondering why the RAGAS metric "context recall" is sometimes different for the same question, the same context and the same ground truth.

I reduced my dataset to 1 question. I see the same context for each test, because the retrieval model is the same, but different answers since I test different LLMs for summarization. The ground truth is also the same for all cases.

The values I get are 0.5 for one test case and 1.0 for another. If I change the question I can get 0 and 1, for example. 

I expect "context recall" values be the same for both cases.

The same problem with "context precision".


Hey, can you share the data point for which you observed this? @checkcode555 so that we can debug the issue

Hi @shahules786, unfortunately I cannot share my data. I was wondering whether someone else experiences this problem as well. 

I have a somewhat similar issue related to how context recall is computed. For an empty list context, the context recall is always 1. The ground truth is not empty, but given that no context was retrieved from the retriever component, shouldn't the context recall be 0?

Hey @roxanamirel feel free to reinstall ragas from source and to resolve your issue.

@shahules786 I reinstalled from source "Successfully installed ragas-0.1.4.dev5+g169c6e5" and I get the same result, context recall 1, where contexts list is empty. Thanks!

Just an update, that it turns to be that LLM is involved in calculating the context recall score. It means that the usage of different LLMs would lead to different scores. What is confusing is that the scores vary a lot and can be even 0 while using gpt4 for exampe and 0.8 while using gpt3.5 for the same input data. The question is then how to interpret these scores and whether it is trustable. I was wondering whether the calculation can be more transparent and intermediate results can be provided (not only the final score as it is at the moment).
Repository: explodinggradients/ragas
Issue: Evaluate not working and set_run_config error 
Root cause: Validation Gap

Hey guys, 

I'm trying to evaluate my RAG and I have a dataset that is well fitted for the evaluate function but unfortunalty I got this error : 

**AttributeError: 'list' object has no attribute 'set_run_config'**
and it comes from : **_[m.init(run_config) for m in metrics]_** (Line 179 from the evaluation.py) 

How can I solve it please?


what is the solution for this error "Evaluation failed: 'CustomOllama' object has no attribute 'set_run_config'" Ragas?

what is the solution ?
Repository: explodinggradients/ragas
Issue: max_worker evaluate option not passed anywhere?
Root cause: Configuration Error

**Describe the bug**
the max_worker option is not passed anywhere when runing evaluate in sync mode (the default). the asyncio pool is way too large by default 


Ragas version: latest
Python version: 3.11

**Code to Reproduce**


**Error trace**

**Expected behavior**
we can control the number of workers during sync evaluate


**Additional context**


will be fixed by #643, i had a monkey patch with that is based on similar approach
```
# monkeypatch concurrency into executor with a semaphore
from ragas.executor import Executor
import asyncio
import typing
def new_wrap_callable_with_index(self, callable: typing.Callable, counter):
    if not hasattr(self, 'sema'):
        self.sema = asyncio.Semaphore(1)

    async def wrapped_callable_async(*args, **kwargs):
        async with self.sema:
            return counter, await callable(*args, **kwargs)

    return wrapped_callable_async
Executor.wrap_callable_with_index = new_wrap_callable_with_index
# end monkeypatch
```
Repository: explodinggradients/ragas
Issue: [answer_correctness] ValueError
Root cause: Algorithmic Error

**Describe the bug**
Encountering ValueError: too many values to unpack (expected 3) while calculating the answer_correctness

Ragas version:
Python version: 0.0.22

**Code to Reproduce**

Use the following dataset:

https://gist.github.com/prasadshreyas/bfe689da8b960c19faa4c245066be079

```python

from ragas.metrics import (
    answer_correctness
)
from datasets import Dataset
from ragas import evaluate

df = pd.read_csv(FILE_NAME)
ds = Dataset.from_pandas(df)


result = evaluate(
    ds,
    metrics=[
        answer_correctness
    ],
)
```

**Error trace**
```python

File "~/miniconda3/lib/python3.11/site-packages/ragas/metrics/_answer_correctness.py", line 129, in AnswerCorrectness._score_batch
  self, dataset, callbacks, callback_group_name):
    123 if prediction:
    124     prediction = [
    125         item.get(key_map[k], np.nan)
    126         for item in prediction
    127         for k in key_map.keys()
    128     ]
--> 129     tp, fp, fn = [
    130         len(item) if isinstance(item, list) else np.nan
    131         for item in prediction
    132     ]
    133     score = tp / (tp + 0.5 * (fp + fn))
    134 else:
ValueError: too many values to unpack (expected 3)

```


**Expected behavior**
No Error


Hey @prasadshreyas , please update the ragas version to 0.1. This should fix your issue. 

Feel free to close it if solved :)

Resolved, thanks!
Repository: explodinggradients/ragas
Issue: Support for Async Embeddings via michaelfeil/infinity
Root cause: Unimplemented Feature Gap

**Describe the Feature**
I would like to integrate https://github.com/michaelfeil/infinity for embeddings inference. This would automatically batch up concurrent request, uses flash-attention2, compatible with cuda, rocm, apple mps and cpu. 
Depending on the usage, you might expect between a 2.5x-22x throughput improvment / speedup over using the default hf embeddings langchain code.

Hey @michaelfeil , thank you for sharing this man. We would love to have this, are you interested in working on this?

@shahules786 Perhaps I'll just push the pure python / async into langchain directly, then it should be reusable, right?

https://github.com/explodinggradients/ragas/blob/41c0c286ae3632a77db13ddd265b7699fe6a4adc/src/ragas/embeddings/base.py#L45C1-L78C61 

he @michaelfeil this would be awesome, like you said if you drop something in langchain, that will be the easiest for you in terms of time spent. What we would love to do is build an integration doc with infinity and showcase how fast it is and how it improves people who are using Ragas as well, hopefully, driving some traffic your way. 

if you check this section we use embed a lot of chunks in sequence and is limiting on how your embedding is served. Maybe we can do a comparison here? would that be something interested?

https://github.com/explodinggradients/ragas/blob/27e1c24e4b53f8f873aa4a15db90f4ee3125c805/src/ragas/testset/docstore.py#L229-L250

we can do other comparisons too but the LLM is the limiting factor for performance so there won't be a lot of diff but the above usecase would be solid for a comparison

let me know if its something that interests you :)

Would be interesting. Fyi, I added the PR for langchain here, took me some hours over the weekend, hope its getting merged soon. https://github.com/langchain-ai/langchain/pull/17671

I would not recommend submitting the nodes (assuming each node has 1 sentence) with ThreadPoolExecutor. At a minimum batch the requests, this will help whatever backend, even API's.

Also is using `async def ` an option for the function you linked above @jjmachan ?

FYI all the thing is now finally in langchain (community, see PR mentined above). Also, you might be interested in in https://github.com/michaelfeil/infinity/blob/1fe3a34e295c95fc4a75297de842ec55c6761457/docs/benchmarks/benchmarking.md for benchmarking. 

@jjmachan It should be now in some versions of langchain.

Hey all, looking forward to contribute this.

Nah, not stale! 


I am still waiting for a freaking PR review

hey @michaelfeil - extremely sorry about this 🙁 - reopening this now and reviewing your PR right now

we have been a bit slow the last couple of months which is why this slipped through the cracks - again extremely sorry for this
Repository: explodinggradients/ragas
Issue: evaluate max_workers is unused, does not limit the number of threads in ThreadpoolExecutor in sync mode
Root cause: Validation Gap

**Describe the bug**
evaluate(..., max_workers=1) does not limit the number of workers.

Ragas version: 0.1.0
Python version: 3.11

**Code to Reproduce**
```
   amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

    result = evaluate(
        amnesty_qa["eval"].select(range(1)),
        is_async=False,
        max_workers=1,
        metrics=metrics,
        llm=watsonx_llm,
        embeddings=sentence_transformers_embedding,
    )
```

**Error trace**
None


**Expected behavior**
A clear and concise description of what you expected to happen.

Documentation of evaluate says:
    max_workers: int, optional
        The number of workers to use for the evaluation. This is used by the
        `ThreadpoolExecutor` to run the evaluation in sync mode.


Expectation is that only 1 worker sending requests when max_workers is set to 1. Api concurrency limit is 8 for me, so it's more than 8 right now.

**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: Add tests for `llm.json_load`
Root cause: Unimplemented Feature Gap

**Describe the Feature**
bugs like the one @pberger514 caught in #593 might occur down the road too. We need tests for these sections of code. The catch here is that these are tests which need LLM calls so have to add a new Pytest mark for this

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Happy to pick this one up 😃

assigned - feel free to give it a go 🙌🏽 

we might need to colaborate to figure out the right pytest magic so that everything is easy to use from the CI. 

I think we will have to test this for a couple of models like OpenAI, Bedrock, Llama2 and Mistral, have it part of the CI

@jjmachan I put a first pass in a draft. Let me know your thoughts, I'm happy to expand to other models if you think this works.
Repository: explodinggradients/ragas
Issue: Seems like `context_recall` is not working in Azure OpenAI
Root cause: Interface Contract Mismatch

**Describe the bug**
Seems like `context_recall` is not working in Azure OpenAI

Ragas version: v0.1.1
Python version: 

**Code to Reproduce**
this was from a user, but will try to reproduce this with amnesty_qa

**Error trace**
```
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'
/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py:276: RuntimeWarning: Mean of empty slice
  value = np.nanmean(self.scores[cn])

{'context_precision': 0.5000, 'faithfulness': 0.5417, 'answer_relevancy': 0.9653, 'context_recall': nan, 'harmfulness': 0.0000, 'context_relevancy': 1.000
```

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I'm facing this issue as well
Repository: explodinggradients/ragas
Issue: context_precision metrics changes?
Root cause: External Dependency Breakage

**Describe the bug**
I was trying to evaluate the metric "context_precision" using ragas. But I hit into below:
```
The metric [context_precision] that that is used requires the following additional columns ['ground_truth'] to be present in the dataset. Looks like you're trying to use 'context_precision' without ground_truth. Please use consider using  `context_utilization' instead.
```
As i remember in version 0.0.22 and even the documentation mentioned that context_precision is measured based on questions and context, it does not involved ground truth. But in latest version, it seems that not the case.

And also I bumped into issue of "'AzureChatOpenAI' object has no attribute 'set_run_config' when i using my own azure GPT4 as llm to parse into evaluate(). This, i did not faced when using version 0.0.22.

Seems like version 0.1.1 have lots of broken things.

Ragas version: 0.1.1
Python version: 3.10

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


are you seeing this warning anywhere?
```py
            logger.warning(
                "Using 'context_precision' without ground truth will be soon depreciated. Use 'context_utilization' instead"
            )
```

as for you second question could you share a code snippet so I can give you a fix?

Yes the warning i saw when i try to run the evaluate with context_precision metric without providing "ground_truth" just like what i have did using v0.0.22 but using latest version i saw that warning which i paste in my question.

For 2nd question, i just simply follow the guide mentioned in here: https://docs.ragas.io/en/v0.1.1/howtos/customisations/azure-openai.html using my azureopenai gpt as llm.
Repository: explodinggradients/ragas
Issue: To disable usage tracking, just need to RAGAS_DO_NOT_TRACK=False in env variable?
Root cause: Documentation Deficiency

**Your Question**
Is it just directly set RAGAS_DO_NOT_TRACK=false in env variable to disable usage tracking? is it work if i install the ragas from pip?


yes it works how ever you install it - no worries at all 🙂 
you can read _analytics.py for more sets.
Repository: explodinggradients/ragas
Issue: About the calculation logic of "ContextPrecision"
Root cause: Documentation Deficiency

**Your Question**
I would like to know the reason for using this to calculate grouped-responses?
`context_lens.insert(0, 0)
context_lens = np.cumsum(context_lens)`

**Code Examples**
`responses = [[i.text for i in r] for r in results.generations]
            context_lens = [len(ctx) for ctx in contexts]
            context_lens.insert(0, 0)
            context_lens = np.cumsum(context_lens)
            grouped_responses = [
                responses[start:end]
                for start, end in zip(context_lens[:-1], context_lens[1:])
            ]`

**Additional context**
My content is as follows：
questions = ['abcdefg?']
contexts = [['abc', 'efg']]
context_lens = 2
context_lens.insert(0, 0) = [0, 2]

This calculation result will be equal to 0.
I want to know what the computational logic is? Or can you tell me if the format of contentx is correct?



Hey @Gaia-Wu I think you're using an older version of ragas. Please update to latest version and follow https://docs.ragas.io/en/latest/concepts/metrics/faithfulness.html (how was this calculated section) under each metric to understand the calculation.

Feel free to reopen if needed :)
Repository: explodinggradients/ragas
Issue: evaluation.py:276: RuntimeWarning: Mean of empty slice   value = np.nanmean(self.scores[cn])
Root cause: Validation Gap

**Describe the bug**
To continue on #528 and #550, unfortunately I'm still encountering nan values sometimes.

Ragas version: v0.1.1
Python version: 3.11
Model: Azure OpenAI endpoint gpt 3.5-turbo-4k

**Code to Reproduce**
The data itself is not publicly available, but the code is a generic evaluate call
`results = evaluate(
                ragas_ds["eval"],
                metrics=[faithfulness, context_precision, context_recall, answer_similarity],
                llm=azure_model,
                embeddings=azure_embeddings,
            )`

**Error trace**
evaluation.py:276: RuntimeWarning: Mean of empty slice   value = np.nanmean(self.scores[cn])

**Expected behavior**
Ideally no nan's or a more concise description of where the 'empty slice' was encountered


**Additional context**
I've ran my evaluation set a couple of times, due to the inconsistency in where the NaNs are placed I suspect is has to do with the llm output format during metric scoring
Out of the metrics I'm testing, faithfulness seems to throw nan's more often than other metrics but as mentioned it's not consistent

I'm happy to answer any questions which could help clarify the issue
Regards, Koen
Repository: explodinggradients/ragas
Issue: TestsetGenerator with custom LLM : 'ChatOpenAI' object has no attribute 'set_run_config'
Root cause: Interface Contract Mismatch

Hi, I'm trying to use the `TestsetGenerator` to generate a dataset from Langchain document. 

Here's my code:
```python
from langchain_openai import ChatOpenAI
from langchain_community.embeddings import HuggingFaceEmbeddings
from ragas.llms.base import LangchainLLMWrapper
from ragas.embeddings.base import LangchainEmbeddingsWrapper
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain.document_loaders import TextLoader

loader = TextLoader(doc_filename)
documents = loader.load()

llm_api = ChatOpenAI(
    model=model,
    openai_api_key=key,
    openai_api_base=base,
)

embeddings = HuggingFaceEmbeddings(
    model_name=model_name_embedding,
)

generator_llm = LangchainLLMWrapper(llm_api)
critic_llm = LangchainLLMWrapper(llm_api)
embeddings_model = LangchainEmbeddingsWrapper(embeddings)

splitter = RecursiveCharacterTextSplitter(chunk_size=250, chunk_overlap=20)
keyphrase_extractor = KeyphraseExtractor(llm=generator_llm)

docstore = InMemoryDocumentStore(
    splitter=splitter,
    embeddings=embeddings_model,
    extractor=keyphrase_extractor,
)

generator = TestsetGenerator(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings=embeddings_model,
    docstore=docstore
)

testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
```

With this code I get the following error: `"AttributeError: 'ChatOpenAI' object has no attribute 'set_run_config'`.

It happens here:
```
in Evolution.set_run_config(self, run_config)
     91     self.docstore.set_run_config(run_config)
     92 if self.generator_llm:
---> 93     self.generator_llm.set_run_config(run_config)
     94 if self.node_filter:
     95     self.node_filter.set_run_config(run_config)
```

Is this an error on my part or a bug?
Thanks.

I tried again and now it seems to be working. 
**However, it doesn't seem to work when I have so many documents that the generation remains at 0%**. If I only have one, it goes to the end of the generation and the time is resonable. Is this normal?

When I have several documents, I get this error during Generation: `ValueError: Node(page_content='something', doc_id='merged') is not in list`. Any solution?
Repository: explodinggradients/ragas
Issue: Testset Generation: Random node selection improvements
Root cause: Unimplemented Feature Gap

**Describe the Feature**
The `TestsetGenerator` currently randomly selects nodes from the docstore _with_ replacement. This means the same node may have multiple testsets generated for it while other nodes have none. Requesting that this be revisited to see if it makes sense to choose nodes from the docstore without replacement. 

**Why is the feature important for you?**
For my usecase, I want to generate a question for each document I provide. This isn't possible without overriding the docstore's `get_random_nodes` implementation to choose without replacement. E.g.:
```
import numpy as np
from ragas.testset.docstore import InMemoryDocumentStore, Node
from ragas.testset.utils import rng

class NoReplacementInMemoryDocumentStore(InMemoryDocumentStore):
    def get_random_nodes(self, k=1) -> List[Node]:
        node_copies = k // len(self.nodes)
        remainder = k % len(self.nodes)

        selected_nodes = self.nodes * node_copies
        if remainder == 0:
            return selected_nodes

        random_nodes = rng.choice(
            np.array(self.nodes),
            size=remainder,
            replace=False
        ).tolist()

        selected_nodes.append(random_nodes)
        return selected_nodes
```



Hey @gleason-m thanks for the feedback. I am revisiting node selection mechanism now, I'll also keep this in mind.