zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Test set generation with llm from LlamaCPP (gguf).
Root cause: Interface Contract Mismatch

**Your Question**
I am trying to generate a test set using this as an llm stable-code-3b-q4_k_m.gguf and this as embeddings sentence-transformers/all-mpnet-base-v2.

Initially, when I was importing embeddings using from llama_index.embeddings.huggingface import HuggingFaceEmbedding, during test set generation there was this error: 'HuggingFaceEmbedding' object has no attribute 'aembed_documents'

After that, I load the embeddings using from ragas.embeddings import HuggingfaceEmbeddings and I no longer get that error.

I get a new one, regarding LlamaCPP. The error is this: 'LlamaCPP' object has no attribute 'agenerate_prompt'.
And I am importing the model using this:
from llama_index.llms.llama_cpp import LlamaCPP

llm = LlamaCPP(
    model_path="../stable-code-3b-q4_k_m.gguf", context_window=4096
)

**Code Examples**

This is my code if you want to reproduce the error:

from llama_index.llms.llama_cpp import LlamaCPP

llm = LlamaCPP(
    model_path="../stable-code-3b-q4_k_m.gguf", context_window=4096
)
from ragas.embeddings import HuggingfaceEmbeddings

embeddings = HuggingfaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2")

from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context

TEST_SIZE = 25

# generator with openai models
generator = TestsetGenerator.from_langchain(
    generator_llm=llm, critic_llm=llm, embeddings=embeddings
)

# set question type distribution
distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}

# generate testset
testset = generator.generate_with_llamaindex_docs(
    documents, test_size=TEST_SIZE, distributions=distribution,
)

test_df = testset.to_pandas()
test_df.head()

**Additional context**
Lastly, as I read in the documentation I tried to pass the llm in a langchain wrapper (from langchain_core.language_models import BaseLanguageModel) but it fails with:

Can't instantiate abstract class BaseLanguageModel with abstract methods agenerate_prompt, apredict, apredict_messages, generate_prompt, invoke, predict, predict_messages

Thank you in advance.
Repository: explodinggradients/ragas
Issue: test
GitHub labels: question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

**Code Examples**
```python
#This is python code
def add(args):
  sum = arg1 + arg2
  return sum

**Additional context**
Anything else you want to share with us?
Repository: explodinggradients/ragas
Issue: AnswerSimilarity: a suggested implementation for Cross Encoder similarity
Root cause: Unimplemented Feature Gap

**Describe the Feature**
The RAGAS documentation (https://docs.ragas.io/en/stable/concepts/metrics/semantic_similarity.html) states that:
> This evaluation utilizes a cross-encoder model to calculate the semantic similarity score.

but does not. It uses cosine similarity between answer and ground truth instead.

**Why is the feature important for you?**
Having a choice between using a document level and token level similarity would be helpful.

**Additional context**
I found the ColBERT scoring function in https://medium.com/@zz1409/colbert-a-late-interaction-model-for-semantic-search-da00f052d30e (look for ColBERT scoring function).

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->

Using the above formula I am filling in the if-block of the `AnswerSimilarity._ascore` method. The only difference between the formula and this implementation is that I normalize the score with the number of tokens in the `ground_truth` to keep the score in the same [0, 1] range as cosine similarity.

```
    async def _ascore(
        self: t.Self, row: t.Dict, callbacks: Callbacks, is_async: bool
    ) -> float:
        assert self.embeddings is not None, "embeddings must be set"

        ground_truth = t.cast(str, row["ground_truth"])
        answer = t.cast(str, row["answer"])

        if self.is_cross_encoder and isinstance(self.embeddings, HuggingfaceEmbeddings):
            words_1 = [w for w in nltk.word_tokenize(ground_truth)]
            words_2 = [w for w in nltk.word_tokenize(answer)]
            embedding_1 = np.array(await self.embeddings.embed_text(words_1))
            embedding_2 = np.array(await self.embeddings.embed_text(words_2))
            similarity = np.array([(np.sum(
                np.max(
                    np.dot(embedding_1, embedding_2.T),
                    axis=1
                )
            ) / len(words_1)])
        else:
            embedding_1 = np.array(await self.embeddings.embed_text(ground_truth))
            embedding_2 = np.array(await self.embeddings.embed_text(answer))
            # Normalization factors of the above embeddings
            norms_1 = np.linalg.norm(embedding_1, keepdims=True)
            norms_2 = np.linalg.norm(embedding_2, keepdims=True)
            embedding_1_normalized = embedding_1 / norms_1
            embedding_2_normalized = embedding_2 / norms_2
            similarity = embedding_1_normalized @ embedding_2_normalized.T
            score = similarity.flatten()

        assert isinstance(score, np.ndarray), "Expects ndarray"
        if self.threshold:
            score = score >= self.threshold

        return score.tolist()[0]
```
Repository: explodinggradients/ragas
Issue: Testset generation example from documentation does not generate 'answer' column
Root cause: Documentation Deficiency

**Question:**
1. The code example does not generate answers in testset generation example. Am I understanding something wrong?
2. Which prompt is used for answer generation? 

**Code example:**
https://docs.ragas.io/en/stable/getstarted/testset_generation.html#data-generation


I too am facing the same issue; I have tried out initiating the critic and generator with gpt4 and gpt3.5 instances with temperatures 0.7 and 0.2 respectively. Followed the exact example from the above documentation..

Facing the same issue as well...

++ 1

Me too, only below column generated :
question | contexts | ground_truth | evolution_type | metadata | episode_done

Same here. Followed documentation closely, but using Claude 3 Haiku via Bedrock. The question generation works as expected but not the 'answer' creation.

Might be just a missing update of the docs, according to [https://github.com/explodinggradients/ragas/issues/875](https://github.com/explodinggradients/ragas/issues/875)

But then, the evaluation step needs to be updated as it is still requesting the 'answer' column.

+1

I too facing the same issue, using Azure open AI api to create test data, answer column is missing.
![image](https://github.com/explodinggradients/ragas/assets/37367345/8972ed5f-89a9-447c-b197-ed76d2dc457f)

Dataset({
    features: ['question', 'contexts', 'ground_truth', 'evolution_type', 'metadata', 'episode_done'],
    num_rows: 2
})


+1

@xdurfina @crystlbat @raffy4284 @dianavintila @chong-w @lukschack @sdiazben @manojpraba @ayoubazaouyat were you folks able to figure this out?

it is an error in the documentation from our end. I will fix this shortly

really sorry about the delay and thanks for pointing it out!

also fix docs for [Compare Embeddings for retriever | Ragas](https://docs.ragas.io/en/stable/howtos/applications/compare_embeddings.html)
Repository: explodinggradients/ragas
Issue: [R-246] Could it be a mistake in the prompts?
Root cause: Validation Gap

Hello,

I assume that there is a minor mistake in find_relevant_context_prompt under the testset/prompt.py. 

relevant contexts seem not right to me. Is it possible to be?
The first one is "relevant_contexts": [1, 2, 3],
The second one is "relevant_contexts": [1, 3],

Also, filter_question_prompt might need to be checked.

Thank you for your time.
```py
find_relevant_context_prompt = Prompt(
    name="find_relevant_context",
    instruction="Given a question and set of contexts, find the most relevant contexts to answer the question.",
    examples=[
        {
            "question": "What is the capital of France?",
            "contexts": [
                "1. France is a country in Western Europe. It has several cities, including Paris, Lyon, and Marseille. Paris is not only known for its cultural landmarks like the Eiffel Tower and the Louvre Museum but also as the administrative center.",
                "2. The capital of France is Paris. It is also the most populous city in France, with a population of over 2 million people. Paris is known for its cultural landmarks like the Eiffel Tower and the Louvre Museum.",
                "3. Paris is the capital of France. It is also the most populous city in France, with a population of over 2 million people. Paris is known for its cultural landmarks like the Eiffel Tower and the Louvre Museum.",
            ],
            **"output": {
                "relevant_contexts": [1, 2],**
            },
        },
        {
            "question": "How does caffeine affect the body and what are its common sources?",
            "contexts": [
                "1. Caffeine is a central nervous system stimulant. It can temporarily ward off drowsiness and restore alertness. It primarily affects the brain, where it alters the function of neurotransmitters.",
                "2. Regular physical activity is essential for maintaining good health. It can help control weight, combat health conditions, boost energy, and promote better sleep.",
                "3. Common sources of caffeine include coffee, tea, cola, and energy drinks. These beverages are consumed worldwide and are known for providing a quick boost of energy.",
            ],
            **"output": {"relevant_contexts": [1, 2]},**
        },
    ],
    input_keys=["question", "contexts"],
    output_key="output",
    output_type="json",
    language="english",
)

```

<sub>[R-246](https://linear.app/exploding-gradients/issue/R-246/could-it-be-a-mistake-in-the-prompts)</sub>

@shahules786 could you check this one?

This was fixed in ragas v0.2 - so closing it for now but if you still face it just comment down below and we'll help you out 🙂 

<!-- From jithin on Linear. LinearCommentId:cecdfdc2-4758-44d8-abe5-504f318b798f: -->
Repository: explodinggradients/ragas
Issue: [R-241] docs on how to custumize LLMs in test set generation
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Was anyone able to generate synthetic dataset using a custom LLM ? The documentation completely skips over the specifics here. It just asks us to either create a langchain instance (supported models are only few) and wrap it around new wrappers OR base them out of BaseRagasLLM 
https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html#llms

But when you do that, you also have to define agenerate_prompt, agenerate_text, generate_prompt and generate_text. We dont know when these functions are called by the test set generation and what to return at each function, etc. Its all a black box with no examples. My experiments tend to repeat multiple times with no output and just fails after a while.

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


<sub>[R-241](https://linear.app/exploding-gradients/issue/R-241/docs-on-how-to-custumize-llms-in-test-set-generation)</sub>

hi，have you solved it?

Unfortunately, I did not receive a solution to this yet. 

Any news about the solutions now? Facing the same problem. 

@projects-g @WGS-note @Zzrjin extremely sorry about the delay but we had improved the docs for this

take a look at https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html#test-set-generation

and it should help

closing this for now but feel free to comment below if you have any issues and I'll reopen
Repository: explodinggradients/ragas
Issue: [R-239] Why the later parts testset will never be accessed?
Root cause: Algorithmic Error

The code in testset/generator.py make me confused:
```py
current_nodes = [
            CurrentNodes(root_node=n, nodes=[n])
            for n in self.docstore.get_random_nodes(k=test_size)
        ]
        total_evolutions = 0
        for evolution, probability in distributions.items():
            for i in range(round(probability * test_size)):
                exec.submit(
                    evolution.evolve,
                    current_nodes[i],
                    name=f"{evolution.__class__.__name__}-{i}",
                )
                total_evolutions += 1
```
**Your Question**
Assuming that in the code, the current_nodes index range generated according to each distribution probability traversal will be concentrated in the front part of the current_nodes list, and the later parts will never be accessed?

**Code Examples**
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.3, multi_context: 0.2})

Using the sample code from Ragas documents, the simple testset will use the No. 1-5 random selected documents, the reasoning testset will use the No. 1-3 random selected documents, the multi_context testset will use the No. 1-2 random selected documents. The No. 6-10 documents will never be used.


<sub>[R-239](https://linear.app/exploding-gradients/issue/R-239/why-the-later-parts-testset-will-never-be-accessed)</sub>

wow that is an embarrassing bug @princepride - thanks a lot for reporting that 😅 

will fix it shortly

> wow that is an embarrassing bug @princepride - thanks a lot for reporting that 😅 
> 
> will fix it shortly

I try to fix this bug and merge it. https://github.com/explodinggradients/ragas/pull/880
Repository: explodinggradients/ragas
Issue: openai.UnprocessableEntityError: Error code: 422
Root cause: Interface Contract Mismatch

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

Ragas version:
Python version:

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**
openai.UnprocessableEntityError: Error code: 422 - {'detail': [{'type': 'string_type', 'loc': ['body', 'input', 'list[str]', 0], 'msg': 'Input should be a valid string', 'input': [791, 1561, 9635, 33617, 617, 2834, 279, 7445, 20904, 264, 3335, 4848, 3115], 'url': 'https://errors.pydantic.dev/2.6/v/string_type'}, {'type': 'string_type', 'loc': ['body', 'input', 'str'], 'msg': 'Input should be a valid string', 'input': [[791, 1561, 9635, 33617, 617, 2834, 279, 7445, 20904, 264, 3335, 4848, 3115]], 'url': 'https://errors.pydantic.dev/2.6/v/string_type'}]}


**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Same here.
Repository: explodinggradients/ragas
Issue: torch.cuda.OutOfMemoryError: CUDA out of memory.
Root cause: Documentation Deficiency

how to call torch.cuda.empty_cache() to free gpu cache?


Is it possible to free gpu cache after every metics or every llm query?

could you please provide a bit more context to tackle this
- which ragas version
- what you're trying to accomplish, code snippet

it would help us answer and solve your question better :) 

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: [R-238] Answer relevance metrics does not require embedding match if answer is non-commital
Root cause: Unimplemented Feature Gap

**Describe the Feature**
The answer relevance metrics multiplies the cosine similarity with noncommittal. So, if we check the answer is noncommittal, we do not need to call the embedding model

**Why is the feature important for you?**
It will improve the performance of the metrics

**Additional context**
None

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


<sub>[R-238](https://linear.app/exploding-gradients/issue/R-238/answer-relevance-metrics-does-not-require-embedding-match-if-answer-is)</sub>

that is a smart point - thanks @rajib76 for suggesting that 🙂 

we will be working on this but let me know if you want to take a stab at it - we can work on this together too

I can take a stab at it Jithin. Let me check the code. I am ok to work with you on this.

I am thinking to make the change as below. But I am not able to find the documentation on how I can build the code and test it.

    def _calculate_score(
        self, answers: t.Sequence[AnswerRelevanceClassification], row: t.Dict
    ) -> float:
        question = row["question"]
        gen_questions = [answer.question for answer in answers]
        committal = np.any([answer.noncommittal for answer in answers])
        if all(q == "" for q in gen_questions):
            logger.warning(
                "Invalid JSON response. Expected dictionary with key 'question'"
            )
            score = np.nan
        else:
            # Changes by Rajib #
            if not committal:
                score = 0
            else:
                cosine_sim = self.calculate_similarity(question, gen_questions)
                score = cosine_sim.mean()
            # Changes by Rajib #
            # 
            # cosine_sim = self.calculate_similarity(question, gen_questions)
            # score = cosine_sim.mean() * int(not committal)

shahul would be the best person to take this forward
Repository: explodinggradients/ragas
Issue: [R-237] ValueError: Unknown format code 'f' for object of type 'str'
Root cause: Validation Gap

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
ValueError: Unknown format code 'f' for object of type 'str'

Ragas version: 0.1.7
Python version: 3.10

**Code to Reproduce**

```

    metrics = [
            answer_relevancy,
            answer_correctness,
            answer_similarity,
            context_recall,
            context_precision,
            context_relevancy, 
        ]

       for question in test_questions:
            response = rag_chain.invoke({"question" : question})
            answers.append(response["answer"].content)
            contexts.append([context.page_content for context in response['docs']])

        # create hf dataset
        response_dataset = Dataset.from_dict({
            "question" : test_questions,
            "answer" : answers,
            "contexts" : contexts,
            "ground_truth" : test_groundtruths
        })

        print("sample response: ")
        print(response_dataset[0])

        results = evaluate(response_dataset, metrics, 
                           llm=llm, embeddings = openai_embed,
                           raise_exceptions=False,
                           )
        print(results)

```


**Error trace**
```
  File "/home/asrst/.cache/pypoetry/virtualenvs/llm-NDDKoWLm-py3.10/lib/python3.10/site-packages/ragas/evaluation.py", line 292, in __repr__
    score_strs = [f"'{k}': {v:0.4f}" for k, v in scores.items()]
  File "/home/asrst/.cache/pypoetry/virtualenvs/llm-NDDKoWLm-py3.10/lib/python3.10/site-packages/ragas/evaluation.py", line 292, in <listcomp>
    score_strs = [f"'{k}': {v:0.4f}" for k, v in scores.items()]
ValueError: Unknown format code 'f' for object of type 'str'

```
**Expected behavior**

output metrics (python dictonary)

**Additional context**
this usually can happen when `v` is a string instead of float. But in what cases ragas returns a string value for `v` in scores.items() ? Ideally it should be numeric for any metric.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<sub>[R-237](https://linear.app/exploding-gradients/issue/R-237/valueerror-unknown-format-code-f-for-object-of-type-str)</sub>

thank you for sharing this @satadda - I don't know why it's spitting a string too - will try to reproduce

thanks a lot for reporting 🙂 

this is fixed with v0.2

<!-- From jithin on Linear. LinearCommentId:644e14f1-2af7-4bed-acd4-4d993d4cb412: -->
Repository: explodinggradients/ragas
Issue: [R-236] testset hard negatives
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Use similarity to also give similar nodes for a given node so each question has a set of hard negatives against it

**Why is the feature important for you?**
Improve dataset.

**Additional context**
N/A



<sub>[R-236](https://linear.app/exploding-gradients/issue/R-236/testset-hard-negatives)</sub>

this is something we have been thinking for a while too - very important feature too

Thanks for proposing this @PrathamSoni 🙂 
btw while we are developing would love to get some early feedback from you if possible - would you be able to help us with that Pratham?

Yes of course! also, maybe this should be a separate issue, but it would be nice to also specify the node_ids at init time from the pre-chunked contexts i have so that i can post associate the contexts. Also, maintaining a list of original node ids for each node when they get merged or synthesized as part of the generation process.

any updates on this
Repository: explodinggradients/ragas
Issue: [R-234] docs: update and fix documentation
GitHub labels: linear

Some of the pages in the documentation are out of date or incorrect and it make it difficult to have a general idea about the capabilities of the library. Here are a few examples:
- [References/Metrics](https://docs.ragas.io/en/stable/references/metrics.html): the summary at the beginning does not include all implemented metrics, missing 4/10 (AspectCritique, ContextRelevancy, ContextUtilization, Faithfulness). Also, there is no info about the definition of the last metrics in the list.
- [Concepts/metrics](https://docs.ragas.io/en/stable/concepts/metrics/index.html#ragas-metrics): also missing some metrics, 2/10 in the page (AspectCritique, ContextUtilization), and 1/10 in the side bar (ContextUtilization)
- [Integrations](https://docs.ragas.io/en/stable/references/integrations/index.html): friendly reminder that the page is empty.

Thanks!

<sub>[R-234](https://linear.app/exploding-gradients/issue/R-234/docs-update-and-fix-documentation)</sub>

Hey @mancaldel 

thank you for keeping us honest - will be updating these shortly Manuel 🙂