zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: [R-222] linear bug report
GitHub labels: bug

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

Ragas version:
Python version:

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<sub>[R-222](https://linear.app/exploding-gradients/issue/R-222/linear-bug-report)</sub>

thank you Linear will be taking this up shortly

<!-- From jithin on Linear. LinearCommentId:f33c1c82-a774-4e65-a312-14991821fb35: -->
Repository: explodinggradients/ragas
Issue: [R-225] test issue
GitHub labels: bug

not_validated

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

Ragas version:
Python version:

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<sub>[R-225](https://linear.app/exploding-gradients/issue/R-225/test-issue)</sub>

The issue could not be validated after 5 minutes. Closing now.
Repository: explodinggradients/ragas
Issue: [R-227] Add an MLFlow Integration
Root cause: Unimplemented Feature Gap

\[work in progress\] - still flushing out the specifics

* @jjmachan will first build a POC

### Todos

- [ ] POC
- [ ] `ragas.integration` and documentation
- [ ] joined blog post?

### Resources

* [https://mlflow.org/docs/latest/llms/rag/notebooks/mlflow-e2e-evaluation.html](https://mlflow.org/docs/latest/llms/rag/notebooks/mlflow-e2e-evaluation.html?highlight=evaluate)

<sub>From [SyncLinear.com](https://synclinear.com) | [R-227](https://linear.app/exploding-gradients/issue/R-227/add-an-mlflow-integration)</sub>

Hi @jjmachan, have you explored this topic further?


not a lot @Tiemen-Vanderstraeten, is this something you would want? would you share a bit more about your usecase if so?

@jjmachan I'm working on a RAG-chatbot and want to use ragas metrics for regression testing. I am currently using LangSmith as a test framework, but I want to migrate to MLFlow since they have now added tracing. It should be possible to use the metrics in MLFlow's evaluate method 

I would possibly like to help enable ragas' metrics for MLFlow, but not super much software development experience myself 

@Tiemen-Vanderstraeten @jjmachan  MLflow just added native integration with RAGAS! https://mlflow.org/docs/latest/genai/eval-monitor/scorers/third-party/ragas/

```
import mlflow
from mlflow.genai.scorers.ragas import Faithfulness, ContextPrecision

traces = mlflow.search_traces()
results = mlflow.genai.evaluate(
    data=traces,
    scorers=[
        Faithfulness(model="openai:/gpt-4"),
        ContextPrecision(model="openai:/gpt-4"),
    ],
)
```
Repository: explodinggradients/ragas
Issue: Question: ragas built-in visualization of results (e.g. HTML exports)?
Root cause: Unimplemented Feature Gap

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**

Let's say I want to evaluate an LLM on a question-answer dataset. Is there like a front end (e.g. HTML exports) for evaluations run with Ragas?

I want to visualize my evaluation dataset:
- Table of question, expected answer, generated answer, pass/fail
- Overall summary metrics like recall, precision, accuracy, F1 score, etc.

**Code Examples**

N/a

**Additional context**

Looking for something like https://github.com/promptfoo/promptfoo, but for ragas.

Hey @jamesbraza - great to see you again 🙂 

you have 2 options
1. if you use any tracing tools - we could use that
2. pandas and matplotlib or streamlit if your local. Actually having a small streamlit app might be interesting 

what do you think?

Hello! Yeah thanks for the response and hope all is well.

> if you use any tracing tools - we could use that

When you say tracing tools, do you mean something like [LangChain's `BaseTracer`](https://github.com/langchain-ai/langchain/blob/v0.1.17rc1/libs/core/langchain_core/tracers/base.py#L43) or LangSmith?

> pandas and matplotlib or streamlit if your local. Actually having a small streamlit app might be interesting

I was hoping for something built into Ragas, I am trying to avoid building my own front end for evaluation visualization

we don't have plans for that now but we can think about it there is more interest. As of now there are tools outside, open-source and other-wise
Repository: explodinggradients/ragas
Issue: [R-244] add ragas as part of CI/CD for users with github action
Root cause: Unimplemented Feature Gap

users will have

1. testset (synthetic or curated)
2. metrics and evaluate function

now this is something people can setup super easily

<sub>From [SyncLinear.com](https://synclinear.com) | [R-244](https://linear.app/exploding-gradients/issue/R-244/add-ragas-as-part-of-cicd-for-users-with-github-action)</sub>

completed!
docs: https://ragas--976.org.readthedocs.build/en/976/howtos/applications/add_to_ci.html#
<!-- From jithin on Linear. LinearCommentId:b83b1433-06c5-48a2-a96e-7217773d9c78: -->
Repository: explodinggradients/ragas
Issue: Faithfulness vs. context recall
GitHub labels: question

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
What is the difference, conceptually, between faithfulness and context recall? Not sure I understand the semantic difference between (1) claims in answer derived from context / total claims in context, and (2) number of statments in answer attributable to context / total statements in answer.

Is one preferred over the other? Is one more stable?

**Code Examples**
N/A

**Additional context**
N/A

I'm not an expert, but the picture from this [video](https://www.youtube.com/live/dt1Iobn_Hw0?si=KGiFvaNvN7qcpsFg&t=3137) helped me to understand the difference between rag metrics. So, I hope this will bring some clarity to you as well.
![image](https://github.com/explodinggradients/ragas/assets/80523757/c450e61b-2c84-4758-90b9-cd430334c7b5)


Ah whoops, I see context recall is based on ground truth rather than the generated answer. Thanks!
Repository: explodinggradients/ragas
Issue: LLM generates trimmed JSON for context recall and faithfulness metrics
Root cause: Configuration Error

Hey,

First of all, thanks for a great tool, it's really helpful!

I was evaluating my RAG pipeline using own test set and Palm (bison@002) model from VertexAI. I noticed that for context recall and faithfulness metrics resulted in a lots of NaNs in the results, so I dive deeper to define what is wrong with those records. It turned out that LLM under the hood was generating invalid, trimmed jsons (a few statements were included, but third-forth statement was just cut in the middle), which furtherly weren't parsed sucessfully.

It seems like for some reason LLM stops generating the output JSON needed for metrics computation. I checked the output seqlen of Palm - it's at least 1024, while those JSONs were around 100 tokens, so I don't think that it is the reason.

Is that some other constraint on output seqlen, or just that is the nature of LLM-based evaluation, and I need to expect that sometimes results will be NaNs because of inconsistent output format in generated text?

ragas version: 0.1.7
Python version: 3.11.9

Like a user, I have two suggestions regarding the issue: 
1. The max_tokens provided is not sufficient to complete the response. 
3. This is an real issue with LLMs that sometimes occurs. In such cases, you can contact the developer and provide them with the prompt that caused the truncated output. This will be helpful for them as they will have more information and may be able to fix the issue. Additionally, you can also try modifying the prompt slightly. Sometimes, even small changes can help (e.g., writing some words in uppercase or removing a word or sentence).

Hi @Red-Caesar,

Thanks for the response!

You are right, I had an opportunity to play with it a bit more, and tried explicilty setting bigger `max_output_tokens` parameter for my LLM (specifically, 2048 for `text-bison` model), and that resolved almost all NaNs for context recall and faithfulness metrics.

I just checked that default `max_output_tokens` for `ChatVertexAI` model is 128 tokens, so I guess that was the main source of issue.

Thanks for the help, closing the issue.
Repository: explodinggradients/ragas
Issue: [R-247] Integrations: wandb and wandb tracer
Root cause: Unimplemented Feature Gap

todo

- [ ] How to do Evaluation Driven Development with the wandb stack?
- [ ] build out `ragas.integration.wandb` to make that easiest for users

<sub>From [SyncLinear.com](https://synclinear.com) | [R-247](https://linear.app/exploding-gradients/issue/R-247/integrations-wandb-and-wandb-tracer)</sub>

notes from init call

* wandb tables is good for visualising Testset Generation dataset
* weave gives you primitives for tracing 

<!-- From jithin on Linear. LinearCommentId:f8c0ac9c-fdbf-4d44-a178-7c8ebb676a59: -->

Hey @jjmachan , can you describe this a little, I would like to take this up

hey @sky-2002 are you still looking to take it up? we can work with the wandb team for this one

Sure, please go ahead, i can contribute if anything is needed in the workflow, or if tasks are lined out.

@sky-2002 didn't quite get it but lets talk more on discord?
Repository: explodinggradients/ragas
Issue: [R-248] setup Dosu in Ragas

- [ ] figure out categories inside Ragas
- [ ] assign devin for labeling those categories

<sub>From [SyncLinear.com](https://synclinear.com) | [R-248](https://linear.app/exploding-gradients/issue/R-248/setup-devin-in-ragas)</sub>
Repository: explodinggradients/ragas
Issue: [BUG] running ragas evaluate tries to post results to local langsmith
Root cause: External Dependency Breakage

I don't have any langchain env vars set but I am getting a network error as seen below.


<img width="1873" alt="Screenshot 2023-08-11 at 6 04 23 PM" src="https://github.com/explodinggradients/ragas/assets/5640648/6a9265fd-54f9-4f19-b8d7-c2c40ce17be5">


hmm this is interesting, most likely due to something with langchain's end but to confirm could you give me your langchain version, python version and ragas version

let me see if I can reproduce this 

Thanks a ton for raising the issue @mikeldking :)

> langchain==0.0.265
> ragas==0.0.10
> langsmith==0.0.22

are you still facing these issue consistently ?

> are you still facing these issue consistently ?

It's not blocking, it just shows up next to the progress bars

@jjmachan this is pretty annoying and pervasive for us. Are we planning on fixing?
Repository: explodinggradients/ragas
Issue: value error: probabilities are less than 0
Root cause: Algorithmic Error

https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/docstore.py#L325 there's a bug here

It can be patched by either, setting `prob[prob < 0] = .0` before normalization or by adding `prob += eps`. Here are the min and max values I have:

```
> prob.min(), prob.max(), prob.sum()
(-2.47917840909896e-05, 0.0005422972470319377, 1.0)
```

Hi @theoden8,

We’ve switched from that implementation. Please refer to the updated [documentation](https://docs.ragas.io/en/stable/concepts/test_data_generation/) and [code](https://github.com/explodinggradients/ragas) for the latest changes.

Thank you!
Repository: explodinggradients/ragas
Issue: documentation on cosine similarity range is wrong
Root cause: Documentation Deficiency

In [doc](https://docs.ragas.io/en/latest/concepts/metrics/semantic_similarity.html ), 

> The concept of Answer Semantic Similarity pertains to the assessment of the semantic resemblance between the generated answer and the ground truth. This evaluation is based on the ground truth and the answer, with values falling within the range of 0 to 1.

This is wrong since cosine similarity can take on negative values
```
a = [1.0]
b = [-1.0]
cosine_sim_a_b = a dot_product b / (a_norm x b_norm) = -1
```

You're right. Cosine smiliarity can take on negative values, althought it is heavily biased towards positive values.  
A few interesting experiments on this -  
1.  https://datascience.stackexchange.com/questions/101862/cosine-similarity-between-sentence-embeddings-is-always-positive#:~:text=This%20range%20is%20valid%20if,line%20but%20in%20opposite%20directions.   
2.  https://stackoverflow.com/questions/60852877/why-is-my-cosine-similarity-always-positive-fasttext  

Keep in mind that vector embeddings are a result of computing the probability of the word in a given context. This means that `beautiful` and `ugly`, even though they are the opposites, they have a medium cosine score, since they're likely to appear in the same context. Completely unrelated phrases like `quantum mechanics cryptography algorithms` and `blue kingfisher eating salmon` have a negative cosine score.  
However, in Dataset Generation, since all of the nodes have the same/related context, it is extremely unlikely to have a negative cosine score.
Repository: explodinggradients/ragas
Issue: Init attribute is mutable and shared (not intended?)
GitHub labels: bug

```
from ragas.testset.filters import EvolutionFilter
a = EvolutionFilter(None)
b = EvolutionFilter(None)
a.evolution_elimination_prompt.name = "a"
print(a.evolution_elimination_prompt.name)
print(b.evolution_elimination_prompt.name)
```
this prints
```
a
a
```
not intended?

Closing this as fixed in new version. 

Please refer to latest docs and usage.
Repository: explodinggradients/ragas
Issue: RAGAS compatibility with mistral models
Root cause: Configuration Error

- I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

Within the base-model file there is a configuration which sets the model temperature to *1e-8*.
This Temperature does not work for Mistral Models.

Due to this setting the Models are unable to produce any results.

When replacing the temperature with 0.01 the models seem to work flawlessly.

Closing this as fixed now.
Repository: explodinggradients/ragas
Issue: Cant' run Quickstart
Root cause: Validation Gap

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Can't run the Quickstart example from Readme.MD

Ragas version: '0.1.8.dev13+gc5eac53'
Python version:  3.11.7


**Code to Reproduce**
Copy and paste

```
from datasets import Dataset 
import os
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness

os.environ["OPENAI_API_KEY"] = "your-openai-key"

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset,metrics=[faithfulness,answer_correctness])
score.to_pandas()
```

**Error trace**
ValueError: Input variables ['question', 'answer', 'sentences'] do not match with the given parameters ['question', 'answer']

**Expected behavior**
Quickstart example running without errors
Repository: explodinggradients/ragas
Issue: Evaluate() function gets unexpected arguments
Root cause: Environment Incompatibility

This is the code: 
result = evaluate(
    amnesty_qa["eval"], metrics=metrics, llm=azure_model, embeddings=azure_embeddings
)
This is the error: 
{
	"name": "TypeError",
	"message": "evaluate() got an unexpected keyword argument 'llm'",
	"stack": "---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[27], line 1
----> 1 result = evaluate(
      2     amnesty_qa[\"eval\"], metrics=metrics, llm=azure_model, embeddings=azure_embeddings
      3 )
      5 result

TypeError: evaluate() got an unexpected keyword argument 'llm'"
}


seems like you are using an old version of  ragas.  Try upgrade your ragas package [ pip3 install ragas==0.1.7 ]  should resolve your problem . 



thanks to @KaimingFanshi for helping @nelagamy ❤️ 

> seems like you are using an old version of ragas. Try upgrade your ragas package [ pip3 install ragas==0.1.7 ] should resolve your problem .

I'm working on openai==0.28.1, and for that I need ragas<=0.0.18,  But ragas is giving the above discussed error. And I can't just upgrade the ragas, since I can only work with lower version of openai.

Any solutions for this....
Repository: explodinggradients/ragas
Issue: Ragas llama_index integration as shown doesn't work for custom LLMs
Root cause: Documentation Deficiency

[X] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
[This document](https://docs.ragas.io/en/latest/howtos/integrations/llamaindex.html) describes how to use Llama index w/OpenAI and Ragas. And [this] shows how we can use a custom LLM but via langchain.

Can you please provide an example of how one can use llama_index without needing LangChain and using a [custom LLM ](https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom/)?


**Code Examples**
I made _some_ progress using"

```
evaluate (data, metrics, llm=LangchainLLMWrapper(mycustomLLM), embeddings=LangchainEmbeddingsWrapper(my_embedding))
```

But halfway through eval it compains of unimplemented methods like  `ragas has no attribute generate_prompt`. I'm not sure what needs to be done, and what unimplemented methods may show up in the future.



this should fix this https://docs.ragas.io/en/stable/howtos/integrations/llamaindex.html

extremely sorry about the late response
Repository: explodinggradients/ragas
Issue: [R-254] Issue in Evaluation using local LLM
Root cause: Validation Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**

> “WARNING:ragas.llms.output_parser:Failed to parse output. Returning None.”

I tried including the trace using Langsmith to check for requests and responses. For the given input prompt I believe it is an issue of context length because I get a blank response. I tried different LLMs but the error remains the same.

**Code Examples**
Hosted LLama 2 model with LLamaCPP. Below is the command I used
`python3 -m llama_cpp.server --model /tmp/llama_index/models/llama-13b.Q5_K_M.gguf --port 8009 --host 129.69.217.24 --chat_format llama-2`
 
Following is a sample testset I am using,
[Ragas_dataset.csv](https://github.com/explodinggradients/ragas/files/15315918/Ragas_dataset.csv)

Can ignore the dataset part, I tried the same with 
`fiqa_eval = load_dataset("explodinggradients/fiqa", "ragas_eval")`
but same issue persists.

Code:
```
from datasets import load_dataset
dataset = load_dataset("csv", data_files="Ragas_dataset.csv")

from tqdm import tqdm
import pandas as pd
from datasets import Dataset
def create_ragas_dataset( eval_dataset):
  rag_dataset = []
  for row in tqdm(eval_dataset):
    rag_dataset.append(
        {"question" : row["question"],
         # "answer" : result["answer"],
         "answer" : row["ground_truth"],
         "contexts" : [row["contexts"]],
         "ground_truth" : row["ground_truth"]
         }
    )
  rag_df = pd.DataFrame(rag_dataset)
  rag_eval_dataset = Dataset.from_pandas(rag_df)
  return rag_eval_dataset

basic_qa_ragas_dataset = create_ragas_dataset(dataset["train"].select(range(2)))

from langchain.chat_models import ChatOpenAI
from ragas.llms import LangchainLLMWrapper

inference_server_url = "http://localhost:8009/v1"

chat = ChatOpenAI(
    model="/tmp/llama_index/models/llama-13b.Q5_K_M.gguf",
    openai_api_key="no-key",
    openai_api_base=inference_server_url,
    max_tokens=5,
    temperature=0,
)

vllm = LangchainLLMWrapper(chat)

from ragas.metrics import (
    context_precision,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

# change the LLM

faithfulness.llm = vllm
context_precision.llm = vllm
context_recall.llm = vllm
harmfulness.llm = vllm

from ragas import evaluate

result = evaluate(
    basic_qa_ragas_dataset, 
    metrics=[faithfulness]
)
result
```
<img width="1029" alt="image" src="https://github.com/explodinggradients/ragas/assets/119347732/57bbba2c-0e8c-478e-9207-f517f5e7078d">


**Additional context**
Please let me know if I should provide more information


<sub>[R-254](https://linear.app/exploding-gradients/issue/R-254/issue-in-evaluation-using-local-llm)</sub>

Having the same issue with langchain Ollama llm wrapper and llama3

Have the same situation with Langchan Azure OpenAI
Repository: explodinggradients/ragas
Issue: [R-253] possible error in answer_correctness prompt
Root cause: Validation Gap

higher score -> [https://smith.langchain.com/public/27aff81a-0a7c-488b-9f7b-8e905f8526c1/r](https://smith.langchain.com/public/27aff81a-0a7c-488b-9f7b-8e905f8526c1/r) lower score -> [https://smith.langchain.com/public/4bc8f83e-7691-4b87-85ad-cfe3fcd30d6e/r](https://smith.langchain.com/public/4bc8f83e-7691-4b87-85ad-cfe3fcd30d6e/r)

<sub>From [SyncLinear.com](https://synclinear.com) | [R-253](https://linear.app/exploding-gradients/issue/R-253/possible-error-in-answer-correctness-prompt)</sub>

I checked the issue. seems like it is gpt-3.5 just getting dumper . The same input with gpt-4 gives correct output [https://smith.langchain.com/public/39986a65-0923-4293-958c-aeeb02c3e102/r](https://smith.langchain.com/public/39986a65-0923-4293-958c-aeeb02c3e102/r) @jjmachan 

<!-- From Shahul ES on Linear. LinearCommentId:287bc182-5bcc-4a52-a062-415e64db1915: -->