zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Assertion error while trying to run AnswerSimilarity
Root cause: Validation Gap

Ragas version: 0.0.22
Python version: 3.10.12
IDE: Google Colab

I am trying the find out the similarity score between the ground truth and the answers. I am using the following code:
```python
from datasets import load_dataset
from ragas.embeddings.base import HuggingfaceEmbeddings
from ragas.metrics import AnswerSimilarity
 
fiqa_eval = load_dataset("explodinggradients/fiqa", "ragas_eval")
embeddings = HuggingfaceEmbeddings(model_name="thenlper/gte-large")
answer_similarity = AnswerSimilarity()
results = answer_similarity.score(fiqa_eval['baseline'])
```
After some time, it is giving the following error:
```
AssertionError                            Traceback (most recent call last)
[<ipython-input-43-2cc7f45a2a25>](https://localhost:8080/#) in <cell line: 1>()
----> 1 results = answer_similarity.score(fiqa_eval['baseline'])
      2 results

2 frames
[/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py](https://localhost:8080/#) in score(self, dataset, callbacks)
     74         with trace_as_chain_group(f"ragas_{self.name}", callback_manager=cm) as group:
     75             for batch in tqdm(self.get_batches(len(dataset))):
---> 76                 score = self._score_batch(dataset.select(batch), callbacks=group)
     77                 scores.extend(score)
     78 

[/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_similarity.py](https://localhost:8080/#) in _score_batch(self, dataset, callbacks, callback_group_name)
     79             scores = np.array(self.embeddings.predict(inputs))
     80         else:
---> 81             embeddings_1 = np.array(self.embeddings.embed_documents(ground_truths))
     82             embeddings_2 = np.array(self.embeddings.embed_documents(answers))
     83             similarity = embeddings_1 @ embeddings_2.T

[/usr/local/lib/python3.10/dist-packages/ragas/embeddings/base.py](https://localhost:8080/#) in embed_documents(self, texts)
    143         )
    144 
--> 145         assert isinstance(embeddings, Tensor)
    146         return embeddings.tolist()
    147 

AssertionError:
```
How to fix this issue? Please advise.

Hey @jerilkuriakose , can you try installing ragas using `pip install "ragas==0.1.0rc1"` and follow this guide? https://docs.ragas.io/en/latest/howtos/customisations/embeddings.html
This should solve your issue. 

closing this, feel free to reopen if issue persists
Repository: explodinggradients/ragas
Issue: AttributeError: 'LangchainLLMWrapper' object has no attribute 'embed_query'
Root cause: Interface Contract Mismatch

**Describe the bug**
Evaluate doesn't work due to compatibility issues with LangchainLLMWrapper, regardless if I wrap the models or not.

Ragas version: 0.0.23.dev52+g08dd0a7
Python version: 3.11.7

**Error trace**
 File "C:\Users\<USER>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\metrics\_answer_relevance.py", line 85, in calculate_similarity
    question_vec = np.asarray(self.embeddings.embed_query(question)).reshape(1, -1)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LangchainLLMWrapper' object has no attribute 'embed_query'

**Infos**
These are the models used in the eval:

```
metrics=[
    faithfulness, # requires col "contexts"
    answer_relevancy, # requires col "contexts"
    harmfulness, #['contexts']
]

result = evaluate(
    embeddings=embeddings_az,
    llm=llm_az,
    dataset=dataset.select(range(1)),
    metrics= metrics
)
```

```
embeddings_az
AzureOpenAIEmbeddings(client=<openai.resources.embeddings.Embeddings object at 0x000001D8D63E5F10>, async_client=<openai.resources.embeddings.AsyncEmbeddings object at 0x000001D8D66088D0>, model='text-embedding-ada-002', deployment='text-embedding-ada-002', openai_api_version='2023-05-15', openai_api_base=None, openai_api_type='azure', openai_proxy='', embedding_ctx_length=8191, openai_api_key='...', openai_organization=None, allowed_special=set(), disallowed_special='all', chunk_size=16, max_retries=2, request_timeout=None, headers=None, tiktoken_enabled=True, tiktoken_model_name=None, show_progress_bar=False, model_kwargs={}, skip_empty=False, default_headers=None, default_query=None, retry_min_seconds=4, retry_max_seconds=20, http_client=None, azure_endpoint='https://....openai.azure.com/', azure_ad_token=None, azure_ad_token_provider=None, validate_base_url=True)

llm_az
AzureOpenAI(client=<openai.resources.completions.Completions object at 0x000001D8D6383810>, async_client=<openai.resources.completions.AsyncCompletions object at 0x000001D8D63BCA50>, openai_api_key='...', openai_proxy='', azure_endpoint='https://....openai.azure.com/', deployment_name='gpt-35-turbo-tw', openai_api_version='2023-05-15', openai_api_type='azure')
```

hey @tenshis 
apologies for the late reply but have you tried wrapping the embeddings like this before passing

```py
from ragas.embeddings.base import LangchainEmbeddingsWrapper

embeddings = LangchainEmbeddingsWrapper(embeddings_az)
``` 
and then pass it

can you check this and let me know if it works?

Hello jjmachan,

Thanks for the answer. I'm afraid there is no LangchainEmbeddingsWrapper in ragas.embeddings.base? 
I am on version 0.1.0rc1 .

oh you have to install from source for that, I've merged the fix #553 

could you install the latest and let me know @tenshis now this notebook should work as it is https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html
Repository: explodinggradients/ragas
Issue: 'list' object has no attribute 'get' in _context_precision.py
Root cause: Validation Gap

**Describe the bug**
In the _calculate_average_precision function, a list is being handled as a dictionary.

Ragas version: 0.1.0rc1
Python version:3.11.5

![image](https://github.com/explodinggradients/ragas/assets/2423660/38f9d3a3-bd45-4c7a-89ef-de2179494941)

```
'list' object has no attribute 'get'
  File "...\LocalCache\local-packages\Python311\site-packages\ragas\metrics\_context_precision.py", line 98, in <listcomp>
    if resp.get("verdict")
       ^^^^^^^^
```

Same issue in the _score and in the _compute_score functions under **_faithfullness.py**

```
Exception has occurred: AttributeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
'list' object has no attribute 'get'
  File "C:\Users\...\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\metrics\_faithfulness.py", line 197, in _score
    p = self._create_nli_prompt(row, statements.get("statements", []))
                                     ^^^^^^^^^^^^^^
```

same

Hey guys @tenshis @Pauldevillers , I'll look into it and raise a fix ASAP.  If anyone would like to work on this, please feel free to raise a PR before that.
Repository: explodinggradients/ragas
Issue: ragas.exceptions.OpenAIKeyNotFound when I try the guide "Bring your own LLMs"
Root cause: Validation Gap

When I try the guide: "Bring your own LLMs" to use ragas, if answer_relevancy is added to metrics, the following message is displayed: 
**ragas.exceptions.OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable**

How to fix it? I used ragas-0.0.22 with python=3.11.

should be fixed with #553 
could you check and reopen if not fixed?

All of sudden, why this is coming up in ragas when using answer relevancy
Repository: explodinggradients/ragas
Issue: RAGAS answer_relevancy integration with OpenAI broken
Root cause: Validation Gap

**Describe the bug**
answer_relevancy metric does not seem to work with open ai key when RAGAS is installed from pip (version 0.0.22)

Ragas version: 0.0.22
Python version: 3.11.7

**Code to Reproduce**
`data = {
        'question': [user_question],
        'answer': st.session_state.answer,
        'contexts' : st.session_state.context,
        'ground_truths': [[user_ground_truth]]
    }
    dataset = Dataset.from_dict(data)

    os.environ["OPENAI_API_KEY"] = open_ai_api_key

    from ragas.metrics import (
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    )

    metrics = []
    for eval_type, eval_config in {
        "faithfulness": [eval_answer_faithfulness, faithfulness],
        "answer_relevancy": [eval_answer_relevancy, answer_relevancy],
        "context_precision": [eval_context_precision, context_precision],
        "context_recall": [eval_context_recall, context_recall],
    }.items():
        if eval_config[0]:
            metrics.append(eval_config[1])
    gpt4 = ChatOpenAI(model_name="gpt-4")

    faithfulness.llm.langchain_llm = gpt4
    answer_relevancy.llm.langchain_llm = gpt4
    context_precision.llm.langchain_llm = gpt4
    context_recall.llm.langchain_llm = gpt4
    result = evaluate(dataset, metrics)
    print(result)`

**Error trace**
`2024-01-31 15:51:30.524 Uncaught app exception
Traceback (most recent call last):
  File "/Users/paavramo/.pyenv/versions/3.11.7/envs/doco-bot/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 535, in _run_script
    exec(code, module.__dict__)
  File "/Users/paavramo/Documents/QLD Treasury/doco-bot/cdk/lib/fargate/doco-bot/pages/Evaluation.py", line 189, in <module>
    result = evaluate(dataset, metrics)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/paavramo/.pyenv/versions/3.11.7/envs/doco-bot/lib/python3.11/site-packages/ragas/evaluation.py", line 93, in evaluate
    [m.init_model() for m in metrics]
  File "/Users/paavramo/.pyenv/versions/3.11.7/envs/doco-bot/lib/python3.11/site-packages/ragas/evaluation.py", line 93, in <listcomp>
    [m.init_model() for m in metrics]
     ^^^^^^^^^^^^^^
  File "/Users/paavramo/.pyenv/versions/3.11.7/envs/doco-bot/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py", line 100, in init_model
    raise OpenAIKeyNotFound
ragas.exceptions.OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable`

**Expected behavior**
Should work as written. When i comment out the references to answer relevancy the solution works as is with no issues with the OpenAI key and I get back valid results.


Hey @wolfman-numba1 , can you install the latest version from the source and try again?
please refer to docs.ragas.io/en/latest while doing so, there are some changes in API which makes it easier for you to custom llms / embeddings
Repository: explodinggradients/ragas
Issue: why?can you help me?
Root cause: External Dependency Breakage

为什么我选择tongyi模型，它返回不出来最后评估的结果，只有openai才可以吗
下面是我在colab上面跑的代码出的错误

```
evaluating with [faithfulness]
  0%|          | 0/1 [00:00<?, ?it/s]ERROR:dashscope:Unsupported atom data type: <class 'langchain_core.prompts.chat.ChatPromptTemplate'>
  0%|          | 0/1 [00:00<?, ?it/s]
---------------------------------------------------------------------------
UnsupportedDataType                       Traceback (most recent call last)
[<ipython-input-81-5fa2ac208107>](https://localhost:8080/#) in <cell line: 3>()
      1 from ragas.llama_index import evaluate
      2 
----> 3 result = evaluate(query_engine, metrics, eval_questions, eval_answers)

25 frames
[/usr/local/lib/python3.10/dist-packages/dashscope/io/input_output.py](https://localhost:8080/#) in resolve_input(input, is_encode_binary, custom_type_resolver)
    109         return custom_type_resolver[type(input)](input)
    110     else:
--> 111         raise UnsupportedDataType('Unsupported atom data type: %s' %
    112                                   type(input))

UnsupportedDataType: Unsupported atom data type: <class 'langchain_core.prompts.chat.ChatPromptTemplate'>
```

which version of Ragas are you using @liuruit ?
llamaindex is not support for v0.1 at the moment sadly - you can track #557 for updates

will get this fixed for you as fast as we can Liuruit :)
Repository: explodinggradients/ragas
Issue: [R-275] Tokens Usage for evaluations and testset generations
Root cause: Unimplemented Feature Gap

I am using RAGAs evaluation metrics to evaluate my RAG pipeline. Is there any way I could know the openai tokens used per api call from RAGAs, lets say for any one metric, i.e., faithfulness. I want to estimate the cost of using RAGAs evaluators.

<sub>[R-275](https://linear.app/exploding-gradients/issue/R-275/tokens-usage-for-evaluations-and-testset-generations)</sub>

this is a valuable suggestion @aliai22 :)

we have thought about it and will be implementing it soon but it might take a couple of weeks to prioritize this. The reason is that I want to take some time to design how stats would be like

basically, we want to track things like
- tokens used
- cost
- performance metrics etc

in a unified way. I'll explore more and send an RFC your way asking for your feedback as soon as I have it ready. would that help?

Yeah, that would be great. Looking forward to hear from you.

Did it get implemented?

I am also looking for this, any update?

Pinging to see if this gets released at any point.

Any updates on this @jjmachan ?

I'm also interested in this! 

hey folks, not yet but I will take this up this week and have something ready for your feedback shortly, my sincere apologies for the delay!

fixed in #1129
Repository: explodinggradients/ragas
Issue: Why do NaN exist in metrics ？
Root cause: Validation Gap

Why do NaN exist in metrics ？Is it because you didn't make some judgment that zero can't be the denominator？I am looking forward to your reply


Similar code exists in the source code：“
                if num_statements:
                    score = faithful_statements / num_statements
                else:
                    score = np.nan ”
Repository: explodinggradients/ragas
Issue: [Question] Does ragas now support async call?


yes we do starting from v0.1 release happening next week
(but you can install from source to get a taste)

> yes we do starting from v0.1 release happening next week (but you can install from source to get a taste)

It will be available via pip install binaries packages also?

yes, will be out in a couple of hours

Thank you for the info. Hope to get a proper guidance/documentation for this as well :)

it actually works under-the-hood. You don't have to do anything. but will be updating the docs over the week

Cool. Thank you
Repository: explodinggradients/ragas
Issue: Feature suggestion - Handling LLM quotas when evaluating 
Root cause: Unimplemented Feature Gap

It would be nice to handle LLM quotas when evaluating a large dataset, for my personal case I cannot increase the default 60 request per min for VertexAI LLM. 

Tracking llm calls for the current minute within .evaluate() might sound a bit overkill. Offering the possibility to set a time.sleep() between each sample might do the trick. 

I don't what know what you guys think. I am the only one to encounter such a problem ?


that is a great idea @0ENZO !
we use tenacity underneath the hood and have this to configure https://github.com/explodinggradients/ragas/blob/main/src/ragas/run_config.py things like this

I'll add sleep to it and that should help you

we have to add something for stats too I guess. so you can see num_tokens, cost, performance figures etc

what do you think about those, have you felt the need for that. If you were to only choose one, which would it be?

Sounds good, thanks ! 

Regarding performance figures, num_tokens.. I haven't had any such needs yet 

hey @0ENZO so after thinking about it a bit more, it seems like a more complicated solution to implement because of how we have things setup.

The core problem here is the contention of resources, we could have fixed it in 2 ways
1. collect all the LLM and embedding calls `ragas` makes and implement something like a leaky bucket so that the #requests per minute is a constant
2. Exponential Backoff as explained [here](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/). We went with this.

so the solution today is configuring the exponential backoff for 60 requests per minute. Right now I don't have a good formula for that but that is something we could find right?

so the solution for your problem today is configuring the `RunConfig` with the correct max_retries and max_wait (and maybe some more, I'll look into that) but what do you think?

also I'm doing some experiments so that I can get you unblocked without much hastle

Do you have a suggestion that I could implement now? I am exceeding my azure gpt4 rate limit of 80k tokens per minute when evaluating 48 questions/answers for all metrics. Is there a way to rate limit the evaluation? Perhaps I should pull out some metrics?

@jjmachan, any suggestions on how we should set the run config? I am also facing this issue with ragas 0.1.7




> we have to add something for stats too I guess. so you can see num_tokens, cost, performance figures etc
> 
> what do you think about those, have you felt the need for that. If you were to only choose one, which would it be?

May I ask if there is any plan for this part

will be fixed with #1156 for documentation on run_config check out [Understand Cost and Usage of Operations | Ragas](https://docs.ragas.io/en/latest/howtos/applications/cost.html) for how to figure out cost

hope this helps @xiaochaohit @bdeck8317 @klangst-ETR
Repository: explodinggradients/ragas
Issue: Testset generation prompt customization
Root cause: Documentation Deficiency

The current SimpleEvolution seed question prompt uses a hardcoded Prompt. I have a usecase where I want to use a customized prompt to generate certain types of questions. The only way I see to accomplish this is at the moment is to derive a class based on SimpleEvolution and copy/paste `_aevolve` using a custom Prompt instead of the seed_question_prompt.

It would be nice to have the ability to inject or otherwise customize a prompt.

actually you can do it another way. just call SimpleEvolution with a different prompt like so

```py
from ragas.testset.evolutions import SimpleEvolution

simple = SimpleEvolution(seed_question_prompt=# your prompt)
``` 
note you will have init the rest of the attributes for `Evolution` and the `Prompt` object should be of the format https://docs.ragas.io/en/latest/concepts/prompts.html

Ahh, did not see this was added with this PR https://github.com/explodinggradients/ragas/pull/530

I'll pull latest. Thanks!
Repository: explodinggradients/ragas
Issue: Add back support for Langchain Evalchains
Root cause: Unimplemented Feature Gap

**Describe the Feature**
with V0.1, support for langchain EvalChain is broken, add that back in

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Any ETA? Or some architecture/code scaffolds so at least I can help accelerate this?

hey @leehanchung , you could invoke (or ainvoke()`) it in a loop and create the dataset to evaluate.

refer this [notebook](https://colab.research.google.com/drive/1C1Epju1lVkXTQi2jBq1njrOrmkfg0eQS?usp=sharing). 


This might take some time for me to fix -worst case 2 weeks. I apologize for the hard time 🙁 

@jjmachan thanks for your work!!

Is there any additional progress made on this issue?
Repository: explodinggradients/ragas
Issue: Metric language adapter AttributeError
Root cause: Validation Gap

I am running the metric adapter as described in the docs, but I am getting an error.

Ragas version: 0.1.0
Python version: 3.8.10

**Code to Reproduce**
```python
from ragas.metrics import (
    faithfulness,
    answer_correctness,
)
from langchain.chat_models import ChatOpenAI
from ragas import adapt

# llm used for adaptation
openai_model = ChatOpenAI(model_name="gpt-4")

adapt(metrics=[faithfulness,answer_correctness], language="hindi", llm=openai_model)
```

**Error trace**
Traceback (most recent call last):
  File "/home/breno/aumo/aumogpt-breno/teste.py", line 11, in <module>
    adapt(metrics=[faithfulness,answer_correctness], language="hindi", llm=openai_model)
  File "/home/breno/.local/lib/python3.8/site-packages/ragas/adaptation.py", line 36, in adapt
    metric.adapt(language, cache_dir=cache_dir)
  File "/home/breno/.local/lib/python3.8/site-packages/ragas/metrics/_faithfulness.py", line 207, in adapt
    self.nli_statements_message = self.nli_statements_message.adapt(
  File "/home/breno/.local/lib/python3.8/site-packages/ragas/llms/prompt.py", line 190, in adapt
    output_keys.append(get_all_keys(example.get(self.output_key)))
  File "/home/breno/.local/lib/python3.8/site-packages/ragas/llms/prompt.py", line 154, in get_all_keys
    for key, value in nested_json.items():
AttributeError: 'list' object has no attribute 'items'


Hey @Breno-de-Angelo , I'm on it. Will raise a fix soon. 

@Breno-de-Angelo I have raised a fix, please reinstall ragas from the source and try again once merged. 

Thank you @shahules786 ! I'll try it here.
Repository: explodinggradients/ragas
Issue: does prompt gets adjusted based on the LLM used ?
Root cause: Unimplemented Feature Gap

lets say i am using llama2 model to evaluate my RAG Applications does ragas takes care to format the prompt how llama2 understands ?
like wrapping the prompt something like https://huggingface.co/blog/llama2#how-to-prompt-llama-2?




Interesting point @VpkPrasanna , We use langchain underneath to wrap all the LLM but I am not sure if langchain handles this properly. I'll try and dig into it. But I would love to hear your thoughts

> Interesting point @VpkPrasanna , We use langchain underneath to wrap all the LLM but I am not sure if langchain handles this properly. I'll try and dig into it. But I would love to hear your thoughts

langchain does not wrap the prompt based on the LLM we use , because it does not know which LLM model we pass so it has to be handled separately 

I met the same problem. When I used LLama2_7b_chat_hf model to evaluate my RAG results, I found some metrics like answer_correctness  were always `np.nan`. I guess the code can not parse the results generated by LLama2.

we can actually do this as part of prompt adaptation. in addition to language. But how it works internally is going to be hard.

1 method I can see is that we save prompts online and when someone adapts, we can download the strings. cheap and fast, harder to scale.

@shahules786 any other ideas you have?