zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Answer Relevancy metric cannot use model temperature
Root cause: Configuration Error

**Describe the bug**
No matter what temperature param set for llm model when using answer relevancy metric test, it won't work. That means the reverse-engineer ‘n’ variants of the question from the generated answer will be identical. When using questions, answers, contexts （in **Code to Reproduce** part） below as input params, if setting a breakpoint on line 106 (_answer_relevance.py) and check variable "response", all "question"  properties are identical. Below are my own result:
[
  {
    "question": "Where is France located and what is its capital?",
    "noncommittal": 0
  },
  {
    "question": "Where is France located and what is its capital?",
    "noncommittal": 0
  },
  {
    "question": "Where is France located and what is its capital?",
    "noncommittal": 0
  }
]

I trace this problem, and find that on line 136 (_answer_relevance.py), it uses self.llm.generate the create result without using temperature param. That means it will using default 1e-8 as temperature to call llm create results. That's why all response are identical. I think it's a bug.

Ragas version: 1.0.3 
Python version: 3.10

**Code to Reproduce**
questions = ["Where is France and what is it’s capital?"]
answers = ["France is in western Europe and Paris is its capital."]
contexts = [["France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower. ", "The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and the vast Palace of Versailles attest to its rich history."]]
cr_data_samples = {
        'question': questions,
        'answer': answers,
        'contexts': contexts
}
cr_dataset = Dataset.from_dict(cr_data_samples)
result = evaluate(
    cr_dataset, metrics=[answer_relevancy], llm=chat
)

**Error trace**
breakpoint on line 106（_answer_relevance.py）
![image](https://github.com/explodinggradients/ragas/assets/8898872/74b3c9f2-c644-471a-9b5b-2af93ef93e4e)
line 136 (_answer_relevancy.py)
![image](https://github.com/explodinggradients/ragas/assets/8898872/5a4c43f7-174e-455b-b7b3-4c88563c7cff)
dive into generate method and find the temperature default value is 1e-8:
![image](https://github.com/explodinggradients/ragas/assets/8898872/0fe1a82d-ac5c-4069-8357-9b178bccd05b)


**Expected behavior**
answer relevancy metric should use model temperature param instead of default 1e-8


<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: Customizing Ragas Evaluation
Root cause: Documentation Deficiency

I want to use my own llm model and embedding for ragas evaluation 
i follow ragas documentation but got this below error

Ragas version: 0.0.22 
Python version: 3.8.10

**Code**
langchain_llm = BaseLanguageModel(model=model_name, api_key = "*********")
langchain_embeddings = Embeddings(model=model_name)
result = evaluate(dataset=dataset,metrics=metrics,llm=langchain_llm,embeddings=langchain_embeddings)

**Error**               
"Can't instantiate abstract class BaseLanguageModel with abstract methods agenerate_prompt, apredict, apredict_messages, generate_prompt, invoke, predict, predict_messages".

Hey, checkout https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html
Repository: explodinggradients/ragas
Issue: TestsetGenerator for Azure OpenAI endpoints
Root cause: Unimplemented Feature Gap

**Your Question**
Hi there, I find the TestsetGenerator feature very interesting and would like to use it. But I can see that this feature is not yet available for Azure OpenAI endpoint users, right? Am I missing something?


Hey PR #670 Addresses this issue. Will make a release this week :)

Good to know! Thanks a lot.
Repository: explodinggradients/ragas
Issue: Metric explainability in evaluate
Root cause: Documentation Deficiency

**Describe the Feature**
Many metrics, such as answer relevance and faithfulness, utilize LLMs to generate intemediary results (ex., faithfulness claims, relevance questions from given answer) that are then evaluated and aggregated in a downstream step. Right now (to the best of my knowledge) these intermediary outputs can't be surfaced, so explainability is lacking. 

For example, given a faithfulness score of 0.75, it would be useful to surface 1) the claims that were extracted from the answer, and 2) the LLM rating for each claim.

**Why is the feature important for you?**
Surfacing these intermediary outputs can increase interpretability and give users a better experience in debugging and improving their own custom prompts that can be used to tune ragas to their particular use case. Ideally, this functionality should be implemented through the `evaluate` function and used like so:

```
   result = evaluate(
       dataset=dataset,
       metrics=metrics,
       explainability=True
   )
```

**Additional context**
I'm trying to use this library and it is difficult to explain or debug why a metric is a particular value.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hey @peanutshawny got your point. But we have something called callbacks which does moreover the same. Checkout out integration page https://docs.ragas.io/en/stable/howtos/integrations/index.html
to write custom callbacks https://python.langchain.com/docs/modules/callbacks/
https://docs.ragas.io/en/latest/howtos/applications/tracing.html
You can log the intermediate results to almost any tracing and logging tool 

Feel free to reopen the issue, if you feel your issue was not addressed feel free to reopen.
Repository: explodinggradients/ragas
Issue: Kernel Crashed while generating the testset
Root cause: Resource Mishandling

[ ] I have checked the [documentation](https://docs.ragas.io/en/latest/getstarted/testset_generation.html) and related resources and couldn't resolve my bug.

**Bug Description**
 I wanted to generate a test dataset to evaluate my RAG application. I followed the instruction from the official documentation. When I tried to use **generator.generate_with_llamaindex_docs** my ipykernel is getting crashed for an unknown reason.


Ragas version: 0.1.3
Python version: 3.10.13
Ipykernel: 6.29.3

**Code to Reproduce**
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.testset.docstore import InMemoryDocumentStore
from langchain.text_splitter import TokenTextSplitter
from ragas.testset.extractor import KeyphraseExtractor

langchain_llm_model = LangchainLLMWrapper(load_llm())
langchain_embed_model = LangchainEmbeddingsWrapper(load_embed_model())

splitter = TokenTextSplitter(chunk_size=256, chunk_overlap=0)
keyphrase_extractor = KeyphraseExtractor(llm = langchain_llm_model)

docstore=InMemoryDocumentStore(
    splitter=splitter,
    embeddings=langchain_embed_model,
    extractor=keyphrase_extractor
)


generator = TestsetGenerator(
    generator_llm=langchain_llm_model,
    critic_llm=langchain_llm_model,
    embeddings=langchain_embed_model,
    docstore=docstore
)

testset = generator.generate_with_llamaindex_docs(
    documents=nodes,
    test_size=10,
) (Kernel error found)

**Error trace**

No error trace found, but the jupyter kernel is getting crashed for an unknown reason.
<img width="449" alt="image" src="https://github.com/explodinggradients/ragas/assets/154510244/f884351c-4997-425e-9463-7dbc25d649b2">
**Expected behavior**
The function should have generated a synthetic test dataset as per the documentation


**Additional context**
Fectching nodes from docstore using llama_index docstore.
Using langchain llms with [HuggingFaceTextGenInference](https://huggingface.co/docs/text-generation-inference/index) (model_name = "TheBloke/Llama-2-13B-chat-GPTQ")
Using langchain embeddings with HuggingFaceEmbeddings (model_name = "BAAI/bge-large-en-v1.5")

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


@shahules786 Please help me to fix this bug.

Hey @Anirban20001962 this could be an issue with async process. Have you tried by passing ` is_async=False` ? 


> Hey @Anirban20001962 this could be an issue with async process. Have you tried by passing ` is_async=False` ?

Yes I have tried that also but still there is kernel error.

same issue here

@shahules786 Did you able to fix the issue?

Hi @Anirban20001962 We are working on it. There are several similar issues that have to be addressed in the PR. thanks for your patience
Repository: explodinggradients/ragas
Issue: Issue with TikToken Dependency 
Root cause: External Dependency Breakage

Hello,

Firstly, thank you for this great framework and library!

I installed your package and followed your notebook on using Ragas with Azure Open AI, but am running into an "Exception In Runner" error with a traceback to 

requests.exceptions.httperror 503 server error service unavailable for url: https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken

I think that the error is caused by a security firewall on my side, but are there any parameters that can be set to avoid having to access the external page?

Thanks in advance!

Did you find solution for this?
I'm also facing the same issue

I couldn't find a direct solution, but if you switch the embedding model to be an open source one, then it works fine.
Repository: explodinggradients/ragas
Issue: Missing copyright notice
GitHub labels: question

- [x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
The Apache-2 license requires redistributions of Ragas to include, among other things, copyright notices, but what is the copyright for Ragas? I don't see it in any source code file or in the LICENSE:

https://github.com/explodinggradients/ragas/blob/0070dbe268ab94cf81bee8d5aee09b76ac313cb7/LICENSE#L189

**Code Examples**
n/a

**Additional context**
There is a copyright statement in the documentation:

https://github.com/explodinggradients/ragas/blob/0070dbe268ab94cf81bee8d5aee09b76ac313cb7/docs/conf.py#L11

Is this correct for the whole project?

thanks for bringing this up, just added it.
Repository: explodinggradients/ragas
Issue: About the prompt of AnswerRelevancy
Root cause: Documentation Deficiency

Hi！When I try to evaluate my RAG application and computing AnswerRelevancy, I found a problem: 
The QUESTION_GEN  Prompt is written in English, so the results it generates are mostly in English. My data is not in English, so the generated results will also be in English.  Is there a bias when calculating the cosine similarity between generated questions (in English) and questions (not in English)?
would you like to modify Prompt to make the generated results adaptive to the language?（Just my opinion. Do you think it is necessary?）

Hey, please use automatic language adaptation to use ragas in languages other than English 
https://docs.ragas.io/en/latest/howtos/applications/use_prompt_adaptation.html
or you can modify prompts
https://docs.ragas.io/en/latest/concepts/prompts.html
@Gaia-Wu
Repository: explodinggradients/ragas
Issue: Evaluation stops after about 30 rows of data
Root cause: Environment Incompatibility

I'm running evaluation on a dataset using GPT 3.5 from Azure. It works well with a sample of the data, but when I try to pass a dataset with about 200 rows, the process never finishes and the progress bar freezes.
So I tried to split the data into chunks to avoid this problem. Yesterday it was working but today the evaluation is stoping in the second chunk (after around 30 - 40 rows). Does anyone know what could be going on?
No erros, it just freezes.

Code for chunking:

for _, chunk in df.groupby(chunks):
    print(f"Chunk {chunk}")
    print(f"-------------------------------")
    try:
        dataset_abi = Dataset.from_pandas(chunk)
        result_abi = evaluate(
            dataset_abi, metrics=metrics, llm=azure_model, embeddings=azure_embeddings)
        df_resultados_item = result_abi.to_pandas()
        resultado_df = pd.concat([resultado_df, df_resultados_item])
        resultado_df.to_excel(args.output_path, index=False)
        print('DF salved')
    except Exception as e:
        print(f"Erro {e}, item {chunk}")
        pass
print('End')


Have you checked your key? Is there a limit on the number of requests?

I managed to resolve the issue by inserting a 'time.sleep(60)' between the chunks. There is a limit on API requests that I was not aware of. Thank you!
Repository: explodinggradients/ragas
Issue: Clarification on Faithfulness
Root cause: Validation Gap

[X] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

Hello,

I'd like to ask how the verdict scores in Faithfulness are implemented. From my understanding of the code there is a prompt that outlines examples on how statements should be generated. Then there's this: 1 = good, 0 = bad, -1 = no verdict/NULL as the value mappings. 

But when the Faithfulness score is computed, 'NULL' values are mapped as np.nan. And this throws an error where the Faithfulness score could not be computed correctly.

Here's the prompt verdict mapping.
`NLI_STATEMENTS_MESSAGE = Prompt(
    name="nli_statements",
    instruction="Natural language inference. Use only 'Yes' (1), 'No' (0) and 'Null' (-1) as verdict.",
    ...`

Here's the compute score snippet.
`faithful_statements = sum(
            verdict_score_map.get(
                statement_with_validation.get("verdict", "").lower(), np.nan
            )
            for statement_with_validation in output
        )`


ragas==0.1.3

@sprt-kmelchor Is there an error trace? 

Hi @shahules786, 

This is what the evaluation returns.

```
/Users/KMelchor/anaconda3/lib/python3.11/site-packages/ragas/evaluation.py:273: RuntimeWarning: Mean of empty slice
  value = np.nanmean(self.scores[cn])
{'faithfulness': nan}
```

I managed to overcome this error by changing the `_compute_score` mapping. If we follow the instruction in the prompt, no-verdict statements are mapped to -1. But when we look at `_compute_score` there is no mapping for -1. This is the mapping currently:

`verdict_score_map = {"1": 1, "0": 0, "null": np.nan}`

So when` _compute_score` encounters a -1 verdict, it couldn't map it and it uses the default value (which is set to np.nan). This throws the whole computation into disarray since there is at least 1 NaN when computing for `faithful_statements`. I added a `"-1": -1 `mapping to `verdict_score_map` and it resolved the issue. 

`verdict_score_map = {"1": 1, "0": 0, "-1": -1, "null": np.nan}`

I'd just like clarification if what I did or what I understand is correct or if this would have effects in other parts of ragas.  

Hi @shahules786,

I've been looking at some of the recent Faithfulness issues and another one seems to have the same conclusion as me (NaNs are being caused by incorrect verdict mapping). You can close this if the PR from #736 pushes through. Thanks!
Repository: explodinggradients/ragas
Issue: Zero division error when evaluating context_recall
Root cause: Validation Gap

Evaluating:   0%|          | 0/30 [00:00<?, ?it/s]
Exception in thread Thread-12:
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\executor.py", line 75, in run
    results = self.loop.run_until_complete(self._aresults())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\executor.py", line 63, in _aresults
    raise e
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\executor.py", line 58, in _aresults
    r = await future
        ^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0\Lib\asyncio\tasks.py", line 615, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\executor.py", line 91, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\metrics\base.py", line 91, in ascore
    raise e
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\metrics\base.py", line 87, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\metrics\_context_recall.py", line 122, in _ascore
    return self._compute_score(response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ragas\metrics\_context_recall.py", line 103, in _compute_score
    score = numerator / denom
            ~~~~~~~~~~^~~~~~~
ZeroDivisionError: division by zero
Repository: explodinggradients/ragas
Issue: AttributeError: type object 'TestsetGenerator' has no attribute 'from_default'
Root cause: Documentation Deficiency

There is an error when load TestsetGenerator models:

from ragas.testset import TestsetGenerator
testsetgenerator = TestsetGenerator.from_default()

AttributeError: type object 'TestsetGenerator' has no attribute 'from_default'

Any suggestions?

Hey @gliu2023 can you follow the updated test generation document here
https://docs.ragas.io/en/latest/getstarted/testset_generation.html
Repository: explodinggradients/ragas
Issue: How to save intermediate evaluation results？
Root cause: Unimplemented Feature Gap

When I evaluated the last piece of data (Evaluating 99%), the program got stuck. I had to run the entire evaluation process from scratch. Can I cache the intermediate results locally, and then continue with the previous results when I evaluate again?

That's interesting @aligeniewcp22 unfortunately we don't have that feature right now. 

I have the same problem. Evaluation is stuck at the very last step. Does happen to me quite frequently, indepentently of the total step of evaluations and/or metric used.
Repository: explodinggradients/ragas
Issue: Multilingual support
GitHub labels: enhancement

**Describe the Feature**
Multilingual support 

**Why is the feature important for you?**
I'm Korean and really want to use RAGAS to evaluation Korean RAG applications.

**Additional context**
If your team guide us how to add codes to support multilingual, many contributor will be appreciated.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hey @sociengineer please refer to out language adaptation feature https://docs.ragas.io/en/stable/howtos/applications/use_prompt_adaptation.html

Feel free to reopen the issue if something doesn't work for you.
Repository: explodinggradients/ragas
Issue: The robustness of RAGAS evaluation
Root cause: Algorithmic Error

I use the GPT-4-turbo-128k as the evaluator and Huggingface BAAI/BGE-m3 as the embedding model. And I do the inference for a part of samples in [Loogle](https://github.com/bigai-nlco/LooGLE). The output results are as following:
{'faithfulness': 0.5714, 'answer_relevancy': 0.7825, 'answer_correctness': 0.5348}
 {'faithfulness': 0.6734, 'answer_relevancy': 0.7039, 'answer_correctness': 0.5164}

Why do the results of the same evaluation indicators (Faithfulness/Answer Relevance/Answer Correctness) differ greatly under the same test data set (200 samples)? Do you have any official robustness testing results?

Hi @aligeniewcp22 We are trying to improve this from our end. There is also issue #660 - For now, it's best to average 3 runs. 

If you find any ways we can improve it, would be happy to hear
Repository: explodinggradients/ragas
Issue: Evaluation taking forever to run
Root cause: Resource Mishandling

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I am running `evaluate()` on a data set with 1 single row. Even though it is estimated at **2 seconds**, it is taking **over 10 minutes** now (and continuing). Is this normal behaviour? 

**Code Examples**
I am using an Azure OpenAI model, which requires a little bit of special endpoint modifications to work. It works perfectly with `langchain` so far. Although I have no idea why the evaluation is taking that long, I suspect it's because of this?

The embedding model:
```
embedding_model = HuggingFaceEmbeddings(
  model_name=EMBEDDING_MODEL,
  model_kwargs={"device": "cuda"},
)
```

The LLM:
```
def update_base_url(request: httpx.Request):
  if request.url.path == "/chat/completions":
    request.url = request.url.copy_with(path="[different endpoint]")

llm = ChatOpenAI(
    default_headers={"Ocp-Apim-Subscription-Key": OPENAI_KEY},
    base_url=OPENAI_ENDPOINT,
    api_key=False,
    http_client=httpx.Client(
      event_hooks={
        "request": [update_base_url],
    }),
    max_retries=10,
  )
```

The evaluation:
```
result_df = evaluate(
  test_dataset,
  llm=llm,
  embeddings=embedding_model,
  metrics=[
    context_precision,
    context_recall,
    faithfulness,
    answer_similarity,
]).to_pandas()
```

**Additional context**

I have noticed a similar issue... one of my evaluations runs smoothly, albeit a bit slower on the last sample, while my other takes forever on the last sample, even with just one sample in the dataset total; if I have say sixteen, it runs through the first fifteen, then takes forever on the last.

Edit: The above only seems to occur when I have multiple evaluation runs in a single script. When I do one at a time, I have no issues.