zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: time out
Root cause: Environment Incompatibility

**Describe the bug**
Traceback (most recent call last):
  File "/root/autodl-tmp/cxl_workspace/com/basic_neural_networks_pytorch/chapter_llm/rag/ragas_tutor.py", line 69, in <module>
    eva()
  File "/root/autodl-tmp/cxl_workspace/com/basic_neural_networks_pytorch/chapter_llm/rag/ragas_tutor.py", line 44, in eva
    result = evaluate(datasets,metrics=[#faithfulness,
  File "/root/autodl-tmp/cxl_workspace/com/basic_neural_networks_pytorch/chapter_llm/rag/ragas/evaluation.py", line 102, in evaluate
    metric.score(dataset)
  File "/root/autodl-tmp/cxl_workspace/com/basic_neural_networks_pytorch/chapter_llm/rag/ragas/metrics/base.py", line 76, in score
    score = self._score_batch(dataset.select(batch), callbacks=group)
  File "/root/autodl-tmp/cxl_workspace/com/basic_neural_networks_pytorch/chapter_llm/rag/ragas/metrics/_answer_relevance.py", line 123, in _score_batch
    results = self.llm.generate(
  File "/root/autodl-tmp/cxl_workspace/com/basic_neural_networks_pytorch/chapter_llm/rag/ragas/llms/openai.py", line 153, in generate
    llm_results = run_async_tasks(
  File "/root/autodl-tmp/cxl_workspace/com/basic_neural_networks_pytorch/chapter_llm/rag/ragas/async_utils.py", line 47, in run_async_tasks
    raise RuntimeError("Fatal error occurred while running async tasks.", e)
RuntimeError: ('Fatal error occurred while running async tasks.', APITimeoutError('Request timed out.'))

Ragas version:0.0.22
Python version:3.10

**Code to Reproduce**
def eva():
    datasets = Dataset.from_json(DATAS_JSON)

    answer_similarity.threshold = None #不使用阈值，可直接记录相似度
    answer_correctness.answer_similarity.threshold = None #同上

    my_aspect = AspectCritique(name="children", definition="Is the submission safe to children?") #自定义一个aspect

    result = evaluate(datasets,metrics=[#faithfulness,
                                        answer_relevancy,
                                        context_precision,
                                        context_recall,
                                        faithfulness,
                                        ])
    
    """
                                        answer_relevancy,
                                        answer_similarity,
                                        answer_correctness,
                                        context_relevancy,
                                        context_precision,
                                        context_recall,
                                        harmfulness,
                                        maliciousness,
                                        coherence,
                                        correctness,
                                        conciseness,
                                        my_aspect
    """
    # print(result)
    reports_manager.gen_report(result)


**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Do I need to install a VPN tool on the server?




the same problem ,with vpn
Repository: explodinggradients/ragas
Issue: Python library must be updated to the latest version
Root cause: Environment Incompatibility

**Describe the bug**
The available version of the python library is old and it must be updated to the latest version of the main branch.
I need the last version to use the evaluation with GPT-4.
Ragas version: v0.0.22 
Python version: 3.9.4

**Code to Reproduce**
from langchain.chat_models import ChatOpenAI

gpt4 = ChatOpenAI(model_name="gpt-4")

from ragas.llms.base import LangchainLLMWrapper

gpt4_wrapper = LangchainLLMWrapper(langchain_llm=gpt4)

**Error trace**

**Expected behavior**
I expect that in ragas.llms.base there is LangchainLLMWrapper as in the branch main of the git repo

**Additional context**


My error is "cannot import name 'LangchainLLMWrapper' from 'ragas.llms.base' (/usr/local/lib/python3.10/dist-packages/ragas/llms/base.py)",  did you solve it

@lqcStar @GabrieleFerrario please install the version from source for now, we are working hard to resolve bugs to make a stable release ASAP.
```
pip install git+https://github.com/explodinggradients/ragas.git
```

@lqcStar yes I solved using the main branch of the repo or if you want in the stable version you can follow this: https://docs.ragas.io/en/stable/howtos/customisations/llms.html
Repository: explodinggradients/ragas
Issue: How can we use Azure OpenAI key with Ragas?
Root cause: Documentation Deficiency

Trying to run an example script the following way and got an error:

```
from ragas import evaluate
from datasets import Dataset, Features, Sequence, Value
import os

# Set the OpenAI API key
os.environ["OPENAI_API_KEY"] = "$YOUR AZURE OPENAI KEY"

# Define the structure of your dataset with the correct types
features = Features({
    'question': Value('string'),
    'contexts': Sequence(Value('string')),  # Contexts as a sequence of strings
    'answer': Value('string'),
    'ground_truths': Sequence(Value('string'))  
})

# Example data (update this with your actual data)
data = {
    'question': ["What is the capital of France?", "What is the boiling point of water?"],
    'contexts': [["Paris is known for its beautiful streets."], ["Water boils at 100 degrees Celsius."]],
    'answer': ["Paris", "100 degrees Celsius"],
    'ground_truths': [["Paris"], ["100 degrees Celsius"]]  # Note the nested lists
}

# Create the dataset
dataset = Dataset.from_dict(data, features=features)

# Evaluate the dataset
results = evaluate(dataset)

# Output the results
print(results)
```


Error:

`openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided:"**********************" . You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}`


Version : Latest installation from source ` git+https://github.com/explodinggradients/ragas.git@5f105c08b7579188aea1113334dae6b6a8a15660#egg=ragas`






Hi @abhijithneilabraham , please refer to https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html

Hi @shahules786 ,
i got the same error even i defined flow according to the guide you pointed.
can you help with more concrete example?
To keep it simple, i defined the data hard coded and sent the data for evaluate as below:
result = evaluate(
        dataset = dataset,
        metrics=[
            context_precision,
            context_recall,
            faithfulness,
            answer_relevancy,
        ],
        llm=azure_llm
    )

It seems Evaluation has been started(8%) and then got the error:
Evaluating:   8%|██████████████████▋        
....

openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: ******************************. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

Found solution- it was required to pass embedding instance as the default embedding is OpenAIEmbedding which require API_KEY which didn't define for embedding. so when i passed my embedding(not of OpenAI) it worked. Thanks anyway!
Repository: explodinggradients/ragas
Issue: ImportError: cannot import name 'LangchainLLM' from 'ragas.llms'
Root cause: Documentation Deficiency

In docs and the examples mentioned, Ragas uses `LangchainLLM` but that throws the following error:

```
ImportError: cannot import name 'LangchainLLM' from 'ragas.llms' 
```

Ragas version: git+https://github.com/explodinggradients/ragas.git@5f105c08b7579188aea1113334dae6b6a8a15660#egg=ragas
Python version: 3.9


Upon quick inspection into the [imports](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/__init__.py) I believe `LangchainLLMWrapper` should be in place for the docs.



Hi @abhijithneilabraham , you're using latest version of ragas but the docs you're referring to are from stable version. Please refer to https://docs.ragas.io/en/latest/

feel free to reopen the issue if not fixed

The docs are similar for the latest version also mentioned here and the issue still persists:

https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html

Oops, sorry. I am raising a fix for this https://github.com/explodinggradients/ragas/pull/473
Repository: explodinggradients/ragas
Issue: Context Precision Prompt Example wrong?
Root cause: Documentation Deficiency

**Describe the bug**

Hi, thanks for the work on this helpful framework!
I was going through the prompts of metrics and I don't get why the second example of Context Precision has verdict 0 for the given context:

```json
{
            "question": """who won 2020 icc world cup?""",
            "context": """Who won the 2022 ICC Men's T20 World Cup?""",
            "answer": """England""",
            "verification": {
                "reason": "the context was useful in clarifying the situation regarding the 2020 ICC World Cup and indicating that England was the winner of the tournament that was intended to be held in 2020 but actually took place in 2022.",
                "verdict": "1",
            },
        },
```

For me, that's a very misleading example.


Ragas version: build from main
Python version: 3.11



Thanks man @almajo , I have raised a fix.
Repository: explodinggradients/ragas
Issue: Answer Semantic Similarity gives output not in range 0-1
Root cause: Validation Gap

**Describe the bug**
The output of the AnswerSimilarity metric should be in range 0-1, but gives also negative output, and not in range 0-1. This affects also answer correctness metric. If i do not put the param "threshold" to None, the output results binary True/False.
Ragas version: 0.0.22
Python version:3.10.6

**Code to Reproduce**
answer_similarity=AnswerSimilarity(llm=ragas_bedrock_model,embeddings= bedrock_embeddings, threshold=None)
answer_similarity.score(ds) 
output: 35.83004634

**Expected behavior**
The score should be between 0 and 1, not specifiyng thresholds values.



Are your embeddings normalized? This might be one reason

I'm passing to the api the embedding model, not embeddings  theirself, so the library should do this. 

Hi @AnnamariaGalante , thanks for reporting this issue. Does this occur in every sample or is it once in a while issue?


Hi @shahules786, It occurs in every sample, using bedrock embeddings. I think this affects answer correctness too. 

that's interesting, Would you like to help us find out the issue here with bedrock? I think this is due to the embedding not being normalised @AnnamariaGalante 

@shahules786 how can i help you to fix this issue?
Repository: explodinggradients/ragas
Issue:  cannot import name 'AzureOpenAIEmbeddings' from 'langchain.embeddings'
Root cause: External Dependency Breakage

This is my first time using ragas, but I encountered an error when importing this package: from ragas import evaluate.

which version of Ragas are you using?
this is mostly likely due to langchain, ragas miss-match. If using ragas <=0.0.22 the only fix would be to find the langchain version that works for this - most likely langchain<0.1

Below is my version information:
ragas                      0.0.20
langchain                  0.0.200
langchain-community        0.0.13
langchain-core             0.1.13
langchainplus-sdk          0.0.20
langsmith                  0.0.83
openai                     1.0.1

I upgrade my langchain version,this problme be solved~

@chenyunsai thanks for the update, closing the issue on that remark.
Repository: explodinggradients/ragas
Issue: OPENAI_API_KEY set in google cloud function cannot be found by RAGAS library
Root cause: Configuration Error

**Describe the bug**
I am deploying a google cloud function using functions_framework, that takes in an OPENAI_API_KEY as a parameter and setting the OPENAI_API_KEY environment before calling the RAGAS evaluation. However I receive the message 

{"error":"OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable"} 

This only happens when I try to set the Environment Variable within the http function using os.environment["OPENAI_API_KEY"] = , if I set the OPENAI_API_KEY environment variable on the google cloud functions console environment variables it works, it also works when I edit the ragas library code and add os.environment["OPENAI_API_KEY"] = , in the _answer_relevance.py file inside the metrics folder.




I faced the same issue, apparently the problem is related to this  _answer_relevance.py file/metric indeed

ragas.exceptions.OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable


i am using open-source llm but still i m getting this error.

tried with ragas==0.0.20/0.0.22

tracking this here folks - this has been fixed for you in main and will add docs soon #503
Repository: explodinggradients/ragas
Issue: context_relevancy Judgment Logic Vulnerability
Root cause: Validation Gap

for context, n_response in zip(contexts, responses):
                context = "\n".join(context)
                overlap_scores = []
                context_sents = sent_tokenize(context)
                for output in n_response:
                    indices = (
                        sent_tokenize(output.strip())
                        if output.lower() != "insufficient information."
                        else []
                    )
                    if len(context_sents) == 0:
                        score = 0
                    else:
                        score = min(len(indices) / len(context_sents), 1)
                    overlap_scores.append(score)
                scores.append(np.mean(overlap_scores))


output.lower() != "insufficient information."  As '.' does not exist, then an exception is recognized
Repository: explodinggradients/ragas
Issue: TestsetGenerator runs into "Invalid json: Extra data"
Root cause: Validation Gap

**Describe the bug**
 **self.critic_llm.generate_text_with_hmpt** in the **_filter_context** function under **testset_generator.py** produces results that have additional symbols after the json part of the answer. Sometimes it even returns the rating for the example context about Albert Einstein, additionally to the actual rating. This leads to the next step, load_as_json, to fail, rendering all TestsetGenerator attempts useless, as none of the documents receive scores.

Ragas version: current main (1.0.23dev)
Python version: 3.11.7

**Code to Reproduce**
`results = self.critic_llm.generate_text_with_hmpt(prompts=[prompt])`
`output = results.generations[0][0].text.strip()`

output
'{"score":2.0, "is_table_present":true}**
**'

`output = load_as_json(output)`

Invalid json: Extra data: line 1 column 39 (char 38)

**Expected behavior**
These two lines need to be more robust to handle possible LLM weirdness. For example, instead of simply stripping the text, grab everything within curly brackets or that looks like json. So instead of:

```
output = results.generations[0][0].text.strip()
output = load_as_json(output)
```

we could have:

```
import re
json_objects = re.findall(r'{[^}]*}', results.generations[0][0].text)
output = json_objects[-1].strip()
output = load_as_json(output)
```


Hey @tenshis , can you update to the latest version and try using our improved version of this? https://docs.ragas.io/en/latest/concepts/testset_generation.html
That should solve your issue

Hi shahules,

This is now working with the newest version (0.1.0rc1) and using gpt4
Repository: explodinggradients/ragas
Issue: Latest tag Incompatible
Root cause: External Dependency Breakage

Doing a `poetry add ragas` installs version 0.0.22 
It downloads required langchain( +core + community), llamaindex and many other packages

But many things have changed in the dependencies, and existing imports fail for this version (at mltiple places, I am using TestsetGenerator).

I also see that the last release was made on Dec 12. Can anything be done regarding it?

Hey @kediaharshit9 sorry for the late reply but you can now try our pre-release version
pip install ragas==0.1.0rc1

closing this since its fixed now :)
Repository: explodinggradients/ragas
Issue: Llama Index | Compatibility Issue
Root cause: External Dependency Breakage

**Describe the bug**
The latest llama index is not compatible with the imports in the test_generator.py file.

Ragas version: 0.0.22
Python version: 3.11


**Error trace**

```
from llama_index.node_parser.simple import SimpleNodeParser
ModuleNotFoundError: No module named 'llama_index.node_parser.simple'
```



tracking this issue here #557 
closing this for now but will get the llamaindex issue sorted for you this week :)
Repository: explodinggradients/ragas
Issue: OPENAI_API_KEY' environment variable
Root cause: Validation Gap

ragas.exceptions.OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable

i am using open-source llm but still i m getting this error.

tried with ragas==0.0.20/0.0.22

Seems to be related to the pip version, if you install from source it works.

still same error

Maybe you can provide your code?


will fix this #503 

closing for now because the fix has been made and the documentation is tracking in the above PR.

really sorry about this guys but with v0.1 you'll have a much simpler experience (I hope 🤞🏽 )
Repository: explodinggradients/ragas
Issue: Example for AzureOpenAI is broken
Root cause: Documentation Deficiency

**Describe the bug**
The example on this page no longer works with the current RAGAS build: https://docs.ragas.io/en/latest/howtos/customisations/embeddings.html#evaluating-with-azure-open-ai-embeddings

For example, **from ragas.llms import LangchainLLM** doesnt exist anymore. Should this be replaced with **from ragas.llms.base import LangchainLLMWrapper** ?
Repository: explodinggradients/ragas
Issue: context_precision, OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable
Root cause: Environment Incompatibility

hello,there was something wrong.

from ragas.metrics import (
answer_relevancy,
faithfulness,
context_recall,
context_precision,
)
error information:cannot import name 'context_precision' from 'ragas.metrics' (c:\python\lib\site-packages\ragas\metrics_init_.py).

so ,in sep 2023,
from ragas.metrics import (
context_relevancy,
answer_relevancy,
faithfulness,
context_recall,
)

there have some changes. so I really want to know the difference. as the same ,i do not have ground_truth.

also  OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable.


Seems to be related to the pip version, if you install from source it works.

AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: adasfafdaf. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

even after installing from source

make some stable release, every new release fucks up

@riyaj8888 please provide a valid open-ai key, your issue must be resolved.

Brother , I am using open source LLM.
Not open ai but still it's giving the same error.
I tried installing from source also

In my occasion, the `evaluate()` somehow reaches `os.environ["OPENAI_API_KEY"]` and OpenAI API even after wrapping open source LLM instance and setting metrics, causing the problem.

Hi @riyaj8888 @Lanture1064 Can you both please share the ragas version you're using? I would highly recommend you to shift to latest version by using `pip install ragas=="ragas==0.1.0rc1`

@shahules786 Thx for reply, and yes I was using pip 0.0.22 version. Trying the newer 0.1.0 ver now

closing this because it has been fixed in the latest and will be released soon. Feel free to reopen if needed.

but the current usage pattern is that the `evaluate()` function takes to additional arguments `llm` and `embeddings`. The doc string explains their behaviour
```
    llm: BaseRagasLLM, optional
        The language model to use for the metrics. If not provided then ragas will use
        the default language model. This can be overridden by the llm specified in
        the metric level with `metric.llm`.
    embeddings: BaseRagasEmbeddings, optional
        The embeddings to use for the metrics. If not provided then ragas will use
        the default embeddings. This can be overridden by the embeddings specified in
        the metric level with `metric.embeddings`.
```

please feel free to share your suggestion here, I'm sorry we are not perfect but we will try our best to get there for you guys :)
Repository: explodinggradients/ragas
Issue: Bug in Answer Similarity Calculation
Root cause: Algorithmic Error

**Describe the bug**
Hi, I've noticed some change in the implementation of AnswerSimilarity in latest version of ragas

in v0.0.22, the implementation is 

def _score_batch(
        self: t.Self,
        dataset: Dataset,
        callbacks: t.Optional[Callbacks] = None,
        callback_group_name: str = "batch",
    ) -> list[float]:
        ground_truths, answers = dataset["ground_truths"], dataset["answer"]
        ground_truths = [item[0] for item in ground_truths]

and it change to

def _score(self, row: t.Dict, callbacks: Callbacks) -> float:
        assert self.embeddings is not None, "embeddings must be set"

        ground_truths, answers = row["ground_truths"], row["answer"]
        ground_truths = [item[0] for item in ground_truths]

row is the a sample in dataset, "ground_truths = [item[0] for item in ground_truths]" only take first character of str, 
so i suppose it is a bug

Ragas version:build from source
Python version:3.11

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.


same issue, also for answer relevancy! it appens on version 0.0.22 but not in 0.0.21

We will add this in the documentation but in the meantime tagging @shahules786 so that he can share the reasoning behind it

We will add this as part of our v0.1 migration docs