zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: AzureOpenAI evals runs forever
Root cause: Configuration Error

**Describe the bug**
I followed the "[Using AzureOpenAI](https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html)" tutorial. Though, I had to eliminate "os.environ["OPENAI_API_BASE"]" as it was giving me the following error:

base_url and azure_endpoint are mutually exclusive (type=value_error)

Once eliminated, the rest of the code ran with no error, but the evaluation step keeps running forever, with no progress being shown.

Ragas version: 0.0.2
Python version: 3.11.5

**Code to Reproduce**

from ragas import evaluate

result = evaluate(
    fiqa_eval["baseline"],
    metrics=metrics,
)

result

**Error trace**

evaluating with [faithfulness]
  0%|          | 0/2 [00:00<?, ?it/s]

**Expected behavior**
Evaluation results from example dataset


**Additional context**
Aside from eliminating the OPENAI_API_BASE environment variable, no changes were made to the code in the AzureOpenAI documentation.


hey @AJAC2000 thanks for raising this. 

could you confirm a few things
1. update to latest version of Ragas v0.0.21 and check again
2. that this happens everytime and not due to openai client waiting for service to respond and then timeout

Hi @AJAC2000, try importing the evaluate function before patching the RagasLLM instance. 
Or if you are using a notebook, try adding :
import nest_asyncio
nest_asyncio.apply()
before calling the evaluate function
Repository: explodinggradients/ragas
Issue: OpenAI API key not found
Root cause: Configuration Error

**Describe the bug**

I have set up my OpenAI API key as follows: 

`os.environ["OPENAI_API_KEY"] = 'my key'`

This is how it is described in the main page of the project and this is how I do with langchain and works as expected. 

However, I get the follower error: 

```
raise OpenAIKeyNotFound
ragas.exceptions.OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable
```

Ragas version: 0.0.20
Python version: 3.10

Why is this happening? How can I fix it? 

Many thanks for creating this library and sharing it with us! 


Hi Always Same Issue ;(

You need to set the environment variable **before** importing any ragas modules. I am not sure if this is intentional or not.

@ceferisbarov Thx that works for me.

yep @ceferisbarov is right and that was a bug but has been fixed with the latest release.

please do reopen if this is still a problem

I'm facing the same issue here with the latest RAGAS version i.e. `0.0.21`
Moving the RAGAS imports after setting the OpenAI API keys solved the issue for me as suggested by @ceferisbarov 

thanks, I change my code to: 

```
import os
os.environ["OPENAI_API_KEY"] = "xxxx"

from datasets import Dataset
from ragas import evaluate

```

than's work. 

Remember to set openai key before you import rages
Repository: explodinggradients/ragas
Issue: feat: Add `Executor` for faster and reliable evalutions
Root cause: Unimplemented Feature Gap

- [ ] make `ascore()` for scoring in singles
- [ ] async coroutine executor that executes tasks with checks and retries - add tests
- [ ] evaluator has an async=True flag that scores these as tasks
- [ ] executor has Threads support - it can now evaluate threads also
- [ ] make sure langsmith is working properly - fix langfuse issues


finished with v0.1
Repository: explodinggradients/ragas
Issue: Issue with column mapping
Root cause: Validation Gap

**Describe the bug**
Column map requires to input all required column names instead of only mapped columns. For example if I have a dataset with columns ["answer", "contexts"] and I need to use a metric like context_recall, Ideally I should only be passing `{"ground_truths":"answer"}` but here I have to pass `{"contexts":"context_v1","ground_truths":"ground_truths"}`


**Code to Reproduce**
```
from datasets import load_dataset
from ragas.metrics import context_recall
from ragas import evaluate

dataset = load_dataset("explodinggradients/WikiEval")
dataset = dataset['train'].map(lambda x : {'ground_truths':[x['answer']]})
ragas_score = evaluate(dataset=dataset,metrics=[context_recall],column_map={"contexts":"context_v1"})
```



@shahules786 I can work on this issue

Raised a PR for this issue. 

this is fixed right @tinomaxthayil ?

This is fixed with  [##331](https://github.com/explodinggradients/ragas/pull/331) and can be closed
Repository: explodinggradients/ragas
Issue: import error:  from ragas.llms import LangchainLLM
Root cause: External Dependency Breakage

**Describe the bug**
Tried to import from ragas.llms import LangchainLLM.
Getting error: ImportError: cannot import name 'AzureOpenAIEmbeddings' from 'langchain.embeddings'

Ragas version: 0.0.20
Python version: 3.11.4

**Code to Reproduce**
from ragas.llms import LangchainLLM

**Error trace**
ImportError: cannot import name 'AzureOpenAIEmbeddings' from 'langchain.embeddings'

**Expected behavior**
AzureOpenAIEmbeddings import without any errors




Did you try 

`from langchain.embeddings.azure_openai import AzureOpenAIEmbeddings`

FYI https://api.python.langchain.com/en/latest/embeddings/langchain.embeddings.azure_openai.AzureOpenAIEmbeddings.html?highlight=azureopenaiembeddings#langchain-embeddings-azure-openai-azureopenaiembeddings

Yes, I tried still. Getting the error ModuleNotFoundError: No module named 'langchain.embeddings.azure_openai'

you should check your langchain version and compare with the langchain docs / reference

Thanks, after updating langchain version, issue got resolved

How do you update it, I update it by"pip install --upgrade langchain",but still " cannot import name 'LangchainLLMWrapper' from 'ragas.llms.base' "
Repository: explodinggradients/ragas
Issue: issues with llama_index dependency
Root cause: Validation Gap

**Describe the bug**
`llama_index` is imported in multiple places without any exception handling, but it is not added to the requirements. I appreciate the idea of avoiding bloated requirements, but we should probably add a user-friendly error message. You actually have done this in some places with the `unstructured` library.

Ragas version: 0.0.20
Python version: 3.10.12

**Code to Reproduce**
```python
from langchain.document_loaders import DirectoryLoader
from ragas.testset import TestsetGenerator

loader = DirectoryLoader("./data", glob="*.txt")
documents = loader.load()

testsetgenerator = TestsetGenerator.from_default()
test_size = 10
testset = testsetgenerator.generate(documents, test_size=test_size)
```

**Error trace**
```
Traceback (most recent call last):
  File "/home/jafar/projects/rag-evaluate/src/generate_testset.py", line 2, in <module>
    from ragas.testset import TestsetGenerator
  File "/home/jafar/projects/rag-evaluate/env/lib/python3.10/site-packages/ragas/testset/__init__.py", line 1, in <module>
    from ragas.testset.testset_generator import TestsetGenerator
  File "/home/jafar/projects/rag-evaluate/env/lib/python3.10/site-packages/ragas/testset/testset_generator.py", line 16, in <module>
    from llama_index.node_parser.simple import SimpleNodeParser
ModuleNotFoundError: No module named 'llama_index.node_parser.simple'
```

**Expected behavior**
The following line should be printed:
```
ValueError: llama_index package not found, please install it with `pip install llama_index`
```
**Additional context**
There is also another issue regarding llama_index versions. I believe that's also relevant to the discussion, so I will create a new issue and attach here.

Now that the PR has been merged, I guess we can close this.
Repository: explodinggradients/ragas
Issue: AttributeError: 'Bedrock' object has no attribute 'validate_api_key' (ragas-0.0.21)
Root cause: Validation Gap

https://github.com/explodinggradients/ragas/blob/3d29c44825eff39775f458af7c2e5d51411ac277/src/ragas/metrics/base.py#L121

```
import boto3
from langchain.llms import Bedrock
# from langchain.chat_models import BedrockChat # Same error using chat model


def get_claude_v2_llm(credentials_profile_name="default"):
    return Bedrock(
    credentials_profile_name=credentials_profile_name,
    # client=bedrock_runtime_client,
    model_id="anthropic.claude-v2",
    model_kwargs={
        "max_tokens_to_sample": 5000,
        "temperature": 0.0,
        "top_k": 250,
    }
)

claude_v2_llm = get_claude_v2_llm(credentials_profile_name="default")



for m in metrics:
    m.__setattr__("llm", claude_v2_llm)


from ragas import evaluate
import nest_asyncio  # CHECK NOTES

result = evaluate(
    fiqa_eval["baseline"],
    metrics=metrics,
)

result
```







ERROR:
    115 def init_model(self):
    116     """
    117     Init any models in the metric, this is invoked before evaluate()
    118     to load all the models
    119     Also check if the api key is valid for OpenAI and AzureOpenAI
    120     """
--> 121     self.llm.validate_api_key()
    122     if hasattr(self, "embeddings"):
    123         # since we are using Langchain Embeddings directly, we need to check this
    124         if hasattr(self.embeddings, "validate_api_key"):

AttributeError: 'Bedrock' object has no attribute 'validate_api_key'

I am also interested in using ragas with Bedrock.

This fixed it:
```
from ragas.llms import LangchainLLM
ragas_bedrock_model = LangchainLLM(claude_v2_llm)
```
Repository: explodinggradients/ragas
Issue: TestsetGenerator silently ignoring test cases
Root cause: Validation Gap

**Describe the bug**
`TestsetGenerator.generate` method takes `test_size` as an argument, but it usually returns less test cases than this number. I understand test cases that do not meet certain requirements are removed, but I had to search through the source code for some time before discovering that. In a similar fashion, `to_pandas` method removes test cases where question is an empty string, but again, no warnings.

Ragas version: 0.0.21.dev8+g13465f0
Python version: 3.10.12

**Code to Reproduce**
```
import argparse

from ragas.testset import TestsetGenerator
from langchain.document_loaders import TextLoader, DirectoryLoader

from config import OPENAI_API_KEY, LLAMA_URL, BGE_URL
from src.llms import LLaMA, BGE, OpenAILLM

parser = argparse.ArgumentParser()
parser.add_argument("--file", help="File path to be loaded. Mutually exclusive with folder argument.")
parser.add_argument("--folder", help="Folder path to be loaded. Mutually exclusive with folder argument.")
parser.add_argument("--outdir", help="Output directory to save to.")
parser.add_argument("--count", help="Number of test cases to be generate.")
args = parser.parse_args()

if args.file:
    loader = TextLoader(args.file)
elif args.folder:
    loader = DirectoryLoader(args.folder)
else:
    raise ValueError("Specify file or folder input by using one of the --file or --folder flags.")
    
documents = loader.load()

llama = LLaMA(url=LLAMA_URL)
openai = OpenAILLM(api_key=OPENAI_API_KEY)
bge = BGE(base_url=BGE_URL)

testsetgenerator = TestsetGenerator(generator_llm=openai, critic_llm=openai, embeddings_model=bge)

print(int(args.count))
testset = testsetgenerator.generate(documents, test_size=int(args.count))

df = testset.to_pandas()
print(df.shape)
df.to_csv(args.outdir)
```

**Expected behavior**
I think it is necessary to either add a warning or make these filters optional.


I can create a PR if you have an idea about how to approach this @jjmachan @shahules786 

Hi @ceferisbarov , thanks for raising this issue. We are aware of this issue, this will be solved in the next release of test generation. 

@shahules786 it this meant to have been solved? Because I'm still experience the same.
Requested for `test_size=500`, got 498.
Requested for `test_size=1000`, got 998.

Ragas version: `0.1.2.dev8+gc18c7f4`
Python version: 3.9.13

@c-yeh Oh, I think your issue is due to the distribution of question types. Will raise a fix for it.
Repository: explodinggradients/ragas
Issue: Outdated Documentations for Azure-Openai Customization ?
Root cause: Documentation Deficiency

**Describe the bug**
In https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html the section says

```
# init and change the embeddings
# only for answer_relevancy
azure_embeddings = OpenAIEmbeddings(
    deployment="your-embeddings-deployment-name",
    model="your-embeddings-model-name",
    openai_api_base="https://your-endpoint.openai.azure.com/",
    openai_api_type="azure",
)
```
 However, the code would raise this exception on the `answer_relevancy` step

```
NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}
```

changing `OpenAIEmbeddings` to `AzureOpenAIEmbeddings` resolved the problem




Can I work on this?

@ajinkyavbhandare  yes, please. We would love to get your help :) 

Thanks a lot @ajinkyavbhandare 

The fix is wrong because the parameters for `AzureOpenAIEmbeddings` are not the same as for `OpenAIEmbeddings`
Repository: explodinggradients/ragas
Issue: [Feature Request] Custom synthetic data generation prompts
Root cause: Unimplemented Feature Gap

Hi!

I apologize if this feature already exists but I found no evidence of it neither in the documentation nor in the source code. 

I'm working with the synthetic data generation feature. I'm using context documents written in italian language but the output questions and the corresponding ground truth answers are in english and it would be great if they were in the same language of the input context documents. 

I've made some changes to the code to use custom prompts and put them here:
[https://github.com/faradri/ragas/tree/testset-prompts](https://github.com/faradri/ragas/tree/testset-prompts)

Thank you

hey @faradri we are looking into how we can best solve for this and will keep you posted about an easier way to do this rather than having to modify the code yourself 

but welcome to the community :)

Hi @jjmachan.

Thank you for your comment and thank you for your welcome! :)
I made a change to the code that seemed to me less "invasive" as possible.
Repository: explodinggradients/ragas
Issue: Test issue

**Describe the bug**
A clear and concise description of what the bug is.

Ragas version:
Python version:

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.


new comment
Repository: explodinggradients/ragas
Issue: TestsetGenerator with own LLM
Root cause: Documentation Deficiency

Hi,

I want to use the Ragas TestsetGenerator to create a synthetic dataset. The function is using OpenAI in the background, but I was wondering if you can set your own LLM to create the synthetic dataset? 

For example for evaluation it is possible to set an own LLM and not an OpenAI Model. Like here:
```
from ragas.llms import LangchainLLM
from src.llm import build_llm


llm = build_llm() # this is loading the Llama 2 model with LlamaCpp from Langchain
llm = LangchainLLM(llm=llm)
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness

# change the LLM

faithfulness.llm = llm
answer_relevancy.llm = llm
context_precision.llm = llm
context_recall.llm = llm
harmfulness.llm = llm
```

hey @weissenbacherpwc welcome to the community :)

this is possible you can create a new testset generation object to achieve it but the issue is that it might change in the future because testset generation is a beta feature.

but check the code [here](https://github.com/explodinggradients/ragas/blob/fe343d6fdcd0e11096fc1b8fbf4f36d289f32eaa/src/ragas/testset/testset_generator.py#L87) . you can create a new `TestsetGenerator` with any `LangchainLLM`
Repository: explodinggradients/ragas
Issue: error: non-openai llm model has no attribute 'validate_api_key'
Root cause: Validation Gap

**Describe the bug**

An error is raised when evaluating Langchain QA Chains with a non-openai llm model.

Ragas version: 0.0.21
Python version: 3.10.0

**Code to Reproduce**

```
# list of metrics we're going to use
metrics = [
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
]

for m in metrics:
    m.__setattr__("llm", bedrock_llm)

faithfulness_chain = RagasEvaluatorChain(metric=metrics[0])
```

**Error trace**

```
AttributeError                            Traceback (most recent call last)
Cell In[25], line 24
     21     m.__setattr__("llm", llm)
     23 # create evaluation chains
---> 24 faithfulness_chain = RagasEvaluatorChain(metric=metrics[0])
     25 answer_relevancy_chain = RagasEvaluatorChain(metric=metrics[1])
     26 context_precision_chain = RagasEvaluatorChain(metric=metrics[2])

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/langchain/evalchain.py:29, in RagasEvaluatorChain.__init__(self, **kwargs)
     27 def __init__(self, **kwargs: t.Any):
     28     super().__init__(**kwargs)
---> 29     self.metric.init_model()

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/metrics/base.py:121, in MetricWithLLM.init_model(self)
    115 def init_model(self):
    116     """
    117     Init any models in the metric, this is invoked before evaluate()
    118     to load all the models
    119     Also check if the api key is valid for OpenAI and AzureOpenAI
    120     """
--> 121     self.llm.validate_api_key()
    122     if hasattr(self, "embeddings"):
    123         # since we are using Langchain Embeddings directly, we need to check this
    124         if hasattr(self.embeddings, "validate_api_key"):

AttributeError: 'BedrockChat' object has no attribute 'validate_api_key'
```

**Expected behavior**
It should not raise an error because models from Amazon do not need an api key and the `validate_api_key` does not exist.


closed with #350
Repository: explodinggradients/ragas
Issue: AttributeError: 'NoneType' object has no attribute 'lower' while using evaluation (from ragas import evaluate).
Root cause: Validation Gap

I'm getting the below error while using evaluate from ragas (from ragas import evaluate).
AttributeError: 'NoneType' object has no attribute 'lower'



thanks to @yuukidach this has been significantly improved. Please feel free to reraise this if you encounter this again @debanshu005
Repository: explodinggradients/ragas
Issue: Will support for custom judge models be added?
Root cause: Documentation Deficiency

Currently it seems that we can only use OpenAI models as LLM judges. Is there a plan to allow for custom LLMs to be used as judges (e.g. by extending a certain class)?

you can with [Bring your own LLMs | Ragas](https://docs.ragas.io/en/stable/howtos/customisations/llms.html)
do check it out and feel free to reopen in case of more doubts 🙂
Repository: explodinggradients/ragas
Issue: How to create/generate dataset with my own data?
Root cause: Documentation Deficiency

Hi everyone,
I'm designing a chatbot and would like to check its responses.
I saw that Ragas uses an example of a dataset with the following layout:
_question, ground_truths, answer, contexts_

I would like to recreate the dataset based on the values ​​of my chatbot, but I have doubts about the file format I need to pass and how to create it.
Do you know how to help me?

Thank you very much and good job
Repository: explodinggradients/ragas
Issue: TestsetGenerator function shows: UserWarning: Invalid json warnings.warn("Invalid json")
Root cause: Validation Gap

**Describe the bug**
LLM does not provide a valid JSON answer to the FILTER_QUESTION prompt. 

Ragas version: 0.0.21
Python version: 3.11.6

**Code to Reproduce**
`test_generator = TestsetGenerator(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings_model=embeddings_model,
    testset_distribution=testset_distribution,
    chat_qa=chat_qa,
    chunk_size=1024,
)

testset = test_generator.generate(documents, test_size=30)`

**Error trace**
```
[/Users/omar/Documents/ai_repos/towardsai_buster_api/env/lib/python3.11/site-packages/ragas/testset/utils.py:16](https://file+.vscode-resource.vscode-cdn.net/Users/omar/Documents/ai_repos/towardsai_buster_api/env/lib/python3.11/site-packages/ragas/testset/utils.py:16): UserWarning: Invalid json
  warnings.warn("Invalid json")
```
**Expected behavior**
The LLM generates a valid JSON. 


**Additional context**
Can be fixed with additional instructions in the prompt like:
`FILTER_QUESTION = HumanMessagePromptTemplate.from_template(
    """\
Determine if the given question can be clearly understood even when presented without any additional context. Specify reason and verdict is a valid json format.
question: What is the keyword that best describes the paper's focus in natural language understanding tasks?
{{"reason":"The specific paper being referred to is not mentioned in the question.", "verdict": "No"}}
WARNING: only output JSON directly without any additional text like '''JSON
question:{question}
"""  # noqa: E501
)`
Repository: explodinggradients/ragas
Issue: RAGAS Evaluate function doesn't work with testset generated by RAGA
Root cause: Interface Contract Mismatch

**Describe the bug**
I have created test set using test_generator.generate(documents, test_size=2) function. I am not able to use this test set with ragas evaluate function 
Ragas version: 0.0.21
Python version: 3.10

**Code to Reproduce**
from langchain.document_loaders import PyPDFDirectoryLoader

data_file = "/content/"
loader = PyPDFDirectoryLoader(data_file+"/")
documents = loader.load()

from ragas.testset import TestsetGenerator
from langchain.embeddings import OpenAIEmbeddings

from ragas.llms import LangchainLLM
from langchain.embeddings import SentenceTransformerEmbeddings
#s
generator_llm = LangchainLLM(llm=llm_wrapper)
critic_llm = LangchainLLM(llm=llm_wrapper)
embeddings_model =  OpenAIEmbeddings()

# Change resulting question type distribution
testset_distribution = {
    "simple": 0.25,
    "reasoning": 0.5,
    "multi_context": 0.0,
    "conditional": 0.25,
}

# percentage of conversational question
chat_qa = 0.2


test_generator = TestsetGenerator(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings_model=embeddings_model,
    testset_distribution=testset_distribution,
    chat_qa=chat_qa,
)

testset = test_generator.generate(documents, test_size=2)

**Error trace**
AttributeError                            Traceback (most recent call last)
[<ipython-input-30-4a046bb39140>](https://localhost:8080/#) in <cell line: 2>()
      1 from ragas import evaluate
----> 2 result = evaluate(
      3     testset, # selecting only 3
      4     metrics=[
      5         context_precision,

2 frames
[/usr/local/lib/python3.10/dist-packages/ragas/validation.py](https://localhost:8080/#) in <dictcomp>(.0)
     10     Remap the column names in case dataset uses different column names
     11     """
---> 12     column_map = {k: v for k, v in column_map.items() if v in dataset.column_names}
     13     inverse_column_map = {v: k for k, v in column_map.items()}
     14     return dataset.from_dict(

AttributeError: 'TestDataset' object has no attribute 'column_names'
**Expected behavior**
the test set generated from ragas test generation function should work with ragas evaluate function


**Additional context**
ragas should work with all llm models like mistral-7b as working with open ai