zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: AzureOpenAIEmbeddings no longer exists on langchain
Root cause: Environment Incompatibility

**Describe the bug**
Bug in import, cannot import AzureOpenAIEmbeddings because it no longer exists
Ragas version: 0.0.21

Langchain version: 0.0.325

**Code to Reproduce**
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)
from ragas import evaluate

**Error trace**
ImportError: cannot import name 'AzureOpenAIEmbeddings' from 'langchain.embeddings'
**Expected behavior**
Import works fine

Try updating your langchain version to the latest version via: 

```python
pip install --upgrade langchain
```

`AzureOpenAIEmbeddings` were added in langchain release: https://github.com/langchain-ai/langchain/releases/tag/v0.0.331rc3
Your installed version of langchain is older, hence does not include the `langchain.embeddings.azure_openai.AzureOpenAIEmbeddings` class.


Be aware, **updating your version of langchain might require some adjustments in your code** regarding the openai evironment variables (`AZURE_OPENAI_...` instead of `OPENAI_...`) and `AzureOpenaiAIChat` / `AzureOpenaiAIEmbeddings` class usage (`azure_endpoint` instead of `openai_api_base`).
Repository: explodinggradients/ragas
Issue: Problem running the Evaluation using my personal dataset
Root cause: Documentation Deficiency

Hi everyone,
I'm using my own personal dataset, but I can't understand in the evaluation code (present below), what I should replace ['baseline']. Do you know how to help me?




_from ragas import evaluate
result = evaluate(
    fiqa_eval["baseline"].select(range(3)),    <---------          
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
    ], 
)_



Hi @MattiaBaldinetti , please refer to https://docs.ragas.io/en/latest/howtos/applications/data_preparation.html

> Hi @MattiaBaldinetti , please refer to https://docs.ragas.io/en/latest/howtos/applications/data_preparation.html

Hi @shahules786 
I would like to use the _train.csv_ file present in my dataset uploaded to HuggingFace:
https://huggingface.co/datasets/mattobaldo/personal_dataset/tree/main

Loading the dataset using load_dataset("mattobaldo/personal_dataset") returns:
_DatasetDict({
    train: Dataset({
        features: ['question', 'ground_truths', 'answer', 'contexts'],
        num_rows: 30
    })
})_

Loading metrics is fine.
The problem is when I launch the Evaluation, as a reference code I am using the one written in the problem above.
Do you know how I can solve it?

Thank you

```python 
from datasets import load_dataset
from ragas import evaluate

your_dataset = load_dataset("mattobaldo/personal_dataset")
result = evaluate(
your_dataset["train"],
metrics=[
context_precision,
faithfulness,
answer_relevancy,
context_recall],)
```

This should do it @MattiaBaldinetti 



Hi @shahules786 and thanks for the reply
I tried to use this code, but I get this error:
_ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Value'>_

What should I do?


Please refer to https://github.com/explodinggradients/ragas/issues/286. 
I would suggest 
1. Joining our discord server (link in README) for such queries
2.  Searching older issues before raising new ones. 
Thank you.  @MattiaBaldinetti
Repository: explodinggradients/ragas
Issue: the trace_as_chain_group context do not contain the code
Root cause: Validation Gap

https://github.com/explodinggradients/ragas/blame/68c7ed67da25c93172f093912646ba6a845a595c/src/ragas/metrics/_answer_correctness.py#L98-L137


- the trace_as_chain_group context do not contain the code
- add the callback to answer_similarity._score_batch method

this has been fixed with #390
Repository: explodinggradients/ragas
Issue: Llama CPP & LlamaIndex Support
Root cause: Interface Contract Mismatch

**Describe the bug**
Currently running a combination of LlamaIndex with local LlamaCPP models does not work properly and throws an error.


Ragas version: 0.0.21
Python version: 3.10

**Code to Reproduce**
 Here is my code for reference:

```
from ragas.metrics import faithfulness
from ragas.llms import LlamaIndexLLM
from ragas.llama_index import evaluate

from RAGIR.utils.util_functions import (
    initialize_llama_cpp,
    initialize_service_context,
    initialize_embedding,
)
from RAGIR.database.setup_database import (
    WeaviateVectorDB
)

client_url = "http://localhost:8080"
embedding_model = "BAAI/bge-small-en-v1.5"
model_path_7b = "/Users/torky/Documents/LlamaCPP-2/llama-2-7b-chat.Q4_0.gguf"
model_path_13b = "/Users/torky/Documents/LlamaCPP-2/llama-2-13b-chat.Q4_0.gguf"
documents_path = "/Users/torky/Documents/thesis-torky/docs/research_papers"
index_name = "IrAnthology"

llama_7b = initialize_llama_cpp(
    model_path=model_path_7b,
    temperature=0,
    max_new_tokens=2048,
    context_window=3900,
    model_kwargs={"n_gpu_layers": 2},
    verbose=False,
)

llama_13b = initialize_llama_cpp(
    model_path=model_path_13b,
    temperature=0,
    max_new_tokens=2048,
    context_window=3900,
    model_kwargs={"n_gpu_layers": 2},
    verbose=False,
)

embedding = initialize_embedding(model_name=embedding_model)

service_context_llama_7b = initialize_service_context(
    llm=llama_7b, embed_model=embedding
)

weaviate_db = WeaviateVectorDB(
    client_url=client_url,
    embedding_model=embedding_model,
    index_name=index_name,
    documents_path=documents_path,
)

query_engine = weaviate_db.get_query_engine(
    prompt_template="",
    alpha=1,
    exists=True,
    service_context=service_context_llama_7b,
    similarity_top_k=1,
    chunk_overlap=0,
    chunk_size=200
    )

faithfulness.llm = llama_13b

questions: list[str] = ["What did the author in 'A Survey on Hallucination in Large Language Models' classify the three types of hallucinations?"]
metrics = [faithfulness]

r = evaluate(
    query_engine=query_engine,
    metrics=metrics,
    questions=questions
)

print(r)
results = r.to_pandas()

results.to_csv("evaluation.csv")

```


**Error trace**

The error I'm getting:
```
AttributeError: 'LlamaCPP' object has no attribute 'validate_api_key' 
```

And in the case in which I switch to LangChainLLMs and use the wrapper, and set 
```
faithfulness.llm = LangChainLLM(llama_13b)
```

```

RuntimeError: ('Fatal error occurred while running async tasks.', AttributeError("'LlamaCPP' object has no attribute 'agenerate'"))
/Users/torky/anaconda3/lib/python3.10/asyncio/base_events.py:688: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>

```
Repository: explodinggradients/ragas
Issue: get_openai_callback() doesn't return results in ragas 0.0.21
Root cause: Interface Contract Mismatch

**Describe the bug**
I'm using OpenAI to do the evaluation (all with default settings). With all input unchanged, in ragas 0.0.19 it does return the cost info. But in 0.0.21 it doesn't
```
from langchain.callbacks import get_openai_callback
...
with get_openai_callback () as cb:
    result = evaluate(
        data,
        metrics=[
            answer_similarity,
            faithfulness,
        ],
    )
    print('\n Done evaluating. OpenAI cost:')
    print(cb)
```

**ragas 0.0.21**
```
Tokens Used: 0
        Prompt Tokens: 0
        Completion Tokens: 0
Successful Requests: 0
Total Cost (USD): $0.0
```

**ragas 0.0.19**
```
  block_group = [InMemoryTable(cls._concat_blocks(list(block_group), axis=axis))]
/Users/myname/.local/share/virtualenvs/ragas-try-XwHlDXQv/lib/python3.11/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by mode='default'.
  table = cls._concat_blocks(blocks, axis=0)

 Done evaluating. OpenAI cost:
Tokens Used: 4955
        Prompt Tokens: 3989
        Completion Tokens: 966
Successful Requests: 6
Total Cost (USD): $0.0079155
```

Ragas version: 0.0.21
Python version: 3.11

**Code to Reproduce**
Share code to reproduce the issue


**Expected behavior**
Expect the cb (on_llm_end of OpenAICallbackHandler) is called correctly
Repository: explodinggradients/ragas
Issue: Erase encode_kwargs of the embedding
Root cause: Algorithmic Error

This post init erase the encode_kwargs of the embedding. 
In the embedding HuggingfaceEmbeddings post init, "convert_to_tensor" = True is added. (its mandatory, otherwise in the "embed_documents" method the check "assert isinstance(embeddings, Tensor)" failed

https://github.com/explodinggradients/ragas/blob/3d29c44825eff39775f458af7c2e5d51411ac277/src/ragas/metrics/_answer_similarity.py#L52-L58
Repository: explodinggradients/ragas
Issue: testgenerator
Root cause: Validation Gap

anaconda3/envs/rag/lib/python3.11/site-packages/ragas/testset/testset_generator.py", line 200, in _filter_question
    return json_results.get("verdict") != "No"
           ^^^^^^^^^^^^^^^^
AttributeError: 'int' object has no attribute 'get'

This error occurred during the execution of testgenerator. After inspecting the code, I found that if the open-source model I am using outputs in an incorrect format, such as a string containing only numbers, then the load_as_json method would directly return an integer, leading to this error. This doesn’t seem to be the correct behavior. Or is it because the model I am using is not powerful enough? What type of model should I choose for dataset generation (I am on a budget haha)? thanks dude

could you help?
`model = LlamaCpp(
        model_path="model/mistral-7b-instruct-v0.1.gguf",
        temperature=0.75,
        max_tokens=2000,
        n_ctx=4096,
        top_p=1,
        verbose=True  # Verbose is required to pass to the callback manager
    )

    # loading files in a directory
    # loader = DirectoryLoader('../doc/', glob="**/*.pdf", show_progress=True)
    loader = DirectoryLoader('doc/', show_progress=True)
    data = loader.load()
    print("Loading Documents nubmers: ", len(data))
    print("data:", data)

    llm = LangchainLLM(model)
    testset_distribution = {
        "simple": 0.25,
        "reasoning": 0.5,
        "multi_context": 0.0,
        "conditional": 0.25,
    }

    testsetgenerator = TestsetGenerator(generator_llm=llm, critic_llm=llm, embeddings_model=embeddings,testset_distribution=testset_distribution,  chunk_size=512)
    testset = testsetgenerator.generate(data, 100)
`
the above code is copied from your docs, and it always encounter the error like this:
`anaconda3/envs/rag/lib/python3.11/site-packages/llama_cpp/llama.py", line 422, in __del__
TypeError: 'NoneType' object is not callable
Exception ignored in: <function _LlamaModel.__del__ at 0x7f41c450aa20>
Traceback (most recent call last):`
This prevents me from continuing my work.

this is a current limitation of ragas and will be fixed in the next version but unfortunately we don't have a work around for this yet :(
Repository: explodinggradients/ragas
Issue: AuthenticationError with "Answer Correctness"
Root cause: Configuration Error

```
from ragas.metrics import AnswerCorrectness
answer_correctness = AnswerCorrectness(
    weights=[0.4,0.6]
)

results = answer_correctness.score(dataset["train"].select(range(3)))
```
`RuntimeError: ('Fatal error occurred while running async tasks.', AuthenticationError("Error code: 401 - {'error': {'message': 'Incorrect API key provided: `

code raise runtime error. i can use other metrics but Answercorectness.
i use azure openAI  api, gpt4, ada-002

similar to #325 

Hi @codinseok , please initialize Azure open ai embeddings as shown here https://docs.ragas.io/en/latest/howtos/customisations/azure-openai.html

Hi @shahules786 , I am using the rest of the metrics(context_precision,faithfulness,answer_relevancy,context_recall) normally. The same problem occurs only in two specific cases. (answer_similarity,answer_correctness)

Also, I tried running the other metrics first and then attempted (answer_similarity, answer_correctness) at the end.

@ZhexuanZhou Thanks!!
I solved problem add this code 

```
# patch the new RagasLLM instance
answer_similarity = AnswerSimilarity(llm=ragas_azure_model, embeddings=azure_embeddings)
```

Thank you @codinseok 

> 此外，我尝试先运行其他指标，然后在最后尝试（answer_similarity、answer_correctness）。



> 嗨，我正常使用其余的指标（context_precision，忠诚度，answer_relevancy，context_recall）。仅在两种特定情况下会出现相同的问题。（answer_similarity，answer_correctness）

另外几种指标报这个错该如何处理
Repository: explodinggradients/ragas
Issue: LangChain RagasEvaluatorChain.evaluate always sends empty dataset to ContextRecall since 0.0.21 due to unsupported EvaluationMode.qcg
Root cause: Configuration Error

**Describe the bug**
LangChain `RagasEvaluatorChain.evaluate()` sends empty dataset to **ContextRecall** metric even though supplied dataset and predictions have properly defined attributes. Version **0.0.21** changed Evaluation Mode of ContextRecall to **qcg** which is not currently supported in `evaluate()` dataset mapping logic.

https://github.com/explodinggradients/ragas/blob/41e9e54b204566425908c1a437d7c1c798578ad3/src/ragas/metrics/_context_recall.py#L84

https://github.com/explodinggradients/ragas/blob/41e9e54b204566425908c1a437d7c1c798578ad3/src/ragas/langchain/evalchain.py#L133

Ragas version:
Python version: 3.11.7

**Code to Reproduce**
```
from ragas.langchain.evalchain import RagasEvaluatorChain
from ragas.metrics import context_recall

metric_chain = RagasEvaluatorChain(metric=context_recall)
results = metric_chain.evaluate(dataset, predictions)
```

**Error trace**
`0it [00:00, ?it/s]`


**Expected behavior**
Should produce ContextRecall metric for each item in provided dataset and predictions.

**Additional context**
Worked fine prior to updating to version 0.0.21
Repository: explodinggradients/ragas
Issue: cannot import name 'evaluate' from 'ragas' (unknown location)
Root cause: Environment Incompatibility

```
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
)

result = evaluate(
    dataset = dataset, 
    metrics=[
        context_precision,
        context_recall,
        faithfulness,
        answer_relevancy,
    ],
)

df = result.to_pandas()
```
```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[<ipython-input-45-48eefc79449c>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from ragas import evaluate
      2 from ragas.metrics import (
      3     faithfulness,
      4     answer_relevancy,
      5     context_recall,

ImportError: cannot import name 'evaluate' from 'ragas' (unknown location)

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
-----------------------------------------------------------------
```

please do `pip install ragas` @andysingal 

> please do `pip install ragas` @andysingal

Already tried but still same error @shahules786 
<img width="1479" alt="Screenshot 2023-12-14 at 11 30 17 AM" src="https://github.com/explodinggradients/ragas/assets/20493493/64b40be2-4d25-4fcd-9fd0-8652f80102ed">


@andysingal  did you fix it ?


RESTART THE KERNEL AND IT'LL BE GONE!
Repository: explodinggradients/ragas
Issue: 'OPENAI_API_KEY' environment variable
Root cause: Configuration Error

```
from ragas import evaluate
import os
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
)
os.environ["OPENAI_API_KEY"] = ""

result = evaluate(
    dataset = dataset, 
    metrics=[
        context_precision,
        context_recall,
        faithfulness,
        answer_relevancy,
    ],
)

df = result.to_pandas()
```
ERROR even after sharing openai key
```
---------------------------------------------------------------------------
OpenAIKeyNotFound                         Traceback (most recent call last)
[<ipython-input-23-f09f3227a916>](https://localhost:8080/#) in <cell line: 11>()
      9 os.environ["OPENAI_API_KEY"] = ""
     10 
---> 11 result = evaluate(
     12     dataset = dataset,
     13     metrics=[

2 frames
[/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py](https://localhost:8080/#) in init_model(self)
     98         if isinstance(self.embeddings, OpenAIEmbeddings):
     99             if self.embeddings.openai_api_key == "no-key":
--> 100                 raise OpenAIKeyNotFound
    101 
    102     def _score_batch(

OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable
```

Please read the docs before raising issues @andysingal  https://docs.ragas.io/en/latest/getstarted/index.html#

oh yes, i read your code but sadly not working as shared in the snapshot

On Thu, Dec 14, 2023 at 1:05 PM Shahul ES ***@***.***> wrote:

> Please read the docs before raising issues @andysingal
> <https://github.com/andysingal>
> https://docs.ragas.io/en/latest/getstarted/index.html#
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/explodinggradients/ragas/issues/383#issuecomment-1855324793>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE4LJNIISYUURAYVRGROX7LYJKT4FAVCNFSM6AAAAABAUJ23WOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJVGMZDINZZGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


really sorry to see this @andysingal are you using the latest version - because this was fixed a while back from how you're using it and a test case added


+1 having the same issue, even after setting os.environ["OPENAI_API_KEY"] = "my-key"
Repository: explodinggradients/ragas
Issue: Get score for each context chunk
Root cause: Unimplemented Feature Gap

Is it possible to retrieve individual scores for each context when using Context Precision and Context Relevancy, rather than a cumulative score? For instance, if I retrieve three distinct chunks of data, I would like to obtain score for each chunk.

It's not supported as of now, could you tell me your use case for this? @leonseet 

To do more granular analysis on each context retrieved.

hey @leonseet - did you find a work around for this?
Repository: explodinggradients/ragas
Issue: The FILTER_QUESTION prompt used for testset_generator.py need to be corrected / improved
Root cause: Documentation Deficiency

**Issue Description**
The FILTER_QUESTION prompt used for test set generation to evaluate the RAG pipeline needs to be corrected / improved. The link to the prompts.py file is as follows:
https://github.com/explodinggradients/ragas/blob/main/src/ragas/testset/prompts.py

**Original FILTER_QUESTION prompt**
"""
Determine if the given question can be clearly understood even when presented without any additional context. Specify reason and verdict is a valid json format.
question: What is the keyword that best describes the paper's focus in natural language understanding tasks?
{{"reason":"The specific paper being referred to is not mentioned in the question.", "verdict": "No"}}
question:{question}
"""

**Suggested Improved FILTER_QUESTION prompt**
"""
The task is to assess whether the given question can be clearly understood without any additional context. Analyze the question provided at the end and provide feedback in a valid JSON format as follows:
{{
  "reason": "Specify the reason if the question can be understood without any additional context.",
  "verdict": "A single word 'Yes' or 'No' indicating whether the question can be understood or not."
}}
question: {question}
"""



Hi @mosama1994 , thanks for your feedback. Test set generation is in active refactoring, I have changed and improved almost all of the prompts in the module. In fact, my latest prompts are very similar to your suggestion. 

Please provide your valuable feedbacks in [RFC](https://github.com/explodinggradients/ragas/issues/380)

Thank you. I was also looking for easier implementation of incorporating open source llm's and embedding models in the framework. That would be great. Currently instead of inheriting the ragas base classes and implementing my own theory, I used hugging face models through langchain and made changes to the Langchain model code. As hugging face is the most widely used open source platform for downloading different models (llms, encoders, rerankers, etc.), it would be great if we had easier implemetation for that.

Hey @mosama1994 - great to meet you 🙂 
could you tell me a bit more about what changes you had to make and why you had to make those changes?

i do understand good HF support is crucial which is why I was curious
Repository: explodinggradients/ragas
Issue: VLLM RAGAS API key error for HuggingFace Models
Root cause: Documentation Deficiency

Does anyone have idea about using RAGAS EVALUATION using VLLM server for HuggingFace models.I was successfully serving my model,but ragas metrics evaluation does recognise vlllm serving and it always asks for open ai api key

cell1:
!python -u -m vllm.entrypoints.openai.api_server \
       --host 0.0.0.0 \
       --model TheBloke/Mistral-7B-Instruct-v0.2-AWQ \
       --dtype half \
       --max-num-batched-tokens 4096 \
       --max-model-len 256 \
       --quantization awq \
       --tensor-parallel-size 1 \
       --port 8010 | grep -q "Uvicorn running" &npx localtunnel --port 8010

cell2:
from langchain.chat_models import ChatOpenAI
from ragas.llms import LangchainLLM

inference_server_url = "http://localhost:8010/v1"

# create vLLM Langchain instance
chat = ChatOpenAI(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    openai_api_key="no-key",
    openai_api_base=inference_server_url,
    max_tokens=256,
    temperature=0.2,
)

# use the Ragas LangchainLLM wrapper to create a RagasLLM instance
vllm = LangchainLLM(llm=chat)


cell3:
evaluate
from ragas import evaluate

result = evaluate(
fiqa_eval["baseline"].select(range(5)), # showing only 5 for demonstration
metrics=[faithfulness],
)

result

OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable




Running this on T4 cuda 12.2 colab free tier

      

Hi @Jaykumaran , apologies for the late response. Were you able to solve this issue? 
@jjmachan 

Hi,thanks I have used RAGAS with VLLM successfully,
but  calculating metric evaluation its very slow.Like when doing faithfulness score calculation.It takes more time,.
I'm hosting my VLLM endpoint in one T4 colab notebook and imferencing ragas in another notebook T4 .Inferencing is fast but calculating scores takes more time
Repository: explodinggradients/ragas
Issue: Issue with faithfulness
Root cause: Validation Gap

I am occasionally encountering an error when evaluating with faithfulness. The error is at line 178, when computing the faithful statements:

    176                 output = output if isinstance(output, list) else []
    177                 faithful_statements = sum(
--> 178                     verdict_score_map.get(dict.get("verdict", "").lower(), np.nan)
    179                     for dict in output
    180                 )

AttributeError: 'int' object has no attribute 'lower'

Hi @Robs1999 , this is interesting. Would be able to provide the sample for which this issue is being encountered?