Repository: promptfoo/promptfoo
Issue: How to Evaluate RAG Applications 
Root cause: Configuration Error

Hello everyone,

I've developed two RAG (Naive RAG and Advance RAG) applications using the following technologies:
**LLM Framework**: llama-index
**LLM Model**: Llama3.1 (8b) using Ollama 
**Embedding Model**: HuggingFace ("BAAI/bge-large-en-v1.5")

I need to evaluate both RAG applications to determine their performance in terms of Accuracy, Response time etc.

I’m considering using **PromptFoo** for evaluating my RAG applications.
How can I integrate PromptFoo into my RAG applications?

Thank you in advance for your assistance!

Best Regards,
Ankur


Have you read the [Evaluating RAG pipelines](https://www.promptfoo.dev/docs/guides/evaluate-rag/) guide?  What have you tried so far?

Hi @thetechinsight,

I'm going to close this issue for now. Please take a look at our Evaluating RAG pipelines guide as well as our [Red Teaming documentation](https://www.promptfoo.dev/docs/category/red-teaming/), which might also be helpful depending on your needs.

If you run into any issues or have further questions, feel free to re-open the issue or reach out directly—we’re happy to help!

@mldangelo @typpo Thank you for prompt reply.

I’m encountering an issue while running promptfoo, and although I believe it might not be directly related to promptfoo, I'm hoping you can help me troubleshoot.

Here is the retriever.py file I am using in the promptfooconfig.yaml:

```
from llama_index.core import StorageContext, load_index_from_storage

def call_api(prompt, options, context):
    storage_context = StorageContext.from_defaults(persist_dir="./naive-rag-db")
    simple_vc_index = load_index_from_storage(storage_context, index_id="naive-rag")
    new_query_engine = simple_vc_index.as_query_engine()
    response = new_query_engine.query(prompt)
    result = {
        "output": response.content,
    }
    return result

```

In this script, I create a vector database and store it locally:

```
               Settings.llm=Ollama(model=selected_model, request_timeout=300.0)
                Settings.embed_model = OllamaEmbedding(
                    model_name="gemma2:2b",
                    base_url="http://localhost:11434",
                    ollama_additional_kwargs={"mirostat": 0},
                )
                Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=20)
                Settings.num_output = 512
                Settings.context_window = 3900

                index = VectorStoreIndex.from_documents(docs, embed_model=Settings.embed_model ,llm=Settings.llm, node_parser=Settings.node_parser, num_output=Settings.num_output, context_window=Settings.context_window,show_progress=True)
                index.set_index_id("naive-rag")
                index.storage_context.persist("./naive-rag-db")
```


However, I’m encountering the following error when running the evaluation and I’m not using OpenAI services directly in my code; however, it seems that llama-index might be using OpenAI internally.

```
 > npx promptfoo@latest eval
[░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0% | ETA: 0s | 0/1 |  ""
[░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0% | ETA: 0s | 0/1 |  ""
Error running Python script: openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-Y8GwA*********}
Stack Trace: Error: openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-Y8GwA******************}
    at PythonShell.parseError (C:\Users\xyz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\python-shell\index.js:303:21)
    at terminateIfNeeded (C:\Users\xyz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\python-shell\index.js:193:32)
    at ChildProcess.<anonymous> (C:\Users\xyz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\python-shell\index.js:185:13)
    at ChildProcess.emit (node:events:519:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
    --Python Traceback: --
    File "C:\Users\XYZ\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\python\wrapper.py", line 35, in <modu>
      result = call_method(script_path, method_name, *data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\XYZ\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\python\wrapper.py", line 24, in call_d
      return method_to_call(*args)
             ^^^^^^^^^^^^^^^^^^^^^
    File "retrieve-naive-rag.py", line 18, in call_api
      response = new_query_engine.query(prompt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

```

Could you please help me diagnose why this error is occurring and how I might resolve it?

Thank you!