Repository: explodinggradients/ragas
Issue: RAGAS Evaluation with OpenAI API Extremely Slow — Any Tips?
Root cause: Resource Mishandling



**Your Question**
Hi all, I’m running RAGAS evaluation  (0.4.0) with the OpenAI API and it’s extremely slow.
Each item takes 30–60 seconds, and my dataset has ~2000 rows, so the full run would take 2–3 days.

I’m using metrics: ContextPrecision, AnswerRelevancy, Faithfulness, LLMContextRecall, and each sample triggers multiple chat.completions and embeddings requests.

I also tried async parallel batching, but after some batches the process gets stuck (possibly rate limits or too many open connections).

Questions:

Is RAGAS evaluation with OpenAI normally this slow?

Any best practices for large datasets? (async patterns, reducing metrics, local LLMs, batching, etc.)

Has anyone solved the “async stuck after a few batches” issue?

Thanks!

**Code Examples**
```
import asyncio
import numpy as np
from openai import AsyncOpenAI

from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory

from ragas.metrics.collections import (
    ContextPrecision,
    AnswerRelevancy,
    Faithfulness
)

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import NonLLMContextRecall
from ragas.metrics import LLMContextRecall

from langchain_openai import ChatOpenAI
import logging
import threading
import time


logger = logging.getLogger(__name__)


# =========================================================
# Heartbeat
# =========================================================
def start_heartbeat():
    def hb():
        while True:
            logger.info("[HEARTBEAT] RAGAS masih berjalan...")
            time.sleep(60)
    threading.Thread(target=hb, daemon=True).start()


def fmt(v, d=3):
    try:
        return round(float(v), d)
    except:
        return 0.0


# =========================================================
# Evaluasi satu item
# =========================================================
async def run_single_sample(ds, i, llm_ragas, embeddings, evaluator_llm):

    q  = ds["question"][i]
    r  = ds["answer"][i]
    gt = ds["ground_truth"][i]
    ctx = ds["contexts"][i]

    

    sample = SingleTurnSample(
        user_input=q,
        response=r,
        reference=gt,
        retrieved_contexts=ctx
    )

    try:
        CP = await ContextPrecision(llm=llm_ragas).ascore(
            user_input=q, reference=gt, retrieved_contexts=ctx
        )

        AR = await AnswerRelevancy(llm=llm_ragas, embeddings=embeddings).ascore(
            user_input=q, response=r
        )

        F = await Faithfulness(llm=llm_ragas).ascore(
            user_input=q, response=r, retrieved_contexts=ctx
        )

        CR = await LLMContextRecall(llm=evaluator_llm).single_turn_ascore(sample)
        #CR=10
    except Exception as e:
        raise RuntimeError(f"Metric error: {e}")

    return {
        "context_precision": float(CP.value),
        "context_recall": float(CR),
        "answer_relevancy": float(AR.value),
        "faithfulness": float(F.value),
    }


# =========================================================
# Evaluasi SEQUENTIAL
# =========================================================
async def run_ragas_evaluation_async(questions, predictions, contexts_used, references, openai_key):

    start_heartbeat()

    logger.info("\n==============================")
    logger.info("📊  RAGAS Evaluation Started (Sequential Mode)")
    logger.info("==============================\n")

    client = AsyncOpenAI(api_key=openai_key)

    llm_ragas = llm_factory("gpt-4.1-mini", client=client)
    embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client)

    evaluator_llm = ChatOpenAI(
        model="gpt-4.1-mini",
        temperature=0,
        openai_api_key=openai_key
    )

    ds = {
        "question": questions,
        "answer": predictions,
        "ground_truth": references,
        "contexts": contexts_used
    }

    total = len(questions)
    results = []

    for i in range(total):

        q = ds["question"][i]
        r = ds["answer"][i]
        ctx = ds["contexts"][i]

        # fallback prevent error
        if not ctx or len(ctx) == 0:
            logger.warning(f"[RAGAS] Context kosong item {i+1}, menggunakan fallback")
            ctx = ["No context retrieved"]
            ds["contexts"][i] = ctx

        logger.info("----------------------------------------------")
        logger.info(f"[RAGAS] ▶ Evaluasi item {i+1}/{total}")
        logger.info("----------------------------------------------")
        logger.info(f"  • Pertanyaan     : {q[:80]}...")
        logger.info(f"  • Jawaban LLM    : {r[:80]}...")
        logger.info(f"  • Jumlah konteks : {len(ctx)}")

                # ===========================================================
        # Tambahkan log konteks (preview)
        # ===========================================================
        if isinstance(ctx, list):
            if len(ctx) == 1:
                preview = ctx[0][:150].replace("\n", " ")
                logger.info(f"  • Context Sample : {preview}...")
            else:
                p1 = ctx[0][:150].replace("\n", " ")
                logger.info(f"  • Context 1      : {p1}...")

                if len(ctx) > 1:
                    p2 = ctx[1][:150].replace("\n", " ")
                    logger.info(f"  • Context 2      : {p2}...")
        else:
            preview = str(ctx)[:150].replace("\n", " ")
            logger.info(f"  • Context Sample : {preview}...")

        # ==========================================================

        t0 = time.time()

        try:
            metric = await run_single_sample(ds, i, llm_ragas, embeddings, evaluator_llm)
        except Exception as e:
            logger.error(f"[RAGAS] ❌ Error item {i+1}: {e}")
            metric = {
                "context_precision": 0.0,
                "context_recall": 0.0,
                "answer_relevancy": 0.0,
                "faithfulness": 0.0,
            }

        dt = time.time() - t0

        logger.info(f"[RAGAS] ✔ Selesai item {i+1} dalam {dt:.2f} s")
        logger.info(f"        Context Precision : {fmt(metric['context_precision'])}")
        logger.info(f"        Context Recall    : {fmt(metric['context_recall'])}")
        logger.info(f"        Answer Relevancy  : {fmt(metric['answer_relevancy'])}")
        logger.info(f"        Faithfulness      : {fmt(metric['faithfulness'])}")

        results.append(metric)

    # =====================================================
    # AGGREGATE
    # =====================================================
    ragas_agg = {
        k: fmt(np.mean([r[k] for r in results]))
        for k in results[0].keys()
    }

    logger.info("\n==============================")
    logger.info("📈  RAGAS Aggregate Scores")
    logger.info("==============================")
    for k, v in ragas_agg.items():
        logger.info(f"{k:20s}: {v}")

    return results, ragas_agg


# =========================================================
# Wrapper sync
# =========================================================
def run_ragas_evaluation(*args, **kwargs):
    return asyncio.run(run_ragas_evaluation_async(*args, **kwargs))
```

```
**SAMPLE RESULTS**
2025-12-04 23:29:59 [INFO] [RAGAS] ▶ Evaluasi item 1643/1819
2025-12-04 23:29:59 [INFO] ----------------------------------------------
2025-12-04 23:29:59 [INFO]   • Pertanyaan     : Bagaimana Anda dapat merancang sistem navigasi yang menggabungkan prinsip-prinsip dari user control and freedom dan aest...
2025-12-04 23:29:59 [INFO]   • Jawaban LLM    : Untuk merancang sistem navigasi yang menggabungkan prinsip-prinsip user control and freedom dan aesthetic and minimalist...
2025-12-04 23:29:59 [INFO]   • Jumlah konteks : 2
2025-12-04 23:30:01 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:04 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:06 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:08 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:10 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:11 [INFO] Retrying request to /embeddings in 0.483860 seconds
2025-12-04 23:30:12 [INFO] HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2025-12-04 23:30:13 [INFO] Retrying request to /embeddings in 0.416737 seconds
2025-12-04 23:30:14 [INFO] HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2025-12-04 23:30:16 [INFO] Retrying request to /chat/completions in 0.386520 seconds
2025-12-04 23:30:19 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:24 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:29 [INFO] [HEARTBEAT] RAGAS masih berjalan...
2025-12-04 23:30:30 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:31 [INFO] [RAGAS] ✔ Selesai item 1643 dalam 31.29 s
2025-12-04 23:30:31 [INFO]         Context Precision : 1.000
2025-12-04 23:30:31 [INFO]         Context Recall    : 1.000
2025-12-04 23:30:31 [INFO]         Answer Relevancy  : 0.767
2025-12-04 23:30:31 [INFO]         Faithfulness      : 0.857
2025-12-04 23:30:31 [INFO] 
----------------------------------------------
2025-12-04 23:30:31 [INFO] [RAGAS] ▶ Evaluasi item 1644/1819
2025-12-04 23:30:31 [INFO] ----------------------------------------------
2025-12-04 23:30:31 [INFO]   • Pertanyaan     : Bagaimana Anda dapat merancang sistem yang sesuai dengan mental model pengguna untuk meningkatkan ease of learning pada ...
2025-12-04 23:30:31 [INFO]   • Jawaban LLM    : Whitney Quesenberry menyatakan bahwa mental model pengguna harus dipahami untuk merancang sistem yang sesuai dengannya. ...
2025-12-04 23:30:31 [INFO]   • Jumlah konteks : 2
2025-12-04 23:30:34 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:36 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:38 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:40 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:42 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:43 [INFO] Retrying request to /embeddings in 0.480786 seconds
2025-12-04 23:30:44 [INFO] HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2025-12-04 23:30:46 [INFO] Retrying request to /embeddings in 0.457862 seconds
2025-12-04 23:30:48 [INFO] HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2025-12-04 23:30:49 [INFO] Retrying request to /chat/completions in 0.494129 seconds
2025-12-04 23:30:51 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:30:57 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:31:02 [INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2025-12-04 23:31:03 [INFO] [RAGAS] ✔ Selesai item 1644 dalam 32.03 s
2025-12-04 23:31:03 [INFO]         Context Precision : 1.000
2025-12-04 23:31:03 [INFO]         Context Recall    : 1.000
2025-12-04 23:31:03 [INFO]         Answer Relevancy  : 0.715
2025-12-04 23:31:03 [INFO]         Faithfulness      : 0.750
2025-12-04 23:31:03 [INFO] 
----------------------------------------------
```


Thanks for raising this @rey1024 

You can use `asyncio.gather()` to run all 4 metrics in parallel. Set `strictness=1` for AnswerRelevancy.
Also, use `ContextRecall` from collections. `LLMContextRecall` is deprecated.


Hi @anistark, Thankyou.
I have implemented your suggestions regarding:

using asyncio.gather() to run all metrics in parallel
setting strictness=1 for AnswerRelevancy
switching to ContextRecall from ragas.metrics.collections
However, after integrating the parallel execution, the evaluation gets stuck during execution.
Below is the log showing where the process hangs:
```
2025-12-09 09:28:32 [INFO] [RUN 1] AC DONE: 0.49746865915891886
2025-12-09 09:28:32 [INFO] [RUN 1] Running CP...
2025-12-09 09:28:34 [INFO] [RUN 0] AC DONE: 0.588152271663352
2025-12-09 09:28:34 [INFO] [RUN 0] Running CP...
2025-12-09 09:28:36 [INFO] [RUN 2] CP DONE: 0.0
2025-12-09 09:28:36 [INFO] [RUN 2] Running AR...
2025-12-09 09:28:45 [INFO] [RUN 2] AR DONE: 0.0
2025-12-09 09:28:45 [INFO] [RUN 2] Running F...
2025-12-09 09:28:45 [INFO] [RUN 3] CP DONE: 0.99999999995
2025-12-09 09:28:45 [INFO] [RUN 3] Running AR...
2025-12-09 09:28:46 [INFO] [RUN 0] CP DONE: 0.99999999995
2025-12-09 09:28:46 [INFO] [RUN 0] Running AR...
2025-12-09 09:28:47 [INFO] [RUN 1] CP DONE: 0.99999999995
2025-12-09 09:28:47 [INFO] [RUN 1] Running AR...
2025-12-09 09:28:54 [INFO] [RUN 0] AR DONE: 0.640611520005186
2025-12-09 09:28:54 [INFO] [RUN 0] Running F...
2025-12-09 09:29:00 [INFO] [RUN 3] AR DONE: 0.567775384572688
2025-12-09 09:29:00 [INFO] [RUN 3] Running F...
```

It looks like the Faithfulness metric never returns a result under concurrent execution.
Every other metric (AnswerCorrectness, ContextPrecision, AnswerRelevancy, ContextRecall) completes normally, but Faithfulness either:

- freezes indefinitely, or
- times out if I wrap it in asyncio.wait_for()

This makes asyncio.gather() unusable in practice because a single hanging metric blocks all scheduled tasks.

This is my final code
```
import asyncio
import numpy as np
from openai import AsyncOpenAI

from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory

from ragas.metrics.collections import (
    ContextPrecision,
    AnswerRelevancy,
    Faithfulness,
    AnswerCorrectness,
    ContextRecall
)

from ragas.dataset_schema import SingleTurnSample
import logging
import time


logging.getLogger("httpx").setLevel(logging.WARNING)
logging.getLogger("openai").setLevel(logging.WARNING)

logger = logging.getLogger(__name__)


def fmt(v, d=3):
    try:
        return round(float(v), d)
    except:
        return 0.0


# =========================================================
# RUN SINGLE SAMPLE (PARALLEL METRIC CALLS)
# =========================================================
async def run_single_sample(i, ds, metrics):
    q  = ds["question"][i]
    r  = ds["answer"][i]
    gt = ds["ground_truth"][i]
    ctx = ds["contexts"][i]

    logger.info(f"[RUN {i}] Start evaluating sample")

    # Prepare empty scores
    AC = CP = AR = F = CR = None

    # -----------------------------------------------------
    # 1. AnswerCorrectness
    # -----------------------------------------------------
    try:
        logger.info(f"[RUN {i}] Running AC...")
        AC = await metrics["AC"].ascore(user_input=q, response=r, reference=gt)
        logger.info(f"[RUN {i}] AC DONE: {AC.value}")
    except Exception as e:
        logger.error(f"[RUN {i}] ERROR in AC: {e}")

    # -----------------------------------------------------
    # 2. ContextPrecision
    # -----------------------------------------------------
    try:
        logger.info(f"[RUN {i}] Running CP...")
        CP = await metrics["CP"].ascore(user_input=q, reference=gt, retrieved_contexts=ctx)
        logger.info(f"[RUN {i}] CP DONE: {CP.value}")
    except Exception as e:
        logger.error(f"[RUN {i}] ERROR in CP: {e}")

    # -----------------------------------------------------
    # 3. AnswerRelevancy
    # -----------------------------------------------------
    try:
        logger.info(f"[RUN {i}] Running AR...")
        AR = await metrics["AR"].ascore(user_input=q, response=r)
        logger.info(f"[RUN {i}] AR DONE: {AR.value}")
    except Exception as e:
        logger.error(f"[RUN {i}] ERROR in AR: {e}")

    # -----------------------------------------------------
    # 4. Faithfulness
    # -----------------------------------------------------
    SEM_F = asyncio.Semaphore(1)

    try:
        logger.info(f"[RUN {i}] Running F...")
        async with SEM_F:
            F = await asyncio.wait_for(
                metrics["F"].ascore(user_input=q, response=r, retrieved_contexts=ctx),
                timeout=20
            )
        logger.info(f"[RUN {i}] F DONE: {F.value}")

    except asyncio.TimeoutError:
        logger.error(f"[RUN {i}] Faithfulness TIMEOUT")
        F = None
    except Exception as e:
        logger.error(f"[RUN {i}] ERROR in F: {e}")
        F = None



    # -----------------------------------------------------
    # 5. ContextRecall
    # -----------------------------------------------------
    try:
        logger.info(f"[RUN {i}] Running CR...")
        CR = await metrics["CR"].ascore(user_input=q, reference=gt, retrieved_contexts=ctx)
        logger.info(f"[RUN {i}] CR DONE: {CR}")
    except Exception as e:
        logger.error(f"[RUN {i}] ERROR in CR: {e}")

    # -----------------------------------------------------
    # Return with fallback -1 for failed metrics
    # -----------------------------------------------------
    return {
        "context_precision": float(CP.value) if CP else -1,
        "context_recall": float(CR) if CR else -1,
        "answer_relevancy": float(AR.value) if AR else -1,
        "faithfulness": float(F.value) if F else -1,
        "answer_correctness": float(AC.value) if AC else -1,
    }

# =========================================================
# MAIN DRIVER
# =========================================================
async def run_ragas_evaluation_async(questions, predictions, contexts_used, references, openai_key):

    logger.info("\n==============================")
    logger.info("📊  RAGAS Evaluation Started (Optimized Parallel Mode)")
    logger.info("==============================\n")

    client = AsyncOpenAI(api_key=openai_key)

    # Semua LLM untuk collections harus pakai InstructorLLM
    llm_ragas = llm_factory("gpt-4o-mini", client=client)
    evaluator_llm = llm_factory("gpt-4o-mini", client=client)
    llm_faith = llm_factory("gpt-4o", client=client)

    embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client)

    # =========================================================
    # METRIC INSTANCES dibuat sekali
    # =========================================================
    metrics = {
        "AC": AnswerCorrectness(llm=llm_ragas, embeddings=embeddings),
        "CP": ContextPrecision(llm=llm_ragas),
        "AR": AnswerRelevancy(llm=llm_ragas, embeddings=embeddings, strictness=1),
        "F":  Faithfulness(llm=llm_faith),
        "CR": ContextRecall(llm=evaluator_llm)
    }

    ds = {
        "question": questions,
        "answer": predictions,
        "ground_truth": references,
        "contexts": contexts_used
    }

    total = len(questions)

    # preview logs
    for i in range(total):
        q = ds["question"][i]
        r = ds["answer"][i]
        ctx = ds["contexts"][i]

        if not ctx:
            ctx = ["No context retrieved"]
            ds["contexts"][i] = ctx

        logger.info("----------------------------------------------")
        logger.info(f"[RAGAS] ▶ Item {i+1}/{total}")
        logger.info(f"  • Pertanyaan : {q[:80]}...")
        logger.info(f"  • Jawaban    : {r[:80]}...")
        logger.info(f"  • Jumlah ctx : {len(ctx)}")


    # =========================================================
    # CONCURRENT EVALUATION
    # =========================================================
    SEM = asyncio.Semaphore(4)
    logger.info("STARTING CONCURRENT EVALUATION")

    async def safe_eval(i):
        logger.info(f"[TASK {i}] Waiting for semaphore...")
        async with SEM:
            logger.info(f"[TASK {i}] Acquired semaphore, starting evaluation")
            t0 = time.time()

            try:
                logger.info(f"[TASK {i}] Calling run_single_sample...")
                result = await run_single_sample(i, ds, metrics)
                logger.info(f"[TASK {i}] run_single_sample finished")

                # ============================================
                # SHOW RAGAS SCORES IMMEDIATELY
                # ============================================
                logger.info(
                    f"[RAGAS][ITEM {i}] "
                    f"CP={result.get('context_precision')}, "
                    f"CR={result.get('context_recall')}, "
                    f"AR={result.get('answer_relevancy')}, "
                    f"FTH={result.get('faithfulness')}, "
                    f"AC={result.get('answer_correctness')}"
                )

            except Exception as e:
                logger.error(f"[RAGAS] ERROR on item {i}: {e}")
                result = {
                    "context_precision": -1,
                    "context_recall": -1,
                    "answer_relevancy": -1,
                    "faithfulness": -1,
                    "answer_correctness": -1
                }

            dt = time.time() - t0
            logger.info(f"[RAGAS] TASK {i} completed in {dt:.2f} seconds")
            return result


    logger.info("[MAIN] Creating tasks...")
    tasks = [asyncio.create_task(safe_eval(i)) for i in range(total)]
    logger.info("[MAIN] Running asyncio.gather...")

    results = await asyncio.gather(*tasks)

    logger.info("[MAIN] FINISHED EVALUATION")


    # =========================================================
    # AGGREGATE
    # =========================================================
    ragas_agg = {k: fmt(np.mean([r[k] for r in results])) for k in results[0].keys()}

    logger.info("\n==============================")
    logger.info("📈  RAGAS Aggregate Scores")
    logger.info("==============================")
    for k, v in ragas_agg.items():
        logger.info(f"{k:20s}: {v}")

    return results, ragas_agg


def run_ragas_evaluation(*args, **kwargs):
    return asyncio.run(run_ragas_evaluation_async(*args, **kwargs))
```
**My Questions:** 
1. Is Faithfulness currently compatible with concurrent execution under asyncio.gather()?
Given the behavior, it appears that the metric internally blocks until completion and does not support parallel async flows.
2. Is there a recommended best practice code snippet for running RAGAS (latest version) with asyncio.gather()?
Specifically for the new collections metrics, including Faithfulness and ContextRecall


> 1. Is Faithfulness currently compatible with concurrent execution under asyncio.gather()?

Yes, should be compatible. There's isn't any metric specfic config as such.

>    Given the behavior, it appears that the metric internally blocks until completion and does not support parallel async flows.

I suspect it might be rate limiting issue. But will need to test and verify.

> 2. Is there a recommended best practice code snippet for running RAGAS (latest version) with asyncio.gather()?
>    Specifically for the new collections metrics, including Faithfulness and ContextRecall

Not yet. Will take a note of it.