Repository: explodinggradients/ragas
Issue: Progress is stuck (or maybe too long) for generate_with_langchain_docs !!!
Root cause: Algorithmic Error

# 🚨 Bug Report: progress is stuck (or maybe too long) for `generate_with_langchain_docs`

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

---

### 📄 Describe the bug

The execution of `generate_with_langchain_docs()` is stuck at 50% with no progress for over an hour. No error is thrown, and CPU usage is minimal. It seems like the generation process either hangs or is too long for practical usage.

---
### 🐍 Python version 
Python 3.132.2

### 💻 Ragas version
Ragas 0.3.0

---

### 📌 Code to Reproduce

```python
from ragas.testset import TestsetGenerator
from pandas import DataFrame, concat
from PDF_load.pdf_loader import get_content
from LLM.settings import generator_llm, generator_embeddings
import pathlib
import asyncio

# Load PDF content
docs = get_content()

# Create generator
generator = TestsetGenerator(
    llm=generator_llm, embedding_model=generator_embeddings
)

output_dir = pathlib.Path("Output")
output_dir.mkdir(exist_ok=True)

async def run():
    docs = get_content()
    for item in docs:
        file_name = item.get("file_name")
        doc = item.get("doc")

        dataset = generator.generate_with_langchain_docs(doc, testset_size=10)
        df = dataset.to_pandas()

        output_path = output_dir / f"{file_name.split('.')[0]}.csv"
        df.to_csv(output_path, index=False)

asyncio.run(run())
```
---
Generating Scenarios:  50%|████████████████                | 1/2 [27:14<27:14, 1634.71s/it]


Same code but different runtime:
Generating Scenarios:  50%|██▌  | 1/2 [08:14<08:14, 494.01s/it]
I think have some problem with google api (maybe).


But now:
Generating Scenarios:  50%|██  | 1/2 [21:19<21:19, 1279.25s/it]
Can someone tell me what happend?  🥲 🥲 🥲 🥲 🥲

Facing the same issue, don't know how long generating scenarios is going to take for 200 docs
Generating Scenarios:   0%|                                                  | 0/3 [00:00<?, ?it/s]

Also is there a intermediate step to resolve this, i am already passing a generator llm, embedding model, as well using default transforms but this still keeps happening . How do i prevent it from using CustomNode Filter
Node 7fa7a57a-f7c9-47d5-9e16-4679c8750eb2 does not have a summary. Skipping filtering.



`MultiHopAbstractQuerySynthesizer` this makes the problem

```py
from ragas.testset.synthesizers import MultiHopAbstractQuerySynthesizer
```

It sends to llm with exactly same prompt infinitely.
the one is `concept_combination_prompt`
and the other is `themes_personas_matching_prompt`

the default_query_distributor function is here (which is used by generator function as `query_distributor` parameter)

```py
def default_query_distribution(
    llm: BaseRagasLLM, kg: t.Optional[KnowledgeGraph] = None
) -> QueryDistribution:
    """ """
    default_queries = [
        SingleHopSpecificQuerySynthesizer(llm=llm),
        MultiHopAbstractQuerySynthesizer(llm=llm),
        MultiHopSpecificQuerySynthesizer(llm=llm),
    ]
    if kg is not None:
        available_queries = []
        for query in default_queries:
            if query.get_node_clusters(kg):
                available_queries.append(query)
    else:
        available_queries = default_queries

    return [(query, 1 / len(available_queries)) for query in available_queries]
```

it makes [(single_query, 0.5), (multi_hop_abstrct_query, 0.5)]
and single_query takes not that much time (about 64s)

```py
for i, (scenario, _) in enumerate(query_distribution):
    # print(scenario)
    exec.submit(
        scenario.generate_scenarios,
        n=splits[i],
        knowledge_graph=self.knowledge_graph,
        persona_list=self.persona_list[:num_personas],
        callbacks=scenario_generation_grp,
    )

try:
    scenario_sample_list: t.List[t.List[BaseScenario]] = exec.results()
except Exception as e:
    scenario_generation_rm.on_chain_error(e)
    raise e
else:
    scenario_generation_rm.on_chain_end(
        outputs={"scenario_sample_list": scenario_sample_list}
    )
```

we have to trace back of the `exec.results()` function i think
`generate_scenarios` makes the problem i think (which send the prompt to llm service)

I encountered a problem when trying to evaluate multi-hop questions using Ragas.

**Single-hop questions → work as expected.**

**Multi-hop questions → failing (so slow)  :<.**

This makes me think the issue might come from the generation or evaluation process for multi-hop rather than the base setup itself.

This is fixed on recent versions. Please update to latest version and try again.

Feel free to reopen if still happening.

tks for support, i will test.