Repository: explodinggradients/ragas
Issue: [R-226] Dead lock and evaluation stuck in the end
Root cause: Resource Mishandling

\[x\] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

Ragas stucks in the final few evaluation cases in azure CI and my local macOS for more than 3 hours. And if you try to terminate it, the stucked in threading waiting for lock.

![image](https://uploads.linear.app/0a832749-6bfc-40ec-ae6f-821fa8081d92/b5bd8e8e-efb3-4e3a-8ffb-8a45e5b3f063/f0ca2a03-4830-4186-9793-bf15cd0ffbff?signature=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwYXRoIjoiLzBhODMyNzQ5LTZiZmMtNDBlYy1hZTZmLTgyMWZhODA4MWQ5Mi9iNWJkOGU4ZS1lZmIzLTRlM2EtOGZmYi04YTQ1ZTViM2YwNjMvZjBjYTJhMDMtNDgzMC00MTg2LTk3OTMtYmYxNWNkMGZmYmZmIiwiaWF0IjoxNzE0MDg4Mzc4LCJleHAiOjE3MTQxNzQ3Nzh9.rUbkFdaPfOewies1cmijcRMFLkpw13IqoaPgd0KS5Z8)

Ragas version: 0.1.7
Python version: 3.11.8

**Code to Reproduce**

```python
# azure_model is GPT3.5 with 300k TPM
# azure_embedding is ada002 with 200k TPM
# dataset is private evaluation dataset

metrics = [
    answer_relevancy,
    answer_correctness
]
run_config = RunConfig(
    timeout=10,
    max_retries=3,
    max_wait=10,
    max_workers=10,
)
result = evaluate(
    dataset=dataset,
    metrics=metrics,
    llm=azure_model,
    embeddings=azure_embedding,
    raise_exceptions=False,
    run_config=run_config,
)
```

**Error trace**

```log
File "/Users/nil/miniconda3/envs/qi_assistant/lib/python3.11/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nil/miniconda3/envs/qi_assistant/lib/python3.11/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/Users/nil/.vscode/extensions/ms-python.debugpy-2024.4.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/Users/nil/.vscode/extensions/ms-python.debugpy-2024.4.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/Users/nil/.vscode/extensions/ms-python.debugpy-2024.4.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/Users/nil/.vscode/extensions/ms-python.debugpy-2024.4.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nil/.vscode/extensions/ms-python.debugpy-2024.4.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/nil/.vscode/extensions/ms-python.debugpy-2024.4.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "ragEval/evaluator.py", line 200, in <module>
    main()
  File "ragEval/evaluator.py", line 181, in main
    details, result = calculate_metrics(file, metrics, azure_model, azure_embedding)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ragEval/evaluator.py", line 123, in calculate_metrics
    result = evaluate(
             ^^^^^^^^^
  File "ragEval/ragas/evaluation.py", line 211, in evaluate
    results = executor.results()
              ^^^^^^^^^^^^^^^^^^
  File "ragEval/ragas/executor.py", line 185, in results
    )
^^^^^^
  File "/Users/nil/miniconda3/envs/qi_assistant/lib/python3.11/threading.py", line 1119, in join
    self._wait_for_tstate_lock()
  File "/Users/nil/miniconda3/envs/qi_assistant/lib/python3.11/threading.py", line 1139, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
```

**Expected behavior**
expect to finish evaluating

**Additional context**
Add any other context about the problem here.

<sub>From [SyncLinear.com](https://synclinear.com) | [R-226](https://linear.app/exploding-gradients/issue/R-226/dead-lock-and-evaluation-stuck-in-the-end)</sub>

Related Issues, can you help with that @jjmachan 
#722 #734 #810


hey @NIL-zhuang taking this up - thank you for the detailed writeup and the PR 🙂 ❤️ 

will keep you updated here

Could you please let us know in which version of ragas is this fix available?

Still I have almost similar issue. Friday my script ran perfect and Now I am trying evaluation with 5 different matrics.  Evaluation script taking huge time in 5th step and starting 4 steps ran very fast. Please let me know which version of RAGAS package is working fine.

**Changes which resolved my issue**: Direct pip install ragas -q which has 0.1.9 version, which dont have thresd_timeout fix. so used directly 
**!pip install git+https://github.com/explodinggradients/ragas**  Which resolved my issue, Now its not failing, but giving 0 value for few matrics

its still stuck showing 100% 

hey @Remya-Ramachandran you will have to install it like @prabham17 suggested to use the latest. will make a release shortly

@prabham17 which LLM are you using? I would love to get on a call to help you debug this too if you want.