Repository: explodinggradients/ragas
Issue: evaluate forcefully stopped
Root cause: Resource Mishandling

    self.gpu_allocator.free(block)
  File "/tmp/.local/lib/python3.10/site-packages/vllm/core/block_manager.py", line 48, in free
    raise ValueError(f"Double free! {block} is already freed.")
ValueError: Double free! PhysicalTokenBlock(device=Device.GPU, block_number=26224, ref_count=0) is already freed.
Exception in callback functools.partial(<function _raise_exception_on_finish at 0x7fe8f56639a0>, request_tracker=<vllm.engine.async_llm_engine.RequestTracker object at 0x7fe8f0f37190>)
handle: <Handle functools.partial(<function _raise_exception_on_finish at 0x7fe8f56639a0>, request_tracker=<vllm.engine.async_llm_engine.RequestTracker object at 0x7fe8f0f37190>)>
Traceback (most recent call last):
  File "/tmp/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 28, in _raise_exception_on_finish
    task.result()
  File "/tmp/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 363, in run_engine_loop
    has_requests_in_progress = await self.engine_step()
  File "/tmp/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 342, in engine_step
    request_outputs = await self.engine.step_async()
  File "/tmp/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 204, in step_async
    return self._process_model_outputs(output, scheduler_outputs)
  File "/tmp/.local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 597, in _process_model_outputs
    self._process_sequence_group_outputs(seq_group, outputs)
  File "/tmp/.local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 487, in _process_sequence_group_outputs
    self.scheduler.free_seq(seq)
  File "/tmp/.local/lib/python3.10/site-packages/vllm/core/scheduler.py", line 312, in free_seq
    self.block_manager.free(seq)
  File "/tmp/.local/lib/python3.10/site-packages/vllm/core/block_manager.py", line 277, in free
    self._free_block_table(block_table)
  File "/tmp/.local/lib/python3.10/site-packages/vllm/core/block_manager.py", line 268, in _free_block_table
    self.gpu_allocator.free(block)
  File "/tmp/.local/lib/python3.10/site-packages/vllm/core/block_manager.py", line 48, in free
    raise ValueError(f"Double free! {block} is already freed.")
ValueError: Double free! PhysicalTokenBlock(device=Device.GPU, block_number=26051, ref_count=0) is already freed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/tmp/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 37, in _raise_exception_on_finish
    raise exc
  File "/tmp/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 32, in _raise_exception_on_finish
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.
INFO 01-12 11:32:29 async_llm_engine.py:134] Aborted request cmpl-6cf32d9e99e34bbfa12257fdfd689a68.