Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Inference stucks when using sglang as backend.
Root cause: Resource Mishandling

I want to use SGLang as the backend to evaluate MLLM performance. The command I executed is `bash run.sh /data1/zzh/huggingface/hub/models--google--gemma-3-4b-it/snapshots/093f9f388b31de276ce2de164bdc2081324b9767 mme image 1`. The `run.sh` is as follows:

```bash
CKPT_PATH=$1
TASK=$2
MODALITY=$3
TP_SIZE=$4
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

python3 -m lmms_eval \
    --model srt_api \
    --model_args modality=$MODALITY,model_version=$CKPT_PATH,tp=$TP_SIZE,host=127.0.0.1,port=30000 \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/ \
    --verbosity DEBUG

```
After executing the above command, **the output gets stuck at 0%** as follows:
```
......
2025-09-15 21:29:15 | INFO     | lmms_eval.evaluator:evaluate:402 - Running on rank 0 (local rank 0)
2025-09-15 21:29:15 | INFO     | lmms_eval.evaluator_utils:from_taskdict:91 - No metadata found in task config for mme, using default n_shot=0
2025-09-15 21:29:15 | DEBUG    | lmms_eval.caching.cache:load_from_cache:34 - requests-mme-0shot-rank0-world_size1-tokenizer is not cached, generating...
2025-09-15 21:29:15 | INFO     | lmms_eval.api.task:build_all_requests:427 - Building contexts for mme on rank 0...
100%|████████████████████████████████████████████████████████████████████████| 2374/2374 [00:00<00:00, 23048.56it/s]
2025-09-15 21:29:15 | DEBUG    | lmms_eval.evaluator:evaluate:460 - Task: mme; number of requests on this rank: 2374
2025-09-15 21:29:15 | INFO     | lmms_eval.evaluator:evaluate:490 - Running generate_until requests
Model Responding:   0%|                                                                    | 0/2374 [00:00<?, ?it/s]
```
After pressing `Ctrl+C`, the error reported is:
```
Traceback (most recent call last):
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/zzh/codes/sglang/python/sglang/srt/managers/scheduler.py", line 2672, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/home/zzh/codes/sglang/python/sglang/srt/managers/scheduler.py", line 814, in event_loop_overlap
    batch = self.get_next_batch_to_run()
  File "/home/zzh/codes/sglang/python/sglang/srt/managers/scheduler.py", line 1550, in get_next_batch_to_run
    need_dp_attn_preparation = require_mlp_sync(self.server_args)
  File "/home/zzh/codes/sglang/python/sglang/srt/utils.py", line 2435, in require_mlp_sync
    return server_args.enable_dp_attention or require_gathered_buffer(server_args)
  File "/home/zzh/codes/sglang/python/sglang/srt/utils.py", line 2431, in require_gathered_buffer
    return require_mlp_tp_gather(server_args) or require_attn_tp_gather(server_args)
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/zzh/codes/sglang/python/sglang/srt/managers/detokenizer_manager.py", line 294, in run_detokenizer_process
    manager.event_loop()
  File "/home/zzh/codes/sglang/python/sglang/srt/managers/detokenizer_manager.py", line 117, in event_loop
    recv_obj = self.recv_from_scheduler.recv_pyobj()
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/zmq/sugar/socket.py", line 989, in recv_pyobj
    msg = self.recv(flags)
  File "zmq/backend/cython/_zmq.py", line 1218, in zmq.backend.cython._zmq.Socket.recv
  File "zmq/backend/cython/_zmq.py", line 1253, in zmq.backend.cython._zmq.Socket.recv
  File "zmq/backend/cython/_zmq.py", line 1408, in zmq.backend.cython._zmq._recv_copy
  File "zmq/backend/cython/_zmq.py", line 179, in zmq.backend.cython._zmq._check_rc
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/PIL/ImageFile.py", line 644, in _save
    fh = fp.fileno()
AttributeError: '_idat' object has no attribute 'fileno'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/__main__.py", line 539, in <module>
    cli_evaluate()
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/__main__.py", line 347, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/__main__.py", line 474, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/utils.py", line 536, in _wrapper
    return fn(*args, **kwargs)
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/evaluator.py", line 268, in simple_evaluate
    results = evaluate(
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/utils.py", line 536, in _wrapper
    return fn(*args, **kwargs)
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/evaluator.py", line 501, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/models/simple/srt_api.py", line 318, in generate_until
    asyncio.run(run(requests))
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/models/simple/srt_api.py", line 304, in _process
    return await self.generate(request)
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/models/simple/srt_api.py", line 177, in generate
    img = self.encode_image(visual)
  File "/home/zzh/llmserving/benchmark/lmms-eval/lmms_eval/models/simple/srt_api.py", line 127, in encode_image
    image.save(output_buffer, format="PNG")
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/PIL/Image.py", line 2588, in save
    save_handler(self, fp, filename)
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/PIL/PngImagePlugin.py", line 1495, in _save
    ImageFile._save(
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/PIL/ImageFile.py", line 648, in _save
    _encode_tile(im, fp, tile, bufsize, None, exc)
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/PIL/ImageFile.py", line 674, in _encode_tile
    errcode, data = encoder.encode(bufsize)[1:]
KeyboardInterrupt
[2025-09-15 21:47:03] INFO:     Shutting down
[2025-09-15 21:47:03] INFO:     Waiting for application shutdown.
[2025-09-15 21:47:03] INFO:     Application shutdown complete.
[2025-09-15 21:47:03] INFO:     Finished server process [416154]
^CException ignored in atexit callback: <function _exit_function at 0x7f8a18e48310>
Traceback (most recent call last):
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/multiprocessing/util.py", line 357, in _exit_function
    p.join()
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/multiprocessing/popen_fork.py", line 43, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
```
One of the errors is:
```

KeyboardInterrupt
Traceback (most recent call last):
  File "/home/zzh/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/PIL/ImageFile.py", line 644, in _save
    fh = fp.fileno()
AttributeError: '_idat' object has no attribute 'fileno'
```
**What is the reason causing the inference process to get stuck, and how should it be resolved?**

The issue has been solved in https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/826.