Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to accelerate multimodal data(image, video) loading?
Root cause: Architectural Constraint

For vlm tasks, we need to load frames locally, which might spend a lot time for IO reading.
i.e. [load_video](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_onevision.py#L447) in llava-onevision.

When training llava-ov, we can use pytorch Dataloader with `num_workers>1` to accelerate data loading. 
However, in lmms-eval, we are loading it in the `generate_until` fuction, which can not set multiple workers.

Any way to accelerate data loading process? With current code,my GPU util is very low.


We do notice this when we performing the video evaluation. Since the structure for our pipeline currently can only take one `doc_to_visual` and read the actual visual during inferencing, the only way to accelerate data loading is to split the process on multiple gpu nodes. We may take a lot into it in future version and see if we have some solution for this. Since this may takes large refactor to our code, we might not update it recently.

> We do notice this when we performing the video evaluation. Since the structure for our pipeline currently can only take one `doc_to_visual` and read the actual visual during inferencing, the only way to accelerate data loading is to split the process on multiple gpu nodes. We may take a lot into it in future version and see if we have some solution for this. Since this may takes large refactor to our code, we might not update it recently.

Hi I am wondering does the current version has resolve this problem? trying to eval on benchmarks like scanrefer, yet super slow even with multiple GPU node.

Currently, in models such as `vllm`, we use threadpool to extract video faster. However, this currently can only apply on models that support batch size larger than 1. To process video faster, you should split the video into chunks and then read each video chunks and then combine them. However, on `lmms-eval`, when applying multiprocessing + threading to read video fast, it would be extremely easy to OOM on cpu so I would prefer not to include this into the main version. A sample code is like this

```python
from qwen_vl_utils import fetch_video

def process_video(video_path: str, fps: int, start_time: float = 0, end_time: float = None, nframes: int = None, max_frames: int = None):
    video_dict = {
        "type": "video",
        "video": f"file://{video_path}",
        "fps": fps,
        "max_pixels": MAX_PIXELS,
        "video_start": start_time,
        "video_end": end_time,
    }
    if nframes is not None:
        video_dict["nframes"] = nframes
        video_dict.pop("fps")
    if max_frames is not None:
        video_dict["max_frames"] = max_frames
    return fetch_video(video_dict)

def process_video_fast(
    video_path: str,
    fps: int,
    nframes: int = None,
    num_threads: int = 16,
    num_chunks: int = 10,
    return_video_length: bool = False,
    disable_tqdm: bool = False,
    max_frames: int = None,
):
    """
    Process video in parallel by splitting it into 10-second chunks.

    Args:
        video_path: Path to the video file
        fps: Frames per second for processing
        num_threads: Number of threads to use for parallel processing

    Returns:
        torch.Tensor: Concatenated video tensor with shape [total_frames, 3, H, W]
    """
    # Get video length
    video_length, total_frames = get_video_length(video_path, return_nframes=True)
    if video_length <= 0:
        raise ValueError(f"Invalid video length: {video_length}")

    if max_frames is not None:
        if nframes is not None:
            nframes = min(nframes, max_frames)
        else:
            fps_frames = fps * video_length
            # If max frames is less than fps * video length, we use max frames
            # else we still let fps=fps
            nframes = max_frames if max_frames < fps_frames else None

    # Define chunk size (10 seconds)
    chunk_size = video_length / num_chunks

    # Calculate number of chunks
    num_chunks = int((video_length + chunk_size - 1) // chunk_size)
    if nframes is not None:
        nframes = min(nframes, total_frames)
        frames_per_chunk = nframes // num_chunks
    else:
        frames_per_chunk = None

    def process_chunk(chunk_idx):
        """Process a single video chunk."""
        start_time = chunk_idx * chunk_size
        end_time = min((chunk_idx + 1) * chunk_size, video_length)

        try:
            chunk_tensor = process_video(video_path, fps, start_time, end_time, frames_per_chunk)
            return chunk_idx, chunk_tensor
        except Exception as e:
            print(f"Error processing chunk {chunk_idx}: {e}")
            return chunk_idx, None

    # Process chunks in parallel using ThreadPoolExecutor
    chunks = []
    with ThreadPoolExecutor(max_workers=num_threads) as executor:
        # Submit all chunk processing tasks
        future_to_chunk = {executor.submit(process_chunk, i): i for i in range(num_chunks)}

        # Collect results as they complete
        pbar = tqdm(total=num_chunks, desc="Processing video chunks", disable=disable_tqdm)
        for future in as_completed(future_to_chunk):
            chunk_idx, chunk_tensor = future.result()
            if chunk_tensor is not None:
                chunks.append((chunk_idx, chunk_tensor))
            pbar.update(1)
        pbar.close()

    # Sort chunks by index to maintain temporal order
    chunks.sort(key=lambda x: x[0])

    # Concatenate all chunks along the first dimension (frames)
    if not chunks:
        raise ValueError("No valid chunks were processed")

    # Extract tensors in order
    tensors = [chunk[1] for chunk in chunks]

    # Concatenate along the first dimension
    final_tensor = torch.cat(tensors, dim=0)

    if return_video_length:
        return final_tensor, video_length
    else:
        return final_tensor
```

If you have enough cpu memory or you don't need too large parallelism, you can try to tune the parameters on your local machine to avoid OOM.

> Currently, in models such as `vllm`, we use threadpool to extract video faster. However, this currently can only apply on models that support batch size larger than 1. To process video faster, you should split the video into chunks and then read each video chunks and then combine them. However, on `lmms-eval`, when applying multiprocessing + threading to read video fast, it would be extremely easy to OOM on cpu so I would prefer not to include this into the main version. A sample code is like this
> 
> from qwen_vl_utils import fetch_video
> 
> def process_video(video_path: str, fps: int, start_time: float = 0, end_time: float = None, nframes: int = None, max_frames: int = None):
>     video_dict = {
>         "type": "video",
>         "video": f"file://{video_path}",
>         "fps": fps,
>         "max_pixels": MAX_PIXELS,
>         "video_start": start_time,
>         "video_end": end_time,
>     }
>     if nframes is not None:
>         video_dict["nframes"] = nframes
>         video_dict.pop("fps")
>     if max_frames is not None:
>         video_dict["max_frames"] = max_frames
>     return fetch_video(video_dict)
> 
> def process_video_fast(
>     video_path: str,
>     fps: int,
>     nframes: int = None,
>     num_threads: int = 16,
>     num_chunks: int = 10,
>     return_video_length: bool = False,
>     disable_tqdm: bool = False,
>     max_frames: int = None,
> ):
>     """
>     Process video in parallel by splitting it into 10-second chunks.
> 
>     Args:
>         video_path: Path to the video file
>         fps: Frames per second for processing
>         num_threads: Number of threads to use for parallel processing
> 
>     Returns:
>         torch.Tensor: Concatenated video tensor with shape [total_frames, 3, H, W]
>     """
>     # Get video length
>     video_length, total_frames = get_video_length(video_path, return_nframes=True)
>     if video_length <= 0:
>         raise ValueError(f"Invalid video length: {video_length}")
> 
>     if max_frames is not None:
>         if nframes is not None:
>             nframes = min(nframes, max_frames)
>         else:
>             fps_frames = fps * video_length
>             # If max frames is less than fps * video length, we use max frames
>             # else we still let fps=fps
>             nframes = max_frames if max_frames < fps_frames else None
> 
>     # Define chunk size (10 seconds)
>     chunk_size = video_length / num_chunks
> 
>     # Calculate number of chunks
>     num_chunks = int((video_length + chunk_size - 1) // chunk_size)
>     if nframes is not None:
>         nframes = min(nframes, total_frames)
>         frames_per_chunk = nframes // num_chunks
>     else:
>         frames_per_chunk = None
> 
>     def process_chunk(chunk_idx):
>         """Process a single video chunk."""
>         start_time = chunk_idx * chunk_size
>         end_time = min((chunk_idx + 1) * chunk_size, video_length)
> 
>         try:
>             chunk_tensor = process_video(video_path, fps, start_time, end_time, frames_per_chunk)
>             return chunk_idx, chunk_tensor
>         except Exception as e:
>             print(f"Error processing chunk {chunk_idx}: {e}")
>             return chunk_idx, None
> 
>     # Process chunks in parallel using ThreadPoolExecutor
>     chunks = []
>     with ThreadPoolExecutor(max_workers=num_threads) as executor:
>         # Submit all chunk processing tasks
>         future_to_chunk = {executor.submit(process_chunk, i): i for i in range(num_chunks)}
> 
>         # Collect results as they complete
>         pbar = tqdm(total=num_chunks, desc="Processing video chunks", disable=disable_tqdm)
>         for future in as_completed(future_to_chunk):
>             chunk_idx, chunk_tensor = future.result()
>             if chunk_tensor is not None:
>                 chunks.append((chunk_idx, chunk_tensor))
>             pbar.update(1)
>         pbar.close()
> 
>     # Sort chunks by index to maintain temporal order
>     chunks.sort(key=lambda x: x[0])
> 
>     # Concatenate all chunks along the first dimension (frames)
>     if not chunks:
>         raise ValueError("No valid chunks were processed")
> 
>     # Extract tensors in order
>     tensors = [chunk[1] for chunk in chunks]
> 
>     # Concatenate along the first dimension
>     final_tensor = torch.cat(tensors, dim=0)
> 
>     if return_video_length:
>         return final_tensor, video_length
>     else:
>         return final_tensor
> If you have enough cpu memory or you don't need too large parallelism, you can try to tune the parameters on your local machine to avoid OOM.

thx for the reply, i just check the scanrefer dataset, seems that they are 32frames not a single video. So i try to wrap the "chunks" with torch dataloader to reduce io overhead. Yet no obvious speedup observed, still large GPU idle bubble. Do you have any idea where the bottlenech will be?

I am not sure if whether io is the bottleneck in this case since only 32 frames are being loaded. Can you do a time count and profile each step in the generate until to locate the time used during generation and in io?