Repository: open-compass/opencompass
Issue: [RFC]For API model backend, submit all input datas in continuous batching way
Root cause: Unimplemented Feature Gap

## Motivation.
1.  When models access inference services via APIs (e.g., using `OpenAISDKStreaming`), the default icl_inferencer (`GenInferencer`) sends requests to the inference service batch by batch according to `batch_size`, which is inefficient. When the data lengths within a batch vary significantly, this request method struggles to fully utilize the inference service resources, leading to low efficiency in testing datasets.
2.  Currently, the API backends supported by OpenCompass all access commercial services (e.g., OpenAI SDK, DouBao). To test locally deployed vLLM or SGLang services, modifications to these commercial API backends are required, preventing out-of-the-box usage.

## Proposed Change.
1.  During testing, it is desired to send requests to the inference service using a continuous batching approach. Specifically, all prompt lists constructed by the retriever are submitted to a thread pool for sending, where the `max_workers` of the thread pool is set to `batch_size`.
2.  Currently, most inference services are compatible with the inference interface format defined by OpenAI. Several generic API model backends (both streaming and non-streaming) that adhere to OpenAI's interface definition can be provided.

### Class Diagram for continuous batching Modifications

<img width="1498" height="846" alt="Image" src="https://github.com/user-attachments/assets/0b9e43e7-53ab-478b-8759-e22fa4798f11" />

*   Add a `generate_single` method to the base class `BaseAPIModel` for executing requests for single cases. The interface definition of `generate_single` is almost identical to the `generate` method (except that the input is a single `PrompType`). Its function is to handle a single inference request. The implementation of `generate_single` in subclasses can directly call the private method `_generate`, which is implemented by all subclasses. All API model backend subclasses that need to support continuous batching inference must implement the `generate_single` method.
*   Within the `GenAPIInferencer.inference` method, execute all prompt lists constructed by the retriever in a thread pool, where the `max_workers` of the thread pool is `batch_size`.
*   Other inferencers besides `GenInferencer` can be extended similarly by referring to `GenAPIInferencer`.

### How to Trigger the New continuous batching Feature
Since inferencer configurations are written in dataset configuration files, triggering continuous batching requires replacing `GenInferencer` with `GenAPIInferencer`. Two optional methods are available:
1.  Add the `--continuous-batch` parameter via the command line. This will replace `GenInferencer` with `GenAPIInferencer` in `fill_infer_cfg`.
2.  When importing dataset configurations in a custom configuration file, replace the inferencer-related configuration in the file with `GenAPIInferencer`.

## Feedback Period.
1~2 weeks

## CC List.

## Any Other Things.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!


We’re currently working on this issue as well.
A simple solution is to set the batch size to a large value and use `num_workers` to control the request rate of `ThreadPoolExecutor`.
Internally, we also have a plan to fully refactor the workflow using async mechanisms (see #1763 ), but since it involves large-scale changes, it has remained in the draft stage.
We will prioritize this requirement later and support not specifying a batch size. For example, when setting `batch_size = -1`, all samples will be passed in directly. This also involves modifying the mechanism for saving intermediate results needed for resumption.


 [#1763 ](https://github.com/open-compass/opencompass/pull/1763) This draft seem to change too many parts(from runner to models) .May I ask what considerations are behind this? As you can see in our Class Diagram, our proposal only need to realize a new `icl_inferencer` and a new base method for `BaseAPIModel`.