Repository: open-compass/opencompass
Issue: [RFC] Add a board for local runner to track the states of all tasks
Root cause: Unimplemented Feature Gap

# Motivation  
The local execution tasks (LocalRunner) of OpenCompass determine whether to enable debug mode by specifying the `--debug` flag. Since debug mode prints all logs of the tasks executed by the runner, it causes excessive on-screen log clutter. Therefore, debug mode is disabled during regular execution. Without debug mode, the execution status of tasks can only be viewed through a progress bar at the granularity of the entire task, as shown below:  

```bash
# Execution command
opencompass --models deepseek_r1_streaming hf_llama_7b  --datasets aime2024_gen gsm8k_gen

# Partial printed logs
11/17 17:07:14 - OpenCompass - INFO - Current exp folder: outputs/default/20251117_170714
11/17 17:07:14 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
11/17 17:07:14 - OpenCompass - INFO - Partitioned into 2 tasks.
launch OpenICLInfer[DeepSeek-R1-0528/gsm8k,DeepSeek-R1-0528/aime2024] on CPU                                                                                                                                                    
launch OpenICLInfer[llama-7b-hf/gsm8k,llama-7b-hf/aime2024] on GPU 0                                                                                                                                                            
 50%|██████████████████████████████████████████████████████████████████████████████████████████████▌                                                                                              | 1/2 [01:01<01:01, 61.31s/it
```  

In non-debug mode, the printed task information is insufficient. When multiple tasks are executed in parallel, it is impossible to monitor the execution status of all tasks simultaneously. If some tasks fail, their logs are easily overwritten by other outputs.  


# Proposed Change  
We aim to introduce a **status dashboard** that can monitor the execution status of all ongoing tasks when running in the local environment (within LocalRunner).  

The status dashboard should include the following information:  
1. **Task Name**: Consistent with the task names printed by OpenCompass (e.g., `OpenICLInfer[DeepSeek-R1-0528/gsm8k]`).  
2. **Task Subprocess ID**: Allows users to detect if the task subprocess has been killed.  
3. **Task Execution Progress**: Displays the progress of inference or evaluation via a progress bar.  
4. **Task Execution Time**: Enables intuitive tracking of how long each task has been running, helping identify task freezes.  
5. **Task Status**: Marks the current phase of the task (e.g., "Model Weight Loading Phase").  
6. **Task Detailed Log Path**: Consistent with the log paths printed by OpenCompass when errors occur (e.g., `outputs/default/20251118_162917/logs/infer/DeepSeek-R1-0528/gsm8k.out`).  
7. **Additional Extended Parameters**: Supports extending other parameters, stored in a dictionary format.  


### Expected Display Effect of the Status Dashboard  

```
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| Task Name                          |   Process | Progress                                              | Time Cost   | Status      | Log Path                                          | Extend Parameters                                    |
+====================================+===========+=======================================================+=============+=============+===================================================+======================================================+
| vllm-api-general-chat/aime2024     |   1682636 | [##############################] 30/30 [30.0 it/s]    | 0:00:03     | finish      | logs/infer/vllm-api-general-chat/aime2024.out     | {'POST': 30, 'RECV': 30, 'FINISH': 30, 'FAIL': 0}    |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| vllm-api-general-chat/GPQA_diamond |   1682701 | [##############################] 198/198 [20.0 it/s]  | 0:00:06     | finish      | logs/infer/vllm-api-general-chat/GPQA_diamond.out | {'POST': 198, 'RECV': 198, 'FINISH': 198, 'FAIL': 0} |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| vllm-api-general-chat/gsm8k        |   1682634 | [##########                    ] 451/1319 [49.9 it/s] | 0:00:11     | inferencing | logs/infer/vllm-api-general-chat/gsm8k.out        | {'POST': 461, 'RECV': 451, 'FINISH': 451, 'FAIL': 0} |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| vllm-api-general-chat/drop         |   1682632 | [                              ] 270/9536 [49.9 it/s] | 0:00:11     | inferencing | logs/infer/vllm-api-general-chat/drop.out         | {'POST': 280, 'RECV': 270, 'FINISH': 270, 'FAIL': 0} |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| triton-api-general/aime2024        |           | NA                                                    | NA          | not start   |                                                   | None                                                 |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| triton-api-general/GPQA_diamond    |           | NA                                                    | NA          | not start   |                                                   | None                                                 |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| triton-api-general/gsm8k           |           | NA                                                    | NA          | not start   |                                                   | None                                                 |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| triton-api-general/drop            |           | NA                                                    | NA          | not start   |                                                   | None                                                 |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
```  


### Architecture Design of the Status Dashboard  

<img width="1906" height="946" alt="Image" src="https://github.com/user-attachments/assets/3b89cde4-8f55-4aa1-8f09-6020015879e2" />

As shown in the diagram above:  
- Before `LocalRunner` starts tasks, a separate thread is launched to run the `task state monitor`, which monitors all tasks.  
- In the subprocess started by `LocalRunner`, a separate thread is launched to run the `state collector` before initializing the task instance. This collector captures the execution status of the task instance throughout its entire lifecycle.  
- The `task state monitor` and `state collector` communicate task status via **temporary files** (intended to be in JSON Lines format) corresponding to each task. The `state collector` appends task status to the temporary file of each task at fixed intervals, while the `task state monitor` reads the latest data from all task temporary files at fixed intervals.  


### Class Diagram for the Changes  

<img width="1580" height="848" alt="Image" src="https://github.com/user-attachments/assets/86374b2b-357c-4e7c-b70b-433c7474ed07" />

As shown in the diagram above:  
- A new member variable `task_states_monitor` is added to `LocalRunner`. In the `LocalRunner.launch` method, an independent thread is used to call `task_states_monitor.launch_state_board` to start the status monitor (not launched in debug mode).  
- A new member variable `state_collector` is added to `BaseTask`, which is initialized by passing an instance via the newly added `init_state_collector` method.  
- Initialization of `TaskStatesCollector` requires passing a temporary file path (for communication with `task_states_monitor`), which is passed via the command line of the task file.  
- The `TaskStatesCollector.update_task_state` method is used to update the task status during the execution of the `BaseTask` instance.  
- The `state_collector` can also be further passed down to the OpenICL layer within the `BaseTask` instance.  


### Command Line Changes  
- For task files that support independent launch via the command line (`openicl_infer.py`, `openicl_attack.py`, `openicl_eval.py`, `subjective_eval.py`), a new command line parameter `--state-tmp-file` needs to be added. This parameter passes the temporary file path for transmitting status information of each task. If no path is specified, `TaskStatesCollector` will not be launched in the task, ensuring no impact on other runners calling this task.  
- When `task_states_monitor` is launched in `LocalRunner`, the `--state-tmp-file /path/to/tmp/file` flag must be added to the locally launched task command constructed as `cmd = task.get_command(cfg_path=param_file, template=tmpl)`.  


# Feedback Period  
1~2 weeks  


# CC List  


# Any Other Things  

### Will you implement it?  
- [x] I would like to implement this feature and create a PR!

A good suggestion, I'm glad to receive your PR.
Why do we need a tmp file to communicate between processes instead of some IPC methods (like multiprocessing.Queue)? What will happen to these tmp files in case of an unexpected interruption?

IPC methods (like multiprocessing.Queue) only work for those subprocess launched by `multiprocessing` lib, but the subprocesses of tasks are launched by `LocalRunner`  using method `subprocess.run`. 
Though, subprocesses can communicate with main process by method `sys.stdout.flush` to broadcast the state infos, but stdout of state info to flush need to be print which means state info will be unexpectedly write to the dumping log file `xxxxx.out`.
Finally, we could only use tmp file method which is the least efficient one to communicate😂