Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once? (2024)

Abstract: Large language models (LLMs) are typically prompted to follow a single instruction per inference call. In this work, we analyze whether LLMs also hold the capability to handle multiple instructions simultaneously, denoted as Multi-Task Inference. For this purpose, we introduce the MTI Bench (Multi-Task Inference Benchmark), a comprehensive evaluation benchmark encompassing 5,000 instances across 25 tasks. Each task in the MTI Bench involves 2 to 3 sub-tasks. As expected, we first demonstrate that Multi-Task Inference reduces the total inference time by \times 1.46 times in average since it does not require multiple inference calls. Interestingly, contrary to the expectation that LLMs would perform better when tasks are divided, we find that state-of-the-art LLMs, such as Llama-2-Chat-70B and GPT-4, show up to 7.3% and 12.4% improved performance with Multi-Task Inference compared to Single-Task Inference on the MTI Bench. We release the MTI Bench dataset and our code at this [link](https://anonymous.4open.science/r/MTI-Bench-6F01).

## introduction large language models (llms) capable of following instructions have demonstrated impressive performance across a wide range of tasks @xcite @xcite @xcite . however, since llms are trained to follow a single instruction per inference call, it is questionable whether they also hold the ability to follow complex instructions that necessitate handling multiple sub-tasks @xcite @xcite . moreover, current evaluation resources are either confined to measuring the llm's capability in following one-step instructions @xcite ; chiang @xcite . @xcite or only diagnose the capability to process multi-step instructions in a particular domain such as commonsense reasoning and arithmetic @xcite @xcite . in this paper, we analyze whether llms hold the capability to handle tasks composed of multiple instructions at one inference call, which we denote as multi-task inference. as shown in figure 1 , we compare the performance and speed with two baselines: (1) single-task inference: addressing sub-tasks sequentially and (2) batch prompting: simultaneously processing multiple instances from the same task @xcite . for this purpose, we construct the mti bench (multi-task inference benchmark), an evaluation dataset featuring 25 tasks, each consisting of 2 to 3 sub-tasks. as shown in figure 2 , the mti bench is divided into two distinct subsets: (a) the multi-step subset, which evaluates the models' ability follow multiple instructions sequentially and (b) the multi-part subset, focusing on the models' capability to handle multiple sub-tasks that do not have a sequential dependency. notably, the mti bench sets itself apart from previous multi-hop reasoning @xcite and multi-turn conversation @xcite evaluation suites by providing annotations to assess the intermediate performance of llms while solving multi-task instructions. this enables researchers to check if llms reach the correct answers and evaluate whether their reasoning process is consistent and logical throughout the process. we evaluate 11 llms capable of following instructions, varying in parameter size. surprisingly, on the mti bench, state-of-the-art llms such as llama-2-chat-70b and gpt-4 show up to 7.3% and 12.4% better performance with multi-task inference compared to single-task in-ference. moreover, multi-task inference requires x1.46 times less average inference time than single-task inference. these results indicate that users could obtain similar performance with substantially less time when querying instructions that necessitate solving multiple sub-tasks. through ablation experiments, we suggest that looking at the next sub-task provides critical clues on the answer format for solving the previous subtask. our contributions are as follows: ## experimental setup in this section, we explain our experimental setup for investigating the multi-task inference capabilities of llms. baseline inference methods in addition to multi-task inference, the method in our main consideration, we compare with single-task in-ference and batch prompting @xcite . figure 1 illustrates a scenario that compares the three inference methods. assuming that we are testing an llm with two instances that consist of 3 sub-tasks, the most naive approach, single-task inference prompts an llm 6 times, where each inference call corresponds to solve a single sub-task. on the other hand, batch prompting groups the same sub-tasks and prompts an llm to solve multiple instances at once. lastly, multi-task inference prompts the llm to solve all the multiple sub-tasks within a single inference call. in general, if n instances consisting of m sub-tasks are given, single-task inference requires n times more inference calls compared to batch prompting and m times more inference calls compared to multi-task inference. test models we evaluate eleven llms capable of following instructions including: (1) gpt-4 (openai, 2023), (2) gpt-3.5 (openai, 2022), (3) tulu (7b, 13b, 30b, 65b) @xcite , (4) vicuna (7b, 13b) @xcite , and (5) llama-2-chat (7b, 13b, 70b) @xcite . for gpt-4 and gpt-3.5, we utilize the 0613 version. reported results represent the average of three runs, except for gpt-4, which were evaluated in a single run to minimize costs. opensource models were run using fp16 precision. all evaluations were conducted in a single-shot setting, incorporating chain-of-thought reasoning. the hyperparameters used for evaluation are detailed in appendix b. ## experimental results in this section, we compare single-task infer-ence, batching prompting and multi-task inference on the mti bench (section 5.1), study the inference latency of each method (section 5.2) and study the efficicacy of multi-task inference on free-form generation (section 5.3). ## analysis of multi-task inference in our previous section, although it is clear that multi-task inference guarantees speed-up (as explained in section 5.2), it is rather unexpected and surprising that larger models show improved performance on the mti bench ( in ## conclusion in this work, we present the mti bench, a comprehensive benchmark consisting of 5,000 in-5613 stances spanning 25 diverse tasks, designed to assess the capability of llms in simultaneous multitasking. our analysis within the benchmark compares multi-task inference, single-task inference and batch prompting. the results indicate a superior performance by multi-task inference, despite reduced inference steps and a 1.46-fold increase in speed, demonstrating its efficiency in handling concurrent tasks.