Paper: TaskBench: Benchmarking Large Language Models for Task Automation (ICLR-2024)

Reviewer comment (Evaluation):
I like the back-instruct, but LLM-generated queries can be biased and not well-aligned with user behavior. In addition, for the benchmark to be faithful (which is very important given that benchmark is what people would use to assess model in a relatively long period), the query and answer should be matched, I doubt whether the rule-based and LLM-based check could always ensure the correctness of the generated instruction. I do think human verification on all the generated instruction can make the benchmark more faithful.

Author response:
Regarding the benchmark's faithfulness and the correctness of generated instructions, we acknowledge your concern. Compared with previous methods, we further employ rule-based and LLM-based checks to filter out low-quality samples. Following your suggestions, we are exploring the integration of human verification to enhance accuracy and reliability. This combined approach will ensure a more robust and faithful benchmark. We conducted a human evaluation of the current version of the dataset and put them in Appendix A.3, please refer to [G1] in General Response.

Reviewer comment (Experiments):
I took a close look at the huggingface dataset produced in this paper, a large portion of them are incorrect. For example, one of the data (id: 10949228), the user query is "I have an image 'example.jpg' containing some information. I want to convert the image content into text and then answer the question: 'What is the main topic of the image?", the tool graph is Image-to-text -> VQA. However, once you do image-to-text (assuming captioning), why VQA? it makes no sense. it can just be a textual QA, right? There are many similar cases in the dataset.

Author response:
Thank you for pointing this out. We acknowledge the error in the specific instance you highlighted (id: 10949228). This example illustrates a mismatch of parameter types between VQA and the image-to-text tool: the output of image-to-text is incorrectly linked to the image input of VQA. We conducted a human verification on the constructed dataset, and the results are shown in [G2] of the general response. In our examination, we found the proportion of error cases is nearly 12%. After human verification, we found that our dataset could provide a better quality. We will continue to improve it in two ways:

- **Improving the quality of tool graphs.** Our analysis shows that these mismatches arise primarily when parameter types are not explicitly defined within the tool graph, leading to incorrect assumptions by the LLM about resource types. To rectify this, we propose introducing resource-type nodes in our tool graph. This will clearly delineate the types of resources each tool handles, ensuring that tool nodes are connected in a way that makes logical sense for the given query. This should significantly reduce such mismatches.
- **Perform full human verification.** We recognize the value of human verification in ensuring the accuracy and faithfulness of our benchmark. We are exploring ways to integrate this into our process, alongside our automated checks.

Reviewer comment (Evaluation):
I don't quite understand the goal of evaluating task decomposition as textual descriptions. I do think evaluating task decomposition is important, but as long as it can build the correct tool graph and predict the correct parameter, it is good enough. So I don't think evaluating textual description is necessary, and given how diverse the textual description could be, the current automatic evaluation metric could be misleading.

Author response:
Thank you for your comments and concerns regarding the necessity of evaluating task decomposition. We appreciate your perspective and it is still challenging to determine the most appropriate metric to measure task decomposition. Since no one has explored the evaluation of task decomposition when conducting task automation, we want to explore a new perspective to analyze task decomposition by using textual descriptions. Just as shown in Figure 1 in our paper, the textual descriptions from task decomposition will instruct LLMs to parse tools and predict parameters. Therefore, we think that evaluating textual descriptions can also draw some inspiration to enhance LLMs in the future and reflect the planning capability of LLMs to obtain the decomposed sub-tasks. However, we also acknowledge your concerns that current metrics based on textual descriptions are still not enough to fully reflect the accuracy of task decomposition. We will continue to design better metrics to evaluate task decomposition, including subjective and objective evaluations.

Reviewer comment (Experiments):
When you build a DAG tool graph, do you consider one graph that contains multiple disconnected DAG?

Author response:
Thanks for your question. Yes, when we sample DAG-structured tool subgraphs from the complete tool graph, we also consider scenarios that include multiple disconnected DAGs. Such scenarios represent a category of practical commands, where the potential tool invocation graph comprises several independent sets of tool invocations. In our approach, Back-Instruct is capable of understanding the dependencies within the tool subgraphs and can create separate sub-commands accordingly. For instance, in the example of Table 12, Audio Noise Reduction, Audio Effects, Audio Splicer, Audio-to-Text, and Image-to-Text, Text Sentiment Analysis, Text Search form two independent groups of tool combinations. Correspondingly, Back-Instruct generates instructions for two sub-commands: audio processing and image processing.

| sampled tool graph | Instruction | Tool Invocation Graph |
| --- | --- | --- | --- |
| {"sampling_method": "dag", "number_of_tools": 7, "sampled_nodes": [{"id": "Audio Effects", "input-type": ["audio", "text"], "output-type": ["audio"]}, {"id": "Audio Noise Reduction", "input-type": ["audio"], "output-type": ["audio"]}, {"id": "Audio Splicer", "input-type": ["audio", "audio"], "output-type": ["audio"]}, {"id": "Audio-to-Text", "input-type": ["audio"], "output-type": ["text"]}, {"id": "Image-to-Text", "input-type": ["image"], "output-type": ["text"]}, {"id": "Text Search", "input-type": ["text"], "output-type": ["text"]}, {"id": "Text Sentiment Analysis", "input-type": ["text"], "output-type": ["text"]}], "sampled_links": [{"source": "Audio Noise Reduction", "target": "Audio Effects"}, {"source": "Audio Splicer", "target": "Audio Noise Reduction"}, {"source": "Audio Splicer", "target": "Audio-to-Text"}, {"source": "Image-to-Text", "target": "Text Sentiment Analysis"}, {"source": "Text Sentiment Analysis", "target": "Text Search"}]} | I have an audio file 'example.wav' which has some background noise, please clean it up, transcribe it to text, and analyze its sentiment. Also, I need to extract text from an image 'example.jpg' and search for that text on the internet. Finally, please combine the cleaned audio file with another audio file 'example2.wav' and apply a reverb effect to create the final audio output. | {"nodes": [{"id": "Audio Effects", "input": ["Audio Splicer", "reverb"]}, {"id": "Audio Noise Reduction", "input": ["example.wav"]}, {"id": "Audio Splicer", "input": ["Audio Noise Reduction", "example2.wav"]}, {"id": "Audio-to-Text", "input": ["Audio Noise Reduction"]}, {"id": "Image-to-Text", "input": ["example.jpg"]}, {"id": "Text Search", "input": ["Image-to-Text"]}, {"id": "Text Sentiment Analysis", "input": ["Audio-to-Text"]}], "links": [{"source": "Audio Noise Reduction", "target": "Audio Splicer"}, {"source": "Audio Noise Reduction", "target": "Audio-to-Text"}, {"source": "Audio Splicer", "target": "Audio Effects"}, {"source": "Audio-to-Text", "target": "Text Sentiment Analysis"}, {"source": "Image-to-Text", "target": "Text Search"}]} |

*Table 12. Back-Instructed on DAG with Multiple Disconnected DAGs*

Reviewer comment (Evaluation):
Are tool graphs with more nodes more challenging? I think analysis of the correlation between the difficulty of the test cases and their numbers of nodes is interesting.

Author response:
Yes. The difficulty of LLM to perform task automation is positively correlated with the number of tool nodes that the tool graphs contain. To make a clear understanding of the correlation between the number of nodes in the tool graph and the performance of LLMs in task automation, we conduct a detailed analysis in Table 8. This analysis includes various metrics such as node set accuracy, edge set accuracy, and graph accuracy, which measure the exact-match accuracy of the node set, edge set, and the entire graph, respectively.

| # tool nodes | # supports | node set accuracy  | edge set accuracy  | graph accuracy |
| --- | --- | --- | --- | --- |
| 1 (single) | 2059 | 96.16 | - | 96.16 |
| 2 | 278 | 86.33 | 84.53 | 84.53 |
| 3 | 1313 | 67.93 | 60.70 | 60.39 |
| 4 | 1280 | 64.29 | 75.62 | 54.37 |
| 5 | 731 | 54.03 | 70.53 | 41.58 |
| 6 | 290 | 50.34 | 39.31 | 39.31 |
| 7 | 151 | 49.66 | 36.42 | 36.42 |
| 8 | 60 | 35.00 | 25.00 | 25.00 |
| 9 | 55 | 38.18 | 21.81 | 21.81 |
| 10 | 64 | 39.06 | 31.25 | 31.25 |
| overall | 6281 | 73.52 | 67.55 | 67.25 |

*Table 13.  Task automation performance with different number of tools on gpt-4*

From Table 13, we observed that as the number of tools in the tool graphs increases, there is a clear downward trend in various performance metrics such as node set accuracy, edge set accuracy, and graph accuracy. This trend confirms that tool graphs with a higher number of tools present more complex challenges for LLMs in task automation. Specifically, the result shows a significant drop in performance metrics when moving from simpler graphs (1-2 tools) to more complex ones (3 or more tools). For instance, while single-node graphs achieve a high graph accuracy of 96.16%, this metric falls to 39.31% for graphs with 6 tools and further decreases to 25.00% for 8-node graphs. This correlation between the number of tools and the difficulty of the test cases can be attributed to the increased complexity in understanding and processing more extensive and intricate links between tools. As the number of tools grows, LLMs must handle a larger set of possible dependencies, which significantly challenges their predictive and analytical capabilities. The results from this analysis underline the importance of continuous advancements in LLM capabilities to be consistent with the increasing complexity of tasks in various domains.

Reviewer comment (Experiments):
One of my concern is that depending on the subgraph sampling procedure (which is not clearly described in the paper), the task might be unnatural (deviating from what users would ask in real-world) while being valid. For example, one can always make a complex task by combining many tools "Tool 1" --> "Tool 2" --> ...."Tool N", while the task is rarely encountered in real-world. Does the sampling procedure take "task naturalness" into account? Also, I'd suggest to include some actual examples in the paper.

Author response:
Thanks for your question. Here we mainly consider "task naturalness" in the instruction generation stage since currently, it is challenging to consider "task naturalness" in sampling without any human supervision. Therefore, to improve the naturalness, we adopt LLM-based and rule-based critics to filter out instances that rarely occur in the real world. We will follow your suggestions to introduce "task naturalness" into the sampling stage by using LLM critics. Besides, following your and other reviewers' suggestions, we have conducted human verification on our datasets to detect errors (Please see [G1] & [G2] of General Response) and further improve the quality of our datasets. Besides, we have presented concrete samples in [G1] of General Response and put them in Appendix A.4.1 of the revised version.

Reviewer comment (Evaluation):
While back-instructing to generate user requests is an intuitive approach. This procedure relies on GPT-4 and there is no guarantee that the generated data is correct (e.g., many of the examples are filtered out by simply using rule-based critics as mentioned). Even if more advanced LLM-based critic is used, there may still be wrong examples. Since the dataset is intended for evaluation, the data quality is of most the utmost importance. Without further (potentially manual) verification, it is not very convincing on the reliability of the benchmark.

Author response:
Thanks for your suggestions. We admit it will be better with further human verification. We have conducted human verification and analyzed error patterns in our datasets (Please see [G1] and [G2] of General Response). Besides, compared with previous datasets (e.g., ToolBench) which all used instruction methods, we further conducted LLM-based and rule-based critics (Table 1) to filter out low-quality samples and reported the agreement of LLM rankings under human and TaskBench (Table 7). Due to the limited rebuttal period, we will continue to refine the reliability of our benchmark by using subjective and objective evaluations.

Reviewer comment (Novelty):
As discussed in related work, there are several parallel efforts in creating benchmarks for LLM tool usage, and perhaps the most related is ToolBench [1]. Can the authors provide more detailed discussion/comparison to the work, and highlight the contribution in this work?

Author response:
Thanks for the suggestion. Due to the page limitation, we have put some discussions between the concurrent work in the appendix in the initial version. To better understand the differences between TaskBench and other concurrent work, we have conducted in-depth comparisons in Appendix A.2 of our latest version from the perspective of datasets and evaluations. Please see [G3] in General Response to the differences and the advantages of our method.

Reviewer comment (Presentation):
Tool graph construction: is there an example on the constructed tool graph before subsampling?

Author response:
Thanks for your question. In Appendix A.10 of the revised version, we have added a visualization of the constructed tool graph on Multimedia domain for reference, where the nodes denote tools, and directed edges indicate that the output type of the source tool matches the input type of the target tool.

Reviewer comment (Evaluation):
In Figure 3, GPT-4 almost reaches 100% in performance, would the benchmark will be saturated soon, perhaps when the next generation LLMs come out?

Author response:
Thanks for your question. We guess this is because our evaluation dataset is also generated by GPT-4, so that its performance will be overestimated when also using GPT-4 for testing. Therefore, we also involve human evaluation to validate its quality. Besides, we believe that when more powerful LLMs come out, the dataset generated by these LLMs will be better and demonstrate better alignments to human preference when compared with GPT-4.

Reviewer comment (Evaluation):
The paper focuses on the evaluation of task solving (agent) capabilities of different LLMs. Although the experiments are enough, the analyses are partial. For instance, which contributes to the performance of different LLMs? What are the intrinsic differences between different existing LLMs in performing agent tasks? Which findings can we derive to better improve the capabilities of current open-source LLMs? Personally, insights from the evaluation results are somehow shallow.

Author response:
Thank you for your feedback. We acknowledge your concerns regarding the perceived partiality in our analyses. To resolve your concerns, we have conducted further analyses of the aspects you highlighted:

**1) Factors Contributing to Task Automation Performance:**

- **Reasoning**: The capacity for complex problem-solving and reasoning varies significantly among LLMs. This ability is crucial for understanding and executing complex tasks. In mathematical and coding tasks, GPT series models show stronger reasoning ability, therefore they also possess corresponding capabilities in task planning and tool utilization.
- **Instruction Following**: The proficiency in comprehending and adhering to user instructions differs, influencing task execution efficiency. Our analysis reveals that models fine-tuned with instruction-following settings, such as Vicuna-13b, WizardLLM-13b, and Nous-Hermes-13b, outperform the baseline Llama-2-13b model in task automation. Specifically,  we observe that WizardLLM-13b can perform better than Vicuna-13b since it fine-tunes more complex instructions. These results also demonstrate the necessity of instruction following.

**2) Intrinsic Differences of LLMs in Performing Agent Tasks:**

- **Code Pre-training:** We note that the models (Code-Llama) with more code-pretraining outperform other open-source LLMs in task automation. Experimental results show an average improvement of 4.45% in tool prediction (\textit{n-f1}) and 12.76% in parameter prediction (\textit{v-f1}) across various domain datasets. We conclude that since task automation usually involves multiple stages and tools, using code-style / structural text as the interface to switch different stages is better.
- **Alignment Techniques**: Besides, the models (e.g., GPT series models) with human alignments (e.g., RLHF) demonstrate stronger task automation capabilities than open-source LLMs. These results also indicate that RLHF allows large language models to develop more generalized reasoning abilities, reducing overfitting to specific instructions.

Based on our analysis, we propose the following improvements for open-source LLMs:

- **Enhanced Code Pre-training**: Strengthening code pre-training can significantly boost the performance of LLMs in task automation.
- **Quality of Instructions**: Developing high-quality (complex and diverse) instructions is crucial for crafting more efficient LLMs.
- **Ongoing Knowledge Acquisition**: Encouraging continuous learning and knowledge updating can make LLMs more adaptable and proficient.

We have incorporated these additional analyses in Section 3.5 of the revised version to offer a more comprehensive view of the capabilities and development of LLMs in task automation. This should provide a deeper understanding and address the concerns you have raised about the initial shallowness of our insights.

Reviewer comment (Experiments):
For an evaluation paper, I think more diverse LLMs should be evaluated as well, such as Claude-2. I also expect a case study to show the performance gap among different LLMs.

Author response:
Thank you for your valuable suggestion. In response, we have incorporated the evaluation results of Claude-2 into our analysis. From Table 5-9, we observed that Claude-2 exhibited superior performance compared to gpt-3.5-turbo, yet it did not match the capabilities of GPT-4. Due to the limited time of the rebuttal period, we will add more diverse LLMs to our paper in the final version.

|  | R1↑  | R2↑  | BsF↑ |
| --- | --- | --- | --- |
| Multimedia | 49.19 | 23.71 | 89.22 |
| Hugging Face | 44.49 | 21.37 | 88.71 |
| Daily APIs | 81.28 | 68.06 | 95.65 |

*Table 5. Task Decomposition Performance of Claude-2.*

|  | n-f1↑ | t-f1↑ | v-f1↑ |
| --- | --- | --- | --- |
| Multimedia | 66.15 | 53.57 | 23.79 |
| Hugging Face | 69.77 | 48.29 | 32.33 |
| Daily APIs | 79.73 | 78.14 | 60.10 |

*Table 6. Tool Invocation and Parameter Prediction Performance of Claude-2 in node-structure Tasks.*

|  | n-f1↑ | e-f1↑ | t-f1↑ | v-f1↑ |
| --- | --- | --- | --- | --- |
| Multimedia | 82.69 | 60.76 | 75.83 | 59.59 |
| Hugging Face | 80.07 | 48.61 | 67.10 | 45.81 |
| Daily APIs | 95.62 | 62.17 | 95.13 | 64.95 |

*Table 7. Tool Invocation and Parameter Prediction Performance of Claude-2 in chain-structure Tasks.*

|  | n-f1↑ | e-f1↑ | t-f1↑ | v-f1↑ |
| --- | --- | --- | --- | --- |
| Multimedia | 85.47 | 63.42 | 77.70 | 60.61 |
| Hugging Face | 83.17 | 52.66 | 69.91 | 47.93 |
| Daily APIs | 93.76 | 55.96 | 92.49 | 64.69 |

*Table 8. Tool Invocation and Parameter Prediction Performance of Claude-2 in DAG-structure Tasks.*

|  | n-f1↑ | e-f1↑ | t-f1↑ | v-f1↑ |
| --- | --- | --- | --- | --- |
| Multimedia | 81.07 | 56.62 | 73.08 | 54.40 |
| Hugging Face | 79.15 | 45.19 | 64.52 | 43.87 |
| Daily APIs | 93.73 | 57.66 | 92.89 | 64.47 |

*Table 9. Tool Invocation and Parameter Prediction Performance of Claude-2 in All Tasks.*

Reviewer comment (Novelty):
Missing discussion with a very relevant work ToolLLM [1]. The workflow is quite similar: preparing high quality tools (or APIs), back-generate the instructions that involve these APIs, and annotate how LLMs solve these instructions. The main difference is that this paper involves a concept of Tool Graph when organizing the structure of tools/APIs. Though ToolLLM can be considered as a concurrent work, I think the authors should discuss their core differences and the unique advantage of this submission (experiments, findings, etc.).

Author response:
Thanks for the suggestion. Due to the page limitation, we have put some discussions between the concurrent work in the appendix in the initial version. To better understand the differences between TaskBench and other concurrent work, we have conducted in-depth comparisons in Appendix A.2 of the latest version from the perspective of datasets and evaluations. Please see [G3] in General Response to the differences and the advantages of our method.

Reviewer comment (Evaluation):
If the authors can provide more examples (predictions and gold answers) in appendix, it would be useful to help readers understand the difficulty of each tasks and some error cases/analysis in main text would be useful. Otherwise, the number itself cannot provide too much information regarding this dataset.

Author response:
Thanks for your suggestions. To resolve your concerns, we have added more examples in Appendix A.4 to illustrate the difficulty and error cases in conducting task automation based on TaskBench. Please also see our response in [G2] for case studies and error analysis.

Reviewer comment (Evaluation):
Comparing to the node prediction, the edge prediction is harder based on the evaluation results. What’s the error types for edge prediction? Does the model make equivalent edge prediction but different connectivity?

Author response:
Thanks for your question. Yes, edge prediction is more challenging compared to node prediction because it requires accurate prediction of both the source and target nodes of an edge. Currently, we summarize the main error types in edge prediction as two parts:

- Mismatch of resource types between source node output and target node input.
- Missing edge prediction between nodes where a resource or temporal dependency exists.

We analyzed 100 randomly sampled examples of edge prediction failures and statistically 27% and 38% of edge prediction errors belong to the above two types respectively.