Beyond Text: Unveiling Multimodal Proficiency of Large Language Models with M ulti API Benchmark (2024)

Abstract: The proliferation of Large Language Models like ChatGPT has significantly advanced language understanding and generation, impacting a broad spectrum of applications. However, these models predominantly excel in text-based tasks, overlooking the complexity of real-world multimodal information. This study introduces MultiAPI , a pioneering comprehensive large-scale API benchmark dataset aimed at expanding LLMs’ proficiency in multimodal contexts. Developed collaboratively through ChatGPT, MultiAPI consists of 187 diverse API calls and 1,799 contextual prompts, offering a unique platform evaluation of tool-augmented LLMs handling multimodal tasks. Through comprehensive experiments, our findings reveal that while LLMs demonstrate proficiency in API call decision-making, they face challenges in domain identification, function selection, and argument generation. What’s more, we surprisingly notice that auxiliary context can actually impair the performance. An in-depth error analysis paves the way for a new paradigm to address these challenges, suggesting a potential direction for future LLM research.

## introduction large language models (llms), such as chatgpt, have emerged as powerful tools in understanding and generating human language @xcite @xcite , playing a pivotal role in diverse open-domain tasks and leaving a significant impact on both industry and academia @xcite @xcite . however, their performance is often confined to the text-based domains and tasks they were trained on, overlooking the multimodal and dynamic nature of real-world information. as people increasingly rely on llms to address their daily challenges, the demand for enhancing the task-handling capabilities of these models grows ever more pressing. in addition to addressing many of people's emerging needs in the real world, enhancing llms with multimodal problem-solving skills could be a significant step towards the realization of agi in an idealized future @xcite . reflecting this demand and vision, recent studies have embarked on two primary approaches to integrate multimodal processing capabilities into existing llms @xcite : 1) joint training or finetuning llms with components for multimodal encoding and generation @xcite @xcite ; 2) introducing auxiliary api tools via natural language interfaces @xcite @xcite , positioning llms as the central decisionmaking entity determining the appropriate tools to employ for the inquiry. joint training of multimodal llms, despite creating more unified models, faces challenges with computational demands and potential loss of the generalization ability @xcite . on the other hand, evolving api functions, which are modularly designed, allow llms to adapt to new tasks by simply altering the api configuration. despite the significant potential and flexibility the tool-augmented llms express on multimodal tasks, their quantitative performance of multimodal tasks when integrated with api tools still remains insufficiently examined. recent studies are very inadequate and merely focus on and gleaning insights from open-domain tasks such as mathematical computations, database searches, and graph reasoning @xcite @xcite . this gap in leveraging api tools to achieve multimodal tasks can be attributed to two primary obstacles: 1) the unavailability of high-quality apiprompt datasets, and 2) the absence of established metrics specifically designed to evaluate the efficacy of llms in multimodal tasks. in this paper, we address the aforementioned challenges by constructing a large-scale api instruction-function dataset and evaluates llms' multimodal performance, called multiapi. based on the huggingface dataset @xcite , we extracted models with high-quality descriptions across 9 domains along with their instructions. these models were initially encapsulated as api functions using chatgpt prompts, followed by meticulous human refinements to ensure executability and consistent arguments across domains. this help create the multiapi benchmark dataset with 187 functional api calls and 1,799 instructions. we subsequently conducted experiments on both api-based llms and open-sourced llms, exploring strategies that were previously proven effective in improving llm prompting such as in-context learning @xcite and chain-of-thought @xcite . our investigation spanned singlestep api call (only 1 api is required to resolve the instruction) and sequential api chain (multiple apis are required) settings, evaluating 4 intuitive aspects: 1) invocation assessment; 2) domain match; 3) function match; and 4) argument match. results revealed that while models accurately make decisions to invoke api functions, they often suffer from selecting the right function and parameters from the correct domain. furthermore, we surprisingly noticed that adding auxiliary context could harm the api call performance. extensive error analyses were conducted to understand the potential cause of such errors, leading us to propose two simple yet effective solutions to mitigate these errors. the experimental results validate the effectiveness of our method. we summarize the contributions of this paper as follows: ## experiments in this section, we extensively test our mutilapi benchmark to evaluate llms' multimodal task handling via tool integration, covering api-based and open-source models. we explore various prompt configurations to find the most effective settings for multimodal tasks. ## improvement framework our analysis in sections 4 and 5 reveals that llms primarily struggle with distinguishing domains and modalities, with argument generation as a significant bottleneck. to mitigate these challenges, we propose two intuitive yet effective solutions: domain description prompting and argument revision. domain description prompting involves adding a sentence to the model's system prompt to clearly define each domain. in addition, in visual analysis tasks, we specify whether the domain conducts global or local image analysis. building on research showing llms' effectiveness in evaluation and revision tasks @xcite , we employ a secondary llm as an argument editor. this llm checks and revises argument predictions to ensure they align with user instructions, reducing task complexity and the context length for the primary llm. to avoid the noise arising from complex interactions between function calling feature and input context, we conducted our experiments using the gpt-3.5 model. ## conclusion in this paper, we presented a comprehensive study on the application of llms to multimodal tasks with external api functions, using the newly introduced multiapi dataset. our findings highlight the capabilities and limitations of llms in function calling. we revealed a significant discrepancy between the models' ability to recognize the need for function calls and their accuracy in selecting appropriate domains, functions, and arguments. this insight led us to propose a novel approach focusing on domain description prompting and argument revision, which demonstrated improved performance in addressing these challenges. our work contributes to the field by introducing the first large-scale multimodal instruction-function benchmark dataset and providing a detailed analysis of llms in multimodal task execution. we hope our dataset and findings could assist the development of tool-augmented llms and more sophisticated models for complex real-world applications.