Paper: LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents (ICLR-2024)

Reviewer comment (Novelty):
In the discussion section, include the following subsections `How does our work differ from LLaVA?` and `How does our work differ from ToolFormer?`

Author response:
Thank you for the suggestion to delineate how our work, LLaVA-Plus, differs from both LLaVA and ToolFormer. We have included a new section at the Sec. E in our paper to comprehensively address these comparisons.

*Comparison with LLaVA:*

- LLaVA primarily supports conversational interactions based on images. Its output, however, is limited to text.
- LLaVA-Plus extends the capabilities of LLaVA to include more diverse functionalities such as segmentation, editing, retrieval, and generation. This expansion is achieved by integrating external tools, thereby enhancing LLaVA's original text-only output to support a broader range of multimodal applications.

*Comparison with ToolFormer:*

- ToolFormer, in contrast, is designed primarily for text-based applications. It supports a range of text-oriented tools like question answering, Wikipedia search, calculators, calendars, and machine translation.

- Unlike LLaVA and LLaVA-Plus, ToolFormer is a pure language model and does not inherently support multimodal functionalities. Even with the integration of image-related tools, ToolFormer lacks the capability to process image inputs directly. This limitation means that ToolFormer cannot independently verify or interact with tool results that are based on visual content.

While LLaVA-Plus shares some foundational aspects with LLaVA and ToolFormer, it distinguishes itself through its enhanced multimodal capabilities and its ability to integrate and leverage external tools for a wider range of applications, including those involving direct image processing and interaction.

Reviewer comment (Presentation):
Figure 2 should be expanded into a bigger pipeline, which shows the different skills in the skill repository. It is hard to tell which part is the contribution of the paper, so emphasizing which modules are added by the paper will help highlight that.

Author response:
Thank you for your valuable suggestion to enhance the clarity and detail of Figure 2 in our paper. We have taken your feedback into account and made significant modifications to the figure.

We have expanded Figure 2 to include a more detailed depiction of the various skills within our skill repository. This expanded view provides a comprehensive understanding of how each skill is integrated into the overall pipeline of LLaVA-Plus.
The primary contributions of our paper are now clearly highlighted in shaded regions within the figure. These areas specifically illustrate the extensions and enhancements that our work, LLaVA-Plus, brings to the existing LLaVA model.

In LLaVA, the process was limited to Step 1 (instruction intake) and Step 4 (response generation). In contrast, LLaVA-Plus significantly expands this pipeline, introducing intermediate steps that enhance the model's ability to process and respond to inputs more effectively. These additions are clearly marked to showcase the advancements we have made.

We believe that these modifications will effectively demonstrate the enhancements introduced by LLaVA-Plus, making it easier for readers to identify and appreciate the novel aspects of our work. Thanks again for your valuable suggestions.

Reviewer comment (Presentation):
Adding a small introduction of the LLaVA paper would help the readers not go back and forth between the two papers to figure out the differences. The authors could consider adding that to a Preliminaries section.

Author response:
Thank you for your valuable suggestion to provide context for our work in relation to the LLaVA paper. In response, we have introduced a new section in our paper titled "Preliminaries: Visual Instruction Tuning in LLaVA" (Section 2.1). This section is dedicated to offering a concise overview of the LLaVA model, aiming to familiarize readers with its foundational concepts and functionalities.

Reviewer comment (Presentation):
Much of the meaningful technical information, such as substantive results, are limited to the appendix. This makes me question the soundness and what the contribution of the paper actually is. We don't even get to the related work until page 6 and so the experiment and results are squeezed at the very end.

Author response:
Thank you for your feedback regarding the organization and presentation of technical information in our paper. We have chosen to present our method before delving into related work, as this structure allows us to establish a clear foundation of our approach, which is crucial for understanding the subsequent discussions and comparisons.

We are aware of the upcoming expansion of page limitations to ten pages for the ICLR camera-ready version. We intend to utilize this additional space to expand on our experiments and results, ensuring that these critical aspects of our paper are given comprehensive coverage.

Reviewer comment (Writing):
The word "planning" is used but the paper doesn't really have anything to do with planning as commonly understood. There is no goal to be trained to achieve. The order of which API should be called first and what goes next shouldn’t be confused with planning approaches.

Author response:
Thanks for the question.

*Clarification on the Use of “Planning”*: In the context of our paper, “planning” refers to the process by which the model determines the most appropriate tool to use for a given task. This usage aligns with similar terminologies employed in projects like HuggingGPT and AutoGPT. To prevent any ambiguity and ensure clarity, we have added a footnote in our paper that explicitly defines our use of “planning” in this specific context.

*Usage of “Tools” in the Paper*: The term “tools” in our research is used to describe the APIs and systems that our model interfaces with to perform various tasks. This terminology is consistent with its usage in notable works such as ToolFormer and Visual ChatGPT. We appreciate your observation regarding the potential for misunderstanding and have added explanatory notes in the paper to clarify this terminology.

*Field of Submission and Representation Learning*: Regarding the field of submission for our paper, we carefully considered the categories listed on the ICLR website and selected "representation learning for computer vision, audio, language, and other modalities" as the most fitting for our work. This choice reflects the multimodal nature of our model and its applicability across various domains. However, we are open to suggestions and willing to reconsider the category if a more suitable one is recommended.

Reviewer comment (Writing):
The use of "tools" is misleading. This leads the reader (at least it led me) to think this was about robotic planning and object-affordance exploitation, but instead this is more about using the appropriate API call to different systems in an AutoGPT-style approach.

Author response:
Thank you for your insightful questions, which touch upon critical aspects of our approach: the use of Large Multimodal Models (LMM) for tool use and the decision to train the model for this purpose.

*LMM vs. LLM for Tool Use*: Our choice to utilize LMMs, as opposed to Language Large Models (LLMs), is grounded in the unique capability of LMMs to process image features. This ability allows for a more comprehensive understanding of visual information, which is crucial in our context. As evidenced in Table 4(a), our LMM-based model demonstrates superior performance compared to GPT4Tools, which employs LLMs. The visual comprehension ability of LMMs significantly aids in the correction and refinement of tool outputs, a key advantage detailed in Tables 11 and 12. This feature plays a vital role in mitigating errors and enhancing the reliability of our model.

*Training vs. Chaining for Tool Use*: Regarding the training methodology, we opted for a trained model approach over the more common chaining method. Chaining often leads to tool conflicts during inference, a problem that our trained model approach successfully mitigates. As demonstrated our paper, our model is adept at selecting and using the appropriate tools in most scenarios, thus reducing the likelihood of error propagation and compounding. 

Thanks again for your suggestions. We have added a new section to discuss the motivation and advantages of our design in Sec. E. A detailed discussion about of motivations is available in **Responses to Common Concerns**.

Reviewer comment (Novelty):
This was submitted to the "representation learning for computer vision, audio, language, and other modalities" but there's nothing about representation learning or conceptual understanding here. All the model does is call upon other unaltered submodels to perform vision-language tasks.

Author response:
Thanks for the question. We would like to show that LLaVA-Plus displays the capabilities to more complex tasks like segmentation, editing, and retrieval. We cite the Society of Mind approach as the high-level idea is shared. Our model aggregate multiple tool outputs, which is similar with the Society of Mind.

Reviewer comment (Experiments):
The approach seems not to be fully generalizable as there appears to be little error checking and validation of the outputs of the individual sub-models invoked, and therefore errors could propagate downstream and compound. The authors mention in the conclusion that LLaVA-Plus still suffers from hallucination and tool conflict. At the very least the tool conflict problem would need to be addressed to drive home the contribution, because otherwise such a plug-and-play architecture is unlikely to generate useful outputs more frequently than a manual chaining together of the individual sub-models.

Author response:
Thanks for your insightful question. Theoretically, we can do the error correction, since we use the LLaVA which is a Large Multimodal Model (LMM). This stems from its ability to process and interpret image inputs directly. The visual information provided by these inputs can play a crucial role in enabling the model to identify and rectify errors or inconsistencies in the outputs of sub-models. A practical instance of this error correction ability is demonstrated in Table 12, focusing on the Grounding DINO tool. We observed that the original Grounding DINO model occasionally produces false positive predictions. However, LLaVA-Plus has shown the capability to filter out these inaccurate predictions effectively. This results in enhanced accuracy and reliability, as the model leverages its multimodal understanding to validate and correct the tool outputs.

Reviewer comment (Evaluation):
"the combinations of these tools lead to emergent abilities that show signs of higher intelligence" - Are the authors simply mentioning the Society of Mind approach here, or do they claim that LLaVA-Plus also displays signs of higher intelligence? Extraordinary claims require extraordinary evidence.

Author response:
Thanks for the question. We would like to show that LLaVA-Plus displays the capabilities to more complex tasks like segmentation, editing, and retrieval. We cite the Society of Mind approach as the high-level idea is shared. Our model aggregate multiple tool outputs, which is similar with the Society of Mind.

Reviewer comment (Presentation):
A Full Dialogue of LLaVA-Plus (Figure 2) - This doesn’t seem like a full dialogue session. It’s a diagram. I would expect to see an actual example at each step along with a representation of the relevant information flow throughout the process.

Author response:
Thank you for your question regarding the depiction of a full dialogue session in LLaVA-Plus as illustrated in Figure 2.

*Clarifying the Dialogue Flow in Figure 2 and Table 1*: To provide a comprehensive example of a dialogue session, we have detailed an actual dialogue in Table 1, which correlates directly with the steps outlined in Figure 2. Here's how the dialogue in Table 1 aligns with the process steps in Figure 2:

  - Step 1 - Initial Query: The first human query presented in Table 1 corresponds to Step 1 in Figure 2. This step initiates the dialogue and sets the context for the interaction.
- Step 2 - Tool Invocation: Following the initial query, the assistant’s action of invoking a specific tool, as detailed in the second paragraph of Table 1, aligns with Step 2 in Figure 2. This demonstrates the assistant's decision-making process in selecting an appropriate tool for the query.
- Step 3 - Tool Output: The second human query in Table 1 represents the tool output, correlating with Step 3 in Figure 2. This step showcases the output provided by the invoked tool in response to the initial query.
- Step 4 - Final Response: Lastly, the concluding response from the assistant, as depicted in the final paragraph of Table 1, corresponds to Step 4 in Figure 2. This step illustrates the assistant’s final response to the human query after processing the tool output.

The dialogue example in Table 1 is designed to provide a real-world representation of the information flow and interaction process depicted abstractly in Figure 2. This example aims to clarify the practical application and functionality of LLaVA-Plus in a full dialogue scenario.

Reviewer comment (Evaluation):
Opensource instruction tuning dataset is one the contributions of this work. However, since the dataset itself was not human evaluated and GPT generated, it is unclear how much hallucination does it contain.

Author response:
**Response***: Thank you for highlighting the importance of data quality, especially in the context of our opensource instruction tuning dataset. We recognize the potential for hallucination in datasets generated by models like GPT, and have taken several measures to ensure the integrity and usefulness of our data.


_Removal of Apologetic Responses_: We have filtered out responses containing phrases like “I’m sorry”, which often indicate uncertainty or errors in model-generated content.

_Exclusion of Language Model Content_: Entries containing references to “AI language model” or “language AI model” have been removed to maintain focus on instructional content rather than meta-discussions about the AI itself.

_Validation of CLIP-Retrieval Data_: For data pertaining to CLIP-Retrieval, we ensured that only items containing correct answers, as verified against ground-truth labels, were retained. Our selection criteria involve keeping items that match any of the listed ground truths.

_Accuracy in Object Detection and Segmentation Data_: We have excluded data where the provided answers contain bounding boxes that do not align with the established ground truths, ensuring accuracy in object detection and segmentation tasks.

Thanks for the suggestions. We have added a new paragraph in Sec. A of the Appendix to discuss the filter rules. Acknowledging your valuable suggestion, we have added a paragraph at the end of the main text of the revised version of our paper to discuss these limitations.

Reviewer comment (Experiments):
It would also be useful to provide dataset stats (# of instructions, # tools, # instructions/tool etc.) for completeness and perhaps human evaluation for a subset.

Author response:
**Response**: Thank you for your valuable suggestion regarding the provision of dataset statistics. To address this, we have included comprehensive details about our dataset in Table 2 of our paper. This table presents a thorough breakdown of the dataset, including the number of instructions and the distribution of these instructions across the different tools we utilized. Regarding the suggestion for human evaluation of a subset of the dataset, we acknowledge its importance in validating the quality and relevance of the data. While this was not included in the current scope of our study, it is certainly a valuable consideration for future work to further establish the dataset's reliability and applicability.

Reviewer comment (Reproducibility):
It is unclear whether addition of a new tool would require doing the instruction tuning from scratch or can the model be fine-tuned for just those new instructions.

Author response:
**Response**: Thank you for your insightful question regarding the incorporation of new tools or skills into our model. We conducted an experiment to understand our model's incremental learning capabilities.

*Impact of Fine-tuning on Specific Tools*: When fine-tuning LLaVA-Plus on a particular tool, we observed that while the model's performance in that specific area improves, it adversely affects its abilities in handling other tools. For instance, after fine-tuning solely on grounding data, the model's proficiency in tagging, captioning, and OCR significantly diminished. This is evident from the following results:

| **model** | **grounding** | **tagging** | **caption** | **OCR** |
| --- | --- | --- | --- | --- |
| **LLaVA-Plus (4 tools)** | 0.77 | 1.00  | 0.83  | 1.00  |
| **LLaVA-Plus (4 tools -> grounding)** | 0.93  | 0.02  | 0.36  | 0.00  |

*Distinguishing Between Skills and Tools*: It's crucial to differentiate between 'skills' and 'tools' in our model's context. Skills represent the model's capabilities and may require a complete retraining process for effective integration of new skills. Conversely, the model is more adaptable when it comes to incorporating upgraded tools. For instance, replacing an existing tool with its advanced version (e.g., upgrading from CLIP-B to CLIP-L) can be done seamlessly without necessitating a full retraining of the model.

Therefore, if the goal is to add a new skill, a comprehensive fine-tuning across all tools would be necessary to maintain the model's overall proficiency. On the other hand, upgrading tools can be achieved with minimal impact on the model’s existing capabilities.

Reviewer comment (Reproducibility):
It is also unclear how the assistant deals with contradicting grounding information from two tools. For instance, if detection and segmentation tools disagree about the presence of a particular object, what does the assistant do?

Author response:
**Response**:
- Thank you for your insightful questions regarding our approach to question rewriting and conflict resolution in tool use.
In our model, each skill is associated with a specific tool, effectively eliminating the possibility of tool conflicts within the same skill. However, conflicts can arise when dealing with different skills due to the inherent ambiguities in human instructions. For instance, a grounding task might yield varying outputs like a simple box or a box with an accompanying mask. To mitigate such conflicts, our data processing pipeline focuses on two main strategies:
we try to avoid ambiguous questions that could leads to conflicts when constructing instruction data. There are two approachs we used during our data processing pipeline. 
1. **Diversifying Questions**: We intentionally diversify the questions in our dataset to reduce ambiguities and improve tool use accuracy. This approach helps in clearly defining the task for each tool, thereby reducing the likelihood of conflicting interpretations.
2. **Quality Control in Data Generation**: Our data is primarily generated using GPT4V, which, while powerful, can produce hallucinations or undesirable data items. To ensure data quality, we apply heuristic methods to filter out low-quality data:
   1. Removal of Apologetic: We have filtered out responses containing phrases like
“I’m sorry”, which often indicate uncertainty or errors in model-generated content.
   2. Exclusion of Language Model Content: Entries containing references to “AI language model”
or “language AI model” have been removed since we plan to train a multimodal model
rather than a pure language model.
   3. Validation of CLIP-Retrieval Data: For data pertaining to CLIP-Retrieval, we ensured that
only items containing correct answers, as verified against ground-truth labels, were retained.
Our selection criteria involve keeping items that match any of the listed ground truths.
   4. Accuracy in Object Detection and Segmentation Data: We have excluded data where the
provided answers contain bounding boxes that do not align with the established ground
truths, ensuring accuracy in object detection and segmentation tasks.

By implementing these measures, we aim to minimize conflicts and ambiguities, ensuring that the model produces reliable and accurate outputs in line with the intended instructions.

Reviewer comment (Presentation):
Table 5: What does “all tools + GPT4” mean? Did the authors mean GPT4Tools?

Author response:
**Response**: Thanks for the question, which help clarify important aspects of our research. In Table 5, “all tools + GPT4” refers to a setup where we integrate the outputs from all the selected tools and then feed this combined information to GPT4. The purpose of this process is for GPT4 to aggregate the diverse inputs and synthesize an answer. This approach is part of an oracle experiment designed to demonstrate the potential effectiveness of tool use in enhancing the performance of a language model like GPT4 in processing and synthesizing multimodal information.

Reviewer comment (Experiments):
Table 4,5: Would have been great to see performances of at least some non-tool based but multimodal models’ performances on Llava-bench just to see how they compare with tool-based approaches on this benchmark.

Author response:
**Response**: Regarding your second query, we have included performance data for the plain LLaVA model in Tables 4 and 5. The plain LLaVA, which is trained on visual instruction data without the integration of any additional tools, serves as a baseline for comparison. This model represents a non-tool based, multimodal approach. By contrasting its performance with that of our tool-enhanced models, we aim to showcase the relative effectiveness and advantages of incorporating tools in multimodal modeling. These comparisons provide valuable insights into how tool-based approaches can enhance the capabilities of multimodal models in processing complex, multimodal datasets.

Reviewer comment (Experiments):
Is it also possible to provide comparisons with GPT4-v (using visual inputs) for all benchmarks?

Author response:
**Response**: Thanks for your question. In response to your question, we have indeed conducted a comparative analysis of our model with GPT4-v on the LLaVA-Bench, as shown in the Table below. These results indicate that GPT4-v performs significantly better than both LLaVA and LLaVA-Plus, even though LLaVA-Plus has been augmented with additional tools.

| Model | Conv | Detail | Reasoning | ALL |
| --- | --- | --- | --- | --- |
| LLaVA | 82.0 | 69.1 | 92.6 | 81.2 |
| LLaVA-Plus | 81.6 | 74.5 | 95.7 | 83.9 |
| GPT4V | 67.3  | 104.3 | 108.4 | 93.3 |

Table R2.1 Results on LLaVA-Bench

Regarding testing on other benchmarks like the SEED-Bench, which contains over 10,000 test examples, we acknowledge the potential value of this comparison. However, conducting such extensive tests with GPT4-v is resource-intensive. We will consider testing on GPT4-v for these benchmarks if the API becomes more accessible and cost-effective.

Reviewer comment (Presentation):
Table 7: The authors should add model sizes for all models if possible.

Author response:
**Response**: Thank you for your valuable suggestion regarding the inclusion of model sizes in our analysis. We have updated Table 7 in our paper. A separate column has now been added to display the size of each model, to the best of our knowledge and available information.

Reviewer comment (Reproducibility):
Instruction tuning training details are missing from the paper and the appendix.

Author response:
**Response**: Thank you for pointing out the need for more detailed information regarding instruction tuning training in our study. To address this, we have introduced a dedicated section in our paper, specifically in Section 2.1, where we comprehensively describe the visual instruction tuning process utilized in the LLaVA model. In this section, we elaborate on the format and structure of the LLaVA data, which is a crucial component of our instruction tuning methodology.

Reviewer comment (Evaluation):
Would be great to see some failure cases in the paper/appendix to understand the limits of the agent's reasoning capabilities.

Author response:
**Response**: Thanks for the valuable question. We have added some failure cases in Figure 16 and Figure 17 to show the limits of the agent’s reasoning capabilities. In Figure 16, the model demonstrates a limitation in comprehending the user’s intent. The task involved detecting an object likely to contain a cold beverage, yet the model defaulted to a generic caption “correspond object” for Grounding DINO, failing to specify the object in question. This particular example is adapted from DetGPT.
In the case depicted in Figure 17, the model struggles with understanding the spatial relationship between two detected objects. Intriguingly, the model's performance in this regard is inconsistent; at times, it successfully discerns and responds accurately to the spatial query, but it also exhibits instances of failure. This variability underscores a need for further refinement in the model's spatial reasoning capabilities. Thanks again for your suggestions, which make our paper more comprehensive.

Reviewer comment (Novelty):
Lack of flexibility. If I understand correctly, once new tools/skills are added, the models must be retrained on the augmented dataset to master this new tool. Is this correct? If this is the case, the overall approach can only teach the model to learn to use a fixed set of tools, instead of "learning to use tools" and "becoming a good agent that can effectively invoke external tools provided in context".

Author response:
**Response:** Thank you for your valuable feedback regarding the flexibility of our model in learning and utilizing new tools and skills.

*Clarifying Skills vs. Tools*: It's important to distinguish between 'skills' and 'tools' within our model's framework. Skills are the model's capabilities, which can be achieved using various tools. When a new skill is required, it is true that the model needs to be retrained to effectively integrate and master this new skill. However, when it comes to replacing or upgrading tools, LLaVA-Plus exhibits greater flexibility. For example, if a more advanced version of a tool (like upgrading from CLIP-B to CLIP-L or switching from EasyOCR to Google OCR API) becomes available, we can seamlessly integrate this new tool into the model without the need for retraining. This approach ensures continuous improvement in the model's performance with minimal retraining requirements. 

*Flexibility vs. Chain Methods*: We acknowledge that our approach might not be as inherently flexible as chain methods in terms of tool use. However, the chain method comes with its own set of challenges, such as tool conflicts during inference. LLaVA-Plus is designed to avoid such conflicts, thereby ensuring more stable and reliable performance. 

*Open-Source LMM Initiative*: LLaVA-Plus is developed as an open-source LMM to further easily enhance the model capabilities. Tool chaining methods frequently require extensive context to describe tools, reducing overall model performance, especially when dealing with a large number of tools. Our framework is designed to bypass these issues, providing a more efficient and effective solution for integrating and utilizing a large tool set. 

We appreciate your insights and hope this clarification highlights the strengths and future potential of our approach.

Reviewer comment (Experiments):
Lack of ablation study and in-depth analysis. This paper doesn't show concrete evidence for the benefits of tool use. Only the absolute performance is reported. I would suggest reporting the tool usage rate and also conducting an ablation study to investigate what is the benefit coming from, the tool using ability, or just adding more tool data during the instruction fine-tuning stage.

Author response:
**Response**: Thank you for your constructive feedback on our paper. In response to your suggestions, we have expanded our analysis to include more concrete evidence demonstrating the benefits of tool integration in our model.

*Reporting Tool Usage Rate*: To address your point about reporting the tool usage rate, we have added new experimental data in Table 13 of Section C in the Appendix of our revised paper. This table provides a detailed breakdown of the tool usage rates, offering a clearer insight into how each tool contributes to the overall performance of our model.

*Examples of Tool Integration Benefits*: We have also included several examples in our paper to illustrate the advantages of integrating tools into the LLaVA-Plus model. A notable example is depicted in Figure 6, where LLaVA-Plus demonstrates the capability to detect and segment objects, a functionality that the original LLaVA model lacks.

*Expanding on New Scenarios*: In addition to these examples, we have dedicated Section D of our paper to present a variety of new scenarios where the tool-enhanced model excels. These scenarios include diverse applications such as meme understanding, image generation, image editing, interactive segmentation, and analyzing social media posts. These examples serve to showcase the versatility and improved performance of our model in handling complex, real-world tasks. Through these additional experiments and expanded discussions, we aim to provide a comprehensive understanding of how tool integration not only enhances the model's existing capabilities but also enables it to tackle new and diverse challenges effectively.