Paper: LMEye: An Interactive Perception Network for Large Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
The experimental part should offer a concise description of the structure of LMEye (BLIP-2).

Author response:
LMEye (BLIP-2-FlanT5-XL) is a derivative of the FlanT5 large language model, operating on an encoder-decoder architecture. It employs the pre-trained vision encoder along with the Q-former to serve as the image encoder, translating image representations into the language space. Specifically, we utilize the T5-Encoder for image comprehension and request processing, employing a probe vector labeled as '<img-q>' to prompt signals essential for the large model's visual information acquisition. Subsequently, we leverage the RVII mechanism to gather visual information relevant to the aforementioned model requests. This process enriches the original image data with dynamically enhanced visual information. The amalgamation of this enhanced visual data and the original image information is then fed into the T5 model.

Reviewer comment (Evaluation):
Whether the inference process will consume some time for different types of models, a brief inference efficiency description is given for the design of the LMEye variant for the encoder-decoder and the decoder-only models.

Author response:
The distinction between the LMEye variants in encoder-decoder and decoder-only language models lies in their operational mechanisms. Encoder-decoder LLMs leverage the encoder specifically to grasp input information, enhancing model inference efficiency by focusing on relevant data. On the other hand, decoder-only LLMs utilize the entire language model to accommodate evolving visual input and human queries. However, our recent advancements have led to an enhanced variant that integrates the foundational layer of decoder-only LLMs for comprehending both image and query data. Moreover, we've implemented the RVII (small size) module across the upper layers of the language model. This strategic integration empowers the language model to continually assimilate dynamically enriched visual information as it delves deeper, akin to imbuing deep thinking with a visual dimension. Consequently, this visual data becomes an integral component of the calculation process within LLMs, augmenting their cognitive capabilities.

Reviewer comment (Experiments):
Would this approach expand to video understanding and provide more detailed description of video and action inference?

Author response:
This question delves into a fascinating area of exploration. Extensive experimentation has convincingly showcased the remarkable scalability of LMEye across various large language models, hinting at its potential extension into the realm of video processing. The approach involves training a dedicated video encoder and converter, leveraging RVII's ability to engage with prolonged encoded signals. Our methodology encompasses aggregating encoding information from brief video frames and subjecting it to rigorous benchmark testing for video action and procedural comprehension.
| Models             | Action Prediction | Procedure Understanding |
|--------------------|-------------------|-------------------------|
| VPGTrans           | 39.54             | 31.88                   |
| LLaMA-Adapter V2   | 38.56             | 19.63                   |
| MiniGPT-4          | 38.22             | 27.1                    |
| InstructBLIP Vicuna| 34.48             | 23.07                   |
| InstructBLIP       | 33.10             | 27.1                    |
| BLIP2              | 32.59             | 23.99                   |
| LLaVa              | 29.71             | 19.13                   |
| mPLUG-Owl          | 26.72             | 26.51                   |
| Video-ChatGPT      | 27.59             | 21.14                   |
| LMEye(ours)        | **46.55**             | **35.15**                   |

The outcomes of our experiments substantiate the feasibility of extending LMEye's capabilities to comprehend video content. Notably, our findings highlight not only the successful extension of this model but also its commendable performance compared to Video-related MLMs. This progress indicates promising avenues for further advancements and applications in this evolving field.

Reviewer comment (Experiments):
To alleviate the Hallucination problem shown in Appendix, could we introduce visual information request during text generation, like ReAct?

Author response:
We are actively considering the integration of visual information retrieval mechanisms into our text generation process. Our current focus is on the development of specific tokens dedicated to retrieving pertinent visual data during generation. This approach mirrors our successful implementation of Tool API tokens and aligns with our ongoing efforts to enrich the generation process with relevant external information sources. Your feedback has guided us towards this promising avenue, and we are committed to exploring and implementing this enhancement in our future work.

Reviewer comment (Experiments):
As the core contribution of the paper, the authors did not carefully explore the contribution of RVII module. For example, the impact of RVII module under the same data and training process.

Author response:
The RVII module is designed to enhance the model's ability to dynamically interact with visual information based on human queries. To evaluate its contribution, we refer to Tables 3 and 4, where we compare the performance of models with and without the RVII module (notably LMEye (xx)*). These tables show that incorporating RVII results in significant improvements for several LLMs in tasks like VQA/OK-VQA/VQA with long answers. Moreover, when comparing LMEye (FlanT5-BLIP-2-XL) with Instructblip (FlanT5-BLIP-2-XL), both of which utilize similar multimodal instruction data, the key difference lies in the inclusion of the RVII structure in our model. This comparison, as shown in the results on evaluation benchmarks, also underscores the effectiveness of the RVII module.

To further explore the contribution of RVII module, we conducted an additional experiment. Here, we fine-tuned LMEye (FlanT5-BLIP-2-XL)* using identical instruction data but without the RVII module. The results of this experiment, detailed in the following table, clearly demonstrate the significant contribution of the RVII module to the model's performance.
| Models(FlanT5-XL) | MMBench.Test | SEEDBench | VCR  | VQA  | OK-VQA |
|-------------------|--------------|-----------|------|------|--------|
| BLIP-2            | -            | 49.74     | 57.3 | 59.95| 48.91  |
| InstructBLIP      | 33.9         | 57.8      | 53.55| 62.56| 50.12  |
| LMEye(ours) w/o RVII | 47.79   | 54.84     | 53.51| 56.72| 49.6   |
| LMEye(ours)       | **62.6**         | **59.7**      | **57.5** | **63.2** | **54**     |

Reviewer comment (Reproducibility):
What are the training parameters in the instruction tuning phase?

Author response:
During the instruction fine-tuning stage, our model utilizes the RVII system along with several linear feature projection layers, which collectively amount to approximately 105 million parameters, with RVII alone comprising about 100 million. Recent findings indicate that for image understanding tasks, it's possible to reduce the number of RVII layers without noticeably affecting performance. By halving the number of RVII layers, we can significantly decrease the total count of trainable parameters, thereby streamlining the model while maintaining its effectiveness.

Reviewer comment (Evaluation):
Since the visual information needs to be fed into LLM twice, will the model run slower?

Author response:
Thank you for raising this point. Intuitively, feeding visual information into the LLM twice suggest a slower process. In practice, the impact on processing speed is minimal. This is due to the efficient design of our model which optimizes the integration of visual data. The length of visual tokens is set to 32 and we only input one <img-q> token as the probing token to obtain the request signal. For encoder-decoder T5 model, we only use the encoder as the the first-stage basic perception and request signal generation. Time is mainly spent in the token generation phase. Moreover, the additional time taken for processing is offset by the significant improvements in accuracy and context relevance of the output. In summary, while there is a slight increase in processing time, it is negligible compared to the enhanced performance and quality of results achieved by our model and we will contiguously optimize the efficiency according to your question.

Reviewer comment (Novelty):
The so called "interactive perception model" is actually a standard and common technique used everywhere. Early since attention was proposed, the visual information is dynamically summarized (attended). Some old works on VQA, such as NMN, have extensively used instruction-based condition to process visual information.

Author response:
While the interactive perception is a concept that has been used in various research domains, our implementation within LMEye exhibits several distinctive and innovative aspects, particularly in its integration with Large Language Models (LLMs). 

**LLM-Centric Approach**: LMEye is uniquely engineered to function synergistically with LLMs. Here, the LLM serves as a central processor, orchestrating a dynamic and efficient interaction with the external visual environment. This is a significant deviation from previous models, which predominantly focused on static visual summarization.

**Dynamic Interaction**: Unlike the methods you mentioned, such as NMN in VQA, LMEye empowers LLMs to actively initiate, gather, and process visual information in real-time, in response to ongoing human instructions. This approach aims to advance the frontiers of AI-Generated Content (AIGC) research by fostering interactions that are more aligned with human intentions and responsive to changes in the environment.

**Performance and Efficiency**: Our comprehensive experimental evaluations demonstrate that LMEye not only significantly enhances zero-shot performance across various multimodal benchmarks but also accomplishes this with fewer parameters than other advanced multimodal large models. This indicates a substantial improvement in both practical effectiveness and efficiency, setting LMEye apart from existing methodologies.

Reviewer comment (Presentation):
Figure 1 could be improved a lot. First, I will suggest including and specifying each component of LMEye. Adding this will help the reader to connect the notation mentioned in Section 3.1 and the flow of the figure.

Author response:
We appreciate your feedback on Figure 1 and acknowledge the need for its enhancement. To address this, we plan to revise Figure 1 to more clearly delineate and label each component of the LMEye system, ensuring that it accurately reflects the architecture and functionality discussed in the text. This revision will aim to enhance the figure’s clarity and its utility as a visual aid for readers. Additionally, we recognize the importance of clear and correct writing for effective communication of our research. To this end, we will conduct a thorough review and revision of the manuscript to correct grammatical errors, including the specific instance you mentioned on page 2.

Reviewer comment (Experiments):
It would be beneficial to understand the contribution of the RVII module if authors include ablations with and without that module in MME and SEED-Bench benchmarks.

Author response:
In our prior paper, we highlighted the efficacy of the RVII module through a comparative analysis involving LMEye (BLIP), InstructBLIP, and BLIP, based on FlanT5-XL. InstructBLIP, by integrating multimodal instruction data to fine-tune the Q-former from BLIP, demonstrated superior performance across various tasks compared to BLIP itself. It's essential to note that the structural composition of InstructBLIP and BLIP remains unchanged. However, LMEye (BLIP) introduces a distinct component known as the Request-based dynamic visual information interactive module (RVII). While trained with instruction data akin to InstructBLIP, LMEye (BLIP) without RVII essentially aligns with the architecture of either BLIP or InstructBLIP. By juxtaposing the performances of these variations in Tables 1, 2, and 3, we can ascertain the specific impact of the RVII module.

To address your constructive feedback, we conducted experiments by training LMEye (BLIP) without the RVII module using our code and instruction data. The subsequent experimental results, outlined in the following table, present  the significance and role of the RVII module in enhancing performance. All models are based on the same vision and language models (FlanT5-XL).
| Models(FlanT5-XL) | MMBench.Test | SEEDBench | VCR  | VQA  | OK-VQA |
|-------------------|--------------|-----------|------|------|--------|
| BLIP-2            | -            | 49.74     | 57.3 | 59.95| 48.91  |
| InstructBLIP      | 33.9         | 57.8      | 53.55| 62.56| 50.12  |
| LMEye(ours) w/o RVII | 47.79   | 54.84     | 53.51| 56.72| 49.6   |
| LMEye(ours)       | **62.6**         | **59.7**      | **57.5** | **63.2** | **54**     |

Reviewer comment (Reproducibility):
I also suggest specifying the Vision Model and Language Model with their corresponding parameters to have a better and fairer comparison while reading Tables 1 and 2.

Author response:
Thank you for your valuable suggestion. We have collected relevant data as shown in the table below. We will also update specific parameter statistics in the next version.
| Method           | Language model | Vision model        |
|------------------|----------------|---------------------|
| MiniGPT-4        | Vicuna 7B      | EVA-G 1.1B         |
| MMGPT            | LLaMA 7B       | CLIP VIT-L/14 428M |
| PandaGPT         | Vicuna 13B     | ImageBind ViT-H/14 1.2B |
| VisualGLM        | ChatGLM 6B     | EVA-CLIP 1.1B      |
| InstructBLIP     | Vicuna 7B       | EVA-G 1.1B         |
| LLaVA            | LLaMA 7B       | CLIP VIT-L/14 428M |
| LLaMA-Adapter-v2 | LLaMA 7B       | CLIP VIT-L/14 428M |
| G2PT             | LLaMA 7B       | ViT-G 1.1B          |
| mPLUG-Owl        | LLaMA 7B       | CLIP VIT-L/14 428M |
| Ottar-I          | LLaMA 7B       | CLIP VIT-L/14 428M |
| Shikra           | Vicuna 7B      | CLIP VIT-L/14 428M |
| Qwen-VL           | Qwen 7B      | ViT-G/16 1.1B |
| LMEye            | FlanT5-xl 3B   | ViT-G 1.1B     |

Reviewer comment (Experiments):
Add ablations without RVII module in LLMEye (BLIP2) in Tables 3 & 4.

Author response:
In our prior paper, we highlighted the efficacy of the RVII module through a comparative analysis involving LMEye (BLIP), InstructBLIP, and BLIP, based on FlanT5-XL. InstructBLIP, by integrating multimodal instruction data to fine-tune the Q-former from BLIP, demonstrated superior performance across various tasks compared to BLIP itself. It's essential to note that the structural composition of InstructBLIP and BLIP remains unchanged. However, LMEye (BLIP) introduces a distinct component known as the Request-based dynamic visual information interactive module (RVII). While trained with instruction data akin to InstructBLIP, LMEye (BLIP) without RVII essentially aligns with the architecture of either BLIP or InstructBLIP. By juxtaposing the performances of these variations in Tables 1, 2, and 3, we can ascertain the specific impact of the RVII module.

To address your constructive feedback, we conducted experiments by training LMEye (BLIP) without the RVII module using our code and instruction data. The subsequent experimental results, outlined in the following table, present  the significance and role of the RVII module in enhancing performance. All models are based on the same vision and language models (FlanT5-XL).
| Models(FlanT5-XL) | MMBench.Test | SEEDBench | VCR  | VQA  | OK-VQA |
|-------------------|--------------|-----------|------|------|--------|
| BLIP-2            | -            | 49.74     | 57.3 | 59.95| 48.91  |
| InstructBLIP      | 33.9         | 57.8      | 53.55| 62.56| 50.12  |
| LMEye(ours) w/o RVII | 47.79   | 54.84     | 53.51| 56.72| 49.6   |
| LMEye(ours)       | **62.6**         | **59.7**      | **57.5** | **63.2** | **54**     |

Reviewer comment (Experiments):
Why not include the Qwen-VL model? I believe it was already published at the moment of the paper submission.

Author response:
Thank you for highlighting the significance of the Qwen-VL model. We understand its recent release and the advancements it brings, particularly through its multi-stage training methods. Initially, our study focused on established models available at the outset of our research. However, we acknowledge the importance of considering Qwen-VL within the context of our work.

In response to your insightful suggestion, we have conducted a comparative performance evaluation in the MMBench and SEED-Bench benchmarks. Our findings reveal noteworthy performance differences: MMBench-Test: 61.8 (Qwen-VL) vs. 62.6; MMBench-Dev: 60.6 (Qwen-VL) vs. 65.5; SEEDBench-Img: 62.3 (Qwen-VL) vs. 59.7. Particularly in SEED-Img, Qwen-VL exhibits advantages over many existing large models, including our own. Moreover, a thorough examination of Qwen-VL's technical report has provided us with advanced insights and methodologies. We intend to integrate these into our variant model, LMEye (Qwen), leveraging Qwen-VL's architecture and training strategies. We believe this integration will significantly augment our research, enhancing the capabilities of our model and contributing substantially to its overall performance. In the revised version of our paper, we will include a detailed analysis and discussion of Qwen-VL's performance, showcasing the potential impact on the research landscape.

Reviewer comment (Writing):
"... which shows supervisor performances on various multimodal scenarios." (Page 2) — Should it be "superior"?

Author response:
We appreciate your feedback on Figure 1 and acknowledge the need for its enhancement. To address this, we plan to revise Figure 1 to more clearly delineate and label each component of the LMEye system, ensuring that it accurately reflects the architecture and functionality discussed in the text. This revision will aim to enhance the figure’s clarity and its utility as a visual aid for readers. Additionally, we recognize the importance of clear and correct writing for effective communication of our research. To this end, we will conduct a thorough review and revision of the manuscript to correct grammatical errors, including the specific instance you mentioned on page 2.