Paper: Voila-A: Aligning Vision-Language Models with User's Gaze Attention (ICLR-2024)

Reviewer comment (Presentation):
The paper's presentation is lacking. Many important sections have been relegated to the appendix, especially the technical details. For example, Section 4.1 is challenging to understand due to the limited text.

Author response:
**Presentation and Technical Details:** 

We recognize the need to enhance our presentation and include crucial information in the main paper by providing detailed explanations where needed. To address your concerns, we have made significant improvements to our manuscript, including extensive reorganization, clarification, and rewriting as we replied in **General Response**.

Reviewer comment (Novelty):
The model heavily depends on the baseline model Otter. The method of injecting gaze information is quite straightforward. The way in which the authors handle catastrophic forgetting can be observed in the literature, thus not introducing technical novelty.

Author response:
**Baseline Selection and Technical Novelty**

We would like to begin by clarifying the rationale behind our choice of Kosmos-2 and Otter as the two baselines for comparison. Our method aims to demonstrate improvements in two key areas: state-of-the-art multimodal conversation capabilities and grounding ability with additional modality input.

First, Otter is based on the Flamingo architecture, a well-recognized milestone in vision-language research. Otter has been fine-tuned on egocentric data (e.g., Ego4D) and instructional data (e.g., COCO), making it more suitable for the scenarios we are targeting compared to other similar VLMs. Second, Kosmos-2, at the time of submission, is a widely known and comprehensively evaluated model with superior grounding capabilities. Since our model has not been pre-trained on massive grounding data like Kosmos-2, comparing our grounding capability to Kosmos-2 presents a significant challenge, requiring us to explore efficient ways to guide VLMs in learning grounding.

Regarding the reliability of our chosen baselines, both Kosmos-2 and Otter are cited extensively (72 and 110 citations, respectively), open-sourced, and actively maintained. We believe that the research community widely recognizes and reviews these models.

In terms of fairness in comparison to the baselines, for grounding, we contend that our method performs on par with, if not surpassing, Kosmos-2, which also has an additional modality indicating user focus and pre-trained on a large grounding dataset. For Otter, our method not only demonstrates significant grounding improvements but also exhibits better helpfulness. The quality of our coreference question responses is comparable to the direct question responses of Otter's base model, which we believe is a fair comparison.

As for technical novelty and contribution, we argue that our model design is far from trivial. Numerous approaches exist for incorporating gaze data, and our final choice was made after conducting a series of experiments detailed in the ablation study. To further elaborate on these aspects, we have included a detailed illustration of different gaze integration methods in Figure 13. Additionally, our work extends beyond model design, as we strive to establish a method for a new problem. The data annotation pipeline and gaze benchmark construction have not been explored by other works, making it difficult to claim that our work lacks novelty.

Reviewer comment (Experiments):
The experimental analysis appears somewhat unfair because the proposed method uses additional modalities to achieve the same results. Therefore, its better performance is not surprising, particularly in cases where the query does not clearly define the object's name, and several other objects are present in the scene, with the gaze heatmap aligning with the queried object. It is also worth mentioning that both baseline methods are still only in ArXiv.

Author response:
In terms of fairness in comparison to the baselines, for grounding, we contend that our method performs on par with, if not surpassing, Kosmos-2, which also has an additional modality indicating user focus and pre-trained on a large grounding dataset. For Otter, our method not only demonstrates significant grounding improvements but also exhibits better helpfulness. The quality of our coreference question responses is comparable to the direct question responses of Otter's base model, which we believe is a fair comparison.

Reviewer comment (Evaluation):
There is uncertainty about the cases in which gaze information was found to be less relevant.

Author response:
**When Gaze is Less Relevant**

You raise an important point regarding the impact of gaze relevance on user experience in real-life scenarios. We would like to address this in two parts. First, when a user's query is clear enough, we observe that models tend to prioritize language instructions in most cases. This can be seen in our ablation studies on coreference and direct query ablations. This behavior may be attributed to the presence of noise in the training data, where gaze traces are not correctly aligned, causing the model to learn robustness in such cases. 

Second, when a user's query is unclear, the model may attempt to connect the irrelevant gaze with the query, potentially leading to inaccurate responses. We acknowledge that our current observations are limited to qualitative cases, and we plan to evaluate this aspect quantitatively as we refine and iterate on our method in the future.

Reviewer comment (Experiments):
Gaze data collection procedure is also very scarce and it is not possible to understand if the annotators have a reliable consensus to use the collected data in model evaluation and comparisons.

Author response:
**Gaze Data Collection Procedure and Sample Size**

In the revised version, we have expanded upon the gaze data collection procedure and provided further information. We appreciate your concerns regarding potential bias in the dataset and the sufficiency of the sample size, and we are more than willing to engage in a discussion on these matters:

1. We acknowledge that bias in any test set is inevitable when the sample size is limited. However, the primary concern should be whether the potential bias significantly impacts the purpose the set is intended to serve.
2. The main motivation for collecting VOILA-GAZE is to evaluate the adaptability of trace-trained VLMs to gaze-directed downstream use cases without a substantial decline in performance.
3. While our dataset's primary limitation is the confined range of scenarios, which may not fully assess a large VLM's knowledge and capabilities, we have made an effort to include a diverse group of participants in terms of gender, age, major, and other factors to address gaze behavior.
4. Based on both quantitative and qualitative experiments, we are confident that the improvements brought about by our pipeline design are both significant and reliable.
5. We are actively working to expand and enhance this benchmark by incorporating a broader range of application scenarios, which will further strengthen the validity of our findings.

We hope this clarification addresses your concerns, and we look forward to any further discussion on this topic.

Reviewer comment (Experiments):
It is doubtful whether 100 gaze samples are sufficient for conducting a comprehensive comparative study. I have reservations about the potential bias in the collected dataset.

Author response:
**Gaze Data Collection Procedure and Sample Size**

In the revised version, we have expanded upon the gaze data collection procedure and provided further information. We appreciate your concerns regarding potential bias in the dataset and the sufficiency of the sample size, and we are more than willing to engage in a discussion on these matters:

1. We acknowledge that bias in any test set is inevitable when the sample size is limited. However, the primary concern should be whether the potential bias significantly impacts the purpose the set is intended to serve.
2. The main motivation for collecting VOILA-GAZE is to evaluate the adaptability of trace-trained VLMs to gaze-directed downstream use cases without a substantial decline in performance.
3. While our dataset's primary limitation is the confined range of scenarios, which may not fully assess a large VLM's knowledge and capabilities, we have made an effort to include a diverse group of participants in terms of gender, age, major, and other factors to address gaze behavior.
4. Based on both quantitative and qualitative experiments, we are confident that the improvements brought about by our pipeline design are both significant and reliable.
5. We are actively working to expand and enhance this benchmark by incorporating a broader range of application scenarios, which will further strengthen the validity of our findings.

We hope this clarification addresses your concerns, and we look forward to any further discussion on this topic.

Reviewer comment (Novelty):
Please comment on Weakness (2) for the technical novelty of the method.

Author response:
**Baseline Selection and Technical Novelty**

We would like to begin by clarifying the rationale behind our choice of Kosmos-2 and Otter as the two baselines for comparison. Our method aims to demonstrate improvements in two key areas: state-of-the-art multimodal conversation capabilities and grounding ability with additional modality input.

First, Otter is based on the Flamingo architecture, a well-recognized milestone in vision-language research. Otter has been fine-tuned on egocentric data (e.g., Ego4D) and instructional data (e.g., COCO), making it more suitable for the scenarios we are targeting compared to other similar VLMs. Second, Kosmos-2, at the time of submission, is a widely known and comprehensively evaluated model with superior grounding capabilities. Since our model has not been pre-trained on massive grounding data like Kosmos-2, comparing our grounding capability to Kosmos-2 presents a significant challenge, requiring us to explore efficient ways to guide VLMs in learning grounding.

Regarding the reliability of our chosen baselines, both Kosmos-2 and Otter are cited extensively (72 and 110 citations, respectively), open-sourced, and actively maintained. We believe that the research community widely recognizes and reviews these models.

In terms of fairness in comparison to the baselines, for grounding, we contend that our method performs on par with, if not surpassing, Kosmos-2, which also has an additional modality indicating user focus and pre-trained on a large grounding dataset. For Otter, our method not only demonstrates significant grounding improvements but also exhibits better helpfulness. The quality of our coreference question responses is comparable to the direct question responses of Otter's base model, which we believe is a fair comparison.

As for technical novelty and contribution, we argue that our model design is far from trivial. Numerous approaches exist for incorporating gaze data, and our final choice was made after conducting a series of experiments detailed in the ablation study. To further elaborate on these aspects, we have included a detailed illustration of different gaze integration methods in Figure 13. Additionally, our work extends beyond model design, as we strive to establish a method for a new problem. The data annotation pipeline and gaze benchmark construction have not been explored by other works, making it difficult to claim that our work lacks novelty.

Reviewer comment (Evaluation):
How do the authors evaluate the interpretability? Several places in the paper mention interpretability; however, its evaluation is unclear given that such a keyword is being used in several different contexts of AI.

Author response:
**Interpretability**

We mentioned interpretability twice in our conclusion, as we believe integrating more human intention signals into the model can enhance mutual understanding between the model and the user. This means the model can better interpret the user's intent based on the additional modality, and the user can understand the model's specific response according to the pre-input gaze grounding signal. However, we acknowledge that using the term "interpretability" might lead to confusion with the traditional interpretation concept in machine learning. Therefore, we have decided to temporarily remove this term and reserve it for further evaluation and demonstration. We genuinely appreciate your valuable feedback on this matter.

Reviewer comment (Experiments):
The data collection section (4.1) lacks detailed information on the methodology and content of the dataset. Providing specific examples and clearer explanations would enhance comprehension.

Author response:
**Automatic data annotation process in Section 4.1**

We acknowledge that our original presentation of this process was not sufficiently clear, making it challenging for readers to follow. In the revised version, and in accordance with your recommendations, we have provided detailed formulations and explanations of the annotation process alongside Figure 3. We have also included final annotated data examples in Figure 11 to enhance comprehension. Furthermore, to facilitate better viewing, we have replaced the original PNG figure with a PDF format, allowing readers to zoom in as needed.

Reviewer comment (Presentation):
Figure 3 needs a caption and more in-depth explanations to convey its intended message.

Author response:
**Automatic data annotation process in Section 4.1**

We acknowledge that our original presentation of this process was not sufficiently clear, making it challenging for readers to follow. In the revised version, and in accordance with your recommendations, we have provided detailed formulations and explanations of the annotation process alongside Figure 3. We have also included final annotated data examples in Figure 11 to enhance comprehension. Furthermore, to facilitate better viewing, we have replaced the original PNG figure with a PDF format, allowing readers to zoom in as needed.

Reviewer comment (Presentation):
The figures need higher resolution for better readability when printed.

Author response:
**Automatic data annotation process in Section 4.1**

We acknowledge that our original presentation of this process was not sufficiently clear, making it challenging for readers to follow. In the revised version, and in accordance with your recommendations, we have provided detailed formulations and explanations of the annotation process alongside Figure 3. We have also included final annotated data examples in Figure 11 to enhance comprehension. Furthermore, to facilitate better viewing, we have replaced the original PNG figure with a PDF format, allowing readers to zoom in as needed.

Reviewer comment (Reproducibility):
It is unclear how the 'Voila Perceiver Resampler' module is integrated with the VLLM.

Author response:
**Voila Perceiver and Resampler Explanation**: 

In the revised version, we have provided a clearer architectural overview to better illustrate how the Voila Perceiver and Resampler components are integrated within the VLM framework. Additionally, we have marked the different fine-tuning stages on the structure to clarify which components are tuned in each phase. We acknowledge that the original terms `gaze linear` and `gaze attention` were not clear, and we have revised the text to specify that they refer to the components tuned during the first stage, without introducing any new formulations. Furthermore, we have rewritten the introductory part of the method section to provide more context on our base structure and input, ensuring a more comprehensive understanding for our readers.

Reviewer comment (Writing):
In Section 4.3, the meaning and specifics of 'gaze linear layer' and 'gaze key linear of attention' need clarification, including which layers these terms refer to and whether there is a specific formula involved.

Author response:
**Voila Perceiver and Resampler Explanation**: 

In the revised version, we have provided a clearer architectural overview to better illustrate how the Voila Perceiver and Resampler components are integrated within the VLM framework. Additionally, we have marked the different fine-tuning stages on the structure to clarify which components are tuned in each phase. We acknowledge that the original terms `gaze linear` and `gaze attention` were not clear, and we have revised the text to specify that they refer to the components tuned during the first stage, without introducing any new formulations. Furthermore, we have rewritten the introductory part of the method section to provide more context on our base structure and input, ensuring a more comprehensive understanding for our readers.

Reviewer comment (Presentation):
Merging Section 5.1.2 with Section 4.1 would improve the overall clarity of the paper.

Author response:
**Other Issues**

1. We have merged Section 5.1.2 with Section 4.1 to streamline the presentation of the related content and improve the overall flow of the manuscript.

Reviewer comment (Evaluation):
The claim that Voila exhibits a 'superior ability to handle multi-turn real conversations' in the last sentence of Section 5.3 needs stronger support or clarification in the results section.

Author response:
**Other Issues**

2. Regarding the phrase "superior ability to handle multi-turn real conversations," we have updated the statement to clarify its meaning and avoid any potential misunderstandings. The updated statement emphasizes that when appending more in-context conversations, our method's performance continues to improve compared to single-turn baselines. This change ensures that readers have a more accurate understanding of the method's capabilities.

Reviewer comment (Presentation):
A clearer depiction of how the ViolapercieverBlock and Resampler are integrated within the larger VLLM framework is needed; a simplified architectural overview would help in understanding the bigger picture.

Author response:
**Voila Perceiver and Resampler Explanation**: 

In the revised version, we have provided a clearer architectural overview to better illustrate how the Voila Perceiver and Resampler components are integrated within the VLM framework. Additionally, we have marked the different fine-tuning stages on the structure to clarify which components are tuned in each phase. We acknowledge that the original terms `gaze linear` and `gaze attention` were not clear, and we have revised the text to specify that they refer to the components tuned during the first stage, without introducing any new formulations. Furthermore, we have rewritten the introductory part of the method section to provide more context on our base structure and input, ensuring a more comprehensive understanding for our readers.

Reviewer comment (Reproducibility):
More details are needed on what exactly is included in the automatic data annotation process and how it is carried out, with specific examples provided in the main paper.

Author response:
**Automatic data annotation process in Section 4.1**

We acknowledge that our original presentation of this process was not sufficiently clear, making it challenging for readers to follow. In the revised version, and in accordance with your recommendations, we have provided detailed formulations and explanations of the annotation process alongside Figure 3. We have also included final annotated data examples in Figure 11 to enhance comprehension. Furthermore, to facilitate better viewing, we have replaced the original PNG figure with a PDF format, allowing readers to zoom in as needed.

Reviewer comment (Evaluation):
For Figures 4 and 5, additional guidance on how to interpret the results is needed, specifically clarification on what constitutes the 'Overall score' and a detailed explanation of how the 'Helpfulness' and 'Grounding score' are calculated.

Author response:
**Interpretation of Figures 4 and 5**:

 In the revised manuscript, we have provided additional insights on interpreting the results, along with explanations for the 'Overall score,' 'Helpfulness,' and 'Grounding score,' which can be found in Figure 12 in the appendix. The primary distinction between these scores is as follows:

1. For Helpfulness, the ground truth answer is provided as a reference, allowing GPT-4 to determine which model's output effectively addresses the user's question.
2. For Grounding, the original captions from COCO and Localized Narratives are given, enabling GPT-4 to decide which one aligns better with the actual content of the image.
3. The Overall score involves both ground truth captions and answers as input, allowing GPT-4 to evaluate which of the two models' answers is superior in general.

Reviewer comment (Evaluation):
Clarification of any specific limitations or potential challenges associated with the proposed approach is needed.

Author response:
**Conclusion section**: 

In the revised conclusion section, we have incorporated a discussion on potential applications, future directions, and limitations of our method. Specifically, our approach can be employed in HMD AR/VR devices to serve as an egoview copilot for everyone, including visually impaired individuals who may not have clear vision but can still use their gaze to convey their intent. Our method offers greater applicability than similar mobile apps that require users to raise their phones to capture a scene for assistance.

The limitations and future directions include:

1. Improving inference efficiency to enable instantaneous responses.
2. Incorporating voice modalities to facilitate seamless interaction.
3. Supporting higher resolutions to accommodate scenarios requiring OCR and screen or UI understanding.

Additionally, we have addressed the hallucination problem and discrepancies with GPT-4V in the appendix. By discussing these aspects, we aim to provide a comprehensive understanding of our method's potential and areas for future improvement.

Reviewer comment (Novelty):
Technical or scientific contribution is very incremental and limited.

Author response:
**Technical Contribution** 

We respectfully disagree with your assessment of our work's technical valuation and contribution, and we would like to provide the following reasons for your reconsideration:

1. *Motivation:* As discussed in the revised version, although there is existing research on gaze-related user interfaces and gaze behavior modeling such as saliency models, there has been no investigation into integrating gaze data into modern models and applications. Our work pioneers this area by providing a scalable solution to address this challenge, paving the way for further advancements with high potential.

2. *Effort and Innovation:* We are the first to define a new gaze-enhanced assistance task that is crucial for the upcoming era of spatial computing. We have invested considerable effort in building a reliable automated data annotation pipeline and providing guidance on potential issues that may arise during this process. Additionally, we have developed a comprehensive solution for scalable training and evaluation of models, established new benchmarks using real-life user data, and offered various insights into model design and training through extensive ablation studies. The work presented is far from trivial and should be recognized as a significant contribution.

3. *Contributions to the Research Community:* We plan to release both datasets, the automated data annotation pipeline, pretrained model weights, and training code. Given the challenges in obtaining gaze collection equipment, invoking massive APIs for automated annotation, and fine-tuning and evaluating comprehensive ablations on models with billions of parameters, we believe the resources we are providing will be valuable and directly enhance future research in this area.

**In light of the points mentioned above, we kindly request that Reviewer 482q reconsider the rating for this work. We also welcome any further questions or suggestions you may have.**

Reviewer comment (Writing):
Writing can be improved; not always easy to follow and clear.

Author response:
**Writing and Presentation:** 

In response to all reviewers' suggestions, we have extensively reorganized, clarified, and rewritten the manuscript, as well as expanded the material in the revised version as we replied in general response. We hope that these changes address your concerns effectively. We remain open to further suggestions and would appreciate any additional feedback on areas that may require improvement or clarification.

Reviewer comment (Experiments):
Baseline methods considered are not comprehensive or fair. Mouse trace data as a proxy for gaze sounds reasonable but there are many off-the-shelf saliency model that are designed to mimic human gaze. Some of the existing saliency model can be used or at least need to be discussed and reviewed in the paper.

Author response:
**Baseline Methods:**
 
We are grateful for your suggestion to discuss saliency models in our paper. In the revised version, we have included a dedicated discussion of saliency models in Section G.4 and clarified their relationship with our proposed method. However, we would like to emphasize that saliency models, which predict human gaze fixation from images, cannot be directly compared to or used in our approach as baselines. In our task, the ground truth gaze, image, and user query are used as inputs to generate a response, which is a different problem setting.

The most relevant work in this context involves using gaze as input for generating captions, such as the work by Sugano and Bulling [1]. However, these methods predate the era of large-scale language models and are designed for single-purpose tasks like captioning. As they have not been pretrained on massive datasets, comparing their performance to our modern approach would be unfair, as their performance is undoubtedly lower than ours.

Instead, we have conducted an ablation study to investigate different aspects of the model design space, such as input types, gaze integration methods in VLMs, tuning techniques, and data variations. We believe that these analyses provide a more meaningful baseline for comparison and offer valuable insights into the impact of various design choices on performance. And we have update a clear illustration of those baselines in Figure 13, it may be helpful for you to check.

[1] Yusuke Sugano and Andreas Bulling. Seeing with humans: Gaze-assisted neural image captioning. ArXiv, abs/1608.05203, 2016.

Reviewer comment (Experiments):
Please address the comments above regarding baseline.

Author response:
**Baseline Methods:**
 
We are grateful for your suggestion to discuss saliency models in our paper. In the revised version, we have included a dedicated discussion of saliency models in Section G.4 and clarified their relationship with our proposed method. However, we would like to emphasize that saliency models, which predict human gaze fixation from images, cannot be directly compared to or used in our approach as baselines. In our task, the ground truth gaze, image, and user query are used as inputs to generate a response, which is a different problem setting.

The most relevant work in this context involves using gaze as input for generating captions, such as the work by Sugano and Bulling [1]. However, these methods predate the era of large-scale language models and are designed for single-purpose tasks like captioning. As they have not been pretrained on massive datasets, comparing their performance to our modern approach would be unfair, as their performance is undoubtedly lower than ours.

Instead, we have conducted an ablation study to investigate different aspects of the model design space, such as input types, gaze integration methods in VLMs, tuning techniques, and data variations. We believe that these analyses provide a more meaningful baseline for comparison and offer valuable insights into the impact of various design choices on performance. And we have update a clear illustration of those baselines in Figure 13, it may be helpful for you to check.

[1] Yusuke Sugano and Andreas Bulling. Seeing with humans: Gaze-assisted neural image captioning. ArXiv, abs/1608.05203, 2016.

Reviewer comment (Reproducibility):
Will the data set VOILA-COCO be released? I did not see this information in the paper.

Author response:
We appreciate your recognition of our work's ideas and experiments. We would like to address your concerns as follows:

**VOILA-COCO dataset release**
We thank the reviewers' interest in the VOILA-COCO and VOILA-GAZE datasets. In response to their inquiries, we would like to confirm our intention to **release both two datasets** upon the publication of our work. Our aim is to foster further research and collaboration within the community by providing these valuable resources. In addition to the datasets, we will also make our annotation pipeline, training code, and model weights publicly available, ensuring that our work is transparent and easily reproducible. To address this concern explicitly, we have included a reproducibility statement in the revised manuscript. We hope that these efforts will contribute positively to the field and facilitate the development of new ideas and methods.

Reviewer comment (Novelty):
Using mouse trace to approximate human gaze/attention is a popular approach in attention related area, however, the authors does not mention existing works like BubbleView https://bubbleview.namwkim.org/ or Salicon http://salicon.net/

Author response:
**Existing works on mouse trace approximation**

We sincerely appreciate your suggestion to consider the relevant background literature on mouse trace approximation. It is encouraging to discover that, despite differing motivations and experimental settings, these works provide robust support for some of our hypotheses. In response to your feedback, we have conducted a comprehensive survey and included an overarching discussion in Section 2, as well as detailed analyses from various perspectives in Appendices G.3 and G.4.

In our revised manuscript, we have incorporated references to prominent works such as SALICON and BubbleView, and provided a comparative analysis to elucidate the connections and distinctions between our approach and these existing methods. We believe this addition strengthens our paper and offers valuable context for readers. Thank you once again for your valuable input.

Reviewer comment (Reproducibility):
The dataset might be an important contribution of this paper. However, it is not clear whether the data set will be released or not.

Author response:
We appreciate your recognition of our work's ideas and experiments. We would like to address your concerns as follows:

**VOILA-COCO dataset release**
We thank the reviewers' interest in the VOILA-COCO and VOILA-GAZE datasets. In response to their inquiries, we would like to confirm our intention to **release both two datasets** upon the publication of our work. Our aim is to foster further research and collaboration within the community by providing these valuable resources. In addition to the datasets, we will also make our annotation pipeline, training code, and model weights publicly available, ensuring that our work is transparent and easily reproducible. To address this concern explicitly, we have included a reproducibility statement in the revised manuscript. We hope that these efforts will contribute positively to the field and facilitate the development of new ideas and methods.